# Apache Airflow POC for Data Reconciliation

- Institution: CERN
- Speaker: Hung Manh Nguyen
- Year: 2026
- Language: English
- Source: [CERN](https://videos.cern.ch/record/3026657)
- Licence: [CERN](https://copyright.web.cern)
- Who can watch: Teens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

**Project Description**: CERN maintains file metadata across Rucio, EOS, and the CERN Tape Archive (CTA), making periodic reconciliation necessary to detect inconsistencies such as dark or missing data. This project investigated Apache Airflow as an alternative to Rundeck for orchestrating the reconciliation workflow. A proof-of-concept Airflow deployment was implemented on CERN's Kubernetes infrastructure using Argo CD, Git-synced DAGs, CERN S3, and temporary PostgreSQL storage. Overall, a representative EOS–CTA reconciliation DAG was developed based on the Rundeck implementations. The results show that Airflow can represent the workflow's dependencies, parallel processing, validation, and cleanup stages clearly while providing centralized execution monitoring, although this comes with additional deployment and operational complexity that must be considered before production adoption. **Supervisor**: Idriss Larbi

00:00:00 Slide 1

00:00:21 Slide 2

00:00:42 Slide 3

00:00:56 Slide 4

00:01:19 Slide 5

00:01:37 Slide 6

00:02:09 Slide 7

00:02:26 Slide 8

00:02:54 Slide 9

00:03:22 Slide 10

00:03:53 Slide 11

00:04:09 Slide 12

00:04:27 Slide 13

00:04:45 Slide 14

00:05:03 Slide 15

00:05:19 Slide 16

## Lectures

1. [Apache Airflow POC for Data Reconciliation](https://invisiblenetwork.ai/learn/cern/3026657/80022b0b99e2), 6 min (CERN's player)