About
**Project Description**: The Hadoop infrastructure at CERN consists of many services (like HDFS, YARN, Spark, ZooKeeper) that all collect logging data in their own files. These services are used to store large amounts of data (from experiments and other sources) and process it efficiently. Before, the logs were located on each server individually, this made looking for (the source of) errors difficult. The goal of the project was to centralise the collection of logging data across all services using Fluent Bit (deployed through Puppet and routing through CERN MONIT infrastructure). We use OpenSearch as a storage backend and visualise the data in Grafana dashboards. It will help Hadoop system administrators investigate incidents faster and detect problematic service behaviour before it becomes a problem.
**Supervisor**: Luis Pigueiras
00:00:00 Slide 1
00:00:02 Slide 2
00:00:16 Slide 3
00:00:46 Slide 4
00:01:16 Slide 5
00:01:35 Slide 6
00:02:00 Slide 7
00:02:18 Slide 8
00:02:42 Slide 9
00:03:17 Slide 10
The talk
Credits and licence
- Institution
- CERN
- Speaker
- Jordie Biemold
- Year
- 2026
- Language
- English
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Teens profiles and older, judged from what the source says about it. How courses are placed