# Implementation of a modern logging pipeline for Hadoop

- Institution: CERN
- Speaker: Jordie Biemold
- Year: 2026
- Language: English
- Source: [CERN](https://videos.cern.ch/record/3026682)
- Licence: [CERN](https://copyright.web.cern)
- Who can watch: Teens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

**Project Description**: The Hadoop infrastructure at CERN consists of many services (like HDFS, YARN, Spark, ZooKeeper) that all collect logging data in their own files. These services are used to store large amounts of data (from experiments and other sources) and process it efficiently. Before, the logs were located on each server individually, this made looking for (the source of) errors difficult. The goal of the project was to centralise the collection of logging data across all services using Fluent Bit (deployed through Puppet and routing through CERN MONIT infrastructure). We use OpenSearch as a storage backend and visualise the data in Grafana dashboards. It will help Hadoop system administrators investigate incidents faster and detect problematic service behaviour before it becomes a problem.

**Supervisor**: Luis Pigueiras

00:00:00 Slide 1

00:00:02 Slide 2

00:00:16 Slide 3

00:00:46 Slide 4

00:01:16 Slide 5

00:01:35 Slide 6

00:02:00 Slide 7

00:02:18 Slide 8

00:02:42 Slide 9

00:03:17 Slide 10

00:03:40 Slide 11

00:03:52 Slide 12

00:04:07 Slide 13

00:04:33 Slide 14

## Lectures

1. [Implementation of a modern logging pipeline for Hadoop](https://invisiblenetwork.ai/learn/cern/3026682/9433be6c361b), 7 min (CERN's player)