# Hadoop Tutorial - Efficient data ingestion

- Institution: CERN
- Department: Workshops
- Taught by: Daniel Lanza Garcia, Zbigniew Baranowski
- Year: 2016
- Language: English
- Source: [CERN](https://videos.cern.ch/record/3002588)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Teens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

The Hadoop ecosystem is the leading opensource platform for distributed storage and processing of "big data". The Hadoop platform is available at CERN as a central service provided by the IT department.

Real-time data ingestion to Hadoop ecosystem due to the system specificity is non-trivial process and requires some efforts (which is often underestimated) in order to make it efficient (low latency, optimize data placement, footprint on the cluster).

In this tutorial attendees will learn about:

The important aspects of storing the data in Hadoop Distributed File System (HDFS).

Data ingestion techniques and engines that are capable of shipping data to Hadoop in an efficient way.

Setting up a full data ingestion flow into a Hadoop Distributed Files System from various sources (streaming, log files, databases) using the best practices and components available around the ecosystem (including Sqoop, Kite, Flume, Kafka).

.

0:00:00 Slide 1

0:00:22 Slide 2

0:00:44 Slide 3

0:03:02 Slide 4

0:03:09 Slide 5

0:03:17 Slide 6

0:03:35 Slide 7

0:03:39 Slide 8

0:03:50 Slide 9

0:04:41 Slide 10

0:05:15 Slide 11

0:05:53 Slide 12

0:06:51 Slide 13

0:07:55 Slide 14

0:09:56 Slide 15

0:11:20 Slide 16

0:15:00 Slide 17

0:15:05 Slide 18

0:15:10 Slide 19

0:15:20 Slide 20

0:15:25 Slide 21

0:15:29 Slide 22

0:15:36 Slide 23

0:15:42 Slide 24

0:15:46 Slide 25

0:15:50 Slide 26

0:15:55 Slide 27

0:15:59 Slide 28

0:16:06 Slide 29

0:16:10 Slide 30

0:16:18 Slide 31

0:16:22 Slide 32

0:16:30 Slide 33

0:16:37 Slide 34

0:16:42 Slide 35

0:16:51 Slide 36

0:17:02 Slide 37

0:17:22 Slide 38

0:17:30 Slide 39

0:17:34 Slide 40

0:17:45 Slide 41

0:17:49 Slide 42

0:17:53 Slide 43

0:17:58 Slide 44

0:18:02 Slide 45

0:18:06 Slide 46

0:18:11 Slide 47

0:18:15 Slide 48

0:18:19 Slide 49

0:18:23 Slide 50

0:18:35 Slide 51

0:18:39 Slide 52

0:18:51 Slide 53

0:18:55 Slide 54

0:19:04 Slide 55

0:20:01 Slide 56

0:22:31 Slide 57

0:24:28 Slide 58

0:25:05 Slide 59

0:25:09…

## Lectures

1. [Hadoop Tutorial - Efficient data ingestion](https://invisiblenetwork.ai/learn/cern/3002588/b738e560f720), 2 h (CERN's player)