About
The Hadoop ecosystem is the leading opensource platform for distributed storage and processing of "big data". The Hadoop platform is available at CERN as a central service provided by the IT department.
Real-time data ingestion to Hadoop ecosystem due to the system specificity is non-trivial process and requires some efforts (which is often underestimated) in order to make it efficient (low latency, optimize data placement, footprint on the cluster).
In this tutorial attendees will learn about:
The important aspects of storing the data in Hadoop Distributed File System (HDFS).
Data ingestion techniques and engines that are capable of shipping data to Hadoop in an efficient way.
Setting up a full data ingestion flow into a Hadoop Distributed Files System from various sources (streaming, log files, databases) using the best practices and components available around the ecosystem (including Sqoop, Kite, Flume, Kafka).
.
0:00:00 Slide 1
0:00:22 Slide 2
0:00:44 Slide 3
0:03:02 Slide 4
0:03:09 Slide 5
The talk
Credits and licence
- Institution
- CERN
- Department
- Workshops
- Taught by
- Daniel Lanza Garcia, Zbigniew Baranowski
- Year
- 2016
- Language
- English
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Teens profiles and older, judged from what the source says about it. How courses are placed