# Big Data Technologies and Physics Analysis with Apache Spark (lecture 2)

- Institution: CERN
- Department: Inverted CSC
- Speaker: Evangelos Motesnitsalis
- Year: 2019
- Language: English
- Subjects: Computer science
- Source: [CERN](https://videos.cern.ch/record/3006487)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

The Large Hadron Collider is scheduled to shut down for a 2 years maintenance period since December 2018. However, the already collected data -which are stored in a dedicated custom storage service- between April 2015 and November 2018, exceed 150 PBs in total. To analyse these data, more and more teams at CERN decide to use Big Data Technologies to perform Physics Analysis and "Data Reduction", i.e. produce smaller reusable datasets for frequent access. These technologies show great potential in speeding up the existing procedures.

This lecture will provide an overview of the latest trending big data technologies in the Hadoop and Spark ecosystems with focus on their main architecture characteristics, and then will target a number of important questions: How can we perform Physics Analysis with Big Data Technologies? What are the problems faced? What are the challenges and the available data sources? What are the other domain in which Big Data Analytics are applied at CERN?.

0:00:00 Slide 1

0:00:24 Slide 2

0:01:09 Slide 3

0:02:53 Slide 4

0:03:09 Slide 5

0:05:11 Slide 6

0:05:15 Slide 7

0:05:54 Slide 8

0:06:06 Slide 9

0:07:30 Slide 10

0:07:48 Slide 11

0:08:03 Slide 12

0:08:45 Slide 13

0:09:17 Slide 14

0:10:57 Slide 15

0:11:03 Slide 16

0:11:28 Slide 17

0:11:32 Slide 18

0:11:40 Slide 19

0:11:58 Slide 20

0:12:06 Slide 21

0:12:18 Slide 22

0:12:22 Slide 23

0:12:33 Slide 24

0:13:26 Slide 25

0:13:34 Slide 26

0:13:40 Slide 27

0:13:54 Slide 28

0:14:00 Slide 29

0:14:04 Slide 30

0:14:24 Slide 31

0:14:45 Slide 32

0:14:59 Slide 33

0:15:12 Slide 34

0:15:16 Slide 35

0:16:08 Slide 36

0:16:23 Slide 37

0:16:50 Slide 38

0:17:49 Slide 39

0:18:20 Slide 40

0:18:24 Slide 41

0:21:54 Slide 42

0:22:36 Slide 43

0:23:13 Slide 44

0:23:43 Slide 45

0:23:49 Slide 46

0:23:56 Slide 47

0:24:11 Slide 48

0:30:38 Slide 49

0:31:03 Slide 50

0:31:07 Slide 51

0:31:40 Slide 52

0:32:58 Slide 53

0:33:04 Slide 54

0:33:41 Slide 55

0:35:15 Slide 56…

## Lectures

1. [Big Data Technologies and Physics Analysis with Apache Spark (lecture 2)](https://invisiblenetwork.ai/learn/cern/3006487/eab31229ebb7), 43 min (CERN's player)