# “Big Data In HEP” - Physics Data Analysis, Machine learning and Data Reduction at Scale with Apache Spark

- Institution: CERN
- Department: other events or meetings
- Speaker: Luca Canali
- Year: 2019
- Language: English
- Subjects: Mathematics, Physics, Computer science
- Source: [CERN](https://videos.cern.ch/record/3002562)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

The field of High Energy Physics is approaching an era were excellent performance of particle accelerators delivers enormous numbers of collisions. The growing size of these datasets could potentially become a limiting factor in the capability to produce scientific results. "Big Data" technologies developed and optimized in industry could help analyzing Petabyte and Exabyte size datasets and enable the next big discoveries.

In this talk, we present the CERN openlab/Intel project to enable LHC-style analysis on Apache Spark at scale: "The CMS Big Data Reduction Facility". The goal was to develop the technical capabilities to provide the physics analyst with a data reduction facility. Working together with CERN Openlab and Intel, CMS replicated a real physics analysis using Spark-based technologies, with the ambition of reducing 1 Petabyte of CMS data in 5 hours to 1 Terabyte directly suitable for final analysis. We will present scaling results and facility improvements achieved by using Intel's CoFluent optimization tool.

We will also discuss how the tools and methods developed in the CERN openlab project and in collaboration with Intel, have allowed to develop an end-to-end data and pipeline for a deep learning research work of interest in High Energy Physics (HEP), in particular applied to improving the accuracy of online event filtering. Apache Spark has been used for the data lifting part of the pipeline, Spark with Analytics Zoo and BigDL have been used to run parallel training of the neural networks on CPU clusters..

0:00:00 Slide 1

0:00:22 Slide 2

0:01:03 Slide 3

0:01:34 Slide 4

0:01:54 Slide 5

0:02:31 Slide 6

0:03:27 Slide 7

0:03:44 Slide 8

0:03:57 Slide 9

0:04:10 Slide 10

0:05:01 Slide 11

0:06:03 Slide 12

0:07:12 Slide 13

0:09:09 Slide 14

0:10:10 Slide 15

0:12:06 Slide 16

0:12:27 Slide 17

0:12:37 Slide 18

0:13:31 Slide 19

0:14:18 Slide 20

0:15:00 Slide 21

0:15:39 Slide 22

0:16:21 Slide 23

0:16:25 Slide 24

0:16:29 Slide 25…

## Lectures

1. [“Big Data In HEP” - Physics Data Analysis, Machine learning and Data Reduction at Scale with Apache Spark](https://invisiblenetwork.ai/learn/cern/3002562/7cc2638e2418), 30 min (CERN's player)