# Efficient Workflow Management in High-Energy Physics

- Institution: CERN
- Department: Inverted CSC
- Speaker: Cedric Verstege
- Year: 2025
- Language: English
- Subjects: Computer science
- Source: [CERN](https://videos.cern.ch/record/3022174)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

In high-energy physics (HEP), efficient workflow management is crucial for processing large datasets, running simulations, and managing computational jobs across distributed environments. This lecture introduces Luigi, a workflow management tool originally developed at Spotify that helps automate and scale complex task pipelines, ensuring dependency resolution and fault tolerance.

Building on Luigi, Law (**L**uigi **a**nalysis **w**orkflow) provides additional abstractions for HEP workflows by incorporating diverse batch job submission systems, like HTCondor, and different execution environments. Additionally, the automatic management of accessing distributed storage locations using the standard WLCG transfer protocols (e.g., WebDAV and XRootD) enables seamless execution of tasks across remote computing resources and ensures efficient resource utilization, simplifying the life of physicists a lot.

The lecture will provide both conceptual insights and practical approaches for managing large-scale workflows in HEP, leveraging modern tools and distributed computing infrastructure to enhance scientific computing. Specifically, it will cover how Luigi and law can be used to construct robust and scalable workflows for HEP applications, from running jobs on distributed batch systems to automatically managing remote data access to conduct complex analyses..

0:00:00 Slide 1

0:01:09 Slide 2

0:02:30 Slide 3

0:03:46 Slide 4

0:05:34 Slide 5

0:07:00 Slide 6

0:09:01 Slide 7

0:12:29 Slide 8

0:13:54 Slide 9

0:18:50 Slide 10

0:21:05 Slide 11

0:22:26 Slide 12

0:23:50 Slide 13

0:25:03 Slide 14

0:26:13 Slide 15

0:27:25 Slide 16

0:29:54 Slide 17

0:31:18 Slide 18

0:32:31 Slide 19

0:34:02 Slide 20

0:35:51 Slide 21

0:37:57 Slide 22

0:39:33 Slide 23

0:43:06 Slide 24

0:46:31 Slide 25

0:48:09 Slide 26

0:49:19 Slide 27

## Lectures

1. [Efficient Workflow Management in High-Energy Physics](https://invisiblenetwork.ai/learn/cern/3022174/e9e0850db102), 51 min (CERN's player)