Efficient Workflow Management in High-Energy Physics

About

In high-energy physics (HEP), efficient workflow management is crucial for processing large datasets, running simulations, and managing computational jobs across distributed environments. This lecture introduces Luigi, a workflow management tool originally developed at Spotify that helps automate and scale complex task pipelines, ensuring dependency resolution and fault tolerance.

Building on Luigi, Law (**L**uigi **a**nalysis **w**orkflow) provides additional abstractions for HEP workflows by incorporating diverse batch job submission systems, like HTCondor, and different execution environments. Additionally, the automatic management of accessing distributed storage locations using the standard WLCG transfer protocols (e.g., WebDAV and XRootD) enables seamless execution of tasks across remote computing resources and ensures efficient resource utilization, simplifying the life of physicists a lot.

The lecture will provide both conceptual insights and practical approaches for managing large-scale workflows in HEP, leveraging modern tools and distributed computing infrastructure to enhance scientific computing. Specifically, it will cover how Luigi and law can be used to construct robust and scalable workflows for HEP applications, from running jobs on distributed batch systems to automatically managing remote data access to conduct complex analyses..

0:00:00 Slide 1

0:01:09 Slide 2

0:02:30 Slide 3

0:03:46 Slide 4

0:05:34 Slide 5

0:07:00 Slide 6

0:09:01 Slide 7

0:12:29 Slide 8

0:13:54 Slide 9