Personalized Opportunistic Computing for CMS at Large Scale

About

In recent years, the WLCG has evolved toward a model of model of centralized scheduling of largely dedicated clusters provided by each member institution, missing some opportunities to harness opportunistically available resources. This talk will present our recent experience in building Lobster, an opportunistic workload management system for the elements of the CMS collaboration at Notre Dame, where our prototype system scavenging cycles from a Tier-3 center runs on O(10K) cores, which is on the scale of the largest Tier-2 centers. I will discuss the challenges inherent in this environment, explain how Parrot, CVMFS, and Work Queue have evolved to meet these needs, and suggest ways to exploit emerging technologies such as containers within the context of scientific workflows.

About the author. **Douglas Thain** is an Associate Professor of Computer Science and Engineering at the University of Notre Dame, where he designs large scale distributed computing systems to power the needs of advanced science and engineering research in fields such as bioinformatics, high energy physics, and molecular dynamics. His team publishes the Cooperating Computing Tools, a collection of open source software that provides application frameworks, workflow systems, and data access tools for clusters, clouds, and grids. Prof. Thain received the Ph.D. from the University of Wisconsin, where he contributed to the HTCondor high-throughput computing system..

0:00:00 Slide 1

0:01:24 Slide 2

0:01:47 Slide 3

0:02:50 Slide 4

0:04:28 Slide 5

0:04:32 Slide 6

0:06:02 Slide 7

0:06:06 Slide 8

0:06:12 Slide 9

0:06:27 Slide 10