# Personalized Opportunistic Computing for CMS at Large Scale

- Institution: CERN
- Department: General
- Speaker: Douglas Prof. THAIN
- Year: 2015
- Language: English
- Subjects: Computer science
- Source: [CERN](https://videos.cern.ch/record/3001579)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

In recent years, the WLCG has evolved toward a model of model of centralized scheduling of largely dedicated clusters provided by each member institution, missing some opportunities to harness opportunistically available resources. This talk will present our recent experience in building Lobster, an opportunistic workload management system for the elements of the CMS collaboration at Notre Dame, where our prototype system scavenging cycles from a Tier-3 center runs on O(10K) cores, which is on the scale of the largest Tier-2 centers. I will discuss the challenges inherent in this environment, explain how Parrot, CVMFS, and Work Queue have evolved to meet these needs, and suggest ways to exploit emerging technologies such as containers within the context of scientific workflows.

About the author. **Douglas Thain** is an Associate Professor of Computer Science and Engineering at the University of Notre Dame, where he designs large scale distributed computing systems to power the needs of advanced science and engineering research in fields such as bioinformatics, high energy physics, and molecular dynamics. His team publishes the Cooperating Computing Tools, a collection of open source software that provides application frameworks, workflow systems, and data access tools for clusters, clouds, and grids. Prof. Thain received the Ph.D. from the University of Wisconsin, where he contributed to the HTCondor high-throughput computing system..

0:00:00 Slide 1

0:01:24 Slide 2

0:01:47 Slide 3

0:02:50 Slide 4

0:04:28 Slide 5

0:04:32 Slide 6

0:06:02 Slide 7

0:06:06 Slide 8

0:06:12 Slide 9

0:06:27 Slide 10

0:07:53 Slide 11

0:08:31 Slide 12

0:09:03 Slide 13

0:09:09 Slide 14

0:10:09 Slide 15

0:10:32 Slide 16

0:10:45 Slide 17

0:10:51 Slide 18

0:10:55 Slide 19

0:11:56 Slide 20

0:13:22 Slide 21

0:15:50 Slide 22

0:17:14 Slide 23

0:18:57 Slide 24

0:19:24 Slide 25

0:19:29 Slide 26

0:20:37 Slide 27

0:21:51 Slide 28

0:22:35 Slide 29

0:22:58 Slide 30…

## Lectures

1. [Personalized Opportunistic Computing for CMS at Large Scale](https://invisiblenetwork.ai/learn/cern/3001579/c62e942a4002), 59 min (CERN's player)