# Cluster Management via Multi-Level Scheduling with Apache Mesos

- Institution: CERN
- Department: CERN Computing Seminar
- Speaker: Benjamin Hindman
- Year: 2015
- Language: English
- Subjects: Computer science
- Source: [CERN](https://videos.cern.ch/record/3003539)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

Existing research has shown the benefits of running multi-level schedulers, either for single node parallel computation or multi-node distributed computation. But, there are some important practical considerations that must be addressed in order to use these multi-level scheduling architectures in multi-user production environments. In this presentation we'll discuss these practical considerations through lessons learned deploying Apache Mesos, a 2-level distributed scheduling system that has been used in organizations such as Twitter, PayPal, and Apple. We'll first highlight the multi-level scheduling systems that influenced Mesos as well as describe the 2-level Mesos architecture in detail. We'll then focus on the 1st-level scheduler of Mesos and the efficient multi-resource fair-sharing algorithm that it employs. Finally, we'll discuss the extensions that have been added over the years (or are being added today) driven by practical needs, from weights, to reservations, to quotas, to optimistic allocations, and deallocation.

About the speaker

Benjamin Hindman is a Founder and Chief Architect at Mesosphere where he leads a team building out core services for the Mesosphere Datacenter Operating System (DCOS). Ben co-created Apache Mesos as a PhD student at UC Berkeley before bringing it to Twitter where it now runs on tens of thousands of machines powering Twitter's datacenters. An academic at heart, his research in programming languages and distributed systems has been published in leading academic conferences.

.

0:00:00 Slide 1

0:01:05 Slide 2

0:01:28 Slide 3

0:01:56 Slide 4

0:02:28 Slide 5

0:05:47 Slide 6

0:06:19 Slide 7

0:06:41 Slide 8

0:07:41 Slide 9

0:08:37 Slide 10

0:08:53 Slide 11

0:09:27 Slide 12

0:09:43 Slide 13

0:10:17 Slide 14

0:10:45 Slide 15

0:11:22 Slide 16

0:12:27 Slide 17

0:12:48 Slide 18

0:13:13 Slide 19

0:14:02 Slide 20

0:14:22 Slide 21

0:14:34 Slide 22

0:14:47 Slide 23

0:15:08 Slide 24

0:16:07 Slide 25

0:16:48 Slide…

## Lectures

1. [Cluster Management via Multi-Level Scheduling with Apache Mesos](https://invisiblenetwork.ai/learn/cern/3003539/e092abb9705c), 1 h 21 min (CERN's player)