Dimensioning storage and computing clusters for efficient High Throughput Computing

About

Scientific experiments are producing huge amounts of data, and they continue increasing the size of their datasets and the total volume of data. These data are then processed by researchers belonging to large scientific collaborations, with the Large Hadron Collider being a good example. The focal point of Scientific Data Centres has shifted from coping efficiently with PetaByte scale storage to deliver quality data processing throughput. The dimensioning of the internal components in High Throughput Computing (HTC) data centers is of crucial importance to cope with all the activities demanded by the experiments, both the online (data acceptance) and the offline (data processing, simulation and user analysis). This requires a precise setup involving disk and tape storage services, a computing cluster and the internal networking to prevent bottlenecks, overloads and undesired slowness that lead to losses cpu cycles and batch jobs failures. In this paper we point out relevant features for running a successful storage setup in an intensive HTC environment.

0:00:00 Slide 1

0:01:05 Slide 2

0:02:09 Slide 3

0:04:17 Slide 4

0:06:25 Slide 5

0:08:33 Slide 6

0:10:42 Slide 7

0:12:50 Slide 8

0:14:58 Slide 9

0:17:06 Slide 10

0:19:14 Slide 11