Next generation WLCG File Transfer Service (FTS)

About

LHC experiments at CERN and worldwide utilize WLCG resources and middleware components to perform distributed computing tasks. One of the most important tasks is reliable file replication. It is a complex problem, suffering from transfer failures, disconnections, transfer duplication, server and network overload, differences in storage systems, etc. To address these problems, EMI and gLite have provided the independent File Transfer Service (FTS) and Grid File Access Library (GFAL) tools. Their development started almost a decade ago, in the meantime, requirements in data management have changed - the old architecture of FTS and GFAL cannot keep support easily these changes. Technology has also been progressing: FTS and GFAL do not fit into the new paradigms (cloud, messaging, for example).

To be able to serve the next stage of LHC data collecting (from 2013), we need a new generation of these tools: FTS 3 and GFAL 2. We envision a service requiring minimal configuration, which can dynamically adapt to the state of its resources (endpoints (SE-s) and network) and which offers an SE-centric configuration model for administrators to use where necessary.

Main problems that we solve: scalability problems of the static channel model, resource and network states are not taken into account when scheduling transfer jobs, supporting multiple database backends (Oracle, MySQL), multiple transfer and control protocols. The new system will be much easier to configure and manage. FTS 3 will provide transparent job submission and monitoring features based on messaging. We report on design and prototype experience. We provide an overview about how FTS fits into the more general suite of data management utilities provided by the EMI project..

0:00:00 Slide 1

0:01:21 Slide 2

0:02:41 Slide 3

0:05:22 Slide 4

0:08:03 Slide 5

0:10:43 Slide 6

0:13:24 Slide 7

0:16:05 Slide 8

0:18:45 Slide 9