About
In today's research landscape, managing and processing a high volume of data has become crucial in many fields. Many researchers make use of remote computing resources to process large data volumes. High-volume data transfers between research institutions and High-Performance Computing Clusters (HPCC) have thus increased in importance, as large data sets can require hours or days to transmit in non-optimized settings. Moreover, adhering to security compliance requires the use of firewalls, which are often costly and / or slow down data transfers.
To resolve these issues, we have developed Hercules and LightningFilter, which make use of the SCION next-generation Internet to achieve security and efficiency. The Hercules data transfer application provides a high-speed implementation, offering sustained transmission and reception speeds of around 100 Gbps including reliable delivery as well congestion control. LightningFilter is an open-source firewall implementation, which can process minimum-sized packets in excess of 100 Gbps on a standard mid-range server. LightningFilter can satisfy firewall compliance rules, and enables ASes to cryptographically verify, restrict, and police the incoming connections, whether from other ASes or specific hosts, allowing the HPCC to implement distinct rate limits for different universities while ensuring a guaranteed throughput for particular hosts. The open-source implementation of these tools facilitates a low-cost yet high-performance file transfer service.
A key component of this architecture is the deployment of data transmission nodes, which play a crucial role in optimizing data flow. These nodes, strategically positioned within the network, facilitate high-speed data transfers and ensure reliable connectivity between the HPCC and researches.
A new development in this infrastructure is the integration of Hercules with the File Transfer Service (FTS) through the gfal2 library. This integration streamlines the data transfer…
The talk
Credits and licence
- Institution
- CERN
- Department
- HEP Computing
- Speaker
- Francois Wirz
- Year
- 2025
- Language
- English
- Subjects
- Computer science
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Teens profiles and older, judged from what the source says about it. How courses are placed