About
Amazon's Simple Storage Service (S3) protocol has become a de-facto standard for object storage, and academic supercomputing centres are increasingly interested in offering S3-compatible storage solutions. This is on the one hand due to more (on-premises) cloud computing at such centres, and on the other hand due to the increasing demands of flexible data access and sharing across computing sites. Certainly, the movement towards FAIR (Findable, Accessible, Interoperable, Reusable) Research Data Management plays a significant role here.
The EXA4MIND Horizon Europe project aims at offering optimised storage back-ends, including object storage and databases, for Extreme-Data applications at European supercomputing centres. As part of this effort, we have conducted performance tests of different open-source middleware alternatives to implement S3 storage. We have performed these benchmarking experiments on bare-metal, virtual-machine and High-Performance Computing (HPC) systems.
We show first results using widely-used open-source S3 middleware such as MinIO, S3Proxy, and VersityGW. It appears that a combination of such middleware and optimised client software, making use of multiple parallel streams, is able to saturate network bandwidths up to several GB/s, possibly enabling direct object-store usage from HPC codes. Typical performance-limiting factors include I/O based on small files in the 100 MB range or below, and a too strongly restricted number of streams.
Acknowledgement: This work received support from the EXA4MIND project ("EXtreme Analytics for MINing Data spaces"), funded by the European Union's Horizon Europe Research and Innovation Programme, under Grant Agreement N° 101092944. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them.
00:00:00 Slide 1
00:00:06 Slide…
The talk
Credits and licence
- Institution
- CERN
- Speaker
- Huseyn Gurbanov
- Year
- 2026
- Language
- English
- Subjects
- Computer science
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed