About
S3 protocol is the de-facto standard data protocol in distributed storage and cloud computing with its wide usage for cloud back-ends, back-up and long-term storage. It is also being adopted in HPC computing systems and AI platforms. This trend is seen in academia and is strengthening in industry with e.g. nVidia assuming S3 to be the default data protocol for accessing data from GPUs for AI data intensive workflows.
S3 services are provided using variety of products and solutions, including prioprietary Amazon S3 cloud infrastructure, closed appliances (EMC's Isilon, DDN's Infinia, NetApp Storage Grid to name the few), as well as open source platforms including Ceph with its S3 RADOS gateway, OpenStack Swift's S3 interface, and standalone products and gateways such as MinIO or Versity.
Despite overall storage 'cloudification', most storage systems in industry and academia is still based on filesystems, including open source ZFS, Lustre (cloud, HTC, HPC, AI) and commercial "usual suspects" DAOS, GPFS, BeeGFS (mostly used in HPC). An open issue is how to enable S3 access to those "legacy systems" in an efficient, scalable way while ensuring non-disruptive filesystems operation, reuse of existing infrastructure and knowledge and avoiding the costly and cumbersome data migration from "legacy" to "modern" systems or duplicated storage of datasets to ensure both S3 and POSIX access.
In our work we focus on examining compatibility and performance of gateway-based system comprised of SaunaFS with HDD disk storage back-end (including CMR and SMR drives) and VersityGW providing S3 protocol integration. The experiment has been part of the technology evaluation conducted by PSNC within the nationally funded R&D project on data center and distributed storage.
Our work proves that such a combined architecture does introduce significant performance limits or overheads on the throughput (GB/s) and I/O capability (IOPS) of the combined storage system compared to its native…
The talk
Credits and licence
- Institution
- CERN
- Speaker
- Krzysztof Wadówka
- Year
- 2026
- Language
- English
- Subjects
- Computer science
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed