About
Subtitle: Superiority of EOS-based Comtrade Distributed File System (CDFS) for Earth Observation Data Storage.
Introduction ------------ The rising quantity of data collected necessitates transitioning to the next generation of reliable, high-performance data storage solutions. Despite the clear need for high-performance storage across many industries, there has yet to be a consensus on the optimal high-performance storage technology that would suit all user needs.
The project's goal was speed-testing to provide a possibility to combine them with the currently available state-of-the-art technology results, technological capabilities and existing expertise.
Methodological approach ----------------------- We choose representative high-performance storage solutions compared on two different platforms for storage clients.
- Linux: - Ceph - EOS - IBM Spectrum Scale - Hadoop - Windows: - Ceph - EOS-drive - EOS through Samba - EOS-wnc - Hadoop
An example of specifications of the testing environment is the following.
- *Management node:* - 32 threads - 2x Intel® Xeon® Silver 4208 Processor - 384 GB of RAM - 2x SSDs of 2 TB. - *Storage nodes:* - 32 threads - 2x Intel® Xeon® Silver 4208 Processor - 64 GB of RAM - 1x SSD of 2 TB for the operation system - 6x HDDs of 2 TB for data. - Client nodes: - 12 threads - 1x Intel® Core™ i5-12400 Processor - 16 GB of RAM - 1x SSD with 1 TB.
Results ------- The results are from the testing performed separately in an isolated environment for each high-performance solution but on the same hardware. For each platform, there were three categories of tests related to file sizes: (1) small, (2) medium, and (3) large files.
According to the results, some high-performance file systems have evident advantages, as shown in our presentation. These results should be the starting point for an even more exact comparison between these file systems. They are good starting points in choosing the right high-performance file system..
0:00:00…
The talk
Credits and licence
- Institution
- CERN
- Department
- HEP Computing
- Speaker
- Gregor Molan
- Year
- 2024
- Language
- English
- Subjects
- Computer science
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed