About
Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to deploy compute resources directly within the storage servers, allowing them to perform data processing operations on the data stored in the FST nodes without having to move the data to a separate computing system. This can reduce latency and improve overall performance, especially when processing large volumes of data. The plugin can be extended to support a variety of data processing tasks, including data filtering, compression, encryption, and machine learning. The computational storage function is defined in a configuration that can be implemented in scripting languages or evolved independently of the storage system in the form of containers. When an FST node receives a request to open a file, the plugin is executed first. It then calls the target program on the storage server by parsing the parameters of the command to open the file. At this time, the input file must be on the FTS storage server, and the plugin also writes the output file to the node. At the end of the task execution, the output file is automatically registered into the MGM server. Client access is fully compatible with XRootD's API and EOS commands. Users can add tasks and parameters to be performed in the open option. The plugin has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the efficiency of data decoding is more than 5 times higher than the original method..
0:00:00 Slide 1
0:01:52 Slide 2
0:03:02 Slide 3
0:04:30 Slide 4
0:05:51 Slide 5
0:07:15 Slide 6
0:08:33 Slide 7
0:10:11 Slide 8
0:11:19 Slide 9…
The talk
Credits and licence
- Institution
- CERN
- Department
- HEP Computing
- Speaker
- Yaodong Cheng
- Year
- 2023
- Language
- English
- Subjects
- Computer science
- Source
- CERN
- Licence
- CERN, as CERN states it. CERN's terms
- Attribution
- © CERN. CERN provides its audiovisual media free of charge for educational and informational use, credited to CERN; this lecture is shown in CERN's own player.
- Who can watch
- CERN gives no rating. Invisible Network shows it to Tweens profiles and older, judged from what the source says about it. How courses are placed