A scheme to implement local server computation on EOS system based on Xrootd plug-in

About

Particle physics computing model has a kind of high statistical calculation, such applications need to access a large amount of data for analysis, the data I/O capability is very high requirements.For example, the LHAASO experiment generates trillions of events each year, and the large raw data needs to be decode to encode and mark before it can be analyzed. In this process, very high I/O bandwidth is required, otherwise an I/O bottleneck will form.When using the EOS file system, the user cannot know the physical storage location of the file, and when the user needs to access the file, it needs to search the MGM, transfer the file from the FST to the client, and the client provides the target file to the user.In this process, if the user needs to perform such IO intensive operations as mentioned above, there are two limitations on I/O bandwidth, one is the storage node's hard disk read and write efficiency, the other is the network bandwidth between the FST and the client.In this case, if the data storage unit and the computing unit can be integrated into one, the data handling can be significantly reduced, and the parallelism and energy efficiency of computing can be greatly improved.Currently, the potential of this kind of integrated memory and computing storage is attracting the attention of many companies and standards bodies.SNIA has formed a working group to establish standards for interoperability between computable storage devices, and the OpenFog Consortium is also working on standards for computable storage. Therefore, we propose a scheme to implement local server computation on EOS system based on Xrootd plug-in.Flags can be added after a file is accessed when a user needs to use computable storage.After receiving the access request, the client will forward the request to the FST where the file is located and perform the default decode calculation in the background on the FST.After testing, we found that using this method to simultaneously decode 10 1G…