Optimized C++ Inference in SOFIE for Quantized Machine Learning Models

About

**Project Description**:

Machine learning models are widely used at CERN for a variety of tasks, including data analysis, pattern recognition, and anomaly detection. These models are often highly sophisticated in order to handle the complexity of the data they process. In addition, when deployed in real-time systems such as trigger pipelines, they must deliver extremely fast inference performance. With the upcoming upgrades of the LHC experiments for the High-Luminosity phase, machine learning models are expected to become even more complex while simultaneously needing to process significantly higher data rates. This places stringent requirements on inference latency, memory usage, and hardware efficiency. SOFIE (System for Optimized Fast Inference code Emit) is a tool that translates trained machine learning models into optimized C++ code with BLAS as its only dependency. In its current implementation, SOFIE-generated code has demonstrated up to 2× faster inference and approximately 50% lower memory consumption on CPUs compared to ONNX Runtime for commonly used models in high-energy physics research. SOFIE is also being extended to support inference on heterogeneous architectures through alpaka, an abstraction library for portable heterogeneous programming. In its next iteration, SOFIE will have integrations with PQuant, a library for end-to-end hardware-aware model compression using pruning and quantization techniques. This integration will enable the generation of C++ inference code from quantized machine learning models.

**Supervisor**: Sanjiban Sengupta, Lorenzo Moneta

00:00:00 Slide 1

00:00:12 Slide 2

00:00:34 Slide 3

00:01:01 Slide 4

00:01:37 Slide 5

00:03:07 Slide 6

00:03:41 Slide 7

00:03:53 Slide 8

00:04:23 Slide 9