Optimizing time series data storage for CERN industrial control systems using TimescaleDB and PostgreSQL

About

CERN is known for producing, processing, and storing vast amounts of data, most of which consists of physics-related information — snapshots of particle collisions recorded by detectors in experiments. Large Detector Control Systems at CERN enable the controlled operation of complex research equipment and generate "conditions" data. This data describes the time evolution of parameters such as voltages, pressures, and temperatures, which are essential for physics data analysis and ensuring reproducibility.

These control systems include over 800 mission-critical SCADA systems built on SIMATIC WinCC Open Architecture software, developed by ETM (Siemens). Together, they produce hundreds of gigabytes of time-series data daily. Storing and querying this data efficiently is the responsibility of the NextGen Archiver module of WinCC OA, a solution developed collaboratively by CERN and Siemens through the CERN openlab project. The ever-growing data volumes and throughput, combined with the need for fast, responsive user interfaces for data visualization, have posed significant challenges for the project.

In this presentation, we will share the lessons learned during the development of the NextGen Archiver. The NextGen Archiver supports multiple database technologies through pluggable backends, offering PostgreSQL and TimescaleDB as alternatives to Oracle Database, which has been used for WinCC OA archiving for over 15 years.

We will walk you through the processes and tools used to evaluate the performance and scalability of various database technologies and schemas considered during the project. Additionally, we will examine the impact of TimescaleDB features, such as compression and continuous aggregates, on improving query performance and reducing storage requirements..

0:00:00 Slide 1

0:05:29 Slide 2

0:06:40 Slide 3

0:08:15 Slide 4

0:09:56 Slide 5

0:13:08 Slide 6

0:15:45 Slide 7

0:16:47 Slide 8