Towards unified full-stack performance analysis and automated computer system design with Adaptyst

About

Slow performance is often a major blocker of new visionary applications in scientific computing and related fields, regardless of whether it is embedded or distributed computing. This issue is becoming more and more challenging to tackle as it is no longer enough to do only algorithmic optimisations, only hardware optimisations, or only (operating) system optimisations: all of them need to be considered together.

Architecting full-stack computer systems customised for a use case comes to the rescue, namely software-system-hardware co-design. However, doing this manually per use case is cumbersome as the search space of possible solutions is vast, the number of different programming models is substantial, and experts from various disciplines need to be involved. Moreover, performance analysis tools often used here are fragmented, with state-of-the-art programs tending to be proprietary and not compatible with each other.

This is why automated full-stack system design is promising, but the existing solutions are few and far between and do not scale. Adaptyst is an open-source project at CERN aiming to solve this problem. It is meant to be a comprehensive architecture-agnostic tool which:

- unifies performance analysis across the entire software-hardware stack by calling state-of-the-art software and APIs under the hood with any remaining gaps bridged by Adaptyst (so that performance can be inspected both macro- and microscopically regardless of the workflow and platform type) - suggests automatically the best solutions of workflow performance bottlenecks in terms of one or more of: software optimisations, hardware choices and/or customisations, and (operating) system design - scales easily from embedded to high-performance/distributed computing and allows adding support for new software/system/hardware components seamlessly by anyone thanks to the modular design

The tool is in the early phase of development and concentrating on profiling at the moment. Given…