EXA4MIND: A Modular Extreme Data Analytics Platform

About

In this talk, we present the project EXA4MIND (Extreme Analytics for Mining Data spaces), which establishes a modular, flexibly-deployable Extreme Data analytics platform. The platform toolkit enables the user to combine different data storage, database systems and powerful computing infrastructures to enable advanced data analytics, machine learning and AI on heterogeneous, large datasets coming from science, industry, and SMEs. Our talk will focus on the design and implementation of distributed data management within the project, where functionalities from data storage and management, over data staging and caching, to FAIR-compliant publication services are provided. For data processing, EXA4MIND provides an "Advanced Query and Indexing System" (AQIS), where analytics workflow orchestration is facilitated leveraging Apache Airflow and DASK. The platform thus helps to automate data ingestion, transfer, caching, querying and computing integration across cloud and supercomputers (HPC clusters), enabling seamless data processing across diverse backends. The system supports persistent identifier assignment and metadata management, in particular for FAIR compliance of research data, and aims at an integration with European data ecosystems (e.g. European Data Spaces, EUDAT, EOSC). Co-designing our systems with four application cases from the molecular-dynamics, automotive, smart viticulture and health sectors, we contribute to the technical groundwork for enabling next-generation data analytics in Europe.

00:00:00 Slide 1

00:00:25 Slide 2

00:00:51 Slide 3

00:01:45 Slide 4

00:02:16 Slide 5

00:02:54 Slide 6

00:03:54 Slide 7

00:04:43 Slide 8

00:05:13 Slide 9

00:05:59 Slide 10

00:07:17 Slide 11