# A journey over the memory managment stack for HPC large applications on moderne architectures

- Institution: CERN
- Department: other events or meetings
- Speaker: Sébastien Valat
- Year: 2019
- Language: English
- Source: [CERN](https://videos.cern.ch/record/3002438)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Teens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

Memory managment has always been an issue for large application but the increase of memory space and intra-node thread-based parallelism now put lot more pressure on this complex part of the operating system stack. Althrough there is a long tradition of algorithm developpements on this topic with behind 60 years of research there is still a lot to do.

This is even more true in large scale application where the size of the code (target was a million line C++/MPI app) and global complexity is a big limitation to apply what should theoritically be the clean way to proceed. We also today need to make global optimization to make the wall stack well interacting not letting a component breaking the performance gained by the top or bottom one.

After making a PhD. on memory management in HPC mostly arround a malloc implementation and various kernels memory managment studies for supercomputers and NUMA architectures I pursued as a post-doc developping a memory profiling tool: MALT. During my time at CERN I added to the list NUMAPROF a NUMA memory profiling tool.

I can over this talk recap the 9 years road I walked on with experience feedback showing sometimes impressive performance gaps on large real applications by considering the path from CPU caches, NUMA layout going through the OS paging system and malloc implementation closing by profiling real applications. I will try to glue the full picture showing the need to keep the global picture to really reach performance..

0:00:00 Slide 1

0:00:25 Slide 2

0:00:57 Slide 3

0:02:03 Slide 4

0:02:17 Slide 5

0:02:59 Slide 6

0:04:12 Slide 7

0:04:29 Slide 8

0:04:43 Slide 9

0:04:51 Slide 10

0:05:10 Slide 11

0:05:30 Slide 12

0:05:35 Slide 13

0:05:49 Slide 14

0:07:02 Slide 15

0:07:17 Slide 16

0:07:25 Slide 17

0:07:31 Slide 18

0:07:40 Slide 19

0:07:46 Slide 20

0:07:52 Slide 21

0:08:24 Slide 22

0:08:31 Slide 23

0:09:27 Slide 24

0:09:47 Slide 25

0:10:23 Slide 26

0:10:36 Slide 27

0:11:05 Slide 28

0:11:19…

## Lectures

1. [A journey over the memory managment stack for HPC large applications on moderne architectures](https://invisiblenetwork.ai/learn/cern/3002438/6102870c2415), 41 min (CERN's player)