# Entropy is all you need? The quest for best tokens and the new physics of LLMs

- Institution: CERN
- Department: EP-IT Data Science Seminars
- Speaker: Pierre-Carl Langlais
- Year: 2024
- Language: English
- Subjects: Computer science
- Source: [CERN](https://videos.cern.ch/record/3018698)
- Licence: [CERN](http://copyright.web.cern.ch)
- Who can watch: Tweens profiles and older (sources give no ratings; placed by Invisible Network from the source's own words)

LLMs currently generate texts one token at a time with fixed hyperparameters. In September 2024, OpenAI released o1, a largely undocumented model that rely on a expanded pre-generation strategies to enhance the quality of even small models ("inference scaling"). Since then, two anonymous researchers publicly released a new method of token selection: Entropix leverage entropy as a metric of token uncertainty to constraint the model to explore different generation strategies. This seminar will present an emerging field of post-training research that holds the promise to significantly enhance the quality of very small model (including a 300m million parameters we adapted to Entropix). We'll also discuss the even more speculative impact of this research direction for pretraining by letting the model adapt in flux different forms of training or optimizer strategies.

SPEAKER'S BIO:

Dr. Pierre-Carl Langlais is a researcher in artificial intelligence, co-fonder of Pleias, a French lab specialised in the training of Small Language Models for document processing and other intensive corporate use cases. A long time activist for open science, Pierre-Carl is an admin on Wikipedia and has co-written an influential report for the EU commission on non-commercial open access publishing. In 2024, he coordinated the release of Common Corpus, the largest open dataset available for the training of large language models.

(*) Groupe de Recherches Interdisciplinaires sur les Processus d'Information et de Communication

Coffee will be served at 10:30. .

0:00:00 Slide 1

0:03:08 Slide 2

0:04:11 Slide 3

0:05:26 Slide 4

0:06:56 Slide 5

0:09:07 Slide 6

0:11:00 Slide 7

0:13:09 Slide 8

0:16:32 Slide 9

0:19:23 Slide 10

0:22:20 Slide 11

0:24:01 Slide 12

0:25:08 Slide 13

0:27:17 Slide 14

0:29:03 Slide 15

0:30:53 Slide 16

0:32:07 Slide 17

0:33:20 Slide 18

0:34:29 Slide 19

0:36:07 Slide 20

0:37:26 Slide 21

0:38:35 Slide 22

0:39:50 Slide 23

0:41:01 Slide 24

0:43:31 Slide 25…

## Lectures

1. [Entropy is all you need? The quest for best tokens and the new physics of LLMs](https://invisiblenetwork.ai/learn/cern/3018698/4b9989d93ead), 59 min (CERN's player)