Podcasts · Into the Impossible With Brian Keating

We Built Something We Can’t Control | A Warning from Top AI Safety Expert - #563

· 1 h 20 min

Episode We Built Something We Can’t Control | A Warning from Top AI Safety Expert - #563

0:00 1:19:36

If anyone builds superintelligent AI before we know how to control it, everyone dies. Nate Soares wrote the book on why that’s not a metaphor. Subscribe if you want science with evidence, not speculation.

Soares runs the Machine Intelligence Research Institute and co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The first word in that title is if. That matters. His argument is not that doom is certain. His argument is that the path we are on leads there, that the driver is asleep at the wheel, and that we still have time to wake him up.

We argue for over an hour. I push on whether LLMs can ever reach superintelligence, whether GPU lock-in is a real ceiling, and what it would actually take to move his p-doom. He pushes back with one clean point: by the time an AI can rediscover general relativity from pre-1911 data the way Einstein did, we will have almost no time left. You don’t wait for that goalpost.

What you’ll hear:

•

Why the bus-racing-toward-a-cliff analogy depends entirely on whether the driver is asleep or awake

•

Whether LLM lock-in is a prison or a temporary inefficiency

•

What the AI that broke out of its virtual machine to solve a hacking problem tells us

•

Why GPT-4o encouraging a teenager toward suicide is not a malice problem but a training problem

•

The difference between an AI doing the right thing too well and an AI that never wanted to do what you asked

•

What Soares actually thinks about aliens, Dyson spheres, and why we should not see stars going out

The first word in the title is if. The second word to watch is would.

CHAPTERS

00:00 The people racing to build superhuman AI say it might kill everyone

00:42 Who coined "AI alignment" and why the first word in the title matters

02:28 Is it already too late for the if?

04:40 The bus, the cliff, and the sleeping driver

05:02 Silicon Valley is spooked. Washington is not.

07:02 Align with who? The rogue actor problem

07:34 Who is holding the leash?

08:24 The AI that edits its own test and deletes the log file

10:04 Controllability vs. making an AI that actually cares

10:44 The move gets harder. The outcome gets easier.

13:04 Are GPUs and LLMs a ceiling or a temporary inefficiency?

16:56 Brian's Einstein test: can an LLM rediscover general relativity?

18:38 Waiting for the goalpost is waiting too long

20:14 How prediction training can push AI beyond humans

21:44 Tycho Brahe, Kepler, and planetary motion as a prediction problem

24:38 Yann LeCun said never. GPT-4 did it half a generation later.

27:28 Can you prove a no-go theorem for superintelligence?

29:14 Training a human takes a light bulb. Training an AI takes a city.

33:28 What would proof of alien life do to p-doom?

35:00 Why interstellar aliens should have Dyson spheres

44:26 What would actually update Soares' p-doom?

49:42 Nobody intended this. Intent doesn't matter.

51:08 The AI hides its tracks before it does what you want

51:34 Sycophancy vs. hallucination: which runs deeper?

51:56 Leaded gasoline and civilizational risk

59:48 Sam Harris: humans have no free will but AIs do

01:00:38 Is alignment really a governance problem?

01:01:48 Unaligned AI vs. AI aligned to the wrong person

01:04:20 2026: 10 to 30% chance of automated AI research this year

01:06:44 What if Soares is wrong?

01:09:18 What gets him out of bed

01:12:38 Watch my conversation with Roman Yampolskiy

Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt

All my top AI episodes in one place: https://briankeating.com/ai

Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu

Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1

Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating

Join this channel for perks, monthly Office Hours,…

Links in the notes 16

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.