# Adam Kucharski: The Uncertain Science of Certainty

An episode of [Ground Truths](https://invisiblenetwork.ai/podcast/1728526108), 29 Jun 2025, 45 min.

Audio: https://api.substack.com/feed/podcast/167052986/8b688fc46d8806d2d206dafddb1b6d89.mp3

“To navigate proof, we must reach into a thicket of errors and biases. We must confront monsters and embrace uncertainty, balancing — and rebalancing —our beliefs. We must seek out every useful fragment of data, gather every relevant tool, searching wider and climbing further. Finding the good foundations among the bad. Dodging dogma and falsehoods. Questioning. Measuring. Triangulating. Convincing. Then perhaps, just perhaps, we'll reach the truth in time.”—Adam Kucharski My conversation with Professor Kucharski on what constitutes certainty and proof in science (and other domains), with emphasis on many of the learnings from Covid. Given the politicization of science and A.I.’s deepfakes and power for blurring of truth, it’s hard to think of a topic more important right now. Audio file (Ground Truths can also be downloaded on Apple Podcasts and Spotify) Eric Topol (00:06): Hello, it's Eric Topol from Ground Truths and I am really delighted to welcome Adam Kucharski, who is the author of a new book, Proof: The Art and Science of Certainty. He’s a distinguished mathematician, by the way, the first mathematician we've had on Ground Truths and a person who I had the real privilege of getting to know a bit through the Covid pandemic. So welcome, Adam. Adam Kucharski (00:28): Thanks for having me. Eric Topol (00:30): Yeah, I mean, I think just to let everybody know, you're a Professor at London School of Hygiene and Tropical Medicine and also noteworthy you won the Adams Prize, which is one of the most impressive recognitions in the field of mathematics. This is the book, it's a winner, Proof and there's so much to talk about. So Adam, maybe what I'd start off is the quote in the book that captivates in the beginning, “life is full of situations that can reveal remarkably large gaps in our understanding of what is true and why it's true. This is a book about those gaps.” So what was the motivation when you undertook this very big endeavor? Adam Kucharski (01:17): I think a lot of it comes to the work I do at my day job where we have to deal with a lot of evidence under pressure, particularly if you work in outbreaks or emerging health concerns. And often it really pushes the limits, our methodology and how we converge on what's true subject to potential revision in the future. I think particularly having a background in math’s, I think you kind of grow up with this idea that you can get to these concrete, almost immovable truths and then even just looking through the history, realizing that often isn't the case, that there's these kind of very human dynamics that play out around them. And it's something I think that everyone in science can reflect on that sometimes what convinces us doesn't convince other people, and particularly when you have that kind of urgency of time pressure, working out how to navigate that. Eric Topol (02:05): Yeah. Well, I mean I think these times of course have really gotten us to appreciate, particularly during Covid, the importance of understanding uncertainty. And I think one of the ways that we can dispel what people assume they know is the famous Monty Hall, which you get into a bit in the book. So I think everybody here is familiar with that show, Let's Make a Deal and maybe you can just take us through what happens with one of the doors are unveiled and how that changes the mathematics. Adam Kucharski (02:50): Yeah, sure. So I think it is a problem that's been around for a while and it's based on this game show. So you've got three doors that are closed. Behind two of the doors there is a goat and behind one of the doors is a luxury car. So obviously, you want to win the car. The host asks you to pick a door, so you point to one, maybe door number two, then the host who knows what's behind the doors opens another door to reveal a goat and then ask you, do you want to change your mind? Do you want to switch doors? And a lot of the, I think intuition people have, and certainly when I…

## Transcript

[0:06] SPEAKER_01: Hello,
[0:06] SPEAKER_01: it's Eric Topol from Ground Truths,
[0:09] SPEAKER_01: and I am really delighted to welcome Adam Kaczarski,
[0:13] SPEAKER_01: who is the author of a new book,
[0:16] SPEAKER_01: Proof,
[0:17] SPEAKER_01: a distinguished mathematician,
[0:19] SPEAKER_01: by the way,
[0:19] SPEAKER_01: the first mathematician we've had on Ground Truths,
[0:22] and a person who I had the real privilege of getting to know a
[0:25] bit through the
[0:26] SPEAKER_01: COVID pandemic.
[0:27] SPEAKER_01: So welcome, Adam.
[0:28] SPEAKER_01: Thanks for having me.
[0:30] SPEAKER_01: Yeah,
[0:30] SPEAKER_01: I mean,
[0:31] SPEAKER_01: I think just to let everybody know,
[0:32] SPEAKER_01: you're a professor at London School of Hygiene and Tropical Medicine.
[0:38] SPEAKER_01: And also noteworthy,
[0:39] SPEAKER_01: you won the Adam Fries,
[0:40] which is one of the most impressive recognitions in the field
[0:44] of mathematics.
[0:45] SPEAKER_01: This is the book.
[0:47] SPEAKER_01: It's a winner, proof.
[0:49] SPEAKER_01: And there's so much to talk about.
[0:50] SPEAKER_01: So,
[0:51] SPEAKER_01: Adam,
[0:51] maybe what I'd start off is the quote in the book that kind of
[0:55] captivates in the
[0:56] SPEAKER_01: beginning is,
[0:57] SPEAKER_01: Life is full of situations that can reveal remarkably large gaps in our
[1:04] SPEAKER_01: understanding of what is true and why it's true.
[1:07] SPEAKER_01: This is a book about these gaps.
[1:09] SPEAKER_01: So what was the motivation when you undertook this very big endeavor?
[1:17] I think a lot of it comes to the sort of work I do in my day
[1:21] job where we have to
[1:23] SPEAKER_00: deal with a lot of evidence under pressure,
[1:24] SPEAKER_00: particularly if you work in outbreaks or emerging health concerns.
[1:29] SPEAKER_00: And often it really pushes the limits,
[1:32] our methodology and how we converge on what's true subject to
[1:37] potential revision in
[1:38] SPEAKER_00: the future.
[1:38] I think particularly having a background in maths,
[1:40] I think you kind of grow up
[1:42] with this idea that you can get to these concrete,
[1:45] almost immovable truths.
[1:46] SPEAKER_00: And then even just looking through the history,
[1:48] SPEAKER_00: realizing that often isn't the case,
[1:49] SPEAKER_00: that there's these kind of very human dynamics that play out around them.
[1:54] SPEAKER_00: And it's something I think that everyone in science can reflect on,
[1:57] SPEAKER_00: that sometimes what convinces us doesn't convince other people.
[2:00] SPEAKER_00: And particularly when you have that kind of urgency of time pressure,
[2:03] SPEAKER_00: working out how to navigate that.
[2:05] SPEAKER_01: Yeah,
[2:05] SPEAKER_01: well,
[2:06] SPEAKER_01: I mean,
[2:06] SPEAKER_01: I think these times,
[2:08] SPEAKER_01: of course,
[2:08] SPEAKER_01: have really gotten us into appreciate,
[2:10] SPEAKER_01: particularly during COVID,
[2:13] SPEAKER_01: the importance of understanding uncertainty.
[2:19] And I think one of the ways that we can dispel what people
[2:25] assume they know is the
[2:27] SPEAKER_01: famous Monty Hall,
[2:30] SPEAKER_01: which you go about,
[2:32] SPEAKER_01: you get into a bit in the book.
[2:34] SPEAKER_01: So I think everybody here is familiar with the show, Let's Make a Deal.
[2:39] SPEAKER_01: And maybe you can just take us through what happens with door numbers,
[2:44] SPEAKER_01: with one of the doors are unveiled and how that changes the mathematics.
[2:50] yeah sure so i think it's it's a problem that's been around for
[2:53] a while and it's
[2:54] it's based on this game show so you've got three doors that are
[2:56] closed um behind
[2:58] two of the doors are a goat and behind one of the doors is a
[3:01] luxury car so
[3:01] obviously you want to win the car uh the host asks you to pick
[3:04] a door so you point
[3:06] to one maybe door number two then the host who knows what's
[3:08] behind the doors opens
[3:09] another door to reveal a goat and then ask you do you want to
[3:12] change your mind do
[3:13] SPEAKER_00: you want to switch doors
[3:15] SPEAKER_00: And a lot of the intuition people have,
[3:17] SPEAKER_00: and certainly when I first came across this problem many years ago,
[3:19] SPEAKER_00: is,
[3:19] SPEAKER_00: well,
[3:20] SPEAKER_00: you've got two doors left,
[3:21] SPEAKER_00: right?
[3:21] SPEAKER_00: You've picked one, there's another one, it's 50-50.
[3:23] SPEAKER_00: And even some quite well-respected mathematicians,
[3:27] SPEAKER_00: people like Paul Erdos,
[3:28] SPEAKER_00: who really published more papers than almost anyone else,
[3:34] SPEAKER_00: That was their initial gut reaction.
[3:35] SPEAKER_00: But if you work through all of the combinations,
[3:38] if you pick this door and then the host does this and you
[3:40] switch or not switch and
[3:42] SPEAKER_00: work through all of those options,
[3:44] you actually double your chances if you switch versus sticking
[3:48] with the door.
[3:48] SPEAKER_00: So something that's counterintuitive,
[3:50] SPEAKER_00: but I think one of the things that really struck me,
[3:53] SPEAKER_00: and even over the years trying to explain it,
[3:54] SPEAKER_00: is
[3:55] convincing myself of the answer which was when i first came
[3:57] across it as a teenager
[3:58] i did quite quickly is very different to convincing someone
[4:01] else and even actually
[4:02] paul odosh uh one of his colleagues kind of showed him the what
[4:06] i'd call proof by
[4:07] exhaustion so go through every combination uh and that didn't
[4:10] really convince him
[4:10] so then he started to simulate and said let's do a computer
[4:14] simulation of the game
[4:14] a hundred thousand times and again you know switching was this
[4:17] optimal strategy but
[4:19] SPEAKER_00: odosh wasn't um
[4:21] SPEAKER_00: wasn't really convinced because I accept that this is the case,
[4:24] SPEAKER_00: but I'm not really satisfied with it.
[4:26] And I think that encapsulates for a lot of people,
[4:29] their experience of proof and evidence.
[4:31] SPEAKER_00: It's kind of,
[4:31] SPEAKER_00: it's a fact and you kind of have to take it as given,
[4:34] but there's actually quite a big bridge often to really
[4:37] understanding why it's true
[4:39] SPEAKER_00: and feeling convinced by it.
[4:41] SPEAKER_00: Yeah,
[4:41] SPEAKER_00: I mean,
[4:41] I think it's a fabulous example because I think everyone would
[4:45] naturally assume
[4:45] SPEAKER_01: it's 50-50 and it isn't.
[4:48] SPEAKER_01: And, you know, I think that gets us to the topic at hand.
[4:53] SPEAKER_01: What I love, there's many things I love about this book.
[4:56] SPEAKER_01: One is that you don't just get into science and medicine.
[5:00] SPEAKER_01: but you cut across all the domains, law, mathematics, AI.
[5:05] SPEAKER_01: So it's a very comprehensive sweep of everything about proof and truth.
[5:10] SPEAKER_01: And it couldn't come at a better time, as we'll get into.
[5:15] Maybe just starting off with math,
[5:17] the term I love,
[5:19] mathematical monsters.
[5:21] SPEAKER_01: Can you tell us a little bit more about that?
[5:25] SPEAKER_00: Yeah,
[5:26] this was a fascinating situation that emerged in the late 19th
[5:29] century where a lot
[5:32] SPEAKER_00: of maths,
[5:32] SPEAKER_00: certainly in Europe,
[5:34] had been derived from geometry because a lot of the ancient
[5:37] Greek influence on how
[5:39] SPEAKER_00: we shape things.
[5:40] SPEAKER_00: And then Newton and his work on rates of change and calculus,
[5:44] SPEAKER_00: it was really the natural world that provided a lot of inspiration,
[5:47] SPEAKER_00: these kind of tangible objects,
[5:48] SPEAKER_00: tangible movements.
[5:50] And as mathematicians started to build out the theory around
[5:54] rates of change and
[5:55] SPEAKER_00: and how we tackle these kinds of situations,
[5:58] SPEAKER_00: they sometimes took that intuition a bit too seriously.
[6:04] SPEAKER_00: There were some theorems that they said were intuitively obvious,
[6:07] SPEAKER_00: some of these French mathematicians.
[6:10] SPEAKER_00: One,
[6:11] SPEAKER_00: for example,
[6:11] SPEAKER_00: is this idea of how things change smoothly over time and how you do those
[6:16] SPEAKER_00: calculations.
[6:16] But what happened was some mathematicians came along and showed
[6:20] that when you have
[6:20] SPEAKER_00: things that can be infinitely small,
[6:23] SPEAKER_00: that intuition didn't necessarily hold in the same way.
[6:27] SPEAKER_00: They came up with these examples that broke a lot of these theorems.
[6:29] SPEAKER_00: A lot of the establishment at the time called these things monsters.
[6:34] They called them these aberrations against common sense and
[6:38] this idea that if
[6:38] SPEAKER_00: Newton had known about them,
[6:40] he never would have done all of his discovery because they're
[6:42] just nuisances and we
[6:43] SPEAKER_00: just need to get rid of them.
[6:44] There's this real tension at the core of mathematics in the
[6:48] late 1800s where
[6:51] Some people just wanted to disregard this and say,
[6:53] look,
[6:53] it works for most of the time.
[6:54] SPEAKER_00: That's kind of good enough.
[6:56] SPEAKER_00: And then others really weren't happy with this quite vague logic.
[7:00] SPEAKER_00: They wanted to put it on a much sturdier ground.
[7:03] And what was remarkable actually is if you trace this then into
[7:05] the 20th century,
[7:06] a lot of these monsters and these particularly in some cases
[7:11] functions which...
[7:13] SPEAKER_00: could almost move constantly.
[7:17] SPEAKER_00: Our intuitive concept of movement is something that's smooth.
[7:21] SPEAKER_00: If you drop an apple, it accelerates at a very smooth rate.
[7:24] It would become foundational in our understanding of things
[7:27] like probability,
[7:28] SPEAKER_00: Einstein's work on atomic theory.
[7:31] A lot of these concepts where geometry breaks down would be
[7:33] really important in relativity.
[7:35] These things that we thought were monsters actually were all
[7:39] around us all the time
[7:41] SPEAKER_00: and science couldn't advance without them.
[7:43] So I think it's just this remarkable example of this tension
[7:46] within a field that's
[7:47] supposedly concrete and the things that were going to be
[7:50] shunned actually turned
[7:51] SPEAKER_00: out to be quite important.
[7:53] It's great how you convey how nature isn't so neat and tidy and
[7:59] things like
[7:59] SPEAKER_01: Brownian motion,
[8:00] SPEAKER_01: understanding that.
[8:01] SPEAKER_01: I mean, just so many things that I think fit into that general category.
[8:06] SPEAKER_01: In the legal, we won't get into too much because that's not the...
[8:10] SPEAKER_01: so much the audience of Ground Truth,
[8:13] SPEAKER_01: but,
[8:14] SPEAKER_01: you know,
[8:14] the classic things about innocent until proven guilty and proof
[8:19] beyond reasonable
[8:19] SPEAKER_01: doubt.
[8:20] I mean,
[8:20] these are obviously really important parts of that overall
[8:24] sense of proof and truth.
[8:26] We're going to get into one thing I'm fascinated by related to
[8:29] that subsequently.
[8:31] SPEAKER_01: And then in science,
[8:33] SPEAKER_01: so before we get into the different types of proof,
[8:36] obviously the pandemic is still fresh in our minds and we're
[8:40] endemic of COVID now.
[8:41] And there are so many things we got wrong along the way of
[8:46] uncertainty and didn't
[8:47] SPEAKER_01: convey that science isn't always evolving differently.
[8:53] SPEAKER_01: search for what is the truth.
[8:57] SPEAKER_01: There's plenty, no shortage of uncertainty at any moment.
[9:00] SPEAKER_01: So can you kind of recap some of the,
[9:03] SPEAKER_01: you did so much work during the pandemic,
[9:05] SPEAKER_01: and obviously some of it's in the book.
[9:07] What were some of the major things that you took out of proof
[9:11] and truth from the pandemic?
[9:14] I think it was almost just the story of two hearts,
[9:17] because on the one hand,
[9:19] SPEAKER_00: Science was the thing that got us where we are today.
[9:22] The reason that so much normality could resume and so much risk
[9:26] was reduced was the
[9:26] development of vaccines and the understanding of treatments and
[9:30] the understanding
[9:30] SPEAKER_00: of variants as they came to the next characteristics.
[9:33] It was kind of this amazing opportunity to see this happen
[9:37] faster than ever happened in history.
[9:39] SPEAKER_00: I think ever in science,
[9:41] it certainly shifted a lot of my thinking about what's possible
[9:44] and even how we
[9:45] SPEAKER_00: should think about these kinds of problems.
[9:47] SPEAKER_00: But also,
[9:47] SPEAKER_00: on the other hand,
[9:48] I think where people might have been more familiar with seeing
[9:51] science progress a
[9:53] SPEAKER_00: bit more slowly and reach consensus around some of these health issues,
[9:57] SPEAKER_00: having that emerge very rapidly can present challenges.
[10:00] Even we found with some of the work we did on alpha and then
[10:04] the delta variants,
[10:06] SPEAKER_00: and it was the early quantification of these.
[10:08] SPEAKER_00: So really, the big question is, is this thing more transmissible?
[10:11] SPEAKER_00: Because at the time,
[10:12] SPEAKER_00: countries were thinking about control measures,
[10:14] SPEAKER_00: thinking about relaxing things.
[10:16] SPEAKER_00: And you've got this just enormous social,
[10:20] SPEAKER_00: economic,
[10:21] SPEAKER_00: health decision-making based around,
[10:25] SPEAKER_00: essentially,
[10:25] SPEAKER_00: is it a lot more spreadable or is it not?
[10:29] SPEAKER_00: And you only had these fragments of evidence.
[10:31] SPEAKER_00: So I think for me, that was really an illustration of the sharp end.
[10:33] And I think what we ended up doing with some of those was
[10:36] rather than arguing over
[10:37] SPEAKER_00: a precise number,
[10:38] SPEAKER_00: something like Delta,
[10:40] SPEAKER_00: Instead, we kind of looked at, well, what's the range that matters?
[10:44] So in the sense of arguing over whether it's 40% or 50% or 30%
[10:49] more transmissible
[10:50] is perhaps less important than being it's substantially more
[10:53] transmissible and it's
[10:54] SPEAKER_00: going to start going up.
[10:55] SPEAKER_00: Is it going to go up extremely fast or just very fast?
[10:59] SPEAKER_00: That's still a very useful conclusion.
[11:01] SPEAKER_00: I think what often created some of the more challenges,
[11:04] SPEAKER_00: I think the things that kind of on reflection people looking back,
[11:07] SPEAKER_00: pick up on, are where there was probably overstated certainty.
[11:11] SPEAKER_00: We saw that around some of the airborne spread,
[11:13] SPEAKER_00: for example,
[11:14] SPEAKER_00: stated as a fact by,
[11:15] SPEAKER_00: in some cases,
[11:16] SPEAKER_00: some organizations.
[11:18] I think in some situations as well,
[11:21] governments had a constraint and presented it as scientific.
[11:25] SPEAKER_00: The UK, for example, would say, testing isn't useful.
[11:30] SPEAKER_00: And what was happening at the time was there wasn't enough tests.
[11:33] SPEAKER_00: So it was more a case of they can't test at that volume.
[11:36] But I think blowing between what the science was saying and
[11:40] what the decision making.
[11:41] SPEAKER_00: And I think also one thing we found in the UK was we made a lot of the
[11:46] SPEAKER_00: epidemiological evidence available.
[11:48] SPEAKER_00: I think that was really something that was important.
[11:51] SPEAKER_00: I found it a lot easier to communicate,
[11:52] SPEAKER_00: if you're talking to the media,
[11:53] SPEAKER_00: to be able to say,
[11:54] SPEAKER_00: look,
[11:54] SPEAKER_00: this is the paper that's out,
[11:55] SPEAKER_00: this is what it means,
[11:56] SPEAKER_00: this is the evidence.
[11:57] I always found it quite uncomfortable having to communicate
[11:59] things where you knew
[12:01] SPEAKER_00: there were reports behind the scenes,
[12:02] SPEAKER_00: but you couldn't actually articulate.
[12:04] SPEAKER_00: But I think what that did is it created this impression that particularly
[12:08] SPEAKER_00: epidemiology was driving the decision-making
[12:11] lot more than it perhaps was in reality because so much of that
[12:14] was being made
[12:15] public and a lot more of the evidence around education or
[12:19] economics was being done
[12:20] SPEAKER_00: behind the scenes.
[12:21] I think that created this kind of asymmetry in public
[12:25] perception about how that was feeding in.
[12:29] It's really hard as well as a scientist when you've got
[12:32] journalists asking you how
[12:33] SPEAKER_00: to run the country to work out those steps of,
[12:37] SPEAKER_00: am I describing
[12:38] SPEAKER_00: the evidence behind what we're seeing?
[12:39] SPEAKER_00: Am I describing the evidence about different interventions?
[12:42] SPEAKER_00: Or am I proposing, to some extent, my value system on what we do?
[12:47] And I think all of that in very intense times can be very easy
[12:53] to get blurred
[12:53] SPEAKER_00: together in public communication.
[12:55] I think we saw a few examples of that where things were being
[12:58] the follow the
[12:59] SPEAKER_00: science on policy type angle,
[13:00] SPEAKER_00: where actually,
[13:01] SPEAKER_00: once you get into what you're prioritizing within a society,
[13:04] SPEAKER_00: quite rightly,
[13:05] SPEAKER_00: you've got other things beyond just the epidemiology driving that.
[13:09] SPEAKER_01: Yeah,
[13:09] SPEAKER_01: I mean,
[13:10] SPEAKER_01: that term that you just used,
[13:12] SPEAKER_01: follow the science,
[13:13] SPEAKER_01: is such an important term because it tells us about the dynamic aspect.
[13:19] SPEAKER_01: It isn't just a snapshot.
[13:21] SPEAKER_01: It's constantly being revised.
[13:23] SPEAKER_01: But during the pandemic,
[13:25] SPEAKER_01: we had things like the six-foot rule that was never supported by data,
[13:31] SPEAKER_01: but yet still today,
[13:33] SPEAKER_01: like if I walk around my hospital and there's still...
[13:37] SPEAKER_01: the footprints of the six foot rule and,
[13:39] SPEAKER_01: you know,
[13:40] SPEAKER_01: not paying attention to the fact that this was airborne and took,
[13:43] SPEAKER_01: you know,
[13:44] SPEAKER_01: years before some of these things were accepted,
[13:46] SPEAKER_01: the flatten the curve stuff with lockdowns,
[13:49] SPEAKER_01: which,
[13:49] SPEAKER_01: you know,
[13:50] SPEAKER_01: I never was supportive of that,
[13:51] SPEAKER_01: but the,
[13:52] SPEAKER_01: You know,
[13:52] SPEAKER_01: perhaps at the worst point,
[13:54] SPEAKER_01: the idea that hospitals would get overrun was an issue,
[13:57] SPEAKER_01: but,
[13:58] SPEAKER_01: you know,
[13:58] SPEAKER_01: it got carried away with school shutdowns for prolonged periods in,
[14:03] SPEAKER_01: you know,
[14:04] SPEAKER_01: in some parts of the world,
[14:05] SPEAKER_01: especially,
[14:06] SPEAKER_01: you know,
[14:06] SPEAKER_01: very stringent lockdowns.
[14:08] SPEAKER_01: But anyway, we learned a lot.
[14:09] But perhaps one of the greatest lessons is that people's
[14:13] expectations about science,
[14:15] you
[14:16] SPEAKER_01: is that it's absolute.
[14:18] SPEAKER_01: And somehow you have this truth that's not there.
[14:22] SPEAKER_01: I mean, it's getting revised.
[14:26] SPEAKER_01: It's kind of on the job training.
[14:28] SPEAKER_01: It's, in this case, on the pandemic revision.
[14:31] SPEAKER_01: But very interesting.
[14:33] SPEAKER_01: And that gets us to,
[14:34] SPEAKER_01: I think,
[14:34] SPEAKER_01: the next topic,
[14:35] which I think is a fundamental part of the book distributed
[14:39] throughout the book,
[14:40] SPEAKER_01: which is the different types of proof.
[14:43] SPEAKER_01: in biomedicine, and of course, across all these domains.
[14:47] SPEAKER_01: And so you take us through things like randomized trials,
[14:51] SPEAKER_01: p-values,
[14:52] SPEAKER_01: 95% confidence intervals,
[14:54] SPEAKER_01: counterfactuals,
[14:55] SPEAKER_01: causation and correlation,
[14:58] SPEAKER_01: peer review,
[14:59] SPEAKER_01: the works,
[15:00] SPEAKER_01: which is great,
[15:01] SPEAKER_01: because a lot of people have misconceptions of these things.
[15:04] SPEAKER_01: So for example,
[15:05] SPEAKER_01: randomized trials,
[15:06] SPEAKER_01: which is like the temple of the randomized trials,
[15:10] SPEAKER_01: they're not as great as a lot of people think.
[15:13] SPEAKER_01: Yes,
[15:13] SPEAKER_01: they can help us establish cause and effect,
[15:16] SPEAKER_01: but they're skewed because of the people who come into the trial.
[15:20] SPEAKER_01: So there may not at all be a representative sample.
[15:25] SPEAKER_01: What are your thoughts about over-deference to randomized trials?
[15:31] Yeah,
[15:31] I think that the story of kind of how we rank evidence in
[15:35] medicines is a fascinating one.
[15:36] I mean,
[15:36] even just how long it took for people to think about these
[15:39] elements of randomization.
[15:40] SPEAKER_00: Fundamentally, the
[15:42] What we're trying to do when we have evidence in medicine or
[15:45] science is prevent
[15:47] SPEAKER_00: ourselves from confusing randomness for a signal.
[15:49] SPEAKER_00: I mean,
[15:50] SPEAKER_00: that's fundamentally...
[15:51] SPEAKER_00: We don't want to mistake something,
[15:53] SPEAKER_00: don't think it's going on where it's not.
[15:55] SPEAKER_00: And the challenge,
[15:56] SPEAKER_00: particularly with any intervention,
[15:57] SPEAKER_00: is you only get to see one version of reality.
[15:59] SPEAKER_00: You can't give someone a drug,
[16:00] SPEAKER_00: follow them,
[16:01] SPEAKER_00: rewind history,
[16:02] SPEAKER_00: not give them the drug,
[16:03] SPEAKER_00: and then follow them again.
[16:04] SPEAKER_00: So one of the things that essentially randomization allows us to do is...
[16:10] SPEAKER_00: If you have two groups,
[16:12] SPEAKER_00: one that's been randomized,
[16:12] SPEAKER_00: one that hasn't on average,
[16:15] SPEAKER_00: um,
[16:15] the difference in outcomes between those groups is going to be
[16:18] down to the,
[16:19] SPEAKER_00: the treatment effect.
[16:20] SPEAKER_00: So.
[16:20] SPEAKER_00: It doesn't necessarily mean in reality that would be the case,
[16:22] SPEAKER_00: but on average,
[16:23] SPEAKER_00: that's the expectation that you'd have.
[16:24] SPEAKER_00: And it's kind of interesting,
[16:25] SPEAKER_00: actually,
[16:25] SPEAKER_00: that the first modern randomized controlled trial in medicine in 1947,
[16:31] SPEAKER_00: this is for TB and streptomycin,
[16:34] SPEAKER_00: the randomization element,
[16:35] SPEAKER_00: actually,
[16:35] SPEAKER_00: it wasn't so much statistical as behavioral.
[16:39] SPEAKER_00: If you have people coming to hospital, you could...
[16:41] SPEAKER_00: to some extent, just say, we'll just alternate.
[16:43] SPEAKER_00: We're not going to randomise.
[16:43] We're just going to first patient,
[16:45] we'll say,
[16:45] is a control,
[16:46] second patient a treatment.
[16:47] SPEAKER_00: But what they found in a lot of previous studies was doctors have bias.
[16:51] Maybe that patient looks a little bit ill or that one maybe is
[16:54] on borderline for eligibility.
[16:56] And often you've got these quite striking imbalances when you
[16:58] allowed it for human judgment.
[16:59] SPEAKER_00: So it was really about kind of shielding against those behavioural
[17:04] SPEAKER_00: elements.
[17:05] SPEAKER_00: But I think there's a few situations.
[17:07] SPEAKER_00: It's a really powerful tool for a lot of these questions.
[17:09] SPEAKER_00: But as you mentioned,
[17:10] SPEAKER_00: one is this issue of you have the population you study on,
[17:14] SPEAKER_00: and then perhaps in reality,
[17:16] SPEAKER_00: how that translates elsewhere.
[17:19] SPEAKER_00: Things like flu vaccines are a good example,
[17:20] which are very dependent on immunity and evolution and what
[17:23] goes on in different
[17:24] SPEAKER_00: populations.
[17:25] SPEAKER_00: Sometimes you've had a result on a vaccine in one place,
[17:27] and then the effectiveness doesn't translate in the same way to
[17:30] somewhere else.
[17:31] SPEAKER_00: I think the other really important thing to bear in mind is
[17:34] SPEAKER_00: As I said,
[17:34] SPEAKER_00: it's the averaging,
[17:36] SPEAKER_00: that you're getting an average effect between two different groups.
[17:38] And I think we see a certain lot of development around things
[17:41] like personalized
[17:41] SPEAKER_00: medicine,
[17:42] where actually you're much more interested in the outcome for
[17:45] the individual.
[17:46] SPEAKER_00: And so what a trial can give you evidence of is,
[17:49] SPEAKER_00: on average,
[17:50] SPEAKER_00: across a group,
[17:51] SPEAKER_00: this is the effect that I can expect this intervention to have.
[17:55] But we've now seen more of the emergence of things like N
[17:57] equals 1 studies where
[17:58] SPEAKER_00: you can actually,
[17:59] SPEAKER_00: over the same individual,
[18:01] SPEAKER_00: particularly with chronic conditions,
[18:03] SPEAKER_00: look at those kind of interventions.
[18:05] And also there's just these extreme examples where you're
[18:08] ethically not going to run a trial.
[18:10] There's never been a trial of whether it's a good idea to have
[18:12] intensive care units
[18:13] SPEAKER_00: in hospitals.
[18:13] SPEAKER_00: Or there's a lot of these kind of historical treatments which are just so
[18:16] SPEAKER_00: overwhelmingly effective.
[18:19] SPEAKER_00: So almost this hierarchy over time,
[18:22] you can see it getting shifted because actually you do have
[18:24] these situations where
[18:26] other forms of evidence can get you either closer to what you
[18:29] need or just more
[18:30] SPEAKER_00: feasibly an answer where it's just not ethical or practical to do an RCT.
[18:36] SPEAKER_01: Yeah,
[18:37] and that brings us to the natural experiments I just wrote
[18:41] about recently,
[18:42] SPEAKER_01: the one with shingles,
[18:43] which there's two big natural experiments to suggest that
[18:47] shingles might reduce the
[18:50] SPEAKER_01: risk of Alzheimer's,
[18:51] SPEAKER_01: an added benefit beyond the shingles that was not anticipated.
[18:56] SPEAKER_01: Your thoughts about natural experiments,
[18:58] because here you're getting a much different type of population
[19:03] assessment,
[19:04] SPEAKER_01: again,
[19:04] SPEAKER_01: not at the individual level,
[19:06] but not necessarily restricted by some potentially skewed
[19:12] enrollment criteria.
[19:14] SPEAKER_00: Yeah, I think this has emerged as a really valuable tool.
[19:17] SPEAKER_00: It's kind of interesting in the book,
[19:19] SPEAKER_00: you're talking to economists like Josh Angrist,
[19:21] SPEAKER_00: that a lot of these ideas emerge in epidemiology,
[19:24] SPEAKER_00: but I think were really then taken up by economists,
[19:26] particularly as they wanted to add more credibility to a lot of
[19:29] these kind of
[19:29] SPEAKER_00: policy questions.
[19:31] SPEAKER_00: And ultimately,
[19:32] SPEAKER_00: it comes down to this issue that for a lot of problems,
[19:34] SPEAKER_00: we can't necessarily intervene and randomise.
[19:38] SPEAKER_00: But there might be a situation that's done it to some extent for us.
[19:41] SPEAKER_00: So the classic example is the Vietnam draft,
[19:44] SPEAKER_00: where it was kind of random birthdays were drawn out of lottery.
[19:46] And so there's been a lot of studies subsequently about the
[19:49] effect of serving in
[19:51] SPEAKER_00: the military on different subsequent lifetime outcomes.
[19:55] SPEAKER_00: Because broadly, those people have been randomized.
[19:58] It was for a different reason,
[20:00] but you've got that element of randomization driving that.
[20:02] SPEAKER_00: And so again, with...
[20:05] SPEAKER_00: some of the recent singles data and other studies,
[20:07] SPEAKER_00: you might have a situation,
[20:08] SPEAKER_00: for example,
[20:08] where there's been an intervention that's somewhat arbitrary in
[20:12] terms of time,
[20:12] SPEAKER_00: like it's a cut off on a birth date,
[20:14] SPEAKER_00: for example.
[20:15] SPEAKER_00: And under certain assumptions,
[20:17] SPEAKER_00: you could think,
[20:17] SPEAKER_00: well,
[20:18] SPEAKER_00: actually,
[20:18] there's no real reason for the person on this day and this day
[20:20] to be fundamentally
[20:21] SPEAKER_00: different.
[20:22] SPEAKER_00: I mean,
[20:23] perhaps there might be effects of kind of cohorts if it's
[20:26] school years or this sort
[20:26] SPEAKER_00: of thing.
[20:27] SPEAKER_00: But generally,
[20:28] SPEAKER_00: this isn't the same as having people who are very,
[20:30] SPEAKER_00: very different ages and very different characteristics.
[20:32] SPEAKER_00: It's just
[20:33] SPEAKER_00: nature,
[20:33] SPEAKER_00: or in this case,
[20:34] SPEAKER_00: just a policy intervention for a different reason,
[20:36] SPEAKER_00: has given you that randomization,
[20:38] SPEAKER_00: which allows you,
[20:39] SPEAKER_00: or kind of pseudo randomization,
[20:41] which allows you to then look at something about the effects of
[20:43] an intervention
[20:44] that you wouldn't as reliably if you were just kind of digging
[20:48] into the data of
[20:49] SPEAKER_00: kind of yes,
[20:49] SPEAKER_00: no,
[20:49] SPEAKER_00: who's received a vaccine.
[20:52] SPEAKER_01: Yeah, no, I think it's really valuable.
[20:54] SPEAKER_01: And now I think increasingly a given priority.
[21:01] SPEAKER_01: If you can find these natural experiments,
[21:03] SPEAKER_01: they're not always so abundant to use,
[21:05] SPEAKER_01: you know,
[21:06] SPEAKER_01: to extrapolate from.
[21:07] SPEAKER_01: But when they are, they're phenomenal.
[21:12] SPEAKER_01: The causation correlations.
[21:15] SPEAKER_01: is so big, the issue there.
[21:18] SPEAKER_01: I mean,
[21:18] SPEAKER_01: Judea Pearl's why book,
[21:22] SPEAKER_01: and you give so many great examples throughout the book of proof.
[21:26] I wonder if you could comment on that a bit more,
[21:29] because this is where associations
[21:32] SPEAKER_01: are confused somehow or other with a direct effect.
[21:37] SPEAKER_01: And we unfortunately make these jumps all too frequently.
[21:43] Perhaps it's the most common problem that's occurring in the
[21:47] way we interpret
[21:48] SPEAKER_01: medical research data.
[21:52] Yeah,
[21:52] I think this is an issue that I think a lot of people get
[21:54] drilled into in their training.
[21:55] SPEAKER_00: You know,
[21:55] just because there's a correlation between things doesn't mean
[21:58] that that thing
[21:58] SPEAKER_00: causes this thing.
[22:00] SPEAKER_00: But it really struck me,
[22:01] SPEAKER_00: as I talk to people researching the book,
[22:03] SPEAKER_00: how in practice in research,
[22:05] SPEAKER_00: there's actually a bit more to it in how it's played out.
[22:07] SPEAKER_00: So first of all,
[22:09] SPEAKER_00: if there's a correlation between things,
[22:11] SPEAKER_00: it doesn't tell you much generally that's useful for intervention.
[22:14] SPEAKER_00: You know,
[22:14] SPEAKER_00: if two things are correlated,
[22:16] it doesn't mean that changing that thing is going to have an
[22:18] effect on that thing,
[22:18] SPEAKER_00: because there might be something that's influencing both of them,
[22:21] SPEAKER_00: you know.
[22:22] If you have more ice cream sales,
[22:23] it will lead to more heat throat cases.
[22:26] It doesn't mean that changing ice cream sales is going to have
[22:28] that effect.
[22:29] SPEAKER_00: But it does allow you to make predictions,
[22:31] SPEAKER_00: potentially,
[22:31] SPEAKER_00: because if you can identify consistent patterns,
[22:33] SPEAKER_00: you can say,
[22:33] SPEAKER_00: okay,
[22:34] SPEAKER_00: if this thing's going up,
[22:35] SPEAKER_00: I'm going to make a prediction that this thing's going up.
[22:37] So one thing I found quite striking actually talking to
[22:39] research in different
[22:40] fields is how many fields choose to focus on prediction because
[22:44] it kind of avoids
[22:45] SPEAKER_00: having to deal with this cause and effect problem.
[22:47] SPEAKER_00: And even in fields like psychology, it was kind of interesting that
[22:52] SPEAKER_00: There's a lot of focus on predicting things like relationship outcomes.
[22:56] But actually,
[22:56] for people,
[22:58] you don't want a prediction about your relationship,
[22:59] right?
[23:00] SPEAKER_00: You want to know, well, how can I do something about it?
[23:02] You don't just want someone to sell you,
[23:03] your relationship's going to go downhill.
[23:05] So there's almost part of the challenge is people just got
[23:09] stuck on prediction
[23:10] SPEAKER_00: because it's an easier field of work,
[23:13] SPEAKER_00: whereas actually some of those problems will involve intervention.
[23:16] I think the other thing that really stood out for me is in
[23:19] epidemiology and a lot
[23:20] SPEAKER_00: of other fields,
[23:21] SPEAKER_00: Rightly, people are very cautious to not get that mixed up.
[23:24] SPEAKER_00: They don't want to mix up correlations or associations with causation.
[23:28] But you've kind of got this weird situation where a lot of
[23:31] papers go out of their
[23:32] SPEAKER_00: way to not use causal language and say,
[23:35] SPEAKER_00: it's an association,
[23:35] SPEAKER_00: it's just an association,
[23:36] SPEAKER_00: it's just an association,
[23:37] SPEAKER_00: we can't say anything about causality.
[23:39] SPEAKER_00: And at the end of the paper,
[23:40] SPEAKER_00: they'll say,
[23:40] SPEAKER_00: well,
[23:41] we should think about introducing more of this thing or
[23:43] restricting this thing.
[23:44] SPEAKER_00: So it's really the whole paper and its purpose is framed around
[23:49] causal intervention but it's extremely careful throughout the
[23:52] paper to not frame it
[23:53] as a causal claim so I think we also you're almost by skirting
[23:57] that too much um
[24:00] SPEAKER_00: we actually avoid the problems that people sometimes care about.
[24:03] And I think a lot of the nice work that's been going on in
[24:05] causal inference is
[24:06] SPEAKER_00: trying to get people to confront this more head on rather than say,
[24:10] SPEAKER_00: okay,
[24:10] SPEAKER_00: you can just stay in this prediction world and that's fine.
[24:14] SPEAKER_00: And then just later, maybe make a policy suggestion off the back of it.
[24:20] SPEAKER_01: Yeah,
[24:20] I think this cause and effect is a very alluring concept to
[24:25] support proof,
[24:26] SPEAKER_01: as you so nicely go through in the book.
[24:30] But of course,
[24:31] one of the things that we use to help us is the biological
[24:36] mechanism,
[24:36] right?
[24:37] SPEAKER_01: So here you have,
[24:38] SPEAKER_01: let's say,
[24:38] SPEAKER_01: for example,
[24:39] you're trying to get a new drug approved by the Food and Drug
[24:43] Administration.
[24:44] And the request is,
[24:46] well,
[24:46] we want two trials,
[24:47] randomized trials,
[24:49] independent.
[24:50] SPEAKER_01: We want to have p-values that are significant.
[24:52] And we want to know the biological mechanism,
[24:54] ideally with the dose response of the drug.
[24:57] But there are many drugs,
[24:59] as you review,
[25:00] that have no biological mechanism established.
[25:03] SPEAKER_01: And even when the tobacco problems were mounting,
[25:08] SPEAKER_01: the actual mechanism of how tobacco use caused cancer wasn't known.
[25:14] SPEAKER_01: So how important is the biological mechanism,
[25:17] especially now that we're well into the AI world where
[25:21] explainability is demanded?
[25:24] SPEAKER_01: And so we don't know the mechanism,
[25:26] SPEAKER_01: but we also don't know the mechanism and lots of things in medicine too,
[25:29] SPEAKER_01: like anesthetics and even things as simple as aspirin,
[25:34] SPEAKER_01: how it works and many others.
[25:36] SPEAKER_01: So how do we deal with this quest for the biological mechanisms?
[25:42] SPEAKER_00: I think that's a really good point.
[25:43] It kind of shows almost a lot of the transition I think we're
[25:46] going through currently.
[25:48] And I think particularly for things like smoking cancer where
[25:50] it's very hard to run a trial.
[25:52] SPEAKER_00: You can't make people randomly take up smoking.
[25:54] SPEAKER_00: Having those additional pieces of evidence,
[25:57] SPEAKER_00: whether it's an analogy with a similar carcinogen,
[25:59] SPEAKER_00: whether it's a biological mechanism,
[26:01] can help almost give you more supports for that argument that
[26:06] there's a cause and
[26:06] SPEAKER_00: effect going on.
[26:08] SPEAKER_00: But I think what I found quite striking,
[26:10] and I realized actually that it's something that had kind of
[26:13] bothered me a bit,
[26:13] SPEAKER_00: and I'd be interested to hear whether it bothers you,
[26:16] SPEAKER_00: but...
[26:17] With the emergence of AI,
[26:18] it's almost a bit of the loss of scientific satisfaction.
[26:21] I think you grow up with learning about how the world works and
[26:25] why this is doing
[26:26] SPEAKER_00: what it's doing.
[26:27] SPEAKER_00: I talked,
[26:27] SPEAKER_00: for example,
[26:27] of some of the people involved with AlphaFold and some of the
[26:31] subsequent work in
[26:32] SPEAKER_00: storing those predictions about structures.
[26:35] SPEAKER_00: and they'd almost kind of made peace with it,
[26:38] SPEAKER_00: which I found kind of interesting,
[26:39] SPEAKER_00: because I think they started off being a bit uncomfortable.
[26:41] SPEAKER_00: You've got these remarkable AI models making these predictions,
[26:45] SPEAKER_00: but we don't understand still biologically what's happening here.
[26:49] SPEAKER_00: But I think they'd just settled in saying,
[26:51] SPEAKER_00: well,
[26:51] SPEAKER_00: biology is really complex on some of these problems,
[26:54] and if we can have a tool that can give us this extremely
[26:56] valuable information,
[26:58] SPEAKER_00: maybe that's okay.
[27:00] SPEAKER_00: And it was just interesting that they'd really kind of gone through that
[27:05] SPEAKER_00: kind of process,
[27:05] which I think a lot of people are still grappling with and that
[27:08] almost that
[27:08] discomfort of using AI and what's going to convince you that
[27:13] that's a useful,
[27:14] SPEAKER_00: reliable prediction,
[27:16] SPEAKER_00: whether it's something like predicting protein folding or getting in the
[27:19] SPEAKER_00: self-driving car.
[27:20] SPEAKER_00: What's the evidence you need to kind of convince you that's reliable?
[27:26] SPEAKER_01: Yeah,
[27:26] SPEAKER_01: no,
[27:26] I'm so glad you brought that up because when Demis Hassabis and
[27:32] John Jumper won the
[27:33] SPEAKER_01: Nobel Prize,
[27:35] SPEAKER_01: you know,
[27:36] the point I made was maybe there should be an asterisk with AI
[27:41] because they don't
[27:42] SPEAKER_01: know how it works.
[27:43] SPEAKER_01: I mean, they had all the rich data from the protein data bank.
[27:47] And they got the transformer model to do it for 200 million
[27:52] protein structure prediction.
[27:54] But they still,
[27:55] to this day,
[27:56] don't fully understand how the model really was working.
[28:01] SPEAKER_01: So it reinforces what you're just saying.
[28:03] SPEAKER_01: And of course, it cuts across so many types of AI.
[28:06] SPEAKER_01: It's just that we tend to hold different standards in medicine,
[28:09] not realizing that there's lots of lack of explainability for
[28:13] routine medical
[28:14] SPEAKER_01: treatments today.
[28:17] SPEAKER_01: One of the things that I found fascinating in your book,
[28:19] SPEAKER_01: because there's different levels of proof,
[28:21] SPEAKER_01: different types of proof,
[28:23] SPEAKER_01: but solid logical systems.
[28:26] SPEAKER_01: And on page 60 of the book,
[28:29] SPEAKER_01: especially pertinent to the US right now,
[28:32] SPEAKER_01: there is a bit about Kurt Godel.
[28:38] SPEAKER_01: And what he did there was he basically, there was a question about
[28:47] SPEAKER_01: a dictatorship in the U S could it ever occur?
[28:52] SPEAKER_01: And, um, he says, Godel says, yes, it can.
[28:56] SPEAKER_01: I can prove it.
[28:59] SPEAKER_01: And he's using the constitution itself to prove it,
[29:03] which I found fascinating because of course we're seeing that
[29:06] emerge right now.
[29:08] SPEAKER_01: Uh, can you give us a little bit more about this?
[29:10] SPEAKER_01: Because this is fascinating about, you know, the fifth amendment.
[29:14] SPEAKER_01: And I mean, I,
[29:15] I never thought that the constitution would allow for a
[29:18] dictatorship to emerge.
[29:23] SPEAKER_00: And this was a fascinating story because Kurt Goodell,
[29:25] who's one of the greatest logical minds of the 20th century and
[29:28] did a lot of work,
[29:29] SPEAKER_00: particularly in the early 20th century,
[29:32] SPEAKER_00: around system of rules,
[29:33] SPEAKER_00: particularly things like mathematics,
[29:35] SPEAKER_00: and whether they can ever
[29:38] SPEAKER_00: be really fully satisfying.
[29:40] SPEAKER_00: Particularly in mathematics,
[29:41] he showed that there was this problem that is very hard to have
[29:45] a set of rules for
[29:45] something like arithmetic that was both complete and covered
[29:48] every situation,
[29:49] SPEAKER_00: but also had no contradictions.
[29:50] SPEAKER_00: I think a lot of countries,
[29:52] if you go back to things like the Napoleonic Code and these
[29:56] attempts to almost
[29:57] SPEAKER_00: write down every possible legal situation that could be imaginable,
[30:01] always just descended into either they needed amendments or
[30:03] they had contradictions.
[30:04] SPEAKER_00: I think Godel's work really summed it up.
[30:07] And there's a story that was in the late 40s when he had his
[30:10] citizenship interview
[30:11] SPEAKER_00: and Einstein and
[30:14] SPEAKER_00: Oscar Morgensen went along as witnesses for him.
[30:17] SPEAKER_00: It's always told as a lighthearted story,
[30:20] SPEAKER_00: as this logical mind,
[30:23] SPEAKER_00: this academic,
[30:25] SPEAKER_00: just saying something silly in front of the judge.
[30:28] SPEAKER_00: Actually,
[30:29] SPEAKER_00: to my own omission,
[30:30] I've in the past given talks and mentioned it in this slightly
[30:33] lighthearted way.
[30:35] But for the book,
[30:36] I got talking to a few people who had taken it more seriously.
[30:38] SPEAKER_00: I realised he has this extremely logically focused mind at the time.
[30:46] SPEAKER_00: And maybe there should have been something more to it.
[30:49] SPEAKER_00: And people who have dug more into possibilities were saying,
[30:53] SPEAKER_00: well,
[30:53] SPEAKER_00: what could he have spotted now?
[30:55] SPEAKER_00: that bothered him.
[30:57] And a lot of his work that he did about consistency in mass was
[31:00] around particularly
[31:02] SPEAKER_00: self-referential statements.
[31:03] SPEAKER_00: So if I say this sentence is false,
[31:06] SPEAKER_00: it's kind of self-referential,
[31:07] SPEAKER_00: and if it is false,
[31:08] SPEAKER_00: then it's true,
[31:09] SPEAKER_00: but if it's true,
[31:10] SPEAKER_00: then it's false,
[31:10] SPEAKER_00: and you get these kind of weird self-referential contradictions.
[31:13] SPEAKER_00: And so one of the theories about Godel was that
[31:17] SPEAKER_00: And in the constitution,
[31:19] it wasn't that there was a kind of rule for someone can become
[31:21] a dictator,
[31:22] but rather people can use the mechanisms within the
[31:24] constitution to make it easier
[31:26] SPEAKER_00: to make further amendments.
[31:27] And he kind of called it a downward cycle of amendment that he
[31:30] had seen happening
[31:31] SPEAKER_00: in Europe in the run up to the war.
[31:34] SPEAKER_00: And again,
[31:35] SPEAKER_00: because this is never fully documented exactly what he thought,
[31:38] SPEAKER_00: but it's one of the theories that it wouldn't just be outright,
[31:41] that it would just be this cyclic process of weakening and
[31:43] weakening and weakening
[31:44] SPEAKER_00: and making it easier to add.
[31:45] And actually,
[31:46] when I wrote that,
[31:47] it was one of the earlier bits of the book that I drafted.
[31:50] SPEAKER_00: I did sort of debate whether including it.
[31:52] I thought,
[31:52] you know,
[31:53] is this actually just a bit in the weeds of kind of American
[31:56] history?
[31:58] SPEAKER_00: And here we are.
[31:59] SPEAKER_00: Yeah, it's remarkable.
[32:00] SPEAKER_01: Yeah, yeah.
[32:01] SPEAKER_01: No,
[32:02] SPEAKER_01: I mean,
[32:02] SPEAKER_01: I found it,
[32:03] SPEAKER_01: you know,
[32:04] SPEAKER_01: it struck me when I was reading this because here back in 1947,
[32:07] SPEAKER_01: there was somebody,
[32:10] SPEAKER_01: predicting that this could happen based on some,
[32:13] SPEAKER_01: you know,
[32:14] SPEAKER_01: if you want to call it loopholes,
[32:17] SPEAKER_01: if you will,
[32:18] SPEAKER_01: or ability to change things,
[32:21] even though you would have thought otherwise that there wasn't
[32:25] any possible
[32:26] SPEAKER_01: capability for that to happen.
[32:28] SPEAKER_01: Now, one of the things I thought was a bit contradictory is
[32:32] SPEAKER_01: is two parts here.
[32:33] SPEAKER_01: One is from Angus Deaton.
[32:37] SPEAKER_01: He wrote, gold standard thinking is magical thinking.
[32:41] SPEAKER_01: And then the other is what you basically are concluding in many respects.
[32:49] SPEAKER_01: To navigate proof, we must reach into a thicket of errors and biases.
[32:56] SPEAKER_01: We must confront monsters,
[32:57] SPEAKER_01: we've talked about a bit,
[32:59] SPEAKER_01: and embrace uncertainty,
[33:01] SPEAKER_01: balancing and rebalancing our beliefs.
[33:03] SPEAKER_01: We must seek out every useful fragment of data,
[33:06] SPEAKER_01: gather every relevant tool,
[33:08] SPEAKER_01: searching wider and climbing further,
[33:10] SPEAKER_01: finding the good foundations among the bad,
[33:12] SPEAKER_01: dodging dogma and falsehoods,
[33:15] SPEAKER_01: questioning,
[33:16] SPEAKER_01: measuring,
[33:16] SPEAKER_01: triangulating,
[33:17] SPEAKER_01: convincing.
[33:18] SPEAKER_01: Then perhaps, just perhaps,
[33:20] SPEAKER_01: will reach the truth in time.
[33:22] SPEAKER_01: So here you have on the one hand,
[33:25] SPEAKER_01: you know,
[33:25] SPEAKER_01: your search for the truth,
[33:28] SPEAKER_01: proof,
[33:28] SPEAKER_01: which I think that little paragraph says it all in many respects.
[33:33] SPEAKER_01: It sums up somewhat to the work that you review here.
[33:36] SPEAKER_01: And on the other, you have this Nobel laureate saying, you know,
[33:40] SPEAKER_01: You don't have to go to extremes here.
[33:42] SPEAKER_01: The enemy of good is perfect, perhaps.
[33:45] SPEAKER_01: I mean, how do you reconcile this sense that you shouldn't go so far?
[33:53] SPEAKER_01: Don't search for absolute perfection of proof.
[33:58] Yeah,
[33:58] I think that encapsulates a lot of what the book is about is
[34:01] that search for certainty.
[34:02] SPEAKER_00: And, you know, how far do you have to go?
[34:05] SPEAKER_00: And I think one of the things that,
[34:07] SPEAKER_00: you know,
[34:08] SPEAKER_00: so then there's a lot of interesting discussion,
[34:09] SPEAKER_00: some fascinating kind of papers around at what point,
[34:15] SPEAKER_00: you know,
[34:15] SPEAKER_00: do you use these studies?
[34:16] SPEAKER_00: What are their flaws?
[34:18] SPEAKER_00: But I think one of the things that does stand out is,
[34:20] SPEAKER_00: you know,
[34:21] SPEAKER_00: across fields,
[34:22] SPEAKER_00: across science,
[34:22] SPEAKER_00: medicine,
[34:23] SPEAKER_00: even if you're going to kind of law AI,
[34:25] SPEAKER_00: the
[34:26] Having these kind of cookie cutter,
[34:28] you know,
[34:28] this is the definitive way of doing it.
[34:31] SPEAKER_00: And if you just follow this simple rule,
[34:34] SPEAKER_00: you know,
[34:34] SPEAKER_00: if you do your p-value,
[34:35] SPEAKER_00: you'll get there and you'll be fine.
[34:37] SPEAKER_00: And I think that's where a lot of the danger is.
[34:39] And I think that's what we've seen over time,
[34:42] certainly in science,
[34:43] people kind of chasing...
[34:44] SPEAKER_00: certain targets and all the behaviors that come around that,
[34:48] SPEAKER_00: or in certain situations,
[34:49] disregarding valuable evidence because you've got this kind of
[34:53] gold standard and
[34:54] SPEAKER_00: nothing else will do.
[34:56] SPEAKER_00: And I think particularly in a crisis,
[34:58] it's very dangerous to have that because you might have a lower
[35:03] level of evidence
[35:04] SPEAKER_00: that demands a certain action.
[35:06] And you almost bias yourself towards inaction if you have these
[35:10] kind of very simple thresholds.
[35:11] SPEAKER_00: So I think for me,
[35:13] SPEAKER_00: across all of these stories and across the whole book,
[35:16] SPEAKER_00: I mean,
[35:17] SPEAKER_00: William Gossett,
[35:18] who did a lot of pioneering work on statistical experiments at
[35:21] Guinness in the
[35:22] SPEAKER_00: early 20th century,
[35:23] SPEAKER_00: he had this nice question he sort of framed is,
[35:25] SPEAKER_00: how much do we lose?
[35:27] SPEAKER_00: And it's kind of, if we're thinking about the problems
[35:31] SPEAKER_00: There's always more studies we can do.
[35:32] SPEAKER_00: There's always more confidence we can have.
[35:35] But whether it's a patient we want to treat or a crisis we need
[35:37] to deal with,
[35:38] we need to work out actually getting that level of proof that's
[35:44] really appropriate
[35:44] SPEAKER_00: for where we are currently.
[35:47] SPEAKER_01: Yeah,
[35:48] SPEAKER_01: that's,
[35:48] SPEAKER_01: I think,
[35:49] exceptionally important that there's this kind of spectrum or
[35:55] continuum in the
[35:58] SPEAKER_01: following science and search for truth.
[36:00] SPEAKER_01: And that distinction, I think, really nails it.
[36:04] SPEAKER_01: Now,
[36:04] one of the things that's unique in the book is you don't just
[36:06] go through all the
[36:07] SPEAKER_01: different types of how you would get to proof,
[36:10] SPEAKER_01: but you also talk about how the evidence is acted odd.
[36:15] And for example,
[36:17] you quote,
[36:17] they spent a lot of time misinforming themselves.
[36:20] SPEAKER_01: This is the whole idea of taking data and torturing it or using it,
[36:28] SPEAKER_01: dredging it,
[36:29] SPEAKER_01: however way you want to support data.
[36:34] SPEAKER_01: either conspiracy theories or alternative facts,
[36:40] SPEAKER_01: basically manipulating,
[36:42] SPEAKER_01: sometimes even emasculating what evidence and data we have
[36:47] SPEAKER_01: And one of the sentences,
[36:50] SPEAKER_01: or I guess this is from Sir Francis Bacon,
[36:53] SPEAKER_01: truth is the daughter of time,
[36:56] SPEAKER_01: but the added part is not authority.
[36:59] SPEAKER_01: So here we have our president here that repeats things that are wrong,
[37:04] SPEAKER_01: fabricated or wrong,
[37:06] SPEAKER_01: and he keeps repeating to the point that people believe it's true.
[37:10] SPEAKER_01: But on the other hand,
[37:11] you could say truth is a daughter of time because you like to
[37:15] not accept any truth
[37:17] SPEAKER_01: immediately.
[37:17] SPEAKER_01: You like to see it get replicated and further supported, backed up.
[37:23] SPEAKER_01: So in that one sentence, truth is a daughter of time, not authority.
[37:28] SPEAKER_01: There's the whole, you know, ball of wax here.
[37:31] SPEAKER_01: Can you take us through that?
[37:32] SPEAKER_01: Because I just think that people don't understand that that that
[37:38] SPEAKER_01: truth being tested over time,
[37:42] SPEAKER_01: but also manipulated by its repetition,
[37:46] SPEAKER_01: this is a part of the big problem that we live in right now.
[37:50] SPEAKER_00: Yeah,
[37:51] and I think it's something that writing the book and actually
[37:53] just reflecting on it
[37:54] subsequently has made me think about a lot in just how people
[37:58] approach these kinds
[37:59] SPEAKER_00: of problems.
[38:00] I think there's some idea that conspiracy theorists are just
[38:04] lazy and have maybe
[38:05] SPEAKER_00: just fallen for a random thing.
[38:08] But talking to people,
[38:11] you really think about these things a lot more in the field.
[38:13] SPEAKER_00: And actually,
[38:13] the more I've ended up engaging with people who believe things
[38:17] that are just
[38:18] SPEAKER_00: outright unevidenced around vaccines or around health issues,
[38:22] SPEAKER_00: they often have this mountain of papers and data to hand.
[38:27] SPEAKER_00: And a lot of it often, they will be peer-reviewed papers.
[38:30] It won't necessarily be supporting the point that they think
[38:33] it's supporting.
[38:33] SPEAKER_01: Right, right.
[38:35] But it's not something that you can just say everything you're
[38:38] saying is false.
[38:39] There's actually often a lot of things that have kind of been
[38:42] put together and then
[38:43] SPEAKER_00: it's just that leap to that conclusion.
[38:45] SPEAKER_00: I think you also see a lot of scientific language kind of borrowed.
[38:50] SPEAKER_00: So I gave a talk earlier this year and it got posted on YouTube.
[38:52] SPEAKER_00: It had conspiracy theories in it and there was a lot of
[38:56] SPEAKER_00: conspiracy theory supporters who piled in the comments.
[39:00] SPEAKER_00: One of the points they made is, scepticism is good.
[39:02] SPEAKER_00: It's the Royal Society.
[39:03] SPEAKER_00: Take no one's word for it.
[39:05] SPEAKER_00: You need this.
[39:07] We're the ones that are doing science and people who just
[39:10] assume that science is
[39:11] SPEAKER_00: settled are in the wrong.
[39:14] SPEAKER_00: Again, you also mentioned that repetition.
[39:16] SPEAKER_00: There's this
[39:17] phenomenon it's kind of the illusory truth problem that if you
[39:21] repeatedly tell
[39:22] someone something's false it will increase their belief in it
[39:25] even if it's
[39:26] something quite outrageous and that mimics that scientific
[39:29] repetition because
[39:30] people kind of say okay but if I've heard it again and again
[39:32] it's almost like if
[39:33] you treat these as mini experiments I'm just accumulating
[39:36] evidence that this thing
[39:37] is true so it's made me think like a lot about how you've got
[39:41] essentially a lot of
[39:42] SPEAKER_00: mimicry of the scientific method of you know
[39:45] amount of data and how you present it and,
[39:47] you know,
[39:48] this kind of skepticism being good.
[39:49] SPEAKER_00: But I think a lot of it comes down to,
[39:53] SPEAKER_00: as well as just looking for these logical flaws,
[39:54] SPEAKER_00: but also ability to be wrong,
[39:56] SPEAKER_00: you know,
[39:57] SPEAKER_00: in not actually seeking out things that confirm.
[39:59] I think all of us,
[39:59] you know,
[40:00] it's something that I've certainly tried to do a lot working on
[40:03] SPEAKER_00: emergencies and one of the scientific advisory groups that I worked on,
[40:08] SPEAKER_00: almost it became a catchphrase.
[40:09] Whenever someone presented something,
[40:10] they finished by saying,
[40:12] tell me why I'm wrong.
[40:14] SPEAKER_00: And if you've got a variant that's more transmissible,
[40:16] SPEAKER_00: I don't want to be right about that,
[40:18] SPEAKER_00: really.
[40:19] SPEAKER_00: And it's something that is quite hard to do.
[40:22] SPEAKER_00: And I found it, particularly for something that's quite high pressure,
[40:26] SPEAKER_00: trying to get a policymaker or someone to write,
[40:27] SPEAKER_00: even just non-publicly,
[40:30] SPEAKER_00: by themselves,
[40:31] write down what you think is going to happen or write down what
[40:34] would convince you
[40:35] SPEAKER_00: that you're wrong about something.
[40:37] I think particularly on contentious issues where someone's got
[40:40] perhaps a lot of
[40:42] SPEAKER_00: public persona wrapped up in something,
[40:44] SPEAKER_00: that's really hard to do.
[40:45] SPEAKER_00: But I think it's those kind of elements that distinguish between
[40:49] kind of getting sucked into a conspiracy theory and really
[40:52] seeking out evidence
[40:53] that supports it and trying to just get your theory stronger
[40:55] and stronger and
[40:57] actually seeking out things that might overturn your belief
[41:01] about the world.
[41:02] SPEAKER_00: And it's often those things that we don't want overturned.
[41:04] I think those are the views that we all have politically or in
[41:07] other ways.
[41:08] SPEAKER_00: And that's often where the problems lie.
[41:11] SPEAKER_01: Yeah,
[41:11] SPEAKER_01: I think this is perhaps one of,
[41:14] SPEAKER_01: if not the most essential part here,
[41:16] SPEAKER_01: is that to try to deal with the different views,
[41:20] SPEAKER_01: we have biases,
[41:22] SPEAKER_01: as you re-emphasized throughout,
[41:25] SPEAKER_01: but if you can use these different types of proof to have a sound
[41:32] SPEAKER_01: discussion,
[41:32] SPEAKER_01: conversation,
[41:34] SPEAKER_01: refutation,
[41:35] SPEAKER_01: whereby you don't summarily dismiss another view,
[41:40] SPEAKER_01: which may be skewed and maybe just absolutely wrong,
[41:44] SPEAKER_01: maybe fabricated,
[41:45] SPEAKER_01: whatever,
[41:46] SPEAKER_01: but that you can engage and say,
[41:48] SPEAKER_01: here's why,
[41:49] SPEAKER_01: these are my proof points,
[41:52] or this is why there's some extent of certainty you can have
[41:58] regarding this view of
[42:00] SPEAKER_01: the data
[42:01] SPEAKER_01: I think this is so fundamental because unfortunately,
[42:05] SPEAKER_01: as we saw during the pandemic,
[42:08] SPEAKER_01: the Strident minority,
[42:12] SPEAKER_01: which were the anti-science,
[42:14] SPEAKER_01: anti-vaxxers,
[42:15] they were summarily dismissed as being kooks and adopting
[42:25] conspiracy theories
[42:26] SPEAKER_01: without the right engagement.
[42:28] SPEAKER_01: and the right debates.
[42:30] SPEAKER_01: And I think this might've helped along the way,
[42:34] no less the fact that a lot of scientists didn't really want to
[42:36] engage in the first
[42:37] SPEAKER_01: place and adopt this methodical proof that you've advocated in the book,
[42:45] SPEAKER_01: so many different ways to support a hypothesis or an assertion.
[42:54] SPEAKER_01: Now, we've covered a lot here, Adam.
[42:56] SPEAKER_01: Have I missed some central parts of the book and the effort?
[43:01] SPEAKER_01: Because it's really quite extraordinary.
[43:03] SPEAKER_01: I know it's your third book, but it's certainly a standout.
[43:08] And it's certainly a standout not just for your books,
[43:10] but books on this topic.
[43:13] SPEAKER_00: Thanks.
[43:14] SPEAKER_00: And as much appreciated, it was not an easy book to write.
[43:16] SPEAKER_00: I think at times I kind of wondered if I should have taken on the topic.
[43:20] SPEAKER_00: But I think your last point kind of speaks to that.
[43:23] I think a core thing is that gap often between what convinces
[43:27] us and what convinces
[43:27] SPEAKER_00: someone else.
[43:28] SPEAKER_00: I think it's often very tempting as a scientist to say,
[43:31] SPEAKER_00: you know,
[43:31] SPEAKER_00: the evidence is clear or,
[43:33] SPEAKER_00: you know,
[43:33] SPEAKER_00: the science has proved this.
[43:36] But even on something like,
[43:37] you know,
[43:37] the vaccines,
[43:38] you do get the loud minority who
[43:40] perhaps think they're putting microchips in people and disease
[43:42] kind of outlandish
[43:43] SPEAKER_00: views,
[43:43] but you actually get a lot more people who might just have some
[43:46] scepticism of
[43:46] SPEAKER_00: pharmaceutical companies or they might have,
[43:48] SPEAKER_00: you know,
[43:49] SPEAKER_00: my wife was pregnant actually at the time during COVID and she was,
[43:52] we weighed it up because there wasn't much data on pregnancy
[43:56] and the vaccine.
[43:57] SPEAKER_00: And I think it's just finding, you know, what is convincing?
[43:59] SPEAKER_00: Is it having more studies from other countries?
[44:02] SPEAKER_00: Is it understanding more about the biology?
[44:05] Is it,
[44:06] you know,
[44:06] understanding how you evaluate some of those safety signals?
[44:09] And I think that's just really important to not just think what
[44:12] convinces us and
[44:13] SPEAKER_00: it's going to be obvious to other people,
[44:15] SPEAKER_00: but actually think where are they coming from?
[44:17] SPEAKER_00: Because ultimately,
[44:18] SPEAKER_00: you know,
[44:19] having proof isn't that good unless it leads to the action that
[44:22] can make lives
[44:23] SPEAKER_00: better.
[44:24] SPEAKER_01: Yeah.
[44:25] SPEAKER_01: Well, look, you've inculcated my mind with this book and proof.
[44:29] SPEAKER_01: Anytime I think of the word proof, I'm going to be thinking about you.
[44:33] SPEAKER_01: So thank you.
[44:33] Thanks for taking the time to have a conversation about your
[44:37] book,
[44:38] your work.
[44:39] And I know we're going to count on you for the astute
[44:43] mathematics and analyses of
[44:45] SPEAKER_01: outbreaks in the future,
[44:47] SPEAKER_01: which we will see.
[44:48] Unfortunately,
[44:49] we're seeing now,
[44:50] in fact,
[44:51] already in this country with measles and whatnot.
[44:55] SPEAKER_01: So thank you, and we'll continue to follow your great work.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.