# How Kubernetes is Built with Kat Cosgrove

An episode of [The Pragmatic Engineer](https://invisiblenetwork.ai/podcast/1769051199), 14 May 2025, 1 h 09 min.

Audio: https://api.substack.com/feed/podcast/163506588/615ad40eefa7e6b752eabb6aa7295be9.mp3

Supported by Our Partners • WorkOS — The modern identity platform for B2B SaaS. • Modal — The cloud platform for building AI applications. • Cortex — Your Portal to Engineering Excellence. — Kubernetes is the second-largest open-source project in the world. What does it actually do—and why is it so widely adopted? In this episode of The Pragmatic Engineer, I’m joined by Kat Cosgrove, who has led several Kubernetes releases. Kat has been contributing to Kubernetes for several years, and originally got involved with the project through K3s (the lightweight Kubernetes distribution). In our conversation, we discuss how Kubernetes is structured, how it scales, and how the project is managed to avoid contributor burnout. We also go deep into: • An overview of what Kubernetes is used for • A breakdown of Kubernetes architecture: components, pods, and kubelets • Why Google built Borg, and how it evolved into Kubernetes • The benefits of large-scale open source projects—for companies, contributors, and the broader ecosystem • The size and complexity of Kubernetes—and how it’s managed • How the project protects contributors with anti-burnout policies • The size and structure of the release team • What KEPs are and how they shape Kubernetes features • Kat’s views on GenAI, and why Kubernetes blocks using AI, at least for documentation • Where Kat would like to see AI tools improve developer workflows • Getting started as a contributor to Kubernetes—and the career and networking benefits that come with it • And much more! — Timestamps (00:00) Intro (02:02) An overview of Kubernetes and who it’s for (04:27) A quick glimpse at the architecture: Kubernetes components, pods, and cubelets (07:00) Containers vs. virtual machines (10:02) The origins of Kubernetes (12:30) Why Google built Borg, and why they made it an open source project (15:51) The benefits of open source projects (17:25) The size of Kubernetes (20:55) Cluster management solutions, including different Kubernetes services (21:48) Why people contribute to Kubernetes (25:47) The anti-burnout policies Kubernetes has in place (29:07) Why Kubernetes is so popular (33:34) Why documentation is a good place to get started contributing to an open-source project (35:15) The structure of the Kubernetes release team (40:55) How responsibilities shift as engineers grow into senior positions (44:37) Using a KEP to propose a new feature—and what’s next (48:20) Feature flags in Kubernetes (52:04) Why Kat thinks most GenAI tools are scams—and why Kubernetes blocks their use (55:04) The use cases Kat would like to have AI tools for (58:20) When to use Kubernetes (1:01:25) Getting started with Kubernetes (1:04:24) How contributing to an open source project is a good way to build your network (1:05:51) Rapid fire round — The Pragmatic Engineer deepdives relevant for this episode: • Backstage: an open source developer portal • How Linux is built with Greg Kroah-Hartman • Software engineers leading projects • What TPMs do and what software engineers can learn from them • Engineering career paths at Big Tech and scaleups — See the transcript and other references from the episode at https://newsletter.pragmaticengineer.com/podcast — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

## Transcript

[0:00] SPEAKER_01: What is Kubernetes?
[0:01] Kubernetes is a tool that makes it easier to manage and scale
[0:05] an application that
[0:06] SPEAKER_00: exists as a swarm of containers.
[0:09] SPEAKER_00: So it will handle things for you like automatically scaling up resources,
[0:13] SPEAKER_00: whether that's networking or storage or whatever,
[0:17] SPEAKER_00: based on demand.
[0:18] SPEAKER_00: And it handles that for you without you having to think about it.
[0:22] SPEAKER_00: You define upper limits,
[0:23] SPEAKER_00: for a particular type of resource, and it'll go up to that as needed.
[0:27] SPEAKER_00: It scales automatically.
[0:28] This is particularly useful if you're trying to keep costs down
[0:32] and keep availability high.
[0:34] The rise in complexity of applications built as microservices
[0:38] is what necessitated
[0:39] SPEAKER_00: something like Kubernetes.
[0:41] SPEAKER_00: You could do all of this before the scaling of applications running as
[0:45] SPEAKER_00: microservices in clusters or swarms.
[0:47] Before we had Kubernetes,
[0:48] it was just very difficult and it was very manual.
[0:51] Kubernetes is just an abstraction on something that we used to
[0:55] do manually.
[0:56] Kubernetes is the second largest open source project in the
[0:58] world after Linux.
[1:00] SPEAKER_01: So what is it, why is it so popular, and how is it built?
[1:03] SPEAKER_01: Today I sit down with Cass Cosgrove,
[1:05] SPEAKER_01: who is the leader of the Kubernetes Release Team subproject.
[1:09] Cass started off as a software engineer specializing in
[1:11] embedded Linux and got
[1:12] SPEAKER_01: involved with Kubernetes by learning K3S,
[1:15] SPEAKER_01: a minimized version of Kubernetes that is designed for smaller,
[1:18] SPEAKER_01: memory-limited embedded devices.
[1:19] Kat was a contributor to Kubernetes for years before becoming a
[1:22] lead on different
[1:23] SPEAKER_01: parts of the project.
[1:24] SPEAKER_01: In our conversation,
[1:25] SPEAKER_01: we cover the architecture of Kubernetes and what Kubernetes pods are,
[1:29] why Kat thinks Kubernetes won and became the de facto
[1:32] infrastructure management solution,
[1:34] SPEAKER_01: in part because of its strong focus on documentation,
[1:37] SPEAKER_01: how to propose changes to Kubernetes via the KEP process,
[1:40] SPEAKER_01: and how Kubernetes release happens in the span of 12 to 14 weeks.
[1:44] SPEAKER_01: and many more topics.
[1:45] If you're interested in backend infrastructure or how a
[1:48] well-organized large open
[1:49] SPEAKER_01: source project operates,
[1:50] SPEAKER_01: then this episode is for you.
[1:52] If you enjoyed the show,
[1:53] please subscribe to the podcast on any podcast platform and on YouTube.
[1:56] SPEAKER_01: Kat, first of all, just welcome to the podcast.
[1:58] SPEAKER_00: Thank you for having me.
[2:00] Unless you're a Kubernetes infrastructure person who actually
[2:06] deals with Kubernetes
[2:07] SPEAKER_01: day in,
[2:07] SPEAKER_01: day out,
[2:08] SPEAKER_01: most software developers,
[2:09] SPEAKER_01: including myself,
[2:10] SPEAKER_01: I've never dealt Kubernetes directly.
[2:13] SPEAKER_01: However,
[2:14] SPEAKER_01: so many of the systems that I deploy to,
[2:16] SPEAKER_01: a lot of the APIs that run today are probably on Kubernetes.
[2:20] SPEAKER_01: I'm pretty sure, for example, use Render.
[2:22] SPEAKER_01: I'm assuming they're going to use Kubernetes.
[2:24] At Uber,
[2:25] we use Kubernetes,
[2:26] and apparently they're one of the biggest deployments.
[2:29] SPEAKER_01: What is Kubernetes?
[2:32] SPEAKER_01: What is the relationship with Docker, if there is any?
[2:37] SPEAKER_01: And as an average developer, when would you be exposed to it, if ever?
[2:41] SPEAKER_00: So I think the average developer,
[2:43] SPEAKER_00: especially if you're a web developer or an application developer,
[2:46] SPEAKER_00: you don't need to worry about what Kubernetes is.
[2:49] SPEAKER_00: You're never going to have to touch it directly.
[2:52] SPEAKER_00: And it's a specialization.
[2:55] Being like a Kubernetes cluster admin or an SRE or something,
[3:00] that's a specialization.
[3:02] And I don't think that the average developer needs to know or
[3:06] care what Kubernetes is.
[3:08] SPEAKER_00: You certainly don't need to know how to use it.
[3:09] SPEAKER_00: You can know what it is, but you don't need to know how to use it.
[3:12] SPEAKER_00: There's so much going on.
[3:14] SPEAKER_00: It's complex.
[3:16] SPEAKER_00: It's notoriously difficult to learn.
[3:18] I don't think that there is a reason to dive into it super deep
[3:22] as an average developer,
[3:23] SPEAKER_00: unless you're trying to make a career switch into Kubernetes,
[3:28] SPEAKER_00: which I couldn't blame you for because it's very lucrative.
[3:31] But Kubernetes is a tool that makes it easier to manage and
[3:37] scale an application
[3:39] SPEAKER_00: that exists as a swarm of containers.
[3:41] SPEAKER_00: Yeah.
[3:43] SPEAKER_00: So it will handle things for you like automatically scaling up resources,
[3:47] SPEAKER_00: whether that's networking or storage or whatever based on demand.
[3:55] SPEAKER_00: And it handles that for you without you having to think about it.
[3:59] You define upper limits for a particular type of resource and
[4:04] it'll go up to that as needed.
[4:06] SPEAKER_00: It scales automatically.
[4:08] SPEAKER_00: This is particularly useful if you're trying to like
[4:11] SPEAKER_00: keep costs down and keep availability high, for instance.
[4:16] SPEAKER_00: So this is direct from the Kubernetes documentation,
[4:20] SPEAKER_00: and this is what Kubernetes components look like.
[4:23] SPEAKER_00: This is a very, very simple...
[4:27] SPEAKER_00: cluster.
[4:28] SPEAKER_00: This box over here on the left is your control plane,
[4:32] SPEAKER_00: and you aren't going to interact directly with this a ton.
[4:38] SPEAKER_00: This is mostly automated.
[4:41] Over here on the right,
[4:41] where it says node,
[4:43] node,
[4:44] node,
[4:44] those are all pods.
[4:47] And within these pods run the kubelet,
[4:51] which is another little control thing.
[4:53] It's interacting with the Kubernetes API to stand up and down
[4:59] all of your containers.
[5:00] SPEAKER_00: And all of your containers exist within here too.
[5:03] SPEAKER_00: So if one goes down, like a pod fails for whatever reason, whether it
[5:11] SPEAKER_00: the code itself,
[5:12] SPEAKER_00: the application crashes,
[5:14] SPEAKER_00: or it runs out of resources or something else weird happens,
[5:17] SPEAKER_00: your deployment will stand it up again,
[5:20] SPEAKER_00: automatically for you.
[5:21] SPEAKER_00: You won't notice it, your users won't notice it, probably.
[5:24] SPEAKER_01: This episode was brought to you by WorkOS.
[5:26] SPEAKER_01: If you're building a SaaS app,
[5:28] at some point your customers will start asking for enterprise
[5:30] features like SAML authentication,
[5:32] SPEAKER_01: SKIM provisioning,
[5:33] SPEAKER_01: and fine-grained authorization.
[5:35] SPEAKER_01: That's where WorkOS comes in,
[5:36] SPEAKER_01: making it fast and painless to add enterprise features to your app.
[5:40] Their APIs are easy to understand and you can ship quickly and
[5:42] get back to building
[5:43] SPEAKER_01: other features.
[5:45] WorkOS also provides a free user management solution called
[5:48] AuthKit for up to 1
[5:49] SPEAKER_01: million monthly active users.
[5:51] It's a drop-in replacement for Auth0 and comes standard with
[5:53] useful features like
[5:54] SPEAKER_01: domain verification,
[5:55] SPEAKER_01: role-based access control,
[5:57] SPEAKER_01: bot protection,
[5:58] SPEAKER_01: and MFA.
[5:59] SPEAKER_01: It's powered by Radix components, which means zero compromises in design.
[6:02] You get limited customizations as well as modular templates
[6:05] designed for quick integrations.
[6:07] SPEAKER_01: Today,
[6:08] SPEAKER_01: hundreds of fast-growing startups are powered by WorkOS,
[6:10] SPEAKER_01: including ones you probably know,
[6:12] SPEAKER_01: like Cursor,
[6:12] SPEAKER_01: Vercel,
[6:13] SPEAKER_01: and Perplexity.
[6:14] SPEAKER_01: Check it out at workos.com to learn more.
[6:17] SPEAKER_01: That is workos.com.
[6:21] This episode is brought to you by Modal,
[6:22] the cloud platform that makes AI development simple.
[6:25] SPEAKER_01: Need GPUs without the headache.
[6:28] SPEAKER_01: With Modal,
[6:28] SPEAKER_01: just add one line of code to any Python function and boom,
[6:31] SPEAKER_01: it's running in the cloud on your choice of CPU or GPU.
[6:34] SPEAKER_01: And the best part, you only pay for what you use.
[6:38] With sub-second container starts and instant scaling to
[6:40] thousands of GPUs,
[6:42] SPEAKER_01: it's no wonder companies like Suno,
[6:43] SPEAKER_01: RAMP,
[6:44] SPEAKER_01: and Substack already trust Modal for their AI applications.
[6:47] SPEAKER_01: Getting an H100 is just a pip install away.
[6:50] SPEAKER_01: Go to modal.com slash pragmatic to get $30 in free credits every month.
[6:55] SPEAKER_01: That is modal.com slash pragmatic.
[6:59] And then so how does a pod and let's say virtual machine
[7:02] connect or like in this case,
[7:03] SPEAKER_01: when we say container,
[7:04] SPEAKER_01: it's probably a virtual machine,
[7:06] SPEAKER_01: right?
[7:06] SPEAKER_00: Nope, there is a difference between a container and a virtual machine.
[7:12] Containers are virtualizing the operating system and
[7:16] applications running within it.
[7:18] A virtual machine is larger and beefier,
[7:21] and it is likely also virtualizing hardware.
[7:27] SPEAKER_00: There are different types of virtualization,
[7:30] SPEAKER_00: and Kubernetes relies on many types of virtualization.
[7:35] Virtual memory is probably the most recent breakthrough in virtualization,
[7:42] SPEAKER_00: I guess,
[7:43] SPEAKER_00: and we cracked that in the 60s,
[7:46] SPEAKER_00: but that's what allows things like containers to exist,
[7:49] SPEAKER_00: virtualized memory.
[7:51] SPEAKER_00: Um, but yeah, a container is a type of virtualization.
[7:54] SPEAKER_00: A virtual machine is a type of virtualization.
[7:56] SPEAKER_00: It's just much, much easier.
[7:58] SPEAKER_00: Um, you, you can run Kubernetes inside of a VM.
[8:02] Like there's nothing stopping you from doing that,
[8:04] but,
[8:05] uh,
[8:05] containers are not VMs themselves.
[8:08] SPEAKER_00: They're, they're smaller, they're more lightweight.
[8:09] SPEAKER_00: They're easier to share around.
[8:11] SPEAKER_00: Um, they're, they're much more configurable as well.
[8:14] SPEAKER_01: Right.
[8:14] SPEAKER_00: Yeah.
[8:15] SPEAKER_00: Yeah.
[8:15] SPEAKER_00: They're, they're,
[8:16] SPEAKER_00: You can just chuck one.
[8:19] The popularity of Docker is largely because of the shareability
[8:24] of Docker files and
[8:26] SPEAKER_00: things like the Docker registry.
[8:27] SPEAKER_00: And now there's a million and one container registries.
[8:31] But shareability,
[8:33] the small compact size of containers is what makes them so attractive.
[8:36] SPEAKER_00: It's why we generally,
[8:39] SPEAKER_00: not always,
[8:39] SPEAKER_00: develop applications as microservices now rather than monoliths.
[8:43] SPEAKER_00: That's the...
[8:45] SPEAKER_00: Containers is what allow us to do that.
[8:48] SPEAKER_00: And the rise in complexity of applications built as microservices is what
[8:54] SPEAKER_00: necessitated something like Kubernetes.
[8:57] SPEAKER_00: You could do all of this before the scaling of applications running as
[9:03] SPEAKER_00: microservices in clusters or swarms.
[9:05] Before we had Kubernetes,
[9:07] it was just very difficult and it was very manual.
[9:10] Kubernetes is just an abstraction on something that we used to
[9:13] do manually.
[9:15] SPEAKER_00: And now since we are automating a lot of it with a very,
[9:19] SPEAKER_00: very complex overlord like Kubernetes,
[9:23] SPEAKER_00: we can scale our applications much larger,
[9:25] SPEAKER_00: much faster.
[9:26] And we can build much more complex applications as much more
[9:30] complex chunks of
[9:31] SPEAKER_00: microservices instead of having to throw a monolith in.
[9:35] So this is like most advances in technology since the ENIAC
[9:41] machine in the 1950s.
[9:43] Kubernetes is just an abstraction layer on something that we
[9:46] used to do by hand
[9:48] SPEAKER_00: that sucked,
[9:49] SPEAKER_00: that was hard to do by hand.
[9:51] SPEAKER_00: And now we've automated it.
[9:52] SPEAKER_00: And the tradeoff there is...
[9:56] SPEAKER_00: When you dig into it, it's confusing.
[9:58] SPEAKER_00: It is confusing.
[10:00] SPEAKER_01: Can we talk about the history?
[10:01] SPEAKER_01: What happened?
[10:02] SPEAKER_01: What was before Kubernetes?
[10:04] SPEAKER_01: And as I understand, the roots are somewhere with Google, right?
[10:07] SPEAKER_01: Because I think it's good to understand what is it that,
[10:10] SPEAKER_01: you know,
[10:10] SPEAKER_01: either sysadmins or other people were doing.
[10:13] SPEAKER_01: And then,
[10:14] SPEAKER_01: you know,
[10:15] SPEAKER_01: why we decided or why some engineers decided like,
[10:17] SPEAKER_01: all right,
[10:17] SPEAKER_01: let's actually build a better tool.
[10:19] SPEAKER_01: And now the whole world just seems to have caught on saying,
[10:21] SPEAKER_01: actually,
[10:22] SPEAKER_01: yeah,
[10:22] SPEAKER_01: we actually want more of this.
[10:23] Yeah,
[10:24] so Google has a history and a reputation of over-engineering things,
[10:28] right?
[10:29] SPEAKER_00: They do.
[10:31] So internal to Google,
[10:33] a long,
[10:33] long time ago,
[10:34] there was a tool called Borg.
[10:37] SPEAKER_00: That is a Star Trek reference, right?
[10:39] SPEAKER_00: You will be assimilated.
[10:40] SPEAKER_00: It's still around.
[10:41] SPEAKER_00: It's still around.
[10:43] And Borg was used internal to Google to manage their own clusters,
[10:48] their own microservices.
[10:50] SPEAKER_00: And somebody decided one day, ooh,
[10:55] SPEAKER_00: this is really good.
[10:55] SPEAKER_00: I think we're onto something.
[10:58] SPEAKER_00: We should open source this.
[11:00] SPEAKER_00: And so they did,
[11:02] and they talked to the Linux Foundation who stood up the Cloud
[11:05] Native Computing
[11:06] SPEAKER_00: Foundation as a directed fund of the LF.
[11:09] And Kubernetes was the first project donated to the Cloud
[11:12] Native Computing Foundation.
[11:13] SPEAKER_00: That was in July, almost 11 years ago.
[11:17] SPEAKER_00: And the first commit was by Joe Beta.
[11:20] SPEAKER_00: A fun thing on the name Kubernetes.
[11:24] SPEAKER_00: It means helmsman in Greek and a typical ship's wheel.
[11:30] Our logo has eight spokes,
[11:33] but ours has seven as a reference to seven of nine from Star Trek.
[11:40] SPEAKER_00: It's like a nod to Borg, the original name.
[11:44] SPEAKER_01: Yeah.
[11:44] SPEAKER_00: Yeah.
[11:46] SPEAKER_00: But it took off pretty quickly.
[11:48] SPEAKER_00: Before that, there were other ways to manage clusters, right?
[11:51] SPEAKER_00: There's Mesosphere and there was Docker Swarm and a couple of others.
[11:56] SPEAKER_00: Like this was very clearly something that the market wanted,
[11:59] SPEAKER_00: that developers needed,
[12:00] SPEAKER_00: that SREs needed,
[12:03] SPEAKER_00: that we didn't really have SRE as a term back then.
[12:06] SPEAKER_00: Back then.
[12:07] Yeah.
[12:08] SPEAKER_00: We certainly do now.
[12:09] SPEAKER_00: SRE is just like a super fancy sysadmin now.
[12:12] But yeah,
[12:14] it was born out of Google solving their own problems and going,
[12:19] you know what?
[12:20] SPEAKER_00: no, people need this, and they decided to ship it for free.
[12:24] And now we're the second largest open source project in the
[12:27] world behind Linux,
[12:30] SPEAKER_01: which we rely on.
[12:32] SPEAKER_01: And as a fun history,
[12:34] SPEAKER_01: I talked with Dave O'Connor,
[12:35] who was one of the first SREs inside of Google when it was
[12:38] internally called like that,
[12:39] SPEAKER_01: but externally the term wasn't popular until much later.
[12:42] SPEAKER_01: And he said this interesting story of how
[12:45] SPEAKER_01: At the time when he joined Google,
[12:47] SPEAKER_01: they already were managing thousands or tens of thousands of servers.
[12:51] And the problems that they were facing is they went to a
[12:54] hosting provider where
[12:56] SPEAKER_01: they wanted to host their servers.
[12:57] SPEAKER_01: And they told them how many they need to manage.
[13:00] SPEAKER_01: And the hosting provider was used to manage maybe 100 at max.
[13:02] SPEAKER_01: And it was very manual.
[13:04] And then he said that they just needed to create the processes,
[13:07] the tools,
[13:07] etc.
[13:08] SPEAKER_01: So my...
[13:09] Is it fair that the understanding is that we had this explosion
[13:12] in both physical servers,
[13:14] SPEAKER_01: then we had virtualization virtual servers,
[13:16] SPEAKER_01: Google figured out how to do it,
[13:18] SPEAKER_01: and then they just turned around and said,
[13:20] SPEAKER_01: okay,
[13:21] SPEAKER_01: let's open it up.
[13:22] SPEAKER_00: Pretty much.
[13:23] SPEAKER_00: Yeah, that's pretty much what happened.
[13:28] SPEAKER_01: Right now,
[13:28] SPEAKER_01: I think it's a really positive thing on Google,
[13:30] SPEAKER_01: but why do you think the motivation might have been for Google?
[13:33] It is amazing that they did it,
[13:35] and the industry has greatly benefited from it.
[13:37] SPEAKER_01: But they could have just said,
[13:38] SPEAKER_01: okay,
[13:38] SPEAKER_01: let's just keep doing what we're doing with Borg and not bother too much.
[13:43] SPEAKER_01: Because they still have Borg, right?
[13:44] SPEAKER_01: Yeah.
[13:44] SPEAKER_00: They do still have Borg.
[13:46] SPEAKER_00: As far as I know, they still have Borg.
[13:47] SPEAKER_00: I haven't asked a Google engineer about the state of Borg in a minute.
[13:50] SPEAKER_00: They have.
[13:50] SPEAKER_00: I recently asked.
[13:52] SPEAKER_00: As far as I know, it's still a thing.
[13:53] SPEAKER_00: Do you mean like why didn't they just like productize Borg and sell it?
[13:58] Either that or just why did they think this would benefit the
[14:02] industry and maybe
[14:03] SPEAKER_01: also their business?
[14:06] SPEAKER_01: Now that we have Google Cloud,
[14:07] SPEAKER_01: I can see how,
[14:08] SPEAKER_01: for example,
[14:09] SPEAKER_01: Kubernetes might have indirectly helped.
[14:11] SPEAKER_01: But I'm just thinking just from a selfish perspective,
[14:14] SPEAKER_01: this is a really unselfish thing from a corporation to do.
[14:18] SPEAKER_01: I'm trying to understand.
[14:18] SPEAKER_01: There's usually a longer term motivation potentially.
[14:21] Or maybe it was just engineers being geeky and thinking this is really,
[14:25] really helpful.
[14:27] I know those engineers quite well,
[14:28] and they are massive dweeps,
[14:31] and I love them.
[14:32] SPEAKER_00: Awesome.
[14:34] SPEAKER_00: So yeah,
[14:35] I totally get how people would think that open sourcing
[14:38] something as successful as
[14:40] SPEAKER_00: Kubernetes may not be a smart business decision.
[14:45] But look at the sheer amount of influence Google has gained
[14:51] over the cloud native
[14:52] SPEAKER_00: space over that.
[14:54] They still have employees sitting on the technical oversight
[14:57] committee for the CNCF.
[14:58] SPEAKER_00: They have Linux Foundation governing board members.
[15:01] SPEAKER_00: They have CNCF governing board members.
[15:03] They have people...
[15:04] I don't think they have somebody sitting on...
[15:06] Oh,
[15:06] no.
[15:06] They do have somebody currently sitting on the Kubernetes
[15:08] steering committee.
[15:09] SPEAKER_00: They still have a degree of control.
[15:12] SPEAKER_00: The way we govern ourselves as a project,
[15:16] no one company can have more than two representatives on the
[15:19] steering committee.
[15:19] SPEAKER_00: So they can't wrest control of the project from the people.
[15:23] SPEAKER_00: anymore, like any more than AWS or Microsoft or Red Hat can.
[15:28] SPEAKER_00: But it has given them a pretty extreme degree of influence.
[15:34] SPEAKER_00: I mean,
[15:34] we're here on this podcast talking about the fact that Google
[15:38] gave us Kubernetes,
[15:39] SPEAKER_00: Google's directly responsible.
[15:41] SPEAKER_01: That's fair.
[15:42] I'm trying to understand because I think it's great that this
[15:45] is happening.
[15:45] SPEAKER_01: And I think the more we understand what the benefits are and why other
[15:49] companies potentially should follow for example spotify that
[15:52] was backstage their
[15:54] developer portal and what they told me is they think it will
[15:57] make it cheaper on the
[15:58] long term for them to actually maintain themselves because now
[16:01] they're going to get
[16:02] other developers from other companies or independent engineers
[16:06] bringing the ideas
[16:07] and also standardization is is another thing so yeah if your
[16:11] project is good people
[16:13] SPEAKER_00: will come contribute to it um i
[16:17] SPEAKER_00: We open source maintainers have mixed feelings about this,
[16:19] SPEAKER_00: but it is a way to get free labor,
[16:23] SPEAKER_00: right?
[16:23] SPEAKER_00: Nobody pays me to maintain Kubernetes.
[16:27] SPEAKER_00: It's not my job.
[16:29] I don't have a cloud provider writing me a paycheck for running
[16:31] the Kubernetes releases.
[16:33] SPEAKER_00: So we do do all of this work for free.
[16:38] Another benefit is that open source software is inherently more
[16:42] secure than closed
[16:43] SPEAKER_00: source proprietary software.
[16:46] SPEAKER_00: That security angle is real hard to argue with.
[16:50] SPEAKER_00: Kubernetes just addressed a nasty CVE very recently in Ingress Nginx,
[16:56] and that would be much slower to address on proprietary
[17:01] software with a smaller
[17:03] SPEAKER_00: team that can't work around the clock.
[17:05] You can't force your employees to work around the clock if it's
[17:09] a small group of
[17:10] SPEAKER_00: people that are all in one time zone.
[17:12] SPEAKER_00: And open source projects,
[17:13] SPEAKER_00: especially one as large as Kubernetes,
[17:15] SPEAKER_00: we can respond around the clock.
[17:18] SPEAKER_01: And how large is Kubernetes,
[17:21] SPEAKER_01: both in terms of the widespread in the industry,
[17:25] SPEAKER_01: the complexity of the software and the team,
[17:27] SPEAKER_01: as you refer to?
[17:32] SPEAKER_00: First,
[17:33] I think I need to explain the different types of people
[17:35] involved in an open source project.
[17:38] SPEAKER_00: Generally, you have users who are people who just consume your tool.
[17:42] SPEAKER_00: You have contributors who are writing code or writing documentation
[17:47] Or in the case of the release team doing some like project
[17:51] management type of role,
[17:53] but they don't have any control over the future or direction of
[17:57] the project and
[17:59] SPEAKER_00: have like maybe limited permissions within repositories or orgs.
[18:03] SPEAKER_00: Maintainers have titles within a project.
[18:07] So SigDocs,
[18:08] Technical Lead,
[18:09] Release Team,
[18:09] Subproject Lead are both titles.
[18:12] You exist in an owner's file and you have control over
[18:15] governance and policy
[18:16] SPEAKER_00: decisions about the project that affect everybody.
[18:21] SPEAKER_00: There are a couple dozen SIGs within Kubernetes,
[18:26] SPEAKER_00: all with two to three chairs and two to four technical leads.
[18:34] SPEAKER_00: Those people would all be considered maintainers.
[18:37] then sub-project owners exist below that before you get down to
[18:41] regular contributors.
[18:42] SPEAKER_00: So there's probably... It's a large project.
[18:45] SPEAKER_00: And everybody in SIG leadership is a project manager, functionally.
[18:49] SPEAKER_00: We're all doing administrative work.
[18:51] SPEAKER_00: We're all managing people.
[18:54] We're all affecting the direction of our various areas of the
[18:58] Kubernetes project.
[18:59] SPEAKER_00: And there's probably...
[19:02] SPEAKER_00: 150,
[19:03] 200 people who would be classed as maintainers and then more
[19:08] than a thousand
[19:09] SPEAKER_00: contributors a month.
[19:10] SPEAKER_01: It is a month.
[19:12] SPEAKER_01: Wow.
[19:12] SPEAKER_00: It's massive.
[19:13] SPEAKER_01: So about like a few dozen, let's say like 20 to 40 SIGs.
[19:19] SPEAKER_01: leaders, about 150, 200 maintainers, and like a little pyramid almost.
[19:23] SPEAKER_01: And then the contributors who are,
[19:25] SPEAKER_01: I'm assuming contributors,
[19:27] SPEAKER_01: some of them will be regular,
[19:28] SPEAKER_01: some of them will be just one-off.
[19:29] SPEAKER_00: Yeah, some of them are one-off.
[19:30] SPEAKER_00: Sometimes we get drive-by contributions like any other big project,
[19:34] SPEAKER_00: and some people just stick around as contributors.
[19:37] SPEAKER_00: And you go through what we call a contributor ladder.
[19:40] SPEAKER_00: So you show up as a contributor at first,
[19:42] you work as a contributor for a while in whatever area
[19:45] interests you most,
[19:46] SPEAKER_00: and
[19:47] And if you stick around long enough and you want it,
[19:49] you will eventually become a maintainer.
[19:51] You'll end up leading a sub project or you'll end up as a
[19:56] technical lead or a SIG chair.
[19:58] SPEAKER_00: That's what happened to me.
[19:59] I was on the Kubernetes release team for three years and then I
[20:04] led a release myself.
[20:06] SPEAKER_00: Kubernetes version 130 was my release.
[20:09] SPEAKER_00: And after that, the SIG release chairs offered me the sub project.
[20:15] SPEAKER_00: And now I just keep running them.
[20:17] SPEAKER_00: And now I'm a maintainer.
[20:18] And because I had run the docs sub team for the Kubernetes
[20:24] release a couple of times,
[20:25] SPEAKER_00: SIG docs asked me to be a technical lead.
[20:27] SPEAKER_00: And that's just, that's how it works.
[20:29] SPEAKER_00: Anybody can show up.
[20:30] SPEAKER_00: Anybody can open an issue.
[20:32] SPEAKER_00: Anybody can...
[20:35] SPEAKER_00: Open a pull request and dive in and help out.
[20:39] SPEAKER_00: You don't even really have to be a Kubernetes expert to be useful.
[20:42] You just have to know how software works or know how to write
[20:45] documentation or know
[20:47] SPEAKER_00: how to manage a team.
[20:50] SPEAKER_01: So Kubernetes,
[20:52] SPEAKER_01: widespread-wise,
[20:53] SPEAKER_01: as I understand,
[20:54] SPEAKER_01: most of the large infrastructure runs on Kubernetes.
[20:58] SPEAKER_01: If it's large, it's the one that makes the most sense.
[21:01] I'm not even sure there's a competitor that is open source that
[21:05] is equally capable.
[21:09] SPEAKER_00: Not that I would say is equally capable.
[21:11] SPEAKER_00: There are certainly other ways to manage clusters.
[21:14] SPEAKER_00: Docker swarm is still a thing.
[21:16] SPEAKER_00: Mesos is still a thing,
[21:18] SPEAKER_00: but yeah,
[21:19] like your people are most commonly using some kind of managed
[21:23] Kubernetes service.
[21:25] SPEAKER_00: Like, um,
[21:26] SPEAKER_00: anything on AWS or GKE, right?
[21:30] SPEAKER_00: Or like Azure has their own service too.
[21:32] SPEAKER_00: I'm pretty sure Oracle does.
[21:34] SPEAKER_00: Red Hat certainly does.
[21:36] SPEAKER_00: And then there are flavors of Kubernetes like VMware's Tanzu, right?
[21:43] And so given that this project is so successful,
[21:47] how does the funding work?
[21:48] SPEAKER_01: I'm going to assume that some of the contributors,
[21:51] SPEAKER_01: as you mentioned,
[21:52] are actually paid full time working at certain companies who
[21:55] have critical
[21:56] SPEAKER_01: infrastructure running on Kubernetes.
[21:58] And these companies figure,
[21:59] well,
[21:59] it's cheaper for us to pay a contributor to...
[22:03] SPEAKER_01: ensure that our use cases are met.
[22:05] SPEAKER_01: But for example,
[22:06] SPEAKER_01: in your case and some of the other contributors,
[22:09] SPEAKER_01: how does it make sense to spend a considerable amount of your time?
[22:12] We're talking about software engineers who are highly skilled
[22:17] spending here,
[22:17] SPEAKER_01: and there will be some goodwill,
[22:19] SPEAKER_01: but what are other ways that the project remains as sustainable as it is?
[22:27] SPEAKER_00: Unfortunately, for a lot of us, this is a hobby, right?
[22:32] SPEAKER_00: I enjoy it.
[22:35] SPEAKER_00: Many of my friends are other maintainers or other contributors.
[22:39] SPEAKER_00: Some of my closest friends are also maintainers.
[22:43] SPEAKER_00: I flew to New York just to hang out with two of them a couple weeks ago.
[22:46] SPEAKER_00: No reason.
[22:47] SPEAKER_00: Not work.
[22:48] SPEAKER_00: Just hanging out with them.
[22:49] SPEAKER_00: And that's not too terribly uncommon.
[22:54] SPEAKER_00: People just think it's fun.
[22:56] SPEAKER_00: Some people also do it as a career bump.
[22:59] This is something that the Kubernetes project doesn't really
[23:01] love to admit,
[23:03] SPEAKER_00: but we are a very,
[23:04] SPEAKER_00: very large and very,
[23:05] SPEAKER_00: very influential project.
[23:08] SPEAKER_00: And having it on your resume can be a career bump,
[23:11] especially for people who are fresh out of college or very
[23:14] early in their career,
[23:15] SPEAKER_00: which is why
[23:16] Shadowing on the Kubernetes release team is such a popular
[23:19] entry point for contributors.
[23:21] We get hundreds of applications for the Kubernetes release team
[23:24] every cycle,
[23:25] SPEAKER_00: and about half of them are students.
[23:29] SPEAKER_00: And that's pretty common, too.
[23:30] SPEAKER_00: They want the career experience.
[23:33] SPEAKER_01: That is encouraging to hear, by the way.
[23:35] SPEAKER_00: Yeah, it's a lot of students.
[23:37] SPEAKER_00: It is so many students.
[23:39] SPEAKER_00: And we do get students onto the team.
[23:42] SPEAKER_00: Obviously,
[23:42] SPEAKER_00: I can't overload the release team with people who have no experience,
[23:46] SPEAKER_00: but we try to fit a handful of them in every single cycle.
[23:51] SPEAKER_00: Which feels good.
[23:52] SPEAKER_00: It's a good entry point for people.
[23:53] So we get to watch them grow up through the project until
[23:56] ultimately they go get a
[23:58] SPEAKER_00: job and leave and we never hear from them again,
[24:00] which is fine and doesn't make us feel bad because we really
[24:02] have more contributors
[24:04] SPEAKER_00: on the release team side than we know what to do with.
[24:06] SPEAKER_00: We cannot have them all on the team.
[24:10] But as for people who are paid for it,
[24:14] that's usually cloud vendors,
[24:16] actually.
[24:18] The big clouds pay a lot of people just to be contributors or
[24:24] just to be maintainers,
[24:25] either because they need something done or they want the
[24:28] influence or because they
[24:30] SPEAKER_00: want the clout.
[24:32] Some people get hired by some company just because they have a
[24:35] maintainer title
[24:36] SPEAKER_00: within the project.
[24:38] SPEAKER_00: And that's it.
[24:39] SPEAKER_00: It's purely clout.
[24:40] SPEAKER_00: And I don't object to that either,
[24:42] as long as somebody's getting paid to do this instead of doing
[24:45] it for free.
[24:46] SPEAKER_00: But the CNCF doesn't pay us.
[24:48] SPEAKER_00: The Linux Foundation doesn't.
[24:49] SPEAKER_00: doesn't pay us.
[24:51] SPEAKER_00: The CNCF does pay our infrastructure costs.
[24:53] So we're not starting a GoFundMe or putting money out of our
[24:57] own pockets to pay for
[24:58] SPEAKER_00: our testing or whatever.
[25:00] SPEAKER_00: The CNCF pays that.
[25:01] SPEAKER_00: That's part of what you get when you donate a project to the CNCF.
[25:05] SPEAKER_00: They take on some of your infrastructure costs,
[25:07] SPEAKER_00: depending on where you are in the sandbox incubating graduated scale.
[25:12] SPEAKER_00: But yeah, most of us do this for fun.
[25:16] SPEAKER_00: Which is not the most healthy thing in the world.
[25:19] SPEAKER_00: And you have to set like really, really, really firm boundaries.
[25:23] SPEAKER_01: Well, yeah, there's that thing that comes into mind.
[25:26] And also there's this famous XKCD comic where there's this big
[25:29] pile of rocks and
[25:30] SPEAKER_01: there's this small thing which says someone's side project.
[25:33] SPEAKER_01: So given how critical Kubernetes is becoming...
[25:37] SPEAKER_01: Globally, I think it's a bit of a mixed feelings.
[25:41] SPEAKER_01: On one end,
[25:42] I think it is cool to hear that it sounds like it is a very
[25:46] independent project
[25:47] SPEAKER_01: that doesn't really have any one vendor going over it.
[25:51] SPEAKER_01: On the other, there is this downside of like, well, you know, like...
[25:57] SPEAKER_00: Yeah, it just makes feelings.
[26:00] It makes me wince for small projects,
[26:03] but everybody starts as a small project.
[26:05] However,
[26:05] Kubernetes is so large and so influential and we have so much momentum.
[26:09] SPEAKER_00: That XKCD comic is not going to become a problem.
[26:13] There is a queue of people waiting in the wings who would love
[26:18] to replace me.
[26:19] SPEAKER_00: And part of our,
[26:20] SPEAKER_00: as a project,
[26:21] our philosophy is that part of our job as maintainers is to be
[26:25] looking for somebody
[26:27] SPEAKER_00: to mentor into replacing us.
[26:30] SPEAKER_00: We have a lot of anti-burnout policies in place.
[26:33] um leading a release like being being the actual point person
[26:37] for a release the the
[26:38] person who ultimately reports reports to me is a very stressful
[26:42] very public
[26:44] miserable job to do it is 14 weeks of having to manage and 14
[26:49] weeks that's the
[26:50] release cycle that's the release cycles yeah like 14 to 16
[26:53] weeks it depends on when
[26:54] SPEAKER_00: it is in the year but 14 to 16 weeks is about how long it takes us
[26:58] And it's 14 weeks of directly managing an unruly group of like
[27:04] 30 people who are
[27:06] trying to manage interactions with potentially hundreds of
[27:10] contributors between them.
[27:12] SPEAKER_00: And it sucks.
[27:13] SPEAKER_00: And by the end of it, you are so tired.
[27:15] So we don't allow you to be involved in the release cycle after
[27:20] you leave one.
[27:21] SPEAKER_00: You've got to take a cycle off.
[27:24] SPEAKER_00: Also, SIG release's charter dictates that we will delay a release.
[27:29] SPEAKER_00: We will push it back before we make our team work nights or weekends.
[27:34] We're never going to ask you to work more than you are
[27:37] comfortable working.
[27:39] SPEAKER_00: And we're very supportive of stepping back if you can't handle it.
[27:43] SPEAKER_00: It doesn't affect your place in the project.
[27:45] SPEAKER_00: You can come back the next cycle or whenever you're ready.
[27:49] SPEAKER_00: We're very encouraging of taking breaks.
[27:52] So we don't really like to allow the burnout to build up to a
[27:56] point where we might
[27:57] SPEAKER_00: topple over like that XKCD comic.
[27:58] SPEAKER_00: Yeah.
[27:59] SPEAKER_00: Um, we're always replacing each other.
[28:01] SPEAKER_01: So like eventually this part is, is just very encouraging to hear.
[28:06] SPEAKER_01: And this is something that I would,
[28:08] I would assume that a lot of commercial companies like a
[28:11] commercial setup would,
[28:13] SPEAKER_01: uh,
[28:14] let me just say hesitate in,
[28:16] in,
[28:17] in many cases again,
[28:18] because of that thing.
[28:18] SPEAKER_01: So,
[28:19] SPEAKER_01: so I guess maybe we're seeing two parts of it,
[28:21] SPEAKER_01: but this is just,
[28:22] SPEAKER_01: it's the first time I'm actually hearing for such a large,
[28:26] SPEAKER_01: such a critical project to actually have it in place,
[28:28] SPEAKER_01: which is just music to my ears.
[28:30] SPEAKER_00: We're unusually well-structured for an open source project.
[28:34] We're,
[28:34] we're unusually well-structured for like an actual product
[28:38] within a company.
[28:39] SPEAKER_00: Actually, we,
[28:41] SPEAKER_00: are very,
[28:41] very well organized by necessity because we're massive and we
[28:45] deal with so many
[28:46] SPEAKER_00: people coming in and out constantly.
[28:48] SPEAKER_00: We have no choice but to be super, super well organized.
[28:52] SPEAKER_01: So why do you think Kubernetes,
[28:56] SPEAKER_01: you've seen the story mostly from the inside and starting from the side,
[29:00] SPEAKER_01: why do you think Kubernetes is one so big?
[29:02] SPEAKER_01: Why has it become the de facto choice from any medium size,
[29:07] SPEAKER_01: sometimes even small size infrastructure,
[29:09] SPEAKER_01: by the way?
[29:09] SPEAKER_01: I've talked with Linear and they chose Kubernetes very early on.
[29:13] SPEAKER_01: Apparently,
[29:15] SPEAKER_01: they thought,
[29:15] SPEAKER_01: you know,
[29:16] SPEAKER_01: some people thought this would be like overkill,
[29:18] SPEAKER_01: but it actually turned out to be a great decision for them.
[29:20] SPEAKER_00: Yeah.
[29:22] SPEAKER_01: What do you think this project specifically won?
[29:25] SPEAKER_01: What did Kubernetes do so well?
[29:27] SPEAKER_00: I think we caught on early because of hype, right?
[29:32] Because of the Google name brand behind us and the association
[29:37] with Docker,
[29:37] SPEAKER_00: which was already very popular.
[29:40] It was a familiar company donating a project that relied on a
[29:44] tool many were
[29:45] SPEAKER_00: already familiar with.
[29:46] SPEAKER_00: And that got us initial, like,
[29:49] SPEAKER_00: hype cycle, right?
[29:50] But then I'm going to toot my own horn and say that the
[29:54] continued popularity of
[29:55] Kubernetes is at least in part due to our truly exceptional documentation.
[30:01] SPEAKER_00: If Kubernetes does something that you can touch as a user,
[30:06] SPEAKER_00: as a cluster admin,
[30:07] SPEAKER_00: we have documented it.
[30:09] Kubernetes uses something in the release cycle called a
[30:12] Kubernetes enhancement proposal,
[30:14] a KEP.
[30:15] SPEAKER_00: This was inspired by Python's PEPs, right?
[30:18] And one of the things we require for a KEP to be considered
[30:21] complete and thus
[30:23] includable in a particular release is that if it has a
[30:26] user-facing change,
[30:28] SPEAKER_00: at all.
[30:28] Even if it's just a feature flag,
[30:31] it must be documented or we do not allow it in the release.
[30:34] Today,
[30:34] as we're recording this,
[30:36] is actually the docs freeze for Kubernetes version 1.33.
[30:40] So today,
[30:41] a lot of camp owners will either merge documentation or I will
[30:46] revert their PRs.
[30:48] SPEAKER_01: Wow.
[30:49] SPEAKER_01: That sounds like tyranny.
[30:52] SPEAKER_01: Does it?
[30:53] SPEAKER_01: Does it?
[30:54] You're never left digging through Kubernetes code to figure out
[30:57] what this is?
[30:59] SPEAKER_01: It sounds amazing as a user and as a developer.
[31:02] SPEAKER_01: I'm cringing.
[31:03] SPEAKER_01: You know, as a dev, like you really just don't want.
[31:06] SPEAKER_01: There's so many excuses we make up for not writing documents.
[31:09] SPEAKER_01: A code will document itself.
[31:11] SPEAKER_01: If it's the documentation, it's not, we're rich and et cetera.
[31:14] SPEAKER_01: No, I'm really impressed.
[31:16] SPEAKER_00: Seriously.
[31:17] SPEAKER_00: Yeah, I know the devs hate it.
[31:18] SPEAKER_00: I know.
[31:19] SPEAKER_00: But our priority isn't making the devs happy.
[31:24] It's building a usable,
[31:27] sustainable project that people want to keep using.
[31:31] SPEAKER_00: That means we got to document that.
[31:33] SPEAKER_00: I hated writing docs when I was an engineer, too.
[31:36] SPEAKER_00: I did.
[31:37] SPEAKER_00: I've slung some truly horrendous code that was entirely undocumented.
[31:42] And I myself looking at it probably could not tell you what on
[31:45] earth it's doing.
[31:47] SPEAKER_00: So we can't have that in a project this large.
[31:50] SPEAKER_01: Yeah, and the problem is that this is complex, right?
[31:53] SPEAKER_01: So it sounds like you're managing to balance.
[31:55] Because what I understand is Kubernetes is just going to become
[31:57] more and more complex.
[31:58] SPEAKER_01: It does.
[32:00] SPEAKER_01: You can tell them more about it,
[32:01] SPEAKER_01: but there's going to be more edge cases,
[32:03] SPEAKER_01: more hardware,
[32:03] SPEAKER_01: more setup,
[32:04] SPEAKER_01: more exotic this and that.
[32:07] SPEAKER_01: But when you have proper documentation, at least that helps with it.
[32:10] SPEAKER_01: If you don't have it, there's no chance.
[32:12] SPEAKER_00: Yeah, there's absolutely no chance.
[32:14] SPEAKER_00: And so that has gotten us over the hump of Kubernetes is hard to use.
[32:18] SPEAKER_00: It is hard to use.
[32:19] SPEAKER_00: It's hard to learn, but it could be so much worse.
[32:22] SPEAKER_00: And the availability of good documentation is...
[32:26] SPEAKER_00: I give a lot of credit to that.
[32:30] SPEAKER_00: Also, GKE has really great tutorials for learning Kubernetes.
[32:35] SPEAKER_01: GKE is Google Kubernetes Engine, right?
[32:37] SPEAKER_01: It's their hosted solution?
[32:38] SPEAKER_00: Yeah, it's their hosted solution.
[32:40] And they absolutely have the most feature-rich
[32:43] Kubernetes-hosted option by virtue of having had.
[32:47] SPEAKER_00: several years of a head start on that.
[32:51] SPEAKER_00: But yeah, they're tutorials.
[32:52] They're just like how to learn Kubernetes are actually quite
[32:55] good for just dipping
[32:57] SPEAKER_00: your toes in the water and our own documentation,
[33:02] SPEAKER_00: our own quick start guides.
[33:03] SPEAKER_00: That's another benefit also of being an open source project.
[33:07] SPEAKER_00: Just like an open source project, the software is more secure.
[33:09] SPEAKER_00: Our documentation can be better.
[33:11] We have a small army of people contributing to it and improving
[33:16] our documentation.
[33:17] SPEAKER_00: on a constant basis.
[33:19] SPEAKER_00: So we don't really tend to slip.
[33:23] SPEAKER_01: I'm not sure if this applies,
[33:24] SPEAKER_01: correct me if I'm not,
[33:25] SPEAKER_01: but a lot of open source projects say like,
[33:27] SPEAKER_01: look,
[33:28] SPEAKER_01: if you want to start contributing and you're not sure just yet,
[33:31] SPEAKER_01: you can always just start with the documentation.
[33:33] SPEAKER_00: Yes.
[33:34] SPEAKER_00: It's an incredible entry point.
[33:35] SPEAKER_01: Which is a win-win.
[33:36] SPEAKER_00: Yeah.
[33:37] Docs are a pretty typical entry point for people who are
[33:40] nervous about contributing
[33:41] SPEAKER_00: to open source for the first time,
[33:43] SPEAKER_00: which is a fear I absolutely get.
[33:44] SPEAKER_00: I was terrified the first time I opened a PR in a Kubernetes repo.
[33:50] SPEAKER_00: It's nerve-wracking, especially if you're super early in your career.
[33:53] You know that you've got all of these people with way more
[33:55] experience than you
[33:56] SPEAKER_00: laser-focused on what you just did.
[33:59] SPEAKER_00: And did you apply a label incorrectly?
[34:01] SPEAKER_00: Are you going to get chewed out for that?
[34:04] SPEAKER_00: But yeah, docs are an easier entry point for that.
[34:07] SPEAKER_00: It feels like a little bit less pressure.
[34:08] And that's pretty common across open source projects,
[34:11] not just Kubernetes.
[34:13] The unique thing about Kubernetes and getting started is the
[34:16] release team.
[34:17] SPEAKER_00: We will let absolutely anybody apply to be on the release team.
[34:23] SPEAKER_00: And it's...
[34:24] SPEAKER_00: 20, 25 spots every cycle, three times a year.
[34:29] SPEAKER_00: So it's pretty competitive, but we will let absolutely anybody apply.
[34:35] SPEAKER_00: You don't have to have contributed to Kubernetes at all before.
[34:39] Every cycle,
[34:39] I've got to add two or three people to the org because they've never...
[34:43] SPEAKER_01: But just to emphasize, do I understand correctly?
[34:48] SPEAKER_01: For example,
[34:49] we recently had an episode with Linux where in order for you to
[34:54] lead a release,
[34:56] SPEAKER_01: you will need to put in many,
[34:57] SPEAKER_01: many years outstanding contributions and so on.
[35:00] SPEAKER_01: And there's no guarantee of doing so.
[35:01] SPEAKER_01: Correct.
[35:02] But for other open source projects,
[35:03] it is also common to have this kind of tenure-based thing.
[35:07] SPEAKER_01: Because what you've said, I've not heard it.
[35:09] And any other projects say,
[35:11] like,
[35:11] we will actually open up the spots and you can apply.
[35:14] And taking up people with little tone of experience sounds like
[35:19] an approach I've
[35:20] SPEAKER_01: just not heard before.
[35:22] The application process we have and the way the release team
[35:25] works is unusual for
[35:27] SPEAKER_00: an open source project.
[35:29] SPEAKER_00: No, tell me more.
[35:29] SPEAKER_00: So we talk about that a lot.
[35:32] SPEAKER_00: I will absolutely talk about the structure of the release team.
[35:34] SPEAKER_00: So the Kubernetes release team has two halves.
[35:37] SPEAKER_00: There's the release team, which I manage, and release engineering.
[35:42] SPEAKER_00: Release engineering is doing the actual cutting of the releases.
[35:47] But the part of the team I manage is made up of several sub
[35:51] teams and a release lead.
[35:54] The sub teams are communications,
[35:56] which is responsible for handling things like feature blogs.
[35:59] So that's working with cap owners or sig leads to make sure
[36:04] that a particularly
[36:04] interesting feature that will make it into that release has a
[36:08] blog that goes out on
[36:10] SPEAKER_00: the Kubernetes blog on a certain schedule.
[36:12] SPEAKER_00: Oh, nice.
[36:13] SPEAKER_00: Yeah, it's very cool.
[36:14] SPEAKER_00: We usually have like between eight to 12 every cycle that end up.
[36:18] SPEAKER_00: And release comms does not write that blog.
[36:20] SPEAKER_00: They're just responsible for managing that it happens.
[36:24] They're also responsible for managing interactions with the
[36:27] Cloud Native Computing
[36:28] Foundation to schedule the release webinar and with the media
[36:32] to handle the webinar.
[36:34] SPEAKER_00: Yeah, we have a release webinar.
[36:36] We also have to manage the media to deal with press embargoes
[36:40] and schedule media
[36:41] SPEAKER_00: interviews for the release lead and anybody else.
[36:44] SPEAKER_01: This sounds like a publicly traded company doing its earnings call.
[36:48] SPEAKER_01: It does, doesn't it?
[36:48] SPEAKER_01: Yeah.
[36:50] SPEAKER_00: We also have release docs.
[36:51] SPEAKER_01: I'm so impressed.
[36:52] SPEAKER_00: Yeah, it's a lot.
[36:54] Release docs manages making sure that every one of those caps
[36:58] that has user facing
[36:59] SPEAKER_00: changes has documentation by its deadline.
[37:03] SPEAKER_00: Enhancements.
[37:04] Manages all of those caps and makes sure that they have met all
[37:09] of the requirements,
[37:10] SPEAKER_00: the checkboxes in the cap,
[37:11] SPEAKER_00: which is like,
[37:12] SPEAKER_00: has the lead opted in?
[37:14] SPEAKER_00: Is your code complete?
[37:15] SPEAKER_00: Is it tested?
[37:17] SPEAKER_00: Did you get a production readiness review?
[37:19] SPEAKER_00: That kind of thing.
[37:20] SPEAKER_00: And release signal,
[37:23] which is responsible for watching our CI signal boards and
[37:28] chasing down SIGs and
[37:29] cap owners to squash bugs to make sure that we don't have a
[37:32] master blocking test
[37:33] SPEAKER_00: failing right before we cut beta 2.
[37:37] They're also responsible for the go-no-go signal on whether or
[37:40] not we can cut a release.
[37:41] SPEAKER_00: And this requires...
[37:45] SPEAKER_00: 20 to 30 people every single cycle.
[37:48] SPEAKER_00: Five of them are leads and are selected.
[37:52] Then the rest are shadows that go through the application
[37:56] process and get pulled in.
[37:58] SPEAKER_00: That is unusual.
[37:59] SPEAKER_00: Most open source projects do not operate that way.
[38:03] SPEAKER_00: Most open source projects, I think, don't need to.
[38:05] That's a consequence of the fact that the project is so large
[38:08] and we have so many contributors.
[38:10] We couldn't manage this with a consistent group of like five to
[38:14] 10 people without
[38:15] SPEAKER_00: catastrophically burning them all out.
[38:18] SPEAKER_00: We just simply couldn't do it.
[38:21] SPEAKER_00: And even with the rotating cast,
[38:23] the release team can be a burnout factory because it's so much
[38:27] people management.
[38:28] SPEAKER_00: It's so much project management.
[38:29] It's so much arguing with people and juggling like maybe
[38:32] conflicting priorities between teams.
[38:35] SPEAKER_00: So we don't like people to stay on the release team for too long.
[38:39] We want to see you move up through the ranks a little,
[38:41] maybe lead a sub team and then leave.
[38:43] SPEAKER_00: Find somewhere else in the project to go.
[38:45] SPEAKER_00: Because if you stick around, you will burn out.
[38:48] SPEAKER_00: I did not, obviously, which is why they gave me the bigger hat.
[38:53] It is very interesting for me to reflect on how much project
[38:57] management and just
[38:57] SPEAKER_01: how much work management is happening within Kubernetes.
[39:00] Whereas when we did the deep dive with the Linux kernel,
[39:04] it was a bit of the opposite.
[39:05] SPEAKER_01: But again,
[39:05] SPEAKER_01: it was pretty clear there that in the case of Linux kernel,
[39:09] a lot of this work that you're mentioning is happening at the
[39:11] actual releases.
[39:12] SPEAKER_01: Let's say, you know, like Red Hat and whoever picks it up.
[39:16] But I think it's just a good reminder of this work for any
[39:20] large project.
[39:21] SPEAKER_01: It's there.
[39:21] SPEAKER_01: You could shift it here and there, and you can play with it.
[39:26] SPEAKER_00: It has to happen.
[39:29] No project with more than two people contributing to it is
[39:33] getting by without some
[39:34] SPEAKER_00: kind of project management work.
[39:36] SPEAKER_00: It's just that we don't have the title project manager.
[39:38] SPEAKER_00: All maintainers are project managers to some degree.
[39:42] We all have some administrative or governance policy position
[39:50] within the project.
[39:51] SPEAKER_00: It is project management.
[39:52] SPEAKER_00: If you are a maintainer, we just don't call them that.
[39:56] SPEAKER_00: Yeah.
[39:56] SPEAKER_00: So, like, the Linux kernel has them, too.
[39:58] SPEAKER_00: Like, somebody is doing that work.
[40:00] SPEAKER_00: They're just not.
[40:01] SPEAKER_01: They don't like that word, I think.
[40:04] SPEAKER_01: That was my sense.
[40:05] SPEAKER_00: Yeah, they don't.
[40:06] I think a lot of engineers are resistant to doing work that
[40:12] they don't consider
[40:13] SPEAKER_00: real engineering.
[40:14] SPEAKER_00: Yeah.
[40:14] SPEAKER_00: You know, get over it.
[40:17] SPEAKER_01: We're kind of seeing this,
[40:18] SPEAKER_01: by the way,
[40:18] SPEAKER_01: like now,
[40:20] SPEAKER_01: you know,
[40:20] SPEAKER_01: like software engineers,
[40:21] SPEAKER_01: like we don't like to say that we do product work,
[40:23] SPEAKER_01: but now product engineer is becoming a thing,
[40:25] SPEAKER_01: which is a software engineer who will talk with customers as well.
[40:29] SPEAKER_01: And who and, you know, same thing.
[40:31] SPEAKER_01: I think there used to be a time,
[40:32] SPEAKER_01: not these days,
[40:33] SPEAKER_01: but maybe 10 or 15 years ago,
[40:35] SPEAKER_01: testing was we had dedicated,
[40:37] SPEAKER_01: you know,
[40:38] SPEAKER_01: testers or whoever and then.
[40:40] SPEAKER_00: Yeah.
[40:42] SPEAKER_01: engineers wouldn't or would object to do testing.
[40:44] SPEAKER_01: But then we would write unit tests and then integration and automate.
[40:48] SPEAKER_01: And they're dispersed.
[40:49] SPEAKER_01: So I think there's clearly a play with words.
[40:51] SPEAKER_01: And what I'm personally seeing is a competent tech professional,
[40:57] SPEAKER_01: may that be a software engineer or someone who is elsewhere,
[41:00] SPEAKER_01: will just take on more and more responsibilities.
[41:02] SPEAKER_01: A great product manager these days will know how or will understand code.
[41:06] SPEAKER_01: They might not code by choice.
[41:08] SPEAKER_01: But they can read it.
[41:10] SPEAKER_01: Same thing,
[41:10] SPEAKER_01: but even just talking with you,
[41:12] SPEAKER_01: you've been a software engineer,
[41:13] SPEAKER_01: you've actually built critical infrastructure,
[41:15] SPEAKER_01: and now you're doing something where we're doing less of that.
[41:19] SPEAKER_00: Yeah, I haven't written a line of code in months.
[41:22] SPEAKER_00: I don't have a reason to.
[41:24] you could if you wanted to or if you needed to yeah like for
[41:28] fun sometimes like
[41:29] i'll do some hobby projects at home or whatever but i don't
[41:32] have a reason to write
[41:32] code i used to be an engineer i can read and understand it just
[41:35] fine i can write it
[41:36] if i need to um but it's not it's not part of my job anymore
[41:40] and like we you see
[41:42] this within engineering at real companies right the more senior
[41:45] your engineering
[41:46] title is the less code writing you're actually doing a
[41:49] principal engineer should
[41:50] SPEAKER_00: not have their
[41:51] fingers in the code nearly as much as somebody who's like an SDE2,
[41:57] right?
[41:58] SPEAKER_00: Like they don't.
[41:59] They're making architectural decisions about the product or
[42:03] they're mentoring juniors.
[42:06] SPEAKER_00: So like same deal within an open source project.
[42:09] SPEAKER_00: I'm the open source project equivalent of a principal engineer.
[42:13] SPEAKER_00: I don't write code anymore.
[42:15] SPEAKER_00: I make decisions about how we get this thing out the door.
[42:17] SPEAKER_01: This episode is brought to you by Cortex.io.
[42:20] SPEAKER_01: Still tracking your services and production readiness in a spreadsheet.
[42:23] SPEAKER_01: Real Microservices is named after TV show characters.
[42:26] SPEAKER_01: You aren't alone.
[42:27] SPEAKER_01: Being woken up at 3 a.m.
[42:29] for an incident and trying to figure out who owns what service,
[42:31] that's no fun.
[42:33] SPEAKER_01: Cortex is the internal developer portal that solves service ownership and
[42:36] SPEAKER_01: accelerates the path to engineering excellence.
[42:39] SPEAKER_01: Within minutes,
[42:40] SPEAKER_01: determine who owns each service with Cortex's AI service ownership model,
[42:44] SPEAKER_01: even across thousands of repositories.
[42:46] SPEAKER_01: Clear ownership means faster migrations,
[42:48] SPEAKER_01: quicker resolutions to critical issues like Lock4J,
[42:51] SPEAKER_01: and fewer add-here pings during incidents.
[42:54] SPEAKER_01: Cortex is trusted by leading engineering organizations like Affirm,
[42:57] SPEAKER_01: TripAdvisor,
[42:58] SPEAKER_01: Grammarly,
[42:59] SPEAKER_01: and SoFi.
[43:00] Solve service ownership and unlock your team's full potential
[43:03] with Cortex.
[43:04] SPEAKER_01: Visit cortex.io slash pragmatic to learn more.
[43:08] SPEAKER_01: That is C-O-R-T-X dot I-O slash pragmatic.
[43:12] SPEAKER_01: Yeah,
[43:12] SPEAKER_01: we used to have a joke internally within,
[43:16] SPEAKER_01: I think,
[43:16] SPEAKER_01: my team or my broader team back when I was a viewer.
[43:19] SPEAKER_01: It just went around.
[43:20] SPEAKER_01: What's the difference between a junior or an entry-level engineer,
[43:24] SPEAKER_01: a senior engineer,
[43:26] SPEAKER_01: and a staff engineer?
[43:27] SPEAKER_01: It's like you get a code review, entry-level engineer comes in,
[43:30] SPEAKER_01: And says, oh, you know, the order is not alphabetized as per our rules.
[43:35] SPEAKER_01: Like, we should change the order of the variables.
[43:38] SPEAKER_01: Senior engineering goes in and on the same call line leaves the comment,
[43:42] SPEAKER_01: actually,
[43:43] we should probably move this out into its own class because it
[43:47] should be encapsulated.
[43:48] SPEAKER_01: The staff engineering comes in.
[43:50] SPEAKER_01: Why are we even doing this?
[43:52] SPEAKER_01: There's a service that does exactly this thing.
[43:54] SPEAKER_01: You should be using that one.
[43:55] SPEAKER_00: And,
[43:56] SPEAKER_01: you know,
[43:58] SPEAKER_01: the thing that goes back to that,
[43:59] SPEAKER_01: the earlier career,
[44:01] SPEAKER_01: you should focus on the code for sure.
[44:04] SPEAKER_01: But after a while, you need to see the bigger pictures.
[44:06] And I often saw some of the most impactful engineers on my team
[44:10] in a given,
[44:11] SPEAKER_01: let's say,
[44:12] SPEAKER_01: quarter,
[44:12] SPEAKER_01: if I had to use a business value,
[44:13] SPEAKER_01: they did not by writing code,
[44:15] sometimes by deleting code or sometimes pointing out the work
[44:17] we should just avoid by,
[44:19] SPEAKER_01: I don't know,
[44:20] SPEAKER_01: like merging with something or just saying no,
[44:23] SPEAKER_01: pushing back on product,
[44:24] SPEAKER_01: et cetera,
[44:24] SPEAKER_01: et cetera,
[44:24] SPEAKER_01: et cetera.
[44:25] SPEAKER_00: Yeah, no one wants to say no is a skill, for sure.
[44:28] SPEAKER_01: Yeah, but it's a baseline, right?
[44:29] SPEAKER_01: You need to know that, but there's a lot more afterwards.
[44:33] SPEAKER_01: So in Kubernetes,
[44:36] SPEAKER_01: if I'm a software engineer or contributor who has an idea,
[44:41] SPEAKER_01: is contributing some code,
[44:42] SPEAKER_01: what does that look in terms of release?
[44:44] SPEAKER_01: How will it eventually get released?
[44:47] SPEAKER_01: That's one part of my question.
[44:48] SPEAKER_01: And the other part is...
[44:51] Does Kubernetes work in a similar way to some other open source
[44:54] project where you
[44:54] SPEAKER_01: kind of open up and you depend on what contributions are?
[44:57] SPEAKER_01: Or do you have a roadmap saying,
[44:59] SPEAKER_01: hey,
[44:59] we'd like to encourage people to build this functionality or
[45:03] solve for this problem?
[45:05] SPEAKER_00: Yeah,
[45:06] SPEAKER_00: so Kubernetes,
[45:06] SPEAKER_00: like I mentioned earlier,
[45:07] SPEAKER_00: has a process called a KEP,
[45:11] SPEAKER_00: Kubernetes Enhancement Proposal.
[45:12] And if there is a feature that you want within Kubernetes,
[45:16] you go open a KEP.
[45:19] It's just an issue in the Kubernetes enhancements proposal or
[45:23] enhancements repo.
[45:24] SPEAKER_00: And there's a template for it.
[45:26] You go open your cap and then a discussion will ensue with
[45:30] whatever SIG owns that.
[45:31] SPEAKER_00: So if it is something within, I don't know.
[45:37] SPEAKER_00: If it's a storage thing,
[45:39] SIG Storage will own that cap and they will come way in and
[45:43] we'll have a discussion
[45:44] SPEAKER_00: about whether or not this is actually useful,
[45:47] SPEAKER_00: whether or not this is viable,
[45:49] SPEAKER_00: whether or not we have the resources to try to get this done right now,
[45:53] which is going to depend on other contributors looking at your
[45:55] proposal and going,
[45:56] SPEAKER_00: oh,
[45:56] SPEAKER_00: yeah,
[45:57] SPEAKER_00: I would like to work on that.
[45:58] SPEAKER_01: And when you say discussion, is it email?
[46:01] SPEAKER_01: Is it async?
[46:02] SPEAKER_01: Is it actually like video call?
[46:05] SPEAKER_00: So because we're an open source project,
[46:07] we prefer to do everything in public in a place where everybody
[46:10] can see it.
[46:11] So that discussion will happen on the GitHub issue or in Slack
[46:15] in a public channel.
[46:17] We very much discourage things from happening behind closed
[46:20] doors unless it's very sensitive,
[46:22] SPEAKER_00: like extremely sensitive in a security type of way,
[46:26] SPEAKER_00: or it could negatively impact somebody's career.
[46:29] SPEAKER_00: So code of conduct stuff is always handled behind closed doors.
[46:33] SPEAKER_00: But discussions about features, that is all very public.
[46:36] SPEAKER_00: GitHub issue or the Slack channel for, for instance, stake storage.
[46:42] SPEAKER_00: If the decision is, yeah, we would like to do this, then work begins.
[46:49] SPEAKER_00: your SIG lead will need to opt in.
[46:52] SPEAKER_00: It's just adding a label,
[46:54] SPEAKER_00: SIG lead opt in to the KEP,
[46:56] SPEAKER_00: and that tracks it for inclusion in a release,
[46:59] SPEAKER_00: at which point you have a whole bunch of deadlines that you have to hit.
[47:04] SPEAKER_00: Code freeze is the big one, obviously.
[47:06] SPEAKER_00: That's, is your code in?
[47:09] Then if it needs documentation or whatever,
[47:13] there's documentation freeze,
[47:14] there's test freeze.
[47:16] SPEAKER_00: These deadlines all exist across the span of 14 weeks.
[47:20] And if you hit all of those marks,
[47:22] your feature is included in the release as an alpha at first.
[47:27] SPEAKER_00: We do allow iterative stages.
[47:29] So you don't have to get the whole thing MVP out the door with
[47:34] your first alpha.
[47:35] SPEAKER_00: You can have an alpha too.
[47:37] Alpha features are off by default,
[47:39] but can be turned on with a feature flag.
[47:42] SPEAKER_00: Once you meet certain requirements to graduate your feature to beta,
[47:46] like you're expecting minimal architectural changes to the way
[47:49] it functions and
[47:50] SPEAKER_00: interacts with the rest of Kubernetes,
[47:52] SPEAKER_00: we'll graduate you to beta.
[47:54] SPEAKER_00: And you are on by default, but can be turned off with a feature flag.
[48:00] Once it's very stable,
[48:01] very usable,
[48:02] and you feel like you're ready to go GA,
[48:05] we graduate to GA.
[48:06] SPEAKER_01: General availability, right?
[48:07] Yeah,
[48:08] we move it to GA in the next release,
[48:10] and it is on by default and cannot be disabled.
[48:14] SPEAKER_01: And then by that time, feature flag is removed, right?
[48:16] SPEAKER_00: Yeah, there's no feature flag at that point.
[48:19] At any given point in time,
[48:20] roughly how many feature flags does Kubernetes have?
[48:23] SPEAKER_01: Is it going to be in the hundreds or thousands?
[48:26] SPEAKER_00: Yeah.
[48:28] SPEAKER_00: At minimum, dozens.
[48:29] SPEAKER_00: I can go look.
[48:31] Let me share my screen and I will show you all of the feature
[48:34] gates we have.
[48:34] SPEAKER_01: I love it.
[48:36] SPEAKER_01: This is what I love about open source.
[48:38] SPEAKER_01: You couldn't do this with commercial closed software.
[48:42] SPEAKER_00: No, not at all.
[48:44] SPEAKER_00: So all of our feature gates are documented.
[48:45] This is usually all the documentation that an alpha requires
[48:49] because it's just like
[48:50] SPEAKER_00: turning a thing off or on.
[48:52] SPEAKER_00: Um, uh, you can, you can see here what the default is.
[48:56] SPEAKER_00: So whether it's functional,
[48:57] SPEAKER_00: whether it's alpha or beta,
[48:59] SPEAKER_00: uh,
[48:59] SPEAKER_00: when it was introduced and,
[49:01] SPEAKER_00: uh,
[49:02] SPEAKER_00: until when,
[49:04] SPEAKER_00: um,
[49:04] SPEAKER_00: some of these are,
[49:05] SPEAKER_00: you know,
[49:05] SPEAKER_00: deprecated and will be removed in the future.
[49:08] SPEAKER_00: Some of them, uh, haven't been graduated since then, but there are, uh,
[49:14] SPEAKER_00: Uh, quite a lot.
[49:16] SPEAKER_00: It doesn't look like hundreds, but it looks like several dozens.
[49:20] SPEAKER_01: It looks, wow.
[49:22] SPEAKER_01: And so those are all features that are,
[49:25] SPEAKER_01: that are big enough to eventually,
[49:28] SPEAKER_01: some of them will graduate.
[49:29] SPEAKER_01: Uh, hopefully most of them will graduate eventually.
[49:32] SPEAKER_00: Yep.
[49:33] And these are all graduated or deprecated and will be removed
[49:36] at some point.
[49:38] SPEAKER_00: Um,
[49:39] SPEAKER_00: Yeah, so it's quite a lot.
[49:41] SPEAKER_00: We talk about what the feature stages mean here.
[49:44] SPEAKER_00: Really, every aspect of this project is so well documented.
[49:48] Even the way the release team runs,
[49:50] the way the project is governed,
[49:52] it's all documented.
[49:53] SPEAKER_00: We don't keep anything as tribal knowledge.
[49:59] SPEAKER_00: We write everything down so that there's no arguing about it.
[50:03] SPEAKER_00: But yeah, there's a ton of them.
[50:06] SPEAKER_01: Also,
[50:07] SPEAKER_01: you documenting this much,
[50:08] it just gives me the idea that if anyone wants to get a bit of
[50:13] a cheat of how a
[50:16] SPEAKER_01: well-run project this size runs for further inspiration,
[50:20] SPEAKER_01: you can just check it out.
[50:21] SPEAKER_01: You can rip off the ideas that make sense.
[50:24] And also you can adjust it for your size or for,
[50:28] again,
[50:28] you can experiment,
[50:29] right?
[50:29] SPEAKER_01: I feel you always want to experiment,
[50:30] SPEAKER_01: but getting inspiration of something that works is a good starting point.
[50:33] SPEAKER_00: Yeah, please do.
[50:34] SPEAKER_00: I mean,
[50:34] SPEAKER_00: and just like,
[50:35] SPEAKER_00: you know,
[50:35] SPEAKER_00: not all products,
[50:37] SPEAKER_00: not all applications need to run on Kubernetes at first.
[50:40] SPEAKER_00: That can very much be overkill from an infrastructure standpoint.
[50:45] Not all open source projects need to be run like Kubernetes to
[50:48] begin with.
[50:49] There are some things that are just good practice that you
[50:51] should ingrain from the
[50:52] SPEAKER_00: very beginning.
[50:53] SPEAKER_00: If it's a user-facing change, it needs documentation.
[50:56] SPEAKER_00: But you don't need a structured release team.
[50:59] the way we have it for your four-person open source project
[51:04] that is releasing one
[51:07] SPEAKER_00: minor update a year.
[51:09] SPEAKER_00: That's unnecessary.
[51:10] SPEAKER_00: And you've got a couple dozen users.
[51:12] But for something the size of Kubernetes or close to the size
[51:15] of Kubernetes,
[51:15] SPEAKER_00: yeah,
[51:16] SPEAKER_00: that's super useful.
[51:18] SPEAKER_00: Rip us off.
[51:19] SPEAKER_00: Absolutely rip us off.
[51:19] SPEAKER_00: That's part of why we document it all.
[51:23] SPEAKER_01: Exactly.
[51:23] SPEAKER_01: This is beautiful, like having this win.
[51:26] SPEAKER_01: And also you don't really need to explain it again and again.
[51:28] SPEAKER_00: Correct.
[51:29] SPEAKER_00: We can just point people at the docs.
[51:30] SPEAKER_00: Yeah.
[51:31] The way every role on the release team has to be run is
[51:34] exhaustively documented.
[51:35] So every single role has a handbook that goes week by week and
[51:39] tells you what to do.
[51:41] SPEAKER_00: which is part of why our releases are so smooth.
[51:44] If something goes wrong,
[51:45] probably it's documented and we know how to solve it.
[51:51] SPEAKER_00: Of course,
[51:51] SPEAKER_00: every release,
[51:51] SPEAKER_00: we get something new and exciting that goes wrong to document,
[51:55] SPEAKER_00: but that's happening less and less over time as we run out of edge cases.
[52:00] SPEAKER_00: So...
[52:01] SPEAKER_01: One thing I'd like to ask you,
[52:05] SPEAKER_01: because I'm really interested on how I was going,
[52:08] SPEAKER_01: how do you think the Gen.AI tools,
[52:10] SPEAKER_01: LLMs that more and more engineers are using to write code,
[52:14] SPEAKER_01: help them write code,
[52:16] SPEAKER_01: will impact how...
[52:18] SPEAKER_01: Kubernetes is developed.
[52:20] SPEAKER_01: Because it's so extensively documented on one end,
[52:22] SPEAKER_01: I'm sure these tools can ingest a lot of things.
[52:26] SPEAKER_01: I wonder if they could be useful.
[52:28] SPEAKER_01: Are you seeing them being useful in,
[52:31] SPEAKER_01: let's say,
[52:32] using features or for the release team to build your own tools
[52:35] to do some of your
[52:36] SPEAKER_01: work easier?
[52:36] SPEAKER_01: Okay,
[52:38] SPEAKER_00: so my personal opinion,
[52:40] SPEAKER_00: I'm going to stress the word personal there,
[52:43] SPEAKER_00: is that the overwhelming majority of Gen.ai tools are a scam.
[52:48] SPEAKER_00: They are a scam.
[52:50] They are designed to filch money out of VCs or the public or
[52:55] whoever else.
[52:56] SPEAKER_01: The people who fund the VCs, the liquidity partner or LPs.
[53:01] SPEAKER_01: Yeah.
[53:01] SPEAKER_00: The investors.
[53:02] SPEAKER_00: I think most of them are a scam.
[53:04] SPEAKER_00: I think a very small minority of them are genuinely useful.
[53:08] SPEAKER_00: And the rest are just, it's a nothing burger.
[53:11] SPEAKER_00: Yeah.
[53:13] As a maintainer within SIG release,
[53:17] we have no use for generative AI or LLM tools.
[53:21] SPEAKER_00: Most of the issues we are dealing with are people problems.
[53:24] SPEAKER_00: Gen AI and LLMs are not going to resolve that.
[53:28] SPEAKER_00: We're people managers.
[53:29] SPEAKER_00: That's what we're doing.
[53:31] SPEAKER_00: So generative AI, not useful there.
[53:34] SPEAKER_00: With my SIG docs hat on,
[53:37] we actually explicitly have a policy of you are not allowed to
[53:40] use tools like that.
[53:41] SPEAKER_00: Um,
[53:42] SPEAKER_00: we caught somebody submitting a blog that had been written with some,
[53:46] some kind of chat GPT tooling and we pulled it and told them
[53:50] not to do that again.
[53:51] SPEAKER_00: Uh,
[53:51] SPEAKER_00: we got a drive by PR from a paid product that claimed to simplify,
[53:58] SPEAKER_00: um,
[53:59] SPEAKER_00: writing documentation,
[54:00] SPEAKER_00: maintaining documentation and
[54:03] SPEAKER_00: handling things like style guide compliance.
[54:05] SPEAKER_00: Kubernetes does have a documentation style guide.
[54:08] SPEAKER_00: We got a drive-by PR from them that stipulated,
[54:11] SPEAKER_00: hey,
[54:11] SPEAKER_00: if you accept this,
[54:13] we grant you a free license for this tool as long as we can use
[54:16] your logo.
[54:18] SPEAKER_00: But here's the thing, it didn't work.
[54:21] It was making changes to a bunch of generated files that a
[54:23] human should never touch.
[54:25] SPEAKER_00: And it wasn't actually enforcing style guide compliance.
[54:29] SPEAKER_00: It made a bunch of mistakes.
[54:31] SPEAKER_00: So even, even when the tool sounds useful,
[54:36] um it it is often not like it doesn't actually do what it says
[54:39] it's doing on the
[54:40] tin and also drive by pr um in exchange for a logo on that
[54:46] doesn't seem right that
[54:48] is that's really poor behavior if there if you own a product
[54:50] and you're listening
[54:51] to this do not do that please we hate it so much because it
[54:55] creates more work for
[54:56] us because we have to have a discussion as leadership about
[54:58] what we're going to do
[54:59] SPEAKER_00: with that like you caused three meetings please don't do that to me
[55:04] However,
[55:05] where I do want to see it is in things that are just toil for us.
[55:11] SPEAKER_00: So things like application of GitHub labels on issues or PRs.
[55:16] SPEAKER_00: I would like something to automatically, if the wrong language works,
[55:21] SPEAKER_00: label gets added to a PR,
[55:23] I would like a tool that removes that label and applies the
[55:26] correct linguistic
[55:27] SPEAKER_00: label or the correct area label,
[55:30] SPEAKER_00: or it probably can't make assumptions about priority.
[55:34] SPEAKER_00: But things like handling labeling,
[55:36] that's really annoying work that we have to do on just about
[55:41] every PR if a new
[55:43] SPEAKER_00: contributor doesn't understand how our labels work.
[55:46] They may not apply any of them,
[55:47] and somebody like me has to go in and do it.
[55:50] So tooling like that would be useful,
[55:52] but we haven't found somebody offering that to us.
[55:59] SPEAKER_01: Yeah,
[56:00] SPEAKER_01: just to add another perspective on the scam part,
[56:04] SPEAKER_01: I understand where you're coming from,
[56:05] SPEAKER_01: by the way,
[56:05] SPEAKER_01: in terms of it is overhyped.
[56:07] I think it's overhyped in so many ways,
[56:09] and I'm a little frustrated that we're
[56:11] We're talking so much about what it could do one day instead of
[56:14] what it is happening.
[56:15] SPEAKER_01: I'm very much focused on how we're using it today.
[56:17] An interesting input I got from John Osterhout,
[56:20] the author of Philosophy of Software Design.
[56:22] SPEAKER_01: He's a professor at Stanford.
[56:24] SPEAKER_01: And he is actually contributing to the Linux kernel.
[56:27] SPEAKER_01: He's implementing a protocol that one of his PhD students came up with.
[56:32] And he told me,
[56:33] I asked him if he uses Gen.AI much,
[56:35] and he said he doesn't accept...
[56:38] he said it was really helpful for him to start contributing to
[56:42] Linux because he
[56:43] SPEAKER_01: said it's so under-documented.
[56:45] His criticism was that there's so little documentation that he
[56:48] actually turned to
[56:49] SPEAKER_01: ChatGPT to try to make sense of it,
[56:50] SPEAKER_01: and it helped him.
[56:51] SPEAKER_01: And his takeaway was...
[56:54] SPEAKER_01: He actually said,
[56:55] SPEAKER_01: I don't really want to popularize this,
[56:57] but I wish they would have added some more comments because
[57:00] then I could have done
[57:01] SPEAKER_01: it without the tool.
[57:03] SPEAKER_01: So I'm actually getting the sense of Kubernetes is very well documented,
[57:06] SPEAKER_01: which will obviously help people who are using these tools to learn.
[57:12] SPEAKER_01: I mean, you can probably just go to the source, to be fair.
[57:14] SPEAKER_00: You can, but I totally get using something like ChatGPT to explain...
[57:20] SPEAKER_00: something complex to you.
[57:22] SPEAKER_00: That is a good use of an LLM to me.
[57:26] SPEAKER_01: Yeah.
[57:26] SPEAKER_00: It's one of the most widespread uses with devs.
[57:29] SPEAKER_00: No, that's great.
[57:31] I don't think you should be using it to generate a blog article
[57:33] or generate the documentation.
[57:35] SPEAKER_00: You certainly shouldn't be using it to translate your docs.
[57:38] The Kubernetes documentation is localized into a couple dozen
[57:41] different languages.
[57:42] SPEAKER_00: And that's all done by humans, right?
[57:45] SPEAKER_00: So I do not think that you should be using generative AI to do that.
[57:49] But using it to explain a complex topic or like look at some
[57:53] code and give you an
[57:54] SPEAKER_00: idea of where it's going,
[57:56] SPEAKER_00: like what it does,
[57:57] SPEAKER_00: that's useful.
[57:58] SPEAKER_00: Like that's a great way to use generative AI to me.
[58:01] SPEAKER_01: when do you think Kubernetes is a great choice to use?
[58:04] SPEAKER_01: And when do you think it can actually be an overkill?
[58:07] SPEAKER_01: I get this a lot where especially small startups are thinking,
[58:11] should we start with Kubernetes or will it just be such an
[58:14] infrastructure overkill
[58:15] SPEAKER_01: that we shouldn't use it?
[58:16] So where would you draw that line or how do you see teams
[58:20] making this decision?
[58:23] SPEAKER_00: I would say that you should think about
[58:27] how quickly do you think you're going to need to scale rapidly,
[58:31] like really rapidly?
[58:34] SPEAKER_00: If you think that you're going to need to do that pretty abruptly,
[58:37] then I would use Kubernetes earlier because migrating over to
[58:43] it can be a pain.
[58:45] SPEAKER_00: What you need to really think about is not so much the
[58:50] SPEAKER_00: Is Kubernetes itself as an infrastructure tool overkill?
[58:53] SPEAKER_00: It's can you afford the people you need to manage it correctly?
[58:59] SPEAKER_00: A mismanaged Kubernetes cluster is a very dangerous thing.
[59:03] SPEAKER_00: Kubernetes is not secure by default.
[59:05] SPEAKER_00: You do require an expert.
[59:08] So my best piece of advice for running Kubernetes early is do
[59:13] not roll your own cluster.
[59:15] SPEAKER_00: Don't do that.
[59:16] SPEAKER_00: Do not do that.
[59:18] Use some kind of managed Kubernetes service that handles a lot
[59:21] of the work for you.
[59:22] And then hire an SRE with experience with GKE or AKS or
[59:29] whatever tooling you're using.
[59:31] SPEAKER_00: But just don't roll your own cluster.
[59:33] SPEAKER_00: Kubernetes is easier to use than it was before.
[59:36] SPEAKER_00: I don't think it's as much easier.
[59:40] overkill early as it used to be like obviously obviously if
[59:44] you're just running a
[59:45] blog or something putting that on kubernetes is ridiculous my
[59:49] personal website does
[59:50] run on kubernetes but that's it's a bit right like i don't need
[59:54] it to right it's a
[59:58] SPEAKER_01: pet project
[59:59] SPEAKER_00: Yeah, it's a pet project.
[1:00:00] SPEAKER_00: It's fun.
[1:00:00] SPEAKER_00: It's easy for me to deploy something to a cluster.
[1:00:03] SPEAKER_00: I can do that faster than I can do it with EC2.
[1:00:07] SPEAKER_00: But that's because that's where my experience is.
[1:00:09] SPEAKER_00: For most people, that's not true.
[1:00:11] SPEAKER_00: So run it on some kind of managed service.
[1:00:16] And if you think that there is a point where you're ever going
[1:00:19] to need to scale
[1:00:20] SPEAKER_00: super fast,
[1:00:21] SPEAKER_00: migrate to Kubernetes before that happens.
[1:00:24] SPEAKER_01: I think the addition there is if you need to scale super fast,
[1:00:29] you can always choose to just stay on a cloud provider and just
[1:00:32] pay massive amounts.
[1:00:34] SPEAKER_01: You will be overcharged for it.
[1:00:36] So I guess Kubernetes makes sense when you actually want to
[1:00:38] control your costs a
[1:00:40] SPEAKER_01: lot more,
[1:00:40] SPEAKER_01: right?
[1:00:41] SPEAKER_00: Yeah, because it's scalable.
[1:00:44] SPEAKER_00: It's not like...
[1:00:45] SPEAKER_00: It scales up and down, right?
[1:00:47] SPEAKER_00: Like your costs aren't going to be consistent.
[1:00:51] SPEAKER_00: And that can make things a little bit difficult to predict.
[1:00:54] But there are companies and tools that help you manage that
[1:00:57] aspect of running Kubernetes.
[1:00:59] But it won't be consistently high,
[1:01:02] at least,
[1:01:02] because of that up and down scalability.
[1:01:05] SPEAKER_00: But yeah, just do it earlier rather than later for me.
[1:01:10] SPEAKER_00: But if it's just a blog,
[1:01:11] SPEAKER_00: if it's just a website or whatever,
[1:01:13] SPEAKER_00: it's just a web application,
[1:01:15] SPEAKER_00: you probably do not need Kubernetes if it's just a web app.
[1:01:20] And so for those who are interested in getting started with
[1:01:22] Kubernetes for building
[1:01:23] SPEAKER_01: their own things as a user,
[1:01:25] SPEAKER_01: where would you recommend them to look?
[1:01:27] And what about for people who would be interested in dipping
[1:01:30] their toes and contributing?
[1:01:31] SPEAKER_01: What's a good place to start?
[1:01:33] For starting to use it,
[1:01:34] I very much recommend our own documentation and GKE's tutorials.
[1:01:40] GKE's tutorials come with a little sandbox so that you can
[1:01:43] actually do the thing,
[1:01:45] SPEAKER_00: which is very helpful.
[1:01:46] SPEAKER_00: That's the way I prefer to learn.
[1:01:47] Personally,
[1:01:48] I need to be told how to do the thing and then actually do it.
[1:01:52] That's not true for everybody,
[1:01:53] but that was a really,
[1:01:56] really great resource for me early on.
[1:01:58] And our own documentation has quick start guides for a variety
[1:02:01] of ways of getting
[1:02:03] SPEAKER_00: going with Kubernetes.
[1:02:05] SPEAKER_00: So I have always just used Minikube.
[1:02:08] There are many ways to get a cluster going and it'll walk you
[1:02:11] through most of them.
[1:02:13] SPEAKER_00: If you would like to get started contributing to Kubernetes,
[1:02:17] you can come talk to me in the Kubernetes Slack and I will be
[1:02:21] happy to help you
[1:02:22] SPEAKER_00: find a place.
[1:02:23] SPEAKER_00: But I will probably recommend that you start with
[1:02:26] SPEAKER_00: documentation in an area that is of specific interest to you.
[1:02:30] SPEAKER_00: So if you're particularly interested in storage or networking,
[1:02:32] I will hand you a storage or networking piece of documentation
[1:02:35] that I think needs updating.
[1:02:38] SPEAKER_00: If you like more of a people-managey thing,
[1:02:41] SPEAKER_00: I will be happy to talk to you about the Kubernetes release team.
[1:02:44] However,
[1:02:45] again,
[1:02:45] we get hundreds of applications and there's like 25 spots.
[1:02:48] So I would not write on that exclusively as a way to get
[1:02:52] involved in the project.
[1:02:53] SPEAKER_00: But yeah,
[1:02:54] SPEAKER_00: come talk to me on the Kubernetes Slack workspace,
[1:02:57] SPEAKER_00: Kat Cosgrove,
[1:02:58] SPEAKER_00: and I am more than happy to help you personally.
[1:03:01] SPEAKER_01: What wonderful.
[1:03:02] SPEAKER_01: And we've already talked,
[1:03:03] SPEAKER_01: I think,
[1:03:04] about the professional benefits that can easily come by being
[1:03:08] associated with a
[1:03:09] SPEAKER_01: project Kubernetes in terms of an active contributor or maintainer.
[1:03:15] SPEAKER_00: Yeah, it's definitely a thing.
[1:03:18] Release team members put Kubernetes release team comms shadow
[1:03:23] on their resume.
[1:03:24] SPEAKER_00: It's on people's LinkedIn.
[1:03:25] If you look at the number of people on LinkedIn who say they
[1:03:27] work at Kubernetes,
[1:03:29] SPEAKER_00: myself included,
[1:03:30] SPEAKER_00: mine says I work at Kubernetes.
[1:03:32] SPEAKER_01: You do, right?
[1:03:33] SPEAKER_00: I do.
[1:03:33] SPEAKER_00: Yeah.
[1:03:34] SPEAKER_00: Yeah, I maintain the thing.
[1:03:35] SPEAKER_00: So I do work there.
[1:03:37] SPEAKER_00: But yeah, people put it on their resumes.
[1:03:40] SPEAKER_00: It's
[1:03:41] SPEAKER_00: It's popular,
[1:03:42] especially if you're a student about to graduate or you're very
[1:03:45] early in your career.
[1:03:46] SPEAKER_00: It can definitely give you a resume boost.
[1:03:48] SPEAKER_00: But what it also gives you is access to a very,
[1:03:52] very large network of people with significantly more experience
[1:03:55] than you who are
[1:03:57] SPEAKER_00: generally quite friendly.
[1:03:58] SPEAKER_00: So it's a really good networking opportunity.
[1:04:00] SPEAKER_00: We all hang out a lot.
[1:04:01] SPEAKER_00: We all talk to each other a lot.
[1:04:05] SPEAKER_00: Right before CubeCon,
[1:04:07] SPEAKER_00: our big conference that we have twice a year,
[1:04:09] SPEAKER_00: there is a maintainer summit.
[1:04:11] And we all get together and hang out and talk to each other for
[1:04:15] a whole day.
[1:04:16] SPEAKER_00: That's our own little miniature conference.
[1:04:18] SPEAKER_00: So there is also the networking opportunity there too.
[1:04:20] SPEAKER_00: It's not just the resume fodder.
[1:04:23] SPEAKER_01: Yeah,
[1:04:23] and I like that you say networking because I feel people are
[1:04:27] starting to realize,
[1:04:28] engineers are starting to realize with the job market becoming
[1:04:30] a bit more frosty that,
[1:04:32] SPEAKER_00: oh,
[1:04:32] SPEAKER_01: having a network could be useful.
[1:04:34] SPEAKER_01: And then there's a question of how do you get a network?
[1:04:36] And just my two cents is the best network I've ever had is it's
[1:04:39] not people who are
[1:04:40] SPEAKER_01: kind of like,
[1:04:41] SPEAKER_01: I know them with drive-by.
[1:04:42] SPEAKER_01: It's people who I actually worked with.
[1:04:43] SPEAKER_01: Like did something together and,
[1:04:45] SPEAKER_01: and,
[1:04:46] SPEAKER_01: you know,
[1:04:46] you can either do it at your full-time job as just assuming you
[1:04:49] have one or on a
[1:04:50] SPEAKER_01: project where you spend extended time and solve the problems,
[1:04:53] SPEAKER_01: which project like Kubernetes or different open source project.
[1:04:57] SPEAKER_00: Yeah, for sure.
[1:04:58] SPEAKER_00: If I've worked with you on the Kubernetes project for a little while,
[1:05:01] SPEAKER_00: I am happy to connect you with anybody else I know.
[1:05:05] If you're looking for a job at Docker or at AWS or Microsoft or whatever...
[1:05:14] SPEAKER_00: I know executives there by virtue of having done this for so long.
[1:05:19] SPEAKER_00: And I'm happy to make connections.
[1:05:21] SPEAKER_00: If I've worked with you for a while, I will not.
[1:05:22] SPEAKER_01: And if that person has actually just, you know, good, solid work.
[1:05:28] SPEAKER_01: So I think it's just one more thing.
[1:05:30] If just don't do anything half ass,
[1:05:31] just like,
[1:05:31] yeah,
[1:05:32] don't have a whole ass.
[1:05:33] SPEAKER_00: Everything.
[1:05:34] Yeah.
[1:05:36] SPEAKER_01: I think the earlier people realize that, the better.
[1:05:40] SPEAKER_01: So to wrap up,
[1:05:42] I've just asked some rapid questions and I'll shoot and then
[1:05:45] you tell me what comes
[1:05:47] SPEAKER_01: to mind.
[1:05:48] SPEAKER_01: What is your favorite framework or programming language?
[1:05:52] SPEAKER_00: Python,
[1:05:54] SPEAKER_00: which I shouldn't say because I'm managing the Kubernetes project,
[1:05:56] SPEAKER_00: which is written in Go,
[1:05:57] SPEAKER_00: but it's Python.
[1:05:58] SPEAKER_01: That was really fast.
[1:05:59] SPEAKER_01: How come?
[1:06:01] It was not the first language I learned,
[1:06:03] but it was one of the earlier ones.
[1:06:06] And I like that even if it's not good at everything,
[1:06:10] I can make it do just about anything.
[1:06:14] SPEAKER_00: So that versatility is really valuable to me as a prototyping tool.
[1:06:18] SPEAKER_00: So I will prototype something in Python.
[1:06:20] And if Python is not the best tool in my toolbox for that
[1:06:25] particular application,
[1:06:26] SPEAKER_00: I'll then go actually rebuild it and go on.
[1:06:29] SPEAKER_00: Or whatever is most applicable.
[1:06:32] SPEAKER_01: I'm coming around to Python.
[1:06:33] SPEAKER_01: I never really use Python.
[1:06:35] SPEAKER_01: I use other languages, C Sharp, JavaScript, etc.
[1:06:38] SPEAKER_01: But you know what's getting me around to Python?
[1:06:40] It's now that I'm going to a lot of the getting started guides on,
[1:06:44] let's say,
[1:06:44] AI tools,
[1:06:45] etc.
[1:06:45] SPEAKER_01: They usually have it in all languages, like from Python.
[1:06:48] SPEAKER_01: Usually Python is the first one.
[1:06:49] SPEAKER_01: And what I've noticed is Python is the...
[1:06:52] the the least verbose and the easiest to read and i just liked
[1:06:58] reading it so now i
[1:06:58] just set up my my thing so i'm i'm actually changing my mind on
[1:07:01] that slowly i'm
[1:07:03] starting to see the the pull yeah you can almost just write
[1:07:06] pseudocode and that's
[1:07:07] like very close to valid python yeah which which rules so it's
[1:07:12] awesome yeah you got
[1:07:13] to do something quick and dirty python what is a book that you
[1:07:16] would recommend
[1:07:18] SPEAKER_00: Well, so I don't read workbooks at all.
[1:07:23] I actually have a personal policy of not reading books for
[1:07:28] anything other than fun.
[1:07:29] SPEAKER_00: Like I only read for pleasure and I read quite a lot.
[1:07:33] SPEAKER_00: Right now,
[1:07:34] SPEAKER_00: I'm rereading A Fire Upon the Deep by Werner Vinge,
[1:07:37] SPEAKER_00: which is very weird,
[1:07:39] SPEAKER_00: very,
[1:07:40] SPEAKER_00: very,
[1:07:40] SPEAKER_00: very strange,
[1:07:41] SPEAKER_00: hard science fiction.
[1:07:42] SPEAKER_00: It's one of my favorite books.
[1:07:43] SPEAKER_00: I would absolutely recommend that if you like hard sci-fi.
[1:07:48] SPEAKER_01: Well,
[1:07:49] SPEAKER_01: Kat,
[1:07:49] SPEAKER_01: thank you so much for going into the details of,
[1:07:52] SPEAKER_01: I guess,
[1:07:53] talking through the surprisingly open nature of Kubernetes and
[1:07:56] very well-documented
[1:07:57] SPEAKER_01: nature of it.
[1:07:58] SPEAKER_01: Thank you.
[1:07:59] SPEAKER_01: Happy to.
[1:08:00] SPEAKER_01: I hope you enjoyed this deep dive into how Kubernetes is built,
[1:08:03] SPEAKER_01: how releases work,
[1:08:04] and all the organizational work that goes into keeping a
[1:08:06] project like this running.
[1:08:07] I found it particularly interesting to learn just how well
[1:08:10] everything in Kubernetes
[1:08:11] SPEAKER_01: is documented,
[1:08:12] SPEAKER_01: including how the project is organized and run.
[1:08:14] SPEAKER_01: If you ever find yourself inside a large project,
[1:08:16] SPEAKER_01: feel free to steal ideas on how to organize things from Kubernetes.
[1:08:21] SPEAKER_01: Thanks a lot for Cap for sharing all these details.
[1:08:23] SPEAKER_01: You can connect with her on the links listed in the show notes below.
[1:08:26] We previously did a deep dive into how Linux was built and into
[1:08:29] a few other related topics.
[1:08:31] You can find all of these The Pragmatic Engineer deep dives in
[1:08:33] the show notes linked below.
[1:08:35] SPEAKER_01: If you've enjoyed the podcast,
[1:08:36] SPEAKER_01: please do subscribe on your favorite podcast platform and on YouTube.
[1:08:39] This helps more people discover the podcast and a special
[1:08:42] thanks if you leave a rating.
[1:08:44] SPEAKER_01: Thanks and see you in the next one.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.