# Observability: the present and future, with Charity Majors

An episode of [The Pragmatic Engineer](https://invisiblenetwork.ai/podcast/1769051199), 22 Jan 2025, 1 h 14 min.

Audio: https://api.substack.com/feed/podcast/155289513/61c0b87b513f3600cd8de81181a5763a.mp3

Supported by Our Partners • Sonar — Trust your developers – verify your AI-generated code. • Vanta —Automate compliance and simplify security with Vanta. — In today's episode of The Pragmatic Engineer, I'm joined by Charity Majors, a well-known observability expert – as well as someone with strong and grounded opinions. Charity is the co-author of "Observability Engineering" and brings extensive experience as an operations and database engineer and an engineering manager. She is the cofounder and CTO of observability scaleup Honeycomb. Our conversation explores the ever-changing world of observability, covering these topics: • What is observability? Charity’s take • What is “Observability 2.0?” • Why Charity is a fan of platform teams • Why DevOps is an overloaded term: and probably no longer relevant • What is cardinality? And why does it impact the cost of observability so much? • How OpenTelemetry solves for vendor lock-in • Why Honeycomb wrote its own database • Why having good observability should be a prerequisite to adding AI code or using AI agents • And more! — Timestamps (00:00) Intro (04:20) Charity’s inspiration for writing Observability Engineering (08:20) An overview of Scuba at Facebook (09:16) A software engineer’s definition of observability (13:15) Observability basics (15:10) The three pillars model (17:09) Observability 2.0 and the shift to unified storage (22:50) Who owns observability and the advantage of platform teams (25:05) Why DevOps is becoming unnecessary (27:01) The difficulty of observability (29:01) Why observability is so expensive (30:49) An explanation of cardinality and its impact on cost (34:26) How to manage cost with tools that use structured data (38:35) The common worry of vendor lock-in (40:01) An explanation of OpenTelemetry (43:45) What developers get wrong about observability (45:40) A case for using SLOs and how they help you avoid micromanagement (48:25) Why Honeycomb had to write their database (51:56) Companies who have thrived despite ignoring conventional wisdom (53:35) Observability and AI (59:20) Vendors vs. open source (1:00:45) What metrics are good for (1:02:31) RUM (Real User Monitoring) (1:03:40) The challenges of mobile observability (1:05:51) When to implement observability at your startup (1:07:49) Rapid fire round — The Pragmatic Engineer deepdives relevant for this episode: • How Uber Built its Observability Platform https://newsletter.pragmaticengineer.com/p/how-uber-built-its-observability-platform • Building an Observability Startup https://newsletter.pragmaticengineer.com/p/chronosphere • How to debug large distributed systems https://newsletter.pragmaticengineer.com/p/antithesis • Shipping to production https://newsletter.pragmaticengineer.com/p/shipping-to-production — See the transcript and other references from the episode at https://newsletter.pragmaticengineer.com/podcast — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

## Transcript

[0:00] SPEAKER_0: The three pillars model.
[0:01] SPEAKER_0: What is that?
[0:02] The famous phrase goes,
[0:03] I think it was coined by Peter Borgen back in 2017,
[0:06] the observability has three pillars,
[0:07] metrics,
[0:08] logs,
[0:08] and traces.
[0:09] And a lot of vendors glommed onto this because cynically speaking,
[0:12] they have a metrics product to sell,
[0:14] a logging product to sell,
[0:15] and a tracing product to sell.
[0:16] SPEAKER_1: But it's actually kind of worse than that, right?
[0:18] Every request that enters your system historically people have stored it in,
[0:22] sure,
[0:22] like metrics,
[0:24] storage and dashboards,
[0:25] and maybe structured logs and unstructured logs,
[0:27] in a tracing tool,
[0:28] in a profiling tool,
[0:30] in an analytics tool again and again and again and again and again.
[0:33] SPEAKER_1: And what connects them?
[0:34] SPEAKER_1: Not much.
[0:35] You,
[0:35] the engineer,
[0:36] sitting in the middle going,
[0:37] "Well,
[0:37] that shape looks like that shape,
[0:39] sothey'reprobablythesamething."
[0:40] SPEAKER_1: Or maybe copy pasting IDs.
[0:42] SPEAKER_1: And the cost multiplier is obscene.
[0:46] Charity Majors is an observability expert and the author of the book,
[0:49] Observability Engineering,
[0:51] she worked at a software engineer at Parse then at Facebook and then
[0:54] co-founded Honeycomb,
[0:55] an observability startup.
[0:57] She believes that observability tools should be a lot easier and faster than
[1:00] they are today,
[1:01] and that every engineer should have the kind of magical observability
[1:04] experience like Devzet Meta have with their internal tools.
[1:08] In today's episode,
[1:08] we go into what is observability and what do things like the three pillar
[1:12] model and cardinality mean?
[1:14] SPEAKER_0: What is observability 2.0 and why is everyone so excited about it?
[1:18] Things engineering teams frequently get wrong about observability and so
[1:22] many more things worth knowing about modern observability practices.
[1:25] If you enjoy the show and would like to support it,
[1:27] please subscribe to the podcast on any podcast platform and on YouTube.
[1:31] SPEAKER_0: Charity, welcome to the podcast.
[1:34] SPEAKER_1: Thank you so much for having me.
[1:35] SPEAKER_1: You and I go way back and I feel like we don't get to talk enough.
[1:38] SPEAKER_1: So this is a real treat for me.
[1:40] SPEAKER_0: We go way back.
[1:41] SPEAKER_0: In fact, the way we started talking was so funny.
[1:46] We both were on our blogs,
[1:47] wewerewritingadvicecolumnswhowerepeople-
[1:52] Oh,
[1:52] that guy said the same advice question to both of us,
[1:53] and we both answered and we were like,
[1:55] "Hey,
[1:56] hi there,
[1:57] friend."
[1:58] Yeah,
[1:59] someone asked something about I think measuring developer productivity
[2:02] actually funny enough.
[2:04] And then I think you wrote something and I wrote something and we didn't know
[2:06] about each other.
[2:07] SPEAKER_0: And then we discovered it later.
[2:09] And I read your article and I'm like,
[2:11] "Oh,
[2:11] Icouldhavewrittenthesame."
[2:12] SPEAKER_0: And I realized we're little bit common souls.
[2:15] SPEAKER_1: Yeah, exactly.
[2:16] SPEAKER_1: That was funny.
[2:19] This episode was brought to you by Sonar,
[2:21] the creators of SonarQube Server,
[2:22] Cloud,
[2:23] ID,
[2:23] and Community Build.
[2:25] Sonar helps prevent bugs,
[2:26] code quality and security issues from reaching production,
[2:29] amplifies developer's productivity in concert with AI assistance and
[2:33] improves the developer experience with streamlined workflows.
[2:37] Sonar analyzes all code regardless of who writes it,
[2:40] your internal team or GenAI resulting in more secure,
[2:43] reliable,
[2:44] and maintainable software.
[2:46] Combining Sonar's AI Code Assurance capability and SonarQube with the
[2:49] power of AI coding assistance like GitHub Copilot,
[2:52] Amazon Q Developer,
[2:54] and Google Gemini Code Assist boosts developer productivity and ensures
[2:58] that code meets rigorous quality and security standards.
[3:01] Join over 7 million developers from organizations like IBM,
[3:04] NASA,
[3:05] Barclays,
[3:05] and Microsoft who use Sonar.
[3:08] SPEAKER_0: Trust your developers, verify your AI-generated code.
[3:11] SPEAKER_0: Visit sonarsource.com/pragmatic to try SonarQube for free today.
[3:16] SPEAKER_0: That is sonarsource.com/pragmatic.
[3:21] SPEAKER_0: Trust isn't just earned.
[3:23] SPEAKER_0: It's demanded.
[3:24] Whether you're a startup founder,
[3:25] navigating your first audit or a seasoned security professional scaling
[3:29] your governance risk and compliance program,
[3:31] proving your commitment to security has never been more critical or more complex.
[3:36] SPEAKER_0: That's where Vanta comes in.
[3:37] Vanta can help you start or scale your security program by connecting with
[3:41] auditors and experts to conduct your audit and set up your security program quickly.
[3:46] Plus,
[3:46] with automation and AI throughout the platform,
[3:48] Vanta gives your time back so you can focus on building your company.
[3:52] Businesses use Vanta to establish trust by automating compliance needs
[3:55] across over 35 frameworks like SOC2 and ISO27001.
[4:00] With Vanta,
[4:00] they centralize security workflows,
[4:02] complete questionnaires up to five times faster,
[4:05] and proactively manage vendor risk.
[4:07] Join over 9,000 global companies to manage risk and prove security in real time.
[4:12] For a limited time,
[4:13] my listeners get $1,000 off Vanta at vanta.com/pragmatic.
[4:17] SPEAKER_0: That is V-A-N-T-A.com/pragmatic for $1,000 off.
[4:23] So to kick things off,
[4:24] can you tell us how you got first exposed to observability?
[4:29] Beforehand,
[4:30] when we talked,
[4:31] I remember that there was something about this startup that you worked at
[4:34] Parse and then it got acquired and you worked at Facebook and then something
[4:38] happened there.
[4:39] SPEAKER_0: How did this go?
[4:40] Well,
[4:41] so around the time that Parse was getting acquired,
[4:43] this was in the free money days,
[4:47] rocket ship growth,
[4:48] and we were taking off and we had 60 some thousand mobile apps when we got acquired.
[4:54] By the time I left,
[4:55] we had over a million mobile apps and the traffic was so unpredictable,
[5:00] A different app would hit the top 10 in iTunes every day,
[5:02] just be like...
[5:03] And-
[5:05] SPEAKER_0: This was back in 2013, 2014, right?
[5:07] SPEAKER_1: Yeah, exactly.
[5:08] SPEAKER_1: Like 2012, 2013.
[5:10] And we had built this on the Ruby on Rails stack,
[5:13] which was a smart enough idea at the time,
[5:15] but fixed pool of workers.
[5:18] And so any app takes off and suddenly boom,
[5:21] Parse goes down because all of the in-flight workers get caught up on threads.
[5:25] SPEAKER_1: Any of the backends get slow, boom, Parse goes down.
[5:28] SPEAKER_1: And I'm like the infrastructure lead, right?
[5:30] And as a reliability engineer,
[5:31] this is just professionally humiliating for me because it's constantly
[5:34] going down.
[5:36] And I tried everything,
[5:38] every tool out there,
[5:41] and the first glimmer of light that we got was we get to Facebook.
[5:44] And I was really cynical about it,
[5:46] because all of the tools have been built in-house for in-house workloads,
[5:50] you have the luxury of forcing people to use your stuff.
[5:53] SPEAKER_1: And I'm like, "This is not built for us."
[5:55] But we started feeding some data sets into a tool there called Scuba,
[5:59] and suddenly we're able to slice and dice in real time on high cardinality dimensions.
[6:04] We were able to instead of just going,
[6:06] "Oh..." Because Parse goes down,
[6:08] we're looking through the logs,
[6:10] and it might be full of a particular request,
[6:13] but that doesn't necessarily mean that that's the app that caused it to go down.
[6:16] SPEAKER_1: There might just be lots of stuff backing up.
[6:17] SPEAKER_1: It might a few very slow requests that's taking.
[6:21] SPEAKER_1: You just don't know.
[6:23] And having the ability to break down slice by things like app ID,
[6:27] user ID,
[6:28] raw query,
[6:29] normalized query,
[6:30] the amount of time it took for us to pinpoint the cause and do something about
[6:33] it dropped like a rock from hours.
[6:38] Sometimes it would just recover and we'd never actually know what happened
[6:41] to seconds.
[6:42] SPEAKER_1: It wasn't even an engineering problem anymore.
[6:44] SPEAKER_1: It was like a support problem.
[6:47] SPEAKER_1: And this was life-changing, right?
[6:51] So I'm coming at this from an operational lens,
[6:55] and Christine,
[6:56] my co-founder,
[6:57] had written the Parse analytics product and she had built it on top of
[7:01] Cassandra and she was constantly being professionally humiliated by the
[7:05] fact that this product she had written for our users to understand their applications,
[7:11] you had to pre-define in advance how to capture the data and what questions
[7:14] you'd be able to ask.
[7:16] SPEAKER_1: And they'd be like, "But I need to ask this new question."
[7:18] SPEAKER_1: And she'd be like, "Ah, how embarrassing."
[7:20] She'd go and look it up by hand in Scuba and reply to them and she'd just like,
[7:25] "Ah."
[7:26] So we had both had this experience from very different sides of the stack and
[7:29] we were just like...
[7:31] So when we left Facebook,
[7:34] it's looking back on how ill prepared we were to start.
[7:37] SPEAKER_1: I had never heard the words product market fit deep in infrastructure.
[7:42] SPEAKER_1: We didn't know that categories existed or Gartner existed.
[7:46] So we were not setting out to build a category,
[7:48] we were setting out to try and explain this life-changing experience that we
[7:54] had had.
[7:54] We knew it wasn't monitoring,
[7:56] we knew it wasn't logging,
[7:58] which is why it took so much longer and was so much harder than it needed to be.
[8:03] And then just to get a sense,
[8:05] so what was Scuba at Facebook or how would you describe it?
[8:08] Because it sounds like it's a mix of just listening to events,
[8:13] but also monitoring and querying.
[8:16] SPEAKER_1: Scuba is this weird beast.
[8:18] SPEAKER_1: It's kind of a columnar store, but it was in memory.
[8:21] SPEAKER_1: They wrote a white paper about it.
[8:23] One of my favorite things was that replication,
[8:27] it's a C++ binary and it shelled out to rsync to do replication.
[8:30] SPEAKER_1: It was quick and dirty.
[8:34] SPEAKER_1: And the evolution of Scuba was actually an interesting one.
[8:37] They developed it about a decade earlier when they were trying to get a handle
[8:40] on their MySQL issues.
[8:42] SPEAKER_1: There was a time when like PHP and MySQL just crashing all the time.
[8:47] Facebook was being professionally humiliated,
[8:49] which is why I think it's so interesting,
[8:51] the genesis of scuba and thus Honeycomb is not connected to the entire 20,
[8:57] 30-year history of telemetry and monitoring and three pillars.
[9:03] SPEAKER_1: It came out of complete left field.
[9:06] SPEAKER_1: It's much closer to the history of business analytics, I think.
[9:13] SPEAKER_0: So that's where it started.
[9:14] And now jumping a little bit more back to basics,
[9:19] you've been in the observability field for a long time now.
[9:22] SPEAKER_0: We can save now it's called that.
[9:23] But what does observability mean to you from a software engineer's point of view?
[9:28] SPEAKER_0: How would you define it?
[9:31] SPEAKER_1: Yeah, there are so many definitions floating around.
[9:33] SPEAKER_1: I'm probably responsible for more than my fair share of them.
[9:40] SPEAKER_1: But really it's about understanding your software.
[9:42] It's about understanding the intersection of your code,
[9:46] your systems and your users,
[9:48] right?
[9:49] SPEAKER_1: That's all it is.
[9:50] And I feel like for a long time the observability field was obsessed with
[9:55] errors and bugs and outages and nines and crashes and all these things.
[10:01] And I feel like one of the directions that we're starting to pull the industry
[10:04] in is that it's not just about problems.
[10:08] Understanding the need to understand impact of what you're putting out into
[10:12] the world is so much bigger and more interesting than that,
[10:14] which is part of why I feel like it's not just an operational tool.
[10:20] SPEAKER_1: This is the tool that underpins or development feedback loops.
[10:25] SPEAKER_0: Yeah.
[10:25] And I guess maybe not just development,
[10:27] but at some level product as well,
[10:29] like how your stuff works.
[10:31] SPEAKER_1: Yes, exactly.
[10:33] And ultimately,
[10:34] Christine and I just did this exercise to refresh our mission and vision and
[10:38] the phrase that we landed on was we're here to help engineers explain and
[10:44] understand our software in the language of the business.
[10:47] SPEAKER_1: I feel like there's this void.
[10:50] If you look at all of the C-level roles out there,
[10:55] the only one that has no template or definition is CTO all over the fucking map.
[11:00] SPEAKER_1: Right?
[11:01] And if you look at VPs and executive teams,
[11:04] engineering VPs have traditionally been like the junior varsity league.
[11:08] SPEAKER_1: They're not really in the core group.
[11:12] SPEAKER_1: Why is that?
[11:13] It's because for so long engineers have been kind of the artist of the company.
[11:18] SPEAKER_1: You just trust us, right?
[11:20] SPEAKER_1: We can't really justify it.
[11:23] SPEAKER_1: Because what is it about executive teams?
[11:25] The point of having an executive team is that you're each other's first team
[11:29] and you co-own the most fundamental decisions about where companies invest
[11:36] their resources,
[11:38] which means you have to have to co-understand them.
[11:43] Everyone needs to understand enough about marketing to be like,
[11:46] "In the upcoming year,
[11:47] our priorities are to move the needle in these ways.
[11:50] SPEAKER_1: So we're going to allocate these."
[11:52] SPEAKER_1: For engineering, it's just like, "We need 20% more people.
[11:55] SPEAKER_1: Why?
[11:56] SPEAKER_1: Well, we just do."
[11:57] SPEAKER_1: That's about as deep as it goes.
[11:59] And so I feel like over the next five years or so,
[12:03] really helping engineering get that,
[12:06] engineering,
[12:06] product,
[12:07] even design,
[12:08] get that first class seat at the table,
[12:11] it all comes down to being able to explain and understand our work in the same
[12:15] language as everyone else,
[12:16] which is money.
[12:17] So you're saying that let's say that the CFO,
[12:19] so the finance team,
[12:20] the CMO,
[12:21] the marketing,
[12:21] then the COO,
[12:22] the operations team,
[12:23] they can all explain,
[12:25] "Here's what we do,
[12:26] here's how that translates to the business.
[12:28] Here's why we are important and here's the things that we are moving for the business.
[12:33] SPEAKER_0: And if I get 20% more people, I can do this for the business."
[12:36] SPEAKER_0: Right?
[12:37] SPEAKER_1: Exactly.
[12:37] And then you're saying that engineering at a lot of startups,
[12:40] if I understand,
[12:41] don't really have that ability and observability or understanding how our
[12:46] stuff connects to the business gets us there.
[12:48] SPEAKER_1: It's a translation layer, yeah.
[12:50] Being able to understand reason about the work that you're doing and tie it
[12:54] back to top level goals.
[12:56] SPEAKER_1: I don't think this is just a startup problem either.
[12:58] I talked to big company execs and nobody wants to say this in public,
[13:01] but there's a lot of...
[13:03] It's hard.
[13:04] SPEAKER_1: It's really freaking hard.
[13:06] SPEAKER_0: Yeah.
[13:07] So going back to observability as a concept,
[13:11] it is an industry and you've written a large book about it.
[13:15] Can we go through some kind of common terms where if I'm a software engineer,
[13:20] I heard about observability,
[13:22] what are some things that I should probably know about?
[13:25] And you mentioned just before you mentioned high cardinality,
[13:28] normalized query,
[13:29] but there's stuff like traces,
[13:30] instrumentation,
[13:31] sampling.
[13:32] SPEAKER_0: What are some of the basics that you think?
[13:34] SPEAKER_0: Look, if you're an engineer, figure these things out.
[13:37] Look it up,
[13:37] read a book,
[13:38] talk to people and then you'll be able to start with.
[13:41] Yeah,
[13:42] I think it's important to know the difference between metrics through small
[13:47] M metrics and big M metrics,
[13:48] right?
[13:49] SPEAKER_1: We use the term metrics a lot just as a syn-
[13:51] SPEAKER_0: Sorry, you say small M metrics.
[13:53] SPEAKER_1: Small M, like the generic term for telemetry.
[13:56] SPEAKER_1: We just like, "Oh, the metrics are blah, blah, blah."
[13:58] SPEAKER_1: And then there's the Metric, which is a number, with some tags appended.
[14:03] SPEAKER_1: And I think this causes a lot of confusion with folks.
[14:06] The metric is a very small,
[14:09] fast,
[14:10] efficient data type,
[14:11] but it's supremely limited because it doesn't sort any contextual
[14:14] relationship data.
[14:18] I think one of the big shifts in the industry right now is away from the sort of
[14:21] three pillars model where every request that enters your system,
[14:25] you store it in a bunch of different places to a model where you have unified storage.
[14:30] So I think understanding the difference between the metric and structured
[14:34] data is pretty key.
[14:36] I think sampling is emerging as a really important lever and it scares a lot of
[14:44] people in large part because the big logging vendors have put decades now
[14:47] into telling people every log is sacred,
[14:50] don't drop a log line,
[14:53] which always reminds me of this sort of Monty Python Every Sperm is Sacred
[14:56] sketch and it's like...
[15:00] Anyway.
[15:01] SPEAKER_1: Yeah.
[15:01] SPEAKER_1: So I think sampling is important.
[15:03] SPEAKER_1: I think understanding the data structures is important.
[15:05] You mentioned something that I think is also kind of a given in the
[15:10] observability industry or people who do it,
[15:12] the three pillars model,
[15:13] what is that?
[15:14] Yeah,
[15:15] so the famous phrase goes,
[15:16] I think it was coined by Peter Borgen back in the 2017.
[15:20] SPEAKER_1: The observability has three pillars, metrics, logs, and traces.
[15:23] And a lot of vendors lumped onto this because cynically speaking,
[15:27] they have a metrics product to sell,
[15:28] a logging product to sell,
[15:30] and a tracing product to sell.
[15:32] SPEAKER_1: But it's actually kind of worse than that.
[15:34] Every request that enters your system,
[15:36] historically,
[15:36] people have stored it in,
[15:38] sure,
[15:39] like metrics,
[15:40] storage and dashboards and maybe structured logs and unstructured logs,
[15:44] in a tracing tool,
[15:45] in a profiling tool,
[15:47] in an analytics tool again and again and again and again and again.
[15:51] SPEAKER_1: And what connects them?
[15:55] The engineer sitting in the middle going,
[15:56] "Well,
[15:57] that shape looks like that shape.
[15:58] SPEAKER_1: So they are probably the same thing," or maybe copy-paste your IDs.
[16:03] And some of the bigger fancier tools like Datadog have built sort of bridges
[16:07] that you can pre-define,
[16:08] like this metric here ties into that log over there.
[16:11] But again,
[16:12] you're in this situation where you're having to define in advance what
[16:15] information is going to be important and where you're going to need to
[16:17] connect it.
[16:18] SPEAKER_1: And the cost multiplier is obscene, right?
[16:23] Some people,
[16:24] every request that enters our system,
[16:25] they're storing it in 15 different tools for 15 different use cases.
[16:29] And the more than you get,
[16:30] the more expensive it gets,
[16:31] the more strong it gets and the harder it gets to correlate everything.
[16:34] So the 1.0 to 2.0 shift,
[16:38] data is data,
[16:39] there are a lot of different ways to climb the mountain,
[16:41] but fundamentally it's about moving from many sources of truth to unified storage.
[16:48] Okay,
[16:48] so just so I get that right,
[16:50] observability 1.0 is just three pillars,
[16:53] the metrics,
[16:54] logs and traces,
[16:54] which it started like that.
[16:57] And apparently it is really good to sell a bunch of services and you can sell
[17:00] expensive services with that as well.
[17:02] SPEAKER_0: So what is 2.0?
[17:03] Okay,
[17:04] because you mentioned unified storage,
[17:05] butcanyouexpandonwhythatis-
[17:08] SPEAKER_1: Yeah.
[17:09] SPEAKER_0: It just sounds a bit too simple.
[17:09] SPEAKER_0: I mean-
[17:10] SPEAKER_1: I know.
[17:10] SPEAKER_0: -it sounds like people could have come up with this earlier, right?
[17:14] SPEAKER_1: Well, they've had.
[17:14] SPEAKER_1: They've had nice things in the business side for decades.
[17:17] They've been extremely expensive,
[17:18] but we're in the very like the cobbler children have no shoes in the software side.
[17:22] It's just like,
[17:23] "Well,
[17:23] wecouldmakeitworkwithourducttapeandbailingwire."
[17:28] I remember Vertica came out 15,
[17:28] 20 years ago,
[17:32] and the business...
[17:33] Anyway,
[17:35] a lot of folks were talking about unified observability,
[17:37] but a lot of situations when you look under a hood,
[17:40] what they're talking about is either a unified bill or best a unified visualization.
[17:45] And the beautiful thing about having unified storage is you have no dead ends.
[17:50] SPEAKER_1: Right?
[17:50] You click on a log,
[17:53] you can turn it into a trace,
[17:54] you can visualize it over time,
[17:56] you can derive your metrics from it.
[17:57] SPEAKER_1: You can derive your SLOs from it.
[17:59] You can take your SLO data,
[18:01] click on it,
[18:02] jump into it,
[18:03] see exactly which events are violating your SLO and why and what's different
[18:08] about them.
[18:09] SPEAKER_1: But you're right, there is a lot more to it.
[18:11] I try to emphasize that because when I started writing about this,
[18:15] I started to notice all these other observability 2.0 articles popping up
[18:19] all over the place where people are like,
[18:20] "Oh yeah,
[18:21] we do that too because blah,
[18:22] blah,
[18:22] blah,
[18:22] blah,
[18:22] blah."
[18:22] SPEAKER_1: And it's like, all right.
[18:24] Some of the other things that I think are associated with 2.0...
[18:27] And by the way,
[18:27] I'm not trying to police this and be like,
[18:29] "OnlyHoneycombdoesthis."
[18:30] SPEAKER_1: I don't think that at all.
[18:31] In fact,
[18:32] one of the most exciting things about this last year to me was that I feel like
[18:36] the batch of baby observability startups that we're seeing are no longer
[18:41] looking like cheaper Datadogs.
[18:43] SPEAKER_1: They're looking more cheaper Honeycombs.
[18:46] SPEAKER_1: They're built on ClickHouse, they're built in columnar stores.
[18:48] SPEAKER_1: They're using OTel Native.
[18:50] SPEAKER_1: They're using wide-structured events organized around units of work.
[18:55] SPEAKER_1: And I'm so ecstatic.
[18:57] SPEAKER_1: This is a better world for the industry and I'm excited.
[19:01] But some other things that I think have in common with this are,
[19:05] I think it really does parallel the shift from observability being an ops tool.
[19:09] It's organized around errors and downtime and outages and crashes and
[19:14] towards being something that underpins the whole development cycle.
[19:19] It's what underpins your feedback loops and allows you to...
[19:23] One of the most exciting use cases I think is in the CI/CD,
[19:26] being able to visualize it as a trace,
[19:28] see where your tests are breaking,
[19:30] where your time is going,
[19:31] because keeping that time between when you're building it and when it's in
[19:35] production and you're looking at it,
[19:37] keeping that as small as possible is the most fundamental part of building
[19:42] great software and great teams,
[19:43] I think.
[19:44] So when you say it underpins development,
[19:46] are you saying that these new tools,
[19:50] either that exist today or that will exist,
[19:52] you're kind of envisioning it as I am coding my stuff and as I push to just
[19:57] development to CI/CD and know there's a crash,
[20:01] something went wrong,
[20:02] I should be able to look at either if it's the build crash,
[20:06] I should be able to look at there.
[20:07] If the app crash,
[20:08] I can just look at this magical dashboard,
[20:10] which in the past I had to look at the logs or if I had a trace,
[20:15] look at the trace.
[20:16] So you're saying that I can just use this to actually develop faster,
[20:20] like me as a dev.
[20:22] SPEAKER_1: Basically, yeah.
[20:23] I mean,
[20:25] it can be hard to generalize because there are so many different ways that
[20:27] people do this and everything.
[20:29] I think that having observability in the pre-production environment is good,
[20:35] but I really think it's so much of it about accelerating time to value,
[20:38] accelerating time to insights,
[20:40] accelerating getting it into production.
[20:43] Some of the most exciting things that I think we're doing right now are about getting...
[20:49] I mean,
[20:49] I usually have my test and prod or live a life shirt on today,
[20:52] I have my database shirt on.
[20:55] But getting things into production immediately and safely,
[21:00] like using progressive deployment,
[21:02] using feature flags,
[21:04] and the combination of observability of the 2.0 framework plus things like
[21:10] progressive deployments,
[21:11] canaries,
[21:12] feature flags.
[21:13] It's greater than the sum of its parts because when you can slice and dice and
[21:13] 20 minutes,
[21:17] break down,
[21:18] say you get your code out within 15,
[21:20] it's so fresh in your head what you did,
[21:22] why you did it,
[21:23] how you did it,
[21:24] and you're looking,
[21:25] but it's not like you're blasting it out to everyone immediately.
[21:28] SPEAKER_1: Of course, of course not.
[21:30] You ship it behind a feature flag,
[21:32] you ship it to a canary,
[21:33] you ship it to 10%,
[21:35] and then you flip it on,
[21:37] you send some test requests,
[21:38] you do it in this very control,
[21:40] it's like using a scalpel.
[21:42] SPEAKER_1: You have this precision tooling that gives you confidence in moving fast.
[21:47] So am I hearing it correctly that what you're saying is when you have these
[21:51] modern and helpful engineering practices,
[21:53] things like feature flags,
[21:54] things like being able to test in production because you have some
[22:00] separation of tendencies,
[22:02] for example,
[22:04] all of those things.
[22:05] If you have those things and then you kind of invest in observability trying
[22:08] to have it better,
[22:10] that is just a much bigger win than let's say just,
[22:13] I don't know,
[22:13] let's get into this,
[22:14] whatever,
[22:14] observability 2.0 is,
[22:16] let's modernize it,
[22:16] but by itself,
[22:17] it's not going to be as big of a wind,
[22:18] right?
[22:19] SPEAKER_1: Correct.
[22:20] SPEAKER_1: Correct.
[22:21] SPEAKER_1: I use this metaphor a lot.
[22:22] SPEAKER_1: I'm super blind.
[22:23] So it's like putting on your glasses before you go barreling down the freeway,
[22:27] right?
[22:27] SPEAKER_1: When you're driving, you don't want to be just veering.
[22:31] SPEAKER_1: It shouldn't feel like you're always course correcting or driving.
[22:35] SPEAKER_1: It should feel like you're just driving, right?
[22:38] And when you're building,
[22:39] you should have these feedback loops that are so intuitive and so fast and so
[22:42] integrated that it feels like you're just building,
[22:44] right?
[22:45] SPEAKER_1: That's the dream.
[22:48] SPEAKER_0: Yeah.
[22:49] Now,
[22:50] one other question I have related to...
[22:54] We've talked about developers of observability,
[22:56] but how do you think o11y observability relates to...
[23:03] The short form.
[23:04] SPEAKER_0: How does it relate to SRE, DevOps, and other roles?
[23:08] Because I feel that there are some companies where they're saying,
[23:14] "Oh,
[23:14] observability?
[23:16] SPEAKER_0: That is owned by our SRE team or our DevOps team."
[23:19] SPEAKER_0: What's your view about this?
[23:28] I think obviously any vendor relationship,
[23:30] any product has to have an owner.
[23:32] SPEAKER_1: I think that's not necessarily a terrible idea.
[23:35] I think that a lot of the center of gravity is moving to platform teams in a lot
[23:39] of places because the arena is really managing that thin line,
[23:45] the fuzzy line between our code and infrastructure code.
[23:49] SPEAKER_1: It's permeable.
[23:50] SPEAKER_1: There's always things that somebody's got to own that line.
[23:54] But the other thing I like about the platform engineering model is that your
[23:58] customers are internal,
[24:00] right?
[24:00] SPEAKER_1: It's a product focused development organization.
[24:03] SPEAKER_1: Your customers are internal.
[24:05] And I like that model because some of the historical flaws of the sort of SRE,
[24:12] DevOps,
[24:12] whatever models have been that,
[24:14] oh,
[24:14] they own it.
[24:15] SPEAKER_1: They own monitoring.
[24:18] And the platform model is so explicitly you own your code and we're here to
[24:22] help you with that.
[24:23] SPEAKER_1: And that is, I think, such a critical change.
[24:26] SPEAKER_0: I could not agree more.
[24:28] It's interesting how it's a platform team,
[24:30] suddenly it's not they,
[24:31] but they kind of either reflect and it's you know that you have skin in the game
[24:35] and you cannot expect them to,
[24:36] for example,
[24:37] go on call for your stuff.
[24:38] SPEAKER_0: Platform teams just don't do that, right?
[24:40] SPEAKER_0: They will build a tooling for you.
[24:46] And to be clear,
[24:46] as someone at is deep in my DNA,
[24:47] and it's not like SREs are going anywhere.
[24:50] The systems are only getting harder and more complex,
[24:52] and there's an area for expertise.
[24:55] SPEAKER_1: And the consultative model is I think a great one.
[24:57] DevOps,
[24:58] if I might go on a little side rant here,
[25:00] I do feel like is there any term in computing that's been more contested than
[25:06] the word DevOps?
[25:07] I'm not sure,
[25:08] but whereas I feel like the DevOps philosophy of being very...
[25:13] You work together,
[25:16] empathy,
[25:17] collaboration is eternal and not restricted to software.
[25:21] I do feel like we're sort of in the waning days of the DevOps movement because
[25:28] it's no longer considered a good thing to do to split up a dev team and an ops
[25:33] team to then collaborate,
[25:35] right?
[25:36] Increasingly,
[25:36] there are only engineers who write code and own their code in production.
[25:40] SPEAKER_1: And I think this is really exciting.
[25:42] I think we can understand why dev versus ops evolved,
[25:49] but it was always kind of a crazy idea that half your engineers could build the
[25:53] software and the other half would understand and operate it.
[25:56] SPEAKER_1: That's just not a great way to break down.
[25:58] SPEAKER_1: It doesn't lead to excellence in either domain.
[26:02] Yeah,
[26:02] I feel it's a little bit similar to waterfall where people still talk about waterfall,
[26:06] but there is no waterfall.
[26:07] Waterfall used to be literally three or 4-year-old projects,
[26:11] long projects,
[26:12] but they don't exist.
[26:13] And so when people talk about waterfall,
[26:15] they talk about a two-month project.
[26:16] SPEAKER_0: That is not waterfall.
[26:18] If it's one or two months,
[26:19] you can call it whatever you want,
[26:21] which is a good thing,
[26:22] right?
[26:22] SPEAKER_0: It's a good thing that we don't have waterfall projects anymore.
[26:24] Even government projects no longer take,
[26:27] I mean,
[26:28] most of them don't take thankfully that long.
[26:30] Yeah,
[26:31] and to be clear,
[26:32] I get just as mad as anyone else when people are like,
[26:35] "DevOpsisdead."
[26:37] SPEAKER_1: No, it's not.
[26:38] SPEAKER_1: That's the wrong takeaway.
[26:39] SPEAKER_1: It's like any good movement eventually fulfills its purpose, right?
[26:44] SPEAKER_1: And I feel like the movement of DevOps is in the fulfillment stage.
[26:50] SPEAKER_0: I agree with that.
[26:52] So I have a question I've been wanting to ask you on why do you think
[26:59] observability is so darn hard?
[27:02] Every single developer has their story of why observability is hard if they
[27:07] actually did it wherever they sit.
[27:09] First of all,
[27:10] what was your first time where you just realized this is just really hard even
[27:14] though it shouldn't be?
[27:16] SPEAKER_1: If I can be honest, I actually always hated observability and monitoring.
[27:23] True story,
[27:29] I would do anything to weasel out of being the one to own it,
[27:32] including three companies in a row.
[27:35] SPEAKER_1: I hired my friend, Ben Hartshorn, who loves this stuff.
[27:37] SPEAKER_1: I'd be like, "Ben, please come work with me because I hate this shit."
[27:39] And he would come in and he would build all the graphs and I would bookmark them.
[27:41] SPEAKER_1: I have always hated this.
[27:42] SPEAKER_0: Oh my gosh, look where that got you.
[27:43] SPEAKER_1: I know, right?
[27:50] SPEAKER_1: Anyway, it's hard.
[27:53] It's hard because software is hard and observability the first line of defense,
[27:59] and it's like not only do you have to build the software,
[28:02] but you also have to have this sort of meta thread that's watching what you're
[28:06] doing and going,
[28:07] "Whatisfuturemegoingtoneedtoknow?"
[28:09] SPEAKER_1: Or "What is future me going to need to understand at 2:00 A.M.?"
[28:13] SPEAKER_1: And that's just a muscle that takes a long time to build it.
[28:17] SPEAKER_1: Right?
[28:18] And I also think that historically we've had a lot of tools that were so...
[28:24] Some of my gripes about the past generation of tooling is just that you really
[28:28] expected people to master at least two disciplines.
[28:32] So many of the tools for so long required you to convert the code that you were
[28:37] writing into physical resources like what is this doing to my CPU and my RAM?
[28:42] It's just like,
[28:45] okay,
[28:46] this is a little too much to ask of your average software engineer.
[28:50] SPEAKER_0: Okay, I'll ask an easier question.
[28:53] One of the biggest frustrations I think like non-technical people like the
[28:57] CEO or CFO have is they look at the bill of an organization.
[29:01] Number one is going to be the cloud costs or infrastructure depending if
[29:03] you're self-hosting.
[29:04] Number two is almost always observability,
[29:07] and it just feels like people feel that it gets out of hand.
[29:10] And when you ask engineers,
[29:12] obviously engineers are told optimize the bill or make it lower.
[29:16] SPEAKER_0: Why do you think the costs just get out of hand in so many cases?
[29:21] SPEAKER_0: I think if they don't get out of hand, that is the exception.
[29:23] SPEAKER_0: What makes it so darn expensive?
[29:26] I mean,
[29:26] what makes it so darn expensive is the complexity of our systems and our high bar.
[29:34] The easiest way to slash observability bill is to give fewer about your
[29:38] customer experience.
[29:40] And to be clear,
[29:43] there's a wide...
[29:44] There are companies who every single request,
[29:47] delivery companies,
[29:48] banks,
[29:49] you don't really have a choice.
[29:50] SPEAKER_1: You have to understand every single request.
[29:52] SPEAKER_1: So there's a certain built-in, and then there are advertising companies.
[29:56] SPEAKER_1: It's more spray and pray and you can get buy and buckets.
[29:59] SPEAKER_1: And so I think that's a legit angle to look at.
[30:03] The second one that I would call out is the thing I mentioned before,
[30:05] which is just the multiplier effect.
[30:08] How many times are you storing a record of this data for every request that
[30:13] enters your assistant?
[30:14] I think that Honeycomb,
[30:16] we've been building this for maybe nine years on January 1st.
[30:21] SPEAKER_0: Wow.
[30:22] SPEAKER_1: I know.
[30:23] We were so small for several years and in the last couple of years it just taken off.
[30:26] But a big part of the driver from 1.0 to 2.0 is people suddenly,
[30:31] money's not free anymore.
[30:32] And they're taking a look at the multiplier effect,
[30:37] and it's just unsustainable.
[30:38] No matter how tightly you're trying to keep control of costs,
[30:41] it's unsustainable if you're storing 15 different copies for every single request.
[30:45] SPEAKER_1: It's just unsustainable.
[30:47] The third one that I'll point out is the one that personally bothers me the most,
[30:51] which is cardinality.
[30:53] If you look at actual observability engineering teams using your
[30:56] traditional three pillars or bottles,
[30:58] they usually,
[30:59] and this is true,
[31:00] spend an outright majority of their time trying to govern cardinality.
[31:06] You can go to bed Friday night,
[31:08] have a $200,000 a month Datadog bill,
[31:11] make no code changes over the weekend and wake up Monday with a $2 million a
[31:16] month Datadog bill just because the cardinality changed out from under you.
[31:20] Can we pause here because I don't think every software engineer will
[31:24] actually understand what cardinality does,
[31:26] especially if you've not worked with observability?
[31:30] SPEAKER_0: Can we break it out?
[31:31] SPEAKER_0: What does it mean?
[31:32] SPEAKER_0: It is wicked important.
[31:35] I know,
[31:35] but the first time you meet at it,
[31:36] it's not an obvious concept,
[31:37] is it?
[31:40] SPEAKER_1: Yeah.
[31:41] SPEAKER_1: No, not at all.
[31:42] Cardinality refers to,
[31:44] I think the mathematicians call it the number of unique items in a set.
[31:48] And basically it means if you've got a collection of a hundred million users,
[31:53] any unique ID is going to be the highest possible cardinality.
[31:56] So like request ID or in America,
[31:59] like social security numbers,
[32:00] if you have a hundred million users,
[32:02] you have a hundred million values.
[32:05] And then the lowest possible cardinality would be a field with just one value.
[32:09] SPEAKER_1: Like species equals human.
[32:11] SPEAKER_1: Right?
[32:12] SPEAKER_0: Yeah.
[32:13] Now,
[32:13] the point of big M metrics tools is they are built to handle low cardinality data,
[32:18] full stop.
[32:20] And there's this very traditional experience that everyone who uses these
[32:26] tools which they start using it,
[32:29] they're happy with it,
[32:30] they append something like host name,
[32:33] and then they get to have more than a hundred hosts and suddenly it all breaks
[32:37] or it becomes absolutely good.
[32:39] SPEAKER_1: And they're just like, "What the hell?"
[32:41] And just to explain behind the scenes of why it gets so expensive,
[32:45] whatnot.
[32:49] SPEAKER_1: There's no relational data in time series databases.
[32:53] Every unique combination of number and value you have to store again and again.
[33:02] SPEAKER_1: So you have the term custom metrics.
[33:07] I always thought that that meant like,
[33:09] "Oh,
[33:09] a custom metric that you've defined in your code.
[33:11] SPEAKER_1: It's a line of code."
[33:12] SPEAKER_1: No, it refers to unique combinations of metrics, values.
[33:18] SPEAKER_0: So basically every single unique combination will take up more space.
[33:21] And this is,
[33:22] as you said,
[33:23] a rookie mistake I've seen when people use one of these.
[33:26] SPEAKER_1: Everybody.
[33:27] Many products is you just literally,
[33:29] you're like,
[33:29] "Okay,
[33:30] I don't know,
[33:32] I want to query for what city,
[33:34] whatcountrydoesthiseventhappen."
[33:37] And then they add IP address,
[33:39] and then IP address is unique to everyone,
[33:41] and suddenly it just adds so much to your bill.
[33:45] SPEAKER_1: Yeah, it can like 100X overnight.
[33:48] And so world-class observability engineering teams end up spending an
[33:52] outright majority of their time just trying to govern,
[33:55] because the irony is that is the most valuable data.
[33:59] The more unique the data is,
[34:00] the more identifying it is,
[34:01] which means the easier it is to debug your systems and understand what's happening.
[34:05] And it's not just storing the data,
[34:07] it's also being able to use it to slice and dice and break down and group by and explore.
[34:13] And so you just can't do this with stuff that's built on the metric data type.
[34:17] SPEAKER_1: It's actually impossible.
[34:20] SPEAKER_0: And this is what we talk about observability 1.0 or so.
[34:22] SPEAKER_0: So how does it change?
[34:26] It seems,
[34:27] this feels a little bit to me like if I had to '60s,
[34:33] you actually had to worry about where your program,
[34:37] you need to tell where it goes on a tape,
[34:38] it goes back and you structured your code accordingly.
[34:42] And it feels to me that if you're using these observability products,
[34:46] when you're thinking about,
[34:47] I want to lock stuff,
[34:48] you need to be thinking about how expensive will this be If I add this field,
[34:53] which is like an IP or a website,
[34:56] oh,
[34:56] I shouldn't add that.
[34:57] SPEAKER_0: I should transform it, which I mean in 2024, that sounds a bit silly.
[35:02] SPEAKER_1: It does, \[inaudible 00:35:03\].
[35:02] Because you're like you're optimizing machine time,
[35:05] even though machine time is cheap,
[35:07] but I guess storage is expensive.
[35:08] SPEAKER_0: And so what's the solution?
[35:11] The solution is we have to move away from tools that are backed by big M metrics.
[35:17] We have to move towards tools that use structured data where you can have high cardinality,
[35:24] where you can store lots of demand.
[35:28] SPEAKER_1: I've often said the bridge from 1.0 to 2.0 is logs, right?
[35:32] SPEAKER_0: Mh-hmm.
[35:33] SPEAKER_1: It's emitting fewer logs, but wider logs.
[35:38] The wider your log,
[35:39] the more context you're attaching to each event,
[35:43] and the more context you have,
[35:45] the better your ability to identify outliers and correlate things.
[35:49] We have this thing called bubble up,
[35:51] which is just like any graph you have,
[35:53] you can be like,
[35:53] "What'sthat?"
[35:54] Draw a little bubble around it and we'll compute for all the dimensions
[35:58] inside the bubble versus the base outside the bubble.
[36:00] SPEAKER_1: So you're like, "What's this little spike?"
[36:02] SPEAKER_1: And you're like, "Oh, I see.
[36:04] All of these are for requests coming from Android devices,
[36:07] from this region going to the secondary with the batch size of blah,
[36:10] blah,
[36:10] blah,
[36:11] andit'stakingthis."
[36:12] And so much of debugging boils down to,
[36:14] here's the thing,
[36:15] I care about why,
[36:16] what are all the ways it's different.
[36:18] And gathering up your data in this way,
[36:20] you ask why it's so expensive and it's because the model does not fit the needs
[36:25] that we have for this data.
[36:27] SPEAKER_0: And do I understand correctly?
[36:28] SPEAKER_0: Because the software engineers always trade off.
[36:30] I think it's easy enough to understand that observability 1.0,
[36:34] it optimizes for...
[36:37] When it stores the data,
[36:38] you can immediately query it and you will get it almost immediately.
[36:42] SPEAKER_0: You don't need too much computation.
[36:44] When you're doing the opposite of,
[36:46] okay,
[36:46] I can just store whatever,
[36:47] it's not going to expand my storage costs.
[36:49] SPEAKER_0: I'm assuming there's a trade-off with, let's say compute, right?
[36:52] You're going to compute it later,
[36:53] or you'll have post-processing or something,
[36:55] right?
[36:56] SPEAKER_0: You don't get anything for free.
[36:57] SPEAKER_0: No, you're shaking your head.
[37:00] SPEAKER_1: So you're right.
[37:00] There are always trade-offs,
[37:01] but it might not be exactly the trade-offs that you think,
[37:03] so this is made possible by the falling costs of storage and compute and all
[37:09] these things.
[37:09] SPEAKER_1: Absolutely no.
[37:11] Metrics were optimized for a world where all of your resources were so expensive.
[37:18] SPEAKER_1: You're just like, "I can't afford to store this NGINX log.
[37:21] SPEAKER_1: I'm going to derive some metrics and store that."
[37:25] SPEAKER_1: Right?
[37:26] SPEAKER_1: Yeah, no, you should be able to slice and dice in real time.
[37:31] Another aspect of this is I don't think you can really move to 2.0 world
[37:34] without taking your wide structured data and feeding it into a columnar store.
[37:38] If you're using a traditional relational database,
[37:39] you have to define and advance the indexes,
[37:43] the schemas,
[37:44] all these things.
[37:45] SPEAKER_1: You want it to be flexible.
[37:46] SPEAKER_1: You want to be able to just drop in, "Oh, this might be useful someday."
[37:49] SPEAKER_1: Drop it in and immediately-
[37:50] Well,
[37:50] yeah,
[37:51] for logging,
[37:51] you shouldn't have to...
[37:52] It feels a royal pain to do it.
[37:55] It feels a little bit like typically what happens when you don't have good
[37:59] logging or good practices,
[38:01] you have a bug,
[38:02] something that crashes with the customer,
[38:04] you realize you have zero logs,
[38:05] you have no way to look at it.
[38:06] SPEAKER_0: So what do you do?
[38:07] You ship something to production back in a mobile app,
[38:10] you add logs,
[38:11] and then you tell the customer,
[38:12] "We now have logs.
[38:13] SPEAKER_0: Can you please update the app and try it again?"
[38:15] And it's like,
[38:17] I mean,
[38:18] it's doable,
[38:18] but it's pretty darn embarrassing,
[38:19] right?
[38:20] SPEAKER_1: Yeah.
[38:20] The goal is to be capturing enough rich telemetry all the time that you can go
[38:26] back and you can just be like,
[38:27] "Oh,
[38:27] whatdidtheydo?"
[38:28] SPEAKER_1: Right?
[38:29] SPEAKER_0: Yeah.
[38:30] So a common worry about any observability these days,
[38:35] I mean,
[38:36] you could build your own observability,
[38:37] but unless you're Facebook or Google,
[38:40] you probably shouldn't do it.
[38:40] And they do it already,
[38:42] but there's this worry about vendor lock-in thinking,
[38:46] okay,
[38:46] whatever I choose,
[38:48] I might be locked in or should I try to choose a vendor that's not locked in?
[38:51] Now you work for a vendor,
[38:52] so you know you're on one end you're going to be biased,
[38:55] on the other end you often speak truth to power.
[38:59] SPEAKER_0: How big of a deal do you think vendor lock in is?
[39:01] SPEAKER_0: Can you avoid it?
[39:02] SPEAKER_0: Should you even want to avoid it?
[39:04] SPEAKER_0: Is it even possible?
[39:06] SPEAKER_1: This is a rare spot of good news in the landscape.
[39:08] SPEAKER_1: Historically, this has been a huge problem.
[39:12] SPEAKER_1: OpenTelemetry is changing everything.
[39:15] The goal with OpenTelemetry has always been you instrument your code with
[39:15] 95% true.
[39:19] OTel and you can basically take your fire hose and point it to whatever vendor
[39:24] you want,
[39:25] which is,
[39:26] there's a little bit,
[39:27] but it is 90,
[39:30] SPEAKER_1: This is a game changer.
[39:31] Forcing vendors to compete for your business based on being excellent and
[39:35] responsive and a good value instead of keeping you locked in their ecosystem is,
[39:42] and honestly,
[39:43] this is the first year where I've really seen this start to come true,
[39:46] and it's really exciting.
[39:48] SPEAKER_0: It's interesting because my next note was exactly this.
[39:50] SPEAKER_0: I made a note, OpenTelemetry.
[39:52] SPEAKER_0: I've heard about it.
[39:53] SPEAKER_0: And again, don't forget, I'm a bit of an outsider for observability.
[39:57] SPEAKER_0: I know software, but not the details of this.
[40:00] SPEAKER_0: What is OpenTelemetry?
[40:02] SPEAKER_0: You said how great it is, but what is it and why should we care about it?
[40:05] Yeah,
[40:06] I mean,
[40:06] it's the inheritor to Google had OpenCensus and OpenTracing,
[40:11] both kind of flopped.
[40:13] SPEAKER_1: Ben Sigelman and the folks at LightStep actually expect this out.
[40:18] SPEAKER_1: Get this.
[40:18] This is also the year where it overtook Kubernetes as the number one CNCF project.
[40:23] SPEAKER_0: OpenTelemetry.
[40:27] SPEAKER_1: OpenTelemetry is now the top project in terms of \[inaudible 00:40:31\].
[40:31] SPEAKER_1: Yeah, it's huge.
[40:32] SPEAKER_1: It's amazing.
[40:33] SPEAKER_1: And it's a few years old now.
[40:36] SPEAKER_1: It gets critiqued a bit for being kind of big and bloated.
[40:41] SPEAKER_1: It does the job it needs to do.
[40:45] SPEAKER_1: I think what most people need to understand is this.
[40:49] SPEAKER_1: It does a lot of jobs.
[40:50] It does them well,
[40:51] and increasingly you don't need to understand everything about it to get the
[40:55] value out,
[40:56] because for a long time it was clunky and you had to really invest into
[41:01] understanding it.
[41:02] Increasingly,
[41:03] it's getting to the point where it just accelerates a lot of the value is you
[41:06] get your data even if you don't think about OTel at all.
[41:10] If you get your data into an OTel-enabled pipeline,
[41:14] it gets consistent naming,
[41:17] consistent structure.
[41:19] SPEAKER_1: There's semantic conventions and stuff.
[41:20] And what this means is then when it gets to the server side,
[41:24] your vendors can do amazing shit for it.
[41:26] SPEAKER_1: We can intuit, we can derive, we can compute, we can calculate.
[41:30] We know what the data is,
[41:32] so we can do a lot of really exciting things with it.
[41:34] And I think you're going to see a lot more of that in the next couple of years.
[41:37] And so just to get a sense of what exactly it is,
[41:39] I'm on the website and it says how OpenTelemetry is a collection of APIs,
[41:43] SDKs,
[41:43] and tools.
[41:45] It's built to make telemetry portable and effective and it writes that you
[41:50] can use it,
[41:50] an instrument generate,
[41:51] collect and export telemetry data.
[41:53] So do I understand it correctly that a lot of languages or frameworks have
[41:57] APIs that you can use and then you can plug in vendors underneath it?
[42:02] SPEAKER_0: Or how do the vendors come in this picture here?
[42:05] SPEAKER_1: Yeah, some vendors support the own collectors.
[42:08] SPEAKER_1: A lot of are contributing back to the core project.
[42:12] But basically the idea is however you do your instrument trying to provide
[42:19] just frameworks for consistency,
[42:20] right?
[42:21] SPEAKER_1: It's a little bit too big to just generalize it.
[42:24] SPEAKER_1: If you're like, "Does the OTel do this?"
[42:25] SPEAKER_1: The answer is probably yes.
[42:28] But basically it's just getting people's telemetry into a consistent
[42:30] format with consistent naming and semantic conventions,
[42:34] which means that we can do a lot of great stuff with it.
[42:36] SPEAKER_0: Okay.
[42:36] So is it safe to say that if I'm working at now a midsize company or a project
[42:41] that I know will need observability or already needs it,
[42:44] it's kind of a safe bet to look at OpenTelemetry,
[42:48] see if I can at least parts of my pipeline adhere to it,
[42:52] because A,
[42:53] this will hopefully make it a bit better.
[42:55] And then B,
[42:56] if the question of portability comes up,
[42:57] it'll be way easier to move vendors if it ever comes up because we know that a
[43:02] lot of companies will never move,
[43:03] butIthinkknowingthatyoucould-
[43:06] SPEAKER_1: How do you \[inaudible 00:43:08\]?
[43:08] Well,
[43:08] it also helps,
[43:09] I mean,
[43:09] I'm talking a little bit against possibly your business,
[43:13] but as any business,
[43:15] the best negotiation is saying,
[43:17] "We could move if we wanted to.
[43:19] SPEAKER_0: So let's talk about should we do a longer term commitment?
[43:23] Do you have new features that you ship that are unique and your competitors
[43:26] don'thaveitandweneedit?"
[43:27] SPEAKER_0: And I have the same thing.
[43:30] It force the vendors to compete on the territory that they should be
[43:33] competing on.
[43:35] What are some common things that you've seen engineering teams get wrong
[43:39] about putting observability in place?
[43:41] SPEAKER_0: So one of the most common ones.
[43:43] SPEAKER_1: Oh, boy.
[43:46] Feeling like they don't need to have any until it's in production and things
[43:50] start breaking.
[43:52] SPEAKER_1: You really want to be developing with it.
[43:54] SPEAKER_1: You want to get in the habit of understanding yourself, like what?
[43:57] SPEAKER_1: You're going to attach a GDB to it.
[44:02] No,
[44:02] you want to debug your software the way you're going to debug it in production,
[44:06] like shift left,
[44:07] shift right,
[44:07] whatever the fuck you want to call it.
[44:08] SPEAKER_1: You want to do that early.
[44:14] Other areas,
[44:16] I think a lot of folks feel like,
[44:21] well,
[44:22] the dashboards versus...
[44:24] A lot of folks get really attached to their dashboards,
[44:25] and I really don't feel like unless your dashboard is dynamic and allows you
[44:29] to ask questions,
[44:30] I feel like it's a really poor view into your software.
[44:34] SPEAKER_1: You want to be like, "And then what?
[44:35] SPEAKER_1: And then what?
[44:36] SPEAKER_1: And then what?"
[44:36] SPEAKER_1: You want to be interacting with your data.
[44:39] If all you're doing is looking at static dashboards,
[44:42] I think it limits your ability to really develop a rich mental model of your software.
[44:49] And it means that there are things that you don't think to ask or graph or
[44:53] dashboard so you don't see them,
[44:55] right?
[44:56] Yeah,
[44:56] it's interesting because I built so many dashboards and whenever you walk
[44:59] into an office where there's a team,
[45:02] usually they have a dashboard and it looks cool.
[45:03] SPEAKER_0: And all of the dashboards we've had always looked cool.
[45:06] SPEAKER_0: And I'll be honest, they were kind of useless.
[45:09] I mean,
[45:09] they were good when someone important walked by,
[45:12] a director or VP,
[45:13] they're like,
[45:14] "Oh,
[45:14] this is the team.
[45:15] SPEAKER_0: Here's their sets."
[45:16] And after a while,
[45:17] what we started to do,
[45:17] because we realized that's what we mostly used it for,
[45:20] we made sure that nice numbers,
[45:21] and it was always green.
[45:23] SPEAKER_0: It sounds silly to say it.
[45:25] SPEAKER_1: No, I get it.
[45:26] SPEAKER_1: I get it.
[45:27] Like a public dashboard,
[45:27] you didn't really...
[45:28] So then we had private ones where we actually had the real stuff.
[45:34] Relatedly,
[45:35] I think that a habit the more and more teams are picking up,
[45:38] which I think is super important,
[45:40] is using SLOs as their entry point instead of using dashboards as their entry point.
[45:45] SPEAKER_0: Oh.
[45:47] SPEAKER_0: So you use it as an entry point for what?
[45:49] SPEAKER_1: For understanding, debugging, interacting.
[45:53] SLOs,
[45:54] I think I have this sticker that I made that's like SLOs are the API's for
[45:57] engineering teams,
[45:59] and SLO is your agreement.
[46:00] You're like,
[46:01] "We will provide a level of service that we all agree internally and
[46:05] externallyisgood."
[46:07] SPEAKER_1: This means we have a budget.
[46:08] We can use what's left over in our budget for running chaos engineering experiments.
[46:13] One of the things that we did at Honeycomb a few years ago was we had Kafka nodes
[46:17] that kept just vanishing on us,
[46:20] and it was really frustrating.
[46:21] And so we took some of our SLO budget and we started killing Kafka machines
[46:28] every day and working on the automated recovery process so that they would.
[46:34] SPEAKER_1: And to this day, every Monday, we kill the oldest Kafka node.
[46:38] SPEAKER_1: We just shoot it in the head.
[46:39] SPEAKER_1: And so we're always testing the bootstrapping process.
[46:43] We stopped getting paged in the middle of the night of Kafka nodes because
[46:46] we're constantly testing this thing.
[46:48] But if you have SLOs,
[46:50] it's also the greatest hedge against micromanagement,
[46:52] I think.
[46:53] Because if you're meeting your obligations,
[46:56] then how you spend your time as a team is below the fold.
[47:01] SPEAKER_1: Nobody should care about that if you are meeting your obligations.
[47:04] It's also a way for you to negotiate and be like,
[47:06] "Hey,
[47:06] we're not meeting your obligations,
[47:08] so we have to put a hold on this feature work because we have to do this
[47:11] reliability work because we have to get ourselves back to a place where we can
[47:16] deliveronourobligations."
[47:17] So there's just so many ways that I think...
[47:19] An antipattern I think is when your SLOs are not derived from the same data
[47:25] that you're using to debug.
[47:27] When it's something that's out there on a satellite it's not connected,
[47:30] it's like,
[47:30] well,
[47:30] now you have one more problem,
[47:32] right?
[47:32] SPEAKER_1: It's like you really want it to be like, "Here's my SLOs.
[47:36] SPEAKER_1: Ooh, I don't understand that my budget is going down faster."
[47:39] SPEAKER_1: Click on it and see why.
[47:43] I really like you saying that SLOs can be a way to avoid micromanagement,
[47:49] and it should be because this is no one's micromanagement.
[47:53] SPEAKER_0: I think it's fair, right?
[47:54] I think micromanagement is warranted when you're doing a terrible job,
[47:57] then the manager or tech leader,
[47:59] whoever,
[48:01] should come and look at you,
[48:02] but otherwise they should leave you alone.
[48:03] SPEAKER_0: And I think it's kind of fair.
[48:05] SPEAKER_0: It's a good way to think of it.
[48:07] I wish more teams would take this and hopefully they can take this as an inspiration.
[48:18] SPEAKER_0: So with Honeycomb, it's an observability startup that you're building.
[48:23] What was a major exciting or interesting engineering challenge that you
[48:28] have to solve to actually build this product?
[48:30] SPEAKER_1: I mean, we had to write our own database.
[48:33] SPEAKER_0: Your own database.
[48:34] SPEAKER_1: Yeah.
[48:34] SPEAKER_0: Really?
[48:34] SPEAKER_0: You're kidding me.
[48:36] SPEAKER_0: Why?
[48:37] Because when we started in 2016,
[48:40] to be clear,
[48:42] I've spent my entire career telling people,
[48:44] "Never write a database.
[48:46] SPEAKER_1: Don't do it.
[48:46] SPEAKER_1: Just never write a database."
[48:48] If you think you want to write a database,
[48:50] trust me,
[48:50] you don't want to write a database.
[48:53] SPEAKER_1: But Christine and I got started and we were just like, "Well, shit."
[48:57] SPEAKER_1: ClickHouse wasn't around.
[49:00] And ironically,
[49:01] we would have I guarantee if ClickHouse was around,
[49:04] we would've used it.
[49:05] And I'm now really grateful that it wasn't and we didn't because being able to...
[49:10] The data model is so custom to us,
[49:13] right?
[49:13] And being able to iterate on it,
[49:15] add traces,
[49:17] add all these things has been a real force multiplier for us.
[49:21] SPEAKER_1: I will say it's why we lost.
[49:23] When I was CEO,
[49:25] our earliest investor,
[49:27] I met with him a year in,
[49:28] I was like,
[49:28] "Well,
[49:28] we're starting to get some interest,
[49:30] butwe'regoingtoneedmoremoney."
[49:33] He's like,
[49:33] "Well,
[49:35] if you're going to succeed,
[49:36] you would've succeeded by now,
[49:37] andI'mnotgivingyouanymoremoney."
[49:39] SPEAKER_1: And I'm like, "Well..." And he's like, "Well, you know what?
[49:42] SPEAKER_1: You shouldn't have spent all this time fucking around writing a database.
[49:45] Youshouldhavefoundproductmarketfitfirstandthenwrittenadatabase."
[49:50] SPEAKER_1: And the thing is, as snotty as he was, he's right.
[49:54] SPEAKER_1: He's absolutely right.
[49:54] SPEAKER_1: That is the common wisdom.
[49:56] SPEAKER_1: That is 99 times out of a hundred, that is the right smart thing to do.
[50:01] SPEAKER_1: We were too dumb to know better, so we accidentally did the right thing.
[50:06] SPEAKER_0: And so how did your database help you?
[50:09] SPEAKER_0: What's it called even?
[50:10] SPEAKER_0: It's internal to you, right?
[50:11] SPEAKER_0: You never opened it up.
[50:12] SPEAKER_1: Yeah.
[50:12] SPEAKER_1: Yeah.
[50:12] SPEAKER_1: It's called Retriever.
[50:13] SPEAKER_1: So all of our services are called after dogs.
[50:16] We have Dalmatian and Poodle and Retriever and Bassett and Hound and all
[50:22] these things.
[50:24] SPEAKER_1: How has it helped us?
[50:25] SPEAKER_1: I mean, people-
[50:29] SPEAKER_0: What kind of database is it?
[50:31] SPEAKER_1: It's a columnar store.
[50:32] SPEAKER_1: It's a columnar store.
[50:33] SPEAKER_1: So it's been through a few different evolutions.
[50:36] And actually,
[50:36] so Sam Stokes gave a great talk at Strange Loop and I think 2018 about this,
[50:44] about the internals.
[50:44] SPEAKER_1: People can go and look it up.
[50:46] And then a couple years later,
[50:47] at the very last Strange Loop,
[50:49] Jessitron gave a talk about how we've evolved,
[50:52] because at some point around 2020 or so,
[50:55] we actually serverless our database.
[50:58] SPEAKER_1: We were like, "Okay."
[50:58] SPEAKER_1: So initially we're using the columnar store.
[51:01] It was all on local SSDs and EC2,
[51:06] but the vast majority of data that gets written to disk never gets queried by anyone,
[51:10] right?
[51:10] SPEAKER_1: And it was really expensive.
[51:12] And we're just like,
[51:13] "You can't build a business on this where 99% of data never gets queried,
[51:17] andyetwehavetopaytostoreit."
[51:18] And so Ian Wilkes,
[51:20] one of our principal engineers who's been here since the very beginning,
[51:23] he actually moved the query planner to Lambda jobs.
[51:28] And shortly after the data gets laid down on the SSDs,
[51:33] we actually age out to S3,
[51:34] and then we do this massive fan out and merge at query time.
[51:41] SPEAKER_0: All right.
[51:43] SPEAKER_0: That's pretty crazy that you built it.
[51:44] SPEAKER_0: I mean, congrats.
[51:49] SPEAKER_1: Yeah.
[51:49] No,
[51:49] I still tell people never write a database,
[51:50] but there's an asterisk once in a while,
[51:52] youreallycan't-
[51:54] Well,
[51:55] what I also like about this is there are startups that succeed because they
[52:02] don't follow the beaten path.
[52:03] In fact,
[52:04] some of the more interesting ones I talk with,
[52:07] we covered some in the news that I'll link it in the show notes below,
[52:10] but there's,
[52:12] Figma,
[52:12] for example,
[52:13] ignored the wisdom of launch in six months.
[52:16] SPEAKER_0: They took three or four years to build their first version.
[52:19] SPEAKER_0: Again, they burned a lot of money, but it was the right thing to do.
[52:22] There was another company,
[52:24] Antithesis,
[52:26] which built a advanced debugging tool for four years.
[52:31] SPEAKER_1: I just talked to the CEO of Antithesis just a few days ago.
[52:35] They also built their own database,
[52:36] but they did something wild,
[52:38] which is they didn't write any tests,
[52:39] and they're having their platform tested.
[52:41] SPEAKER_0: It is wild.
[52:43] SPEAKER_0: But again, he told me the same thing.
[52:45] SPEAKER_0: People say, "Don't write your own thing."
[52:46] SPEAKER_0: So I think it's kind of a little bit reassuring that-
[52:48] SPEAKER_1: I didn't know the rules write the rules.
[52:54] When you know where you want to go,
[52:54] I mean,
[52:55] there's a chance that you might run out of money,
[52:57] whatnot,
[52:57] but when you know,
[52:58] just do it,
[52:58] I mean,
[52:58] take all the advice,
[52:59] but you don't need to control C,
[53:00] control V to succeed.
[53:05] SPEAKER_1: Yeah.
[53:05] SPEAKER_1: No, 100%.
[53:05] SPEAKER_1: I'm a big fan of the innovation token metaphor.
[53:08] You've got two or three innovation tokens as startups who spend them wisely.
[53:12] SPEAKER_1: And we definitely spent two on our internal storage engine.
[53:18] SPEAKER_0: Of the three that you had in total back in the time.
[53:19] SPEAKER_1: Of the three, yeah.
[53:20] But it pays off every time we go to write a feature,
[53:23] we're not fighting our storage engine.
[53:25] We can build our storage engine to do what we want to do,
[53:27] and it's actually an incredible force multiplier.
[53:30] SPEAKER_0: Okay.
[53:30] So let's jump into a pretty interesting topic,
[53:33] which is observability and AI.
[53:36] SPEAKER_0: LLMs are a super hot topic these days, and they're everywhere.
[53:41] SPEAKER_0: How do you think about observability and AI systems?
[53:47] SPEAKER_1: Yeah, that's such a good question.
[53:49] So I feel like there are three places where AI really intersects with observability.
[53:54] SPEAKER_1: Number one is when you're building and training a model.
[53:56] SPEAKER_1: Number two is when you're developing with LLMs.
[53:59] And number three is the everyone problem of we're all now dealing with this
[54:06] influx of software of unknown origin.
[54:08] It used to be you could pretty much guarantee that someone somewhere
[54:13] understood the software at some point,
[54:15] and you can no longer take that for granted.
[54:20] And that is,
[54:20] I think,
[54:21] so funny because it really hearkens back to the origin story of Honeycomb at Parse.
[54:26] We had developers all over the world just writing snippets of JavaScript and
[54:29] uploading and we just had to make it work,
[54:31] right?
[54:31] MongoDB queries,
[54:33] they'd write them and upload them,
[54:34] we just had to make it work.
[54:35] And so many of the things we just sort of forged in fire to understand this
[54:40] unknown software is like,
[54:41] "Oh,
[54:41] nowthisisaworldwideproblemthateveryoneishaving."
[54:43] SPEAKER_1: So it's a little fun.
[54:46] SPEAKER_1: I wrote a blog post about observability in AI last week, and...
[54:51] SPEAKER_0: Oh, cool.
[54:52] Yeah,
[54:53] it's not...
[54:54] I don't think it's super mind-blowing or anything,
[54:56] but a couple of the conclusions that we're coming to is basically,
[55:00] first of all,
[55:00] if you can compute the answer,
[55:01] you should probably compute the answer.
[55:03] We see a lot of observability vendors out there who are like,
[55:06] "AI,
[55:07] thisandthat."
[55:08] SPEAKER_1: And it's like, yeah, but it's actually, it's a guess.
[55:12] And if you had gathered the context,
[55:14] then you could have computed it and it would've been faster,
[55:16] cheaper,
[55:16] easier,
[55:17] and better.
[55:17] But instead,
[55:18] you're like,
[55:18] "We put AI on it and it's a get," that's not better.
[55:21] SPEAKER_1: It's not better.
[55:21] SPEAKER_1: Just it has AI on it.
[55:23] SPEAKER_1: AI can make things worse too.
[55:26] There are certain problems where AI is the right tool for their job when it
[55:29] comes to calculation computation,
[55:30] not one of them.
[55:34] Another thing that we're really seeing a lot of,
[55:37] and unfortunately we have actually some customers who are really
[55:42] sophisticated model builders and stuff,
[55:44] but everybody in the AI community is so tight-lipped,
[55:48] they don't want to talk about anything.
[55:51] SPEAKER_1: And it's like, "Guys, come on."
[55:54] But I feel like one of the early lessons for Philip and me has been that you
[56:01] can't have good AI observability in isolation.
[56:06] SPEAKER_1: You have to have it embedded in good software observability.
[56:11] There are all these startups out there that are raising just buckets of cash
[56:14] to solve the problem of AI observability,
[56:16] and they're all focusing on the sort of self-contained models,
[56:21] but the inputs from all these different services and data and humans and stuff.
[56:27] SPEAKER_1: It's a trace, right?
[56:28] SPEAKER_1: It's a trace shaped problem.
[56:30] You have to be able to trace it all the way from all these inputs up here in
[56:33] software land through the model to the human feedback.
[56:39] SPEAKER_1: It's a classic high card now.
[56:41] And by AI observability,
[56:42] you mean that the problem is like,
[56:43] "Okay,
[56:43] I have an LLM or a system that uses LLMs in my software and I want to add
[56:49] observabilitytoittounderstandhowitworks."
[56:52] And you're saying that a lot of the startups are focused on,
[56:54] "All right,
[56:54] let's just wrap the model into observability inputs and outputs and we'll
[56:58] seeit."
[56:58] Whereas this thing,
[57:00] it actually it is built around other parts of your software.
[57:03] You want to see the user interface,
[57:05] you want to connect to customer support tickets,
[57:08] that kind of stuff.
[57:09] SPEAKER_1: Yeah, absolutely.
[57:10] SPEAKER_1: It's a trace shaped problem.
[57:14] SPEAKER_1: Yeah, it's a trace shape problem.
[57:16] It's a high cardinality shaped problem and it's a high dimensionality shape problem.
[57:20] SPEAKER_1: This is a software problem with non-deterministic elements.
[57:25] SPEAKER_1: It's not an AI problem is how I think of it.
[57:28] Yeah,
[57:29] well,
[57:29] there's going to be a lot of money in it for sure just because how much money
[57:33] there is in AI.
[57:34] But I really liked your third point,
[57:37] which is because you said that the three buckets are,
[57:39] number one is observability for these LLM models or when companies are
[57:43] writing AI.
[57:43] Number two was observability for developers who are writing code using LLMs.
[57:48] And number three,
[57:49] which I think is the most important is observability for this code generated
[57:52] by AI,
[57:53] which we know is hard to tell if it's good quality or not,
[57:57] but there's going to be just more of it.
[57:58] And this is what you said that back at Parse,
[58:00] you were used to just adding observability for basically all sorts of code,
[58:05] like JavaScript and whatever that people upload it to power their mobile apps.
[58:10] SPEAKER_1: Yep, exactly, exactly.
[58:12] SPEAKER_1: Production is where code meets reality.
[58:15] It's where it doesn't matter how pretty it looked,
[58:17] doesn't matter how great.
[58:20] You don't know if your code is good or not until you've watched it run in production.
[58:24] So I feel like so many things in the age of AI,
[58:27] this is not new,
[58:29] it's just a really intensified version of what we're actually dealing with.
[58:33] SPEAKER_0: Yeah.
[58:35] And what it tells me is any company that wants to use these AI agents,
[58:40] maybe these new AI agents or just more AI code,
[58:44] probably the prerequisite of doing that and not like burning,
[58:48] have good observability so that when stuff breaks and from there on,
[58:52] it might be a next step of figuring out,
[58:54] can there be a feedback loop?
[58:55] Can I actually allow some of this AI to push production and all of those things?
[58:59] But without that you're going to be flying blind,
[59:01] which is just stupid to do with something as unreliable as an LLM.
[59:06] SPEAKER_1: Exactly.
[59:07] A different topic,
[59:08] but also pretty important one,
[59:11] every company these days has a choice of building,
[59:14] buying or using open source.
[59:16] Now in the case of observability,
[59:17] usually this goes down to should we buy or should we use open source?
[59:21] SPEAKER_0: Because again, building from scratch doesn't really make sense.
[59:24] SPEAKER_0: All of these have upsides.
[59:25] Do you predict any of these to gain more momentum based on what you're seeing?
[59:30] Do you see more companies might be going to use open source and try to host them,
[59:35] or maybe more of them are giving up and going to vendors and maybe using OpenTelemetry,
[59:40] those kinds of things?
[59:42] I think insofar as OpenTelemetry is open source,
[59:46] I think it has a really bright future and I'm so relieved to see it when it comes
[59:51] to using open source,
[59:53] running your own versus using vendors.
[59:56] SPEAKER_1: The main trend that I'm seeing in this space is consolidation.
[59:59] I think it's part of people just trying to get a handle on their bills,
[1:00:03] which totally makes sense.
[1:00:05] If you're paying five different vendors and each of those vendors is going
[1:00:09] for your 15%,
[1:00:10] I feel like a reasonable benchmark for observability spend is like 15 to 20%
[1:00:17] of your cloud spend depending on the type of business or whatever.
[1:00:21] SPEAKER_1: I think it's a good rule of thumb.
[1:00:22] But if you're paying for five different vendors and each of them are gunning
[1:00:25] for that 15 to 20% of your cloud spend,
[1:00:28] that's nuts.
[1:00:30] SPEAKER_1: Right?
[1:00:30] SPEAKER_1: You can't do that.
[1:00:32] SPEAKER_1: So there's a big consolidation in the industry.
[1:00:35] The only open source vendor that's really involved is Grafana,
[1:00:39] and Grafana is doing very well.
[1:00:40] SPEAKER_1: They just raised a huge round.
[1:00:44] SPEAKER_1: I think different models... Although I guess should count Prometheus too.
[1:00:51] I think of Prometheus and Datadog as the last best capital metrics products
[1:00:58] that will ever be built.
[1:01:01] Nobody's ever going to try and launch another big project because what's the point?
[1:01:05] These are mature,
[1:01:06] they're great,
[1:01:07] honestly,
[1:01:07] and they're just very mature technologies and metrics,
[1:01:13] for all the shit talking that I do,
[1:01:15] metrics have a place in the ecosystem right now.
[1:01:19] It's like 80% of what people use is metrics and 20% is structured data and they
[1:01:23] need to invert that.
[1:01:24] SPEAKER_1: It needs to be 80% structured data and 20% metrics.
[1:01:27] But metrics are still great for cheaply plotting trends over long periods of
[1:01:35] time or for counters.
[1:01:39] Counters are a very essential metric thing or at a certain scale,
[1:01:46] which is a lot higher than a lot of people think.
[1:01:49] SPEAKER_1: Structured data is too expensive and you should use metrics.
[1:01:51] SPEAKER_1: So there are niche use cases for metrics.
[1:01:53] SPEAKER_1: And so Prometheus I think will continue to be a contender.
[1:01:55] SPEAKER_1: I think Grafana is great.
[1:01:57] I think that Datadog,
[1:01:59] et cetera,
[1:02:00] but in general,
[1:02:01] most people are using vendors and most people don't want to have to deal with
[1:02:04] this stuff under the hood.
[1:02:06] When everything breaks at 2:00 A.M.,
[1:02:08] you don't want to also be dealing with a broken observability tool on top of
[1:02:12] your other broken software.
[1:02:15] SPEAKER_1: So I mean, it makes sense to me.
[1:02:17] SPEAKER_1: And I'm not saying that just because I'm a vendor.
[1:02:21] SPEAKER_1: I mean, I wouldn't say that, I wouldn't.
[1:02:24] SPEAKER_0: Yeah, a question I got actually from a reader on social media.
[1:02:28] SPEAKER_0: What about front end and mobile observability?
[1:02:31] Because when we talk about it,
[1:02:32] it feels so much a bit as about the backend.
[1:02:35] SPEAKER_1: Yes.
[1:02:36] SPEAKER_1: Yes.
[1:02:37] So we actually launched our RUM replacement this year for front end trying to
[1:02:43] do the same thing for RUM that we have done for the backend.
[1:02:47] SPEAKER_0: What does RUM stand for?
[1:02:48] SPEAKER_1: Real user monitoring, and it's basically the front end.
[1:02:52] It's organized around browser sessions and user sessions instead of
[1:02:56] backend requests.
[1:02:57] But I think it's so critical that I feel like so often the borders of tools are
[1:03:02] what create silos.
[1:03:04] So if you've got one team over here that's using a completely different view
[1:03:08] into their software,
[1:03:09] then the other teams,
[1:03:11] it's like you spend more time arguing about the nature of reality than trying
[1:03:14] to solve the problem together when you have a common...
[1:03:17] Again,
[1:03:18] back to that unified storage,
[1:03:19] right?
[1:03:20] Many different views,
[1:03:21] many different entry points,
[1:03:22] but a unified view all the way from your mobile device or your browser to the
[1:03:27] database and back,
[1:03:28] it's such a powerful thing.
[1:03:32] Mobile is a different beast and we've started dipping our toe into it this year.
[1:03:39] It's no surprise that mobile is the only standalone solution out there
[1:03:45] that's sort of left.
[1:03:45] SPEAKER_1: And I think a lot of folks are trying to figure out how.
[1:03:47] I mean,
[1:03:48] you come from a mobile background,
[1:03:49] so you probably know all the reasons for this better than I do.
[1:03:52] SPEAKER_0: Yeah.
[1:03:52] SPEAKER_0: I mean, right, there was Crashlytics, which was acquired.
[1:03:56] It's such a weird thing because what I've heard is Crashlytics around like
[1:03:56] 2012 or something was acquired by Twitter and for a while kept alive and then
[1:04:06] it was destroyed.
[1:04:07] SPEAKER_0: And two things happened there.
[1:04:08] I think they bought it for about $300 million,
[1:04:09] and apparently it was such a low return that the VCs really got spooked and
[1:04:14] they never invested in another company that was doing this because they said
[1:04:17] if this is the biggest exit done was there and it always feels that mobile has
[1:04:22] been left by itself.
[1:04:24] SPEAKER_0: There are some tools, but none of them are really first class.
[1:04:27] None of them really come from the vendors that are doing the proper backend
[1:04:32] observability monitoring.
[1:04:33] And then I think everyone thinks it's such a small market,
[1:04:37] which I disagree with honestly.
[1:04:39] It's not small,
[1:04:40] but I do have a theory for why this is,
[1:04:43] and it's because the build pipeline is so alien and different because you
[1:04:49] can't use CI/CD,
[1:04:50] right?
[1:04:50] You've got the Apple Store gating or you've got the Android diaspora and the
[1:04:55] inability to fold it into the best practices of software development,
[1:05:01] I think that's why it's out in the island by itself.
[1:05:03] No,
[1:05:04] mobile remains surprisingly archaic in this,
[1:05:07] and a lot of it has to do with Apple and then Google going along,
[1:05:11] not allowing binary code to be shipped directly without their permission.
[1:05:16] Obviously companies,
[1:05:19] every team is going around,
[1:05:20] they're doing feature flags and there's some JavaScript here,
[1:05:23] JavaScript there,
[1:05:24] but it's all hush,
[1:05:24] hush under the hood.
[1:05:26] SPEAKER_0: Apple kind of knows, but it's different.
[1:05:29] SPEAKER_0: It's a fun, fun world.
[1:05:33] SPEAKER_0: Every area has its own challenges.
[1:05:35] SPEAKER_1: Every are does.
[1:05:36] SPEAKER_1: Yeah.
[1:05:37] And the last question I got from,
[1:05:40] again,
[1:05:40] someone on social media from this person wrote,
[1:05:46] "I'm starting a new company today.
[1:05:48] What is the right time to start investing in observability and how can I
[1:05:52] designforitupfront?"
[1:05:53] SPEAKER_0: Startup, fresh idea.
[1:05:58] SPEAKER_1: I think it's as integral to building software as tests.
[1:06:02] SPEAKER_1: I do.
[1:06:03] I think it's the same sort of...
[1:06:05] The best time to instrument,
[1:06:08] the best time to write tests,
[1:06:09] the best time to instrument is while you're writing the code.
[1:06:11] Anything that you try to slap on after the fact is not going to get at that
[1:06:15] original intent of what you were trying to do when you had it in your head.
[1:06:20] And I think that done correctly,
[1:06:21] it actually accelerates your development.
[1:06:24] SPEAKER_1: It doesn't hold you back.
[1:06:25] It accelerates your ability to get stuff out,
[1:06:28] to understand,
[1:06:29] to keep moving.
[1:06:30] And so yeah,
[1:06:31] I would say as soon as the code that you're writing is real,
[1:06:35] something you intend to put in front of a user,
[1:06:37] as soon as you start thinking about writing tests,
[1:06:40] you should be thinking about writing observability.
[1:06:44] It sounds like to me we're saying when you're prototyping and you're doing
[1:06:47] throw away stuff,
[1:06:48] and again,
[1:06:48] you wouldn't write any tests,
[1:06:50] don't worry about it,
[1:06:51] but when you're like,
[1:06:52] "Okay,
[1:06:52] this might go out," I really like the test analogy.
[1:06:55] SPEAKER_0: And you know the interesting thing with tests is?
[1:06:57] SPEAKER_0: There's two sides of people.
[1:06:58] SPEAKER_0: When you say, "Oh, tests will speed you up the long term."
[1:07:03] People who haven't written tests or haven't seen it,
[1:07:05] they're like,
[1:07:05] "That's BS.
[1:07:06] SPEAKER_0: It takes time.
[1:07:07] SPEAKER_0: It cannot slow me.
[1:07:08] SPEAKER_0: I'm just going to skip it."
[1:07:10] And then people who've seen it,
[1:07:11] they're like,
[1:07:11] "No,
[1:07:11] no,
[1:07:12] no,
[1:07:12] youdonotunderstand."
[1:07:14] And now you see the two startup founders,
[1:07:16] one of them is starting by writing the tests and they actually do get sped up
[1:07:20] where the other one says,
[1:07:21] "It'ssilly."
[1:07:22] SPEAKER_0: There's a comic about pushing the car on the square wheel.
[1:07:29] SPEAKER_0: Someone brings a round one saying, "Oh, let's change it."
[1:07:31] SPEAKER_0: "No, no, no.
[1:07:32] SPEAKER_0: It will take too much time."
[1:07:34] SPEAKER_1: So real.
[1:07:36] SPEAKER_0: I feel this might be the case with observability.
[1:07:39] SPEAKER_0: Once you've seen it, you probably cannot unsee it.
[1:07:41] SPEAKER_1: Exactly.
[1:07:43] SPEAKER_0: So let's close with some rapid questions.
[1:07:45] SPEAKER_0: I'll just shoot a question and you tell me whatever comes to mind.
[1:07:50] SPEAKER_0: People told me I need to ask something around management for you.
[1:07:53] SPEAKER_0: So I'll ask, do you like being an engineer or a manager more?
[1:07:58] SPEAKER_1: I love being an engineer.
[1:08:00] SPEAKER_1: I love it.
[1:08:01] One of the hardest things about the early years of Honeycomb were really rough.
[1:08:05] I didn't expect to have to be CEO and I wasn't a very good one,
[1:08:09] but so much of my identity came from being an engineer,
[1:08:12] and it was really,
[1:08:13] really hard for me to move past that.
[1:08:15] I'll say that when I was an engineering manager,
[1:08:17] I hated it,
[1:08:18] but now I look back,
[1:08:19] I'm like,
[1:08:19] "Oh,
[1:08:19] I kind of missed that," which is why I actually have this open calendar link
[1:08:23] where people can set up time to just bring their problems to me and talk about them,
[1:08:27] and I get such a kick out of it.
[1:08:28] SPEAKER_1: I miss those aspects of doing engineering management.
[1:08:31] SPEAKER_1: But I think being an engineer is you get paid to solve puzzles all day.
[1:08:37] That dopamine hit of just figuring things out,
[1:08:40] making things work,
[1:08:41] yet I would just go around feeling high all day.
[1:08:43] SPEAKER_1: It was really fun.
[1:08:46] SPEAKER_0: Yeah.
[1:08:48] SPEAKER_0: What is a controversial thing that you believe is true?
[1:08:52] SPEAKER_1: Absolutely nothing.
[1:08:53] SPEAKER_1: Gergely, how could you say such a thing to me?
[1:08:58] SPEAKER_0: What am I thinking?
[1:08:59] SPEAKER_0: All right.
[1:09:00] SPEAKER_1: What are you thinking?
[1:09:01] SPEAKER_1: A controversial thing that I believe to be true.
[1:09:03] I actually wrote an article yesterday I got out something about founder mode
[1:09:09] and I...
[1:09:13] So I'm not sure if this counts as controversial or not,
[1:09:16] but I think it counts as controversial is Silicon Valley like YC.
[1:09:20] I do not think that it's a good idea for the CEO to have to approve everything
[1:09:24] that goes out.
[1:09:24] SPEAKER_1: I think that is egotistical.
[1:09:27] SPEAKER_1: I think it is wrong.
[1:09:29] I think it hobbles good decision making and judgment in other parts of the organization.
[1:09:34] SPEAKER_1: I wouldn't want to work for someone like that.
[1:09:37] There's this group of folks who just idolize Steve Jobs,
[1:09:40] Gergely,
[1:09:41] and I think that Steve Jobs obviously very successful person,
[1:09:44] and I think in large part,
[1:09:46] despite the fact that he was a raging control freak asshole,
[1:09:49] I think he was successful despite that,
[1:09:51] not because of that.
[1:09:52] It makes me really sad to see so many bright people be like,
[1:09:55] "Oh,
[1:09:56] well,
[1:09:56] Ishouldalsobeanassholeandcontrolfreak."
[1:09:59] Think of how many brilliant,
[1:10:00] wonderful people probably left because they couldn't stand to be
[1:10:03] micromanaged in that way.
[1:10:04] Well,
[1:10:05] I think it also comes to show there's just so many ways to succeed that there's
[1:10:09] no one way.
[1:10:11] SPEAKER_1: There's so many ways, there's so many ways.
[1:10:11] SPEAKER_0: Then also to be fair, so many personalities, right?
[1:10:13] SPEAKER_0: So I'm going to just do a wild stab.
[1:10:17] SPEAKER_0: You don't operate like that, right?
[1:10:21] SPEAKER_1: No, you don't.
[1:10:21] For better or for worse,
[1:10:22] I've been a worker and an employee much longer than I've been a founder at a C-level.
[1:10:27] SPEAKER_1: And I don't know.
[1:10:29] It's so clear to me that the way that you bring out greatness in people is by
[1:10:34] supporting them and empowering them and giving them agency and giving them control.
[1:10:38] SPEAKER_1: And yes, be in the details.
[1:10:41] The advice of hire great people and get out of their way is terrible advice.
[1:10:46] You need to be doing the work to create alignment,
[1:10:48] to make sure that you have a shared view of what good looks like,
[1:10:51] what great looks like,
[1:10:52] so that you can course correct early when you start to diverge.
[1:10:55] So yes,
[1:10:56] be in the details,
[1:10:57] but you don't take people's agency away from them.
[1:11:00] So if you were not building an observability startup,
[1:11:04] what would you be doing?
[1:11:05] SPEAKER_1: I would be a staff engineer someplace.
[1:11:07] SPEAKER_1: In fact, that's what I plan to do next, is just go be an 5:00 P.M.
[1:11:12] I go home and I turn my brain off and it's like,
[1:11:14] "It's your problem now,
[1:11:15] bud."
[1:11:16] SPEAKER_1: It's going to be amazing.
[1:11:20] For some reason,
[1:11:21] I want to believe you,
[1:11:22] but I don't think you're going to be that even later.
[1:11:25] SPEAKER_0: The turning of the brain at 5:00 P.M.
[1:11:29] SPEAKER_0: I mean specifically.
[1:11:30] SPEAKER_1: Oh, that part, yeah.
[1:11:31] SPEAKER_1: Yeah.
[1:11:31] SPEAKER_1: Maybe not.
[1:11:33] SPEAKER_1: I mean it though.
[1:11:34] SPEAKER_1: I plan on going and being an engineer for a while.
[1:11:36] SPEAKER_1: I think you can only... I hate the term thought leader.
[1:11:40] It just makes me a little nauseous and I really think that the farther
[1:11:44] detached you get from the work,
[1:11:47] just the lower quality.
[1:11:48] SPEAKER_1: I don't know.
[1:11:49] For me,
[1:11:49] at some point,
[1:11:50] I need to circle back and do something with my hands in order to not cringe when
[1:11:54] I hear myself speaking.
[1:11:57] SPEAKER_0: I absolutely hear you.
[1:11:58] SPEAKER_0: So you're a big fan of whiskey.
[1:12:02] SPEAKER_1: I am.
[1:12:02] SPEAKER_0: What is your current favorite?
[1:12:03] For the longest time,
[1:12:04] I really like peated Scotches and it,
[1:12:07] I'm still kind of coming to terms with my identity as being more of a bourbon
[1:12:11] person now.
[1:12:13] SPEAKER_1: Yeah, no, interesting.
[1:12:15] SPEAKER_1: I really like WhistlePig, and I really like the Rye WhistlePig too.
[1:12:23] SPEAKER_1: My favorite ever is impossible to find now though.
[1:12:26] SPEAKER_1: It's called George T. Stagg, and it is like 190 proof.
[1:12:30] SPEAKER_1: It's so good.
[1:12:32] You can't find it,
[1:12:33] but they started making something called Stagg Jr.,
[1:12:35] which is like 80% is good.
[1:12:38] SPEAKER_1: So if you haven't tried it, try Stagg Jr.
[1:12:40] SPEAKER_0: All right.
[1:12:40] SPEAKER_0: And what's a nonfiction book you'd recommend for software engineers?
[1:12:44] SPEAKER_1: The book is called Fluke by Brian Klaas.
[1:12:49] SPEAKER_1: Fluke: Chance, Chaos, and Why Everything We Do Matters.
[1:12:53] I'm not a religious person,
[1:12:54] but I do really look for ways of having meaning in my life.
[1:13:00] And the takeaway from you is that everything we do really does matter because
[1:13:06] you don't know when the thing...
[1:13:08] A straight comment that you drop in someone's presence sets off something in
[1:13:12] their head that changes their life or causes them to stop happening or causes
[1:13:17] them to get sober,
[1:13:18] or all these ripple effects happen.
[1:13:21] Like the things that we do and say,
[1:13:24] yeah,
[1:13:24] maybe 95 don't actually set off,
[1:13:29] but things that we do really do matter.
[1:13:31] SPEAKER_1: It's called Fluke.
[1:13:33] SPEAKER_1: I've read it three times in the past year.
[1:13:34] SPEAKER_1: Cannot recommend it highly enough.
[1:13:38] SPEAKER_1: Yeah.
[1:13:41] Well,
[1:13:41] Charity,
[1:13:42] this was a really interesting and fun conversation as usual.
[1:13:46] SPEAKER_1: Thank you so much.
[1:13:46] SPEAKER_1: It's so nice to see you.
[1:13:47] SPEAKER_1: I wish we lived closer so we could get together more often.
[1:13:50] It's always a delight getting to spend a little bit of time just shooting the shit.
[1:13:54] SPEAKER_0: Thank you to Charity for sharing all these interesting details.
[1:13:57] Charity is a prolific writer,
[1:13:58] and you can read more from her on her blog at charity.wtf,
[1:14:02] which is linked in the show notes below.
[1:14:04] For a deep dive on how to build an observability startup and for details on how
[1:14:08] a scale up managed to have a $65 million observability built for a single year.
[1:14:14] SPEAKER_0: See The Pragmatic Engineer Deepdives linked in the show notes below.
[1:14:17] If you enjoyed this podcast,
[1:14:18] please do subscribe on your favorite podcast platform and on YouTube.
[1:14:22] SPEAKER_0: Thank you and see you in the next one.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.