Yours

Podcasts · Technology · Education

Data Engineering Podcast

Tobias Macey

300 episodes

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Episodes page 2 of 6

  1. Enabling Agents In The Enterprise With A Platform Approach

    · 54 min · Episode 470

    Summary In this episode of the Data Engineering Podcast Arun Joseph talks about developing and implementing agent platforms to empower businesses with agentic capabilities. From leading AI engineering at Deutsche…

  2. Dagster's New Era: Modularizing Data Transformation in the Age of AI

    · 1 h 02 min · Episode 469

    Summary In this episode of the Data Engineering Podcast we welcome back Nick Schrock, CTO and founder of Dagster Labs, to discuss the evolving landscape of data engineering in the age of AI. As AI begins to impact data…

  3. AI and the Lakehouse: How Starburst is Pioneering New Workflows

    · 44 min · Episode 468

    Summary In this episode of the Data Engineering Podcast Alex Albu, tech lead for AI initiatives at Starburst, talks about integrating AI workloads with the lakehouse architecture. From his software engineering roots to…

  4. Amazon S3: The Backbone of Modern Data Systems

    · 1 h 01 min · Episode 467

    Summary In this episode of the Data Engineering Podcast Mai-Lan Tomsen Bukovec, Vice President of Technology at AWS, talks about the evolution of Amazon S3 and its profound impact on data architecture. From her work on…

  5. Scaling Data Operations With Platform Engineering

    · 42 min · Episode 466

    Summary In this episode of the Data Engineering Podcast Chakravarthy Kotaru talks about scaling data operations through standardized platform offerings. From his roots as an Oracle developer to leading the data…

  6. From Data Discovery to AI: The Evolution of Semantic Layers

    · 50 min · Episode 465

    Summary In this episode of the Data Engineering Podcast, host Tobias Macy welcomes back Shinji Kim to discuss the evolving role of semantic layers in the era of AI. As they explore the challenges of managing vast data…

  7. Balancing Off-the-Shelf and Custom Solutions in Data Engineering

    · 46 min · Episode 464

    Summary In this episode of the Data Engineering Podcast Tulika Bhatt, a senior software engineer at Netflix, talks about her experiences with large-scale data processing and the future of data engineering technologies.…

  8. StarRocks: Bridging Lakehouse and OLAP for High-Performance Analytics

    · 1 h 00 min · Episode 463

    Summary In this episode of the Data Engineering Podcast Sida Shen, product manager at CelerData, talks about StarRocks, a high-performance analytical database. Sida discusses the inception of StarRocks, which was…

  9. Exploring NATS: A Multi-Paradigm Connectivity Layer for Distributed Applications

    · 1 h 13 min · Episode 462

    Summary In this episode of the Data Engineering Podcast Derek Collison, creator of NATS and CEO of Synadia, talks about the evolution and capabilities of NATS as a multi-paradigm connectivity layer for distributed…

  10. Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog

    · 57 min · Episode 461

    Summary In this episode of the Data Engineering Podcast Viktor Kessler, co-founder of Vakmo, talks about the architectural patterns in the lake house enabled by a fast and feature-rich Iceberg catalog. Viktor shares…

  11. Simplifying Data Pipelines with Durable Execution

    · 40 min · Episode 460

    Summary In this episode of the Data Engineering Podcast Jeremy Edberg, CEO of DBOS, about durable execution and its impact on designing and implementing business logic for data systems. Jeremy explains how DBOS's…

  12. Overcoming Redis Limitations: The Dragonfly DB Approach

    · 44 min · Episode 459

    Summary In this episode of the Data Engineering Podcast Roman Gershman, CTO and founder of Dragonfly DB, explores the development and impact of high-speed in-memory databases. Roman shares his experience creating a…

  13. Bringing AI Into The Inner Loop of Data Engineering With Ascend

    · 53 min · Episode 458

    Summary In this episode of the Data Engineering Podcast Sean Knapp, CEO of Ascend.io, explores the intersection of AI and data engineering. He discusses the evolution of data engineering and the role of AI in…

  14. Astronomer's Role in the Airflow Ecosystem: A Deep Dive with Pete DeJoy

    · 52 min · Episode 457

    Summary In this episode of the Data Engineering Podcast Pete DeJoy, co-founder and product lead at Astronomer, talks about building and managing Airflow pipelines on Astronomer and the upcoming improvements in Airflow…

  15. Accelerated Computing in Modern Data Centers With Datapelago

    · 56 min · Episode 456

    Summary In this episode of the Data Engineering Podcast Rajan Goyal, CEO and co-founder of Datapelago, talks about improving efficiencies in data processing by reimagining system architecture. Rajan explains the shift…

  16. The Future of Data Engineering: AI, LLMs, and Automation

    · 1 h 00 min · Episode 455

    Summary In this episode of the Data Engineering Podcast Gleb Mezhanskiy, CEO and co-founder of DataFold, talks about the intersection of AI and data engineering. He discusses the challenges and opportunities of…

  17. Evolving Responsibilities in AI Data Management

    · 39 min · Episode 454

    Summary In this episode of the Data Engineering Podcast Bartosz Mikulski talks about preparing data for AI applications. Bartosz shares his journey from data engineering to MLOps and emphasizes the importance of data…

  18. CSVs Will Never Die And OneSchema Is Counting On It

    · 55 min · Episode 453

    Summary In this episode of the Data Engineering Podcast Andrew Luo, CEO of OneSchema, talks about handling CSV data in business operations. Andrew shares his background in data engineering and CRM migration, which led…

  19. Breaking Down Data Silos: AI and ML in Master Data Management

    · 58 min · Episode 452

    Summary In this episode of the Data Engineering Podcast Dan Bruckner, co-founder and CTO of Tamr, talks about the application of machine learning (ML) and artificial intelligence (AI) in master data management (MDM).…

  20. Building a Data Vision Board: A Guide to Strategic Planning

    · 50 min · Episode 451

    Summary In this episode of the Data Engineering Podcast Lior Barak shares his insights on developing a three-year strategic vision for data management. He discusses the importance of having a strategic plan for data,…

  21. How Orchestration Impacts Data Platform Architecture

    · 1 h 00 min · Episode 450

    Summary The core task of data engineering is managing the flows of data through an organization. In order to ensure those flows are executing on schedule and without error is the role of the data orchestrator. Which…

  22. An Exploration Of The Impediments To Reusable Data Pipelines

    · 52 min · Episode 449

    Summary In this episode of the Data Engineering Podcast the inimitable Max Beauchemin talks about reusability in data pipelines. The conversation explores the "write everything twice" problem, where similar pipelines…

  23. The Art of Database Selection and Evolution

    · 1 h 00 min · Episode 448

    Summary In this episode of the Data Engineering Podcast Sam Kleinman talks about the pivotal role of databases in software engineering. Sam shares his journey into the world of data and discusses the complexities of…

  24. Bridging Code and UI in Data Orchestration with Kestra

    · 45 min · Episode 447

    Summary In this episode of the Data Engineering Podcast, Anna Geller talks about the integration of code and UI-driven interfaces for data orchestration. Anna defines data orchestration as automating the coordination…

  25. Streaming Data Into The Lakehouse With Iceberg And Trino At Going

    · 40 min · Episode 446

    In this episode, I had the pleasure of speaking with Ken Pickering, VP of Engineering at Going, about the intricacies of streaming data into a Trino and Iceberg lakehouse. Ken shared his journey from product…

  26. An Opinionated Look At End-to-end Code Only Analytical Workflows With Bruin

    · 56 min · Episode 445

    Summary The challenges of integrating all of the tools in the modern data stack has led to a new generation of tools that focus on a fully integrated workflow. At the same time, there have been many approaches to how…

  27. Feldera: Bridging Batch and Streaming with Incremental Computation

    · 48 min · Episode 444

    Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The…

  28. Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent

    · 49 min · Episode 443

    Summary Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the challenges and innovations in data migrations. Gleb shares his experiences building and scaling data platforms at companies…

  29. Bring Vector Search And Storage To The Data Lake With Lance

    · 58 min · Episode 442

    Summary The rapid growth of generative AI applications has prompted a surge of investment in vector databases. While there are numerous engines available now, Lance is designed to integrate with data lake and lakehouse…

  30. The Role of Python in Shaping the Future of Data Platforms with DLT

    · 54 min · Episode 441

    Summary In this episode of the Data Engineering Podcast, Adrian Broderieux and Marcin Rudolph, co-founders of DLT Hub, delve into the principles guiding DLT's development, emphasizing its role as a library rather than…

  31. Build Your Data Transformations Faster And Safer With SDF

    · 43 min · Episode 440

    Summary In this episode of the Data Engineering Podcast Lukas Schulte, co-founder and CEO of SDF, explores the development and capabilities of this fast and expressive SQL transformation tool. From its origins as a…

  32. Scaling Airbyte: Challenges and Milestones on the Road to 1.0

    · 57 min · Episode 439

    Summary Airbyte is one of the most prominent platforms for data movement. Over the past 4 years they have invested heavily in solutions for scaling the self-hosted and cloud operations, as well as the quality and…

  33. Enhancing Data Accessibility and Governance with Gravitino

    · 39 min · Episode 438

    Summary As data architectures become more elaborate and the number of applications of data increases, it becomes increasingly challenging to locate and access the underlying data. Gravitino was created to provide a…

  34. The Evolution of DataOps: Insights from DataKitchen's CEO

    · 54 min · Episode 437

    Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to simplify the lives of data engineers. Chris explains the…

  35. Achieving Data Reliability: The Role of Data Contracts in Modern Data Management

    · 49 min · Episode 436

    Summary Data contracts are both an enforcement mechanism for data quality, and a promise to downstream consumers. In this episode Tom Baeyens returns to discuss the purpose and scope of data contracts, emphasizing…

  36. How Generative AI Is Impacting Data Engineering Teams

    · 55 min · Episode 435

    Summary Generative AI has rapidly gained adoption for numerous use cases. To support those applications, organizational data platforms need to add new features and data teams have increased responsibility. In this…

  37. The Role of Product Managers in Data-Centric Organizations

    · 53 min · Episode 434

    Summary In this episode Praveen Gujar, Director of Product at LinkedIn, talks about the intricacies of product management for data and analytical platforms. Praveen shares his journey from Amazon to Twitter and now…

  38. Neon: A Serverless And Developer Friendly Postgres

    · 58 min · Episode 433

    Summary Postgres is one of the most widely respected and liked database engines ever. To make it even easier to use for developers to use, Nikita Shamgunov decided to makee it serverless, so that it can scale from zero…

  39. Improve Data Quality Through Engineering Rigor And Business Engagement With Synq

    · 1 h 00 min · Episode 432

    Summary This episode features an insightful conversation with Petr Janda, the CEO and founder of Synq. Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data…

  40. Stitching Together Enterprise Analytics With Microsoft Fabric

    · 53 min · Episode 431

    Summary Data lakehouse architectures have been gaining significant adoption. To accelerate adoption in the enterprise Microsoft has created the Fabric platform, based on their OneLake architecture. In this episode…

  41. Being Data Driven At Stripe With Trino And Iceberg

    · 53 min · Episode 430

    Summary Stripe is a company that relies on data to power their products and business. To support that functionality they have invested in Trino and Iceberg for their analytical workloads. In this episode Kevin Liu…

  42. X-Ray Vision For Your Flink Stream Processing With Datorios

    · 42 min · Episode 429

    Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming…

  43. Practical First Steps In Data Governance For Long Term Success

    · 1 h 01 min · Episode 428

    Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of…

  44. Data Migration Strategies For Large Scale Systems

    · 1 h 00 min · Episode 427

    Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been…

  45. Zenlytic Is Building You A Better Coworker With AI Agents

    · 54 min · Episode 426

    Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration…

  46. Release Management For Data Platform Services And Logic

    · 20 min · Episode 425

    Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services…

  47. Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

    · 54 min · Episode 424

    Summary Artificial intelligence has dominated the headlines for several months due to the successes of large language models. This has prompted numerous debates about the possibility of, and timeline for, artificial…

  48. Build Your Second Brain One Piece At A Time

    · 50 min · Episode 423

    Summary Generative AI promises to accelerate the productivity of human collaborators. Currently the primary way of working with these tools is through a conversational prompt, which is often cumbersome and unwieldy. In…

  49. Making Email Better With AI At Shortwave

    · 54 min · Episode 422

    Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized…

  50. Designing A Non-Relational Database Engine

    · 1 h 16 min · Episode 421

    Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode…

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.