Podcasts · Technology · Education

Data Engineering Podcast

Tobias Macey

300 episodes

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Episodes page 5 of 6

  1. An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality

    · 1 h 04 min · Episode 320

    Summary The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution.…

  2. Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations

    · 1 h 10 min · Episode 319

    Summary AirBnB pioneered a number of the organizational practices that have become the goal of modern data teams. Out of that culture a number of successful businesses were created to provide the tools and methods to…

  3. An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

    · 1 h 06 min · Episode 317

    Summary Data has permeated every aspect of our lives and the products that we interact with. As a result, end users and customers have come to expect interactions and updates with services and analytics to be fast and…

  4. Understanding The Role Of The Chief Data Officer

    · 47 min · Episode 318

    Summary The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. As a result, not everyone understands what the responsibilities of the role are, when you…

  5. Bringing Automation To Data Labeling For Machine Learning With Watchful

    · 1 h 20 min · Episode 316

    Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and…

  6. Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery

    · 53 min · Episode 315

    Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking…

  7. Useful Lessons And Repeatable Patterns Learned From Data Mesh Implementations At AgileLab

    · 49 min · Episode 314

    Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large…

  8. Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus

    · 59 min · Episode 313

    Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. For…

  9. Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda

    · 41 min · Episode 312

    Summary Exploratory data analysis works best when the feedback loop is fast and iterative. This is easy to achieve when you are working on small datasets, but as they scale up beyond what can fit on a single machine…

  10. What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta

    · 1 h 05 min · Episode 311

    Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Because of its…

  11. Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

    · 58 min · Episode 310

    Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved…

  12. Writing The Book That Offers A Single Reference For The Fundamentals Of Data Engineering

    · 1 h 01 min · Episode 309

    Summary Data engineering is a difficult job, requiring a large number of skills that often don’t overlap. Any effort to understand how to start a career in the role has required stitching together information from a…

  13. Making The Total Cost Of Ownership For External Data Manageable With Crux

    · 1 h 07 min · Episode 307

    Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate…

  14. Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast

    · 57 min · Episode 308

    Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. This podcast does its best to explore this fractal ecosystem, and…

  15. Charting the Path of Riskified's Data Platform Journey

    · 40 min · Episode 306

    Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that…

  16. Maintain Your Data Engineers' Sanity By Embracing Automation

    · 1 h 05 min · Episode 305

    Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide…

  17. Be Confident In Your Data Integration By Quickly Validating Matching Records With data-diff

    · 1 h 11 min · Episode 304

    Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. While it is straightforward to know whether a synchronization process succeeded, it is not always clear whether…

  18. The View From The Lakehouse Of Architectural Patterns For Your Data Platform

    · 59 min · Episode 303

    Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform…

  19. Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform

    · 1 h 07 min · Episode 301

    Summary The proliferation of sensors and GPS devices has dramatically increased the number of applications for spatial data, and the need for scalable geospatial analytics. In order to reduce the friction involved in…

  20. Strategies And Tactics For A Successful Master Data Management Implementation

    · 1 h 09 min · Episode 302

    Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what…

  21. Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

    · 43 min · Episode 300

    Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. In order to lower the barrier to entry Ryan Buick created the Canvas…

  22. Level Up Your Data Platform With Active Metadata

    · 53 min · Episode 299

    Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great…

  23. Discover And De-Clutter Your Unstructured Data With Aparavi

    · 49 min · Episode 298

    Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data…

  24. Hire And Scale Your Data Team With Intention

    · 1 h 01 min · Episode 297

    Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone…

  25. Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault

    · 54 min · Episode 296

    Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely…

  26. Bringing The Modern Data Stack To Everyone With Y42

    · 59 min · Episode 295

    Summary Cloud services have made highly scalable and performant data platforms economical and manageable for data teams. However, they are still challenging to work with and manage for anyone who isn’t in a technical…

  27. Data Cloud Cost Optimization With Bluesky Data

    · 1 h 03 min · Episode 293

    Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model…

  28. A Multipurpose Database For Transactions And Analytics To Simplify Your Data Architecture With Singlestore

    · 41 min · Episode 294

    Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of…

  29. Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy

    · 1 h 11 min · Episode 292

    Summary The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to…

  30. Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

    · 1 h 07 min · Episode 291

    Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to…

  31. Designing And Deploying IoT Analytics For Industrial Applications At Vopak

    · 48 min · Episode 289

    Summary Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a…

  32. Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way

    · 58 min · Episode 290

    Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds…

  33. Scaling Analysis of Connected Data And Modeling Complex Relationships With The TigerGraph Graph Database

    · 40 min · Episode 288

    Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. These connections are best represented and analyzed as graphs to provide…

  34. Exploring The Insights And Impact Of Dan Delorey's Distinguished Career In Data

    · 1 h 01 min · Episode 287

    Summary Dan Delorey helped to build the core technologies of Google’s cloud data services for many years before embarking on his latest adventure as the VP of Data at SoFi. From being an early engineer on the Dremel…

  35. Leading The Charge For The ELT Data Integration Pattern For Cloud Data Warehouses At Matillion

    · 53 min · Episode 286

    Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. Matillion was an early innovator of that approach and in this episode CTO Ed Thompson explains how…

  36. Evolving And Scaling The Data Platform at Yotpo

    · 1 h 04 min · Episode 285

    Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. Yotpo has been on a journey to evolve and…

  37. Operational Analytics At Speed With Minimal Busy Work Using Incorta

    · 1 h 11 min · Episode 284

    Summary A huge amount of effort goes into modeling and shaping data to make it available for analytical purposes. This is often due to the need to simplify the final queries so that they are performant for…

  38. Gain Visibility Into Your Entire Machine Learning System Using Data Logging With WhyLogs

    · 59 min · Episode 283

    Summary There are very few tools which are equally useful for data engineers, data scientists, and machine learning engineers. WhyLogs is a powerful library for flexibly instrumenting all of your data systems to…

  39. Connecting To The Next Frontier Of Computing With Quantum Networks

    · 40 min · Episode 282

    Summary The next paradigm shift in computing is coming in the form of quantum technologies. Quantum procesors have gained significant attention for their speed and computational power. The next frontier is in quantum…

  40. What Does It Really Mean To Do MLOps And What Is The Data Engineer's Role?

    · 1 h 16 min · Episode 281

    Summary Putting machine learning models into production and keeping them there requires investing in well-managed systems to manage the full lifecycle of data cleaning, training, deployment and monitoring. This…

  41. DataOps As A Service For Your Data Integration Workflows With Rivery

    · 58 min · Episode 280

    Summary Data engineering is a practice that is multi-faceted and requires integration with a large number of systems. This often means working across multiple tools to get the job done which can introduce significant…

  42. Synthetic Data As A Service For Simplifying Privacy Engineering With Gretel

    · 49 min · Episode 279

    Summary Any time that you are storing data about people there are a number of privacy and security considerations that come with it. Privacy engineering is a growing field in data management that focuses on how to…

  43. Accelerate Development Of Enterprise Analytics With The Coalesce Visual Workflow Builder

    · 43 min · Episode 278

    Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to…

  44. Repeatable Patterns For Designing Data Platforms And When To Customize Them

    · 47 min · Episode 277

    Summary Building a data platform for your organization is a challenging undertaking. Building multiple data platforms for other organizations as a service without burning out is another thing entirely. In this episode…

  45. Eliminate The Bottlenecks In Your Key/Value Storage With SpeeDB

    · 47 min · Episode 276

    Summary At the foundational layer many databases and data processing engines rely on key/value storage for managing the layout of information on the disk. RocksDB is one of the most popular choices for this component…

  46. Building A Data Governance Bridge Between Cloud And Datacenters For The Enterprise At Privacera

    · 1 h 03 min · Episode 275

    Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management…

  47. Exploring Incident Management Strategies For Data Teams

    · 57 min · Episode 274

    Summary Data assets and the pipelines that create them have become critical production infrastructure for companies. This adds a requirement for reliability and management of up-time similar to application…

  48. Accelerate Your Embedded Analytics With Apache Pinot

    · 1 h 13 min · Episode 273

    Summary Data and analytics are permeating every system, including customer-facing applications. The introduction of embedded analytics to an end-user product creates a significant shift in requirements for your data…

  49. Accelerating Adoption Of The Modern Data Stack At 5X Data

    · 54 min · Episode 272

    Summary The modern data stack is a constantly moving target which makes it difficult to adopt without prior experience. In order to accelerate the time to deliver useful insights at organizations of all sizes that are…

  50. Taking A Multidimensional Approach To Data Observability At Acceldata

    · 1 h 03 min · Episode 271

    Summary Data observability is a term that has been co-opted by numerous vendors with varying ideas of what it should mean. At Acceldata, they view it as a holistic approach to understanding the computational and…

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.