Podcasts · Technology · Education
Data Engineering Podcast
Tobias Macey
300 episodes
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Episodes page 4 of 6
-
An Exploration Of The Composable Customer Data Platform
Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. When it was difficult to wire together the event…
-
Mapping The Data Infrastructure Landscape As A Venture Capitalist
Summary The data ecosystem has been building momentum for several years now. As a venture capital investor Matt Turck has been trying to keep track of the main trends and has compiled his findings into the MAD (ML,…
-
Unlocking The Potential Of Streaming Data Applications Without The Operational Headache At Grainite
Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable…
-
Aligning Data Security With Business Productivity To Deploy Analytics Safely And At Speed
Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his…
-
Use Your Data Warehouse To Power Your Product Analytics With NetSpring
Summary With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and…
-
Exploring The Nuances Of Building An Intentional Data Culture
Summary The ecosystem for data professionals has matured to the point that there are a large and growing number of distinct roles. With the scope and importance of data steadily increasing it is important for…
-
Building A Data Mesh Platform At PayPal
Summary There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at…
-
The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse
Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain…
-
Let The Whole Team Participate In Data With The Quilt Versioned Data Hub
Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large…
-
Reflecting On The Past 6 Years Of Data Engineering
Summary This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done.…
-
Let Your Business Intelligence Platform Build The Models Automatically With Omni Analytics
Summary Business intelligence has gone through many generational shifts, but each generation has largely maintained the same workflow. Data analysts create reports that are used by the business to understand and…
-
Safely Test Your Applications And Analytics With Production Quality Data Using Tonic AI
Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to…
-
Building Applications With Data As Code On The DataOS
Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. Unfortunately it has also introduced new overhead to manage the full…
-
Automate Your Pipeline Creation For Streaming Data Transformations With SQLake
Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration…
-
Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI
Summary Making effective use of data requires proper context around the information that is being used. As the size and complexity of your organization increases the difficulty of ensuring that everyone has the…
-
Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams
Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his…
-
An Exploration Of Tobias' Experience In Building A Data Lakehouse From Scratch
Summary Five years of hosting the Data Engineering Podcast has provided Tobias Macey with a wealth of insight into the work of building and operating data systems at a variety of scales and for myriad purposes. In…
-
Simple And Scalable Encryption Of Data In Use For Analytics And Machine Learning With Opaque Systems
Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around…
-
Making Sense Of The Technical And Organizational Considerations Of Data Contracts
Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to…
-
Revisit The Fundamental Principles Of Working With Data To Avoid Getting Caught In The Hype Cycle
Summary The data ecosystem has seen a constant flurry of activity for the past several years, and it shows no signs of slowing down. With all of the products, techniques, and buzzwords being discussed it can be easy…
-
Convert Your Unstructured Data To Embedding Vectors For More Efficient Machine Learning With Towhee
Preamble This is a cross-over episode from our new show The Machine Learning Podcast, the show about going from idea to production with machine learning. Summary Data is one of the core ingredients for machine…
-
Run Your Applications Worldwide Without Worrying About The Database With Planetscale
Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of…
-
Business Intelligence In The Palm Of Your Hand With Zing Data
Summary Business intelligence is the foremost application of data in organizations of all sizes. The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing…
-
Adopting Real-Time Data At Organizations Of Every Size
Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. The promise of insights that are always accurate and up to date is appealing to organizations, but the…
-
Supporting And Expanding The Arrow Ecosystem For Fast And Efficient Data Processing At Voltron Data
Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and…
-
Analyze Massive Data At Interactive Speeds With The Power Of Bitmaps Using FeatureBase
Summary The most expensive part of working with massive data sets is the work of retrieving and processing the files that contain the raw information. FeatureBase (formerly Pilosa) avoids that overhead by converting…
-
A Look At The Data Systems Behind The Gameplay For League Of Legends
Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. In…
-
Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet
Summary The problems that are easiest to fix are the ones that you prevent from happening in the first place. Sifflet is a platform that brings your entire data stack into focus to improve the reliability of your data…
-
Taking A Look Under The Hood At CreditKarma's Data Platform
Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s…
-
Build Data Products Without A Data Team Using AgileData
Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With the rise in cloud platforms and self-serve data technologies the barrier of entry is…
-
Build Better Data Products By Creating Data, Not Consuming It
Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. There is an undeniable amount of value and utility in that information, but…
-
Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg
Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically…
-
Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense
Summary Business intelligence has grown beyond its initial manifestation as dashboards and reports. In its current incarnation it has become a ubiquitous need for analytics and opportunities to answer questions with…
-
Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt
Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread…
-
How To Bring Agile Practices To Your Data Projects
Summary Agile methodologies have been adopted by a majority of teams for building software applications. Applying those same practices to data can prove challenging due to the number of systems that need to be…
-
Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB
Summary The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases…
-
An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem
Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development…
-
Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks
Summary Logistics and supply chains are under increased stress and scrutiny in recent years. In order to stay ahead of customer demands, businesses need to be able to react quickly and intelligently to changes, which…
-
Making The Open Data Lakehouse Affordable Without The Overhead At Iomete
Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data…
-
Investing In Understanding The Customer Journey At American Express
Summary For any business that wants to stay in operation, the most important thing they can do is understand their customers. American Express has invested substantial time and effort in their Customer 360 product to…
-
Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee
Summary The global economy is dependent on complex and dynamic networks of supply chains powered by sophisticated logistics. This requires a significant amount of data to track shipments and operational…
-
Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin
Summary Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low…
-
Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations
Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the…
-
Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language
Summary Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. To help support the…
-
Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica
Summary In order to improve efficiency in any business you must first know what is contributing to wasted effort or missed opportunities. When your business operates across multiple locations it becomes even more…
-
Building A Shared Understanding Of Data Assets In A Business Through A Single Pane Of Glass With Workstream
Summary There is a constant tension in business data between growing siloes, and breaking them down. Even when a tool is designed to integrate information as a guard against data isolation, it can easily become a silo…
-
Build Confidence In Your Data Platform With Schema Compatibility Reports That Span Systems And Domains Using Schemata
Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. As they scale, the problems of visibility and dependency management can increase at an exponential…
-
Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data
Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered…
-
A Reflection On Data Observability As It Reaches Broader Adoption
Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their…
-
Introduce Climate Analytics Into Your Data Platform Without The Heavy Lifting Using Sust Global
Summary The global climate impacts everyone, and the rate of change introduces many questions that businesses need to consider. Getting answers to those questions is challenging, because the climate is a…
More in Technology
All-In with Chamath, Jason, Sacks & Friedberg
All-In Podcast, LLC
Machine Gods
NPR
Acquired
Ben Gilbert and David Rosenthal
Hard Fork
The New York Times
TED Radio Hour
NPR
Search Engine
PJ Vogt
Dwarkesh Podcast
Dwarkesh Patel
The AI Daily Brief: Artificial Intelligence News and Analysis
Nathaniel Whittemore
Lex Fridman Podcast
Lex Fridman
The Vergecast
The Verge
Waveform: The MKBHD Podcast
MKBHD
Better Offline
Cool Zone Media and iHeartPodcasts
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.