Podcasts · Technology · Education
Data Engineering Podcast
Tobias Macey
300 episodes
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Episodes page 5 of 6
-
An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality
Summary The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution.…
-
Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations
Summary AirBnB pioneered a number of the organizational practices that have become the goal of modern data teams. Out of that culture a number of successful businesses were created to provide the tools and methods to…
-
An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications
Summary Data has permeated every aspect of our lives and the products that we interact with. As a result, end users and customers have come to expect interactions and updates with services and analytics to be fast and…
-
Understanding The Role Of The Chief Data Officer
Summary The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. As a result, not everyone understands what the responsibilities of the role are, when you…
-
Bringing Automation To Data Labeling For Machine Learning With Watchful
Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and…
-
Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery
Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking…
-
Useful Lessons And Repeatable Patterns Learned From Data Mesh Implementations At AgileLab
Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large…
-
Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus
Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. For…
-
Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda
Summary Exploratory data analysis works best when the feedback loop is fast and iterative. This is easy to achieve when you are working on small datasets, but as they scale up beyond what can fit on a single machine…
-
What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta
Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Because of its…
-
Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster
Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved…
-
Writing The Book That Offers A Single Reference For The Fundamentals Of Data Engineering
Summary Data engineering is a difficult job, requiring a large number of skills that often don’t overlap. Any effort to understand how to start a career in the role has required stitching together information from a…
-
Making The Total Cost Of Ownership For External Data Manageable With Crux
Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate…
-
Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast
Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. This podcast does its best to explore this fractal ecosystem, and…
-
Charting the Path of Riskified's Data Platform Journey
Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that…
-
Maintain Your Data Engineers' Sanity By Embracing Automation
Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide…
-
Be Confident In Your Data Integration By Quickly Validating Matching Records With data-diff
Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. While it is straightforward to know whether a synchronization process succeeded, it is not always clear whether…
-
The View From The Lakehouse Of Architectural Patterns For Your Data Platform
Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform…
-
Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform
Summary The proliferation of sensors and GPS devices has dramatically increased the number of applications for spatial data, and the need for scalable geospatial analytics. In order to reduce the friction involved in…
-
Strategies And Tactics For A Successful Master Data Management Implementation
Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what…
-
Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas
Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. In order to lower the barrier to entry Ryan Buick created the Canvas…
-
Level Up Your Data Platform With Active Metadata
Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great…
-
Discover And De-Clutter Your Unstructured Data With Aparavi
Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data…
-
Hire And Scale Your Data Team With Intention
Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone…
-
Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault
Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely…
-
Bringing The Modern Data Stack To Everyone With Y42
Summary Cloud services have made highly scalable and performant data platforms economical and manageable for data teams. However, they are still challenging to work with and manage for anyone who isn’t in a technical…
-
Data Cloud Cost Optimization With Bluesky Data
Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model…
-
A Multipurpose Database For Transactions And Analytics To Simplify Your Data Architecture With Singlestore
Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of…
-
Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy
Summary The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to…
-
Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte
Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to…
-
Designing And Deploying IoT Analytics For Industrial Applications At Vopak
Summary Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a…
-
Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way
Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds…
-
Scaling Analysis of Connected Data And Modeling Complex Relationships With The TigerGraph Graph Database
Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. These connections are best represented and analyzed as graphs to provide…
-
Exploring The Insights And Impact Of Dan Delorey's Distinguished Career In Data
Summary Dan Delorey helped to build the core technologies of Google’s cloud data services for many years before embarking on his latest adventure as the VP of Data at SoFi. From being an early engineer on the Dremel…
-
Leading The Charge For The ELT Data Integration Pattern For Cloud Data Warehouses At Matillion
Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. Matillion was an early innovator of that approach and in this episode CTO Ed Thompson explains how…
-
Evolving And Scaling The Data Platform at Yotpo
Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. Yotpo has been on a journey to evolve and…
-
Operational Analytics At Speed With Minimal Busy Work Using Incorta
Summary A huge amount of effort goes into modeling and shaping data to make it available for analytical purposes. This is often due to the need to simplify the final queries so that they are performant for…
-
Gain Visibility Into Your Entire Machine Learning System Using Data Logging With WhyLogs
Summary There are very few tools which are equally useful for data engineers, data scientists, and machine learning engineers. WhyLogs is a powerful library for flexibly instrumenting all of your data systems to…
-
Connecting To The Next Frontier Of Computing With Quantum Networks
Summary The next paradigm shift in computing is coming in the form of quantum technologies. Quantum procesors have gained significant attention for their speed and computational power. The next frontier is in quantum…
-
What Does It Really Mean To Do MLOps And What Is The Data Engineer's Role?
Summary Putting machine learning models into production and keeping them there requires investing in well-managed systems to manage the full lifecycle of data cleaning, training, deployment and monitoring. This…
-
DataOps As A Service For Your Data Integration Workflows With Rivery
Summary Data engineering is a practice that is multi-faceted and requires integration with a large number of systems. This often means working across multiple tools to get the job done which can introduce significant…
-
Synthetic Data As A Service For Simplifying Privacy Engineering With Gretel
Summary Any time that you are storing data about people there are a number of privacy and security considerations that come with it. Privacy engineering is a growing field in data management that focuses on how to…
-
Accelerate Development Of Enterprise Analytics With The Coalesce Visual Workflow Builder
Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to…
-
Repeatable Patterns For Designing Data Platforms And When To Customize Them
Summary Building a data platform for your organization is a challenging undertaking. Building multiple data platforms for other organizations as a service without burning out is another thing entirely. In this episode…
-
Eliminate The Bottlenecks In Your Key/Value Storage With SpeeDB
Summary At the foundational layer many databases and data processing engines rely on key/value storage for managing the layout of information on the disk. RocksDB is one of the most popular choices for this component…
-
Building A Data Governance Bridge Between Cloud And Datacenters For The Enterprise At Privacera
Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management…
-
Exploring Incident Management Strategies For Data Teams
Summary Data assets and the pipelines that create them have become critical production infrastructure for companies. This adds a requirement for reliability and management of up-time similar to application…
-
Accelerate Your Embedded Analytics With Apache Pinot
Summary Data and analytics are permeating every system, including customer-facing applications. The introduction of embedded analytics to an end-user product creates a significant shift in requirements for your data…
-
Accelerating Adoption Of The Modern Data Stack At 5X Data
Summary The modern data stack is a constantly moving target which makes it difficult to adopt without prior experience. In order to accelerate the time to deliver useful insights at organizations of all sizes that are…
-
Taking A Multidimensional Approach To Data Observability At Acceldata
Summary Data observability is a term that has been co-opted by numerous vendors with varying ideas of what it should mean. At Acceldata, they view it as a holistic approach to understanding the computational and…
More in Technology
All-In with Chamath, Jason, Sacks & Friedberg
All-In Podcast, LLC
Machine Gods
NPR
Acquired
Ben Gilbert and David Rosenthal
Hard Fork
The New York Times
TED Radio Hour
NPR
Search Engine
PJ Vogt
Dwarkesh Podcast
Dwarkesh Patel
The AI Daily Brief: Artificial Intelligence News and Analysis
Nathaniel Whittemore
Lex Fridman Podcast
Lex Fridman
The Vergecast
The Verge
Waveform: The MKBHD Podcast
MKBHD
Better Offline
Cool Zone Media and iHeartPodcasts
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.