Podcasts · Technology · Education
Data Engineering Podcast
Tobias Macey
300 episodes
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Episodes page 3 of 6
-
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer
Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the…
-
Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary
Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data…
-
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+
Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned…
-
Reconciling The Data In Your Databases With Datafold
Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming,…
-
Version Your Data Lakehouse Like Your Software With Nessie
Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The…
-
When And How To Conduct An AI Program
Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment…
-
Find Out About The Technology Behind The Latest PFAD In Analytical Database Development
Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of…
-
Using Trino And Iceberg As The Foundation Of Your Data Lakehouse
Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors…
-
Data Sharing Across Business And Platform Boundaries
Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be…
-
Tackling Real Time Streaming Data With SQL Using RisingWave
Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream…
-
Build A Data Lake For Your Security Logs With Scanner
Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much…
-
Modern Customer Data Platform Principles
Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with…
-
Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel
Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further…
-
Designing Data Platforms For Fintech Companies
Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite,…
-
Troubleshooting Kafka In Production
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has…
-
Adding An Easy Mode For The Modern Data Stack With 5X
Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending…
-
Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack
Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical…
-
Designing Data Transfer Systems That Scale
Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building…
-
Addressing The Challenges Of Component Integration In Data Platform Architectures
Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that…
-
Unlocking Your dbt Projects With Practical Advice For Practitioners
Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a…
-
Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine
Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work…
-
Shining Some Light In The Black Box Of PostgreSQL Performance
Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this…
-
Surveying The Market Of Database Products
Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even…
-
Defining A Strategy For Your Data Products
Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the…
-
Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable
Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing…
-
Using Data To Illuminate The Intentionally Opaque Insurance Industry
Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he…
-
Building ETL Pipelines With Generative AI
Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it…
-
Powering Vector Search With Real Time And Incremental Vector Indexes
Summary The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for…
-
Building Linked Data Products With JSON-LD
Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata…
-
An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem
Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This…
-
Eliminate The Overhead In Your Data Integration With The Open Source dlt Library
Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source…
-
Building An Internal Database As A Service Platform At Cloudflare
Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In…
-
Harnessing Generative AI For Creating Educational Content With Illumidesk
Summary Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take…
-
Unpacking The Seven Principles Of Modern Data Pipelines
Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at…
-
Quantifying The Return On Investment For Your Data Team
Summary As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data?…
-
Strategies For A Successful Data Platform Migration
Summary All software systems are in a constant state of evolution. This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this…
-
Build Real Time Applications With Operational Simplicity Using Dozer
Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data…
-
Datapreneurs - How Todays Business Leaders Are Using Data To Define The Future
Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. Bob Muglia has had a front-row seat to many of the major shifts driven by technology over his career.…
-
Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling
Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. The major strategies in use today were created…
-
How Data Engineering Teams Power Machine Learning With Feature Platforms
Summary Feature engineering is a crucial aspect of the machine learning workflow. To make that possible, there are a number of technical and procedural capabilities that must be in place first. In this episode Razi…
-
Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh
Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical…
-
How Column-Aware Development Tooling Yields Better Data Models
Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have…
-
Build Better Tests For Your dbt Projects With Datafold And data-diff
Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data…
-
Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service
Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a…
-
A Roadmap To Bootstrapping The Data Team At Your Startup
Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and…
-
Keep Your Data Lake Fresh With Real Time Streams Using Estuary
Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads,…
-
What Happens When The Abstractions Leak On Your Data
Summary All of the advancements in our technology is based around the principles of abstraction. These are valuable until they break down, which is an inevitable occurrence. In this episode the host Tobias Macey…
-
Use Consistent And Up To Date Customer Profiles To Power Your Business With Segment Unify
Summary Every business has customers, and a critical element of success is understanding who they are and how they are using the companies products or services. The challenge is that most companies have a multitude of…
-
Realtime Data Applications Made Easier With Meroxa
Summary Real-time capabilities have quickly become an expectation for consumers. The complexity of providing those capabilities is still high, however, making it more difficult for small teams to compete. Meroxa was…
-
Building Self Serve Business Intelligence With AI And Semantic Modeling At Zenlytic
Summary Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes.…
More in Technology
All-In with Chamath, Jason, Sacks & Friedberg
All-In Podcast, LLC
Machine Gods
NPR
Acquired
Ben Gilbert and David Rosenthal
Hard Fork
The New York Times
TED Radio Hour
NPR
Search Engine
PJ Vogt
Dwarkesh Podcast
Dwarkesh Patel
The AI Daily Brief: Artificial Intelligence News and Analysis
Nathaniel Whittemore
Lex Fridman Podcast
Lex Fridman
The Vergecast
The Verge
Waveform: The MKBHD Podcast
MKBHD
Better Offline
Cool Zone Media and iHeartPodcasts
Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.