Podcasts · Technology · Education

Data Engineering Podcast

Tobias Macey

300 episodes

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Episodes page 3 of 6

  1. Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

    · 56 min · Episode 420

    Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the…

  2. Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

    · 51 min · Episode 419

    Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data…

  3. Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

    · 56 min · Episode 418

    Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned…

  4. Reconciling The Data In Your Databases With Datafold

    · 58 min · Episode 417

    Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming,…

  5. Version Your Data Lakehouse Like Your Software With Nessie

    · 41 min · Episode 416

    Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The…

  6. When And How To Conduct An AI Program

    · 46 min · Episode 415

    Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment…

  7. Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

    · 56 min · Episode 414

    Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of…

  8. Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

    · 59 min · Episode 413

    Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors…

  9. Data Sharing Across Business And Platform Boundaries

    · 1 h 00 min · Episode 412

    Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be…

  10. Tackling Real Time Streaming Data With SQL Using RisingWave

    · 57 min · Episode 411

    Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream…

  11. Build A Data Lake For Your Security Logs With Scanner

    · 1 h 03 min · Episode 410

    Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much…

  12. Modern Customer Data Platform Principles

    · 1 h 02 min · Episode 409

    Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with…

  13. Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

    · 50 min · Episode 408

    Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further…

  14. Designing Data Platforms For Fintech Companies

    · 48 min · Episode 407

    Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite,…

  15. Troubleshooting Kafka In Production

    · 1 h 15 min · Episode 406

    Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has…

  16. Adding An Easy Mode For The Modern Data Stack With 5X

    · 56 min · Episode 405

    Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending…

  17. Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

    · 51 min · Episode 404

    Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical…

  18. Designing Data Transfer Systems That Scale

    · 1 h 04 min · Episode 403

    Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building…

  19. Addressing The Challenges Of Component Integration In Data Platform Architectures

    · 30 min · Episode 402

    Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that…

  20. Unlocking Your dbt Projects With Practical Advice For Practitioners

    · 1 h 16 min · Episode 401

    Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a…

  21. Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

    · 1 h 08 min · Episode 400

    Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work…

  22. Shining Some Light In The Black Box Of PostgreSQL Performance

    · 55 min · Episode 399

    Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this…

  23. Surveying The Market Of Database Products

    · 47 min · Episode 398

    Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even…

  24. Defining A Strategy For Your Data Products

    · 1 h 04 min · Episode 397

    Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the…

  25. Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable

    · 1 h 08 min · Episode 396

    Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing…

  26. Using Data To Illuminate The Intentionally Opaque Insurance Industry

    · 52 min · Episode 395

    Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he…

  27. Building ETL Pipelines With Generative AI

    · 52 min · Episode 394

    Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it…

  28. Powering Vector Search With Real Time And Incremental Vector Indexes

    · 59 min · Episode 393

    Summary The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for…

  29. Building Linked Data Products With JSON-LD

    · 1 h 02 min · Episode 392

    Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata…

  30. An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem

    · 1 h 01 min · Episode 391

    Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This…

  31. Eliminate The Overhead In Your Data Integration With The Open Source dlt Library

    · 42 min · Episode 390

    Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source…

  32. Building An Internal Database As A Service Platform At Cloudflare

    · 1 h 01 min · Episode 389

    Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In…

  33. Harnessing Generative AI For Creating Educational Content With Illumidesk

    · 55 min · Episode 388

    Summary Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take…

  34. Unpacking The Seven Principles Of Modern Data Pipelines

    · 47 min · Episode 387

    Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at…

  35. Quantifying The Return On Investment For Your Data Team

    · 1 h 02 min · Episode 386

    Summary As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data?…

  36. Strategies For A Successful Data Platform Migration

    · 1 h 10 min · Episode 385

    Summary All software systems are in a constant state of evolution. This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this…

  37. Build Real Time Applications With Operational Simplicity Using Dozer

    · 41 min · Episode 384

    Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data…

  38. Datapreneurs - How Todays Business Leaders Are Using Data To Define The Future

    · 55 min · Episode 383

    Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. Bob Muglia has had a front-row seat to many of the major shifts driven by technology over his career.…

  39. Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling

    · 1 h 13 min · Episode 382

    Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. The major strategies in use today were created…

  40. How Data Engineering Teams Power Machine Learning With Feature Platforms

    · 1 h 04 min · Episode 381

    Summary Feature engineering is a crucial aspect of the machine learning workflow. To make that possible, there are a number of technical and procedural capabilities that must be in place first. In this episode Razi…

  41. Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

    · 50 min · Episode 380

    Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical…

  42. How Column-Aware Development Tooling Yields Better Data Models

    · 46 min · Episode 379

    Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have…

  43. Build Better Tests For Your dbt Projects With Datafold And data-diff

    · 48 min · Episode 378

    Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data…

  44. Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service

    · 54 min · Episode 377

    Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a…

  45. A Roadmap To Bootstrapping The Data Team At Your Startup

    · 43 min · Episode 376

    Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and…

  46. Keep Your Data Lake Fresh With Real Time Streams Using Estuary

    · 56 min · Episode 375

    Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads,…

  47. What Happens When The Abstractions Leak On Your Data

    · 27 min · Episode 374

    Summary All of the advancements in our technology is based around the principles of abstraction. These are valuable until they break down, which is an inevitable occurrence. In this episode the host Tobias Macey…

  48. Use Consistent And Up To Date Customer Profiles To Power Your Business With Segment Unify

    · 55 min · Episode 373

    Summary Every business has customers, and a critical element of success is understanding who they are and how they are using the companies products or services. The challenge is that most companies have a multitude of…

  49. Realtime Data Applications Made Easier With Meroxa

    · 45 min · Episode 372

    Summary Real-time capabilities have quickly become an expectation for consumers. The complexity of providing those capabilities is still high, however, making it more difficult for small teams to compete. Meroxa was…

  50. Building Self Serve Business Intelligence With AI And Semantic Modeling At Zenlytic

    · 49 min · Episode 371

    Summary Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes.…

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them. Explicit shows are for adult profiles only, and children’s profiles hear only children’s shows. More about this list.