Finish Slime #13
Data Observability, dbt, Cube.& Droughty, DuckDB, Polars & Mosaic
Data Engineering, Analytics. No ML, no AI. The weekly dose of the data content you actually want to read!
Data observability is still an obscure trend. This article highlights 61 different use cases of observability, increasing revenue, saving time, and many others.
The data team at Compass built their new data platform on top of Databricks. They faced several challenges, including multiple overlapping analytics systems, inefficient ad hoc reporting processes, and the need for a unified machine learning infrastructure—and, of course, scale.
Instacarts ad system is a complex ecosystem that generates data, powering performance metrics and insights for ad campaigns. To overcome challenges with their previous ETL stack, Instacart implemented modular pipelines in Spark, enhancing testing, monitoring, and integrating streaming for incremental processing capabilities.
Let’s build a semantic layer with dbt Metrics, cube, and Droughty (paid, optional).
This article challenges the idea of real-time data and proposes that fresh data instead of real-time data is valuable. Indeed it is even precious to the data engineer because it leads to faster development cycles and more robust data flows.
Matt offers his perspective on DuckDB: DuckDB has the potential to be a cost-effective alternative to managed OLAP systems. It could serve as a staging layer for data transformation or provide a platform for tutorials and learning in the data engineering field.
Polars is the new pandas. The polars cookbook is a great place to start experiencing this new technology.
Mosaic is an extensible architecture for linking data visualizations, tables, input widgets, and other data-driven components, leveraging a backing database for scalable static and interactive views processing. With Mosaic, you can visualize and explore millions and even billions of data points at interactive rates.
Exclusive Gift 🎁
We got an exclusive gift for you! Data Engineering Reading Guide - the best books, articles & resources you must read!
And, of course, our (public)
How did you like this edition?