Finish Slime - #2

dbt 1.5, data spring cleaning, duckDB, data alerts, big data, and dashboards are dead.

Simple. So you don’t slip in data engineering. No opinions. Just great content, shared. 

News

dbt Core v1.5 was released with features to help govern critical models across data practitioners. The goal is to manage the complexity of people by clarifying ownership, communicating across teams, and managing change.

Tutorials & Show Cases

Traditional retailers have streamlined operations. Their method of refreshing inventory every few months can be applied to data teams to address data debt incrementally.

dbt has rapidly gained popularity. It can potentially be an alternative to Spark for data transformations on top of a data lake. The advantages of dbt over Spark include a broader target audience, simple execution, and the fact that most companies don't have big data. And that’s where Duckdb enters the picture as well.

Tools & Resources

Dashboards are not dead, but they have fallen out of favor. There are now many alternatives to sharing data. The fundamental problem with dashboards is in relationships, communication, processes, and people. 

Managing error alerting workflow in larger data teams is difficult as it can lead to several adverse outcomes, such as overlooked important issues or loss of confidence in writing tests.

In this article, an example of a better alerting setup and steps to improve the alerting workflow are explored.

Big Data isn’t dead; it never was. Organizations just have different needs. They have big, medium, and small data needs. The data infrastructure community should optimize for helping people get insights from their data independent of size. And that’s currently a problem.