Finish Slime - #6

Data Engineering, Analytics. No ML, no AI. The weekly dose of the data content you actually want to read!

dbt at Zendesk — Part I: Setting foundations for scalability

Zendesk used to have a manual process for data transformation, which was time-consuming and error-prone. They decided to use dbt to automate their data transformation process. Dbt helps Zendesk to create reusable and scalable data transformation pipelines. They improved their data transformation process's speed, accuracy, and scalability by using dbt.

Creating Harmony: Collaborative Database Schema Design for Better Data Teamwork

A collaborative database is a database that is designed to be used by multiple people or teams. It can help improve collaboration by providing a single source of truth for data and making it easier for people to share and access data. Collaborative databases can also help improve data quality by providing a way to track changes to data and making it easier to identify and correct errors.

Understanding database Indexes in PostgreSQL

An index is a data structure that helps PostgreSQL find rows in a table more quickly. Indexes are created on one or more columns of a table. When you run a query that uses an indexed column, PostgreSQL can use the index to find the rows that match the query criteria more quickly than if it had to scan the entire table.

Wrangling BigQuery at Reddit

Reddit uses BigQuery to store and analyze data. They have many queries running on BigQuery and needed a way to manage the workload. They created a system that uses BigQuery logs to track queries and identify resource usage. This system allows them to identify queries that are using too many resources and take action to reduce the load.

The State of Data Engineering 2023

In 2023, data engineering is still a rapidly growing field. Many new technologies and trends are emerging, such as lakehouses, data mesh, and federated learning. Data engineers are in high demand, and they need to be able to adapt to the changing landscape.

How Scale Kills Data Teams

As data teams grow, they often struggle to meet the scale demands. These problems can make it difficult for data teams to deliver the insights that businesses need. To avoid these problems, data teams need to adopt a scalable data architecture and processes.

5 Proven Best Practices for Measuring Data Team ROI

Data leaders are comfortable with metrics and numbers, but measuring data team ROI is universally challenging. Measuring ROI is difficult because data teams sit between technology and business. Data teams can increase the ROI of a business function’s operations, help identify or uncover net-new growth initiatives, find ways to scale initiatives and build new capabilities. Once you know how a data team drives impact across business functions, you can quantify that impact.

Pandas 2.0 and its Ecosystem (Arrow, Polars, DuckDB)

Pandas is a popular Python library for data manipulation and analysis. In version 2.0, Pandas introduced a new ecosystem of tools and libraries that make it easier to work with large datasets. These tools include Arrow, Polars, and DuckDB.

Arrow is a columnar storage format that can efficiently store and process large datasets. Polars is a Pandas-compatible library that provides additional features for working with large datasets. DuckDB is a fast, in-memory database that stores and queries large datasets.

The combination of Pandas, Arrow, Polars, and DuckDB provides a powerful data manipulation and analysis toolkit. These tools can be used to process large datasets quickly and efficiently, and they can be used to build a variety of data-driven applications.