- The Finish Slime
- Posts
- Finish Slime - #4
Finish Slime - #4
ELT 101, Dynamic Data Masking, Incremental Loads, MetaFlow, PostgreSQL.
Simple. Data Engineering, Analytics, no ML, no AI, no opinions. Just great content, shared.
Tutorials & Show Cases
Snowflake's Dynamic Data Masking is a data security feature that allows you to alter sections of data to keep their anonymity using a predefined masking strategy. It is a policy-based security feature that keeps the data in the database unchanged while hiding sensitive data in the query result set over specific database fields.
Writing efficient and correct incremental pipelines is difficult and an advanced use case. Incremental loads are necessary when working with large fact/event tables and can save time and money. There are two main ways to read data incrementally: maximal timestamp and date partitions.
Metaflow can efficiently load and process large amounts of tabular data from S3 using its optimized S3 client and Apache Arrow. This approach is becoming popular due to versatile tools like Apache Arrow and the ability of a single EC2 instance to handle large amounts of data.
Tools & Resources
ETL (Extract-Transform-Load) is a legacy architecture pattern for organizing data pipelines. It involves taking data from source systems, transforming it, and loading it into a target system like a data warehouse. ELT (Extract-Load-Transform) is a newer architecture pattern that is replacing ETL. The critical difference is that ELT involves loading the data into a target system before transforming it.
The article discusses the importance of leveraging technical skills for maximum impact in data roles. It emphasizes the need to focus on building what the organization needs, making good architectural decisions, and finding opportunities for scale.
PostgreSQL is a popular database management system (DBMS) that uses Multi-Version Concurrency Control (MVCC) to allow multiple queries to read and write to the database simultaneously. But the PostgreSQL MVCC implementation is among the worst. Read why.