- Knowledge Sharing - Data Engineering skilling
- Table of Content (ToC)
- Overview
- References
- Articles and posts
- Idempotency with Spark
- Complete SQL and PySpark guide
- End-to-end project in the AI era
- Full open source lakehouse
- Data Engineering - How to transition from one role to the next
- The Essential Skill Stack: Foundational Knowledge Every Data Engineer Must Hav
- Data Engineering is not one job anymore
- Claude Skills in Data Engineering
- Spark on Luminousmen on Substack
- Awesome open source Data Engineering - Resources by Gunnar Morling
- Data ingestion methods
- Data Talks Club - Zoomcamp
- Reliable Data Engineering
- Data Expert - Resources by Zach Wilson
- Awesome Data Engineering - Resources by Igor Barinov
- Data Engineering resources by Amjń
- Data Engineering resources by Ahmed Alsaket
- Data Engineering articles by Saurav Singh
- Data Engineering articles by Mayurkumar Surani
- Data Engineering Q&A by Sachin Chandrashekhar
- Data Engineering on DataBricks by Jakub Lasak
- Data Engineering illustrations by Riya Khandelwal
- Books
- Curricula
- Specific topics
- DataBricks - Data Engineering
- Web sites
Created by gh-md-toc
This project aims at collecting in a single place training resources for the up skilling of data engineers.
Even though the members of the GitHub organization may be employed by some companies, they speak on their personal behalf and do not represent these companies.
- Data Engineering Helpers - Knowledge Sharing - Cheat sheets
- Material for the Data platform - Architecture principles
- Material for the Data platform - Data contracts
- Material for the Data platform - Data products
- Material for the Data platform - Data quality
- Material for the Data platform - Semantic layer
- Material for the Data platform - Data lakehouse
- Material for the Data platform - Data management
- Material for the Data platform - Modern Data Stack (MDS) in a box
- Material for the Data platform - Data life cycle
- Material for the Data platform - Metadata
- Link to the article on Medium
- Title: Mastering PySpark & SQL: A Complete Hands-On Guide for Data Engineers
- Link to the article on Medium
- Title: How Important It Has Become to Build a Pet Project or Product-Equivalent E2E Project on Your Own in the AI Era
- Link to the article
- Author: Viktor Gamov
- Date: April 2026
- Title: Building a Streaming Lakehouse with Open Source: Kafka to Iceberg to Trino to Superset
- GitHub - Flink Trino Superset pipeline
- Soda blog - Data Engineering fundamentals
- Date: May 2026
- Author: Fabiana Perraz
- Excerpts:
Key Takeaways:
- Data engineering is the discipline of building systems that move data reliably from source to use: ingestion, transformation, validation, storage, and consumption.
- The job has shifted from "moving data" to "ensuring data is correct, fresh, and usable" at every pipeline stage.
- Data engineering is evolving toward greater automation, observability, and governance.
- Most data failures are silent: pipelines run successfully but produce incorrect outputs. Engineers' job is to design systems that fail visibly.
Core skills of a Data Engineer:
- Strong SQL capability
- Python or a similar general-purpose language
- How data should be modeled
- Distributed systems thinking
- Cloud infrastructure knowledge
- Reason about data quality
- Communication
- Publisher: Substack - Luminousmen
- Date: Aug. 2025
- Link to the article on Substack
- Post on LinkedIn about the article
- Author: Kirill Bobrov (Kirill Bobrov on LinkedIn, Kirill Bobrov on Substack)
- Title: Data Engineering Design Patterns You Must Learn in 2026
- Date: Jan. 2026
- Article on Medium
- GitHub - Igor Barinov - Awesome Data Engineering
- Author: Igor Barinov
- Title: Data Engineering Resources
- Date: Aug. 2025
- Author: Ahmed Alsaket (Ahmed Alsaket on LinkedIn)
- Post on LinkedIn
- Scrimba - Master Python
- Scrimba - Learn SQL
- Scrimba - Learn MySQL
- Scrimba - Learn MongoDB
- Scrimba - Dominate PySpark
- Scrimba - Learn Bash, Airflow & Kafka
- Scrimba - Learn Git & GitHub
- Scrimba - Learn CICD basics
- Scrimba - Decode Data Warehousing
- Scrimba - Learn DBT
- Scrimba - Learn Data Lakes
- Scrimba - Learn DataBricks
- Scrimba - Learn Azure Databricks
- Scrimba - Learn Snowflake
- Scrimba - Learn Apache NiFi
- Scrimba - Learn Debezium
- Scrimba - Reddit ETL Pipeline
- Scrimba - Surfline Dashboard
- Scrimba - Finnhub Streaming Data Pipeline
- Scrimba - Audiophile End-To-End ELT Pipeline
- Scrimba - Streamify
- Medium - Mayur Surani's page
- Author: Mayurkumar Surani (Mayurkumar Surani on LinkedIn, Mayurkumar Surani on Medium)
- Publisher: Medium
- A few posts:
- Medium - Mayurkumar Surani - May 2025 - End-to-End ETL Pipeline with AWS, PySpark, and Databricks
- Medium - Mayurkumar Surani - May 2025 - Top 20 PySpark Functions Every Data Engineer Should Master
- Medium - Mayurkumar Surani - Aug. 2025 - Mastering Databricks and DBT: An End-to-End Production-Grade Data Engineering Project
- Medium - Mayurkumar Surani - Aug. 2025 - Bronze Layer - Dynamic Incremental Data Ingestion with Databricks Autoloader Part-02
- Author: Sachin Chandrashekhar (Sachin Chandrashekhar on LinkedIn)
- Date: Aug. 2025
- LinkedIn post
- Author:
- Date: Aug. 2025
- GitHub - Databricks apparel streaming
- LinkedIn post - Senior Databricks Data Engineer interview
- Date: Apr. 2026
- Substack - 7 Questions That Expose How You Debug Slow Pipelines
- Author: Riya Khandelwal (Riya Khandelwal on LinkedIn)
- Date: Aug. 2025
- GitHub - Data Engineering helpers - Skilling - Dedicated readme for material published by Riya Khandelwal
- Authors: Joe Reis and Matt Housley
- Date: July 2022
- LinkedIn media - Copy of the book
- Book on Amazon
- Book on O'Reilly
- Print length: 447 pages
- ISBN-10: 1098108302
- ISBN-13: 978-1098108304
- Author: Martin Kleppmann
- Book on Amazon
- Book on O'Reilly
- Date: Jan. 2026 (2nd version)
- Author: Andreas Kretz (Andreas Kretz on LinkedIn)
- LinkedIn service page - Data Engineering training program
- LinkedIn learning - Transition from data science to data engineering
- Level: beginners
- Duration: 47 minutes
- Author: Pooja Jain
- GitHub - Abel Tavares - Batch data pipeline
- Date: Nov. 2025
- Author: Abel Tavares
- Post on LinkedIn
- Overview:
- Abel built a batch data pipeline with observability, data quality,
and lineage tracking using open-source tools.
- Apache Airflow for orchestration
- DuckDB for fast analytical processing
- Delta Lake for ACID transactions on data lakes
- MinIO (S3-compatible) for storage
- Trino Software Foundation for distributed SQL queries
- Metabase for dashboards
- Soda core for data quality checks
- Marquez + OpenLineage for end-to-end lineage
- Prometheus Group + Grafana Labs for monitoring
- Abel built a batch data pipeline with observability, data quality,
and lineage tracking using open-source tools.
- The pipeline follows the Medallion Architecture (Bronze → Silver → Gold) with automated data cleaning, validation, and business aggregations.
- Everything runs with Docker, including automated dashboard generation in Metabase and Grafana.
- Author: Pooja Jain
- Linked post