data-engineer-handbook Documentation

This is a repo with links to everything you'd ever want to learn about data engineering

43,003 stars Jupyter Notebook 1 file · ~6,982 tokens #apachespark#awesome#bigdata#data#dataengineering#sql
RAW Doc

The Data Engineering Handbook

This repo has all the resources you need to become an amazing data engineer!

Join the free Databricks AI boot camp on August 3rd [here](https://learn.dataexpert.io/program/the-one-week-beginners-databricks-boot-camp-7129) and signup for Databricks Free Edition [here](https://signup.databricks.com/?provider=DB_FREE_TIERutm_source=github&utm_medium=video&utm_campaign=DataExpert)

Getting started

If you are new to data engineering, start by following this 2024 breaking into data engineering roadmap

If you are here for the 4-week free beginner boot camp you can check out:

If you are here for the 6-week free intermediate boot camp you can check out

For more applied learning:

  • Check out the projects section for more hands-on examples!
  • Check out the interviews section for more advice on how to pass data engineering interviews!
  • Check out the books section for a list of high quality data engineering books
  • Check out the communities section for a list of high quality data engineering communities to join
  • Check out the newsletter section to learn via email

Resources

Great [list of over 25 books](books.md)

Top 3 must read books are:

Great [list of over 10 communities to join](communities.md):

Top must-join communities for DE:

Top must-join communities for ML:

Companies:

Data Engineering blogs of companies:

Data Engineering Whitepapers:

Social Media Accounts

Here's the mostly comprehensive list of data engineering creators:
(You have to have at least 5k followers somewhere to be added!)

YouTube

Name YouTube Channel Follower Count
ByteByteGo ByteByteGo 1,000,000+
Data with Baraa Data with Baraa 195,000+
Zach Wilson Data with Zach 150,000+
Shashank Mishra E-learning Bridge 100,000+
Seattle Data Guy Seattle Data Guy 100,000+
TrendyTech TrendyTech 100,000+
Darshil Parmar Darshil Parmar 100,000+
Andreas Kretz Andreas Kretz 100,000+
The Ravit Show The Ravit Show 100,000+
Guy in a Cube Guy in a Cube 100,000+
Adam Marczak Adam Marczak 100,000+
nullQueries nullQueries 100,000+
TECHTFQ by Thoufiq TECHTFQ by Thoufiq 100,000+
SQLBI SQLBI 100,000+
Alex Freberg Alex The Analyst 100,000+
Ankur Ranjan Big Data Show 100,000+
Prashanth Kumar Pandey ScholarNest 77,000+
ITVersity ITVersity 67,000+
Soumil Shah Soumil Shah 50,000
Ansh Lamba Ansh Lamba 18,000+
Azure Lib Azure Lib 10,000+
Advancing Analytics Advancing Analytics 10,000+
Kahan Data Solutions Kahan Data Solutions 10,000+
Ankit Bansal Ankit Bansal 10,000+
Mr. K Talks Tech Mr. K Talks Tech 10,000+
Samuel Focht Python Basics 10,000+
Mehdi Ouazza Mehdio DataTV 3,000+
Alex Merced Alex Merced Data N/A
John Kutay John Kutay N/A
Emil Kaminski Databricks For Professionals 5,000+

LinkedIn

Name LinkedIn Profile Follower Count
Zach Wilson Zach Wilson 400,000+
Chip Huyen Chip Huyen 250,000+
Shashank Mishra Shashank Mishra 100,000+
Seattle Data Guy Ben Rogojan 100,000+
TrendyTech Sumit Mittal 100,000+
Darshil Parmar Darshil Parmar 100,000+
Andreas Kretz Andreas Kretz 100,000+
ByteByteGo (Alex Xu) Alex Xu 100,000+
Azure Lib (Deepak Goyal) Deepak Goyal 100,000+
Alex Freberg Alex Freberg 100,000+
SQLBI (Marco Russo) Marco Russo 50,000+
Ankit Bansal Ankit Bansal 50,000+
Marc Lamberti Marc Lamberti 50,000+
Ankur Ranjan Ankur Ranjan 48,000+
ITVersity (Durga Gadiraju) Durga Gadiraju 48,000+
Prashanth Kumar Pandey Prashanth Kumar Pandey 37,000+
Alex Merced Alex Merced 30,000+
Ijaz Ali Ijaz Ali 24,000+
Mehdi Ouazza Mehdi Ouazza 20,000+
Ananth Packkildurai Ananth Packkildurai 18,000+
Ansh Lamba Ansh Lamba 13,000+
Manojkumar Vadivel Manojkumar Vadivel 12,000+
Advancing Analytics Simon Whiteley 10,000+
Li Yin Li Yin 10,000+
Jaco van Gelder Jaco van Gelder 10,000+
Joseph Machado Joseph Machado 10,000+
Eric Roby Eric Roby 10,000+
Simon Späti Simon Späti 10,000+
Constantin Lungu Constantin Lungu 10,000+
Lakshmi Sontenam Lakshmi Sontenam 9,500+
Dani Pálma Daniel Pálma 9,000+
Soumil Shah Soumil Shah 8,000+
Arnaud Milleker Arnaud Milleker 7,000+
Dimitri Visnadi Dimitri Visnadi 7,000+
Lenny Lenny A 6,000+
Dipankar Mazumdar Dipankar Mazumdar 5,000+
Daniel Ciocirlan Daniel Ciocirlan 5,000+
Hugo Lu Hugo Lu 5,000+
Tobias Macey Tobias Macey 5,000+
Marcos Ortiz Marcos Ortiz 5,000+
Julien Hurault Julien Hurault 5,000+
John Kutay John Kutay 5,000+
Hassaan Akbar Hassaan Akbar 5,000+
Subhankar Subhankar 5,000+
Nitin Nitin N/A
Hassaan Hassaan 5000+
Javier de la Torre Javier 5000+

X/Twitter

Name X/Twitter Profile Follower Count
ByteByteGo alexxubyte 100,000+
Dan Kornas @dankornas 66,000+
Zach Wilson EcZachly 30,000+
Seattle Data Guy SeattleDataGuy 10,000+
SQLBI marcorus 10,000+
Joseph Machado startdataeng 5,000+
Alex Merced @amdatalakehouse N/A
John Kutay @JohnKutay N/A
Mehdi Ouazza mehd_io N/A

Instagram

Name Instagram Profile Follower Count
Sundas Khalid sundaskhalidd 300,000+
Zach Wilson eczachly 150,000+
Andreas Kretz learndataengineering 5,000+
Alex Merced @alexmercedcoder N/A

TikTok

Name TikTok Profile Follower Count
Zach Wilson @eczachly 70,000+
Alex Freberg @alex_the_analyst 10,000+
Mehdi Ouazza @mehdio_datatv N/A

Great Podcasts

Great [list of 20+ newsletters](newsletters.md)

Top must follow newsletters for data engineering:

Glossaries:

Design Patterns

Courses / Academies

Certifications Courses

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#apachespark Browse apachespark catalog → #awesome Browse awesome catalog → #bigdata Browse bigdata catalog → #data Browse data catalog → #dataengineering Browse dataengineering catalog → #sql Browse sql catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →