# Technical Documentation: DataTalksClub/data-engineering-zoomcamp

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/DataTalksClub/data-engineering-zoomcamp)

### Overview (README.md)

<p align="center">
  <img width="100%" src="/images/architecture/arch_v5_workshops.png" alt="Data Engineering Zoomcamp Overview">
</p>

<h1 align="center">
    Data Engineering Zoomcamp: A Free 9-Week Course on Data Engineering Fundamentals
</h1>

<p align="center">
Master the fundamentals of data engineering by building an end-to-end data pipeline from scratch. Gain hands-on experience with industry-standard tools and best practices.
</p>

<p align="center">
<a href="https://courses.datatalks.club/register/de-zoomcamp/"><img src="https://user-images.githubusercontent.com/875246/185755203-17945fd1-6b64-46f2-8377-1011dcb1a444.png" height="50" /></a>
</p>

<p align="center">
<a href="https://datatalks.club/slack.html">Join Slack</a> •
<a href="https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG">#course-data-engineering Channel</a> •
<a href="https://t.me/dezoomcamp">Telegram Announcements</a> •
<a href="https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb">Course Playlist</a> •
<a href="https://datatalks.club/faq/data-engineering-zoomcamp.html">FAQ</a>
</p>

<p align="center">
<a href="https://github.com/DataTalksClub/data-engineering-zoomcamp/pulls"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=for-the-badge" /></a>
<a href="https://datatalks.club/slack.html"><img src="https://img.shields.io/badge/Slack-Join%20Community-4A154B?style=for-the-badge&logo=slack" /></a>
</p>

## Quick Links

| Resource | Link |
|----------|------|
| Course materials | [GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp) |
| Video lectures | [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) |
| Documentation | [Zoomcamp Logistics](https://datatalks.club/docs/courses/zoomcamp-logistics/) · [Data Engineering Zoomcamp](https://datatalks.club/docs/courses/data-engineering-zoomcamp/) |
| Course platform (deadlines, homework) | [courses.datatalks.club](https://courses.datatalks.club/) |
| Slack channel | [#course-data-engineering](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) |
| Announcements | [Telegram](https://t.me/dezoomcamp) |
| FAQ | [FAQ document](https://datatalks.club/faq/data-engineering-zoomcamp.html) |

## About the Course

This free 9-week course teaches the fundamentals of data engineering by building an end-to-end data pipeline from scratch. It consists of structured modules, hands-on workshops, and a final project, giving you practical experience with industry-standard tools and best practices.

## Who Should Join

This course is for developers, analysts, and data scientists who want to learn how to build data pipelines and work with the modern data engineering stack. No prior data engineering experience is necessary.

## Prerequisites

To get the most out of this course, you should have:

- Basic coding experience
- Familiarity with SQL
- Experience with Python (helpful but not required)

No prior data engineering experience is necessary.

## How to Take the Course

There are two ways to follow the course: live and self-paced.

| | Live Cohort | Self-Paced |
|-|-|-|
| Start | January 2027 | Anytime |
| Lectures | Pre-recorded | Pre-recorded |
| Homework | Graded | Available but not scored |
| Leaderboard | ✅ Yes | ❌ No |
| Peer Review | ✅ Yes | ❌ No |
| Certificate | ✅ Yes | ❌ No |
| Cost | Free | Free |
| Register | [Sign up here](https://courses.datatalks.club/register/de-zoomcamp/) | Just start learning! |

> [!IMPORTANT]
> "Live cohort" does not mean live classes. All lectures are pre-recorded. "Live" means working alongside others with deadlines, scored homework, a leaderboard, peer review, and a certificate at the end.

Self-paced steps:

1. Follow the materials on [GitHub](https://github.com/DataTalksClub/data-engineering-zoomcamp)
2. Ask questions and share progress in [Slack](https://datatalks.club/slack.html)
3. Do the homework (self-checked) and build a project for your portfolio

## Syllabus

### [Module 1: Containerization and Infrastructure as Code](01-docker-terraform/)

- Introduction to GCP
- Docker and Docker Compose
- Running PostgreSQL with Docker
- Infrastructure setup with Terraform
- Homework

### [Module 2: Workflow Orchestration](02-workflow-orchestration/)

- Data Lakes and Workflow Orchestration
- Workflow orchestration with Kestra
- Homework

### [Workshop 1: Data Ingestion](cohorts/2026/workshops/dlt.md)

- API reading and pipeline scalability
- Data normalization and incremental loading
- Homework

### [Module 3: Data Warehousing](03-data-warehouse/)

- Introduction to BigQuery
- Partitioning, clustering, and best practices
- Machine learning in BigQuery

### [Module 4: Analytics Engineering](04-analytics-engineering/)

- Analytics Engineering and Data Modeling
- dbt (data build tool) with DuckDB & BigQuery
- Testing, documentation, and deployment

### [Module 5: Data Platforms](05-data-platforms/)

- Building end-to-end data pipelines with Bruin
- Data ingestion, transformation, and quality
- Deployment to cloud (BigQuery)

### [Module 6: Batch Processing](06-batch/)

- Introduction to Apache Spark
- DataFrames and SQL
- Internals of GroupBy and Joins

### [Module 7: Streaming](07-streaming/)

- Introduction to Kafka
- Kafka Streams and KSQL
- Schema management with Avro

## Final Project

The [final project](projects/) applies all the concepts learned in a real-world scenario, including a peer review and feedback process.

## Certificate

<p align="center">
<img src="images/de-zoomcamp-certificate.jpg" alt="Data Engineering Zoomcamp certificate of completion awarded after finishing the final project and peer reviews" title="Data Engineering Zoomcamp Certificate of Completion" width="500" />
</p>

Certificates are awarded to learners who complete the final project during a live cohort. See [Certification](https://datatalks.club/docs/courses/zoomcamp-logistics/certification/) for how certification works and how to get your certificate.

## Instructors

- [Alexey Grigorev](https://linkedin.com/in/agrigorev)
- [Michael Shoemaker](https://www.linkedin.com/in/michaelshoemaker1/)
- [Will Russell](https://www.linkedin.com/in/wrussell1999/)
- [Anna Geller](https://www.linkedin.com/in/anna-geller-12a86811a/)
- [Juan Manuel Perafan](https://www.linkedin.com/in/jmperafan/)
- [Arsalan Noorafkan](https://www.linkedin.com/in/arsalan0/)

Past instructors:

- [Victoria Perez Mola](https://www.linkedin.com/in/victoriaperezmola/)
- [Ankush Khanna](https://linkedin.com/in/ankushkhanna2)
- [Sejal Vaidya](https://www.linkedin.com/in/vaidyasejal/)
- [Irem Erturk](https://www.linkedin.com/in/iremerturk/)
- [Luis Oliveira](https://www.linkedin.com/in/lgsoliveira/)
- [Zach Wilson](https://www.linkedin.com/in/eczachly)

## Testimonials

> Thank you for what you do! The Data Engineering Zoomcamp gave me skills that helped me land my first tech job.
>
> — [Tim Claytor](https://www.linkedin.com/in/claytor/) ([Source](https://www.linkedin.com/feed/update/urn:li:activity:7396882073308938240?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7396882073308938240%2C7396889959711793152%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287396889959711793152%2Curn%3Ali%3Aactivity%3A7396882073308938240%29))

> Three months might seem like a long time, but the growth and learning during this period are truly remarkable. It was a great experience with a lot of learning, connecting with like-minded people from all around the world, and having fun. I must admit, this was really hard. But the feeling of accomplishment and learning made it all worthwhile. And I would do it again!
>
> — [Nevenka Lukic](https://www.linkedin.com/in/nevenka-lukic/) ([Source](https://www.linkedin.com/posts/nevenka-lukic_data-engineering-zoomcamp-final-project-activity-7181985646033461248-Lc1O?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4))

> One of the significant things I inferred from the Zoomcamp is to prioritize fundamentals and principles over ever-evolving tools and tech stacks. Hugely grateful to Alexey Grigorev for putting together this incredible course and offering it for free.
>
> — [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi/) ([Source](https://www.linkedin.com/posts/activity-7325692407675604992-XSKI?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4))

> Such a fun deep dive into data engineering, cloud automation, and orchestration. I learned so much along the way. Big shoutout to Alexey Grigorev and the DataTalksClub team for the opportunity and guidance throughout the 3 months of the free course.
>
> — [Assitan NIARE](https://www.linkedin.com/in/assitan-niar%C3%A9-data/) ([Source](https://www.linkedin.com/posts/activity-7317441554023874561-E3wm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4))

> If you're serious about breaking into data engineering, start here. The repo's structure, community, and hands-on focus make it unparalleled.
>
> — [Wady Osama](https://www.linkedin.com/in/wadyosama/) ([Source](https://www.linkedin.com/posts/wadyosama_dataengineering-zoomcamp-dezoomcamp-activity-7292126824711520258-puJm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4))

## Community & Support

### Getting Help on Slack

Join the [#course-data-engineering](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel on [DataTalks.Club Slack](https://datatalks.club/slack.html) for discussions, troubleshooting, and networking.

To keep discussions organized:

- Follow [our guidelines](https://datatalks.club/docs/courses/zoomcamp-logistics/asking-questions/) when posting questions.
- Review the [community guidelines](https://datatalks.club/slack/guidelines.html).

### Learning in Public

Share your progress as you go — see the [learning in public guide](https://datatalks.club/docs/courses/zoomcamp-logistics/learning-in-public/).

## Sponsors

A special thanks to our course sponsors for making this initiative possible!

<p align="center">
  <a href="https://kestra.io/">
    <img height="120" src="images/kestra.svg">
  </a>
</p>

<p align="center">
  <a href="https://getbruin.com/">
    <img height="110" src="images/bruin.svg">
  </a>
</p>

<p align="center">
  <a href="https://dlthub.com/">
    <img height="90" src="images/dlthub.png">
  </a>
</p>

Interested in supporting our community? Reach out to [alexey@datatalks.club](mailto:alexey@datatalks.club).

## FAQ

A few common questions. For everything else, see the full [Data Engineering Zoomcamp FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html).

Q: Is this course really free?<br/>
A: Yes. All videos, materials, and homework are free and open-source.

Q: Do I need prior data engineering experience?<br/>
A: No. You just need basic coding experience and some familiarity with SQL. Python helps but isn't required.

Q: What does "live cohort" mean? Are there live classes?<br/>
A: No mandatory live classes. All lectures are pre-recorded. "Live" means deadlines, scored homework, a leaderboard, peer review, and certificate eligibility.

Q: Can I take it self-paced, and will I get a certificate?<br/>
A: Yes, you can start anytime. Certificates require completing the final project and peer reviews during a live cohort.

## About DataTalks.Club

<p align="center">
  <img width="40%" src="https://github.com/user-attachments/assets/1243a44a-84c8-458d-9439-aaf6f3a32d89" alt="DataTalks.Club">
</p>

<p align="center">
<a href="https://datatalks.club/">DataTalks.Club</a> is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
</p>

<p align="center">
<a href="https://datatalks.club/">Website</a> •
<a href="https://datatalks.club/slack.html">Join Slack Community</a> •
<a href="https://us19.campaign-archive.com/home/?u=0d7822ab98152f5afc118c176&id=97178021aa">Newsletter</a> •
<a href="http://lu.ma/dtc-events">Upcoming Events</a> •
<a href="https://www.youtube.com/@DataTalksClub/featured">YouTube</a> •
<a href="https://github.com/DataTalksClub">GitHub</a> •
<a href="https://www.linkedin.com/company/datatalks-club/">LinkedIn</a> •
<a href="https://x.com/DataTalksClub">X</a>
</p>

All the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more.

At DataTalks.Club, we organize online events, community activities, and free courses. You can learn more about what we do at [DataTalks.Club docs](https://datatalks.club/docs/general/).

---

### Cohorts 2022 Week 2 Data Ingestion Airflow Docs 1 Concepts (`cohorts/2022/week_2_data_ingestion/airflow/docs/1_concepts.md`)

## Airflow concepts


### Airflow architecture


Ref: https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html

* **Web server**:
GUI to inspect, trigger and debug the behaviour of DAGs and tasks. 
Available at http://localhost:8080.

* **Scheduler**:
Responsible for scheduling jobs. Handles both triggering & scheduled workflows, submits Tasks to the executor to run, monitors all tasks and DAGs, and
then triggers the task instances once their dependencies are complete.

* **Worker**:
This component executes the tasks given by the scheduler.

* **Metadata database (postgres)**:
Backend to the Airflow environment. Used by the scheduler, executor and webserver to store state.

* **Other components** (seen in docker-compose services):
    * `redis`: Message broker that forwards messages from scheduler to worker.
    * `flower`: The flower app for monitoring the environment. It is available at http://localhost:5555.
    * `airflow-init`: initialization service (customized as per this design)

All these services allow you to run Airflow with CeleryExecutor. 
For more information, see [Architecture Overview](https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html).


### Project Structure:

* `./dags` - `DAG_FOLDER` for DAG files (use `./dags_local` for the local ingestion DAG)
* `./logs` - contains logs from task execution and scheduler.
* `./plugins` - for custom plugins


### Workflow components

* `DAG`: Directed acyclic graph, specifies the dependencies between a set of tasks with explicit execution order, and has a beginning as well as an end. (Hence, “acyclic”)
    * `DAG Structure`: DAG Definition, Tasks (eg. Operators), Task Dependencies (control flow: `>>` or `<<` )
    
* `Task`: a defined unit of work (aka, operators in Airflow). The Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
    * Common Types: Operators (used in this workshop), Sensors, TaskFlow decorators
    * Sub-classes of Airflow's BaseOperator

* `DAG Run`: individual execution/run of a DAG
    * scheduled or triggered

* `Task Instance`: an individual run of a single task. Task instances also have an indicative state, which could be “running”, “success”, “failed”, “skipped”, “up for retry”, etc.
    * Ideally, a task should flow from `none`, to `scheduled`, to `queued`, to `running`, and finally to `success`.


### References

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html

https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html

---

--- METRICS ---
- Files Extracted: 2
- Estimated Token Budget: ~3858 tokens
- Canonical Reference: https://codewiki.google/github.com/DataTalksClub/data-engineering-zoomcamp