15. Apache Airflow Core: DAGs, Operators, Sensors, Task Execution Engine & Celery/Kubernetes Executors • Data Engineering, Big Data & ETL Infrastructure Architecture • Rishabh Ki Kaksha

Data Engineering, Big Data & ETL Infrastructure Architecture › 15. Apache Airflow Core: DAGs, Operators, Sensors, Task Execution Engine & Celery/Kubernetes Executors

50 mins

← Prev Next →

Course Syllabus

Module 1: Modern Data Engineering Architecture & The Data Stack

01. The Modern Data Stack: OLTP vs OLAP, ETL vs ELT & Batch vs Stream Processing Architecture 02. Dimensional Modeling: Kimball Methodology, Star Schema, Snowflake Schema & Fact vs Dimension Tables 03. Columnar Storage Formats: Apache Parquet, ORC, Snappy Compression & Dictionary Encoding Internals

Module 2: Distributed Processing with Apache Spark & PySpark

04. Apache Spark Core Architecture: Driver, Executors, RDDs, DAG Scheduler & Spark Catalyst Optimizer 05. High-Performance PySpark DataFrames: Transformations (Narrow vs Wide), Actions & Lazy Evaluation 06. Spark Optimization Masterclass: Partitioning, Coalesce vs Repartition, Broadcast Joins & Skew Data Handling 07. Spark Structured Streaming: Event-Time Processing, Watermarking & Windowed Aggregations

Module 3: Real-Time Event Streaming with Apache Kafka

08. Apache Kafka Architecture: Brokers, Topics, Partitions, Replicas, ISR (In-Sync Replicas) & KRaft Consensus 09. Kafka Producers & Consumers: Partitioning Keys, Consumer Groups, Rebalancing & Offset Commit Management 10. Schema Evolution & Governance: Confluent Schema Registry with Apache Avro and Protobuf Schemas

Module 4: Modern Data Lakehouse Architecture (Delta Lake, Apache Iceberg)

11. Data Lakehouse Architecture: Evolution from Data Lakes (S3/HDFS) to ACID Lakehouses 12. Delta Lake Architecture: Transaction Log (_delta_log), Time Travel, Schema Enforcement & ACID Guarantees 13. Apache Iceberg: Metadata Layer, Hidden Partitioning, Snapshot Isolation & Compaction Optimization 14. The Medallion Architecture: Bronze (Raw Ingestion) -> Silver (Enriched) -> Gold (Business Aggregates)

Module 5: Workflow Orchestration with Apache Airflow & Dagster

15. Apache Airflow Core: DAGs, Operators, Sensors, Task Execution Engine & Celery/Kubernetes Executors 16. Production Airflow Best Practices: Idempotent Tasks, Dynamic DAG Generation, Backfilling & SLA Alerts 17. Modern Data Orchestration with Dagster: Software-Defined Assets, Declarative Scheduling & Type Safety

Module 6: Cloud Data Warehousing (Snowflake / BigQuery) & dbt Transformations

18. Cloud Data Warehouses: Snowflake Multi-Cluster Shared Data Architecture vs Google BigQuery Dremel Engine 19. Data Transformations with dbt (data build tool): Modular SQL Models, Jinja Macros, Materializations & Testing 20. End-to-End Enterprise Data Pipeline Project: Kafka -> Spark -> Delta Lake -> dbt -> Snowflake Dashboard

Interactive Lesson Studio

# 15. Apache Airflow Core: DAGs, Operators, Sensors, Task Execution Engine & Celery/Kubernetes Executors

> Data Engineering & Big Data Notes by Rishabh - Topic 15

---

### 1. Airflow Core Architecture 💨

```
[ DAG Definitions (Python code in Git) ] ➔ [ Airflow Scheduler (Evaluates DAG Dependency Graph) ]
                                                   ⬇
                                        [ Metadata Database (Postgres) ]
                                                   ⬇
                                 [ Executor: Celery Worker Pool / Kubernetes Pods ]
```

---

### 2. Production Airflow 2.x TaskFlow API DAG

```python
from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(
    schedule="0 2 * * *", # Daily at 2:00 AM
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={"retries": 3, "retry_delay": timedelta(minutes=5)}
)
def enterprise_etl_pipeline():
    @task
    def extract_from_postgres() -> dict:
        return {"records_extracted": 50000}

@task
    def transform_spark(metadata: dict) -> str:
        return "s3://lakehouse/silver/users"

@task
    def load_to_snowflake(s3_path: str):
        print(f"🚀 Loaded {s3_path} to Snowflake!")

# Clean declarative dependency chaining:
    meta = extract_from_postgres()
    silver_path = transform_spark(meta)
    load_to_snowflake(silver_path)

etl_dag = enterprise_etl_pipeline()
```

---

### 3. Interview Ke 3 Sawal Is Topic Se

1. **CeleryExecutor vs KubernetesExecutor me kya difference hai?** -> CeleryExecutor pre-warmed worker processes maintain karta hai (Low task latency); KubernetesExecutor har task ke liye isolated on-demand pod spawn karta hai (Dynamic scaling & resource isolation).
2. **Airflow Sensor me `mode="reschedule"` vs `mode="poke"` me kya difference hai?** -> `poke` worker slot occupy karke sleep karta hai (wastes worker concurrency); `reschedule` worker slot release kar deta hai aur check interval par dobara wake up hota hai.
3. **XCom (Cross-Communication) Airflow me large datasets transfer ke liye kyu nahi use karna chahiye?** -> Kyunki XCom metadata database me store hota hai; large data transfer ke liye intermediate S3 path URI share karna best practice hai.

---
**Next Topic:** 16. Production Airflow Best Practices: Idempotent Tasks, Dynamic DAG Generation, Backfilling & SLA Alerts

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track