TL;DR — ETL Orchestrator

Design an ETL orchestrator that schedules and manages data pipeline DAGs for thousands of pipelines processing petabytes daily. Key features: Define and schedule DAGs with complex dependencies. Multiple triggers: cron, data-availability, event-driven.

MEDIUM45 min

ETL Orchestrator

DAG schedulingdata lineagemonitoringbackfill

Key Points

Define and schedule DAGs with complex dependencies
Multiple triggers: cron, data-availability, event-driven
Failure handling: retry, skip, backfill, alerting

Key Constraints

Active DAGs

10K

Tasks/day

Max depth

50 levels

Hints (0/3)

Canvas

Build your design

Drag components from the palette to build your solution for "ETL Orchestrator"

ETL Orchestrator

▶3D Simulate

Intermediate45 min read+100 XP

TL;DR — ETL Orchestrator

MEDIUM45 min

ETL Orchestrator

DAG schedulingdata lineagemonitoringbackfill

Key Points

Define and schedule DAGs with complex dependencies
Multiple triggers: cron, data-availability, event-driven
Failure handling: retry, skip, backfill, alerting

Key Constraints

Active DAGs

10K

Tasks/day

Max depth

50 levels

Hints (0/3)

Canvas

Build your design

Drag components from the palette to build your solution for "ETL Orchestrator"