Master Cloud Data Engineering with AWS through hands-on, self-paced learning. Build production-ready skills in data lakehouses, ETL pipelines, streaming analytics, and cloud-native architectures.
This section gives you a quick orientation on how the training is structured, what to study first, and where to find each key document.
You will build a full Cloud Data Engineering stack with practical focus:
- Data lakes and lakehouse design patterns
- Batch and streaming ETL/ELT pipelines
- Pipeline orchestration and automation
- Data quality, governance, and security
- Cloud-native infrastructure (IaC, containers, serverless)
- Realistic local operation with LocalStack and Docker
PHASE 1: Foundation (Modules 01-04)
├─ Cloud fundamentals, storage, SQL, Python for data
└─ Milestone: strong cloud and data foundations
PHASE 2: Core Data Engineering (Modules 05-10 + Checkpoint 01)
├─ Lakehouse, ETL, batch, streaming, data quality, orchestration
└─ Checkpoint 01: Serverless Data Lake
PHASE 3: Cloud-Native Platform (Modules 11-14 + Checkpoint 02)
├─ IaC, serverless processing, containers, governance
└─ Checkpoint 02: Real-time Analytics Platform
PHASE 4: Advanced Tracks (Modules 15-18 + Checkpoint 03)
├─ Track A: Real-time analytics
├─ Track B: Security and compliance
├─ Track C: Cost optimization
└─ Checkpoint 03: Enterprise Data Lakehouse
PHASE 5: Bonus (Modules 22-23)
└─ Databricks and Snowflake (optional)
nan-data-engineering-labs/
├── modules/ -> modules and checkpoints
├── docs/ -> technical guides and troubleshooting
├── scripts/ -> setup, validation, utilities
├── shared/ -> shared resources
├── LEARNING-PATH.md -> complete dependency-based learning route
├── GETTING_STARTED.md -> onboarding guide
├── STATUS.md -> project progress
├── Makefile -> daily operating commands
└── docker-compose.yml -> local services stack
| Module | Focus |
|---|---|
| 01 | Cloud Fundamentals (AWS basics, IAM) |
| 02 | Storage Basics (S3, data formats) |
| 03 | SQL Foundations |
| 04 | Python for Data |
| 05 | Data Lakehouse Architecture |
| 06 | ETL Fundamentals |
| Checkpoint 01 | Serverless Data Lake |
| 07 | Batch Processing |
| 08 | Streaming Basics |
| 09 | Data Quality |
| 10 | Workflow Orchestration |
| 11 | Infrastructure as Code |
| 12 | Serverless Processing |
| Checkpoint 02 | Real-time Analytics Platform |
| 13 | Container Orchestration |
| 14 | Data Catalog and Governance |
| 15 | Real-time Analytics (Track A) |
| 16 | Data Security and Compliance (Track B) |
| 17 | Cost Optimization (Track C) |
| 18 | Advanced Architectures |
| Checkpoint 03 | Enterprise Data Lakehouse |
| Bonus 01 | Databricks Lakehouse |
| Bonus 02 | Snowflake Data Cloud |
Option A -- DevContainer (recommended)
git clone git@github.com:nanlabs/nan-data-engineering-labs.git
cd nan-data-engineering-labs
code .
# Command Palette -> "Dev Containers: Reopen in Container"Option B -- Local
git clone git@github.com:nanlabs/nan-data-engineering-labs.git
cd nan-data-engineering-labs
bash scripts/setup-environment.sh
# Start local services (LocalStack, Kafka, Spark, PostgreSQL, Trino, MinIO)
make up
# Check progress
make progress
# Start Module 01
cd modules/module-01-cloud-fundamentals
cat README.md| Document | Description |
|---|---|
| GETTING_STARTED.md | Onboarding and setup |
| LEARNING-PATH.md | Complete learning path with dependencies |
| STATUS.md | Project progress |
| docs/CHARTER.md | NaNLABS Lab Charter v1 |
| docs/setup-guide.md | Detailed setup instructions |
| docs/localstack-guide.md | Working with LocalStack |
| docs/localstack-alternatives.md | AWS service to local tool mapping |
| docs/troubleshooting.md | Common issues and solutions |
| Category | Technologies and Concepts |
|---|---|
| Data Storage | S3, Data Lakes, Delta Lake, Parquet, Avro, Data Lakehouses |
| Data Processing | Spark, PySpark, Batch Processing, Stream Processing, Kafka |
| Data Pipelines | ETL/ELT, Orchestration, Airflow, Step Functions |
| Data Quality | Great Expectations, Schema Validation, Data Contracts |
| Infrastructure | Terraform, CloudFormation, Docker, Kubernetes |
| Serverless | Lambda, Glue, Kinesis, DynamoDB |
| Analytics | Athena, Trino, SQL Optimization, Real-time Dashboards |
| Governance | Data Catalogs, Lineage, Security, Compliance |
| Architecture | Medallion, Data Mesh, Event-Driven, Cost Optimization |
┌─────────────────────────────────────────────────────────┐
│ Docker Compose │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LocalStack │ │ Kafka │ │ Spark │ │
│ │ AWS Services │ │ Streaming │ │ Processing │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ PostgreSQL │ │ Trino │ │ MinIO │ │
│ │ Data Warehouse│ │ Query Eng. │ │ S3 Storage │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────┘
No prerequisites -- complete in any order:
- Module 01: Cloud Fundamentals (AWS basics, IAM)
- Module 02: Storage Basics (S3, data formats)
- Module 03: SQL Foundations (analytical SQL)
- Module 04: Python for Data (Pandas, data manipulation)
- Module 05: Data Lakehouse Architecture (requires 02)
- Module 06: ETL Fundamentals (requires 02, 04)
- Checkpoint 01: Serverless Data Lake (requires 01-06)
- Module 07: Batch Processing (requires 02, 04, 05)
- Module 08: Streaming Basics (requires 04, 06)
- Module 09: Data Quality (requires 04, 06)
- Module 10: Workflow Orchestration (requires 06)
- Module 11: Infrastructure as Code (requires 01, 02)
- Module 12: Serverless Processing (requires 06, 11)
- Checkpoint 02: Real-time Analytics Platform (requires 07-12)
- Module 13: Container Orchestration (requires 11)
- Module 14: Data Catalog and Governance (requires 05, 09)
- Module 15: Real-time Analytics (requires 08, 10) -- Track A
- Module 16: Data Security and Compliance (requires 01, 14) -- Track B
- Module 17: Cost Optimization (requires 11) -- Track C
- Module 18: Advanced Architectures (requires 05, 07, 08, 14)
- Checkpoint 03: Enterprise Data Lakehouse (requires 13-18)
- Bonus 01: Databricks Lakehouse (requires 05, 07)
- Bonus 02: Snowflake Data Cloud (requires 03, 06)
See LEARNING-PATH.md for dependency diagram.
make progressmake up # Start all services
make down # Stop all services
make restart # Restart all services
make logs # View service logs
make clean # Stop and remove all datamake progress # Show learning progress
make validate MODULE=<name> # Validate specific modulemake setup # Run initial setup
python scripts/validate_learning_labs.py --strict-core --strict-headings- Docker and Docker Compose -- For LocalStack and services
- Python 3.9+ -- For scripts and data processing
- Git -- For version control
- 10 GB Disk Space -- For Docker images and datasets
- 8 GB RAM -- Minimum for running services
- AWS CLI -- For LocalStack interaction
- Terraform -- For IaC modules
- VS Code -- With Python and Docker extensions
- Basic programming (any language -- we teach Python)
- Basic SQL (SELECT, WHERE, JOIN -- we teach advanced)
- Command line usage (bash/terminal)
- No AWS account required -- everything runs locally
Everything runs locally using free, open-source tools:
| Service | Local Alternative | Cost |
|---|---|---|
| AWS S3, Lambda, DynamoDB, etc. | LocalStack Community | Free |
| Athena | Trino | Free |
| Kinesis/MSK | Apache Kafka | Free |
| EMR/Glue | Spark Standalone | Free |
| Redshift | PostgreSQL | Free |
Contributions welcome:
- Bug reports in exercises or validation
- Documentation improvements
- New exercise ideas
See CONTRIBUTING.md.
| Lab | Focus |
|---|---|
| nan-python-engineering-labs | Python engineering fundamentals |
| nan-ai-engineering-labs | AI/ML engineering |
| nan-ai-native-engineering-labs | AI-native workflows and tooling |
MIT License -- Free to use for learning and teaching. See LICENSE.
Progress table will be auto-generated here by scripts/progress.py
Made with contributors-img.