Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.

The simplest way to scale Python.

Data pipelines from re-usable components

The open-source Useful SDK. One python decorator in the Useful library allows for full observability of Python functions within an ETL.

Data Cleaning for Pyspark

A project structure for doing and sharing data engineer work.

Lien de l'application

Master the AWS Data Stack! 🚀 This repository features 15+ Industrial Data Engineering Projects covering Serverless ETL, Real-Time Streaming, & Data Warehousing. Hands-on labs for S3, Lambda, Spark, Airflow, Snowflake, Redshift, Kinesis, & Glue. Includes production-grade CICD pipelines. A complete roadmap to becoming a top Data Professional.

End To End MLOPS Project With ETL Pipelines- Building Network Security System

Lightweight, production-hardened Python utilities for data engineering: resilient HTTP, PostgreSQL pooling, and structured JSON logging with automatic secret redaction.

Python ETL pipeline normalizing Ahmedabad transit APIs (BRTS, AMTS, Metro) into unified JSON/GTFS datasets for GraphHopper and OpenTripPlanner

DataSift auto applies a data pre-processing pipeline to Data Science Projects.

Big Data ETL pipeline for Brazilian e-commerce data. Implements data ingestion, transformation, and storage using Apache Spark, Hadoop, and SQL. Designed for scalable data processing and analytics.

This project demonstrates a comprehensive data warehousing and analytics solution, from building a data warehouse to generating actionable insights. Designed as a portfolio project, it highlights industry best practices in data engineering and analytics.

Complete portfolio of data engineering projects from Udacity's Data Engineering with AWS Nanodegree.

A robust ETL pipeline using MySQL for data transformation and Python for programmatic visualization of the 2018 NYC Squirrel Census (3,000+ records).

🚀 A comprehensive showcase of projects and skills from the IBM Data Engineering Professional Certificate! 📚 Features include: 🔄 ETL pipelines, 🗄️ data warehousing, ⚡ big data processing with Spark/Hadoop, 🛠️ database administration, and 📈 business intelligence dashboards. Built with 🦾 to demonstrate real-world data engineering capabilities!

End-to-end Azure Data Factory project transforming raw sales data into customer-level insights using pivot transformation and storing results in Blob Storage.

🗄️ IBM Relational Database Administrator with GenAI Certificate Portfolio – A comprehensive collection of projects, labs, and assignments showcasing expertise in relational database administration, 🏘️data warehousing, 🔁ETL pipelines, and 🤖Generative AI integration for modern database management.

Modern Data Warehouse and Analytics Project implementing Medallion Architecture (Bronze, Silver, Gold) with ETL pipelines, SQL data modeling, and analytical reporting.