Skip to content

About

A collection of PySpark projects showcasing scalable data processing, transformation pipelines, and big data analytics using Apache Spark.

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Latest commit

 

History

26 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PySpark Pipeline

Scale. Transform. Analyse Big Data.

A curated collection of end-to-end PySpark projects designed to handle large-scale data processing. This repository demonstrates how to build scalable data pipelines for cleaning, transforming, and analysing massive datasets using Apache Spark. It includes practical implementations of Spark SQL, DataFrames, structured streaming, and performance tuning - ideal for working with real-world big data workflows.


1. Getting Started with PySpark

1.1 SparkSession, RDDs, and DataFrames

✦ Creating your First RDD with SparkSession
✦ Word Counter Application
✦ DataFrame Basics
✦ Processing Sports Items Data with RDDs, DataFrames, and Spark SQL
✦ Age Group Segmentation of Air Passengers Using Spark RDDs
✦ Survey Answer Frequency Analysis via Spark RDDs

1.2 DataFrame Operations (Select, Filter, Join, GroupBy)

✦ DataFrame Operations


2. Working with Data

2.1 Reading & Writing Data (CSV, JSON, Parquet, Hive)

✦ Create and Work with an RDD created from a text file
✦ Read Modes in PySpark

2.2 Schema Inference & Manual Schema Definition

✦ Schema Inference And Definition

2.3 Handling Missing or Corrupted Data


3. Transformations & Actions

3.1 Lazy Evaluation and Execution Plans

✦ Filter Log Lines Containing INFO Using RDD

3.2 Narrow vs Wide Transformations

✦ GroupByKey Vs ReduceByKey

3.3 Chaining and Optimization Techniques


4. PySpark SQL

4.1 SQLContext and SparkSQL Queries

4.2 Temporary Views and Tables

✦ Basics of Views in PySpark

4.3 Using SQL and DataFrame APIs Together


5. Performance Tuning

5.1 Partitioning & Caching Strategies

5.2 Broadcast Joins & Skew Handling

5.3 Catalyst Optimizer and Tungsten Engine


6. Machine Learning with PySpark (MLlib)

6.1 ML Pipelines: Transformers & Estimators

6.2 Feature Engineering (VectorAssembler, StringIndexer)

✦ Data Cleaning and Transformation Pipeline

6.3 Regression, Classification & Clustering Models

✦ Full Workflow for Regression or Classification in PySpark MLlib

6.3.1 Regression

✦ Mumbai House Price Regression Using PySpark ML Components
✦ Bengaluru House Price Prediction

6.3.2 Classification


About

A collection of PySpark projects showcasing scalable data processing, transformation pipelines, and big data analytics using Apache Spark.

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Used by

Contributors

Languages