A curated collection of end-to-end PySpark projects designed to handle large-scale data processing. This repository demonstrates how to build scalable data pipelines for cleaning, transforming, and analysing massive datasets using Apache Spark. It includes practical implementations of Spark SQL, DataFrames, structured streaming, and performance tuning - ideal for working with real-world big data workflows.
✦ Creating your First RDD with SparkSession
✦ Word Counter Application
✦ DataFrame Basics
✦ Processing Sports Items Data with RDDs, DataFrames, and Spark SQL
✦ Age Group Segmentation of Air Passengers Using Spark RDDs
✦ Survey Answer Frequency Analysis via Spark RDDs
✦ Create and Work with an RDD created from a text file
✦ Read Modes in PySpark
✦ Schema Inference And Definition
✦ Filter Log Lines Containing INFO Using RDD
✦ Data Cleaning and Transformation Pipeline
✦ Full Workflow for Regression or Classification in PySpark MLlib
✦ Mumbai House Price Regression Using PySpark ML Components
✦ Bengaluru House Price Prediction