Apache Spark Standalone Cluster on Docker

The project was featured on an article at MongoDB official tech blog! 😱

The project just got its own article at Towards Data Science Medium blog! ✨

Introduction

This project gives you an Apache Spark cluster in standalone mode with a JupyterLab interface built on top of Docker. Learn Apache Spark through its Scala, Python (PySpark) and R (SparkR) API by running the Jupyter notebooks with examples on how to read, process and write data.

TL;DR

curl -LO https://raw.githubusercontent.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker/master/docker-compose.yml
docker-compose up

Contents

• Quick Start
• Tech Stack
• Metrics
• Contributing
• Contributors
• Support

Quick Start

Cluster overview

Application	URL	Description
JupyterLab	localhost:8888	Cluster interface with built-in Jupyter notebooks
Spark Driver	localhost:4040	Spark Driver web ui
Spark Master	localhost:8080	Spark Master node
Spark Worker I	localhost:8081	Spark Worker node with 1 core and 512m of memory (default)
Spark Worker II	localhost:8082	Spark Worker node with 1 core and 512m of memory (default)

Prerequisites

• Install Docker and Docker Compose, check infra supported versions

Download from Docker Hub (easier)

1. Download the docker compose file;

curl -LO https://raw.githubusercontent.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker/master/docker-compose.yml

2. Edit the docker compose file with your favorite tech stack version, check apps supported versions;
3. Start the cluster;

docker-compose up

4. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
5. Stop the cluster by typing ctrl+c on the terminal;
6. Run step 3 to restart the cluster.

Build from your local machine

Note: Local build is currently only supported on Linux OS distributions.

1. Download the source code or clone the repository;
2. Move to the build directory;

cd build

3. Edit the build.yml file with your favorite tech stack version;
4. Match those version on the docker compose file;
5. Build up the images;

chmod +x build.sh ; ./build.sh

6. Start the cluster;

docker-compose up

7. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
8. Stop the cluster by typing ctrl+c on the terminal;
9. Run step 6 to restart the cluster.

Tech Stack

• Infra

Component	Version
Docker Engine	1.13.0+
Docker Compose	1.10.0+

• Languages and Kernels

Spark	Hadoop	Scala	Scala Kernel	Python	Python Kernel	R	R Kernel
3.0	3.2	2.12.10	0.10.9	3.7.3	7.19.0	3.5.2	1.1.1
3.1	3.2	2.12.10	0.10.9	3.7.3	7.19.0	3.5.2	1.1.1
3.2	3.2	2.12.10	0.10.9	3.7.3	7.19.0	3.5.2	1.1.1
2.x	2.7	2.11.12	0.6.0	3.7.3	7.19.0	3.5.2	1.1.1

• Apps

Component	Version	Docker Tag
Apache Spark	2.4.0 | 2.4.4 | 3.0.0 | 3.3.1	<spark-version>
JupyterLab	2.1.4 | 3.0.0 | 3.5.0	<jupyterlab-version>-spark-<spark-version>

Metrics

Image	Size	Downloads
JupyterLab
Spark Master
Spark Worker

Contributing

We'd love some help. To contribute, please read this file.

Contributors

A list of amazing people that somehow contributed to the project can be found in this file. This project is maintained by:

André Perez - dekoperez - [email protected] Zhiwei Zhang - [email protected]

Support

Support us on GitHub by staring this project ⭐

Support us on Patreon. 💖