Distributed Python Tester

This is the final project for the Distributed Systems UC for the second year of my Computer engeneering degree.

A distributed service for executing Python test suites across multiple worker nodes. Each node exposes a Flask API, joins a peer mesh, and can execute part of an evaluation locally or delegate work to other nodes. An Nginx container sits in front of the cluster and forwards client traffic to the bootstrap node.

The system accepts either:

• A .zip file containing one or more Python projects
• A JSON payload with GitHub repository URLs to download and evaluate

Results are persisted per node and can be queried while an evaluation is running or after it completes.

Architecture

The project is built from the following components:

• src/run_node.py: container entrypoint; starts a MeshNode process and the Flask API.
• src/node/meshNode.py: peer discovery, synchronization, health checks, evaluation lifecycle, and statistics.
• src/node/evaluator.py: extracts projects, detects tests, balances work across nodes, and aggregates results.
• src/testing/test_running.py: creates temporary virtual environments, installs project requirements, and runs pytest.
• src/app/routes.py: HTTP API exposed by each node.
• docker-compose.yml: starts a 4-node cluster plus Nginx.
• default.conf: Nginx reverse proxy configuration.

Default cluster topology:

• cnode1 on port 5001
• cnode2 on port 5002
• cnode3 on port 5003
• cnode4 on port 5004
• nginx on port 80

How It Works

1. A node starts with its own address and, optionally, a bootstrap node.
2. New nodes join the mesh through /join and receive the current peer list.
3. When an evaluation request arrives, the service:
   • extracts or downloads the target projects
   • detects Python projects by looking for requirements.txt and tests
   • splits test files across the least-loaded nodes
   • runs pytest in isolated temporary virtual environments
   • merges partial results and persists the evaluation
4. Nodes periodically ping peers and sync evaluations to tolerate failures.

Requirements

• Docker
• Docker Compose
• Bash, for boot.sh

If you want to run Python files directly outside Docker, install the dependencies in requirements.txt.

Running The Cluster

Option 1: helper script

chmod +x boot.sh
./boot.sh

What the script does:

• stops and removes the current compose stack
• detects the host machine IP
• exports HOST_REAL_IP
• rebuilds and starts the cluster in detached mode

Option 2: manual start

export HOST_REAL_IP=$(hostname -I | awk '{print $1}')
docker compose up -d --build

Stopping the cluster

docker compose down --volumes --timeout 0

Running A Single Local Docker Node

For ad hoc testing, src/local_node.py can start standalone Docker containers without Compose.

Examples:

python src/local_node.py 5001
python src/local_node.py 5002 5001
python src/local_node.py 5003 192.168.1.20 5001

Meaning:

• python src/local_node.py 5001: start a new network on port 5001
• python src/local_node.py 5002 5001: join the local bootstrap node on port 5001
• python src/local_node.py 5003 192.168.1.20 5001: join a bootstrap node on another machine

API

When the full stack is running, the easiest entrypoint is http://localhost through Nginx.

POST /evaluation

Submit a new evaluation.

Supported content types:

• multipart/form-data with a file field containing a zip archive
• application/json with repository URLs

Zip upload example:

curl -X POST http://localhost/evaluation \
  -F "file=@projects.zip"

GitHub repositories example:

curl -X POST http://localhost/evaluation \
  -H "Content-Type: application/json" \
  -d '{
    "auth_token": "ghp_your_token",
    "projects": [
      "https://github.com/owner/project-a",
      "https://github.com/owner/project-b"
    ]
  }'

Response:

{
  "id": "evaluation-uuid"
}

GET /evaluation

Returns all known evaluations aggregated from the cluster.

curl http://localhost/evaluation

GET /evaluation/<eval_id>

Returns the complete data for a specific evaluation.

curl http://localhost/evaluation/<eval_id>

Typical fields include:

• status
• _run_info
• module_results
• per-project summaries

GET /stats

Returns node-level and global statistics collected from the cluster.

curl http://localhost/stats

GET /network

Returns the current peer view for the cluster.

curl http://localhost/network

GET /peers

Returns the peer list from the node that answered the request.

curl http://localhost/peers

Internal endpoints

These routes are used for peer-to-peer coordination and internal execution:

• GET /evaluation_req
• GET /evaluation_req/<eval_id>
• GET /stats_req
• POST /join
• POST /new_peer
• GET /ping
• POST /run_tests
• GET /load
• GET /kill

Evaluation Output

The exact response depends on the project under evaluation, but completed evaluations follow this general structure:

{
  "status": "done",
  "_run_info": {
    "errors": 0,
    "%passed": 87.5,
    "%failed": 12.5,
    "nota": 17.5,
    "eval_time": 4.21
  },
  "module_results": {
    "project-a": {
      "tests/test_example.py": {
        "%passed": 100.0,
        "%failed": 0.0
      }
    }
  }
}

Important details:

• each test module is executed independently
• project requirements are installed before running tests
• temporary virtual environments are created under /tmp
• each node persists evaluations in files like evaluations_<host>_<port>.json

Repository Structure

.
├── src/
│   ├── app/                # Flask app and HTTP routes
│   ├── load_balance/       # Least-loaded distribution strategy
│   ├── node/               # Mesh node, evaluator, persistence
│   ├── testing/            # Pytest execution helpers and result aggregation
│   ├── local_node.py       # Run standalone Docker nodes
│   └── run_node.py         # Main runtime entrypoint
├── docs/                   # Project report and specification PDFs
├── Dockerfile
├── docker-compose.yml
├── default.conf
├── boot.sh
└── requirements.txt

Notes And Limitations

• Nginx currently forwards requests to cnode1, which acts as the public entrypoint.
• GitHub-based evaluation depends on GitHub API availability and a valid token when private repositories or rate limits are relevant.
• Test execution installs project dependencies dynamically, so failures in pip install will fail that project evaluation.
• Persistence is file-based and local to each node container.

Project Documents

Additional project material is available in docs/Projecto CD 2025-3.pdf, docs/Relatorio.pdf, and docs/Porotocolo.pdf.