BuddyNet

BuddyNet is a distributed data-processing platform for crunching large tabular datasets, such as CSV files, across multiple machines on the same LAN. One machine runs a lightweight registry that lets the others discover each other; everyone else runs the Tray Application, which contributes compute and lets you submit jobs through a local web UI.

Development Approach

BuddyNet was developed using a spec-driven development workflow with Kiro, an agentic development environment by AWS.

For key features, we used structured specifications to define requirements, technical design decisions, and implementation tasks before development. This helped us keep the implementation aligned across the team and provided clear documentation for both development and review.

Kiro was used as a development tool throughout the project, while architecture, implementation decisions, testing, and validation remained part of the team's engineering process.

Components

• registry - a small Spring Boot service that machines on the LAN register with and discover each other through. Distributed as a Docker image (sfirat/registry). This is the one piece of infrastructure someone on the network needs to run.
• tray-app and tray-app/frontend - the app every contributing machine runs: a system-tray app with an embedded React UI for building and submitting jobs, plus worker logic that executes chunks of a job locally or claims them from peers.
• installer - a Go terminal UI (buddynet) that installs the registry as a Docker container and manages it afterward (start/stop/restart, logs, update, uninstall).

Documentation

• Registry overview and docs: registry/README.md
• Registry API contract: registry/docs/API_CONTRACT.md
• Registry configuration: registry/docs/CONFIGURATION.md
• TrayApplication overview and backend docs: tray-app/README.md
• Frontend setup and docs: tray-app/frontend/README.md
• Frontend API contract: tray-app/frontend/docs/API_CONTRACT.md
• Frontend structure: tray-app/frontend/docs/STRUCTURE.md

What You Need

To run...	You need
The registry, via the installer TUI	Docker + Go
The registry, without Docker	Java 21 + Git
The Tray Application	Java 21 + Git
The frontend dev server	Node/npm

Both registry/ and tray-app/ ship a Maven wrapper (mvnw / mvnw.cmd), so you do not need Maven installed globally. The wrapper downloads Maven on first use.

Quick Start

The registry only needs to run on one machine on the LAN.

Note: the hosted install script (buddynet.selimhan.dev) is no longer available. Build the installer from source instead (requires Go), or skip the installer entirely and run the registry directly (next section).

cd installer && go build -o buddynet ./cmd/installer
./buddynet

The first run walks you through a guided install: it checks for Docker, pulls the sfirat/registry image, and starts the container.

Run buddynet again any time afterward to manage it: start/stop/restart, view live logs, update to a new image, or uninstall. Note down the address it reports; that is the registry.url every Tray Application instance needs.

Running The Registry Without Docker

Linux/macOS:

cd registry && ./mvnw spring-boot:run

Windows PowerShell:

cd registry; .\mvnw.cmd spring-boot:run

Defaults come from registry/src/main/resources/application.properties (server.port=1881, registry.staleness-threshold-ms=5000). You can override them without editing the file:

cd registry && ./mvnw package -DskipTests
java -jar target/registry-0.0.1-SNAPSHOT.jar --server.port=1881 --registry.staleness-threshold-ms=5000

PowerShell:

cd registry; .\mvnw.cmd package -DskipTests
java -jar target\registry-0.0.1-SNAPSHOT.jar --server.port=1881 --registry.staleness-threshold-ms=5000

Running The Tray Application

Every machine that should contribute compute or submit jobs runs this app.

Build it:

cd tray-app && ./mvnw package

PowerShell:

cd tray-app; .\mvnw.cmd package

On first run the app creates ~/.config/tray-app/application.properties with a commented template. Set registry.url there:

registry.url=https://your-registry-address
# Optional:
# registry.heartbeat-interval-ms=1000
# node.interface=wlan0
# node.port=8080
# node.display-name=Alice's Laptop

Until registry.url is set, the app runs but stays unregistered and cannot see other nodes. After startup, the frontend settings drawer can update the registry URL, display name, advertised network interface, and backend port; the registry URL and display name apply immediately, while interface and port changes require a restart.

Run the packaged app:

cd tray-app && java -jar target/tray-app-0.0.1-SNAPSHOT.jar

PowerShell:

cd tray-app; java -jar target\tray-app-0.0.1-SNAPSHOT.jar

During development:

cd tray-app && ./mvnw spring-boot:run

PowerShell:

cd tray-app; .\mvnw.cmd spring-boot:run

The BuddyNet tray icon opens the job-submission UI in your browser. The backend starts at node.port and increments if that port is busy. If no system tray is available (headless session, unsupported desktop), the app still runs — open http://localhost:8080 (or whichever port the log reports) directly.

Running The Frontend Dev Server

The packaged TrayApplication builds and serves the frontend automatically. Use the standalone frontend dev server when working on the React UI.

Linux/macOS:

cd tray-app/frontend && npm install && npm run dev

Windows PowerShell:

cd tray-app\frontend; npm install; npm run dev

Repository Layout

registry/            Spring Boot discovery service
tray-app/            Spring Boot tray app, workers, jobs, REST API
tray-app/frontend/   React UI embedded in the tray-app jar
installer/           Go TUI that installs/manages the registry container

Development

Each module has its own wrapper and focused docs for API contracts, configuration, verification, benchmarking, and architecture.

Quick test reference:

cd registry && ./mvnw test
cd tray-app && ./mvnw test
cd tray-app/frontend && npm test

PowerShell:

cd registry; .\mvnw.cmd test
cd ..\tray-app; .\mvnw.cmd test
cd frontend; npm test

Full Verification

Before opening or merging a pull request, it is recommended to run:

mvn -f registry/pom.xml verify
mvn -f tray-app/pom.xml verify

The test command runs the test suite, while verify runs a more complete Maven lifecycle check.

Test Reports

Maven generates test reports under each application's target directory:

registry/target/surefire-reports/
tray-app/target/surefire-reports/

Benchmarking & Performance Instrumentation

The Tray Application has built-in timing instrumentation for measuring where time goes during distributed query execution. It is off by default and produces zero extra output during normal operation. Use it to find bottlenecks and validate optimizations.

What you can measure

When enabled, the app emits [benchmark] log lines:

• Per-job summary — file, chunk count, chunk size, local/remote task split, total wall time, and merge time.
• Per-drain summary — per-phase timing (execute, and claim+download/report for remote tasks) with avg/max/total across the tasks in that drain.
• Baseline comparison (optional) — runs the same query against the whole file with a single in-process DuckDB connection (no chunking, no HTTP) and reports how much overhead distribution adds. This is the apples-to-apples "what could this machine do on its own" number.
• Per-task DuckDB sub-phases (optional, DEBUG) — duckdb-init (connection open), duckdb-view (CSV view creation), duckdb-query (query + result materialization).

Enabling it

Pass the flags when starting the app (no code or properties changes needed):

# from tray-app/
./mvnw spring-boot:run -Dspring-boot.run.arguments="--benchmark.enabled=true --benchmark.baseline=true"

Or, if running a packaged jar:

java -jar target/tray-app-0.0.1-SNAPSHOT.jar --benchmark.enabled=true --benchmark.baseline=true

• benchmark.enabled=true — per-job and per-drain timing summaries.
• benchmark.baseline=true — also run the single-process DuckDB baseline for comparison.

To see the per-task DuckDB sub-phase breakdown, also raise the log level for the collector:

--logging.level.com.amazon.aws.trayapp.instrumentation.LoggingTaskMetricsCollector=DEBUG

You can instead set these in src/main/resources/application.properties if you prefer not to pass arguments each run, but leave them false on commit so normal runs stay quiet.

Running a benchmark

1. Start the app with benchmarking enabled (see above).
2. Open the GUI (served at http://localhost:8080/ by default), select a CSV file, build a pipeline, and submit the job. The chunk size can be tuned in the job request; larger chunks mean fewer tasks and less per-task overhead.
3. Watch the application console for the [benchmark] lines.

Example output for an aggregation query over a 2.5 GB CSV with 256 MB chunks:

[benchmark] job complete — file=big_3gb.csv (2.5GB) | chunks=11 | chunkSize=256MB | local=11/11 | remote=0/11 | query="SELECT age, AVG(salary) ..." | wall=6941ms | merge=3ms
[benchmark] baseline (local DuckDB, no chunking): 1554ms
[benchmark] distributed: 6941ms | overhead: 5387ms (+346%)
[benchmark] drain: tasks=11 (local=11 remote=0) | execute: avg=613ms max=993ms total=6743ms

With the DEBUG level raised, each task additionally logs:

[<taskId>] duckdb-init=5ms (local)
[<taskId>] duckdb-view=73ms (local)
[<taskId>] duckdb-query=236ms (local)
[<taskId>] execute=584ms (local)

Suggested queries

These exercise different parts of the system. The pipeline GUI generates the SQL for you; the SQL is shown here for reference:

• Aggregation (small result): SELECT age, AVG(salary) AS avg_salary FROM data GROUP BY age ORDER BY age ASC — dominated by per-chunk compute; the canonical benchmark.
• Filter + sort (large result): SELECT age, salary FROM data WHERE salary > 50000 ORDER BY salary DESC — stresses result serialization and the merge step rather than compute.
• Multi-column GROUP BY: SELECT age, salary, COUNT(*) AS cnt, SUM(salary) AS total FROM data GROUP BY age, salary ORDER BY total DESC — more partial-result rows; stresses the merge harder.

Interpreting the numbers

• If execute dominates the drain, you are compute-bound — chunk size and DuckDB are the levers.
• If claim+download or report dominate (remote tasks), you are network-bound — overlapping transfer with compute is the lever.
• The baseline comparison tells you the cost of distribution on this machine. On a single machine the baseline will usually win; the distributed system is designed to pay off across multiple machines.

DuckDB cold-start micro-benchmark

A standalone measurement of the per-task DuckDB connection cost (fresh connection per query vs. reusing one) is available as a disabled-by-default test:

mvn -f tray-app/pom.xml test -Dtest=DuckDbColdStartBenchmarkTest -DrunDuckDbBenchmark=true

It prints total/avg/min/max for both strategies and the time saved by reuse.

Team

BuddyNet was built by Derin Aksan, Ali Bakir, Selimhan Fırat, Eren Meric, and Kutay Ozbagci, with the mentorship of Amazon Web Services (AWS).

License

Released under the MIT License — see LICENSE.