Skip to content

Add API benchmark suite - #2007

Closed
adityawrk wants to merge 2 commits into
SecureBananaLabs:mainfrom
adityawrk:codex/api-benchmark-suite-30
Closed

adityawrk wants to merge 2 commits into
SecureBananaLabs:mainfrom
adityawrk:codex/api-benchmark-suite-30

Conversation

@adityawrk

@adityawrk adityawrk commented May 30, 2026 •

Copy link
Copy Markdown

/claim #30

Closes #30.

Summary

Adds a reproducible autocannon benchmark suite under benchmarks/ for every mounted /api/ route in the Express API. The runner can either start the local app in benchmark mode or target an existing host with BENCHMARK_TARGET_URL.

What is included:

  • npm run benchmark for the full local/staging benchmark suite
  • npm run benchmark:smoke for low-concurrency CI regression checks
  • autocannon 8.0.0 as the configured benchmark engine
  • Route coverage enforcement against mounted Express /api routes
  • Endpoint inventory for auth, users, jobs, proposals, payments, reviews, messages, notifications, uploads, search, and admin metrics
  • Unique per-run JSON and multipart payloads aligned with the current controllers and validators
  • Dedicated benchmark bearer token support for /api/admin/metrics
  • p50/p95/p99 latency, p99 response-header timing as the TTFB signal, sustained RPS, peak RPS, error rate, and status distribution
  • JSON and Markdown output under benchmarks/results/
  • Reviewable thresholds in benchmarks/thresholds.json
  • CI smoke workflow that fails on route coverage gaps, p99, TTFB p99, or error-rate threshold failures and uploads the latest reports as artifacts

Generated result files are intentionally ignored by git; the latest local Markdown summary is pasted below for review.

Validation

node --check benchmarks/run-benchmark.js
node --check benchmarks/routes.js
node --check apps/api/src/app.js
npm run benchmark:smoke
npm run benchmark
node --test apps/api/src/tests/*.test.js
git diff --check

Results:

  • npm run benchmark:smoke: 20 endpoints, 80 requests, 0% error rate, all thresholds passed
  • npm run benchmark: 20 endpoints, 1600 requests, 0% error rate, all thresholds passed
  • node --test apps/api/src/tests/*.test.js: 1 passing health test
  • fractional request env values such as BENCHMARK_REQUESTS=0.5 now fail validation instead of producing corrupt reports

Demo

Short demo video: https://github.com/adityawrk/bug-bounty/releases/download/bounty-30-demo-20260530/securebanana-30-demo.mp4

Benchmark Summary

API Benchmark Report

Generated: 2026-05-30T13:13:17.669Z

Mode: default
Target: local loopback app

Environment

Field Value
Node.js v25.2.1
Platform darwin arm64
CPU concurrency 10
Benchmark tool autocannon 8.0.0
Benchmark target local-app
Rate limiter disabled yes
Requests per endpoint 80
Concurrency 20
Timeout 10000 ms
RPS sample interval 1000 ms

Summary

Endpoints Requests Errors Error rate Sustained RPS Peak RPS
20 1600 0 0% 79.52 80

Endpoint Metrics

Route Requests p50 ms p95 ms p99 ms TTFB p99 ms Sustained RPS Peak RPS Error rate
POST /api/auth/register 80 1.38 12.2 14.44 14.43 80 80 0%
POST /api/auth/login 80 1.27 13.18 15.6 15.6 80 80 0%
GET /api/auth/oauth/:provider/callback 80 1 10.8 12.4 12.4 80 80 0%
POST /api/auth/refresh 80 2.01 25.24 27.83 27.83 80 80 0%
POST /api/users 80 1.46 18.89 20.97 20.97 79.21 80 0%
GET /api/users 80 2.27 24.64 28.4 28.39 79.21 80 0%
POST /api/jobs 80 1.32 23.92 25.74 25.73 79.21 80 0%
GET /api/jobs 80 1.82 25.82 28.74 28.73 79.21 80 0%
POST /api/proposals 80 1.54 21.11 22.96 22.96 79.21 80 0%
GET /api/proposals 80 1.74 18.61 21 21 79.21 80 0%
POST /api/payments 80 1.58 17.29 19.12 19.12 79.21 80 0%
POST /api/reviews 80 1.58 21.24 23.5 23.5 79.21 80 0%
GET /api/reviews 80 1.79 19.76 22.27 22.27 80 80 0%
POST /api/messages 80 1.65 18.76 20.59 20.59 79.21 80 0%
GET /api/messages 80 1.83 20.93 23.59 23.58 79.21 80 0%
POST /api/notifications 80 1.39 16.18 18.37 18.37 80 80 0%
GET /api/notifications 80 1.68 27.15 29.99 29.99 80 80 0%
POST /api/uploads 80 2.03 23.91 26.66 26.66 80 80 0%
GET /api/search 80 1.23 24.19 26.02 26.02 79.21 80 0%
GET /api/admin/metrics 80 2.11 25.11 27.7 27.7 79.21 80 0%

Threshold result: all configured thresholds passed.

Benchmark Environment

Hardware

  • CPU model & core count: Apple M4, 10 physical / 10 logical cores reported by macOS
  • RAM (total & available during benchmark): 16 GB total; available memory not pinned for this local run
  • Storage type (SSD / NVMe / HDD): local Apple SSD/NVMe-class storage
  • Network interface (Ethernet / WiFi / loopback): loopback
  • Machine type (local workstation / cloud VM / CI runner -- include instance type if cloud): local workstation
  • OS & version: macOS 26.2, arm64

Runtime

  • Node.js version (or relevant runtime): Node.js v25.2.1, npm 11.11.1
  • Any resource limits applied (Docker memory cap, cgroup limits, etc.): none intentionally applied
  • Other significant processes running during benchmark (yes / no -- if yes, describe): yes, normal Codex desktop and shell activity

If submitted by or with an AI agent

  • Agent or tool name (e.g. Claude Code, Devin, Copilot Workspace, AutoGPT): OpenAI Codex
  • Underlying model and version (e.g. claude-sonnet-4-5, gpt-4o -- if known): GPT-5-based Codex coding agent
  • Inference provider (e.g. Anthropic, OpenAI, Azure, self-hosted): OpenAI
  • Orchestration framework if any (e.g. LangChain, AutoGen, custom): none beyond Codex shell/GitHub tooling
  • Execution mode (fully autonomous / human-supervised / human-initiated per step): human-initiated, agent-executed implementation
  • Did the agent have shell/tool access during execution (yes / no): yes
  • Did the agent have internet access during execution (yes / no): yes, for GitHub issue/PR checks and npm package installation
  • Were benchmark commands run by the agent directly or handed off to the human to run: run directly by the agent locally
  • Any known agent constraints or sandboxing that may have affected execution: local benchmark/test commands required loopback bind permission; validation was rerun with local loopback binding allowed. No production secrets or staging targets were used.

@adityawrk

Copy link
Copy Markdown
Author

Follow-up after adversarial review: I updated the benchmark runner to use autocannon 8.0.0 as the configured load engine, added route coverage enforcement against mounted Express /api routes, made benchmark count env vars integer-validated, generated per-run unique payload IDs, kept the rate-limiter bypass behind BENCHMARK_DISABLE_RATE_LIMIT, and uploaded smoke reports as CI artifacts. Fresh local validation is in the PR body: smoke 20 endpoints/80 requests/0% errors and full 20 endpoints/1600 requests/0% errors.

@adityawrk adityawrk closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark APIs with p50, p95, p99 latency, RPS, error rate and TTFB

1 participant