Benchwarmer

A SQLite-backed job queue for plain Ruby — a polling worker, built-in cron scheduling, and a Sinatra web UI for inspecting past runs, enqueuing jobs, and re-running failures. No Rails, no Redis, no ActiveJob. Jobs sit on the bench until a worker puts them on the ice.

Benchwarmer targets small apps that need background work — nightly data syncs, batch backfills, static-site builds — especially long-running shell-out jobs and ordered pipelines ("sync data, then build the site"). It is the plain-Ruby answer to plainjob.

Features

• SQLite backend in the queue's own database file — no server to run.
• A single polling worker: atomic claim-by-UPDATE, timeout-requeue of crashed jobs, automatic retention cleanup.
• Batching + ordering: a parent job can enqueue children, and a child can be ordered after: a sibling so it only runs once that sibling succeeds.
• Retries with lineage: attempts + backoff, and re-running a failed job from the UI creates a fresh row linked back to the original.
• Captured output and error/backtrace per job, streamed live into the row.
• Built-in 5-field cron scheduling (no external cron daemon).
• A web UI (Sinatra + Phlex) to browse runs, enqueue jobs, and retry failures.

Requirements

• Ruby >= 3.2
• SQLite >= 3.35 (for UPDATE ... RETURNING, used by the atomic claim). The sqlite3 gem >= 2.x vendors a modern libsqlite3, so this normally just works; Benchwarmer asserts the version at connect time with a clear error otherwise.

Install

The gem installs as benchwarmer-rb; in code it is require "benchwarmer".

# Gemfile
gem "benchwarmer-rb"

bundle install

Quickstart

1. Define job classes. A job is a plain class that subclasses Benchwarmer::Job and implements #perform. perform can take named parameters (delivered from an enqueued instance's constructor kwargs) or positional arguments (splatted from an args: array) — either way they round-trip through JSON.

require "benchwarmer"

class GreetJob < Benchwarmer::Job
  def self.description = "Say hello (shown in the UI enqueue form)"

  def perform(name:, language: "en")
    append_output("Hello, #{name}!\n") # streams into the job's row
  end
end

Every subclass registers itself automatically under its job_type ("GreetJob" -> "greet", dropping a trailing Job and underscoring). There is no registry to build or pass around — the worker and web UI read this global registry by default.

2. Enqueue. The queue owns its own SQLite file. Enqueue by passing a job class (with positional args:) or a job instance (carrying its keyword arguments):

queue = Benchwarmer::Queue.new(db_path: "db/benchwarmer.db")

queue.enqueue(GreetJob.new(name: "World"))         # kwargs -> perform(name:)
queue.enqueue(GreetJob.new(name: "World", language: "fr"))
queue.enqueue(PositionalJob, args: ["a", 1])       # array -> perform(*args)

# enqueue also takes: delay: (seconds), run_at:, parent_id:, after:,
#                     max_attempts:, scheduled_job_id:

Instance kwargs are serialized as a JSON object and delivered back as keyword arguments; an args: array is serialized as a JSON array and splatted as positional arguments. Args must be JSON-serializable — anything else raises an ArgumentError at enqueue time, not later inside the worker. (You can also enqueue by type string — queue.enqueue("greet", args: {name: "World"}) — which is how the web form submits; the class/instance forms are preferred in code.)

3. Run a worker. Worker#run blocks, polling for runnable jobs and running them one at a time. It traps INT/TERM to shut down cleanly (the in-flight job always finishes first). No registry: needed — it defaults to the global auto-registry.

Benchwarmer::Worker.new(
  db_path: "db/benchwarmer.db",
  poll_interval: 5 # seconds between polls
).run

4. Mount the web UI. The web app is a separate entry point, require "benchwarmer/web", mounted from a config.ru. Only the queue factory is injected; the enqueue form's job list comes from the auto-registry:

# config.ru
require "benchwarmer/web"
require_relative "jobs" # your job classes (auto-registered on load)

Benchwarmer::Web.set :queue_factory, -> { Benchwarmer::Queue.new(db_path: "db/benchwarmer.db") }

run Benchwarmer::Web

bundle exec rackup config.ru   # http://localhost:9292

queue_factory is a lambda called once per request. The UI lists past runs with filters and status tabs, shows per-job detail (args, timing, attempts, error, output, parent/children, retry lineage), and has forms to enqueue a job or retry a failed one.

A working demo of everything below lives in examples/.

Consuming-app integration pattern

Benchwarmer stays generic; your app supplies job classes and two small processes: a worker (bin/worker) that builds the registry, upserts schedules, and calls Worker#run, and a web process (config.ru) that mounts Benchwarmer::Web.

Long-running CLI tasks (the ShellJob pattern)

For minutes-long shell-outs (data syncs, site builds), shell out through a small SubprocessRunner and stream the command's output into the job row. Inject the runner so it can be faked in tests:

require "open3"

# Runs a CLI command, streaming combined stdout+stderr line-by-line to on_output,
# and returns the Process::Status.
class SubprocessRunner
  def run(*argv, chdir: Dir.pwd, on_output: ->(_line) {})
    Bundler.with_unbundled_env do
      Open3.popen2e(*argv, chdir: chdir) do |stdin, out, wait_thr|
        stdin.close
        out.each_line { |line| on_output.call(line) }
        wait_thr.value
      end
    end
  end
end

class ShellJob < Benchwarmer::Job
  class CommandFailed < StandardError; end

  def initialize(runner: SubprocessRunner.new) # injectable for tests
    @runner = runner
  end

  def run!(*argv, chdir: Dir.pwd)
    status = @runner.run(*argv, chdir: chdir, on_output: ->(line) { append_output(line) })
    return if status.success?

    raise CommandFailed, "#{argv.join(" ")} exited #{status.exitstatus}"
  end
end

class SyncJob < ShellJob
  def perform = run!("bin/sync-data", "--full")
end

Why it's built this way:

• popen2e merges stdout and stderr so interleaved progress lines are captured in order; each line is appended to the row via append_output, which keeps only the trailing 64 KB.
• Bundler.with_unbundled_env drops Benchwarmer's own bundler environment so the shelled-out tool resolves its own gems / Ruby.
• stdin is closed so a CLI that unexpectedly prompts fails fast instead of hanging the worker forever.
• Shelling out (rather than doing the work in-process) keeps the worker fork-safe when the target CLI forks its own children.

The runner is injected through the job's constructor. The worker builds jobs with its job_factory: (default ->(klass) { klass.new }), which is the seam for passing real collaborators in production and fakes in tests. Collaborators are dependency-injected here; a job's serialized params (the kwargs it was enqueued with) are separate data, delivered straight to #perform:

Benchwarmer::Worker.new(
  db_path: "db/benchwarmer.db",
  job_factory: ->(klass) { klass.new(runner: RealRunner.new) }
)

Pipelines (the batching + after: pattern)

A parent job enqueues children; pass after: a sibling's id to order a child after that sibling succeeds. Inside a job, enqueue_child sets parent_id for you and returns the child's id. It takes the same targets as enqueue — a job class, a job instance, or a type string:

class NightlyJob < Benchwarmer::Job
  def perform
    sync_id = enqueue_child(SyncJob)
    enqueue_child(BuildJob, after: sync_id) # runs only after sync succeeds
  end
end

The parent succeeds as soon as the children are enqueued; its detail page shows the whole batch. Each step is its own row, so:

• you get a visible per-step status, timing, and output — not one giant blob;
• you can retry a single failed step from the UI without silently re-running or skipping the others;
• if a step exhausts its attempts, the worker's cascade marks any job ordered after it as failed with Benchwarmer::DependencyFailed (recovery is enqueuing a fresh pipeline, or retrying the failed step). This is why per-step rows and retries matter: the alternative — a parent that runs each step inline, or a job that enqueues the next step on success — couples the steps together and loses per-step retry.

Cron scheduling

Upsert a recurring schedule by a stable name (calling it again with the same name updates it in place). The worker's scheduler tick enqueues due schedules and recomputes the next run:

queue.schedule(name: "nightly", type: NightlyJob, cron: "30 2 * * *") # or type: "nightly"
queue.unschedule(name: "nightly")

type: accepts a job class (stored as its job_type) or the type string directly, and args: may be a JSON array (positional) or object (named), just like enqueue.

The built-in parser is a 5-field cron (minute hour day-of-month month day-of-week) supporting:

Syntax	Meaning
*	every value
5	a specific value
1-5	an inclusive range
1,3,5	a list
*/15	steps (also 0-30/10, 5/10)

Day-of-week is 0-7 (both 0 and 7 are Sunday). When both day-of-month and day-of-week are restricted, a day matches if either matches (standard Vixie-cron behavior). There is no support for L/W, #, or named months / weekdays — this is meant for nightly/hourly schedules, not exotic ones.

Timezone: cron is evaluated in the process's local timezone. A schedule like "30 2 * * *" means 2:30am wherever the worker runs. Missed windows (machine down overnight) fire once on the next boot, not replayed. Set TZ explicitly in your deployment (e.g. TZ=America/New_York) so the schedule doesn't drift when the host's zone changes. next_run_at is stored in UTC.

Operational notes

• The queue gets its own SQLite database file (e.g. db/benchwarmer.db) — never point it at your app's primary database. It writes in WAL mode and prunes its own rows.
• Jobs run serially (concurrency = 1) by design. A single worker loop runs one job at a time, inline. The target workload is pipelines whose steps contend over shared resources (a sync writing a file a build reads), so serialization is a feature, not a limitation. There is no thread pool.
• job_timeout (default 1 hour) vs. long jobs. If the worker crashes, a job stuck in running is requeued (or failed, if out of attempts) once its claimed_at is older than job_timeout. Set job_timeout: comfortably above your longest expected job so a legitimately slow shell-out isn't requeued while still running. The trade-off: after a real crash, a stale job waits up to job_timeout before rerunning.
• Retention. Succeeded jobs are pruned after 7 days and failed jobs after 30 days by default; override via retention: { succeeded: <seconds>, failed: <seconds> }. Pruning runs at most hourly.
• Web UI auth. The UI is an ops tool. Set BENCHWARMER_WEB_PASSWORD to require HTTP Basic auth (any username, that password). Set BENCHWARMER_SESSION_SECRET to a stable secret so sessions (and CSRF tokens) survive restarts; without it a random secret is generated per boot. Both are read from the environment.
• SQLite >= 3.35 is required (see Requirements).

Development

bundle install
bundle exec rake test   # minitest suite (in-memory SQLite, injected clock)
bin/check               # standardrb style gate (no autofix)

To try it end-to-end, run the demo in examples/ from the repo root:

bundle exec ruby examples/bin/worker      # worker process
bundle exec rackup examples/config.ru     # web UI at http://localhost:9292

Then open the UI, enqueue a nightly job, and watch the sync then build children run in order with their output streaming into each row.

Previewing the UI

For a quick look at the UI without running a worker, bundle exec rake dev:ui seeds a throwaway SQLite db with data covering every UI feature (queued/running/succeeded/failed jobs, batches, retries, cron schedules) and serves the web UI at http://127.0.0.1:6767. Press Ctrl-C to stop; the seeded db is deleted automatically.

License

MIT — see LICENSE.