atago tests real CLI behavior from plain YAML: commands, files, snapshots, services, and interactive terminals. It runs your actual binary — in any language — and asserts what a user observes. No test code, no shell DSL.
Documentation: https://nao1215.github.io/atago/
No install, no fictional binary — if you have Go, paste this. It records a real run of a command you already have, then replays it as a test:
go run github.com/nao1215/atago@latest record --out demo.atago.yaml -- git --version
go run github.com/nao1215/atago@latest run demo.atago.yaml.
PASSED 1 scenario: 1 passed, 0 failed, 0 errored, 0 skipped
record runs git --version once and writes a spec from what it observed — the
exit code, the version line on stdout, an empty stderr. run replays it. Open
demo.atago.yaml and you have a real test you can tighten, not YAML you wrote
from scratch. (Swap git --version for any command you have: go version,
jq --version, ls -la.)
Then point it at your own tool:
atago record --out mytool.atago.yaml -- mytool convert input.txt # turn a real run into a spec
atago run mytool.atago.yaml # replay it as a test
atago run --report junit specs/ # or run a whole suite in CIPick the tool that owns your layer:
| You are testing | Use |
|---|---|
| An HTTP/gRPC API server — scenario-based API testing | runn |
| A whole platform — integration suites across HTTP, gRPC, Kafka, databases, and more | venom |
| Shell functions and scripts — BDD-style unit tests | ShellSpec |
| Bash scripts — TAP-style tests | Bats |
| A CLI product — exit codes, output, generated files, snapshots, interactive prompts and TUIs | atago |
If the server or the platform is the system under test, use runn or venom. atago points the other way: the CLI is the product, and HTTP, database, SSH, gRPC, browser, and mock servers appear only as peers your CLI talks to.
go install github.com/nao1215/atago@latestOn macOS, Homebrew works too:
brew install --cask nao1215/tap/atagoOn Windows, Scoop installs it from this repository's own bucket:
scoop bucket add nao1215 https://github.com/nao1215/atago
scoop install nao1215/atagoOn Arch Linux, install the atago-bin package from the AUR:
yay -S atago-bin # or: paru -S atago-binatago is in the aqua standard registry. Add it to your aqua.yaml, then install it:
aqua g -i nao1215/atago
aqua imise can install it through the same registry with the aqua backend:
mise use -g aqua:nao1215/atagoIf your installed mise release does not see nao1215/atago yet, update mise or enable floating registries with mise settings registry_floating=true.
The release page contains prebuilt binary archives for Linux, macOS, and Windows (amd64/arm64; .tar.gz, or .zip on Windows), plus .deb, .rpm, and .apk packages for Linux. Requires Go 1.26 or later when building from source.
Runs on Linux, macOS, and Windows (CI tests all three).
You don't write the first spec — your tool does. atago record -- <command> runs it once and generates a spec from what it observed (exit code, output, created files):
$ atago record --out mytool.atago.yaml -- mytool convert input.txt
recorded: exit 0, 2 lines of stdout, 1 file created
wrote mytool.atago.yaml
$ atago run mytool.atago.yaml
.
PASSED 1 scenario: 1 passed, 0 failed, 0 errored, 0 skipped (12ms)Interactive tools record too: atago record --pty -- <command> runs it in a real terminal, lets you drive one session by hand, and writes a pty: step that replays your keystrokes as expect/send pairs. It works on Linux, macOS, and Windows (a ConPTY); on POSIX a password prompt becomes an ${env:...} placeholder automatically, while on Windows — where a ConPTY exposes no echo state — you convert a secret send to ${env:...} by hand. A --pty session is bounded by --timeout (default 30s): if the program never exits, atago kills it, writes whatever was captured, and fails instead of hanging forever:
$ atago record --pty --out wizard.atago.yaml -- mytool initPrefer a blank template? atago init scaffolds one. Either way, the shape is always the same: declare a command, run it, assert on what it produced.
version: "1"
suite:
name: example
scenarios:
- name: echo greets the world
steps:
- run:
shell: true # portable: echo is a shell builtin on Windows
command: echo "hello atago"
- assert:
exit_code: 0
stdout:
contains: atago
stderr:
empty: trueatago run accepts spec files and directories (searched recursively for *.atago.yaml; the *.atago.yml spelling is accepted too). Each scenario runs in its own temporary directory, and progress streams as a dot per scenario (. pass, F fail, E error, s skip):
$ atago run ./specs
.....................................................................................................
PASSED 160 scenarios: 160 passed, 0 failed, 0 errored, 0 skipped (20.5s)Scenarios run concurrently by default (--parallel N, defaulting to your CPU count; set --parallel 1 to serialize). Workdirs are isolated, but the host network is shared — so if two scenarios each start a background service:, give them distinct ports, or one scenario's requests can reach the other's server.
When a check fails, atago prints exactly what was expected and what happened; multi-line mismatches render a colorized unified diff:
FAILED: demo / greeting matches its golden
Step:
assert stdout snapshot
Diff (-expected +actual):
--- snapshot (golden)
+++ actual
@@ -1,3 +1,3 @@
hello
-WORLD
+world
bye
Hint:
stdout did not match snapshot "snaps/greeting.txt" (update with --update-snapshots if intended)
fixture: writes input files into the isolated workdir; file:/dir: assertions check what the command produced, and snapshot: pins output to a committed golden file (volatile details like temp paths, UUIDs, and timestamps are normalized). A fixture's source is one of content: (inline text), base64: (inline bytes), from: (copy an existing file), or symlink: (link to a target):
scenarios:
- name: the generator writes the expected files
steps:
- run:
command: mytool generate --out site
- assert:
file:
path: site/index.html
contains:
- "<html"
- assert:
stdout:
snapshot: snapshots/generate.txt # record/refresh with `atago snapshot update`See files_and_fixtures, snapshot, and dir_tree for whole-tree golden manifests.
A pty step runs the command in a real pseudo-terminal and drives it with a declarative expect/send session — wizards, REPLs, and TTY-detection branches, no expect(1) scripting:
scenarios:
- name: the init wizard completes
steps:
- pty:
command: mytool init
session:
- expect: "Project name:"
- send: "demo\n"
- expect: "created demo/"
- assert:
exit_code: 0For full-screen TUIs, expect_screen: waits on the LIVE rendered frame during the session, and screen: asserts the final rendered frame after exit:
- pty:
command: mytool dashboard
session:
- expect_screen:
contains: "Ready"
stable_for: 100ms
- send: "q"
- assert:
screen:
contains: "Summary"Named keys (send: {key: enter}) and rendered-screen checks cover full TUIs — including control-byte aliases like ctrl-space, ctrl-[, and ctrl-_, plus modified key events like ctrl-hyphen/ctrl-minus for apps that distinguish the physical Ctrl+- key — see pty, pty_screen, and the cross-platform pty_portable. A tool that draws its interface on stderr and prints its result on stdout is testable as one step — the UI on the rendered screen, the result in the redirected file: pty_stdout_split. pty steps and atago record --pty run on Linux, macOS, and Windows (where they drive a ConPTY pseudo-console); only signal: stays POSIX-only. The pty/pty_screen examples skip on Windows because their inner commands ([ -t 0 ], cat -v, a SIGINT trap) are POSIX-specific, not because the pty mechanism is.
The same YAML also drives HTTP, database, SSH, gRPC, headless-browser, and offline mock-server peers — as dependencies of the CLI under test. atago init --template <name> scaffolds each:
$ atago init --list-templates
browser drive a headless Chrome; assert page content (needs Chrome on PATH)
cli run a command; assert exit code/stdout/stderr (runs as-is)
db run SQL; assert on rows via bundled SQLite (runs as-is)
grpc call a unary gRPC method via server reflection (edit target first)
http call an HTTP API; assert status and JSON body (edit base_url first)
mock stub an HTTP API offline and assert what the client sent (needs curl on PATH)
services test against a background server: readiness, retry, teardown (runs as-is)
ssh run a command on a remote host over SSH (edit host/user first)Every feature has a commented, runnable spec under examples/, tested in CI on Linux, macOS, and Windows. The cookbook collects 50+ copyable recipes for common jobs — converting images, driving prompts and TUIs, simulating API failures offline, proving idempotency — and indexes every spec by task and by feature.
Selection flags compose with any spec: --filter NAME (repeatable, and comma-separated for OR — --filter a,b or --filter a --filter b runs scenarios whose name contains a or b), --tag T and --skip-tag T (tags match exactly, not by substring — atago list shows the available tags), --parallel N, --fail-fast, and --rerun-failed. While authoring, --verbose traces every command, capture, and assertion verdict — for passing scenarios too.
Real E2E suites flake (timing, ports, external tools). --retry-failed N re-runs failed scenarios in a fresh workdir and reports recovered ones as flaky, loud in every report format; silent retries are explicitly a non-goal. A flaky scenario still fails the run, because one whose answer depends on how many times it ran has not been shown to work — pass --allow-flaky for a suite whose instability you already know about and accept. --repeat N does the opposite job: run each scenario N times to detect flakiness before it reaches CI.
atago run --ci --retry-failed 2 ./specs # retry, report instability loudly, still fail
atago run --ci --retry-failed 2 --allow-flaky ./specs # ...and keep CI green when that is accepted
atago run --repeat 20 --filter "race prone" ./specs # flake detectionsetup-atago installs a released binary:
name: behavior-specs
on: [push, pull_request]
jobs:
atago:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: nao1215/setup-atago@v0
- run: atago run --ci --report gha ./specsOn GitLab CI (or any CI that starts from a container image), use the published GHCR image:
image: ghcr.io/nao1215/atago:latest
stages: [test]
behavior-specs:
stage: test
script:
- atago run --ci --report junit ./specs > junit.xml
artifacts:
when: always
reports:
junit: junit.xmlThe image contains atago and ca-certificates; if your scenarios drive git,
jq, a browser, or your own CLI binary, build FROM ghcr.io/nao1215/atago:latest
and layer those tools on top.
--report json|junit|gha|tappicks the report format; the JSON shape is stable and versioned.--cienables deterministic, color-free output. It also turns an empty selection into a hard error: a--filter/--tag/--skip-tagthat matches no scenario fails the run (exit 3) instead of passing an empty suite, so a typo cannot silently disable your specs. Without--cithe same case is a warning that still exits 0.--artifacts-dir DIRpersists the exact payloads a failed assertion compared — plus, for a failed scenario, its background services' logs and each mock server's recorded requests — so a failure stays reviewable after the job ends.- Environment variable names listed under
secrets:are masked as***in all reports and snapshots.
explain describes what a spec does, doc generates Markdown (with fixtures, expected payloads, and golden files inlined), manifest emits a stable JSON summary for tooling, and list shows scenarios, tags, and artifacts. All of them load and validate the spec first — exit code 2 on a schema error — so any of them doubles as a lint step in CI:
atago explain spec.atago.yaml
atago doc --out docs/specs.md ./specs
atago manifest ./specs
atago list ./specsScenario names say what happens; an optional description: on the suite and on
a scenario says why it matters. Both are prose for the reader of the generated
Markdown — rendered under the matching heading, never expanded, and with no
effect on the run. Maintenance notes stay in YAML comments, which never appear
in the generated page.
snapshot matchers compare output against committed golden files; ANSI colors, temp paths, UUIDs, timestamps, ports, and CRLF are normalized so snapshots stay stable across machines. Record or refresh them with:
atago snapshot update spec.atago.yamlFor volatile patterns the built-ins do not cover — auto-increment IDs, request identifiers, epoch times — declare spec-wide scrub: rules that rewrite each regex match to a placeholder before the compare (applied after secrets: masking):
scrub:
- {pattern: 'id=\d+', placeholder: 'id=<ID>'}See scrub.
A JSON Schema lives at schema/atago.schema.json. With the YAML language server you get completion and validation as you type — step types, every matcher, and the ${workdir} / ${env:NAME} / ${name} / $${...} expansion rules. atago init and atago record already emit this header as the first line of every generated spec, so scaffolded specs get completion out of the box. To add it to an existing spec, use the absolute URL (it resolves in any project, unlike a repo-relative path):
# yaml-language-server: $schema=https://raw.githubusercontent.com/nao1215/atago/main/schema/atago.schema.json
version: "1"atago completion <bash|zsh|fish|powershell> prints a completion script for your shell.
| Code | Meaning |
|---|---|
0 |
all scenarios passed |
1 |
one or more failed |
2 |
spec error (YAML syntax or schema/semantic validation) |
3 |
CLI-invocation error (unknown subcommand, bad flag, or no matching spec files) |
4 |
execution error |
5 |
internal error |
6 |
security policy violation |
Ctrl-C/SIGTERM stops the run cleanly: in-flight processes, services, and sessions are torn down, partial results are reported, and the run exits 4.
These suites run real programs of every shape: the author's Go tools (atago tests itself) and unmodified third-party binaries — git and jq, interactive TUIs (fzf, htop, yazi), the python3 REPL, servers driven as scenario services (redis, gitea, grafana, prometheus), cloud and IaC CLIs tested offline (aws-cli, terraform, ecspresso), crypto tools (openssl, age, sops), and document/media pipelines (pandoc, ffmpeg). Most were migrated from ShellSpec. The generated real-world index includes deep behavior docs for those binaries, including Yazi file operations, chooser/cwd handoff, tabs, and sort modes.
atago (愛宕) is named for Mount Atago in Kyoto, whose shrine enshrines a deity of fire prevention. A test runner should do the same job: catch the sparks so a project never catches fire.
Issues and pull requests are welcome; see CONTRIBUTING.md. Contributions are not only about code: a GitHub Star also motivates development.
The atago project is licensed under the terms of MIT LICENSE.
Thanks goes to these wonderful people (emoji key):
 CHIKAMATSU Naohiro 💻 📖 |