xtest: Record and Replay GUI Macros in Docker (Firefox example)

This project runs a GUI app in a Docker container on an isolated X server (Xvfb), exposes it through VNC/noVNC, and records/replays mouse/keyboard automation against the app window.

The included example is Firefox (docker-compose-firefox.yml), but the recording/replay flow is generic and can be reused for other GUI apps by changing compose environment values.

What Is Included

• Containerized GUI stack: Xvfb + window manager + x11vnc + noVNC
• Host-side recorder: record-macro.py
• Host-side replayer: replay-macro.py
• Timed macro converter/player: timed_xmacro.py
• Screenshot assertions during replay via Check ... lines (check-image.sh)
• #APP metadata stored inside recordings so replay can find/start the right window

Quick Start (Firefox)

Start the container:

docker compose -f docker-compose-firefox.yml up --build

Open the GUI in your browser:

• http://localhost:6080/vnc.html

The compose file starts Firefox and exposes the isolated display through noVNC.

Record a Macro

Record against the Firefox compose file:

./record-macro.py docker-compose-firefox.yml

During recording:

1. Open noVNC (http://localhost:6080/vnc.html)
2. Interact with the app
3. Press Pause to stop recording (default stop key)
4. Press F2 anytime to capture a screenshot and insert a Check ... line (default screenshot/check key)

Default hotkeys:

• STOP_KEYSYM=Pause
• CHECK_KEYSYM=F2

Override if your browser/keyboard/noVNC intercepts them:

STOP_KEYSYM=F9 CHECK_KEYSYM=F3 ./record-macro.py docker-compose-firefox.yml

Notes:

• record-macro.py auto-arms the recorder and injects the stop key for xmacrorec2.
• CHECK_KEYSYM and STOP_KEYSYM must be different.
• F2 is the default to avoid Firefox focus behavior triggered by F6.

Output Files

For docker-compose-firefox.yml, the derived run name is firefox, so outputs go to:

• recordings/firefox/firefox.xmacro
• recordings/firefox/firefox-check-001.png
• recordings/firefox/firefox-check-002.png

Replay a Macro

Replay the recorded macro:

./replay-macro.py docker-compose-firefox.yml

Optional speed override:

REPLAY_SPEED=2.0 ./replay-macro.py docker-compose-firefox.yml
REPLAY_SPEED=0.5 ./replay-macro.py docker-compose-firefox.yml

replay-macro.py:

• Loads recordings/<run-name>/<run-name>.xmacro
• Reads #APP metadata from the recording
• Focuses/positions the app window in the container
• Replays actions using xdotool
• Executes screenshot checks when it encounters Check ... lines

If a Check fails, replay exits non-zero.

#APP Metadata in Macros

Recorded macros include metadata lines like:

#APP startcommand='firefox https://browserbench.org/Speedometer3.1/ --no-default-browser-check'
#APP windowtitle=Firefox
#APP windowclass=firefox

These are used by replay (and screenshot checks) to find the app window by class/title and optionally start the app if the window is missing.

Supported fields:

• startcommand (optional): shell command to launch inside the container
• windowtitle: matcher for xdotool search --name
• windowclass: matcher for xdotool search --class

Recording-Time Metadata Source

record-macro.py reads defaults from the compose file environment:

• APP_STARTCOMMAND
• APP_WINDOW_TITLE
• APP_WINDOW_CLASS

You can override them at record time with host environment variables:

APP_STARTCOMMAND='xterm' \
APP_WINDOW_TITLE='xterm' \
APP_WINDOW_CLASS='xterm' \
./record-macro.py docker-compose-firefox.yml

Check Screenshot Assertions

During recording, pressing F2 inserts a line like:

Check firefox/firefox-check-001.png

During replay, the container script:

1. Finds the app window (using #APP class/title metadata)
2. Captures the current window image
3. Compares it to the reference image
4. Fails replay if the RMSE exceeds the threshold

Tuning Image Comparison

Default threshold:

• CHECK_MAX_RMSE=0.01

Example override:

CHECK_MAX_RMSE=0.02 ./replay-macro.py docker-compose-firefox.yml

Ignore dynamic regions (toolbars, timestamps, animations):

CHECK_IGNORE_RECT=0,0,420,40 ./replay-macro.py docker-compose-firefox.yml

Multiple rectangles (semicolon-separated):

CHECK_IGNORE_RECT="0,0,420,40;300,580,120,30" ./replay-macro.py docker-compose-firefox.yml

Rectangle format is x,y,width,height.

Macro Format Notes

The recorded .xmacro files are timed macros generated by timed_xmacro.py and include:

• #WAIT_SEC <seconds> comments between events
• #APP ... metadata lines
• xmacro events (MotionNotify, ButtonPress, KeyStrPress, etc.)
• Check <path>.png assertion lines

Window Positioning / Deterministic Layout

The container can force the app window to a fixed size/position before replay/checking. Configure in compose environment:

• AUTO_POSITION (1 or 0)
• WINDOW_X
• WINDOW_Y
• WINDOW_WIDTH
• WINDOW_HEIGHT
• APP_WINDOW_CLASS
• APP_WINDOW_TITLE

This helps keep screenshots and click coordinates stable.

Useful Low-Level Tool (timed_xmacro.py)

timed_xmacro.py also exposes subcommands directly:

• record: read xmacrorec2 output and write timed macro
• replay: emit timed xmacro lines to stdout
• replay-xdotool: emit normalized xdotool actions to stdout
• app-meta: read effective #APP metadata from a macro file

Example:

python3 timed_xmacro.py app-meta --input recordings/firefox/firefox.xmacro --format tsv

Troubleshooting

• If the stop key does not work in noVNC/browser, set STOP_KEYSYM to another key (for example F9).
• If screenshot/check hotkey affects app UI (Firefox F6 focuses toolbar), use a different CHECK_KEYSYM. The default is F2.
• If screenshot checks fail due to minor UI variation, increase CHECK_MAX_RMSE slightly or mask dynamic regions with CHECK_IGNORE_RECT.
• If window geometry or theme changes move UI elements, rebuild/restart and re-record the macro.

Security / Demo Caveats

• X access control is disabled (Xvfb -ac) for local convenience.
• VNC/noVNC is exposed without a password in this demo setup.
• Use only on a trusted local machine/network unless you harden the configuration.