remote: make remote_setup reliably provision the GPU node

[ykhrustalev]

Problem

lqh remote_setup failed to provision a GPU node end-to-end in two independent ways. Both produced confusing downstream failures rather than a clear error.

1. uv not found when the login shell isn't bash

ssh_helpers.ssh_run wraps every remote command in bash -lc — deliberately, because venv activate, &&-chaining, source, export, and the #!/bin/bash launcher all need bash. But that makes bash's dotfiles the only source of truth for PATH. Astral's uv installer keys off $SHELL: for a fish user it writes ~/.config/fish/conf.d/uv.env.fish and never touches ~/.bashrc. So bash -lc 'command -v uv' found nothing even though uv was installed — and bootstrap silently downgraded to pip (or failed if pip was missing too).

2. Wrong lqh installed on the remote → ModuleNotFoundError: No module named 'lqh.train'

_find_lqh_package_root() only located the local source when lqh was an editable checkout (it required a pyproject.toml next to lqh/). For a normal pip install (lqh in site-packages) it returned None, and bootstrap fell back to pip install lqh[train] from PyPI. lqh on PyPI is an unrelated package — so the remote got the wrong code and python -m lqh.train blew up.

Fix

Find uv wherever it actually is. New _locate_uv tries command -v uv, then probes the canonical install dirs directly (~/.local/bin, ~/.cargo/bin, /snap/bin, /usr/local/bin, /opt/homebrew/bin, /home/linuxbrew/.linuxbrew/bin, /usr/bin). Returns the absolute path, which is threaded into <uv> venv and <uv> pip install so bash never has to resolve uv on PATH.

Always install the lqh that's running locally. The importable lqh/ directory is always available (Path(lqh.__file__).parent). bootstrap_remote now always rsyncs it; for the [train] dependency metadata it uses the checkout's pyproject.toml when present, else synthesizes a minimal one from importlib.metadata. The PyPI fallback is removed entirely — the remote always gets exactly the local code. compute_local_lqh_hash works for non-checkout installs too, so remote_status drift detection no longer reports "pypi".

Harden the bash wrapper. ssh_run now invokes /usr/bin/env bash -lc … instead of bash -lc … — skips shell-level name resolution so a function/alias named bash can't shadow the binary, and works on non-FHS layouts (NixOS).

Deliberately not done

• No login-shell detection. An earlier iteration of this branch added <shell> -lc 'command -v uv' probing. Reviewed and dropped — the canonical-dirs probe already covers every install method in real use, and the shell-aware layer added a fragile MOTD parser and extra round-trips for an edge case nobody hits.
• No python3 path resolution. /usr/bin/python3 is on bash's PATH on every Linux/macOS GPU host; bare python3 -m venv works.

Test plan

• TestLocateUv — PATH hit, /snap/bin fallback, probe covers every canonical location, not-found.
• TestLocalLqhRoot / TestSynthesizePyproject — the importable-root resolver and the synthetic pyproject.toml (built structure, base + train deps).
• TestBootstrapRemote — uv path threaded into <uv> venv / <uv> pip install; python3 -m venv + plain pip when uv is absent; synthesizes pyproject.toml and installs from the synced tree (never PyPI) when there's no checkout.
• TestSSHRun::test_invokes_bash_via_env — pins the /usr/bin/env bash -lc … form.

Full suite: 276 passed, 0 failed, 26 skipped.

Session