Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 15 additions & 17 deletions .github/actions/setup-uv-python/action.yml
Original file line number Diff line number Diff line change
@@ -1,33 +1,31 @@
name: "Setup uv & Python"
description: "Installs uv, sets up Python via uv, and configures PYO3_PYTHON environment variable"
description: "Installs a UV-managed Python for PyO3 builds"
inputs:
python-version:
description: 'Python version to install (e.g., "3.12")'
required: true
default: "3.12"
outputs:
python-path:
description: "Path to the uv-managed Python interpreter used for PyO3"
value: ${{ steps.python.outputs.python-path }}
resolved-version:
description: "Full Python version reported by the selected interpreter"
value: ${{ steps.python.outputs.resolved-version }}
python-abi-version:
description: "Major.minor Python ABI version used in libpython names"
value: ${{ steps.python.outputs.python-abi-version }}
python-prefix:
description: "Prefix of the uv-managed Python installation"
description: "Prefix of the UV-managed Python installation"
value: ${{ steps.python.outputs.python-prefix }}
runs:
using: "composite"
steps:
- name: Install uv
uses: astral-sh/setup-uv@v3
- name: Install Python via uv and set PYO3_PYTHON
- name: Install managed Python
id: python
shell: bash
run: bash scripts/ci_setup_uv_python.sh "${{ inputs.python-version }}"
- name: Fix libpython install name for portable builds
if: runner.os == 'macOS'
shell: bash
run: bash scripts/ci_fix_libpython_install_name.sh "${{ steps.python.outputs.python-prefix }}" "${{ steps.python.outputs.python-abi-version }}"
env:
REQUESTED_PYTHON: ${{ inputs.python-version }}
run: |
set -euo pipefail
uv python install --no-config --managed-python "${REQUESTED_PYTHON}"
python="$(uv python find --no-config --no-project --managed-python "${REQUESTED_PYTHON}")"
prefix="$(dirname "$(dirname "${python}")")"
{
printf 'PYO3_PYTHON=%s\n' "${python}"
printf 'R2X_PYTHON_VERSION=%s\n' "${REQUESTED_PYTHON}"
} >> "${GITHUB_ENV}"
printf 'python-prefix=%s\n' "${prefix}" >> "${GITHUB_OUTPUT}"
4 changes: 0 additions & 4 deletions .github/build-setup.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
# Build setup steps for r2x CI
# These steps are used as a reference for setting up the build environment.

- name: Setup uv & Python
uses: ./.github/actions/setup-uv-python
with:
python-version: ${{ env.R2X_PYTHON_VERSION }}
- name: PyO3 diagnostics
shell: bash
run: bash scripts/ci_pyo3_diagnostics.sh
23 changes: 4 additions & 19 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Setup Python
uses: actions/setup-python@v6
- name: Setup uv & Python
uses: ./.github/actions/setup-uv-python
with:
python-version: "3.12"
- name: Install ShellCheck
Expand All @@ -31,15 +31,9 @@ jobs:
run: bash scripts/ci_check_shell_scripts.sh
- name: Python script tests
run: |
PYTHONPATH=scripts python3 -m unittest \
scripts.tests.test_python_version_sh \
scripts.tests.test_resolve_pyo3_python \
scripts.tests.test_ci_python_version \
PYTHONPATH=scripts uv run --no-config --no-project --managed-python --python 3.12 -- python -m unittest \
scripts.tests.test_format_benchmark_summary \
scripts.tests.test_compare_benchmark_summary \
scripts.tests.test_patch_dist_installer \
scripts.tests.test_detect_uv_python \
scripts.tests.test_fix_python_dylib
scripts.tests.test_compare_benchmark_summary

lint:
name: Lint (stable, Python 3.12)
Expand All @@ -56,9 +50,6 @@ jobs:
uses: ./.github/actions/setup-uv-python
with:
python-version: "3.12"
- name: PyO3 diagnostics
shell: bash
run: bash scripts/ci_pyo3_diagnostics.sh
- name: Format check
run: cargo fmt --all -- --check
- name: Clippy
Expand Down Expand Up @@ -120,9 +111,6 @@ jobs:
uses: ./.github/actions/setup-uv-python
with:
python-version: ${{ matrix.python-version }}
- name: PyO3 diagnostics
shell: bash
run: bash scripts/ci_pyo3_diagnostics.sh
- name: Test
env:
PYTHON_PREFIX: ${{ steps.python.outputs.python-prefix }}
Expand Down Expand Up @@ -171,8 +159,5 @@ jobs:
uses: ./.github/actions/setup-uv-python
with:
python-version: ${{ env.R2X_PYTHON_VERSION }}
- name: PyO3 diagnostics
shell: bash
run: bash scripts/ci_pyo3_diagnostics.sh
- name: Cargo publish dry-run
run: cargo publish --workspace --dry-run
9 changes: 1 addition & 8 deletions .github/workflows/release-plz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,7 @@ jobs:
group: release-plz-pr-${{ github.ref }}
cancel-in-progress: true
steps:
- &checkout
name: Checkout repository
- name: Checkout repository
uses: actions/checkout@v7
with:
fetch-depth: 0
Expand All @@ -52,11 +51,6 @@ jobs:
with:
python-version: ${{ env.R2X_PYTHON_VERSION }}

- &pyo3_diagnostics
name: PyO3 diagnostics
shell: bash
run: bash scripts/ci_pyo3_diagnostics.sh

- name: Run release-plz (release-pr)
uses: release-plz/action@v0.5
with:
Expand Down Expand Up @@ -89,7 +83,6 @@ jobs:
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.ref || github.ref_name }}
- *setup_rust
- *setup_python
- *pyo3_diagnostics
- name: Run release-plz (release)
uses: release-plz/action@v0.5
with:
Expand Down
39 changes: 4 additions & 35 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,6 @@ jobs:
with:
python-version: ${{ env.R2X_PYTHON_VERSION }}

- name: PyO3 diagnostics
shell: bash
env:
CARGO_BUILD_TARGET: ${{ join(matrix.targets, ' ') }}
run: bash scripts/ci_pyo3_diagnostics.sh

- name: Install Rust toolchain
if: ${{ matrix.container }}
uses: actions-rust-lang/setup-rust-toolchain@v1
Expand All @@ -176,7 +170,7 @@ jobs:
run: |
${{ matrix.packages_install }}

- name: Build and fix Python library paths (container)
- name: Build runtime binaries (container)
if: ${{ matrix.container }}
shell: bash
env:
Expand All @@ -188,29 +182,19 @@ jobs:
echo "Building dist-profile binaries first..."
TARGETS="${{ join(matrix.targets, ' ') }}"
for target in $TARGETS; do
cargo build --profile dist -p r2x --target "$target"
cargo build --profile dist -p r2x --bins --target "$target"
done

echo "Fixing Python library paths..."
while IFS= read -r -d '' bin; do
./scripts/fix_python_dylib.sh "$bin"
done < <(find target -path "*/dist/r2x" -type f -print0)

- name: Build and fix Python library paths (host)
- name: Build runtime binaries (host)
if: ${{ !matrix.container && runner.os != 'Windows' }}
shell: bash
run: |
echo "Building dist-profile binaries first..."
TARGETS="${{ join(matrix.targets, ' ') }}"
for target in $TARGETS; do
cargo build --profile dist -p r2x --target "$target"
cargo build --profile dist -p r2x --bins --target "$target"
done

echo "Fixing Python library paths..."
while IFS= read -r -d '' bin; do
./scripts/fix_python_dylib.sh "$bin"
done < <(find target -path "*/dist/r2x" -type f -print0)

- name: Build artifacts (container)
if: ${{ matrix.container }}
shell: bash
Expand Down Expand Up @@ -351,21 +335,6 @@ jobs:

cp dist-manifest.json "$BUILD_MANIFEST_NAME"

- name: Patch shell installer for Python runtime bootstrap
shell: bash
run: |
shopt -s nullglob
installers=(target/distrib/*-installer.sh)

if [ ${#installers[@]} -eq 0 ]; then
echo "No shell installer artifacts found in target/distrib"
exit 1
fi

for installer in "${installers[@]}"; do
./scripts/patch_dist_installer.sh "$installer"
done

- name: "Upload artifacts"
uses: actions/upload-artifact@v7
with:
Expand Down
81 changes: 43 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,13 @@ r2x --version
```

> [!NOTE]
> Pre-built binaries require Python shared libraries at runtime.
> If `r2x --version` fails with a missing `libpython` error, run
> `uv python install 3.12` to make the shared library available.
> Source builds can target supported Python versions by setting
> `R2X_PYTHON_VERSION=<version>` at build time.
> On first run, R2X uses UV to provision its managed CPython runtime and
> creates the R2X virtual environment. It does not require `python3.12` in
> `PATH` or `~/.local/bin`. UV owns the CPython installation; R2X owns only
> the plugin virtual environment, which references that interpreter. Use
> `uv python list` to inspect UV's available interpreters.
> Source builds select a supported version with `PYO3_PYTHON` from
> `uv python find --managed-python` and `R2X_PYTHON_VERSION`.

## Upgrading

Expand Down Expand Up @@ -203,16 +205,19 @@ r2x run plugin r2x-reeds.reeds-parser --show-help
r2x run r2x-reeds.reeds-parser --repeat 10 --benchmark solve_year=2030

# Compare two benchmark outputs (baseline vs current)
python3 scripts/compare_benchmark_summary.py --baseline baseline.txt --current current.txt
uv run --no-config --no-project --managed-python --python 3.12 -- \
python scripts/compare_benchmark_summary.py --baseline baseline.txt --current current.txt

# Emit machine-readable status line in stderr
python3 scripts/compare_benchmark_summary.py \
uv run --no-config --no-project --managed-python --python 3.12 -- \
python scripts/compare_benchmark_summary.py \
--baseline baseline.txt \
--current current.txt \
--print-status-line

# Fail when regression exceeds 15%
python3 scripts/compare_benchmark_summary.py \
uv run --no-config --no-project --managed-python --python 3.12 -- \
python scripts/compare_benchmark_summary.py \
--baseline baseline.txt \
--current current.txt \
--fail-on-regression-pct 15
Expand Down Expand Up @@ -320,23 +325,27 @@ r2x read system.json --exec script.py -i
r2x config show

# Set values
r2x config set python-version 3.13
r2x config set cache-path /path/to/cache

# Reset everything
r2x config reset -y
```

R2X uses the Python major.minor ABI that its PyO3 runtime was built against.
Use `uv python list` and `uv python install` to inspect or manage interpreter
installations directly. A configured Python patch version must use that same
major.minor ABI.

<details>
<summary>Python and virtual environment management</summary>

The `r2x python` command provides shortcuts for Python runtime management:
The `r2x python` command manages R2X's UV-backed plugin environment:

```bash
# Install a Python version
r2x python install 3.13
# Create or refresh the managed venv with UV
r2x python install

# Show installed Python versions
# Show the R2X venv's Python configuration
r2x python show

# Get the Python executable path
Expand Down Expand Up @@ -483,25 +492,24 @@ uv python install 3.12

```bash
git clone https://github.com/NatLabRockies/r2x-cli && cd r2x-cli
R2X_PYTHON_VERSION=3.12 cargo install --path crates/r2x-cli --force --locked
PYO3_PYTHON="$(uv python find --managed-python 3.12)" \
R2X_PYTHON_VERSION=3.12 cargo install --path crates/r2x-cli --bins --force --locked
```

To build against another supported Python version:

```bash
uv python install 3.13
R2X_PYTHON_VERSION=3.13 cargo install --path crates/r2x-cli --force --locked
PYO3_PYTHON="$(uv python find --managed-python 3.13)" \
R2X_PYTHON_VERSION=3.13 cargo install --path crates/r2x-cli --bins --force --locked
```

`cargo` now resolves `R2X_PYTHON_VERSION` through `uv python find` automatically.
For patch requests (for example `3.13.1`), r2x falls back to the matching ABI
request (`3.13`) if the exact patch is unavailable.
If the requested interpreter is missing, the build fails with
fallback-aware guidance such as
`uv python install <requested> || uv python install <abi>` and
`uv python find <requested> || uv python find <abi>`.
Set `PYO3_PYTHON` from `uv python find --managed-python` so PyO3 builds
against the UV-managed interpreter. R2X passes the requested version directly
to UV. Install that exact version first with `uv python install <version>`.

This places the `r2x` binary in `~/.cargo/bin/`.
This places `r2x` and its adjacent `r2x-runtime` payload in `~/.cargo/bin/`.
Run `r2x`; keep both files together when moving the installation.

```bash
r2x --version
Expand All @@ -511,25 +519,23 @@ r2x --version
<summary>Manual build (custom install path)</summary>

```bash
R2X_PYTHON_VERSION=3.12 cargo build --release
PYO3_PYTHON="$(uv python find --managed-python 3.12)" \
R2X_PYTHON_VERSION=3.12 cargo build --release -p r2x --bins
```

The binary lands at `target/release/r2x`. Copy it wherever you
like.
The build produces `target/release/r2x` and `target/release/r2x-runtime`.
Copy them to the same directory, then invoke `r2x`.

The justfile also honors `R2X_PYTHON_VERSION`, for example
`R2X_PYTHON_VERSION=3.13 just test`.

`R2X_PYTHON_VERSION` and `r2x config set python-version` accept major.minor
versions such as `3.13` and patch versions such as `3.13.1`. r2x uses the
requested version for uv, then uses the matching major.minor ABI version for
`libpython` paths.
`R2X_PYTHON_VERSION` selects the Python ABI for a source build. The installed
CLI accepts only the same major.minor ABI for `r2x config set python-version`;
a patch version such as `3.12.1` is allowed for a binary built against `3.12`.

When `R2X_PYTHON_VERSION` is set for a just task, an interpreter with the same
major.minor ABI must exist; install it first with `uv python install <version>`
(or for patch requests, `uv python install <patch> || uv python install <major.minor>`).
This avoids accidentally building PyO3 against a different Python ABI than the
one requested.
When `R2X_PYTHON_VERSION` is set for a just task, install that version first
with `uv python install <version>`. This avoids accidentally building PyO3
against a different Python ABI than the one requested.
If you also set `PYO3_PYTHON` manually, it must point to an interpreter with the
same major.minor ABI as `R2X_PYTHON_VERSION`.

Expand All @@ -554,9 +560,8 @@ just all # fmt + clippy + test

- If the build fails with a Python error, verify
`R2X_PYTHON_VERSION` is set to a supported version (for example `3.12`,
`3.13`, or `3.13.1`) and `uv python find <version>` returns a valid path.
You may need `uv python install <version>` first, or for patch requests,
`uv python install <patch> || uv python install <major.minor>`.
`3.13`, or `3.13.1`) and `uv python find --managed-python <version>`
returns a valid path. You may need `uv python install <version>` first.
- If `r2x` is not found after install, check that `~/.cargo/bin`
is in your `$PATH`.
- On HPC systems with older glibc, building from source is
Expand Down
Loading
Loading