Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,11 @@ yt map './my_job' --src //tmp/in --dst //tmp/out \
--local-file target/x86_64-unknown-linux-musl/release-worker/my_job
```

## Or let the binary launch itself
## Or let a static binary launch itself

The cluster starts a job with `YT_JOB_ID` in its environment, so one binary can
be both the launcher and the job — and upload *itself*, which means the cluster
can never be running a stale worker:
The cluster starts a job with `YT_JOB_ID` in its environment, so a static Linux
x86-64 binary can be both the launcher and the job — and upload *itself*, which
means the cluster can never be running a stale worker:

```rust
fn main() {
Expand All @@ -84,9 +84,11 @@ fn main() {
}
```

If it fails, the error carries the job's own stderr rather than a state string.
See [examples/src/bin/selfrun.rs](examples/src/bin/selfrun.rs); the full
walkthrough is [docs/writing-a-job.md](docs/writing-a-job.md).
If the launcher comes from `cargo run`, build a static worker separately and
set `YT_WORKER_BINARY` so it uploads that artifact; rebuild the worker whenever
its source changes. If it fails, the error carries the job's own stderr rather
than a state string. See [examples/src/bin/selfrun.rs](examples/src/bin/selfrun.rs);
the full walkthrough is [docs/writing-a-job.md](docs/writing-a-job.md).

## Build and test

Expand Down
22 changes: 12 additions & 10 deletions crates/ytsaurus-client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,10 +256,11 @@ waits forever without a word.
[`examples/cypress.rs`](examples/cypress.rs) runs all of it, ending with three
transactions competing for one lock.

## One binary, two roles
## One static binary, two roles

`upload_current_exe` uploads the *running* executable, so the same program can
launch the operation and be the job it runs:
When the running executable is a static Linux x86-64 binary,
`upload_current_exe` uploads it, so the same program can launch the operation
and be the job it runs:

```rust
fn main() {
Expand All @@ -268,13 +269,14 @@ fn main() {
}
```

There is no second artifact to forget to rebuild. The running executable has to
be something a node can exec, so its ELF header is checked first — Linux,
x86-64, statically linked — and refused with `ClientError::NotAWorker` when it
is not, rather than failing on the node minutes later. On macOS the launcher is
Mach-O and cannot be the uploaded file: build the worker with
`scripts/build-worker.sh` and upload that with `upload_worker`. The source is
still one file — see
That direct-static path has no second artifact to forget to rebuild. The running
executable has to be something a node can exec, so its ELF header is checked
first — Linux, x86-64, statically linked — and refused with
`ClientError::NotAWorker` when it is not, rather than failing on the node
minutes later. The default launcher built with `cargo run` is Mach-O on macOS
or normally dynamically linked on Linux, so it cannot be the uploaded file.
Build the worker with `scripts/build-worker.sh` and upload that with
`upload_worker`. The source is still one file — see
[`examples/src/bin/selfrun.rs`](../../examples/src/bin/selfrun.rs).

## Talking to a real installation
Expand Down
6 changes: 4 additions & 2 deletions crates/ytsaurus-job/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,10 @@ writer.finish()
}
```

With `ytsaurus-client`'s `upload_current_exe`, the binary uploads itself —
there is no second artifact to forget to rebuild.
When that launcher is a static Linux x86-64 binary,
`ytsaurus-client`'s `upload_current_exe` uploads itself — there is no second
artifact to forget to rebuild. A `cargo run` launcher instead needs a
separately built static worker.

## Design notes

Expand Down
4 changes: 3 additions & 1 deletion docs/go-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,9 @@ cargo run -p ytsaurus-client --example table_usage
cargo run -p ytsaurus-client --example schema
cargo run -p ytsaurus-client --example sort_reduce
cargo run -p ytsaurus-client --example vanilla
cargo run -p ytsaurus-examples --bin selfrun # on Linux; see the README on macOS
# `cargo run` is the host launcher; upload the static worker built above.
YT_WORKER_BINARY=target/x86_64-unknown-linux-musl/release-worker/selfrun \
cargo run -p ytsaurus-examples --bin selfrun
```

[`tests/e2e/README.md`](../tests/e2e/README.md) holds the output of each, from
Expand Down
33 changes: 18 additions & 15 deletions docs/writing-a-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ From an empty file to a running operation. Assumes you can reach a cluster —
`export YT_PROXY=http://localhost:8000` for the local one in
[`tests/e2e/README.md`](../tests/e2e/README.md).

The shape this guide builds towards is **one binary that is both the launcher
and the job**: it uploads itself, starts the operation, and is what the cluster
runs. The `yt` CLI can do the launching instead, and §5 covers that too.
The direct-static shape this guide builds towards is **one binary that is both
the launcher and the job**: it uploads itself, starts the operation, and is
what the cluster runs. A `cargo run` launcher uploads a separately built static
worker from the same source; the `yt` CLI can do the launching instead, and §5
covers that too.

## 0. What a job actually is

Expand Down Expand Up @@ -181,8 +183,8 @@ fn main() {
fn launch() -> Result<(), ytsaurus_client::ClientError> {
let client = ytsaurus_client::Client::from_env()?;

// Uploads *this very binary*, so what runs on the cluster is what you
// just built.
// Uploads this static binary, so what runs on the cluster is what you just
// built.
client.upload_current_exe("//tmp/my_job")?;

let spec = ytsaurus_client::MapSpec::new("./my_job", ["//tmp/input"], ["//tmp/output"])
Expand All @@ -194,9 +196,9 @@ fn launch() -> Result<(), ytsaurus_client::ClientError> {
}
```

That is the whole pattern, and it removes a whole class of bug: there is no
second artifact to forget to rebuild, so "the cluster is running last week's
worker" cannot happen.
For a static launcher, that is the whole pattern, and it removes a whole class
of bug: there is no second artifact to forget to rebuild, so "the cluster is
running last week's worker" cannot happen.

`upload_current_exe` checks the running executable's ELF header before
uploading — Linux, x86-64, statically linked — because everything it rejects
Expand All @@ -208,13 +210,14 @@ cause:
so a Linux node cannot exec it. Build the worker with scripts/build-worker.sh …
```

**On macOS the launcher cannot be the uploaded file.** A Mach-O binary is not
something a node can exec, and a Linux binary is not something macOS can run.
The source stays one file; you build it twice and point the launcher at the musl
build:
**The default launcher built with `cargo run` cannot be the uploaded file.** On
macOS it is Mach-O; on a typical Linux host it is dynamically linked. A cluster
node can run neither. The source stays one file; build the static musl worker
and point the host launcher at it:

```sh
scripts/build-worker.sh my_job
# Rebuild this worker whenever its source changes.
YT_WORKER_BINARY=target/x86_64-unknown-linux-musl/release-worker/my_job \
cargo run -p ytsaurus-examples --bin my_job
```
Expand All @@ -228,9 +231,9 @@ match std::env::var("YT_WORKER_BINARY") {
}
```

On Linux x86-64, run the musl build itself and `upload_current_exe` needs no
help. [`examples/src/bin/selfrun.rs`](../examples/src/bin/selfrun.rs) is the
runnable version of all of this.
On Linux x86-64, you can instead run the musl build itself, and
`upload_current_exe` needs no help. [`examples/src/bin/selfrun.rs`](../examples/src/bin/selfrun.rs)
is the runnable version of both forms.

### Uploading it only when it changed

Expand Down
38 changes: 20 additions & 18 deletions examples/src/bin/selfrun.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//! `selfrun` — one binary that is both the launcher and the job.
//! `selfrun` — one Rust source, two roles.
//!
//! The cluster starts a job by exec'ing an uploaded binary with `YT_JOB_ID` in
//! its environment. So a program can ask which role it is playing, and be both:
Expand All @@ -8,28 +8,30 @@
//! ──────────── ──────────────
//! selfrun ./selfrun
//! is_inside_job() -> false is_inside_job() -> true
//! uploads *itself* ──────────────────► maps rows
//! uploads the worker ─────────────────► maps rows
//! starts the operation writes the output table
//! waits, reads the result back
//! ```
//!
//! The binary that runs on the cluster is the one you just built, because it is
//! the same file. There is no second artifact to forget to rebuild.
//! A static Linux x86-64 build can be both the launcher and the job. When
//! `cargo run` is the launcher, it instead builds a host executable (normally a
//! dynamically linked debug binary). Build the static worker separately and
//! tell the launcher to upload it; both artifacts still come from this source.
//!
//! ```sh
//! export YT_PROXY=http://localhost:8000
//! scripts/build-worker.sh selfrun # the binary the cluster will run
//! cargo run -p ytsaurus-examples --bin selfrun
//! YT_WORKER_BINARY=target/x86_64-unknown-linux-musl/release-worker/selfrun \\
//! cargo run -p ytsaurus-examples --bin selfrun
//! ```
//!
//! ## Two builds on macOS
//! ## Running the static build directly
//!
//! `upload_current_exe` uploads the running executable, which works when the
//! launcher is itself a Linux x86-64 static binary — build it with
//! `scripts/build-worker.sh` and run *that*. On macOS the running executable is
//! Mach-O, which no node can exec, and the client refuses it by inspecting the
//! ELF header rather than letting the job fail later. Set `YT_WORKER_BINARY` to
//! the musl build in that case; the source is still one file.
//! `upload_current_exe` also works when the launcher is itself the Linux x86-64
//! static binary produced by `scripts/build-worker.sh`. That is an option on a
//! Linux x86-64 host; macOS cannot run the Linux binary, and a normal Linux
//! `cargo run` build is dynamically linked. In either case the client checks
//! the ELF header before uploading instead of letting the job fail later.
//!
//! The mapper counts hosts: `{url, size}` in, `{host, size}` out.

Expand All @@ -41,8 +43,8 @@ use ytsaurus_yson::{YsonFormat, to_vec};
/// Where the demo keeps its tables, and the worker.
const BASE: &str = "//tmp/ytsaurus_rs_selfrun";

/// Points at a cross-compiled build of this same source, for hosts that cannot
/// run a Linux binary themselves.
/// Points at the static worker built from this source when the running launcher
/// is not itself a Linux x86-64 static binary.
const WORKER_OVERRIDE: &str = "YT_WORKER_BINARY";

#[derive(Deserialize)]
Expand Down Expand Up @@ -109,11 +111,11 @@ fn launch() -> Result<(), ClientError> {
client.row_count(&format!("{BASE}/input"))?
));

step("Uploading this very binary");
step("Uploading the worker");
let remote = format!("{BASE}/selfrun");
match std::env::var(WORKER_OVERRIDE) {
// A cross-compiled build of this same source, for a host whose own
// binaries a cluster node cannot run.
// A static build of this same source, for a launcher a cluster node
// cannot run (including a normal `cargo run` build on Linux).
Ok(path) if !path.trim().is_empty() => {
client.upload_worker(&path, &remote)?;
done(&format!("{path} -> {remote}"));
Expand Down Expand Up @@ -146,7 +148,7 @@ fn launch() -> Result<(), ClientError> {
let output = client.read_table(format!("{BASE}/output"))?;
done(&format!("{rows} rows, {} bytes", output.len()));

println!("\nOne binary, two roles. Output at {BASE}/output");
println!("\nOne source, two roles. Output at {BASE}/output");
Ok(())
}

Expand Down
11 changes: 5 additions & 6 deletions tests/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ cargo run -p ytsaurus-client --example raw # commands the crate does no
cargo run --release -p ytsaurus-client --example streaming # a table bigger than the program
cargo run --release -p ytsaurus-client --example profile # what the pilot spends on decoding

# One binary that is both launcher and job. On macOS the launcher cannot be the
# uploaded file, so point it at the musl build of the same source.
# One source, two build outputs: `cargo run` makes a host launcher on every
# platform, so point it at the static musl worker built above.
YT_WORKER_BINARY=target/x86_64-unknown-linux-musl/release-worker/selfrun \
cargo run -p ytsaurus-examples --bin selfrun
```
Expand Down Expand Up @@ -168,17 +168,16 @@ refused before it can be uploaded:
so a Linux node cannot exec it. Build the worker with scripts/build-worker.sh …
```

and the real one-binary path — the binary uploading *itself* — was verified by
running the musl build as the launcher inside Linux, which is what a Linux
developer's machine would do:
The optional direct-static path — the binary uploading *itself* — was also
verified by running the musl build as the launcher inside Linux:

```sh
docker cp target/x86_64-unknown-linux-musl/release-worker/selfrun yt.backend:/tmp/selfrun
docker exec -e YT_PROXY=http://localhost:80 yt.backend /tmp/selfrun
```

```text
== Uploading this very binary
== Uploading the worker
ok /tmp/selfrun -> //tmp/ytsaurus_rs_selfrun/selfrun
== Waiting for it
ok completed
Expand Down
Loading