fix: two files shipped in 0.3.0 that cannot compile from a tarball

`ytsaurus-skiff/tests/wire.rs` and `ytsaurus-job/tests/skiff_reader_tests.rs`
`include_str!` reference vectors from `tests/skiff-go-interop/`, which is
outside their crates. Those macros resolve at compile time, so both files
compile in this repository and cannot compile from the published `.crate` — the
fixtures are not in the tarball and no consumer can put them there. Verified
against the actual artifacts downloaded from crates.io: of the nine published,
`ytsaurus-skiff` and `ytsaurus-job` are the two whose `cargo test --no-run`
fails.

Consumers are unaffected: cargo does not build a dependency's tests. It bites
whoever unpacks the crate and runs its suite, and vendoring or packaging
workflows that do the same.

Four files of this kind were found and excluded before 0.3.0 went out. These two
were missed, and the reason is worth recording: the search was a line-based
grep for

    include_str!("../../../tests/…

which does not match

    hex_fixture(include_str!(
        "../../../tests/…"
    ))

— the same macro with a newline in it. The audit that caught the other four did
not catch these either.

So `scripts/check-package-includes.sh` replaces the grep. It asks cargo which
files actually ship, parses each with a multi-line-aware regex, resolves every
include target and fails if one escapes its crate root. It runs in CI, and it
was checked to fail: re-including `tests/wire.rs` makes it exit 1 naming the
file and the three fixtures it reaches.

0.3.0 is published and cannot be replaced. This lands the fix so it is correct
from here, and so nothing else ships with it.

Author: sshaplygin

.github/workflows/ci.yml

@@ -86,6 +86,15 @@ jobs:
           cargo xtask generate-protos
           git diff --exit-code -- crates/ytsaurus-proto/src/generated
 
+      # `include_str!`/`include_bytes!` resolve at compile time, so a published
+      # file reaching outside its own crate builds here and cannot build from a
+      # tarball. `cargo package` does not catch it — it verifies by building the
+      # library, not the tests — and 0.3.0 shipped two such files because the
+      # check for them was a line-based grep and the macros had newlines in
+      # them. This parses instead.
+      - name: No published file reaches outside its own crate
+        run: ./scripts/check-package-includes.sh
+
       - name: cargo fmt --check
         run: cargo fmt --all -- --check
 

crates/ytsaurus-job/Cargo.toml

@@ -24,7 +24,7 @@ categories = ["encoding"]
 # time, so shipping that file gives anyone who unpacks the crate and runs
 # `cargo test` a build error and no way to fix it. The other e2e tests build the
 # example workers, which *are* published, and are left in.
-exclude = ["examples/selfrun.rs", "tests/cat_e2e.rs"]
+exclude = ["examples/selfrun.rs", "tests/cat_e2e.rs", "tests/skiff_reader_tests.rs"]
 
 [dependencies]
 ytsaurus-yson.workspace = true

crates/ytsaurus-skiff/Cargo.toml

@@ -19,12 +19,14 @@ readme = "README.md"
 keywords = ["ytsaurus", "skiff", "serialization", "mapreduce"]
 categories = ["encoding"]
 
-# `tests/cpp_interop.rs` `include_str!`s the C++ reference vectors from
-# `tests/skiff-cpp-interop/`, which is outside this package and so is not in the
-# tarball. `include_str!` resolves at compile time, so shipping the file would
-# hand anyone who unpacks the crate a build error they cannot fix. The vectors
-# and the test both live in the repository, where they run.
-exclude = ["tests/cpp_interop.rs"]
+# `tests/cpp_interop.rs` and `tests/wire.rs` `include_str!` reference vectors
+# from `tests/skiff-cpp-interop/` and `tests/skiff-go-interop/`, which are
+# outside this package and so are not in the tarball. `include_str!` resolves at
+# compile time, so shipping either hands anyone who unpacks the crate a build
+# error they cannot fix. The vectors and the tests both live in the repository,
+# where they run. `scripts/check-package-includes.sh` is what keeps this list
+# honest.
+exclude = ["tests/cpp_interop.rs", "tests/wire.rs"]
 
 [dependencies]
 ytsaurus-yson.workspace = true

scripts/check-package-includes.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+#
+# No published file may `include_str!`/`include_bytes!` something outside its own
+# crate.
+#
+# Those macros resolve at **compile time**, so a file that reaches into
+# `tests/skiff-go-interop/` or `tests/cluster-e2e/fixtures/` compiles here and
+# cannot compile from a `.crate` tarball — the fixture is not in it, and no
+# consumer can put it there. `cargo package` does not catch this: it verifies by
+# building the library, not the tests.
+#
+# `ytsaurus-skiff` 0.3.0 and `ytsaurus-job` 0.3.0 shipped with exactly that
+# defect. It was looked for beforehand, with a line-based grep, which matched
+#
+#     include_str!("../../../tests/…")
+#
+# and missed
+#
+#     hex_fixture(include_str!(
+#         "../../../tests/…"
+#     ))
+#
+# — the same macro with a newline in it. Hence a real parse below rather than a
+# pattern, and hence this running in CI rather than being remembered.
+#
+# The fix for a violation is an `exclude` entry in that crate's Cargo.toml, the
+# way `ytsaurus-skiff` and `ytsaurus-job` already carry one.
+
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+python3 - <<'PY'
+import re, subprocess, sys, json
+from pathlib import Path
+
+# Multi-line by construction: `re.S` plus `\s*` across the paren and the string.
+PAT = re.compile(r'include_(?:str|bytes)!\s*\(\s*"([^"]+)"', re.S)
+
+meta = json.loads(subprocess.run(
+    ["cargo", "metadata", "--no-deps", "--format-version", "1"],
+    capture_output=True, text=True, check=True).stdout)
+
+bad = []
+for pkg in meta["packages"]:
+    if pkg.get("publish") == []:            # publish = false
+        continue
+    root = Path(pkg["manifest_path"]).parent
+
+    # Ask cargo what actually ships, so an `exclude`d file is not reported.
+    listed = subprocess.run(
+        ["cargo", "package", "--list", "--allow-dirty", "-p", pkg["name"]],
+        capture_output=True, text=True)
+    if listed.returncode != 0:
+        print(f"warning: could not list package {pkg['name']}", file=sys.stderr)
+        continue
+    shipped = {line.strip() for line in listed.stdout.splitlines() if line.strip()}
+
+    for rel in sorted(shipped):
+        if not rel.endswith(".rs"):
+            continue
+        f = root / rel
+        if not f.is_file():
+            continue
+        try:
+            src = f.read_text()
+        except (OSError, UnicodeDecodeError):
+            continue
+        for m in PAT.finditer(src):
+            target = (f.parent / m.group(1)).resolve()
+            try:
+                target.relative_to(root.resolve())
+            except ValueError:
+                bad.append((pkg["name"], rel, m.group(1)))
+
+if bad:
+    print("ERROR: published files include data from outside their own crate.")
+    print("These compile here and cannot compile from a crates.io tarball.\n")
+    for name, rel, target in bad:
+        print(f"  {name}: {rel}")
+        print(f"      -> {target}")
+    print("\nAdd the file to that crate's `exclude` in Cargo.toml.")
+    sys.exit(1)
+
+print("OK: no published file reaches outside its own crate")
+PY