dataforge: msd dataset — remote discovery, on-demand AV1 convert, raw budget (msd 2/3) - #223
Open
pablo-rerun wants to merge 4 commits into
Open
dataforge: msd dataset — remote discovery, on-demand AV1 convert, raw budget (msd 2/3)#223pablo-rerun wants to merge 4 commits into
pablo-rerun wants to merge 4 commits into
Conversation
Three Monado SLAM sequences ship as Info-ZIP spanned archives (`<seq>.z01 … <seq>.zip`). Python's zipfile opens the closing volume — its central directory is intact — and then fails on the first member whose data crosses a volume boundary, so it cannot read them at all. conda-forge's `7zip` package provides `7zz`, which can. Re-locked on this aarch64 host; the diff touches only the two 7zip records plus one `purls` metadata normalization pixi 0.77 applies to pytorch-cpu. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The three readers a Monado SLAM sequence needs before anything can be logged: * `calibration.json` in basalt's format, as pyserde dataclasses. `T_imu_cam` is the camera's pose in the IMU frame, and MSD's rig frame *is* that frame, so it goes into simplecv's `Extrinsics` as `world_R_cam`/`world_t_cam` — the RoboCap loader's convention, one inversion away (Kalibr says `T_cam_imu`). `kb4` becomes a `Fisheye62Parameters`, `pinhole-radtan8` a `PinholeParameters`; radtan8's `rpmax` has no home in Rerun and pyserde drops it with every other unknown key. * The csvs. Camera indices carry a filename, so they are typed rows; the inertial ones run at 1 kHz, where an hour is 3.6M rows and a pyserde row costs 9.8 s and 2.1 GB of live objects against `numpy.loadtxt`'s 0.8 s and one array. Timestamps get their own int64 pass rather than a cast down from the float table, so no stamp can be rounded. * The archive. A plain `.zip` is stdlib; the `*_long_session` sequences ship as Info-ZIP spanned sets, which `zipfile` opens and then fails to read, so those go through `7zz` — one camera directory extracted, yielded, and deleted at a time, keeping peak scratch at one camera rather than the whole sequence. The split-archive test builds a real multi-volume fixture with Info-ZIP's `zip -s`, and first asserts that `zipfile` really cannot read it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…eprints MSD is ~350 GB of PNG archives, so nothing is kept: `discover()` enumerates the *remote* HuggingFace tree (one listing call per collection, isolated so a test can hand it a literal tree), `download()` fetches only the few-kilobyte device calibration and proves the machine has `av1_nvenc` before any of it, and `convert()` pulls exactly one sequence, streams its PNGs into the encoder, writes the rrd, and deletes the archive again. One headset is one dataset (`msd-index`, `msd-g2`, `msd-odyssey`): a catalog dataset holds one default blueprint and the three have two, four and two cameras. `--raw-budget-gb` caps the scratch directory so a batch run cannot fill a disk; a sequence whose archives alone exceed it — the Index and G2 long sessions at 66 and 55 GB — warns and converts anyway, while leftovers from an earlier failure that would breach it are an error naming the files. A failed convert keeps the archive (a retry then skips the download) and clears the scratch. `video_time` is the device clock minus `t0`, and `t0` is the earliest sample of any stream *including* `gt` — the gt layer is a sibling rrd and both must share the origin, which is the only reason the base converter opens that csv. No `ViewCoordinates` at the root, for the same reason RoboCap has none: the layer that establishes a world frame owns it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`msd` joins the registry, so `--device index|g2|odyssey` works on download, convert, register and view. The README gains the dataset row and a run section: the per-device table, the raw-budget knobs, the NAS output root, and the fact that `--sequence` belongs to the verb and so goes *before* the dataset subcommand. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three. Stacks on #222. Adds the Monado SLAM Datasets (
collabora/monado-slam-datasets, CC-BY 4.0) as themsddataset with one catalog dataset per headset:msd-index(Valve Index, 2×960², kb4),msd-g2(Reverb G2, 4×640×480, radtan8, magnetometer),msd-odyssey(Odyssey+, 2×640×480, radtan8, magnetometer).How it works
discover()enumerates the remote HF tree.convert()fetches one sequence's archive, streams its PNGs out of the zip straight into theav1_nvencpipe encoder, remuxes the mp4 with exact csv timestamps, writes the base rrd atomically, and deletes the archive and scratch.raw_budget_gb(50) refuses to fetch when leftovers would breach the cap; the two split long sessions that exceed it alone are processed with a warning as the accepted exception..z01…zip) go through the conda-forge7zipCLI one camera directory at a time; single zips use the stdlib. Tested with a real Info-ZIP multi-volume fixture.calibration.jsonvia pyserde: kb4 →Fisheye62Parameters, radtan8 →PinholeParameterswith Brown-Conrady;T_imu_camis the camera pose in the rig (IMU) frame, referenceimu_00.video_timeis zero-based at the earliest sample of any stream includinggt(so the gt layer in 3/3 shares the origin);start_time_nskeeps the device clock. IMU at 1 kHz, magnetometer at 50 Hz, onlydata.csvfiles are read.downloadfetches just the calibration, checksav1_nvenc, and prints the plan.Evidence
pixi lock --checkclean; the lock diff is the7ziprecords only.MIO09_short_1_updown(Index): 8.6 s wall for download + encode + write, 2.98 MB rrd on/mnt/nas/datasets/msd-rrd/base/, 186 samples per camera, 3467 IMU rows, no raw left on disk./tmp/rerun-viewer-validation/msd-mio09-base/).Note:
--sequencebelongs to the verb, so it goes before the subcommand:dataforge-convert --sequence MIO09_short_1_updown msd --device index.🤖 Generated with Claude Code