Skip to content

BF: Preserve MGH tag data across an unmodified round-trip (gh-1402) - #1540

Open
CedricConday wants to merge 1 commit into
nipy:masterfrom
CedricConday:bf/mgh-preserve-tags
Open

BF: Preserve MGH tag data across an unmodified round-trip (gh-1402)#1540
CedricConday wants to merge 1 commit into
nipy:masterfrom
CedricConday:bf/mgh-preserve-tags

Conversation

@CedricConday

Copy link
Copy Markdown
Contributor

Closes #1402.

Problem

FreeSurfer may write variable-length tag data after the MGH footer —
TAG_CMDLINE history, TAG_AUTO_ALIGN, TAG_MRI_FRAME, embedded colour
tables. MGHHeader.from_fileobj reads exactly footer_dtype.itemsize bytes and
writeftr_to writes back only that struct, so everything after the footer is
silently dropped on a load/save round-trip.

This is reproducible with the file already in the repo:

import gzip, tempfile, nibabel as nib
d = tempfile.mkdtemp()
img = nib.load('nibabel/tests/data/test.mgz')
img.to_filename(f'{d}/rt.mgz')
len(gzip.open('nibabel/tests/data/test.mgz', 'rb').read())  # 23215
len(gzip.open(f'{d}/rt.mgz', 'rb').read())                  # 784  -> 22431 bytes lost

That file's tags are [(41, 7), (42, 22400)]TAG_PEDIR and TAG_MRI_FRAME.

It presents confusingly, which is probably why the original thread stalled: the
header compares as identical in both nibabel and mri_info, and only the file
size differs. That is exactly what the reporter described — "Both nibabel and
FreeSurfer's mri_info says that both images have identical headers. However
after decompressing the mgz to mgh I noticed the size of both files are not the
same."
— and it is consistent with their recon-all failure.

Fix

Retain the trailing bytes on read and write them back after the footer. They are
not interpreted.

Tags are not portable between volumes. TAG_MRI_FRAME stores per-frame
acquisition parameters — 11200 bytes per frame in the bundled file — and
TAG_AUTO_ALIGN holds a geometry-specific transform. Writing them next to a
different volume produces a file FreeSurfer will misparse: it would read
per-frame state for frames that no longer exist. So the bytes are recorded
together with the header state they were read with, and written back only while
that header is unchanged:

img = nib.load('.../test.mgz')          # nframes 2, 22431 tag bytes
nib.funcs.four_to_three(img)[0]         # nframes 1 -> tags dropped, not rewritten
img.slicer[:2, :2, :2]                  # tags dropped
img.to_filename(...)                    # unchanged header -> all 22431 bytes preserved

Editing voxel data while leaving the header alone — the case in the issue — keeps
the tags.

Two supporting changes:

  • The read is bounded by MAX_TAG_BYTES. A file whose dims field understates
    the data makes the voxel data itself look like tag data; unbounded, a 597 KB
    .mgz can expand to over a gigabyte resident, on what is otherwise a lazy
    header read. The bound is deliberately generous (64 MiB) because
    TAG_MRI_FRAME grows with the frame count — a long BOLD run legitimately
    carries megabytes of tags. Over the bound, it warns and keeps none.
  • writeftr_to truncates. Writing a smaller image over a reused stream (e.g.
    to_file_map with a BytesIO) previously left bytes from the larger image
    behind; harmless before, but they would now be read back as tag data and
    become sticky. ImageOpener gains a truncate passthrough for this.

Tests

  • test_tag_data_roundtrip — byte-exact round-trip; tags survive a data-only
    edit; tagless files are untouched.
  • test_tag_data_not_reused_across_volumes — uses the bundled test.mgz, and
    asserts tags are dropped for four_to_three, a reshape and a dtype change.
  • test_tag_data_bounded — over-long trailing region warns and keeps no tags.
  • test_tag_data_not_laundered_from_reused_stream — the truncation case.

Verified against master: the round-trip above loses 22431 bytes there and 0
here. Full suite 5290 passed, 270 skipped; nibabel/freesurfer is also green
under --parallel-threads 16.

One judgement call worth flagging: because the fingerprint is the header, editing
only voxel data keeps the tags. That is right for TAG_MRI_FRAME and
TAG_CMDLINE, but a segmentation whose labels are rewritten in place would keep
a colour table that no longer matches. Distinguishing those needs the tags to be
parsed, which is a larger change than this. Happy to go further if you'd prefer.


AI-assisted, human-reviewed.

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.53%. Comparing base (014fc4a) to head (56918f3).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1540      +/-   ##
==========================================
+ Coverage   95.48%   95.53%   +0.04%     
==========================================
  Files         209      209              
  Lines       30050    30144      +94     
  Branches     4494     4510      +16     
==========================================
+ Hits        28694    28798     +104     
+ Misses        925      914      -11     
- Partials      431      432       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CedricConday

Copy link
Copy Markdown
Contributor Author

Same as on #1539test-package (archive) fails on the archive filename:

error: Source distribution `archive/nibabel-archive.tgz` has a non-PEP 625-compliant filename; only `.tar.gz` and `.zip`

Not from this diff, which doesn't touch packaging. Happy to rebase whenever it's convenient.

FreeSurfer writes optional, variable-length tag data after the MGH footer
(TAG_CMDLINE history, auto-align matrices, embedded colour tables, ...).
`MGHHeader.from_fileobj` reads exactly `footer_dtype.itemsize` bytes and
`writeftr_to` writes back only that struct, so everything following the
footer was silently dropped on load/save. Round-tripping the bundled
nibabel/tests/data/test.mgz loses 22431 of its 23215 bytes.

The resulting file compares as having an identical header while being
smaller on disk, and downstream FreeSurfer tools that read the tags (e.g.
recon-all) fail on it.

Retain the trailing bytes on read and write them back after the footer. The
contents are not interpreted.

Tags are not portable between volumes: TAG_MRI_FRAME carries per-frame state
sized by the frame count and TAG_AUTO_ALIGN holds a geometry-specific
transform, so emitting them alongside a different volume would produce a
file FreeSurfer misparses. The bytes are therefore recorded with the header
state they were read with and written back only while the header is
unchanged, so deriving an image (`slicer`, `four_to_three`, a new shape or
data type) drops them rather than corrupting them.

The read is bounded by MAX_TAG_BYTES so that a file whose `dims` field
understates the data -- which would make the voxel data itself look like tag
data -- cannot pull an unbounded amount into memory; such a file warns and
keeps no tags. The bound is deliberately generous, as TAG_MRI_FRAME grows
with the frame count.

`writeftr_to` now truncates, so writing a smaller image over a reused stream
no longer leaves bytes from the previous image to be read back as tags.
`ImageOpener` gains a `truncate` passthrough for that.

Closes nipygh-1402

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@effigies
effigies force-pushed the bf/mgh-preserve-tags branch from 266c072 to 56918f3 Compare August 24, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nibabel copying mgz header incorrectly

1 participant