Anchor overlap-DCT illumination fit in un-overlapped tile interiors

[arjunrajlab]

Problem

A user reported that Stitch Refinement + Illumination Correction left a visible illumination pattern in the YFP channel and asked whether the worker corrects every channel. It does: one model is fitted and applied per channel. The leftover pattern is a systematic over-correction that affects every channel. It only shows up clearly in low-contrast background channels like YFP once display contrast is stretched.

Cause: the overlap-DCT step is fitted only from tile-overlap pixels, and overlaps only cover the tile margins (about 38% of each 128×128 model tile on a 49-tile, ~10%-overlap grid). Nothing constrains the low-order DCT correction in the tile interior, so it extrapolates there and makes the flat field too peaked. The log-median base field alone is flat (about +0.3% center vs edge). The DCT step takes it to −2.5 to −6.4%: tile centers darker than edges after correction. Photobleaching in overlaps was also measured (later tile is 1.3% dimmer in YFP and 2.9% in A594), but it isn't the main driver: DAPI over-corrects with no measurable bleaching.

Change

• fit_overlap_dct builds a coverage mask from the pairs that actually contributed overlap rows. It adds a fixed quadratic prior (anchorᵀ·anchor) that pulls the DCT correction toward zero at uncovered pixels, so the field stays at the log-median base where there's no overlap evidence.
• The prior goes to robust_ridge through a new prior= argument, separate from the observations, so Huber IRLS never reweights it and the residual diagnostics stay overlap-only. (My first version appended zero-response rows; IRLS down-weighted them in a data-dependent way. I caught that in self-review and changed it before opening this PR.)
• The total prior weight is INTERIOR_ANCHOR_WEIGHT = 0.3 × the overlap sample count, evaluated on at most 4,096 interior pixels.
• New diagnostics: overlap_coverage_fraction, interior_anchor_weight, interior_anchor_samples.
• Worker version 1.0.3 → 1.1.0. Docs updated.

Results (49-tile, 7-Z, 5-channel ND2; center/edge residual of the median normalized corrected tile, plus median overlap seam mismatch)

Channel	Before: center/edge, seam	After (w=0.3): center/edge, seam
DAPI	−6.4%, 0.81%	+0.2%, 0.80%
YFP	−4.2%, 1.10%	−0.2%, 1.11%
CY3	−4.3%, 1.60%	−0.0%, 1.62%
A594	−4.4%, 2.71%	+0.2%, 2.66%
CY5	−2.5%, 1.31%	+0.2%, 1.31%

Weights from 0.03 to 0.3 all give a center/edge residual within ±0.8%, so the result isn't sensitive to the exact weight.

Tests

• New test_overlap_dct_does_not_overcorrect_unobserved_tile_interior: a 5×5 grid with ~10% overlaps and a known flat field. It fails with the anchor disabled (field 7–9% too peaked) and passes with it.
• ./build_workers.sh --build-and-run-tests illumination_correction: 24 passed.
• Ran the real worker end to end on the dataset locally, v1.0.3 vs v1.1.0. Results are in the follow-up comment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BTfgrwBPb5wjNxG2W5XeEK

[arjunrajlab]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T01:28:40.276024Z	cfce753	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: cfce753a31

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[arjunrajlab]

End-to-end check: I ran the real worker locally on the 49-tile ND2 dataset with v1.0.3 and with v1.1.0 (cfce753, same interface settings). Both jobs succeeded.

Comparing the two output pyramids at Z index 4, with the same tight 1–99% contrast on both:

• YFP: v1.0.3 shows a regular grid of dark tile centers and bright tile edges across the whole mosaic. v1.1.0 removes that pattern. What's left are some small tile-to-tile brightness steps (straight-edged, constant within each tile), which line up with the overlap photobleaching measured in the description. They aren't addressed here.
• DAPI: no visible difference, so no regression.

Codex review finished with no findings (👍). (Before/after images were shared outside GitHub; gh can't attach images to comments.)

🤖 Generated with Claude Code