fix: warn on implicit zero-fill and preserve empty dicts in Batch - #1296
fix: warn on implicit zero-fill and preserve empty dicts in Batch#1296Lidang-Jiang wants to merge 5 commits into
Conversation
Addresses thu-ml#1088 and thu-ml#1089: - Issue thu-ml#1089: Empty dicts in batch lists were silently dropped during stack_, causing index position misalignment. Now all entries are preserved in the batch_list. - Issue thu-ml#1088: When __setitem__ or stack_ fills missing numeric keys with 0, a UserWarning is now emitted to alert users that the zero value may mask truly missing data. Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ddce36a42e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| has_any_nonempty = False | ||
| for batch in batches: | ||
| if isinstance(batch, dict): | ||
| batch_list.append(Batch(batch) if len(batch) > 0 else Batch()) |
There was a problem hiding this comment.
Reconcile nested schemas before keeping empty stack entries
Keeping top-level empty dicts as Batch() placeholders causes mixed nested schemas to fail in cases that previously worked. For example, Batch.stack([{"info": {"a": 1}}, {}, {"info": {"b": 2}}]) now routes info through the partial-key assignment path; the first value allocates only subkey a, and assigning the third value then raises ValueError("Creating keys is not supported by item assignment.") when subkey b appears. This breaks stacking whenever an empty entry sits between batches whose nested keys differ.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch. When empty dicts are preserved between batches whose nested keys differ, the element-by-element assignment path fails because __setitem__ doesn't support creating new keys.
Fixed in 0953587: for partial keys containing nested Batch/dict values, we now collect all entries and delegate to recursive Batch.stack instead of assigning one-by-one. This handles differing nested schemas correctly.
Verified locally:
>>> Batch.stack([{'info': {'a': 1}}, {}, {'info': {'b': 2}}])
Batch(info: Batch(a: array([1, 0, 0]), b: array([0, 0, 2])))- Remove stale type: ignore comments that mypy now flags as errors - Remove associated TODO comment since ignores are no longer needed
…tive The for-loop variable `value` from the shared-keys block retained type `list[Any]` in mypy's analysis, causing the later `isinstance(value, Batch)` check to be flagged as unreachable. Rename to `val` to give mypy a fresh binding. Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
When empty dicts are preserved between batches whose nested keys
differ (e.g. stack([{"info": {"a": 1}}, {}, {"info": {"b": 2}}])),
the previous element-by-element assignment would fail with
ValueError("Creating keys is not supported by item assignment.")
because the first value allocated only subkey 'a', and the third
value tried to create subkey 'b'.
Fix by detecting nested Batch/dict values in partial keys and using
recursive Batch.stack instead of element-by-element assignment.
Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
Summary
Addresses #1088 and #1089:
stack_, causing index position misalignment. Now all entries are preserved via_validate_and_convert_batches().__setitem__orstack_fills missing numeric keys with 0, aUserWarningis now emitted via_warn_numeric_zero_fill()to alert users that the zero value may mask truly missing data.The fix is backward compatible — the zero-fill behavior itself is unchanged (changing it would break existing code), but users are now explicitly warned when it happens.
Design decisions
_validate_and_convert_batches()and_warn_numeric_zero_fill()as module-level helpers to keepstack_()complexity under ruff's C901 limit (max 20).UserWarning(notDeprecationWarning) since the zero-fill behavior is not being deprecated, only made visible.Before (bug reproduction)
After (with fix)
Test results (75 passed)
Test plan
TestBatchNoneAndEmptyHandlingcovering empty dict preservation and warning behaviorruff checkandblackpass