perf(gc): don't mint per-object pointer masks for single-slot payloads (#7812)

`interp.ts` spent ~19% of its runtime in `layout_forget_object`, plus ~6% in
the hashbrown probe underneath it, against a `LAYOUT_SLOT_MASKS` that had grown
past 400,000 live entries. Instrumented on the isolated FIB half, the
`PER_OBJECT_LAYOUTS_NONEMPTY` fast path fired 52 times in 15,000,000 calls.

The interpreter allocates `{ names: [p], vals: [a], parent }` per interpreted
call, so `layout_note_slot`'s "first pointer into a POINTER_FREE object" arm
minted 1.8M masks over payloads of exactly one slot. A mask over one slot can
skip nothing -- the tracer tag-checks that slot either way -- but the entry it
creates keeps the emptiness flag armed, which puts a two-map hash probe back on
every allocation in the program for as long as it lives.

Both mint sites now decline the mask below DEFAULT_MASK_MIN_SLOTS and use
GC_LAYOUT_UNKNOWN, the tag-checked scan-all-slots state that is already the
established fallback on this path. The tag check is exact here: neither site is
reachable for an object with an intact typed descriptor, so no raw-f64 slot can
be misread as a pointer.

Quiet M1 mini, best-of-5, interleaved against the same binaries with the policy
disabled: interp 1.894 -> 1.697, iso_miss 2.371 -> 2.157, new bench/mask_tax
probe 0.1218 -> 0.1049 with its numeric-element control flat at 1.000. No
regression on the rest of the 19-benchmark corpus, including tree, tree_wide,
retain*, cycles and deeplist.

Co-authored-by: Ralph Küpper <ralph@skelpo.com>

Author: proggeramlug

changelog.d/7812-single-slot-pointer-mask.md

@@ -0,0 +1,89 @@
+### GC: stop minting per-object pointer masks for single-slot payloads
+
+`interp.ts` — the tree-walking interpreter that best resembles real software in
+the benchmark corpus — spent **~19% of its runtime in `layout_forget_object`**,
+plus another ~6% in the hashbrown probe underneath it. That is side-table
+bookkeeping, not user work, and by design it should have been ~zero: #7510's
+`PER_OBJECT_LAYOUTS_NONEMPTY` flag exists so that the allocation, store, death
+and relocation paths can skip both per-object layout maps whenever they are
+empty, which "on a monomorphic workload they are".
+
+They were not. Instrumented on `iso_FIB.ts` (the isolated FIB half):
+
+```
+forget_total=15,000,000  fast=52  slow=14,999,948
+residency: masks=313,875 -> 381,505 -> 400,430 (still climbing)
+```
+
+The disarmed fast path fired **52 times in 15 million calls**. Every other call
+took two `RefCell` round-trips and two hashes against a 400k-entry, cache-cold
+map — once per allocation, program-wide.
+
+**Cause.** `layout_note_slot`'s "first pointer stored into a `POINTER_FREE`
+object" arm minted a per-object entry in `LAYOUT_SLOT_MASKS`. The interpreter
+allocates `{ names: [p], vals: [a], parent }` per interpreted call, so it minted
+two masks per call — **1.8M of them**, each a mask over a payload of exactly
+**one slot**. A mask over one slot cannot skip anything: the tracer consults
+`layout_pointer_bearing_bits` on that slot either way, so the entry was the
+mask's entire contribution. The entries also outlive their arrays — they are
+only reclaimed when the recycled address is allocated over — so residency grew
+without bound, and a single live entry anywhere keeps the flag armed for every
+allocation in the program. This is #7510's "one immortal entry nullifies
+`is_empty()`" a second time, from the other direction.
+
+**Fix.** Both mint sites (`layout_note_slot` and
+`layout_rebuild_from_slots_with_policy`) now decline the mask when the payload
+is below `DEFAULT_MASK_MIN_SLOTS` (2, i.e. single-slot payloads only) and use
+`GC_LAYOUT_UNKNOWN` — the tag-checked scan-all-slots state — instead. That state
+is already the established fallback on this exact path, and the tag check is
+exact here: neither site is reachable for an object with an intact typed
+descriptor, so there are no raw-f64 slots whose bits could be misread as a
+pointer. `PERRY_LAYOUT_MASK_MIN_SLOTS` overrides the threshold for bisection.
+
+Two details worth keeping:
+
+- An array reports its `length`, but **only for a store into an already-formed
+  array**. Every append protocol notes the slot *before* bumping `length`, so
+  mid-construction `length` is the pre-append value; judging on it stranded
+  every incrementally built array — a `push` loop, a JSON parse — in the scan
+  state regardless of final size. Capacity is not a substitute either:
+  `MIN_ARRAY_CAPACITY` is 16, so a one-element literal reports 16 and the
+  distinction disappears entirely.
+- An object reports a bound derived from `GcHeader::size`, not `field_count`,
+  because `size` is maintained for every GC allocation whatever its
+  type-specific header holds.
+
+Both directions of error are *correct*, only differently priced: over-estimating
+mints a mask that was not needed (the old behaviour), and under-estimating
+routes the object to a scan that visits a superset of what the mask would have
+selected. Neither can hide a live child.
+
+**Measured** (quiet M1 mini, best-of-5, interleaved against the same binaries
+with the policy disabled, outputs byte-identical to node and exit codes checked):
+
+| bench | before | after |
+|---|--:|--:|
+| `interp` | 1.894 | **1.697** |
+| `iso_miss` | 2.371 | **2.157** |
+| `bench/mask_tax` (new probe) | 0.1218 | **0.1049** |
+| `bench/mask_tax_nopointer` (control) | 0.0929 | 0.0929 |
+
+No regression anywhere on the 19-benchmark corpus, including the GC-heavy
+`tree`, `tree_wide`, `retain*`, `cycles` and `deeplist`. The correctness canary
+(`iso_miss` printing `checksum 437840 misses 0`) holds plain and under
+`PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800`,
+`PERRY_GC_VERIFY_EVACUATION=1` and `PERRY_GC_FORCE_EVACUATE=1`.
+
+**New probe.** `gc-handoff/bench/mask_tax.ts` reduces the interpreter's
+environment chain to the shape that mints the masks, with
+`mask_tax_nopointer.ts` as a numeric-element control that holds flat at 1.000.
+The arrays have to genuinely escape: a first version kept them in a local,
+codegen scalar-replaced the array away, and the probe measured a 1.000 ratio
+while the bug was fully intact.
+
+**Left on the table, deliberately.** Raising the threshold to 9 or above pays
+roughly twice as much (`interp` 1.619, `iso_miss` 2.046) with still no
+regression on the corpus, but 21 tests in this crate encode "a small mixed
+payload uses a mask" as a precondition (5 do at 2, 11 at 3, saturating at 21
+from 9). That is a contract change worth making deliberately rather than as a
+side effect of a perf patch.

crates/perry-runtime/src/gc/layout.rs

@@ -909,16 +909,25 @@ pub(crate) fn layout_note_slot(parent_user: usize, slot_index: usize, value_bits
                         set_layout_state(header, GC_LAYOUT_SIDE_MASK);
                     }
                 } else if (*header)._reserved & GC_LAYOUT_STATE_MASK == GC_LAYOUT_POINTER_FREE {
-                    if super::layout_tables::immortal_layout_scope_active() {
-                        // An object built inside an `ImmortalLayoutScope` is
+                    if super::layout_tables::immortal_layout_scope_active()
+                        || super::layout_tables::layout_prefers_scan_over_mask(
+                            header,
+                            parent_user,
+                            slot_index,
+                        )
+                    {
+                        // Two reasons to decline the mask, one fallback. An
+                        // object built inside an `ImmortalLayoutScope` is
                         // rooted for the life of the process, so the entry it
                         // would mint here is never removed — and one such
                         // entry disables `PER_OBJECT_LAYOUTS_NONEMPTY` for
-                        // every allocation the program will ever make. Take
+                        // every allocation the program will ever make (see
+                        // `ImmortalLayoutScope`). And a payload too small for
+                        // the mask to earn its side-table entry
+                        // (`layout_prefers_scan_over_mask`) skips nothing the
+                        // tag-checked scan would not check anyway. Both take
                         // the same `GC_LAYOUT_UNKNOWN` fallback the `else`
-                        // arm below uses for this exact situation; see
-                        // `ImmortalLayoutScope` for why that is the safe
-                        // state and not a weaker one.
+                        // arm below uses for this exact situation.
                         set_layout_state(header, GC_LAYOUT_UNKNOWN);
                     } else {
                         let mut mask = LayoutSlotMask::Inline(0);
@@ -1354,13 +1363,17 @@ pub(super) unsafe fn layout_rebuild_from_slots_with_policy(
     if mask.is_empty() {
         set_layout_state(header, GC_LAYOUT_POINTER_FREE);
         slot_masks_remove(user_ptr as usize);
-    } else if super::layout_tables::immortal_layout_scope_active() {
-        // Same reasoning as the `layout_note_slot` branch: an object built
-        // inside an `ImmortalLayoutScope` never dies, so the mask it would
-        // install here is a permanent tenant of a side table whose emptiness
-        // is a process-wide fast path. Falling back to the tag-checked scan is
-        // sound *for this rebuild specifically* because the mask above is
-        // itself derived from `layout_pointer_bearing_bits` — exactly the test
+    } else if super::layout_tables::immortal_layout_scope_active()
+        || slot_count < super::layout_tables::layout_mask_min_slots()
+    {
+        // Same two reasons as the `layout_note_slot` branch, same fallback. An
+        // object built inside an `ImmortalLayoutScope` never dies, so the mask
+        // it would install here is a permanent tenant of a side table whose
+        // emptiness is a process-wide fast path; and too few slots means the
+        // mask cannot earn its side-table entry — the tag-checked scan is
+        // exact and costs the program nothing globally. Falling back is sound
+        // *for this rebuild specifically* because the mask above is itself
+        // derived from `layout_pointer_bearing_bits` — exactly the test
         // `GC_LAYOUT_UNKNOWN` re-runs per slot. (This is why the scope may not
         // be applied to a TYPED descriptor, whose raw-f64 slots the tag test
         // would misread; see `ImmortalLayoutScope`.)

crates/perry-runtime/src/gc/layout_tables.rs

@@ -31,6 +31,7 @@
 
 use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layout_hint, hot_typed_layouts};
 use super::layout::{LayoutSlotMask, TypedLayoutDescriptor};
+use super::types::{GcHeader, GC_HEADER_SIZE, GC_TYPE_ARRAY, GC_TYPE_OBJECT};
 use std::cell::{Cell, RefCell};
 
 thread_local! {
@@ -301,6 +302,71 @@ pub(crate) fn per_object_layout_table_sizes() -> (usize, usize) {
     )
 }
 
+/// Smallest payload slot count for which minting a **per-object pointer mask**
+/// is worth its side-table entry. Below it the object takes
+/// `GC_LAYOUT_UNKNOWN` — the tag-checked scan-all-slots state — instead.
+///
+/// The two sides are not symmetric. A mask's benefit is bounded by the object:
+/// it can skip at most `slots - pointers` tag checks per trace. Its cost is
+/// **program-global and unbounded** — one live entry arms
+/// [`PER_OBJECT_LAYOUTS_NONEMPTY`], which puts a two-map hash probe back on
+/// every allocation anywhere in the program for as long as that entry lives
+/// (see the module docs, and #7510's "one immortal entry nullifies
+/// `is_empty()`"). At the bottom of the range the asymmetry is total rather
+/// than merely lopsided: over a **single** slot a mask cannot skip anything at
+/// all, because the tracer consults `layout_pointer_bearing_bits` on that one
+/// slot either way, so the entry is the mask's entire contribution.
+///
+/// A tag check is exact at both mint sites: neither is reached for an object
+/// with an intact typed descriptor, so there are no raw-f64 slots whose bits a
+/// tag check could misread as a pointer. #7630 recorded the same conclusion for
+/// the materialiser cohort — "a pointer mask can never skip anything a tag
+/// check would not reject anyway ... the mask machinery buys nothing here".
+///
+/// `PERRY_LAYOUT_MASK_MIN_SLOTS` overrides it for bisection.
+#[inline(always)]
+pub(in crate::gc) fn layout_mask_min_slots() -> usize {
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    /// `usize::MAX` = "not yet read from the environment".
+    static N: AtomicUsize = AtomicUsize::new(usize::MAX);
+    match N.load(Ordering::Relaxed) {
+        usize::MAX => {
+            let v = std::env::var("PERRY_LAYOUT_MASK_MIN_SLOTS")
+                .ok()
+                .and_then(|s| s.parse::<usize>().ok())
+                .unwrap_or(DEFAULT_MASK_MIN_SLOTS);
+            N.store(v, Ordering::Relaxed);
+            v
+        }
+        v => v,
+    }
+}
+
+/// Only single-slot payloads take the scan. This is deliberately the
+/// *provable* end of the range: at one slot the mask demonstrably skips
+/// nothing, so no judgement about tracing cost is being made.
+///
+/// Measured on the 19-benchmark corpus (quiet M1 mini, best-of-5, interleaved
+/// against the same binaries with the policy disabled):
+///
+/// | bench | before | after |
+/// |---|--:|--:|
+/// | `interp` | 1.894 | **1.697** |
+/// | `iso_miss` | 2.371 | **2.157** |
+/// | `bench/mask_tax` | 0.1218 | **0.1049** |
+/// | `bench/mask_tax_nopointer` (control) | 0.0929 | 0.0929 |
+///
+/// Every other benchmark — including the GC-heavy `tree`, `tree_wide`,
+/// `retain*`, `cycles`, `deeplist` — is unchanged within noise.
+///
+/// Raising it pays roughly twice as much and costs test churn, both measured:
+/// `9` and above gives `interp` 1.619 / `iso_miss` 2.046 with still no
+/// regression on the corpus, but 21 tests in this crate encode "a small mixed
+/// payload uses a mask" as a precondition (5 do at `2`, 11 at `3`, saturating
+/// at 21 from `9`). That is a contract change worth making on purpose rather
+/// than as a side effect of a perf patch.
+pub(in crate::gc) const DEFAULT_MASK_MIN_SLOTS: usize = 2;
+
 /// True when either per-object side table may hold an entry. `false` is a
 /// proof of emptiness (see [`PER_OBJECT_LAYOUTS_NONEMPTY`]); `true` is only a
 /// hint, so every caller still has to handle a miss.
@@ -499,3 +565,68 @@ pub(in crate::gc) fn layout_forget_object(user_ptr: usize) {
 pub(in crate::gc) fn test_per_object_tables_are_empty() -> bool {
     hot_layout_slot_masks().borrow().is_empty() && hot_typed_layouts().borrow().is_empty()
 }
+
+/// An upper bound on the payload slots the tracer would enumerate for
+/// `user_ptr`, or `usize::MAX` when this module cannot cheaply tell.
+///
+/// Both directions of error are *correct*, only differently priced, which is
+/// what lets this be a bound rather than an exact count: over-estimating mints
+/// a mask that was not needed (the pre-existing behaviour), and
+/// under-estimating routes the object to `GC_LAYOUT_UNKNOWN`, where the tracer
+/// scans every slot and so visits a superset of what a mask would have
+/// selected. Neither can hide a live child.
+///
+/// An array reports its `length` — exactly the range the tracer walks, and so
+/// exactly the bound on what a mask could skip — but **only for a store into an
+/// already-formed array**. A store at the append position (`slot_index >=
+/// length`) reports `usize::MAX` instead, because every append protocol writes
+/// the element and notes the slot *before* bumping `length` (see
+/// [`layout_all_pointer_array_append`]): mid-construction `length` is the
+/// pre-append value, usually 0 or 1, and judging on it would strand every
+/// incrementally built array — a `push` loop, a JSON parse — in the scan state
+/// no matter how large it eventually grew. Capacity is not a substitute:
+/// `MIN_ARRAY_CAPACITY` is 16, so a one-element literal reports 16 and the
+/// distinction this is drawing disappears.
+///
+/// An object reports the bound derived from [`GcHeader::size`] rather than its
+/// `field_count`: `size` is maintained for every GC allocation whatever its
+/// type-specific header says, so this stays correct for a payload that is not a
+/// well-formed `ObjectHeader`, and it errs high — towards the old mask path.
+#[inline]
+pub(in crate::gc) unsafe fn layout_payload_slot_count(
+    header: *const GcHeader,
+    user_ptr: usize,
+    slot_index: usize,
+) -> usize {
+    match (*header).obj_type {
+        GC_TYPE_ARRAY => {
+            let arr = user_ptr as *const crate::array::ArrayHeader;
+            let length = (*arr).length as usize;
+            let capacity = (*arr).capacity as usize;
+            if length > capacity || length > 16_000_000 || slot_index >= length {
+                usize::MAX
+            } else {
+                length
+            }
+        }
+        GC_TYPE_OBJECT => {
+            let size = (*header).size as usize;
+            match size.checked_sub(GC_HEADER_SIZE) {
+                Some(payload) => payload / 8,
+                None => usize::MAX,
+            }
+        }
+        _ => usize::MAX,
+    }
+}
+
+/// True when `user_ptr` is small enough that a tag-checked scan of every slot
+/// beats a per-object pointer mask. See [`layout_mask_min_slots`].
+#[inline]
+pub(in crate::gc) unsafe fn layout_prefers_scan_over_mask(
+    header: *const GcHeader,
+    user_ptr: usize,
+    slot_index: usize,
+) -> bool {
+    layout_payload_slot_count(header, user_ptr, slot_index) < layout_mask_min_slots()
+}