perf(gc): don't mint per-object pointer masks for single-slot payloads

`interp.ts` spent ~19% of its runtime in `layout_forget_object`, plus ~6% in
the hashbrown probe underneath it, against a `LAYOUT_SLOT_MASKS` that had grown
past 400,000 live entries. Instrumented on the isolated FIB half, the
`PER_OBJECT_LAYOUTS_NONEMPTY` fast path fired 52 times in 15,000,000 calls.

The interpreter allocates `{ names: [p], vals: [a], parent }` per interpreted
call, so `layout_note_slot`'s "first pointer into a POINTER_FREE object" arm
minted 1.8M masks over payloads of exactly one slot. A mask over one slot can
skip nothing -- the tracer tag-checks that slot either way -- but the entry it
creates keeps the emptiness flag armed, which puts a two-map hash probe back on
every allocation in the program for as long as it lives.

Both mint sites now decline the mask below DEFAULT_MASK_MIN_SLOTS and use
GC_LAYOUT_UNKNOWN, the tag-checked scan-all-slots state that is already the
established fallback on this path. The tag check is exact here: neither site is
reachable for an object with an intact typed descriptor, so no raw-f64 slot can
be misread as a pointer.

Quiet M1 mini, best-of-5, interleaved against the same binaries with the policy
disabled: interp 1.894 -> 1.697, iso_miss 2.371 -> 2.157, new bench/mask_tax
probe 0.1218 -> 0.1049 with its numeric-element control flat at 1.000. No
regression on the rest of the 19-benchmark corpus, including tree, tree_wide,
retain*, cycles and deeplist.

Author: Ralph Küpper

changelog.d/7801-single-slot-pointer-mask.md

@@ -0,0 +1,89 @@
+### GC: stop minting per-object pointer masks for single-slot payloads
+
+`interp.ts` — the tree-walking interpreter that best resembles real software in
+the benchmark corpus — spent **~19% of its runtime in `layout_forget_object`**,
+plus another ~6% in the hashbrown probe underneath it. That is side-table
+bookkeeping, not user work, and by design it should have been ~zero: #7510's
+`PER_OBJECT_LAYOUTS_NONEMPTY` flag exists so that the allocation, store, death
+and relocation paths can skip both per-object layout maps whenever they are
+empty, which "on a monomorphic workload they are".
+
+They were not. Instrumented on `iso_FIB.ts` (the isolated FIB half):
+
+```
+forget_total=15,000,000  fast=52  slow=14,999,948
+residency: masks=313,875 -> 381,505 -> 400,430 (still climbing)
+```
+
+The disarmed fast path fired **52 times in 15 million calls**. Every other call
+took two `RefCell` round-trips and two hashes against a 400k-entry, cache-cold
+map — once per allocation, program-wide.
+
+**Cause.** `layout_note_slot`'s "first pointer stored into a `POINTER_FREE`
+object" arm minted a per-object entry in `LAYOUT_SLOT_MASKS`. The interpreter
+allocates `{ names: [p], vals: [a], parent }` per interpreted call, so it minted
+two masks per call — **1.8M of them**, each a mask over a payload of exactly
+**one slot**. A mask over one slot cannot skip anything: the tracer consults
+`layout_pointer_bearing_bits` on that slot either way, so the entry was the
+mask's entire contribution. The entries also outlive their arrays — they are
+only reclaimed when the recycled address is allocated over — so residency grew
+without bound, and a single live entry anywhere keeps the flag armed for every
+allocation in the program. This is #7510's "one immortal entry nullifies
+`is_empty()`" a second time, from the other direction.
+
+**Fix.** Both mint sites (`layout_note_slot` and
+`layout_rebuild_from_slots_with_policy`) now decline the mask when the payload
+is below `DEFAULT_MASK_MIN_SLOTS` (2, i.e. single-slot payloads only) and use
+`GC_LAYOUT_UNKNOWN` — the tag-checked scan-all-slots state — instead. That state
+is already the established fallback on this exact path, and the tag check is
+exact here: neither site is reachable for an object with an intact typed
+descriptor, so there are no raw-f64 slots whose bits could be misread as a
+pointer. `PERRY_LAYOUT_MASK_MIN_SLOTS` overrides the threshold for bisection.
+
+Two details worth keeping:
+
+- An array reports its `length`, but **only for a store into an already-formed
+  array**. Every append protocol notes the slot *before* bumping `length`, so
+  mid-construction `length` is the pre-append value; judging on it stranded
+  every incrementally built array — a `push` loop, a JSON parse — in the scan
+  state regardless of final size. Capacity is not a substitute either:
+  `MIN_ARRAY_CAPACITY` is 16, so a one-element literal reports 16 and the
+  distinction disappears entirely.
+- An object reports a bound derived from `GcHeader::size`, not `field_count`,
+  because `size` is maintained for every GC allocation whatever its
+  type-specific header holds.
+
+Both directions of error are *correct*, only differently priced: over-estimating
+mints a mask that was not needed (the old behaviour), and under-estimating
+routes the object to a scan that visits a superset of what the mask would have
+selected. Neither can hide a live child.
+
+**Measured** (quiet M1 mini, best-of-5, interleaved against the same binaries
+with the policy disabled, outputs byte-identical to node and exit codes checked):
+
+| bench | before | after |
+|---|--:|--:|
+| `interp` | 1.894 | **1.697** |
+| `iso_miss` | 2.371 | **2.157** |
+| `bench/mask_tax` (new probe) | 0.1218 | **0.1049** |
+| `bench/mask_tax_nopointer` (control) | 0.0929 | 0.0929 |
+
+No regression anywhere on the 19-benchmark corpus, including the GC-heavy
+`tree`, `tree_wide`, `retain*`, `cycles` and `deeplist`. The correctness canary
+(`iso_miss` printing `checksum 437840 misses 0`) holds plain and under
+`PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800`,
+`PERRY_GC_VERIFY_EVACUATION=1` and `PERRY_GC_FORCE_EVACUATE=1`.
+
+**New probe.** `gc-handoff/bench/mask_tax.ts` reduces the interpreter's
+environment chain to the shape that mints the masks, with
+`mask_tax_nopointer.ts` as a numeric-element control that holds flat at 1.000.
+The arrays have to genuinely escape: a first version kept them in a local,
+codegen scalar-replaced the array away, and the probe measured a 1.000 ratio
+while the bug was fully intact.
+
+**Left on the table, deliberately.** Raising the threshold to 9 or above pays
+roughly twice as much (`interp` 1.619, `iso_miss` 2.046) with still no
+regression on the corpus, but 21 tests in this crate encode "a small mixed
+payload uses a mask" as a precondition (5 do at 2, 11 at 3, saturating at 21
+from 9). That is a contract change worth making deliberately rather than as a
+side effect of a perf patch.

crates/perry-runtime/src/gc/layout.rs

@@ -908,7 +908,13 @@ pub(crate) fn layout_note_slot(parent_user: usize, slot_index: usize, value_bits
                     if (*header)._reserved & GC_LAYOUT_STATE_MASK != GC_LAYOUT_SIDE_MASK {
                         set_layout_state(header, GC_LAYOUT_SIDE_MASK);
                     }
-                } else if (*header)._reserved & GC_LAYOUT_STATE_MASK == GC_LAYOUT_POINTER_FREE {
+                } else if (*header)._reserved & GC_LAYOUT_STATE_MASK == GC_LAYOUT_POINTER_FREE
+                    && !super::layout_tables::layout_prefers_scan_over_mask(
+                        header,
+                        parent_user,
+                        slot_index,
+                    )
+                {
                     let mut mask = LayoutSlotMask::Inline(0);
                     mask.set_slot(slot_index);
                     masks.insert(parent_user, mask);
@@ -1338,6 +1344,11 @@ pub(super) unsafe fn layout_rebuild_from_slots_with_policy(
     if mask.is_empty() {
         set_layout_state(header, GC_LAYOUT_POINTER_FREE);
         slot_masks_remove(user_ptr as usize);
+    } else if slot_count < super::layout_tables::layout_mask_min_slots() {
+        // Too few slots for the mask to earn its side-table entry; the
+        // tag-checked scan is exact and costs the program nothing globally.
+        set_layout_state(header, GC_LAYOUT_UNKNOWN);
+        slot_masks_remove(user_ptr as usize);
     } else {
         set_layout_state(header, GC_LAYOUT_SIDE_MASK);
         slot_masks_insert(user_ptr as usize, mask);

crates/perry-runtime/src/gc/layout_tables.rs

@@ -31,6 +31,7 @@
 
 use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layouts_nonempty, hot_typed_layouts};
 use super::layout::{LayoutSlotMask, TypedLayoutDescriptor};
+use super::types::{GcHeader, GC_HEADER_SIZE, GC_TYPE_ARRAY, GC_TYPE_OBJECT};
 use std::cell::{Cell, RefCell};
 
 thread_local! {
@@ -50,6 +51,71 @@ thread_local! {
     pub(in crate::gc) static PER_OBJECT_LAYOUTS_NONEMPTY: Cell<bool> = const { Cell::new(false) };
 }
 
+/// Smallest payload slot count for which minting a **per-object pointer mask**
+/// is worth its side-table entry. Below it the object takes
+/// `GC_LAYOUT_UNKNOWN` — the tag-checked scan-all-slots state — instead.
+///
+/// The two sides are not symmetric. A mask's benefit is bounded by the object:
+/// it can skip at most `slots - pointers` tag checks per trace. Its cost is
+/// **program-global and unbounded** — one live entry arms
+/// [`PER_OBJECT_LAYOUTS_NONEMPTY`], which puts a two-map hash probe back on
+/// every allocation anywhere in the program for as long as that entry lives
+/// (see the module docs, and #7510's "one immortal entry nullifies
+/// `is_empty()`"). At the bottom of the range the asymmetry is total rather
+/// than merely lopsided: over a **single** slot a mask cannot skip anything at
+/// all, because the tracer consults `layout_pointer_bearing_bits` on that one
+/// slot either way, so the entry is the mask's entire contribution.
+///
+/// A tag check is exact at both mint sites: neither is reached for an object
+/// with an intact typed descriptor, so there are no raw-f64 slots whose bits a
+/// tag check could misread as a pointer. #7630 recorded the same conclusion for
+/// the materialiser cohort — "a pointer mask can never skip anything a tag
+/// check would not reject anyway ... the mask machinery buys nothing here".
+///
+/// `PERRY_LAYOUT_MASK_MIN_SLOTS` overrides it for bisection.
+#[inline(always)]
+pub(in crate::gc) fn layout_mask_min_slots() -> usize {
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    /// `usize::MAX` = "not yet read from the environment".
+    static N: AtomicUsize = AtomicUsize::new(usize::MAX);
+    match N.load(Ordering::Relaxed) {
+        usize::MAX => {
+            let v = std::env::var("PERRY_LAYOUT_MASK_MIN_SLOTS")
+                .ok()
+                .and_then(|s| s.parse::<usize>().ok())
+                .unwrap_or(DEFAULT_MASK_MIN_SLOTS);
+            N.store(v, Ordering::Relaxed);
+            v
+        }
+        v => v,
+    }
+}
+
+/// Only single-slot payloads take the scan. This is deliberately the
+/// *provable* end of the range: at one slot the mask demonstrably skips
+/// nothing, so no judgement about tracing cost is being made.
+///
+/// Measured on the 19-benchmark corpus (quiet M1 mini, best-of-5, interleaved
+/// against the same binaries with the policy disabled):
+///
+/// | bench | before | after |
+/// |---|--:|--:|
+/// | `interp` | 1.894 | **1.697** |
+/// | `iso_miss` | 2.371 | **2.157** |
+/// | `bench/mask_tax` | 0.1218 | **0.1049** |
+/// | `bench/mask_tax_nopointer` (control) | 0.0929 | 0.0929 |
+///
+/// Every other benchmark — including the GC-heavy `tree`, `tree_wide`,
+/// `retain*`, `cycles`, `deeplist` — is unchanged within noise.
+///
+/// Raising it pays roughly twice as much and costs test churn, both measured:
+/// `9` and above gives `interp` 1.619 / `iso_miss` 2.046 with still no
+/// regression on the corpus, but 21 tests in this crate encode "a small mixed
+/// payload uses a mask" as a precondition (5 do at `2`, 11 at `3`, saturating
+/// at 21 from `9`). That is a contract change worth making on purpose rather
+/// than as a side effect of a perf patch.
+pub(in crate::gc) const DEFAULT_MASK_MIN_SLOTS: usize = 2;
+
 /// True when either per-object side table may hold an entry. `false` is a
 /// proof of emptiness (see [`PER_OBJECT_LAYOUTS_NONEMPTY`]); `true` is only a
 /// hint, so every caller still has to handle a miss.
@@ -227,3 +293,68 @@ pub(in crate::gc) fn layout_forget_object(user_ptr: usize) {
 pub(in crate::gc) fn test_per_object_tables_are_empty() -> bool {
     hot_layout_slot_masks().borrow().is_empty() && hot_typed_layouts().borrow().is_empty()
 }
+
+/// An upper bound on the payload slots the tracer would enumerate for
+/// `user_ptr`, or `usize::MAX` when this module cannot cheaply tell.
+///
+/// Both directions of error are *correct*, only differently priced, which is
+/// what lets this be a bound rather than an exact count: over-estimating mints
+/// a mask that was not needed (the pre-existing behaviour), and
+/// under-estimating routes the object to `GC_LAYOUT_UNKNOWN`, where the tracer
+/// scans every slot and so visits a superset of what a mask would have
+/// selected. Neither can hide a live child.
+///
+/// An array reports its `length` — exactly the range the tracer walks, and so
+/// exactly the bound on what a mask could skip — but **only for a store into an
+/// already-formed array**. A store at the append position (`slot_index >=
+/// length`) reports `usize::MAX` instead, because every append protocol writes
+/// the element and notes the slot *before* bumping `length` (see
+/// [`layout_all_pointer_array_append`]): mid-construction `length` is the
+/// pre-append value, usually 0 or 1, and judging on it would strand every
+/// incrementally built array — a `push` loop, a JSON parse — in the scan state
+/// no matter how large it eventually grew. Capacity is not a substitute:
+/// `MIN_ARRAY_CAPACITY` is 16, so a one-element literal reports 16 and the
+/// distinction this is drawing disappears.
+///
+/// An object reports the bound derived from [`GcHeader::size`] rather than its
+/// `field_count`: `size` is maintained for every GC allocation whatever its
+/// type-specific header says, so this stays correct for a payload that is not a
+/// well-formed `ObjectHeader`, and it errs high — towards the old mask path.
+#[inline]
+pub(in crate::gc) unsafe fn layout_payload_slot_count(
+    header: *const GcHeader,
+    user_ptr: usize,
+    slot_index: usize,
+) -> usize {
+    match (*header).obj_type {
+        GC_TYPE_ARRAY => {
+            let arr = user_ptr as *const crate::array::ArrayHeader;
+            let length = (*arr).length as usize;
+            let capacity = (*arr).capacity as usize;
+            if length > capacity || length > 16_000_000 || slot_index >= length {
+                usize::MAX
+            } else {
+                length
+            }
+        }
+        GC_TYPE_OBJECT => {
+            let size = (*header).size as usize;
+            match size.checked_sub(GC_HEADER_SIZE) {
+                Some(payload) => payload / 8,
+                None => usize::MAX,
+            }
+        }
+        _ => usize::MAX,
+    }
+}
+
+/// True when `user_ptr` is small enough that a tag-checked scan of every slot
+/// beats a per-object pointer mask. See [`layout_mask_min_slots`].
+#[inline]
+pub(in crate::gc) unsafe fn layout_prefers_scan_over_mask(
+    header: *const GcHeader,
+    user_ptr: usize,
+    slot_index: usize,
+) -> bool {
+    layout_payload_slot_count(header, user_ptr, slot_index) < layout_mask_min_slots()
+}

crates/perry-runtime/src/gc/tests/layout_trace/array_layout.rs

@@ -52,20 +52,25 @@ fn test_layout_mask_overflow_fields_and_array_grow_transfer() {
         assert_ne!((*child_header).gc_flags & GC_FLAG_MARKED, 0);
     }
 
-    let arr = crate::array::js_array_alloc_with_length(1);
+    // Two elements, not one: a mask over a single-slot payload can skip
+    // nothing, so `layout_note_slot` now leaves such an array in the
+    // tag-checked `GC_LAYOUT_UNKNOWN` state and mints no mask to grow or
+    // transfer. Two slots is the smallest payload that still exercises the
+    // grow/transfer path this test is about.
+    let arr = crate::array::js_array_alloc_with_length(2);
     crate::array::js_array_set_f64(
         arr,
         0,
         f64::from_bits(STRING_TAG | (child as u64 & POINTER_MASK)),
     );
     let grown = crate::array::js_array_grow(arr, 128);
-    assert_eq!(test_layout_pointer_slot_count(grown as usize, 1), Some(1));
+    assert_eq!(test_layout_pointer_slot_count(grown as usize, 2), Some(1));
 
-    let moved = crate::array::js_array_alloc_with_length(1);
+    let moved = crate::array::js_array_alloc_with_length(2);
     unsafe {
         layout_transfer(grown as *mut u8, moved as *mut u8);
     }
-    assert_eq!(test_layout_pointer_slot_count(moved as usize, 1), Some(1));
+    assert_eq!(test_layout_pointer_slot_count(moved as usize, 2), Some(1));
 
     clear_marks();
     clear_mark_seeds();
@@ -213,7 +218,13 @@ fn test_array_mixed_bulk_producers_preserve_pointer_layout() {
     let set = crate::set::js_set_alloc(4);
     let set = crate::set::js_set_add(set, child_box);
     let set_arr = crate::set::js_set_to_array(set);
-    assert_eq!(test_layout_pointer_slot_count(set_arr as usize, 1), Some(1));
+    // A one-element result carries no mask: over a single slot a mask selects
+    // exactly what the tracer's tag check already selects, so it is pure
+    // side-table cost and `layout_note_slot` declines it. What this test is
+    // actually about — that the bulk producer leaves a layout the tracer can
+    // follow to the child — is asserted below, unchanged: one slot read, child
+    // marked.
+    assert_eq!(test_layout_pointer_slot_count(set_arr as usize, 1), None);
     assert_array_root_trace_reads(set_arr, 1);
     unsafe {
         assert_ne!((*child_header).gc_flags & GC_FLAG_MARKED, 0);
@@ -224,7 +235,11 @@ fn test_array_mixed_bulk_producers_preserve_pointer_layout() {
     let map = crate::map::js_map_alloc(4);
     let map = crate::map::js_map_set(map, 7.0, child_box);
     let entries = crate::map::js_map_entries(map);
-    assert_eq!(test_layout_pointer_slot_count(entries as usize, 1), Some(1));
+    // One entry, so the outer array is single-slot and carries no mask for the
+    // same reason as the set above; the pair it holds is two slots and still
+    // does. Both are traced either way, which is what the reads assertion and
+    // the child's mark bit below check.
+    assert_eq!(test_layout_pointer_slot_count(entries as usize, 1), None);
     let pair_box = crate::array::js_array_get_f64(entries, 0);
     let pair = (pair_box.to_bits() & POINTER_MASK) as *mut crate::array::ArrayHeader;
     assert_eq!(test_layout_pointer_slot_count(pair as usize, 2), Some(1));
@@ -235,14 +250,20 @@ fn test_array_mixed_bulk_producers_preserve_pointer_layout() {
     clear_marks();
     clear_mark_seeds();
 
-    let overwritten = crate::array::js_array_alloc_with_length(1);
+    // Two slots, so this still goes through the mask: clearing the last
+    // pointer empties it and restores `GC_LAYOUT_POINTER_FREE`, which is the
+    // transition being asserted. A single-slot array never mints a mask now,
+    // and `GC_LAYOUT_UNKNOWN` is one-way — such an array keeps being scanned
+    // after the pointer is overwritten. That costs one tag check on one slot,
+    // which is the whole reason the mask was not worth minting for it.
+    let overwritten = crate::array::js_array_alloc_with_length(2);
     crate::array::js_array_set_f64(overwritten, 0, child_box);
     assert_eq!(
-        test_layout_pointer_slot_count(overwritten as usize, 1),
+        test_layout_pointer_slot_count(overwritten as usize, 2),
         Some(1)
     );
     crate::array::js_array_set_f64(overwritten, 0, 99.0);
-    assert_numeric_array_trace_free(overwritten, 1);
+    assert_numeric_array_trace_free(overwritten, 2);
 
     clear_marks();
     clear_mark_seeds();