test all nouns

Author: ttuff

PROMPT_LOG.md

@@ -2333,3 +2333,149 @@ secrets, credentials, private tokens, or unrelated transcript text.
   scientifically declared cross-grid alignment. Each must satisfy source,
   CRS, boundary, units, temporal-support, missingness, and QA contracts before
   a dependency design is promoted to executable.
+
+## 2026-08-26 - Lightweight semantic grammar and analysis coach
+
+### User goal
+
+- Make the existing `pipe(noun) | verb() | verb()` grammar semantically
+  inspectable and helpful to scientists and future agents without adding a DSL,
+  rewriting user order, or requiring new user-facing object classes.
+- Add noun and verb metadata, state tracking, deterministic explanations and
+  suggestions, metadata-only validation, order knowledge, useful scientific
+  errors, diagrams, tests, and readable grammatical keywords.
+
+### Design and implementation
+
+- Audited the pipe, 62 callable names visible in `cubedynamics.verbs`, 38
+  explicit `verbs.__all__` exports, documented state/event/project verbs,
+  loaders, output metadata, dimensions, provenance, and information-changing
+  operations. Recorded the baseline in `docs/project/grammar_inventory.md`.
+- Added `cubedynamics.grammar`, a lightweight registry and inference layer with
+  nine semantic states: observation, continuous field, categorical field,
+  condition, event, feature, relationship, summary, and network. Verb contracts
+  declare human descriptions, accepted/returned states, requirements,
+  preservation/removal effects, ownership categories, and examples.
+- Registered every callable in `verbs.__all__` plus documented state, event,
+  synchrony, and biology verbs. Conceptual future verbs are present only in
+  order-rule metadata with `implemented=False`; they are never offered as
+  runnable suggestions.
+- Extended `Pipe` with immutable `semantic_state` and `semantic_trace`, plus
+  deterministic `explain()`, `suggest()`, and `validate()` methods. Pipe still
+  calls each stage exactly once in written left-to-right order. Plain,
+  unregistered Python callables remain valid and need no subclass or registry
+  step.
+- Added semantic preflight errors for incompatible kinds, missing temporal or
+  spatial support, removed time variation, unordered time, and obvious CRS or
+  dimension conflicts for aligned-state overlap. Underlying execution errors
+  remain unchanged outside known semantic incompatibilities.
+- Added the four requested order categories and the complete starter response
+  set, including both directions for near/density, intersect/density,
+  intersect/summarize, clip/summarize, filter/change, subtract/divide,
+  normalize/threshold, and upstream/intersect. Rules are neutral,
+  deterministic, and do not mutate pipelines.
+- Added semantic metadata to scientific nouns, state/event/overlap outputs, and
+  spatial block summary/comparison outputs so metadata survives unwrap and
+  re-pipe workflows. Added backward-compatible `over=` to `mean`, `variance`,
+  `anomaly`, and `zscore`; `dim=` remains supported and conflicts are explicit.
+
+### Documentation and validation
+
+- Added `docs/concepts/semantic_grammar.md` with state and architecture
+  diagrams, explain/suggest/validate examples, order categories, and a
+  near→density versus density→near interpretation table. Updated Pipe, grammar,
+  public API, README, and MkDocs navigation pages.
+- Added `tests/test_grammar_semantics.py` for exact order/no rewriting, state
+  transitions, explanations, suggestions, reports, semantic errors, lost
+  information, all order rules, public verb registry coverage, CRS conflicts,
+  metadata propagation, and keyword compatibility. Expanded noun metadata
+  assertions and retained focused regression coverage for VirtualCube,
+  synchrony, spatial-block, plotting, and public APIs.
+- Final focused affected-area run: 84 passed. An additional post-stall group
+  covering plotting, shape/statistical verbs, vignettes, and VirtualCube paths
+  passed 21 tests. `mkdocs build --strict`, Python compilation, and
+  `git diff --check` passed.
+- The required offline suite reached 349 passed, 3 skipped, and 9 deselected
+  with no failures, then encountered the repository's pre-existing Matplotlib
+  backend stall and was interrupted after about 97 seconds. The six semantic
+  compatibility regressions found on the first broad run were corrected; their
+  complete affected suites subsequently passed.
+
+### Known limitations and next work
+
+- Runtime inference is strongest for xarray, VirtualCube, EventResult, and
+  metadata-bearing project outputs. Custom non-xarray objects fall back to an
+  observation state unless their project adds semantic attrs or registry
+  metadata.
+- Some broad project/integration contracts are intentionally conservative and
+  should be refined alongside their owning workflows. Unit compatibility is
+  reported and preserved, but cross-object unit conversion/reconciliation
+  awaits verbs that actually combine numeric quantities.
+- High-value next work is to let external verb packages contribute registry
+  entries through a documented registration hook, then implement vetted
+  feature/network nouns and verbs such as near, density, intersect, summarize,
+  and upstream against explicit CRS, boundary, unit, and provenance contracts.
+
+## 2026-08-26 - Interrupted-work audit and Phase 1 source-QA completion
+
+### User goal
+
+- Re-read the interrupted publication, Decision Lab, and semantic-grammar
+  briefs and complete work left unfinished when earlier tasks hit usage limits.
+- Keep examples observational, reproducible, validated, and suitable for the
+  publication website.
+
+### Audit and decisions
+
+- Re-audited the three pasted briefs and the resulting repository changes. The
+  semantic grammar is implemented and tested. The Decision Lab intentionally
+  contains one executable real-data vignette and four clearly labeled
+  dependency designs because the required feature/network nouns and verbs do
+  not yet exist; no speculative APIs or fake data were promoted as runnable.
+- Identified the remaining Phase 1 publication gap: gridMET and Sentinel-2 had
+  no reviewed numerical/visual baseline, and their dataset pages referenced
+  text files masquerading as PNG previews. PRISM also retained an obsolete
+  placeholder despite already having reviewed QA evidence.
+
+### Implementation
+
+- Added small real-data gridMET and Sentinel-2 fixtures with checksums, source
+  requests, and provenance under `data/source_qa/`, plus a reproducibility
+  README and `scripts/build_phase1_qa_fixtures.py`.
+- Expanded `scripts/run_source_qa.py` to validate PRISM, gridMET maximum
+  temperature, and Sentinel-2 B04/B08 plus derived NDVI. Checks cover checksum,
+  source, CRS, dates, finite data, coordinate order and resolution, bounds,
+  broad physical/scale limits, and source-specific invariants. It writes JSON,
+  figures, and a consolidated manifest.
+- Replaced gridMET, Sentinel-2, and PRISM placeholder previews with reviewed
+  real-data QA figures and documented exact evidence and limitations.
+- Removed older FIRED, Landsat, and fire-workflow text files masquerading as
+  PNGs. Their pages now state the missing QA evidence explicitly instead of
+  displaying or soliciting an unvalidated screenshot; the asset policy now
+  prohibits synthetic or text image placeholders.
+- Found duplicate Sentinel-2 catalog records for identical acquisitions. The
+  loader now keeps the newest `s2:generation_time` record using STAC metadata
+  only, preserving lazy imagery, and records the selection in output attrs.
+- Corrected the Sentinel-2 citation from a stale MODIS reference to the
+  Copernicus/ESA Level-2A product.
+
+### Validation
+
+- A bounded live Sentinel-2 request to the Planetary Computer passed and
+  confirmed unique, strictly ordered acquisition times after deduplication.
+- Offline source QA passed for all three source adapters, and the gridMET and
+  Sentinel-2 figures were visually inspected for readable, plausible output.
+- The combined affected-area suite passed 84 tests. A final focused
+  deduplication/live request passed 2 tests; its only warning is an upstream
+  Planetary Computer Pydantic deprecation.
+- `scripts/run_decision_qa.py` passed. `scripts/run_vignettes.py` executed all
+  nine supported notebooks offline and confirmed their static plot outputs.
+- `mkdocs build --strict`, Python compilation, and `git diff --check` passed.
+
+### Limitations
+
+- The baselines validate the named variables, places, and time windows, not
+  every product permutation. gridMET maximum temperature is representative of
+  the adapter; Sentinel-2 pixel-level cloud masking is not yet implemented.
+- gridMET still downloads annual files before local AOI selection. Live tests
+  remain separate from deterministic offline fixture checks.

README.md

@@ -77,9 +77,13 @@ cube = data.temperature(
 
 result = (
     pipe(cube)
-    | v.anomaly(dim="time")
-    | v.variance(dim="time", keep_dim=False)
-).unwrap()
+    | v.anomaly(over="time")
+    | v.variance(over="time", keep_dim=False)
+)
+
+print(result.explain())
+validated = result.validate()
+summary = result.unwrap()
 ```
 
 Loading names the scientific noun and source flavor. The pipe stays focused on
@@ -111,6 +115,7 @@ clipped = (pipe(cube) | clip_values(0, 1)).unwrap()
 Key entry points:
 
 - [Core grammar versus project verbs](https://cu-esiil.github.io/cubedynamics/concepts/core_and_projects/)
+- [Semantic grammar and analysis coaching](https://cu-esiil.github.io/cubedynamics/concepts/semantic_grammar/)
 - [Narrative, executable vignettes](https://cu-esiil.github.io/cubedynamics/vignettes/)
 - [Write a custom verb project](https://cu-esiil.github.io/cubedynamics/extending/custom_verbs/)
 - [Public API and stability](https://cu-esiil.github.io/cubedynamics/project/public_api/)

data/source_qa/README.md

@@ -0,0 +1,33 @@
+# Phase 1 real-data QA fixtures
+
+These small NetCDF files are bounded extracts from observational products used
+to keep publication QA deterministic and offline-capable. They are not
+synthetic data and are not alternative production backends.
+
+| Fixture | Product | Coverage | Purpose |
+| --- | --- | --- | --- |
+| `gridmet_badlands_july_2001.nc` | gridMET daily maximum temperature | Badlands, South Dakota; 10 days in July 2001 | Source, grid, time, units, range, and visual checks |
+| `sentinel2_badlands_june_2023.nc` | Sentinel-2 L2A B04/B08 reflectance | Badlands, South Dakota; bounded June 2023 scenes | Source, CRS, bands, acquisition identity, scale, NDVI, and visual checks |
+
+Each fixture has a sibling provenance JSON containing its source request,
+creation method, and SHA-256 checksum. The QA runner refuses a fixture whose
+bytes no longer match that record.
+
+PRISM uses the existing real-data fixture in `data/vignettes/`. To rebuild the
+two fixtures in this directory, run:
+
+```bash
+python scripts/build_phase1_qa_fixtures.py
+```
+
+That command uses public upstream services and therefore requires network
+access. Routine validation is fully offline:
+
+```bash
+python scripts/run_source_qa.py
+```
+
+The reviewed report is published at `docs/data/phase1_qa.md`. A passing
+fixture is evidence for the exact product, variable, place, and time window
+listed above; it is not a claim that every variable or geography has received
+scientific validation.

data/source_qa/gridmet_badlands_july_2001.provenance.json

@@ -0,0 +1,20 @@
+{
+  "created_at": "2026-08-26T12:55:49.518657+00:00",
+  "fixture_sha256": "a36b2c4e1164d0a85429bc855fd94b32eb708a0f72e6be96f4959fc92bfe2721",
+  "product": "gridMET daily maximum temperature",
+  "provider": "University of Idaho",
+  "query": {
+    "bbox": [
+      -102.5,
+      43.6,
+      -101.9,
+      44.2
+    ],
+    "end": "2001-07-10",
+    "start": "2001-07-01",
+    "variable": "tmmx"
+  },
+  "source_file_sha256": "a0e07daeb1aa8b58b66f01dcc99e776ab71a163aa80dbef448c972b74fed313a",
+  "source_url": "https://www.northwestknowledge.net/metdata/data/tmmx_2001.nc",
+  "transformation": "cropped and renamed dimensions/variable; values unchanged"
+}

data/source_qa/sentinel2_badlands_june_2023.provenance.json

@@ -0,0 +1,29 @@
+{
+  "catalog": "Microsoft Planetary Computer STAC",
+  "fixture_sha256": "eb7c02b01bc741483d28714f7081e08639282685705d5d7a15ff44bf7e97672a",
+  "product": "Sentinel-2 Level-2A surface reflectance",
+  "provider": "European Union Copernicus Programme / ESA",
+  "query": {
+    "bands": [
+      "B04",
+      "B08"
+    ],
+    "cloud_lt": 80,
+    "edge_size": 64,
+    "end": "2023-06-10",
+    "lat": 43.89,
+    "lon": -102.18,
+    "resolution": 10,
+    "start": "2023-06-01"
+  },
+  "retrieved_at": "2026-08-26T12:55:51.112347+00:00",
+  "scene_cloud_cover_percent": [
+    72.819281,
+    13.389252
+  ],
+  "scene_ids": [
+    "S2B_MSIL2A_20230602T173909_R098_T13TGJ_20240911T145543",
+    "S2A_MSIL2A_20230607T173911_R098_T13TGJ_20240927T100945"
+  ],
+  "transformation": "selected B04/B08, cropped to 640 m square, converted to float32; values unchanged"
+}

docs/api/pipe.md

@@ -12,14 +12,18 @@ surface.
 
 1. Wrap a value with :func:`cubedynamics.pipe` to opt into piping.
 2. Apply verbs or callables with ``|``; each stage receives the previous value.
-3. Finish with :meth:`Pipe.unwrap` (or rely on rich reprs in notebooks).
+3. Optionally inspect the semantic trace with ``explain()``, ``suggest()``, or
+   ``validate()``.
+4. Finish with :meth:`Pipe.unwrap` (or rely on rich reprs in notebooks).
 
 ```python
 import cubedynamics as cd
 from cubedynamics import pipe, verbs as v
 
 cube = cd.gridmet(lat=40.0, lon=-105.2, start="2020-06-01", end="2020-07-01", variable="tmmx")
-result = (pipe(cube) | v.mean(dim="time") | v.zscore(dim="time")).unwrap()
+analysis = pipe(cube) | v.anomaly(over="time") | v.mean(over=("y", "x"))
+print(analysis.explain())
+result = analysis.unwrap()
 ```
 
 Rich repr example that keeps the viewer attached:
@@ -44,6 +48,15 @@ Container holding the wrapped value. Key methods:
 * ``__or__(func)`` – apply ``func`` to the wrapped value and return a new Pipe; respects passthrough verbs that attach viewers without replacing the object.
 * ``unwrap()`` – return the wrapped value.
 * ``v`` – property alias for the wrapped value.
+* ``semantic_state`` – metadata-only description of the current result.
+* ``semantic_trace`` – immutable record of executed pipe stages.
+* ``explain()`` – deterministic plain-language read-back of the analysis.
+* ``suggest()`` – a short list of compatible, implemented next verbs.
+* ``validate()`` – structured metadata, dimension, provenance, and order checks.
+
+These inspection methods never reorder or execute pipeline stages. The result
+of each verb remains the object on which the next written verb acts. See
+[Semantic grammar and analysis coaching](../concepts/semantic_grammar.md).
 
 ### ``cubedynamics.piping.Verb``
 Lightweight wrapper for callables used in pipe chains. A verb can flag itself as

docs/assets/datasets/README.md

@@ -1,8 +1,9 @@
 # Dataset preview assets
 
-Each dataset page in `docs/datasets/` references a static PNG preview at
-`docs/assets/datasets/<dataset-slug>-preview.png`.
+Dataset pages may reference reviewed static figures stored here or in
+`docs/assets/source_qa/`. Every image must be a genuine render from the named
+real dataset and accompanied by reproducible source/provenance and QA details.
 
-Codex does not add image binaries. After running the regeneration steps on a
-dataset page, save a 1200×700 px screenshot to the corresponding path so the
-placeholder link renders on the documentation site.
+Do not add text placeholders with image extensions. If reviewed visual
+evidence is not available, the dataset page should say so explicitly and omit
+the image until the source-specific QA is complete.