[WIP] Add Rule: usage constraints as their own content-addressed entity

docs/comparison-with-undata.md

@@ -0,0 +1,51 @@
+# NeuroGhost vs undata
+
+Both tackle the same core problem — neuroscience data standards (BIDS, NWB,
+DANDI, openMINDS, AIND, ...) each redefine the same concepts under different
+names, and there's no machine-readable way to know two things are "the same".
+undata (`/Users/dorota/repronim_various/undata`) is a much larger, production-
+platform take on this; NeuroGhost deliberately borrows its core identity idea
+and leaves the rest out. This is a quick reference for what's shared and what
+isn't, and why.
+
+## Adopted from undata
+
+| Idea | undata | NeuroGhost |
+|---|---|---|
+| Content-addressed identity | `sha256` hash from semantic content, two-mode (ontology-anchored / structural fallback) | Same principle, **single mode only** — structural content hash (`name`/`description`/`range`/`units`/etc.), no ontology-anchored mode (see below) |
+| Identity ≠ provenance | `ProvenanceEntry` list per entity, accumulates across sources | Same — `ProvenanceEntry` list, same idea, thinner field set (see below) |
+| LinkML as intermediate representation | Every adapter emits a `SchemaDefinition`; one standard extractor classifies | Every converter emits LinkML YAML; `parse_linkml()`/`build_registry_entities()` does the extraction+hashing |
+| PROV-O–grounded provenance fields | `generated_at`, `attributed_to`, `activity`, `derived_from` | Same four fields, same PROV-O predicates (`slot_uri: prov:...`) |
+
+## Deliberately not adopted
+
+| undata has | NeuroGhost's choice | Why |
+|---|---|---|
+| Two-mode hashing (ontology-anchored vs structural) | Structural only | Explicit decision: no ontology store, no ontology-anchored identity, for now |
+| Align **before** Commit (alignment can inform/merge the hash) | Align **after** Commit (`align.py` runs post-ingestion) | No complex alignment exists yet — pure hash equality already handles exact duplicates; align-before-commit only matters once alignment can detect and merge *near*-duplicates |
+| Knowledge service (13+ ontologies, embeddings, LLM verification, multi-precision SKOS) | `SkosMapping` model field exists but isn't wired into ingestion at all | Same reason as above — no ontology grounding in scope |
+| `StorageBackend` protocol (pluggable file/Postgres backends) | One embedded LadybugDB file, no abstraction | NeuroGhost is a single-file registry driven by a GitHub Action, not a multi-user service |
+| Task manager, GraphQL API, OIDC auth, curator roles, staged→curated lifecycle | None of it | All production-platform machinery for a hosted, multi-user service; out of scope |
+| `ProvenanceEntry.class`/`name` (which source-schema class/slot this came from) and `source_ref` (repo/commit/file/checksum) | Considered, then dropped — see `docs/ingestion.md`'s note on `source_class`/`source_slot` | Redundant with graph structure (`HAS_PROPERTY` edges) or not yet needed |
+
+## Where NeuroGhost has gone further
+
+**`Rule`** (usage constraints — `required`/`multivalued`/`pattern`/min/max —
+as their own content-addressed entity, referencing the `RegistryProperty` via
+`applies_to`) isn't something undata's VISION.md describes as a distinct
+entity type. It falls out of the same "identity ≠ usage" principle undata
+established for provenance, just applied one level further: a property's
+*core meaning* is separate not only from *where it came from* but also from
+*how a given source happens to use it*. See `docs/ingestion.md`'s
+[Rule section](ingestion.md#rule-usage-constraints-as-their-own-entity).
+
+## Entity type mapping
+
+| undata | NeuroGhost | Status |
+|---|---|---|
+| `Element` | `RegistryProperty` | Built |
+| `Schema` | `RegistryClass` | Built |
+| — | `Rule` | Built (NeuroGhost-specific, see above) |
+| `Value` / `ValueSet` | `ValueSet` | Stub only |
+| `Transform` | `Transform` | Stub only |
+| `OntologyAnnotation` (multi-precision SKOS) | `SkosMapping` | Modeled, not wired into ingestion |

docs/ingestion.md

@@ -54,9 +54,10 @@ LinkML YAML
 intermediate dict            {classes: {...}, slots: {...}}
     │  build_registry_entities()  — compute content hashes
     ▼
-RegistryProperty / RegistryClass instances (Pydantic, hash_id already set)
+RegistryProperty / RegistryClass / Rule instances (Pydantic, hash_id already set)
     │  write_registry_entities()  — write-if-new, attach-provenance-if-new
     │  write_structural_edges()   — HAS_PROPERTY, SUBCLASS_OF
+    │  write_rule_edges()         — APPLIES_TO_P
     ▼
 LadybugDB graph
 ```
@@ -92,8 +93,12 @@ Converts the dict into real, hash-identified objects:
 - **Every entity gets one `ProvenanceEntry`** for this ingestion — `source`,
   `attributed_to` (agent), `generated_at`, `activity`, `registry_version` —
   entirely separate from the hash computation.
+- **A property that declares `required`/`multivalued`/`pattern`/min/max also
+  gets a `Rule`**, `applies_to` = that property's `hash_id`. A plain property
+  with none of these gets no `Rule` at all — see [Rule](#rule-usage-constraints-as-their-own-entity)
+  below.
 
-### 3. `write_registry_entities()` + `write_structural_edges()` (`neuro_ghost/db.py`)
+### 3. `write_registry_entities()` + `write_structural_edges()` + `write_rule_edges()` (`neuro_ghost/db.py`)
 
 For each entity: does a node with this `hash_id` already exist?
 - **No** → create it.
@@ -102,9 +107,11 @@ For each entity: does a node with this `hash_id` already exist?
   the same file from the same source twice adds nothing the second time).
 
 Then `HAS_PROPERTY` and `SUBCLASS_OF` edges get created from the resolved
-hash references. These two functions are shared between `ingest_linkml.py`
+hash references, and `APPLIES_TO_P` from each `Rule` to the property it
+constrains. All three writer functions are shared between `ingest_linkml.py`
 and `seed.py` — schema.org is ingested through the exact same path, just with
-`source="schema.org"`.
+`source="schema.org"` (though `seed.py` never builds any `Rule`s — schema.org's
+base vocabulary doesn't declare `required`/`multivalued`/etc. on its slots).
 
 ## The data model
 
@@ -117,16 +124,47 @@ and `seed.py` — schema.org is ingested through the exact same path, just with
 | `class_uri` / `slot_uri` | `RegistryClass` / `RegistryProperty` | Ontology IRI preserved from the source. **Not** part of the hash — two schemas using different IRIs (or none) for the same content still collapse to one node. |
 | `provenance` | both | List of `ProvenanceEntry`. Accumulates, never affects `hash_id`. |
 | `source`, `attributed_to`, `generated_at`, `activity`, `derived_from`, `registry_version` | `ProvenanceEntry` | PROV-O–grounded (`slot_uri: prov:wasAttributedTo` etc.). `registry_version` lives here, not on the entity — the same entity can be attested by different sources at different times, each under a different registry version, so a single scalar on the entity doesn't fit (same reasoning as dropping `source_label`). |
+| `applies_to`, `required`, `multivalued`, `pattern`, `minimum_value`, `maximum_value` | `Rule` | Identity-defining — see [Rule](#rule-usage-constraints-as-their-own-entity). |
 
 Field names were deliberately aligned with LinkML's own metamodel
 (`description`, `range`, `class_uri`/`slot_uri`, `is_a`, `abstract`) rather
 than inventing parallel terminology — e.g. `parse_linkml()` already produces
 `is_a` straight from `SchemaView`, so there's no translation step.
 
-**Deliberately not modeled yet:** `required`/`multivalued` used to live on
-`RegistryProperty` directly, which meant a property required in one schema's
-usage and optional in another's could never share a hash. They've been
-removed entirely — that's a **`Rule`** concern (still a stub), not identity.
+## Rule: usage constraints as their own entity
+
+`required`/`multivalued` used to live on `RegistryProperty` directly, which
+meant a property required in one schema's usage and optional in another's
+could never share a `hash_id`. They're not modeled on `RegistryProperty` at
+all now — a `Rule` is a separate content-addressed entity that references
+the property it constrains:
+
+```
+RegistryProperty "age"  (hash_id = A)
+        ▲
+        │ APPLIES_TO_P
+        │
+Rule { applies_to: A, required: true }  (hash_id = B)
+        ├── ProvenanceEntry{source: "bids", ...}
+        └── ProvenanceEntry{source: "dandi", ...}
+```
+
+- `applies_to` (the constrained property's `hash_id`) **is** part of the
+  `Rule`'s own hash — identical constraints on two *different* properties
+  are different rules, by design.
+- Same collapsing behavior as everything else: two sources declaring the
+  exact same constraints on the exact same property produce one `Rule` node
+  with two `ProvenanceEntry` records, not two `Rule`s
+  (`test_identical_rule_from_two_sources_shares_one_hash_id`).
+- A property with none of `required`/`multivalued`/`pattern`/min/max gets no
+  `Rule` at all — `build_registry_entities()`'s `_has_constraints()` gate
+  keeps unconstrained properties from accumulating empty, meaningless rules.
+- Scoped to `RegistryProperty` only for now (`applies_to: RegistryProperty`
+  in the meta-model) — class-level rules are a possible future extension,
+  not built.
+- Written through the same `write_registry_entities()`/`entity_exists()`/
+  `write_provenance()` functions as `RegistryClass`/`RegistryProperty` (now
+  with an optional third `rules` argument) — no separate code path.
 
 ## Alignment
 
@@ -169,7 +207,12 @@ the other layer does with it.
 takes this one step further, end to end: two schemas declare the exact same
 `age` slot except one marks it `required: true`. Ingesting both must produce
 exactly one `RegistryProperty` node, not two — proving `required` doesn't
-leak into identity, in the real graph, not just in an isolated object.
+leak into identity, in the real graph, not just in an isolated object. The
+same test also confirms where `required: true` *does* end up: one `Rule`,
+`APPLIES_TO_P` the "age" property.
+`test_identical_rule_from_two_sources_shares_one_hash_id` proves the
+content-addressing guarantee applies to `Rule` itself, the same way it does
+to `RegistryProperty`/`RegistryClass`.
 
 ## Open question: when should ProvenanceEntry.registry_version be set?
 
@@ -201,8 +244,8 @@ registry_version` stays `None` for now, pending a decision on the above.
 - **`derived_from`** on `ProvenanceEntry` is never populated — nothing yet
   detects "this hash supersedes that one" (would need an anchor like
   `(name, source)` to correlate an edit against prior content).
-- **`Rule`/`Transform`/`ValueSet`** are stubs (`hash_id`/`name`/`description`
-  only).
+- **`Transform`/`ValueSet`** are still stubs (`hash_id`/`name`/`description`
+  only) — `Rule` is real now (see above).
 - **`SemanticIdentity`/`PRIOR_VERSION*`** tables in `db.py`'s DDL are dead
   (superseded by content-hash identity) but not yet removed.
 - **`pandas`** isn't in `requirements.txt`, so `align.py`'s embedding cache

neuro_ghost/db.py

@@ -85,7 +85,11 @@ def bump_version(ver: str, bump: str = "patch") -> str:
 # .derived_from) so this module doesn't need to import schema_registry_utils.
 
 LIST_FIELDS = {"provenance", "skos_mappings", "properties", "relations", "mixins"}
-HAS_PROVENANCE_REL = {"RegistryClass": "HAS_PROVENANCE", "RegistryProperty": "HAS_PROVENANCE_P"}
+HAS_PROVENANCE_REL = {
+    "RegistryClass":    "HAS_PROVENANCE",
+    "RegistryProperty": "HAS_PROVENANCE_P",
+    "Rule":             "HAS_PROVENANCE_RULE",
+}
 
 
 def scalar_fields(entity) -> dict:
@@ -145,19 +149,22 @@ def write_provenance(conn, label: str, hash_id: str, prov) -> bool:
 
 
 def write_registry_entities(conn, properties: dict, registry_classes: dict,
+                             rules: dict | None = None,
                              dry_run: bool = False) -> dict:
     """
-    Write (or reuse) each property/class node by hash_id, then attach this
-    ingestion's ProvenanceEntry to every one of them. Existing nodes are
-    never overwritten — a hash match means identical content, so there is
-    nothing to update; only a new ProvenanceEntry may need attaching.
-
-    `properties`/`registry_classes` are name -> entity dicts (values just
-    need .hash_id and .provenance; shared by ingest_linkml.py and seed.py).
+    Write (or reuse) each property/class/rule node by hash_id, then attach
+    this ingestion's ProvenanceEntry to every one of them. Existing nodes
+    are never overwritten — a hash match means identical content, so there
+    is nothing to update; only a new ProvenanceEntry may need attaching.
+
+    `properties`/`registry_classes`/`rules` are name -> entity dicts (values
+    just need .hash_id and .provenance; shared by ingest_linkml.py and
+    seed.py). `rules` is optional since seed.py doesn't build any yet.
     """
     stats = {
         "properties_new": 0, "properties_existing": 0,
         "classes_new":    0, "classes_existing":    0,
+        "rules_new":      0, "rules_existing":       0,
         "provenance_added": 0,
     }
 
@@ -181,9 +188,36 @@ def write_registry_entities(conn, properties: dict, registry_classes: dict,
                 if write_provenance(conn, "RegistryClass", rc.hash_id, prov):
                     stats["provenance_added"] += 1
 
+    for rule in (rules or {}).values():
+        is_new = not entity_exists(conn, "Rule", rule.hash_id)
+        if is_new and not dry_run:
+            create_entity_node(conn, "Rule", rule)
+        stats["rules_new" if is_new else "rules_existing"] += 1
+        if not dry_run:
+            for prov in rule.provenance:
+                if write_provenance(conn, "Rule", rule.hash_id, prov):
+                    stats["provenance_added"] += 1
+
     return stats
 
 
+def write_rule_edges(conn, rules: dict) -> int:
+    """APPLIES_TO_P: each Rule to the RegistryProperty it constrains."""
+    rels = 0
+    for rule in rules.values():
+        already = conn.execute("""
+            MATCH (r:Rule {hash_id: $r})-[:APPLIES_TO_P]->(p:RegistryProperty {hash_id: $p})
+            RETURN r.hash_id LIMIT 1
+        """, {"r": rule.hash_id, "p": rule.applies_to}).has_next()
+        if not already:
+            conn.execute("""
+                MATCH (r:Rule {hash_id: $r}), (p:RegistryProperty {hash_id: $p})
+                CREATE (r)-[:APPLIES_TO_P]->(p)
+            """, {"r": rule.hash_id, "p": rule.applies_to})
+            rels += 1
+    return rels
+
+
 def write_structural_edges(conn, registry_classes: dict) -> int:
     """
     HAS_PROPERTY (from each class's own `properties`) + SUBCLASS_OF (from
@@ -426,6 +460,7 @@ def _build_registry_ddl(yaml_path: str | Path = SCHEMA_YAML) -> list[str]:
     "CREATE REL TABLE IF NOT EXISTS HAS_PROVENANCE     (FROM RegistryClass    TO ProvenanceEntry)",
     "CREATE REL TABLE IF NOT EXISTS HAS_PROVENANCE_P   (FROM RegistryProperty TO ProvenanceEntry)",
     "CREATE REL TABLE IF NOT EXISTS HAS_PROVENANCE_R   (FROM Relation         TO ProvenanceEntry)",
+    "CREATE REL TABLE IF NOT EXISTS HAS_PROVENANCE_RULE (FROM Rule            TO ProvenanceEntry)",
     "CREATE REL TABLE IF NOT EXISTS MIXIN              (FROM RegistryClass    TO RegistryClass)",
     "CREATE REL TABLE IF NOT EXISTS SUBCLASS_OF        (FROM RegistryClass    TO RegistryClass)",
 
@@ -462,6 +497,7 @@ def _build_registry_ddl(yaml_path: str | Path = SCHEMA_YAML) -> list[str]:
 
     # --- Infrastructure edges ---
     "CREATE REL TABLE IF NOT EXISTS APPLIES_TO         (FROM Rule             TO RegistryClass)",
+    "CREATE REL TABLE IF NOT EXISTS APPLIES_TO_P       (FROM Rule             TO RegistryProperty)",
     "CREATE REL TABLE IF NOT EXISTS PROV_GENERATED     (FROM RegistryClass    TO SchemaActivity)",
     "CREATE REL TABLE IF NOT EXISTS PROV_GENERATED_P   (FROM RegistryProperty TO SchemaActivity)",
     "CREATE REL TABLE IF NOT EXISTS PROV_GENERATED_R   (FROM Rule             TO SchemaActivity)",