Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/execute.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,10 @@ not opened until that code requests it. Core values otherwise lower to exact CDe
index in the frozen Store table, never a path. The worker reconstructs one local
owner boundary: function, Method, and managed owners each deliver/invoke/normalize
once rather than gaining a second generic signature boundary.
Transient `Ref(value)` and `Mat(value)` assertions are invocation-only graph
nodes. They cross intact to that worker-owned boundary and must be consumed by
signature activation there; unconsumed assertions remain forbidden in result
graphs and persistent formats.

Core Execute does not inject a root `ManagedConfig` into a managed target. Direct
managed targets and managed calls nested inside ordinary transported callables use
Expand Down Expand Up @@ -327,7 +331,9 @@ earlier restores nor replays the workload.
Store-table-relative publication status only; it never carries a live Repo,
Store, `StoreReport`, argument, or result. A delivered worker failure preserves
known publication evidence but does not imply rollback, retry, or successful
caller refresh.
caller refresh. Coordinator recovery reports this bounded category as
`core execution worker failed: <reason>` without including submitted values or
remote exception payloads.

`PreparedCoreCall` contains only invocation bytes, frozen storage setup, and
opaque update descriptors. It never retains caller Objects or refresh progress.
Expand Down
79 changes: 79 additions & 0 deletions docs/graph_querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,85 @@ Graph traversal records typed V2 `Parameter` and container paths. Materializing

Store indexes are acceleration only. Rebuild scans authoritative definition and reference records, announces visible progress, and can safely replace a missing or stale derived index. A query may fail closed when current metadata is incompatible; it never treats an incompatible index as empty or current authority.

## Reference-Aware Containment

Nested containment finds authoritative stored CDef roots with a non-empty typed
path to a target. The CDef entry is
`repo.query(target).nested(...)`; `target` may be a concrete CDef, an exact
`ObjectRef`, or an exact `StateRef`. The reference entry is
`repo.references().containing(target, ...)`. It is an adapter for the same
immutable nested query, so an unfiltered reference builder produces the same
owners, occurrences, policies, source scope, and explanation as the CDef entry.
It rejects already-filtered reference builders rather than reinterpreting an
authority predicate as containment. Ordinary `Repo.references()` lookup,
including `contains`, `exact`, aliases, and metadata predicates, is unchanged.

`edges="materialize"` is the compatibility default and permits only owning
materializing hops. `edges="ref"` permits only retained `Ref` hops, and
`edges="all"` permits either kind at every hop. `contains_ref=False` is the
default. Set `contains_ref=True` to retain only paths that contain at least one
selected `Ref` hop; it narrows the selected edge policy and never widens it.
Therefore `edges="materialize", contains_ref=True` is deterministically empty,
while `edges="ref", contains_ref=True` has the same non-empty path membership
as `edges="ref"`. An independently stored target does not match through its own
empty root path; it qualifies only through another stored root's non-empty path.

```python
from dryml.artifacts import CachedDataset
from dryml.core import Definition

# This is definition-only: construction and the query do not open the Dataset
# or compute the cache. `source` need not be an independently stored root.
source = Definition(MyDataset, "training").concretize()
cached = CachedDataset(source, repo=repo)
repo.save_object(cached)

owners = (
repo.query(source)
.nested(edges="ref")
.owners()
.defs()
)
assert list(owners) == [cached.definition]
```

For an exact reference target, `object_refs()` and `state_refs()` return only
their respective complete target identities. A StateRef comparison includes its
complete ObjectRef and state identity; an ObjectRef comparison includes its
complete object identity. Neither terminal coerces an exact reference into its
CDef, so a different checkpoint of the same object, or a different object with
the same CDef, does not match. `owners().defs()` returns the enclosing stored
CDefs for any target kind. Exact references are terminal values: containment
does not load Objects, read payloads, resolve target authority, or infer that a
target is independently stored, loadable, restorable, or eligible for cleanup.

Raw nested execution returns one occurrence per qualifying owner-to-target
path. Each occurrence retains the target plus `hops`, whose ordered entries pair
the direct typed path segment with its literal `materialize` or `ref` kind. This
distinguishes mixed paths such as `ref -> materialize -> ref` without treating a
reference on an unrelated branch as evidence for a materializing path.

Use `in_store(store)` after `nested()` or on the unfiltered reference builder
before `containing()` to restrict roots to that exact connected Store handle.
Source restriction happens before replica merging, counts, and occurrence
limits. Owners and target projections are structurally or completely typed
deduplicated as appropriate; identical replica witnesses merge their source
evidence. Results use canonical owner/path/hop/terminal ordering. A raw
`max_occurrences(n)` cap is one global, post-deduplication truncation, not a
claim that the graph is complete. Direct owner and exact-target projections are
existential and are not truncated by that raw-path cap; projections made from an
already bounded occurrence result remain bounded.

Reference-aware, reference-bearing, and exact-reference containment performs
authoritative root verification. `scan_policy("allow")` permits it and
`scan_policy("warn")` emits the existing warning before the scan;
`scan_policy("forbid")` and `require_indexed()` reject it. `explain()` reports
the target kind, edge policy, reference filter, source scope, scan reason, and
derived index generation evidence without performing the required residual
scan unless `analyze=True`. The legacy eligible materialize-only CDef paths keep
their indexed behavior. Store indexes remain candidate-only derived state, never
authority for root membership or a no-match conclusion.

## Template Selectors

`Template.as_selector()` creates a loose ordinary Selector that preserves known
Expand Down
35 changes: 35 additions & 0 deletions docs/query_index_backend_contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,38 @@ SQLite connections are process-local. Rebuild and mutation use staged activation
`dryml.core.query.sqlite.open_connection(config, *, readonly=False)` is the public standalone connection owner. `config` must be a `SQLiteQueryIndexConfig`, `readonly` must be exactly `bool`, and `config.path` must be present; validation occurs before importing SQLite or creating a parent/database. Invalid argument types raise `TypeError`, while an absent configured path raises actionable `QueryIndexError`. The connection applies the same foreign-key, timeout, trusted-schema, journal, and durability configuration as the query-index connection manager and always closes its connection. Writable opening creates missing parent directories; read-only opening requires an existing database and creates neither the target nor its parent. SQLite open/configuration errors remain SQLite errors, while a missing optional backend is reported as `QueryIndexUnavailable`.

The outer `open_connection` context is a handle scope, not a commit scope. Its connection uses standard-library deferred transactions and creator-thread affinity; callers use an inner `with con:` block to commit on successful exit or roll back on exceptional exit. When the outer scope ends, any transaction still open is rolled back rather than committed, then the handle is closed on normal exit, exceptions, and cancellation. Manager-cached index connections retain autocommit mode because index read views and mutations issue explicit `BEGIN`, `BEGIN IMMEDIATE`, `COMMIT`, and `ROLLBACK` statements. Manager cache keys remain process/thread/access-mode local and handles are never returned across keys. Cached handles disable SQLite creator-thread enforcement only so manager teardown can close an inactive worker's handles from a coordinator thread; callers must not race handle use with teardown, and a failed close remains visible and retained for recovery.

## Reference-Aware Containment

Reference-aware nested queries do not add persisted rows, schema versions, or a
second authority format. Existing sidecars can remain in place and reopen with
their current compatibility rules. Missing, dirty, corrupt, stale, or
incompatible sidecars follow the existing visible rebuild or fallback policy;
they never replace current authoritative root capture or turn a required
containment scan into an empty result.

Eligible materialize-only CDef containment continues to use the indexed
definition/owner and captured materializing-edge paths. Reference-inclusive,
reference-bearing, and exact ObjectRef/StateRef containment instead captures
each selected Store's authoritative roots inside its read fence, then verifies
the retained graph in memory. The derived index can prioritize that work but
cannot exclude a root. `refresh=False` disables refresh, not this authority
check. Each source contributes one valid cut; federated queries may combine
separate source cuts, but no occurrence mixes two sources.

`in_store()` selects an exact connected source before capture, replica
deduplication, cardinality, ordering, or the global raw occurrence limit. A
reference-aware residual has the explanation scan reason
`reference-aware containment requires authoritative root verification`.
`scan_policy("allow")` permits the residual, `"warn"` reports it before the
first authority or recovery scan, and `"forbid"` and `require_indexed()` reject
it before recovery. The valid materialize-plus-reference-bearing combination is
deterministically empty and requires no scan.

The residual only inspects retained definition structure. Exact ObjectRef and
StateRef terminals retain their complete identities and declared hop kinds; they
are not dereferenced, loaded, or promoted to CDef children. Consequently an
occurrence is a definition-graph fact, not evidence of target ownership,
independent storage, payload availability, loadability, restoration, or cleanup
eligibility. Ordinary reference-authority lookup remains a separate query
contract.
5 changes: 4 additions & 1 deletion docs/signatures.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,10 @@ delivery. It recursively walks closure globals, defaults, annotations, instance
attributes, and `__slots__`; live Repo/Store resources are rejected rather than
being carried by a pickle fallback. Selected declaration Stores are represented by
the frozen execution Store table, never filesystem paths. This transport remains
internal and bounded; it is not a general pickle or RPC format.
internal and bounded; it is not a general pickle or RPC format. Explicit
`Ref(value)` and `Mat(value)` assertions remain transient while crossing an
invocation and are consumed by the reconstructed worker-local plan. They cannot
cross the result boundary or enter persistent identity data.

The callable graph includes globals, defaults, annotations, closure cells, bound
receivers, instance attributes, and `__slots__` in one pass, preserving aliases
Expand Down
2 changes: 2 additions & 0 deletions src/dryml/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@
"EdgeKind": "dryml.core.cdef_graph",
"iter_direct_cdef_edges": "dryml.core.cdef_graph",
"Arg": "dryml.core.query",
"ContainmentEdgePolicy": "dryml.core.query",
"DefinitionPath": "dryml.core.query",
"DefinitionQuery": "dryml.core.query",
"DefinitionResultSet": "dryml.core.query",
Expand Down Expand Up @@ -286,6 +287,7 @@ def __getattr__(name: str) -> object:
"EdgeKind",
"iter_direct_cdef_edges",
"Arg",
"ContainmentEdgePolicy",
"DefinitionPath",
"DefinitionQuery",
"DefinitionResultSet",
Expand Down
4 changes: 2 additions & 2 deletions src/dryml/core/execute.py
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,7 @@ def recover(
outcome = decoded.value
if not outcome["success"]:
raise CoreExecutionError(
f"core execution worker outcome failed: {outcome['reason']}",
f"core execution worker failed: {outcome['reason']}",
phase="invoke", evidence=decoded.evidence,
)
updates = tuple((item["target"], StateRef.from_data(item["state"])) for item in outcome["updates"])
Expand Down Expand Up @@ -761,7 +761,7 @@ def invoke_prepared_call(invocation: bytes) -> Any:
repo=context.repo,
)
if not outcome["success"]:
raise CoreCallCodecError(f"core execution worker outcome failed: {outcome['reason']}")
raise CoreCallCodecError(f"core execution worker failed: {outcome['reason']}")
return outcome["result"]


Expand Down
Loading
Loading