Improve ingestion & query service#75
Open
tekrajchhetri wants to merge 16 commits into
Open
Conversation
Track all mutating actions (ingestion, named-graph registration, crash recovery) as W3C PROV-O triples in a dedicated provenance named graph (https://brainkb.org/provenance/), queryable via SPARQL. Postgres keeps job execution state; Oxigraph is the provenance source of truth. - core/provenance.py: PROV-O builders (IngestionActivity/RegistrationActivity/ RecoveryActivity) with typed user/system agents, plus Graph Store HTTP write and CONSTRUCT->JSON-LD retrieval helpers. Writes are best-effort. - insert.py: emit provenance from run_ingest_job (all terminal states), create_named_graph, and recover_stuck_jobs; stop embedding PROV into domain data (files upload unmodified); add GET /provenance/job and /provenance/named-graph (application/ld+json). - PROVENANCE_MODEL.md: design and reference.
- query_provenance_jsonld: Oxigraph returns HTTP 406 for Accept application/ld+json (it serializes Turtle/N-Triples/N-Quads/RDF-XML only), so request Turtle and convert to JSON-LD locally with rdflib. Keeps the JSON-LD API contract independent of the triplestore's output formats. - construct_for_job: also traverse inbound prov:used so a job's provenance bundle includes the recovery activity (and its system agent) that acted on the job, not just forward links.
Track which triples each ingestion job adds, not just activity-level metadata. Validated end-to-end against a live Oxigraph. - Each job stages triples in a per-job delta graph (https://brainkb.org/provenance/delta/{job_id}), then merges into the target via SPARQL ADD (idempotent set union). The delta graph is preserved as the exact change record; a brainkb:IngestionDelta PROV-O entity records the derivation, target, delta graph, and added triple count. - Gated by TRACK_TRIPLE_DELTAS (default on); disable to upload directly to the target (delta graphs persist, roughly doubling stored triples). - New endpoints: GET /provenance/delta (added triples as JSON-LD), /provenance/delta/history (change history for a graph), /provenance/delta/compare (diff two jobs' deltas: A-only/B-only/shared). - provenance.py: delta_graph_for, merge_delta_into_target, count_graph_triples, construct_delta_content, delta_history_for_graph, compare_deltas; job CONSTRUCT now includes the delta entity. - PROVENANCE_MODEL.md: document the delta model and endpoints.
Registration was recorded twice: once in the named-graph registry graph (metadata/named-graph, via named_graph_metadata) and again as a separate RegistrationActivity in the provenance graph. Both asserted the registration timestamp and made the graph IRI a subject. Keep the registry graph as the single home for registration facts and attribute it to the registering user there (prov:wasAttributedTo); drop the duplicate RegistrationActivity from the provenance graph. - shared.py: named_graph_metadata takes an optional agent_uri and adds prov:wasAttributedTo on the registry entry. - insert.py: create_named_graph passes the agent URI; remove the duplicate build_registration_provenance write (and its import). - provenance.py: remove build_registration_provenance (now unused). - query.py: /query/registered-named-graphs returns registered_by. - PROVENANCE_MODEL.md: document registration living in the registry graph.
Scopes (require_scopes) — GET reads require 'read', mutations require 'write':
- read added: GET /insert/jobs, /insert/user/jobs/detail,
/insert/jobs/check-recoverable, /provenance/job, /provenance/named-graph,
/provenance/delta, /provenance/delta/history, /provenance/delta/compare,
/query/registered-named-graphs
- write added: POST /insert/jobs/recover, POST /register-named-graph
- unchanged: /insert/{raw,files}/knowledge-graph-triples (write),
/query/taxonomy (read), /query/sparql/ (write+admin, arbitrary query),
/register + /token (public)
Docstrings: document the difference between /query/registered-named-graphs
(the registry/catalog of graphs) and /provenance/named-graph (the PROV-O
ingestion/activity history of a graph); expand descriptions on all provenance
and delta endpoints.
Arbitrary SPARQL is a powerful, unrestricted capability; require the 'admin' scope (dropping the redundant 'write'), and keep it off the default 'read' scope used by fixed-shape read endpoints.
Users/teams create owner-controlled spaces, keep them private, or publish them publicly for anyone (incl. unauthenticated clients) to read. Supports a decentralized, IRI-addressable model (https://brainkb.org/space/{slug}). Storage split (confirmed): Postgres holds identity/teams/enforcement (spaces, space_members, space_graphs); Oxigraph holds all KG data + provenance plus a best-effort RDF mirror of each space manifest (metadata/spaces graph). - core/spaces.py: CRUD, per-request authorization (public=anonymous read; private=members; write=owner/editor), and SPARQL-Update RDF mirror. Legacy unmapped graphs fall through to existing scope checks (backward compatible). - core/routers/spaces.py: POST/GET /spaces, GET /spaces/{slug}, PATCH visibility, POST/DELETE members, POST graphs, GET /spaces/{slug}/data (public = anonymous). - security.py: get_current_user_optional (token used if present, never 401) for anonymous public reads. - insert.py: ingestion now enforces space write-authorization on the target graph, in addition to the write scope and user-identity check. - main.py: create spaces tables on startup; mount spaces router. - SPACES_MODEL.md: design + endpoints. Validated end-to-end against the live stack (15/15): private owner ingest/read, outsider ingest/read denied (403), anonymous read denied while private, public flip enables anonymous read + listing, public grants read-not-write, RDF mirror.
Stop leaking private graph existence via the registry listing: graphs in a private space the caller is not a member of are omitted. Public-space and legacy (unmapped) graphs remain listed. The endpoint now loads the caller identity (get_current_user) and excludes hidden graphs via spaces.hidden_graphs_for(). Also clarify in SPACES_MODEL.md that job-scoped provenance is intentionally owner-only and ingestion is always restricted to activated JWT users with valid credentials + space owner/editor membership (never anonymous); only reads of public spaces are anonymous. Validated live: owner sees private+public graphs; non-member sees only public.
- query_service/README.md: document auth/scopes, ingestion + jobs, PROV-O provenance, triple-level delta endpoints, and private/public spaces; add architecture note (Postgres = identity/enforcement, Oxigraph = graph data + provenance). - readme.md: expand the Query Service bullet to mention provenance, deltas, and spaces, with pointers to the model docs.
Search over the knowledge graphs that respects space visibility. Hybrid design: Postgres holds a full-text locator index (graph_search_index: subject, text, named graph, owning space) populated at ingest; a query runs in Postgres (fast, filtered by space visibility/membership) to locate subjects, then the matched triples are fetched from Oxigraph (the source of truth for KG data). - core/search.py: index_graph_subjects (indexes a graph's/-delta's subject literals), reindex_graph_space, and search() with the access filter (anonymous -> public spaces only; authenticated -> public + member spaces + legacy). Data for the located subjects is fetched from Oxigraph as JSON-LD. - main.py: create graph_search_index (GIN full-text) on startup; mount router. - run_ingest_job: index the target graph's subjects after merge (best-effort). - spaces.attach_graph: point existing index rows at the space so search picks up the workspace immediately (inline SQL to avoid an import cycle). - routers/search.py: GET /search (optional auth; space-scoped or full). - READMEs updated. Validated live (9/9): anon finds public term + data from Oxigraph; anon/outsider cannot see private term; owner finds private; scoped search respects membership; anon scoped to a private space returns nothing.
Indexing a large graph inline was slow and delayed ingest jobs. Move it off the ingest path into an in-process async task queue with durable status. - core/indexing.py: asyncio queue + single background consumer, durable index_tasks table, atomic queued->running claim (safe across gunicorn workers), and startup recovery (re-queue tasks left over from a crash). Supports 'ingest' (one graph) and 'backfill' (reindex every user graph, with progress). - main.py: create index_tasks table; start the consumer on startup. - insert.py: ingest now ENQUEUES indexing (non-blocking) instead of awaiting it, so jobs finish without waiting on indexing. - routers/search.py: POST /search/reindex (admin, background backfill) and GET /search/index-tasks (status). - README updated. Validated live: ingest job completes in ~1s while indexing runs in background; background task indexes the subject and it becomes searchable; backfill reindexed 3/3 graphs in the background.
Ingestion stays submit-and-forget/background, but a burst of concurrent submissions could previously run unbounded and exhaust memory / the DB pool / Oxigraph and crash the worker. Add a per-worker asyncio.Semaphore limiter (MAX_CONCURRENT_INGEST_JOBS, default 3): run_ingest_job now acquires a slot before processing; excess jobs return immediately and wait as 'pending' until a slot frees (backpressure without a queue). Effective global cap ~= cap x workers. Validated live: 6 concurrent submissions all accepted in ~0.07s (submit-and- forget intact) and all completed 'done', throttled, no crash.
SPARQL (Oxigraph) + SQL (Postgres) queries to verify ingested graphs, provenance, per-job deltas, spaces manifest/membership, and the search index, with instructions to run via the API or directly against Oxigraph/Postgres.
…I access only Authorization now comes from the user's roles (joined by email to Web_user_profile -> Web_user_role), mapped to capabilities, layered with space membership. JWT scopes remain only an API-access gate. - core/rbac.py: roles->capabilities policy; delegated grants (user_capability_grants); SuperAdmin>=Admin>write>read>none hierarchy; admin-intrinsic caps (grant, sparql_admin) are NOT delegatable (no escalation); query_service never assigns roles (role assignment stays Django-owned). - Space types: 'individual' (any write-capable user) vs 'team' (Admin/SuperAdmin or granted create_team_space). - Enforcement: create space (by type), ingest, recover, arbitrary SPARQL, space management, and reads (no-role -> public content only). - Admin endpoints: GET/POST /admin/capabilities[/grant|/revoke]. - main.py: space_type column + user_capability_grants table. - RBAC_MODEL.md: full model. Validated live (14/14): no-role denied (public read only); Lab Member creates private + ingests but not team; Admin creates team + SPARQL; delegated grant upgrades Lab Member to create team spaces; non-admins can't grant; admin-intrinsic caps rejected for delegation.
Within a space, restrict an action to a global role, a space role, or specific
members — e.g. 'only Admins may write here', 'only these Lab Members may read',
'let this member manage'. Layers on top of capabilities + owner/editor/viewer
membership; owner and global Admin/SuperAdmin always bypass (no lockout).
- space_access_rules table (action, subject_type[global_role|member|space_role],
subject_value).
- spaces.py: rule CRUD, matches_access_rule (pure match) and
space_action_permitted (owner/admin bypass; no-rules -> allow for read/write).
- Enforced on: reads (get space / space data), ingest (insert raw+files), and
manage (members/visibility/graphs, via _can_manage grant).
- Endpoints: GET/POST/DELETE /spaces/{slug}/access-rules (manager only; GET
member/manager).
- RBAC_MODEL.md updated.
Validated live (9/9): write Admin-only rule blocks a Lab Member editor but owner
bypasses; member rule then allows the Lab Member; read Admin-only rule blocks a
member read (owner bypass); manage rule grants a member management.
POST /api/admin/users/activate and /deactivate (by email), Admin-gated, using the existing jwt_user_repo.activate_user/deactivate_user. Enables admins to activate accounts via API (e.g. after password self-registration) instead of only the UI/DB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes several improvements to existing implementations, such as introducing private, public graph/space. Some major changes are: