feat: query optimization spike#141
Open
mbarlow12 wants to merge 12 commits into
Open
Conversation
…gem>llm) SRC_PRIORITY and DEFAULT_PRIORITIES had gem before wm, disagreeing with the Alembic migration seed. Swap them to match the authoritative order. Add guard test to detect future drift across all four definitions.
…Task 1) - Add PORTABLE_REAL to db/model/types.py (Float with postgresql/sqlite variants) - Add OGFieldResourceQueryView ORM model (EAV/long shape, composite PK) - Add VALUE_TEXT/NUM/JSON_FIELDS and FIELD_TO_VALUE_COLUMN routing constants - Register model in db/model/__init__.py so it lands in StitchBase.metadata - Add Alembic migration 4a65b82737c6 (down_revision=6de2b873bacb), table ships empty - Add TDD test file (5 checks): metadata presence, not-a-view, PK columns, row round-trip on SQLite, and field routing constant coverage
Add og_field_query_view_refresh.py with refresh_resources() and rebuild_all() to populate og_field_resource_query_view. The unpivot preserves existing coalescing semantics exactly: value is None is the only skip condition; empty string and empty list are present and stored. Reuses Task 1 routing constants (FIELD_TO_VALUE_COLUMN). Covered by 13 integration tests (the 10 required cases plus empty-string presence, empty-ids no-op, and no-active-memberships).
Add og_field_query_view_actions with query_v2_ids (phase 1), hydrate_v2 (phase 2), and query_v2 (drop-in for og_field_resource_actions.query()). Phase 1 builds an unfiltered resource_id universe, coalesces involved fields over licensed rows via a row_number() window (min priority then source_id, rn==1), LEFT-JOINs them so nulls survive, filters/sorts on the coalesced values, counts before pagination. Phase 2 fetches and pivots the same winner per (resource_id, column), coercing year fields to int and returning null-shells for resources with no licensed rows. Two deliberate divergences from query(): params.source is ignored (only licensed_sources filters), and same-key duplicate sources resolve by lowest source_id rather than max(value). Tests: ported TestResourceQueryAction cases, divergence tests, and a 56-case equivalence harness asserting query_v2 == query (SQLite >= 3.30 guarded).
Implements filter_options_v2 reading the precomputed query-view projection, mirroring filter_options() semantics (distinct, non-NULL/non-empty, ordered). Extracts _coalesced_value_cte helper shared with query_v2_ids. Adds 25 tests: 5 ported + 20 equivalence (5 fields × 4 license combos).
query_v2_ids built one windowed-subquery CTE per involved field and LEFT JOINed each to the universe. SQLite cannot index a windowed CTE, so the join degraded to a nested-loop scan of the materialized CTE per universe row -> O(n^2) (filter-independent: ~30s at 32k rows, minutes at 80k). Replace the per-field windowed CTEs with a single windowed pass + GROUP BY pivot (a plain derived table the planner can hash/auto-index for the universe join). Same rn=1 winner primitive (priority, source_id), same null-shell/licensing contracts; now linear (~140ms at 32k, ~0.5s at 80k). Equivalence harness still green.
scripts/test_query.py: builds a >75k-row on-disk SQLite DB (mixed sources, repointed, multi-source, inactive memberships), runs rebuild_all, and times query_v2() across a param matrix. Default mode times only the new query; --compare-old also runs the old query() and asserts identical ids/total.
…ships Skip rather than fall back to a possibly-duplicate source key, so --compare-old can never hit the documented same-key tiebreak divergence and report a false mismatch.
…metics - query_v2_ids: derive universe from active memberships of non-repointed resources (not the projection) so all-null sources match query()'s null-shell - migration: value_json JSONB on Postgres to match model PORTABLE_JSON - annotate params: OGFieldQueryParams; drop dead 'or None'; priority-rebuild note - add all-null-source equivalence test (query_v2 == query)
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-141-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-141-entity-linkage.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-141.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-141-stitch-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_141 | stitch-dev.postgres.database.azure.com |
5432 |
pr_141 |
Jobs (2)
| job | image | postgres_db | api_url | auth_mode |
|---|---|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-141@sha256:d4d86e06e0fae07cbd31078a9b5ad13a88599623b8150b1f4cdc7a4c06c6dcc4 |
pr_141 |
||
| seed | ghcr.io/rmi/stitch-seed:pr-141@sha256:afbfdc727b2f3b3d55034a31d833c310c6b1fbd1a9bd2679b6dc9b75d951f5bb |
https://pr-141-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 |
stitch-client-bearer-token |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-06-18T18:36:56Z | 2026-06-18T18:36:37Z | 24a5a56 | ghcr.io/rmi/stitch-api:pr-141 |
ghcr.io/rmi/stitch-api:pr-141@sha256:d4d86e06e0fae07cbd31078a9b5ad13a88599623b8150b1f4cdc7a4c06c6dcc4 |
| 2026-06-18T18:36:56Z | 2026-06-18T18:36:37Z | 24a5a56 | ghcr.io/rmi/stitch-entity-linkage:pr-141 |
ghcr.io/rmi/stitch-entity-linkage:pr-141@sha256:7ba2f34c7f7adf0e299dc66341101d1765802981ca3fef90ed0fbbebc30e3f33 |
| 2026-06-18T18:36:58Z | 2026-06-18T18:36:37Z | 24a5a56 | ghcr.io/rmi/stitch-seed:pr-141 |
ghcr.io/rmi/stitch-seed:pr-141@sha256:afbfdc727b2f3b3d55034a31d833c310c6b1fbd1a9bd2679b6dc9b75d951f5bb |
| 2026-06-18T18:36:58Z | 2026-06-18T18:36:37Z | 24a5a56 | ghcr.io/rmi/stitch-stitch-llm:pr-141 |
ghcr.io/rmi/stitch-stitch-llm:pr-141@sha256:8c82a8d2b8fe382b74fa94db13ee6edb08ba66c73cb0b93c072d7ed2ca7b2335 |
CD summary
|
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_141 | stitch-dev.postgres.database.azure.com |
5432 |
pr_141 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-141@sha256:bd7d0e65980f4943fc3e147c86dd3c82461c7353c4ba24f2a7ad05a6879100b6 |
pr_141 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-06-18T18:42:40Z | 2026-06-18T18:42:24Z | 1b6600f | ghcr.io/rmi/stitch-api:pr-141 |
ghcr.io/rmi/stitch-api:pr-141@sha256:bd7d0e65980f4943fc3e147c86dd3c82461c7353c4ba24f2a7ad05a6879100b6 |
| 2026-06-18T18:42:38Z | 2026-06-18T18:42:24Z | 1b6600f | ghcr.io/rmi/stitch-entity-linkage:pr-141 |
ghcr.io/rmi/stitch-entity-linkage:pr-141@sha256:bbfaf6225b72c33ef67896744547d977b7ada6799f95894c6b5daee718720979 |
| 2026-06-18T18:42:40Z | 2026-06-18T18:42:24Z | 1b6600f | ghcr.io/rmi/stitch-seed:pr-141 |
ghcr.io/rmi/stitch-seed:pr-141@sha256:64bb32b10ff6f1898ba82f5dd6d2890c89c18b488985893fee0cf54d59e55f9e |
| 2026-06-18T18:42:41Z | 2026-06-18T18:42:24Z | 1b6600f | ghcr.io/rmi/stitch-stitch-llm:pr-141 |
ghcr.io/rmi/stitch-stitch-llm:pr-141@sha256:dd4795dccd202d079a0212bbb8c90e2dead9cecc9b8d5025846c26ed1d7e6a55 |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-141-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-141-entity-linkage.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-141.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-141-stitch-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_141 | stitch-dev.postgres.database.azure.com |
5432 |
pr_141 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-141@sha256:7b62953a4ab2581d715e685a3c15e5fa8f9efa3249b304cf6d69de7a5cf798d6 |
pr_141 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-06-18T18:41:19Z | 2026-06-18T18:41:01Z | 78aa2ef | ghcr.io/rmi/stitch-api:pr-141 |
ghcr.io/rmi/stitch-api:pr-141@sha256:7b62953a4ab2581d715e685a3c15e5fa8f9efa3249b304cf6d69de7a5cf798d6 |
| 2026-06-18T18:41:21Z | 2026-06-18T18:41:01Z | 78aa2ef | ghcr.io/rmi/stitch-entity-linkage:pr-141 |
ghcr.io/rmi/stitch-entity-linkage:pr-141@sha256:230aed218a9c4185fb26cf1da1665c89c2adfabab366a78435e8f86f59473f58 |
| 2026-06-18T18:41:19Z | 2026-06-18T18:41:01Z | 78aa2ef | ghcr.io/rmi/stitch-seed:pr-141 |
ghcr.io/rmi/stitch-seed:pr-141@sha256:9841c859f83336a18ac756a2661cad6dabbffe89b8300c33888c8863a641c23b |
| 2026-06-18T18:41:25Z | 2026-06-18T18:41:01Z | 78aa2ef | ghcr.io/rmi/stitch-stitch-llm:pr-141 |
ghcr.io/rmi/stitch-stitch-llm:pr-141@sha256:2efd995128f96acdeb9f21eaaf406b009449c68eb9a9b29de8a8d9ae4890569f |
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-141-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-141-entity-linkage.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-141.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-141-stitch-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_141 | stitch-dev.postgres.database.azure.com |
5432 |
pr_141 |
Jobs (2)
| job | image | postgres_db | api_url | auth_mode |
|---|---|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-141@sha256:980f9421d8cd9c582ed086156173bc299d9e3c7d6ea2e4eed4683131dd1e45d9 |
pr_141 |
||
| seed | ghcr.io/rmi/stitch-seed:pr-141@sha256:af061ab8d4d9671d09f1fda518b5b8a7e42aad7f17e42aee74b5908d06408cf4 |
https://pr-141-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 |
stitch-client-bearer-token |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-06-19T04:57:41Z | 2026-06-19T04:57:27Z | 903d904 | ghcr.io/rmi/stitch-api:pr-141 |
ghcr.io/rmi/stitch-api:pr-141@sha256:980f9421d8cd9c582ed086156173bc299d9e3c7d6ea2e4eed4683131dd1e45d9 |
| 2026-06-19T04:57:45Z | 2026-06-19T04:57:27Z | 903d904 | ghcr.io/rmi/stitch-entity-linkage:pr-141 |
ghcr.io/rmi/stitch-entity-linkage:pr-141@sha256:7273902ef543d4bf3c62a20f865a90d66b6cdb4342bfdba3927ab472bc3f5d55 |
| 2026-06-19T04:57:45Z | 2026-06-19T04:57:27Z | 903d904 | ghcr.io/rmi/stitch-seed:pr-141 |
ghcr.io/rmi/stitch-seed:pr-141@sha256:af061ab8d4d9671d09f1fda518b5b8a7e42aad7f17e42aee74b5908d06408cf4 |
| 2026-06-19T04:57:42Z | 2026-06-19T04:57:27Z | 903d904 | ghcr.io/rmi/stitch-stitch-llm:pr-141 |
ghcr.io/rmi/stitch-stitch-llm:pr-141@sha256:571a77f3e3aeb4abe998a606bc37c48d9f6f89bd6d6798490f74c10cdb114cc9 |
AlexAxthelm
requested changes
Jun 22, 2026
Collaborator
There was a problem hiding this comment.
I don't have any particular changes to request in code, but I think it's worth discussing the architectural differences from my parallel PR around the same issue (#135)
In either case, we're introducing and Entity-Attribute-Value table, which allows for fast query, since it's the SQL idiom.
The major difference I see is that you're keeping the primary store in the current source table, and then computing what is effectively a search/filter index in the OGFieldResourceQueryView, where I'm using the EAV as a primary data store.
Of the two, I have a preference for my design, since it avoids the need to maintain any syncing utilities (which are workarounds for SQLite not having Materialized Views?) at a modest read-time cost, and introducing a break between the storage model and logical model (I'm not entirely certain what the performance trade off is, I haven't run any formal benchmarking yet).
The other big difference is that you're keeping the current baseline migration, while I'm willing to start over in the face of a big restructuring
Marking as "changes requested" mostly to raise discussion on how we want to go here (cc @jdhoffa )
Collaborator
There was a problem hiding this comment.
There's a lot to like about this test script. Loading data directly into sqlite for testing is fast (but does couple the test script a bit tighter to the DB schema). I think a lot of this could get integrated in followups to #134
|
|
||
|
|
||
| # Field → value-column routing constants (single source of truth) | ||
| VALUE_TEXT_FIELDS = ( |
Collaborator
There was a problem hiding this comment.
You landed on largely the same boilerplate EAV construction I did in #135. Being boring is good on this.
|
|
||
| async def rebuild_all(session: AsyncSession, *, batch_size: int = 1000) -> None: | ||
| """Full rebuild of the projection (used by stress script and backfill).""" | ||
| await session.execute(delete(OGFieldResourceQueryView)) |
Collaborator
There was a problem hiding this comment.
I'm not totally sure how SQLAlchemy would handle this, but throwing around a delete on a table like this can cause a lot of problems on a Postgres instance (where CASCADE exists). More idiomatic would be to delete all rows (but keep the table), or as you pointed out, MV.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For the spike, we've added a new table (for sqlite compatibility) with a minimal set of sync actions to keep data up-to-date with resources, sources, and memberships. In PG, this would make more sense as a materialized view with incremental updates on write. Using a view would also remove the need for the additional sync functionality.
There's also a stress test script at
deployments/api/scripts/test_query.pythat profiles the query over 6 test cases. It builds (and optionally persists) a sqlite db file, seeds it with random, reasonably representative data, and runs the queries.Over ~80k sources & resources with multiple layers of merging, I'm seeing 0.3-0.9s for all 6 variants.
Overall table/view structure is essentially identical to #135 for the main query table. It's used exclusively to fetch the appropriate set of resource ids that can then be used to hydrate the actual response.
Note: this is a large changeset since it's purely additive. We can integrate by swapping
queryinog_field_resource_actions.pywith thev2call before fully deleting the old implementation and tests. Majority of added lines come fromscripts/test_query.pyand additional modules undertests/.Implemented mostly by Claude Code.