Skip to content

Commit 01f33b7

Browse files
define-nullclaudemo4islona
authored
Bump sqd-query-engine to 6df565a, version 2.16.0-rc.1 (#76)
## Summary Bump `sqd-query-engine` (the experimental dynamic query engine) from `54b9020` to [`6df565a`](subsquid/sqd-query-engine@6df565a), the current upstream master tip, and set the worker version to `2.16.0-rc.1`. The previous pin dated from the initial integration in July and had been orphaned upstream (not reachable from master). Upstream range: 52 commits. Full list: subsquid/sqd-query-engine@54b9020...6df565a ## The catalog format changed Two of those commits reshape the dataset description the engine loads, and `DatasetDescription` is `deny_unknown_fields`, so a stale catalog is a load error rather than a silently ignored key: - [`c31a4a0`](subsquid/sqd-query-engine@c31a4a0) makes `filters` a required, closed list — a table that accepts none says `filters: []`. - [`b72c404`](subsquid/sqd-query-engine@b72c404) groups a table's keys into `request` (what a client may send), `output` (what a client may see) and the storage keys. `query_name` → `request.name`, `field_name` → `output.name`, `json_encoding: hex` → `encoding: hex_bytes`; `stats`, `dictionary`, `children` and `parent_key` are gone. Both surfaces are now closed: a column is filterable only if `request.filters` names it, and selectable only if `output.fields` does. The four inlined test catalogs still spoke the flat layout, which is what the first CI run caught — 23 failures, all of them `missing field 'filters'` or an unknown key. They are rewritten here; no production code needed changes. Note for whoever ships this: the catalogs the worker fetches at runtime have to be republished in the new layout too, or the worker hits the same load error outside the tests. ## One bad schema no longer takes the bundle down Separate commit. Parsing a staged schema failed the whole bundle, which is how a stale catalog for a kind this worker holds no chunk of would stop it serving the kinds it does hold. `load_dir` now drops the schema it cannot parse, warns, and counts it (`schema_bundle_schemas_skipped`), leaving the rest installed. That is not a loosening: the id is left out of `bundle_ids`, and `DatasetsIndex` already refuses an assignment whose chunks reach for an id the bundle does not offer — so the refusal lands on the assignment that actually depends on the schema rather than on every assignment. A bundle in which nothing parses stays a permanent fault, since no query could be served from it. The store path had this shape already: `read_store` skips a file it cannot read and lets a later bundle reinstall the id. ## Test plan - [x] `cargo check --all-targets --locked` - [x] `cargo test --all-features --lib --bins --test state_pbt --test state_regression` - [x] Conformance tier (`e2e`, `query_surface`, `query_concurrency`), default and `--all-features` - [x] New: a bundle carrying one unparseable schema installs the other two and offers only their ids - [x] New: a bundle in which nothing parses is still refused as permanent, with nothing left in the store 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Evgeny Formanenko <mo4islona@gmail.com>
1 parent 20aa750 commit 01f33b7

6 files changed

Lines changed: 121 additions & 24 deletions

File tree

‎Cargo.lock‎

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[package]
22
name = "sqd-worker"
33
license = "AGPL-3.0-or-later"
4-
version = "2.15.0"
4+
version = "2.16.0-rc.1"
55
edition = "2021"
66

77
[features]
@@ -73,7 +73,7 @@ sqd-network-transport = { git = "https://github.com/subsquid/sqd-network.git", r
7373

7474
sqd-query = { git = "https://github.com/subsquid/data.git", rev = "aa62e79", features = ["parquet"] }
7575
sqd-polars = { git = "https://github.com/subsquid/data.git", rev = "aa62e79" }
76-
sqd-query-engine = { git = "https://github.com/subsquid/sqd-query-engine.git", rev = "54b9020" }
76+
sqd-query-engine = { git = "https://github.com/subsquid/sqd-query-engine.git", rev = "6df565a" }
7777

7878
sql_query_plan = {git = "https://github.com/subsquid/qplan.git", rev = "658f88f" }
7979

‎src/controller/experimental_engine.rs‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -224,16 +224,22 @@ mod tests {
224224
name: evm
225225
tables:
226226
blocks:
227+
output:
228+
name: block
229+
fields: [number]
227230
block_number_column: number
228-
field_name: block
229231
sort_key: [number]
230232
columns:
231233
number:
232234
type: uint64
233235
logs:
236+
request:
237+
name: logs
238+
filters: []
239+
output:
240+
name: log
241+
fields: [transaction_index, log_index, data]
234242
block_number_column: block_number
235-
query_name: logs
236-
field_name: log
237243
item_order_keys: [transaction_index, log_index]
238244
sort_key: [block_number, transaction_index, log_index]
239245
columns:
@@ -245,7 +251,7 @@ tables:
245251
type: uint32
246252
data:
247253
type: string
248-
json_encoding: hex
254+
encoding: hex_bytes
249255
weight: data_size
250256
data_size:
251257
type: uint64
@@ -473,14 +479,17 @@ tables:
473479
name: evm
474480
tables:
475481
blocks:
482+
output:
483+
name: block
484+
fields: [number, hash]
476485
block_number_column: number
477486
sort_key: [number]
478487
columns:
479488
number:
480489
type: uint64
481-
stats: true
482490
hash:
483491
type: string
492+
encoding: hex_bytes
484493
"#;
485494
let app = axum::Router::new()
486495
.route(

‎src/controller/schema_bundle.rs‎

Lines changed: 85 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -719,6 +719,7 @@ async fn load_dir(
719719
joined(
720720
tokio::task::spawn_blocking(move || {
721721
let mut schemas = HashMap::new();
722+
let mut skipped = Vec::new();
722723
// Reading back what was just staged is disk; what the yaml says is the bundle.
723724
for entry in std::fs::read_dir(&dir).map_err(BundleFault::transient)? {
724725
let path = entry.map_err(BundleFault::transient)?.path();
@@ -732,18 +733,31 @@ async fn load_dir(
732733
let yaml = std::fs::read_to_string(&path)
733734
.with_context(|| format!("couldn't read schema {}", path.display()))
734735
.map_err(BundleFault::transient)?;
735-
let description = sqd_query_engine::metadata::parse_dataset_description(&yaml)
736-
.map_err(|e| {
737-
BundleFault::permanent(anyhow::anyhow!(
738-
"couldn't parse schema {}: {e:?}",
739-
path.display()
740-
))
741-
})?;
742-
schemas.insert(id, Arc::new(description));
736+
// A schema is load-bearing only for the chunks that name it, and the assignment
737+
// check downstream already refuses those. Failing the bundle instead would let
738+
// one unreadable schema — a kind this worker holds no chunk of — stop it serving
739+
// the datasets it does hold.
740+
match sqd_query_engine::metadata::parse_dataset_description(&yaml) {
741+
Ok(description) => {
742+
schemas.insert(id, Arc::new(description));
743+
}
744+
Err(e) => {
745+
metrics::SCHEMA_BUNDLE_SCHEMAS_SKIPPED.inc();
746+
tracing::warn!(
747+
%id,
748+
error = ?e,
749+
"Dropping a schema the bundle carries but this engine can't parse",
750+
);
751+
skipped.push(format!("{id}: {e:?}"));
752+
}
753+
}
743754
}
744755
if schemas.is_empty() {
756+
// `unpack` has already refused a bundle with no `<id>.yaml` entries, so there
757+
// were some and every one of them failed: nothing here can serve a query.
745758
return Err(BundleFault::permanent(anyhow::anyhow!(
746-
"no schemas found in {dir}"
759+
"no schema in {dir} could be parsed: {}",
760+
skipped.join("; ")
747761
)));
748762
}
749763
Ok(schemas)
@@ -791,6 +805,9 @@ pub(crate) mod test_support {
791805
name: evm
792806
tables:
793807
blocks:
808+
output:
809+
name: block
810+
fields: [number]
794811
block_number_column: number
795812
sort_key: [number]
796813
columns:
@@ -934,6 +951,65 @@ mod tests {
934951
assert!(format!("{fault:#}").contains("zstd"), "{fault:#}");
935952
}
936953

954+
/// A stale catalog for a kind this worker holds no chunk of used to stop it serving the
955+
/// kinds it does hold: one parse fault was a verdict on the whole bundle.
956+
#[tokio::test]
957+
async fn one_unparseable_schema_leaves_the_rest_of_the_bundle_usable() {
958+
let dir = tempfile::tempdir().unwrap();
959+
let registry = store(&dir);
960+
let solana = SCHEMA.replace("name: evm", "name: solana");
961+
let archive = targz(&[
962+
("7.yaml", SCHEMA.as_bytes()),
963+
(
964+
"9.yaml",
965+
b"name: fuel\ntables:\n blocks:\n field_name: block\n",
966+
),
967+
("12.yaml", solana.as_bytes()),
968+
]);
969+
let bundle = served_bundle(archive).await;
970+
971+
registry
972+
.ensure(&bundle, &reqwest::Client::new())
973+
.await
974+
.unwrap();
975+
976+
assert_eq!(registry.get_by_id(SchemaId::new(7)).unwrap().name, "evm");
977+
assert_eq!(
978+
registry.get_by_id(SchemaId::new(12)).unwrap().name,
979+
"solana"
980+
);
981+
assert!(registry.get_by_id(SchemaId::new(9)).is_err());
982+
983+
// The id the bundle carries but cannot serve is not one it offers, so an assignment
984+
// whose chunks need it is refused rather than answered from a schema that never loaded.
985+
assert_eq!(
986+
*registry.bundle_ids(),
987+
HashSet::from([SchemaId::new(7), SchemaId::new(12)]),
988+
);
989+
assert_eq!(stored(&dir), vec!["12.yaml", "7.yaml"]);
990+
assert_eq!(registry.installed_hash(), Some(bundle.hash));
991+
}
992+
993+
#[tokio::test]
994+
async fn a_bundle_no_schema_of_which_parses_is_still_a_permanent_fault() {
995+
let dir = tempfile::tempdir().unwrap();
996+
let registry = store(&dir);
997+
let archive = targz(&[(
998+
"9.yaml",
999+
b"name: fuel\ntables:\n blocks:\n field_name: block\n",
1000+
)]);
1001+
let bundle = served_bundle(archive).await;
1002+
1003+
let fault = registry
1004+
.prepare_bundle(&bundle, &reqwest::Client::new())
1005+
.await
1006+
.unwrap_err();
1007+
1008+
assert!(fault.is_permanent(), "{fault:#?}");
1009+
assert!(format!("{:#}", fault.into_error()).contains("field_name"));
1010+
assert!(stored(&dir).is_empty());
1011+
}
1012+
9371013
#[tokio::test]
9381014
async fn preparing_a_bundle_does_not_make_it_active() {
9391015
let dir = tempfile::tempdir().unwrap();

‎src/metrics.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ lazy_static::lazy_static! {
6969
pub static ref SCHEMA_BUNDLE_LOADED: Gauge = Default::default();
7070
pub static ref SCHEMA_BUNDLE_FAILURES: Counter = Default::default();
7171
pub static ref SCHEMA_BUNDLE_MISMATCHES: Counter = Default::default();
72+
pub static ref SCHEMA_BUNDLE_SCHEMAS_SKIPPED: Counter = Default::default();
7273

7374
static ref QUERY_EXECUTED: Family<QueryExecutedLabels, Counter> = Default::default();
7475
static ref QUERY_RESULT_SIZE: Histogram = Histogram::new(std::iter::empty());
@@ -193,6 +194,11 @@ pub fn register_metrics(registry: &mut Registry, version: String) {
193194
"Number of times a schema bundle failed to install",
194195
SCHEMA_BUNDLE_FAILURES.clone(),
195196
);
197+
registry.register(
198+
"schema_bundle_schemas_skipped",
199+
"Number of schemas dropped from a bundle because they could not be parsed",
200+
SCHEMA_BUNDLE_SCHEMAS_SKIPPED.clone(),
201+
);
196202
registry.register(
197203
"schema_bundle_mismatches",
198204
"Number of pairs the scheduler published that do not hold together",

‎tests/harness/corpus.rs‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,22 @@ pub const SCHEMA_YAML: &str = r#"
1717
name: evm
1818
tables:
1919
blocks:
20+
output:
21+
name: block
22+
fields: [number]
2023
block_number_column: number
21-
field_name: block
2224
sort_key: [number]
2325
columns:
2426
number:
2527
type: uint64
2628
logs:
29+
request:
30+
name: logs
31+
filters: []
32+
output:
33+
name: log
34+
fields: [transaction_index, log_index, data]
2735
block_number_column: block_number
28-
query_name: logs
29-
field_name: log
3036
item_order_keys: [transaction_index, log_index]
3137
sort_key: [block_number, transaction_index, log_index]
3238
columns:
@@ -38,7 +44,7 @@ tables:
3844
type: uint32
3945
data:
4046
type: string
41-
json_encoding: hex
47+
encoding: hex_bytes
4248
weight: data_size
4349
data_size:
4450
type: uint64

0 commit comments

Comments
 (0)