feat(ingress): Gorilla delta-of-delta timestamp encoding for QWP/WS ingress - #168

Draft

jovfer wants to merge 427 commits into

mainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 26, 2026

Correct polars example run commands to use the stable toolchain

mtopolnik
and
claude
committed
3ea6442
Add column-major failover e2e: qwp_column_sidecar + test

mtopolnik
and
claude
committed
eaa55bd
Gate QwpWsSymbolHasher re-export on `arrow` to fix unused import

mtopolnik
and
claude
committed
b718fe7
Run rustfmt on qwp_column_sidecar

mtopolnik
and
claude
committed
0c3d983
Add arrow SEND variant to qwp_column_sidecar

mtopolnik
and
claude
committed
fffd77e
Add polars SEND variant (opt-in) to qwp_column_sidecar

mtopolnik
and
claude
committed
3dc29d9
Add egress (read-side) failover sidecar + test

mtopolnik
and
claude
committed
d5e799f
Normalize UUID in sliced Arrow round-trip test

mtopolnik
and
claude
committed
40c112f
refactor(features): rework client feature taxonomy
bluestreak01
committed
cca5bc8
Apply rustfmt to questdb-rs after feature rework

mtopolnik
and
claude
committed
5dda5a1
Fix QWP orphan drainer retry handling
jerrinot
committed
f8f06e0
Fix QWP UDP narrow decimal null bitmap
jerrinot
committed
68c92be
Fix QWP decimal NaN test under qwp-ws-only build

mtopolnik
and
claude
committed
39c4875
Fix QWP/WS marker snapshot retention
jerrinot
committed
865ea87
Reset QWP WS decimal scale on clear
jerrinot
committed
d35d7c7
Fix Arrow null_count OOB and Polars retry hang

kafka1991
and
claude
committed
804d2ed

Commits on Jun 27, 2026

Commits on Jun 28, 2026

Commits on Jun 29, 2026

rename `flush_arrow_batch_server_stamped` to `flush_arrow_batch_at_now()`
kafka1991
committed
5597f8d
add some validations
kafka1991
committed
3212362
add system tests
kafka1991
committed
138ca0f
feat(error): unify the client error model across Rust, FFI, C and C++

jerrinot
and
claude
committed
74f7522
fix(ingress): drain in-memory store-and-forward queue on column-sender pool teardown

bluestreak01
and
claude
committed
e22eeb0
docs(ingress): correct SFA wait-timeout recovery guidance

bluestreak01
and
claude
committed
10711b0
test(e2e): durable-ack orphan-drainer regression tests (19b9fda)

bluestreak01
and
claude
committed
a9204bf
test(e2e): C and C++ binding variants of the durable-ack orphan tests

bluestreak01
and
claude
committed
2cbb100

Commits on Jun 30, 2026

adaptive ingress max_rows
kafka1991
committed
69046fa
adaptive ingress max_rows
kafka1991
committed
0e6c28e
fix one failed test
kafka1991
committed
cccc264
ingress: complete the QWP/WebSocket pooled row-sender API for C/C++

jerrinot
and
claude
committed
ac0b31e
ingress: remove the unused hand-driven direct column sender from C/C++
jerrinot
committed
ce01824
fix(cpp): prevent pooled sender handle escapes
jerrinot
committed
fb03889
Rename reader drop-on-return APIs
jerrinot
committed
c5f19b9

Commits on Jul 1, 2026

Use neutral QWP ack level naming
jerrinot
committed
765d566
Hide pooled sender must-close state
jerrinot
committed
90385af
Expose pooled row sender FSN progress
jerrinot
committed
df3e96f
Document pooled row sender FSN usage
jerrinot
committed
845483d
Expose column SFA FSN progress
jerrinot
committed
6f38e38

Commits on Jul 2, 2026

arrow/polars: consolidate deps and widen supported version ranges

kafka1991
and
claude
committed
30a9777
failover cargo fmt
kafka1991
committed
82a8128
revert precommit.sh
kafka1991
committed
2419a88
fix example
kafka1991
committed
27d9bc1
fix run_version_matrix
kafka1991
committed
b118fad
ingress: drop the vestigial sf_ prefix from the pooled column-sender C/C++ API
jerrinot
committed
e99e8ae
Clarify column sender naming in docs and FFI errors
jerrinot
committed
9be071f
tests
jerrinot
committed
3ca7e33

Commits on Jul 3, 2026

Align QWP store-and-forward with NACK v2 policy
jerrinot
committed
5deae40
update partial readme
kafka1991
committed
c68b941
refactor(ingress): align column-sender vocabulary with the row sender

glasstiger
and
claude
committed
fe50e3e
refactor(db): hoist the shared pool config out of the column sender

glasstiger
and
claude
committed
0cf763f
refactor(ingress): move TimestampUnit next to AckLevel

glasstiger
and
claude
committed
096ba66

Commits on Jul 4, 2026

Commits on Jul 5, 2026

Commits on Jul 6, 2026

Commits on Jul 7, 2026

Commits on Jul 8, 2026

Commits on Jul 9, 2026

test(qwp-ws-fuzz): widen bounce-variant drain budget to 4 min
kafka1991
committed
294c81a
add direct c api
kafka1991
committed
e530e7e
Fix direct column-sender capacity handling; honour the server batch cap
kafka1991
committed
3c83f14
long CI timeout
kafka1991
committed
68db801
Fix pool store-and-forward recovery
jerrinot
committed
4a04a14
update test assumptions
jerrinot
committed
23aaf9e
Reject standalone durable wait without request_durable_ack opt-in
jerrinot
committed
2692bf9
test: tighten Arrow SFA terminal rejection coverage
jerrinot
committed
8c86837
ci: clarify Arrow dependency comments
jerrinot
committed
214830a

Commits on Jul 10, 2026

Persist the SFA symbol dictionary so delta frames can replay after recovery

glasstiger
and
claude
committed
3b31d9b
update submodule
glasstiger
committed
47f8818
Merge branch 'jh_conn_pool_refactor' of https://github.com/questdb/c-questdb-client into jh_conn_pool_refactor

glasstiger
and
claude
committed
9a12440
Add per-pool connection-count diagnostics (dbg_pool_counts)
kafka1991
committed
b5f389b
Add long-running soak / stress harness
kafka1991
committed
b2edeb6
Fix managed slot paths in SFA tests
jerrinot
committed
5e1a97c
Fix numpy chunk split advancing by wire stride

glasstiger
and
claude
committed
f6d96eb
Keep split failures in-doubt after a partial commit

glasstiger
and
claude
committed
70d70af
Fix numpy f32 column type in the C ABI docs

glasstiger
and
claude
committed
3a2b2d0
Don't label a role reject a protocol violation

glasstiger
and
claude
committed
51b292b
Validate recovered SFA dict before spawning runner

glasstiger
and
claude
committed
1dffdf8
Reject a conflicting SFA dict redefinition as torn

glasstiger
and
claude
committed
dad40ba
Harden the persisted SFA symbol dictionary

glasstiger
and
claude
committed
222d899
docs: lead README with QWP/WebSocket, deemphasize ILP

jerrinot
and
claude
committed
60ce253
Reject overflowing 10-byte QWP varints

glasstiger
and
claude
committed
d42a294
Harden SFA side-file cleanup and recovery allocs

glasstiger
and
claude
committed
6b72eb0
Merge origin/jh_conn_pool_refactor into jh_conn_pool_refactor

glasstiger
and
claude
committed
1221cb3
Make QWP reader a first-class client surface
jerrinot
committed
9fd93b7
Document Arrow dict-memo key soundness

glasstiger
and
claude
committed
fadc477
Harden QWP reconnect catch-up frame builder

glasstiger
and
claude
committed
96bafb6
Validate UTF-8 for column-sender symbol dictionaries

glasstiger
and
claude
committed
c3173ee
Remove redundant compatibility facades
jerrinot
committed
aa21a6b
Rename QWP/WebSocket scheme qwpws/qwpwss to ws/wss

jerrinot
and
claude
committed
a97e9bc
Fix failover client formatting
jerrinot
committed
413079c
Run soak via GitHub Actions (daily 2h + per-merge 30min)
kafka1991
committed
20ea5e3
soak: refresh workload Cargo.lock against the updated questdb-rs
kafka1991
committed
5a2b4b6
Document column-append buffer lifetime in C header

glasstiger
and
claude
committed
2180198
Test columnar symbol FFI and SFA recovery gaps

glasstiger
and
claude
committed
aa1300f
Latch poison flag on rollback truncate failure

glasstiger
and
claude
committed
396660a
Pool the column-sender symbol id table across flushes

glasstiger
and
claude
committed
f95ccbc
Clarify symbol dict-size cap and null-count docs

glasstiger
and
claude
committed
25a4be1
Merge branch 'jh_conn_pool_refactor' of https://github.com/questdb/c-questdb-client into jh_conn_pool_refactor
glasstiger
committed
b0d7b2a
Cap the QWP/WS sender symbol-dict heap size

glasstiger
and
claude
committed
93732b7
Correct the ABI doc's symbol-code validation

glasstiger
and
claude
committed
a769f11
Make flush-time symbol indexing fallible

glasstiger
and
claude
committed
71139fb
Free dead symbol-dict state after recovery

glasstiger
and
claude
committed
cd85dd9
questdb-rs: bound background QWP connects
jerrinot
committed
421ad08
Lock the column encoder delta section to the Java golden

glasstiger
and
claude
committed
eb96ecf
Pool the Arrow symbol id buffers across flushes

glasstiger
and
claude
committed
29dc483
Stream the dense symbol dictionary straight to the wire

glasstiger
and
claude
committed
824d176
Switch symbol-dict persistence to per-entry CRC

glasstiger
and
claude
committed
3c56131
Merge branch 'jh_conn_pool_refactor' of https://github.com/questdb/c-questdb-client into jh_conn_pool_refactor
glasstiger
committed
b9920fd
update submodule
glasstiger
committed
d070fde
Validate UTF-8 for Arrow-ingested symbols

glasstiger
and
claude
committed
8375dbf
Test the dense symbol-dict fallback keeps lockstep

glasstiger
and
claude
committed
844e048
Make QWP symbol-dict allocations fallible under panic=abort

glasstiger
and
claude
committed
44dc26a
Cover FFI symbol_i8/i16 at the C ABI

glasstiger
and
claude
committed
7b8c2ae

Commits on Jul 12, 2026

Cover column SFA write-ahead/rollback lockstep

glasstiger
and
claude
committed
8618a08
Address minor symbol-dict review findings

glasstiger
and
claude
committed
ea3d0ef

Commits on Jul 13, 2026

soak-full: build QuestDB master, not a release (mirror fuzz)
kafka1991
committed
acb1f77
fix soak test script
kafka1991
committed
cefbc5e
add column_sender_chunk_at_now for column_sender
kafka1991
committed
6fd90e4
soak: preserve server data across restarts; harden orchestrator
kafka1991
committed
1ae5673
soak: quiet egress transient-error log spam
kafka1991
committed
7506914
add feature gate
kafka1991
committed
97a485c
1. column_sender_chunk_at_scalar_nanos 2. column-sender support np.arrow
kafka1991
committed
47bc655
export reader_server_info_zone_id
kafka1991
committed
f92ce02
soak(ci): trim QuestDB data dirs from the published artifact
kafka1991
committed
8697222
soak: bound egress scan + survive reconcile query errors
kafka1991
committed
a2578ab
add connect event
kafka1991
committed
598b864
connect events lifetime
kafka1991
committed
68995a0
add connection_listener_from_c
kafka1991
committed
a9a0ee8
Fix C++ string view overload resolution
jerrinot
committed
73e4655
doc: QWP real-network bench plan + AWS rig scripts (Step-8 axis)

jovfer
and
claude
committed
bf33adb
bench: C QWP DataFrame bench twins (qwp_ingress_c / qwp_egress_c)

jovfer
and
claude
committed
edae2c3
fix(bench): C bench build on Linux under strict -std=c11

jovfer
and
claude
committed
b4ca657
fix(net_bench): client bootstrap survives a half-built .venv

jovfer
and
claude
committed
c0871fb
soak: cap egress scan window at the QWP reader LIMIT max (10000)
kafka1991
committed
8302168
fix dataframe in flight error
kafka1991
committed
94bab28
soak: drain the leg's borrowed sender/reader before the final quiesce sample
kafka1991
committed
e5e00fe
C++ server_info: owning snapshot + zone parity

jerrinot
and
claude
committed
eee1768

Commits on Jul 14, 2026

Commits on Jul 15, 2026

fix one c++ compile issue
kafka1991
committed
a94d54b
soak: make I4 fd check trend-based + loosen RSS warmup-ramp tolerance
kafka1991
committed
2da9068
add direct_column_sender_from_conf ffi ABI
kafka1991
committed
e41cce1
add direct_column_sender_from_opts ffi ABI
kafka1991
committed
27e9041
Enable chrono-tz for Arrow egress
jerrinot
committed
bee4c04
Add width-agnostic decimal column views
jerrinot
committed
e91fe3f
align pool conf keys and defaults with the Java client
kafka1991
committed
2e8fae2
update failover cargo lock
kafka1991
committed
e97f88c
Pool connection events: register listener at connect time

jerrinot
and
claude
committed
4692c49
fix(ingress): allocate SF segment inline when rotation finds no hot spare

jovfer
and
claude
committed
1f46964
fix(ingress): make the 4MiB QWP/WS socket buffer request macOS-only

jovfer
and
claude
committed
b374c90
bench(net): QWP network bench — multi-client rig, multi-sender axis, branch-ref pins

jovfer
and
claude
committed
4ff7cbb

Commits on Jul 16, 2026

drain error notifications on wait
kafka1991
committed
3bc2803
change author's name
kafka1991
committed
f283ebe
fix(ingress): split SFA flushes at the sf_max_bytes segment cap, not max_buf_size
kafka1991
committed
da47fde
fix(db): gate query-pool fields out of the QuestDb Debug impl without _egress
kafka1991
committed
12923a5
code fmt
kafka1991
committed
386bb6d
Restore chunk clear/reuse for C/C++ and re-expose Chunk::clear

mtopolnik
and
claude
committed
e77ab80
Update questdb module
mtopolnik
committed
9c86024
chore(lint): fix clippy 1.97 new lints across examples, benches and tests
kafka1991
committed
4d62ee9
Scope pooled sender ack observation to the borrowing lease
kafka1991
committed
ff14373
code fmt
kafka1991
committed
260292a
Add shared-pool example for ingest and query

mtopolnik
and
claude
committed
121bc67
Use the Chunk API in the shared-pool example
mtopolnik
committed
94f2434
Add shared-pool chunk ingestion and query example

mtopolnik
and
claude
committed
5bf9474
Remove superseded shared-pool example

mtopolnik
and
claude
committed
0da1f55
Report unexpected result columns as client errors

mtopolnik
and
claude
committed
3d519f9
Deliver pool rejections through a handler; make wait a pure ack barrier
kafka1991
committed
181223c
Add C++ shared-pool chunk and query example

mtopolnik
and
claude
committed
9099f32
fix(ingress): keep SFA recovery off mmap so power-loss damage cannot SIGBUS the host
kafka1991
committed
dd38fdb
Merge branch 'jh_conn_pool_refactor' of github.com:questdb/c-questdb-client into jh_conn_pool_refactor
kafka1991
committed
7ac0063
chore(lint): fix clippy 1.97 byte_char_slices in ffi column-sender tests
kafka1991
committed
834c322
fix docstr
kafka1991
committed
d8dac23

Commits on Jul 17, 2026

chore(lint): fix clippy 1.97 useless_borrows_in_formatting in tls_proxy
kafka1991
committed
da98382
Expose lease-scoped rejection polling on the pooled QWP sender
kafka1991
committed
927982c
ipv6 format support
kafka1991
committed
2158029
docs: prepare QWP client release
jerrinot
committed
aaa0b80
test(system): enable faulthandler so a native fault prints thread stacks
kafka1991
committed
fabd16c
perf(ingress): inline the row-API op-state checks

jovfer
and
claude
committed
085c949
feat(ingress): Gorilla delta-of-delta timestamp encoding for QWP/WS ingress

jovfer
and
claude
committed
ed8c654