feat(ingress): Gorilla delta-of-delta timestamp encoding for QWP/WS ingress - #168
Draft
jovfer wants to merge 427 commits into
Draft
feat(ingress): Gorilla delta-of-delta timestamp encoding for QWP/WS ingress#168jovfer wants to merge 427 commits into
jovfer wants to merge 427 commits into
Conversation
The qwp_ingress_polars / qwp_egress_polars example headers and the matching [[example]] comments in Cargo.toml instructed `cargo +nightly run` and claimed the polars dependency requires a nightly toolchain. That is inaccurate: polars 0.52 builds on stable (verified with `cargo +stable check --features polars` on rustc 1.94.1), and CI already builds these examples on the linux-stable / linux-beta legs. Drop the `+nightly` from the run commands and remove the nightly note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover the c-questdb-client Rust column-major ingest API (QuestDb -> borrow_column_sender -> ColumnSender) in the cross-repo Enterprise failover e2e, alongside the existing row-major sidecar. - failover_clients: new qwp_column_sidecar binary. Same line protocol as qwp_sidecar (CONNECT/SEND/FLUSH/CLOSE/EXIT), but it drives a store-and-forward column sender: SEND accumulates a LONG `v` column + designated timestamp (the row sidecar's schema, so the asserting query is unchanged), FLUSH builds a Chunk and flush_and_wait at AckLevel::Ok. request_durable_ack + sf_dir make the SF backend retain the frame until durably acked and replay it to a reconnected primary. Chunk-only for now; arrow / polars source variants are a follow-up. - enterprise_e2e: CClientRustColumnSidecar + build_qwp_column_sidecar (builder refactored into _build_failover_bin), and the c_client_rust_column_sidecar[_binary] fixtures. - test_kill9_primary_failover_no_data_loss_c_client_rust_columnar mirrors the row-major test body (kill -9 P1, wipe disk + object store, P2 on the same port) over the columnar connect string (qwpws + pool_size=1/pool_max=1), asserting a dense [0..N) sequence. Picked up automatically by the dispatched Enterprise pipeline via `pytest -m c_client`; no CI YAML change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The buffer.rs re-export of QwpWsSymbolHasher was gated on `_sender-qwp-ws`, but its only consumer is the `arrow`-gated column_sender::arrow_batch. A `_sender-qwp-ws` build without `arrow` (default features, and the almost-all-features clippy CI leg) saw an unused import, which `-D warnings` turns into a hard error. Gate the re-export on `arrow` to match its consumer (`arrow` implies `sync-sender-qwp-ws`, so the symbol is always in scope when arrow_batch is). Verified `RUSTFLAGS=-D warnings cargo build` passes both with and without `arrow`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cargo fmt collapses the `let table = state.table.clone().ok_or_else(...)` binding onto one line, satisfying the `failover_clients: fmt` CI check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the column sidecar's SEND with an optional input-shape selector
(`SEND <table> <count> <start> [chunk|arrow]`). `chunk` (default) keeps the
borrowed-slice Chunk path; `arrow` builds an Arrow RecordBatch (v LONG + ts
microsecond timestamp) and ingests via flush_arrow_batch_at_column_and_wait.
Both encode to the same column-major QWP/WS wire and store-and-forward backend,
so the failover contract is identical.
- failover_clients: enable questdb-rs `arrow` feature, add arrow-array /
arrow-schema (same major so the RecordBatch type unifies).
- CClientRustColumnSidecar.send(src=...) forwards the shape; the columnar
failover test is parametrized over {chunk, arrow}.
Both variants pass the kill-9 failover locally against a forked ENT primary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `polars` input shape to the column sidecar's SEND, behind an optional crate feature (the polars dependency tree is large, so it is off by default). The Polars arm builds a DataFrame (v LONG + ts microsecond Datetime) and ingests via flush_polars_dataframe, which owns its commit and ACKs each checkpoint at the server OK watermark -- so the store-and-forward backend still replays to the successor after the kill, the same failover contract as chunk/arrow. - failover_clients: optional `polars` feature (= questdb-rs/polars + polars dep, pinned to questdb-rs's version so the DataFrame type unifies). The cfg-gated arm/import/builder keep the default no-polars build compiling and the match exhaustive. - The Enterprise harness builds the sidecar with --features polars when C_QUESTDB_CLIENT_COLUMN_POLARS is set; the columnar test's polars param is skipped unless that env is set, so default CI runs chunk + arrow and the polars failover is opt-in. All four (row + chunk + arrow + polars) pass the kill-9 failover locally against a forked ENT primary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New `qwp_egress_sidecar` bin -- a Rust port of the Java QwpEgressSidecarMain, driving questdb::egress::Reader. Speaks the exact protocol lib.egress_sidecar.py already drives (CONNECT/QUERY/SHOW_ZONE/SERVER_INFO/CLOSE), so the Python driver slots in unchanged: QUERY executes the SQL, sums the streamed row count, and replies `OK <rows> <ms>`; SERVER_INFO/SHOW_ZONE expose the bound endpoint's zone/role. No Cargo change -- failover_clients already enables sync-reader-ws. - CClientRustEgressSidecar (subclass of lib.egress_sidecar.EgressSidecar, overriding only the launch) + c_client_rust_egress_sidecar[_binary] fixtures. - test_primary_failover_egress_read_c_client_rust: a primary + replica share the object store; seed on the primary, wait for the replica to catch up, bind the reader (addr=p1,r1, target defaults to any), read the full count from p1, kill -9 p1, and assert the next query rebinds to the replica and returns every row. Unlike the ingress tests there is no client replay -- the survivor already holds the replicated data; the point is the reader's transparent endpoint walk. Passes locally against a forked ENT primary+replica alongside the row/columnar suite (4 passed, polars skipped without its opt-in feature). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sliced-batch C Data Interface test compared raw `as_py()` values across the round-trip. The `arrow.uuid` kind is a plain FixedSizeBinary(16) column tagged only with `ARROW:extension:name` field metadata: the source array surfaces it as `bytes`, but on import pyarrow recognises the canonical extension and materialises `uuid.UUID`. Both encode the same 16 bytes, so the comparison failed on representation alone (`UUID(...) != b'...'`), deterministically, across every linux toolchain and the docker job. Collapse UUID scalars to their big-endian bytes before comparing, so the assertion checks value and window offset without being sensitive to extension-type promotion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename and restructure the Cargo feature flags for clarity, consistency, and lean single-direction builds. Back-compat aliases keep existing Cargo.toml's building. Renames (with deprecated aliases -> new name): - sync-reader-ws -> sync-reader-qwp-ws (symmetry with sync-sender-qwp-ws) - compression-zstd -> sync-reader-zstd (egress/reader family) - chrono_timestamp -> chrono-timestamp (kebab-case) - json_tests -> json-tests (kebab-case) Structural: - QWP/UDP removed from the `sync-sender` umbrella and default build; it is best-effort (no acks/auth/TLS) and now opt-in via `sync-sender-qwp-udp`. Added explicitly to questdb-rs-ffi (C ABI exposes all transports) and to `almost-all-features` to preserve CI coverage. - `ndarray` promoted from an implicit dep-feature to an explicit feature; `chrono` hidden behind `dep:chrono`. Arrow/Polars directional split: - `arrow`/`polars` are now umbrellas over `*-ingress` / `*-egress`, so a read-only build skips the ~6.7k-line ingress Arrow encoder and a write-only build skips the egress reader/convert stack. - Extracted the only shared code into transport-neutral modules: `arrow_meta` (field-metadata keys) and `polars_ffi` (arrow<->polars_arrow FFI bridges), gated by hidden `_arrow` / `_polars` dep bundles. - Shared reconnect plumbing gated on any(polars-ingress, polars-egress). Verified: default, almost-all-features, every directional combo and both umbrellas, plus --tests and the FFI crate all compile clean (no unexpected_cfgs). Also fixes a corrupted test fn name (test_chrono_timestamp).
The feature-taxonomy rework left several call sites and cfg attributes formatted in a way rustfmt rejects, breaking the `questdb-rs: fmt` CI step (`cargo fmt --check`). Multi-feature `cfg`/`cfg_attr` predicates need the multiline `any(...)` layout, and several `with_metadata` / `md.insert` calls fit on one line. Run `cargo fmt` over questdb-rs to bring db.rs, ingress.rs, and arrow_batch.rs back to canonical formatting. Whitespace and token reflow only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The qwp_column_dec64_nan_roundtrip_preserves_following_column test built its buffer via `Buffer::new_qwp().as_qwp()`. `as_qwp()` is compiled only under `_sender-qwp-udp`, or in tests when both `_sender-qwp-ws` and `_sender-http` are enabled. The CI matrix builds a combo with QWP/WS, the reader, and arrow but neither HTTP nor QWP/UDP, so the method is absent there and `cargo test` failed to compile. Construct `QwpBuffer` directly and call `encode_datagrams` on it, exactly as the sibling decimal round-trip tests do. `QwpBuffer` and `encode_datagrams` are available in every QWP test build, so the test compiles under all feature combos while exercising the identical encoder path that the decimal-null bitmap fix touches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review fixes for the columnar ingest / Arrow tracks. - Arrow ingest: the variable-length offset writer sized its table from the array's declared null_count but filled it from the live validity bitmap. A producer whose null_count over-reported nulls drove more emits than the table held and back-patched past it -- an out-of-bounds write that aborts under the panic=abort FFI (under-reporting misaligned the byte region instead). Reconcile the two while emitting: an O(1) per-row bound check and a post-loop equality check turn the mismatch into a clean ArrowIngest error. No popcount, no hot-path cost. Adds over-/under-report regression tests. - Polars ingest: flush_polars_dataframe re-drove the uncommitted tail forever when the server accepted connections but never advanced acks (reborrow returns as soon as a replacement opens, never consulting the reconnect budget), causing an unbounded hang and duplicate writes. Bound the retries by the reconnect budget, refreshed on every checkpoint that makes progress. - Frame-size estimate: a deferred Arrow column resolved to symbols emits up to 5 bytes/row, which get_buffer_memory_size undercounts. Add the symbol worst case so the up-front try_reserve stays an upper bound and per-column writes never fall back to an aborting realloc. - Harden the aligned QWP-bitmap fast path with a runtime length check instead of debug_assert, matching the bool path. - Column-sender config: reject qwp_ws_progress=manual with initial_connect_retry=async, matching build(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
borrow_column_sender() is now always store-and-forward, mirroring the row-major sender: in-memory queue when no sf_dir, disk-backed when set. In-memory SF pools freely up to pool_max; disk SF stays single-borrower. Both column pools are now lazy (like borrow_row_sender), reusing the shared connect_sfa_background / open_configured_qwp_ws_queue machinery. Add borrow_direct_column_sender(): an always-direct (non-SF) pool, independent of sf_dir, used by DataFrame ingestion (Polars/Pandas), which drives its own replay and wants a plain pipelined connection. Threaded a ColumnPoolKind through the borrow/return/reborrow/reap paths. Config: relax the column-sender conf to accept sf_* / sender_id keys without sf_dir (passthrough to the SenderBuilder), matching row-major; rename ParsedConf.store_and_forward -> sf_disk (disk/single-borrower). Tests: migrate the direct-backend and Polars tests onto the direct pool; parametrize the store_and_forward_* behaviours over disk + in-memory; add durable-ack success and SF auto-reconnect/replay coverage for the in-memory backend. Fix a redundant into_iter clippy warning.
…lures Three independent CI failures from build 246215: 1. QWP/WS fuzz suite — "invalid GeoHash precision: 0" The column-major buffer retains columns across flushes. A geohash column's pinned precision was reset to the invalid sentinel 0 on clear/rollback, so a reused buffer that omitted the column in a later batch serialized an all-null geohash column with precision 0, which the server rejects. Pin precision off `cells.is_empty()` (start of a fresh batch) instead of the 0 sentinel, and stop resetting precision on clear/rollback so a retained, fully-omitted column keeps a valid precision. Adds a regression test. 2. reader_c_smoke + test_reader_mock — lazy-pool stragglers The pool was made lazy (connect opens nothing; first borrow opens the connection), matching the row-major/direct pools and the Rust reader_pool tests. The C smoke test and reader-mock tests still assumed eager-open: the smoke expected connect to fail against a closed port (now succeeds; the failure moves to first borrow), and the mock scripts reserved a stale "column eager-open" entry that the reader borrow then consumed, reading a non-responding script and surfacing EAGAIN. Align the tests and the connect_reader docs (reader.h / egress.rs / db.rs) with the lazy design. 3. test_reader_mock arm64 UBSan — misaligned typed-pointer load Zero-copy scalar getters formed a typed `const T*` over an unaligned borrowed slice and did `base + row` + memcpy, which the optimizer lowered into an alignment-assuming load (UB; traps under -fsanitize=alignment on arm64). Route the per-element getters through a `detail::load_unaligned<T>` byte-wise helper, fix the bulk-API test helper to read through a byte pointer, and tighten the values<T>() docs. Full C/C++ ASan+UBSan suite (ctest -E rust_tests) and the Rust lib tests pass.
Give the store-and-forward and direct column senders distinct types with
honest, non-overlapping surfaces instead of one handle that exposed both:
- SF: flush + wait (queue owns delivery; wait is an ack barrier)
- Direct: flush + flush_and_wait + commit
`sync` is removed everywhere; the ambiguous single `column_sender` /
`BorrowedColumnSender` handle is gone.
Rust core: SfColumnSender / DirectColumnSender replace BorrowedColumnSender
(no more Deref leak); borrow_column_sender returns SF,
borrow_direct_column_sender returns Direct.
FFI crate: sf_column_sender / direct_column_sender opaque handles, 21
sf_/direct_ exports via shared CsHandle generics; owned direct
borrow added in questdb::ffi_support.
C header: column_sender typedef split; declarations + docs updated.
C++ wrapper: sf_/direct_ conn + borrowed_ guards + pool::borrow_* methods;
friend decls updated in line_sender_core.hpp.
Consumers: C/C++ tests, Python ctypes, examples, failover sidecar migrated
(flush_and_wait sites -> direct handle or flush+wait).
Verified: questdb-rs 1542 tests, questdb-rs-ffi 99 tests, and the C/C++
ctest suite (9/9) all pass; cdylib links cleanly.
- cargo fmt the two split files (db.rs, ffi column_sender.rs) - drop redundant closures in the sf/direct borrow entry points (pass the ffi_support borrow fns directly to borrow_cs)
- decoder bench: gate the polars-only imports as all(arrow, polars) to match their use sites (polars enables arrow-egress, not the arrow umbrella the usages are cfg'd on), fixing unused_imports under --features polars - failover sidecars: collapse nested if-let into a let-chain (clippy::collapsible_if) and rustfmt the column sidecar
The PooledQwpMock sets a short ~50ms read timeout on accepted sockets so its post-handshake read loop can poll for shutdown. That timeout also applied to the handshake read in read_request_until_blank, which propagated a transient WouldBlock/TimedOut up to upgrade_mock_stream_with_request().unwrap(), panicking the handler thread and flaking pooled_reader_rejects_prepare_after_db_close_and_closes_safely on slow CI agents (macOS surfaces the timeout as WouldBlock / EAGAIN). Retry on WouldBlock/TimedOut during the handshake (matching the steady-state read loop), bounded by a 5s overall deadline so a stuck client can't hang the handler.
The questdb submodule was 86 commits behind origin/master (a clean fast-forward, 0 commits ahead -- no divergence). Bump the gitlink 99126149 -> f8cf9e46 (current master HEAD) so it tracks current OSS master, as configured in .gitmodules (branch = master).
test_sender_kill9_sf_recovery_replays_c_client_rust published all 20k
rows in one flush (~320 KiB) while capping segments at sf_max_bytes=64
KiB. A single QWP publication must fit inside one SF segment, so the
queue rejected it:
PayloadExceedsByteCapacity { payload_len: 320058, max_bytes: 65504 }
Publish in 1k-row chunks (~16 KiB each, under the 64 KiB cap); the
sequence of publications also seals multiple .sfa segments, so the
recovery walk genuinely scans a multi-segment ring.
Caught by the enterprise-e2e-c-client pipeline on questdb-enterprise
PR #1094 (build 246294).
test_zone_failover_stays_in_zone_then_crosses_c_client_rust: two zone-A servers + one zone-B server (zone-A first in addr=, zone=A). The Reader binds zone A, stays in zone A across an intra-zone failover (kill the bound zone-A host -> surviving zone-A sibling), and crosses to zone B only once both zone-A hosts are gone (zone= is a preference, not a hard filter). Extends _egress_connect_string with zone/target/timeout knobs and adds a _wait_for_zone poll helper. Mirrors the Enterprise Java test_zone_failover.py suite; the binding under test is the Rust egress Reader.
SHOW_ZONE captured the zone value from the first batch then broke out of the loop, dropping the cursor before it was fully read. The Rust Reader poisons a connection whose cursor is dropped un-drained, so the *next* SHOW_ZONE failed with 'Reader connection is closed and cannot be reused'. This surfaced in test_zone_failover_stays_in_zone_then_crosses_c_client_rust (build 246314): the first SHOW_ZONE passed but poisoned the connection, and the _wait_for_zone poll that followed errored out. Drain the cursor to completion (capture the value once, keep calling next_batch until None). SHOW PARAMETERS returns a tiny result, so this is cheap.
Align the row-major and store-and-forward column-major senders on a single delivery-wait surface. Row-major: - Remove `Sender::await_acked_fsn(fsn, timeout) -> Result<bool>` and the `line_sender_qwpws_await_acked_fsn` C ABI entry point. - Add `Sender::wait(ack_level: AckLevel, timeout: Duration) -> Result<()>` (and on the `BorrowedRowSender` handle), mirroring `SfColumnSender::wait`. It waits for the cumulative `published_fsn` boundary at the requested AckLevel. Re-export `AckLevel` at `crate::ingress`. Column-major (store-and-forward only; direct sender untouched): - `SfColumnSender::wait` / `ColumnSender::wait` gain a per-call `timeout`, replacing the `set_sfa_sync_timeout_for_test` hook. Shared timeout semantics: `timeout` is a no-progress deadline (fires only if the ack watermark fails to advance for that long); `Duration::ZERO` waits indefinitely; on expiry returns `ErrorCode::FailoverRetry` with frames retained for replay. The direct sender keeps `commit`/`flush_and_wait` and its `request_timeout`-bounded behaviour. FFI/bindings: `line_sender_qwpws_wait` and `sf_column_sender_wait` take a `uint64_t timeout_millis`; C++ `wait(level, milliseconds = zero)` and Python `wait(ack_level=0, timeout_millis=0)` stay source-compatible. Failover sidecar AWAIT_ACKED maps a timeout back to its bool reply. Also clear all pre-existing rustdoc broken-link / link-target / bare-URL warnings (public and --document-private-items builds). Tests: questdb-rs 889 + questdb-rs-ffi 52 pass; clippy/fmt/doc clean.
The qwp_column_sidecar binary still called SfColumnSender::wait(AckLevel) with one argument, breaking the failover_clients clippy CI gate. Pass Duration::ZERO (wait indefinitely, preserving prior block-until-ack behaviour).
…t sender Extend the c_client e2e suite with deterministic scenarios ported from the Enterprise reference suite (questdb-ent/e2e/tests), all driven by the Rust qwp_sidecar via the shared lib.shared_fixtures harness. Deduplicated against what this branch already has (test_sender_kill9_sf_recovery_replays is not re-added). test_failover.py (+6, -m c_client): - failover_during_active_send - two_failovers_in_one_scenario - no_request_durable_ack_loses_rows (negative/honesty test) - orphan_drainer_durable_ack_survives_kill - sender_repeated_sigkill_no_state_corruption (multi-cycle SF recovery) - partial_ack_sealed_segment_replay_dedup_collapses New files (-m c_client): - test_failover_graceful.py: graceful_failover_round_trip (demote/promote, write-rejection + connection-survival probes) - test_switch.py: write_path_across_switch, disturbance_honesty_guard - test_switch_roundtrip_crash_repro.py: roundtrip_post_switch_write_no_crash New file (separate cadence, -m "c_client_rust and fuzz"): - test_failover_fuzz.py: random_failover (parametrized). Tagged fuzz + c_client_rust but NOT c_client, so -m c_client stays deterministic. Registers the fuzz marker in pyproject.toml. Reuses jh's _connect_string (username= keyword, sender_id/sf_max_bytes kwargs) and second-sidecar spawn pattern. -m c_client now selects 18 deterministic tests.
… _egress query_pool_min/query_pool_max on DbInner are #[cfg(feature = "_egress")], but the Debug impl read them unconditionally, breaking the no-default-features sync-sender CI build (E0609).
The connection-pool refactor removed the chunk clear-and-reuse path: `column_sender_chunk_clear` (C/FFI) and `column_chunk::clear` (C++) were dropped, and the Rust `Chunk::clear` was demoted to a crate-private `clear_after_successful_flush`. Restore the C/C++/FFI surface and the examples/tests that exercise it (reverted to their prior form), and bring `Chunk::clear` back as a public method the FFI crate can call. `Chunk::clear` documents the safe-Rust caveat: a chunk borrows its column buffers for its entire lifetime, so clearing does not enable streaming reuse — every buffer ever appended must stay alive until the chunk is dropped. Reuse is only feasible when the whole dataset is resident for the chunk's lifetime; a bounded-batch source should create a fresh chunk per batch. The internal flush path and the FFI reuse path erase the lifetime and manage buffer validity manually, so they reuse one chunk and retain its descriptor-vec capacity. The qwp_ws_l1_quotes example reuses a single chunk again, matching the case it was written for: the full dataset is generated up front and kept in memory, so each flush borrows a slice of the same long-lived arrays. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ests CI clippy (-D warnings) moved to Rust 1.97, which added manual_is_multiple_of (7 sites: % ... == 0 on unsigned counters) and byte_char_slices (2 sites: &[b'='][..] -> &b"="[..]). Swept all targets for both patterns, not just the two the aborted CI run reported.
Rejection diagnostics and the wait() sync boundaries were connection-level state on pooled store-and-forward senders, so a rejection for frames a previous lease published without waiting would fail the next lease's wait() (inviting a duplicating re-flush), and a rejection nobody waited for was silently lost. - Rebase the sync boundaries past everything already published and drain the rejection ring when a sender is borrowed: wait() now covers only the borrowing lease's own publications and short-circuits when it published nothing. - Recheck must_close when popping a parked free-list entry and retire the connection (releasing its disk slot) instead of lending a terminal connection to the next borrower. - Report rejections no lease observed -- drained at borrow, return, reap and pool close, or skipped by a wait() scan -- at warn level, counted in QuestDb::unobserved_rejections_total and the C ABI's questdb_db_unobserved_rejections_total.
Adds `qwp_ws_shared_pool`, an example that shares one `QuestDb` pool between an ingestion thread and a query thread over an `Arc`, and registers it in Cargo.toml behind the `sync-sender-qwp-ws` and `sync-reader-qwp-ws` features. `QuestDb` is `Send + Sync` but not `Clone`, so the intended usage is one pool per process shared through an `Arc`, with each thread taking its own short-lived borrow. The borrowed handles are neither `Send` nor `Sync`, which is easy to state but easier to understand from working code; the example shows the borrow-per-use shape, including borrowing per poll rather than holding a reader across sleeps, since a borrow occupies its pool slot until `Drop`. The example also demonstrates that an `AckLevel::Ok` ack means the server accepted the frame, not that the rows are queryable. A WAL table applies writes asynchronously, so the query thread polls a run-scoped count and prints how far visibility trails the acked watermark, converging once ingestion stops. The run marker is bound into the query so a leftover table cannot fake progress, and the watermark is sampled after the query returns so the reported lag is a true upper bound rather than a negative value clamped to zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shows the intended deployment shape for the QuestDb pool: one pool per process, wrapped in an Arc and shared across threads, with each worker taking its own short-lived borrow. An ingestion thread builds column-major batches with the Chunk API and publishes them through a BorrowedSender, checkpointing on AckLevel::Ok every eight batches. A query thread polls the same table through a BorrowedReader, watching rows become visible as the WAL is applied, then reports per-symbol stats through a bound parameter. The main thread runs its DDL through a third borrow. The synthetic feed is paced at one batch per 100 ms. Unpaced, the run ingests in 40 ms and the query thread polls twice, which hides the concurrency the example exists to show. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
qwp_ws_chunk_and_query covers the same ground: one Arc-shared pool, a Chunk-based ingestion thread, and a concurrent query thread. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The example matched each result column with an `unreachable!()` fallback, so a schema that disagreed with the program would abort the process. An example gets copied into real code, where a panic on unexpected server output is the wrong default. The query helpers now return questdb::Result and construct the failure with Error::new(ErrorCode::InvalidApiCall, ..), keeping ingestion and queries in the crate's single error vocabulary. InvalidApiCall matches how the crate already reports a caller-side mistake, such as a borrow that exceeds the pool cap. follow_trades keeps a boxed application error: its visibility deadline is this example's own policy, not a client failure, and no ErrorCode describes it. questdb::Error converts into that type, so the client calls inside it still propagate with `?`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Align the pooled sender's error model with the Java client: data-fate notification and the synchronization barrier are now orthogonal channels instead of both riding on wait()'s error-ring polling. - Every server rejection a store-and-forward runner records is published to a pool-wide rejection source at record time. With a handler (QuestDb::connect_with_handlers / questdb_db_connect_with_handlers) it is delivered on a dedicated dispatcher thread through a bounded drop-oldest inbox; without one it is logged (warn for retriable policies -- the frames are replayed, not lost -- error for terminal), so silence is never the default. Delivery and overflow are counted by rejection_events_delivered / rejection_events_dropped. - wait() / flush-and-wait is now a watermark check plus a terminal-latch check, mirroring Java's awaitAckedFsn: server rejections no longer surface from the ack barrier, retiring the range-scoped error-ring polling and the unobserved_rejections_total counter the previous commit introduced. Borrow-time terminal recheck and the lease-scoped sync-boundary rebase stay. - The connection-event dispatcher is generalized over the event type and reused for rejection delivery.
Ports questdb-rs/examples/qwp_ws_chunk_and_query.rs to C++, showing the intended deployment shape for the pool: one pool per process, shared across threads, with each worker taking its own short-lived borrow. An ingestion thread builds column-major batches with the column_chunk API and publishes them through a borrowed_sender, checkpointing on qwpws_ack_level::ok every eight batches. A query thread polls the same table through a pooled reader, watching rows become visible as the WAL is applied, then reports per-symbol stats through a bound parameter. The main thread runs its DDL through a third borrow. Three things diverge from the Rust original, each because C++ wants a different idiom. The pool is shared by reference rather than through an Arc, and is declared before the futures so the workers are joined before it is destroyed and no borrow outlives it. The workers run under std::async rather than std::thread, because an exception escaping a std::thread calls std::terminate, while future::get() re-raises on the main thread the way join().expect(..)? does in Rust. The Rust column_type_error helper has no counterpart, because column::get<T> already reports a kind mismatch as a questdb::error. One chunk is reused across all batches rather than built per batch. A successful flush clears the chunk while retaining the table name and the descriptor-vec capacity, so nothing points into the row buffers when they are refilled. Rust cannot express this: Chunk<'a> binds the buffer lifetime into the type and clear() cannot shrink 'a, so the borrows outlive the runtime invariant that makes the reuse sound. Verified against a local QuestDB. The run ingests all 100k rows and reports per-symbol stats identical to the Rust example, whose seeded xorshift64* feed it reproduces exactly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…SIGBUS the host Store-and-forward slots can live on unstable hardware; a restart after a power loss is exactly when an unreadable sector is most likely, and an mmap page fault has no error channel — it kills the host application. - Segment recovery scans (scan_file_metadata, open_existing) now read through positional read() syscalls, never a mapping: a bad sector surfaces as Err(EIO) and the existing recovery diagnostics take over. One streaming scanner (bounded 64 KiB CRC chunk) serves both the File and byte-slice paths, so scanning no longer buffers whole frames. - open_existing re-reserves the segment's blocks before mapping: a power loss can persist the file size without its extents, silently re-opening the SIGBUS-on-ENOSPC window for the recovered active segment's appends. The macOS reservation now allocates only the st_blocks shortfall — F_PREALLOCATE is append-past-EOF, not idempotent like posix_fallocate. - The ACK watermark record moves from mmap+atomics to plain positional file I/O (writes are off the send path, on ACK-driven trim only), with real zeroes instead of set_len at creation and a write-failure latch that degrades to in-memory state instead of faulting. The zero-copy mapped send path is deliberately untouched: the recovery scan probes every recovered byte via read() before anything is mapped.
…client into jh_conn_pool_refactor
Clippy 1.97 extended byte_char_slices to bare byte-char arrays (not just &[...] borrows), catching five test literals the earlier sweep skipped on the borrowed-form assumption.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
qwp_sender_poll_error drains the borrow's connection diagnostics ring (pull parity with the standalone sender); borrowing rebases the ring so a lease only ever polls rejections recorded since its borrow — earlier entries were already pushed to the pool error handler at record time. qwp_sender_error_events_dropped exposes the ring's drop counter.
Add the release runbook and compatibility matrix, align Rust/C/C++ client documentation and examples, and enforce release documentation and MSRV checks in CI.
The Windows integration run died with 0xC0000005 (access violation) inside TestQwpWsFuzz.test_non_ascii_values with no diagnostics; faulthandler turns the next occurrence into per-thread Python stacks naming the ctypes call.
Every row-API builder call (table/symbol/column/at, ~16x per s2-wide row) runs the op-order state check. The check itself is a two-instruction bitmask test, but it compiled as a real outlined call twice over: check_op -> OpState::check -> OpCase::allows, with the error-formatting machinery bloating check() past the inline threshold. - #[inline(always)] on QwpWsColumnarBuffer::check_op (both impls), OpState::check and OpCase::allows: the op argument is a compile-time constant at every call site, so after inlining the whole check const-folds to a single state compare. - Split the error construction into a #[cold] #[inline(never)] bad_op_error() so the inlined hot path stays two instructions and the formatting code drops out of the icache footprint. Measured on the QWP ingress row bench (s2-wide, batch=10k, tmpfs server, Ryzen 9950X3D): row-build floor 174 -> 147 ns/row (-15%); e2e row-flush x1 3.07M -> 3.62M rows/s (+18%); x8 unchanged (server-bound). perf shows the OpState symbols gone from the profile; cycles redistribute to append_symbol/fill_batch (the real work). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ngress Add the Gorilla codec (ingress/gorilla.rs, exact mirror of the egress decoder's bit format) and wire it through every QWP/WebSocket encode path: row-API buffer (publish + both replay builders), columnar chunk encoder (incl. designated timestamps, column and scalar), Arrow batch bodies (micro/nano/second-widening), and NumPy temporal dtypes (direct and unit-converting). WS frames now always set FLAG_GORILLA (0x04) and TIMESTAMP/TIMESTAMP_NANOS columns carry a per-column encoding discriminator (0x00 raw / 0x01 Gorilla), falling back to raw when a column has <= 2 non-null values or a delta-of-delta overflows i32 — matching the Java client's QwpWebSocketEncoder byte for byte. QWP/UDP datagrams and DATE columns stay raw. Frame-size estimates and the WS columnar size hint gain the discriminator byte so up-front try_reserve stays an upper bound. Java-golden wire fixtures are regenerated for the Gorilla-era format; the old->new byte delta was verified to be confined to flags/payload_len/timestamp sections only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Gorilla delta-of-delta compression for timestamp columns on the QWP ingress side, matching the Java client's wire behavior (
QwpWebSocketEncoder/QwpGorillaEncoder) bit for bit.ingress/gorilla.rs— exact mirror of the egress decoder's bit format (LSB-first DoD buckets: 1/9/12/16/36 bits), round-trip tested against the production egress decoder as reference.FLAG_GORILLA(0x04) and writes a per-column encoding discriminator (0x00raw /0x01Gorilla) forTIMESTAMP/TIMESTAMP_NANOScolumns: row-API buffer (publish + both replay builders), columnar chunk encoder (incl. designated ts, column and scalar), Arrow batch bodies (micro/nano/second-widening), NumPy temporal dtypes (direct and theDatetime*→microsconverting family).try_reserveremains a strict upper bound.Typical effect: a regular-interval designated-ts column drops from 8 bytes/row to ~1 bit/row (golden fixture: 81 → 19 bytes for 10 rows).
Notes for reviewers
qwp_ws_java_golden.rs,interop/qwp-unified-ingress/*.hex): the old fixtures were pre-Gorilla captures. The old↔new byte delta was verified (independently, twice) to be confined to exactly the flags byte (0x08→0x0C), the payload_len field, and the timestamp column sections. The regenerated bytes are Rust-derived; a fresh capture from the Java client would restore strict cross-client provenance and is a recommended follow-up.#[cfg(feature = "_egress")]because the reference decoder lives behind that feature and sender-only CI combos must still compilecargo test.qwp_ws_real_server_ack_order_and_reject_probe, env-gated) has a post-parse-error re-ack assertion that fails against a 9.4.4-SNAPSHOT dev server both at this head and at the base commit (bisected in a clean worktree) — pre-existing server-side behavior, unrelated to this change. The same run confirmed the server accepts and acks Gorilla-flagged frames.Verification
--features almost-all-features,arrow,polars: 2230 passed / 0 failed / 21 ignored;questdb-rs-ffibuilds, clippy clean.Deferred follow-ups
🤖 Generated with Claude Code