ch-arrow-http

ClickHouse HTTP client for Apache Arrow Stream format.

Send RecordBatch data to ClickHouse over HTTP using the ArrowStream format — no native TCP protocol needed. Works behind reverse proxies, supports authentication via headers, and uses reqwest with retry middleware.

Write operations (insert, insert_many, execute) are idempotent-safe: a deterministic query_id is computed from the request body and SQL, sent as a URL parameter. ClickHouse deduplicates by query_id, so retries (e.g. on network errors) never produce duplicate data.

Usage

use ch_arrow_http::{ChConfig, HttpClient, ChError};

let config = ChConfig::from_url("http://localhost:8123")?;
let client = HttpClient::new(config)?;

let batch = /* your RecordBatch */;

// Insert — returns a stream that yields once
let mut stream = client.insert("INSERT INTO my_table", batch).await?;
while let Some(result) = stream.next().await {
    result?;
}

// Query — returns a single RecordBatch
let result = client.query("SELECT * FROM my_table").await?;

// Execute — DDL or INSERT without return data
client.execute("OPTIMIZE TABLE my_table").await?;

Configuration

ChConfig is deserializable via serde (JSON, YAML, TOML) and has a bon-rs builder.

Builder

use ch_arrow_http::ChConfig;

let config = ChConfig::builder()
    .url("https://user:pass@clickhouse.example.com:8443/mydb")
    .timeout_ms(60_000)
    .retry_max_retries(5)
    .build();

From URL

let config = ChConfig::from_url(
    "http://admin:secret@localhost:8123/analytics?max_threads=4"
)?;

Serde (JSON/YAML/TOML)

url: http://localhost:8123
username: default
password: ""
database: default
compression: lz4
timeout_ms: 30000

Compression

Arrow IPC payloads are compressed by default using LZ4_FRAME, matching ClickHouse's default.

• Writes (INSERT): RecordBatch data is compressed with LZ4_FRAME before sending. ClickHouse ≥23.8 auto-decompresses on receipt. Set compression: None for older servers.
• Reads (query): The client automatically requests compressed responses from ClickHouse via output_format_arrow_compression_method. The Arrow StreamReader decompresses transparently — no client-side changes needed.
• Available methods: Lz4 (default), Zstd, None.

Configure via ChConfig:

use ch_arrow_http::{ChConfig, CompressionMethod};

// Default (Lz4)
let config = ChConfig::from_url("http://localhost:8123")?;

// Zstd
let config = ChConfig::builder()
    .url("http://localhost:8123")
    .compression(CompressionMethod::Zstd)
    .build();

// Disable compression
let config = ChConfig::builder()
    .url("http://localhost:8123")
    .compression(CompressionMethod::None)
    .build();

Note: Arrow IPC compression for INSERT requires ClickHouse ≥23.8. Older versions should use compression: None. Query responses are always decompressed automatically by the client regardless of the server version.

How it works

The crate:

1. Serializes RecordBatch into Arrow IPC Stream format via arrow::ipc::writer::StreamWriter
2. Computes a deterministic query_id from the request body and SQL (xxhash64), sent as a URL parameter — makes write retries safe via ClickHouse deduplication
3. Sends a POST request to ClickHouse HTTP interface with X-ClickHouse-Format: ArrowStream header
4. Deserializes responses via arrow::ipc::reader::StreamReader

The format is always specified via the X-ClickHouse-Format header, never embedded in the SQL string. This works with any SQL — CTEs, subqueries, WITH clauses, etc.

API

Method	Returns	Description
HttpClient::new(config)	ChResult<Self>	Build client from config
client.insert(query, batch)	Stream<Item=ChResult<()>>	Insert a single batch (idempotent)
client.insert_many(query, batches)	Stream<Item=ChResult<()>>	Insert multiple batches (idempotent)
client.execute(query)	ChResult<()>	Execute DDL/INSERT (idempotent)
client.query(query)	ChResult<RecordBatch>	SELECT query, returns ArrowStream

Integration tests

Integration tests use testcontainers to spin up a real ClickHouse instance in Docker:

cargo test --test roundtrip
cargo test --test insert_many
cargo test --test types
cargo test --test execute
cargo test --test query_errors
cargo test --test settings

Requires Docker to be running.

License

MIT OR Apache-2.0