Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Fixed

- Parquet readers now support Zstandard-compressed files. The native crate had
disabled the `parquet` crate's default features without explicitly enabling
its `zstd` feature, so reads failed at runtime with `Disabled feature at
compile time: zstd`.

## [0.7.2] - 2026-07-08

### Fixed
Expand Down
6 changes: 6 additions & 0 deletions docs/parquet_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ parquet_bytes = File.read!("/data/events.parquet")
batch = ExArrow.Stream.next(stream)
```

### Compression support

ExArrow reads uncompressed and Zstandard-compressed Parquet files. Compression
is decoded in native memory as record batches are requested; it does not change
the lazy row-group streaming behavior.

### Schema introspection

`ExArrow.Stream.schema/1` never fails for Parquet streams (the schema is
Expand Down
2 changes: 2 additions & 0 deletions lib/ex_arrow/parquet/reader.ex
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ defmodule ExArrow.Parquet.Reader do
Parquet file reader: open a `.parquet` file or an in-memory binary and
receive an `ExArrow.Stream` that yields record batches.

Uncompressed and Zstandard-compressed Parquet files are supported.

The stream interface is identical to `ExArrow.IPC.Reader` and ADBC streams —
use `ExArrow.Stream.schema/1`, `ExArrow.Stream.next/1`, and
`ExArrow.Stream.to_list/1` to consume it.
Expand Down
66 changes: 65 additions & 1 deletion native/ex_arrow_native/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion native/ex_arrow_native/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ arrow-buffer = { version = "56", default-features = false }
arrow-data = { version = "56", default-features = false }
arrow-select = { version = "56", default-features = false }
arrow-ord = { version = "56", default-features = false }
parquet = { version = "56", default-features = false, features = ["arrow"] }
parquet = { version = "56", default-features = false, features = ["arrow", "zstd"] }
arrow-flight = { version = "56", features = ["flight-sql"] }
futures = "0.3"
tokio = { version = "1", features = ["rt-multi-thread", "net", "sync", "time", "macros"] }
Expand Down
15 changes: 15 additions & 0 deletions test/ex_arrow/parquet_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,21 @@ defmodule ExArrow.ParquetTest do
assert ExArrow.RecordBatch.num_rows(rt_batch) == ExArrow.RecordBatch.num_rows(batch)
end

test "reads a Zstandard-compressed file" do
path = Path.expand("../fixtures/parquet_zstd.parquet", __DIR__)

assert {:ok, stream} = Parquet.Reader.from_file(path)
assert {:ok, schema} = Stream.schema(stream)
assert Schema.field_names(schema) == ["id", "name"]

rows =
stream
|> Stream.to_list()
|> Enum.sum_by(&ExArrow.RecordBatch.num_rows/1)

assert rows == 3
end

test "from_file returns error for missing file" do
assert {:error, _msg} = Parquet.Reader.from_file("/tmp/this_does_not_exist_xyz.parquet")
end
Expand Down
7 changes: 6 additions & 1 deletion test/fixtures/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,10 @@

- **IPC file format (golden):** File-format bytes are produced by `ExArrow.Native.ipc_test_fixture_file_binary/0` (schema: `id` int64, `name` utf8; one batch of 2 rows). Tests use this for `ExArrow.IPC.File.from_binary/1` and for compatibility checks.
- **IPC from_file:** Tests that need a path write a temp file with `ExArrow.Native.ipc_file_writer_to_file/3` and remove it in an `after` block.
- **Zstandard Parquet:** `parquet_zstd.parquet` contains three rows with `id` (int64) and `name` (utf8) columns. It verifies compatibility with compressed files produced outside ExArrow. Regenerate it with:

No pre-generated `.arrow` files are committed; the single file-format fixture is generated in Rust for reproducibility.
```sh
duckdb -c "COPY (SELECT * FROM (VALUES (1::BIGINT, 'alpha'), (2::BIGINT, 'beta'), (3::BIGINT, 'gamma')) AS t(id, name)) TO 'test/fixtures/parquet_zstd.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);"
```

No pre-generated `.arrow` files are committed; the IPC file-format fixture is generated in Rust for reproducibility.
Binary file added test/fixtures/parquet_zstd.parquet
Binary file not shown.