feat: add support for reading deletion-vector tables

[lukany]

Add support for reading deletion-vector tables (modern Databricks / Unity
Catalog) by switching load_as_polars to Polars' native Delta reader
(pl.scan_delta) by default. The pyarrow reader rejects such tables with
DeltaProtocolError; the native reader reads them. Native DV support landed in
polars 1.40.0, so the floor is raised to >=1.40.0.

partition_filter (and PartitionFilterOperator) are kept, but now route to the
pyarrow reader, which pushes the predicate into delta-rs's file enumeration and
prunes partitions — the native reader does not do this efficiently
(pola-rs/polars#20998). The native reader skips non-matching partitions' data but
still handles per-file metadata for every partition, which on object storage is a
network request per file, so reading a few partitions of a many-partition table
can be far slower. The pyarrow path cannot read
deletion-vector tables and raises DeltaProtocolError for them; read those
natively and .filter() the returned LazyFrame instead.

Passing a cached DeltaTable to scan_delta bypasses Polars' own protocol check,
which would silently return all-null columns for column-mapping (and
reader-version-2) tables; load_as_polars now guards the protocol and raises
DeltaProtocolError for them.

[mbelak-dtml]

issue: polars native scan_delta currently does not support partition pruning. In effect, it means that reading a single partition from a Delta table with a large number of partitions can be orders of magnitude slower compared to the previous approach with pyarrow.

See pola-rs/polars#20998

I suggest keeping both approaches, with a note that if pyarrow partition filtering is used, tables which use deletion vectors cannot be read (and enforce this with an exception).

[lukany]

issue: polars native scan_delta currently does not support partition pruning. In effect, it means that reading a single partition from a Delta table with a large number of partitions can be orders of magnitude slower compared to the previous approach with pyarrow.

See pola-rs/polars#20998

I suggest keeping both approaches, with a note that if pyarrow partition filtering is used, tables which use deletion vectors cannot be read (and enforce this with an exception).

Good catch!

I've reproduced that behavior locally and created and opt-in benchmark test for pytest (opt-in because it can be flaky as it depends on speed - I found no way to verify that the native polars reads the partition metadata other than by simulating high number of partitions and demonstrating the slowdown.

Ready for review.

[mbelak-dtml]

suggestion: remove explicit mention of only version 1 or {MAX_SUPPORTED_READER_VERSION} as this formulation is coupled to the assumption that MAX_SUPPORTED_READER_VERSION == 3 which might change in deltalake (probably will change in the near future as deltalake is adding support for Delta features).

I would unify this version check with the check above (rough formulations):

if version > MAX_SUPPORTED_READER_VERSION or version == NOT_SUPPORTED_READER_VERSION:
    f'The table requires reader version {version}, which the native '
    f'Polars Delta reader does not support. Supported reader versions are: <= {MAX_SUPPORTED_READER_VERSION}, != {NOT_SUPPORTED_READER_VERSION}'

[mbelak-dtml]

suggestion: I would not mention the number from a specific use case here. Rather, I would make a general statement that the slowdown can be one or more orders of magnitude when reading a small number of partitions from a table with a huge total number of partitions.

[mbelak-dtml]

nitpick: to showcase that the operations produce the same results between the polars-native way and the pyarrow-options-way, it would be nicer if the queries was equivalent.

I think it's good to showcase that IN operator is supported for partition filtering as it is the way to implement an 'OR' condition (since multiple partition filters have 'AND' semantics).

Suggested change

	('country', PartitionFilterOperator.EQUAL, 'CZ'),
	('country', PartitionFilterOperator.IN, ['CZ', 'SK']),