Skip to content

feat: Support time travel for reading Lance dataset#18

Open
fightBoxing wants to merge 1 commit intomainfrom
feature/time-travel
Open

feat: Support time travel for reading Lance dataset#18
fightBoxing wants to merge 1 commit intomainfrom
feature/time-travel

Conversation

@fightBoxing
Copy link
Copy Markdown
Collaborator

Summary

Implements time travel support for reading Lance datasets, addressing #5.

Changes

Configuration Options

  • read.version (Integer): Specify a dataset version number to read a specific historical version
  • read.timestamp (Long): Specify a timestamp in milliseconds to derive the dataset version (best-effort)

Implementation Details

  • Updated LanceOptions with readVersion and readTimestamp fields, including validation
  • Updated LanceSource.open() to use ReadOptions.Builder.setVersion() when a version is specified
  • Updated LanceDynamicTableFactory to expose new config options for SQL DDL usage
  • Updated LanceSource.Builder with version() and timestamp() methods

Usage Examples

DataStream API:

LanceSource source = LanceSource.builder()
    .path("/path/to/dataset")
    .version(3)  // Read version 3
    .build();

SQL DDL:

CREATE TABLE lance_table (
    id BIGINT,
    name STRING
) WITH (
    'connector' = 'lance',
    'path' = '/path/to/dataset',
    'read.version' = '3'
);

Testing

  • Added 12 unit tests covering version/timestamp time travel, validation, and builder API

Closes #5

- Add read.version option for version-based time travel
- Add read.timestamp option for timestamp-based time travel
- Update LanceSource to use ReadOptions for opening specific dataset versions
- Update LanceDynamicTableFactory with new config options
- Add comprehensive unit tests for time travel configuration
- Validate mutual exclusivity of version and timestamp options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support time travel for reading Lance dataset

1 participant