Skip to content

Conversation

@alex-fung
Copy link
Contributor

user will need to add primary_key_columns

@alex-fung alex-fung requested a review from Copilot March 8, 2025 04:49
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Overview

This PR adds support for deduplication on data append operations by introducing a new flag (dedupe_on_append) in the push method and a corresponding API call that accepts primary key columns.

  • Added a new method (_import_data_from_append_only_dedupe_presigned_url) for handling dedupe imports.
  • Extended the push method to include dedupe_on_append and primary_key_columns parameters.

Reviewed Changes

File Description
chakra_py/client.py Added dedupe functionality and parameters to support de-duplication logic

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

chakra_py/client.py:334

  • Consider adding validation to ensure primary_key_columns is not empty when dedupe_on_append is True, to prevent potential issues when interacting with the deduplication API.
if dedupe_on_append:

Comment on lines +283 to 285
primary_key_columns: list[str] = [],
) -> None:
"""Push data to a table."""
Copy link

Copilot AI Mar 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a mutable default (an empty list) for primary_key_columns may lead to unexpected behavior if modified. Consider using None as the default and initializing an empty list within the function body.

Suggested change
primary_key_columns: list[str] = [],
) -> None:
"""Push data to a table."""
primary_key_columns: Optional[list[str]] = None,
) -> None:
"""Push data to a table."""
if primary_key_columns is None:
primary_key_columns = []

Copilot uses AI. Check for mistakes.
@alex-fung alex-fung merged commit ef3ec31 into main Mar 8, 2025
5 checks passed
@alex-fung alex-fung deleted the afung/import_parquet_append_only branch March 8, 2025 04:52
@alex-fung alex-fung requested a review from 0xnirmal March 8, 2025 04:52
@alex-fung
Copy link
Contributor Author

feel free to comment, will follow up after

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants