Skip to content

Add builder to help create Schemas for shredding (ShreddedSchemaBuilder) #8940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
XiangpengHao wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add builder to help create Schemas for shredding (ShreddedSchemaBuilder) #8940

XiangpengHao wants to merge 4 commits into from
+460 −24

Conversation

@XiangpengHao
Copy link
Contributor

@XiangpengHao XiangpengHao commented Dec 1, 2025 •
edited
Loading

Rationale for this change

Basically a helper to simplify this:

let shredding_type = ShredTypeBuilder::default()
    .with_path("a", &DataType::Int64)
    .with_path("b.c", &DataType::Utf8)
    .with_path("b.d", &DataType::Float64)
    .build();

assert_eq!(
    shredding_type,
    DataType::Struct(Fields::from(vec![
        Field::new("a", DataType::Int64, true),
        Field::new(
            "b",
            DataType::Struct(Fields::from(vec![
                Field::new("c", DataType::Utf8, true),
                Field::new("d", DataType::Float64, true),
            ])),
            true
        ),
    ]))
);

What changes are included in this PR?

  1. Added ShredTypeBuilder
  2. Updated existing tests cases to use this new primitive

Are these changes tested?

Yes

Are there any user-facing changes?

Add a new public interface

@github-actions github-actions bot added the parquet-variant parquet-variant* crates label Dec 1, 2025
klion26
klion26 approved these changes Dec 2, 2025
View reviewed changes
Copy link
Member

@klion26 klion26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, LGTM, left some nit comments to be considered.

parquet-variant-compute/src/shred_variant.rs
if child_fields.is_empty() {
None
} else {
Some(DataType::Struct(Fields::from(child_fields)))
Copy link
Member

@klion26 klion26 Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems we can construct DataType::Struct and primitive types (use "" as the path), and no List types, not sure if we need to add the information struct information in the builder name or anywhere.

Copy link
Contributor

@alamb alamb Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what you mean by this

Copy link
Member

@klion26 klion26 Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShredTypeBuilder is primarily used for building Struct data types. Since Variants can be shredded as either Structs or Arrays, the previous comment raises the question of whether we should inform users that this cannot be used to build Array schemas.

parquet-variant-compute/src/shred_variant.rs
}

#[test]
fn test_variant_schema_builder_conflicting_path() {
Copy link
Member

@klion26 klion26 Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice tests!

@alamb
Copy link
Contributor

alamb commented Dec 3, 2025

FYI @mhilton -- this PR may be interesting to you

alamb
alamb approved these changes Dec 3, 2025
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @klion26 and @XiangpengHao -- this is great

The only thing I think we should consider is supporting FieldRefs directly rather than assuming DataType / nullable fields.

I left some other comments that might be useful but are not necessary in my opinion

parquet-variant-compute/src/shred_variant.rs
if child_fields.is_empty() {
None
} else {
Some(DataType::Struct(Fields::from(child_fields)))
Copy link
Contributor

@alamb alamb Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what you mean by this

parquet-variant-compute/src/shred_variant.rs
Field::new("age", DataType::Int64, true),
]));
let schema2 = ShredTypeBuilder::default()
.with_path("id", &DataType::Int32)
Copy link
Contributor

@alamb alamb Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this certainly is much nicer now

@alamb alamb changed the title Add variant ShredTypeBuilder Add builder to help create Schemas for shredding (ShredTypeBuilder) Dec 3, 2025
@alamb
Copy link
Contributor

alamb commented Dec 3, 2025

FYI @scovich in case you have seen similar APIs or have ideas

@alamb alamb mentioned this pull request Dec 3, 2025
Open
mhilton
mhilton reviewed Dec 4, 2025
View reviewed changes
Copy link
Contributor

@mhilton mhilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice if this builder had a similar structure to VariantBuilder with new_object and new_list. A list can only have one type so it wouldn't work exactly the same though. However such an API and the proposed API aren't mutually exclusive.

parquet-variant-compute/src/shred_variant.rs

/// Internal tree node structure for building variant schemas.
#[derive(Clone)]
enum VariantSchemaNode {
Copy link
Contributor

@mhilton mhilton Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be nice if this type also supported an Array type too. Although the path syntax would need to be extended a little. Possibly you could use .0 to indicate an array?

Copy link
Contributor Author

@XiangpengHao XiangpengHao Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow List is not supported by shred_variant yet, maybe we can wait until array shredding is implemented

@scovich
Copy link
Contributor

scovich commented Dec 4, 2025

Traveling this week but would love to take a look when I get a chance, hopefully next week

@XiangpengHao XiangpengHao changed the title Add builder to help create Schemas for shredding (ShredTypeBuilder) Add builder to help create Schemas for shredding (ShreddedSchemaBuilder) Dec 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb approved these changes

+2 more reviewers

@mhilton mhilton mhilton left review comments

@klion26 klion26 klion26 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Variant] easier way to construct a shredded schema

5 participants

@XiangpengHao @alamb @scovich @mhilton @klion26