Skip to content

Graphql loaders#2674

Open
wyatt-joyner-pometry wants to merge 2 commits into
db_v4from
graphql-loaders
Open

Graphql loaders#2674
wyatt-joyner-pometry wants to merge 2 commits into
db_v4from
graphql-loaders

Conversation

@wyatt-joyner-pometry

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add load_nodes_from_parquet and load_edges_from_parquet GraphQL endpoints. It is specifically load_nodes_from_parquet rather than load_nodes, because I only written for this capacity at first.

Note that I am also not allowing shared_metadata, since we'll have to come up with a format for how to interpret types.

Claude mostly wrote the AppConfig changes for the allowed_parquet_paths, but there are tests for negative and positive cases of writing to allowed parquet paths.

Why are the changes needed?

We need endpoints for loading parquet files over GraphQL.

Does this PR introduce any user-facing change? If yes is this documented?

I've copied over the descriptions from the corresponding Python API.

How was this patch tested?

Added tests alongside the other existing GraphQL tests.

Are there any further changes required?

add support for shared_metadata
add support for other input formats like csv and raw blob
(more generally) add support for column name-mapping and casting

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 958e9e9 Previous: 9823ef7 Ratio
lotr_graph/num_edges 4 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph/num_nodes 4 ns/iter (± 0) 1 ns/iter (± 0) 4
lotr_graph/graph_latest 3 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph_materialise/materialize 7751732 ns/iter (± 43009) 1564816 ns/iter (± 35303) 4.95
lotr_graph_window_100/num_nodes 13 ns/iter (± 0) 5 ns/iter (± 0) 2.60
lotr_graph_window_100_materialise/materialize 7838908 ns/iter (± 31657) 1669150 ns/iter (± 10700) 4.70
lotr_graph_window_10/has_node_existing 139 ns/iter (± 9) 62 ns/iter (± 11) 2.24
lotr_graph_window_10_materialise/materialize 3282184 ns/iter (± 16164) 971980 ns/iter (± 4278) 3.38
lotr_graph_subgraph_10pc_materialise/materialize 2031221 ns/iter (± 25736) 334634 ns/iter (± 1287) 6.07
lotr_graph_subgraph_10pc_windowed/has_node_existing 140 ns/iter (± 8) 62 ns/iter (± 14) 2.26
lotr_graph_subgraph_10pc_windowed_materialise/materialize 1236749 ns/iter (± 8571) 230399 ns/iter (± 2617) 5.37
lotr_graph_window_50_layered/num_edges_temporal 141706 ns/iter (± 4023) 70121 ns/iter (± 7586) 2.02
lotr_graph_window_50_layered/has_node_existing 400 ns/iter (± 24) 129 ns/iter (± 12) 3.10
lotr_graph_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_window_50_layered/graph_latest 84023 ns/iter (± 1434) 36649 ns/iter (± 916) 2.29
lotr_graph_window_50_layered_materialise/materialize 28757836 ns/iter (± 478543) 3488825 ns/iter (± 24948) 8.24
lotr_graph_persistent_window_50_layered/num_edges_temporal 584336 ns/iter (± 6121) 192686 ns/iter (± 1569) 3.03
lotr_graph_persistent_window_50_layered/has_node_existing 415 ns/iter (± 373) 174 ns/iter (± 83) 2.39
lotr_graph_persistent_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_persistent_window_50_layered/graph_latest 131300 ns/iter (± 2402) 57549 ns/iter (± 4809) 2.28
lotr_graph_persistent_window_50_layered_materialise/materialize 49193142 ns/iter (± 455487) 5298035 ns/iter (± 147912) 9.29

This comment was automatically generated by workflow using github-action-benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant