Graphql loaders#2674
Open
wyatt-joyner-pometry wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 958e9e9 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
7751732 ns/iter (± 43009) |
1564816 ns/iter (± 35303) |
4.95 |
lotr_graph_window_100/num_nodes |
13 ns/iter (± 0) |
5 ns/iter (± 0) |
2.60 |
lotr_graph_window_100_materialise/materialize |
7838908 ns/iter (± 31657) |
1669150 ns/iter (± 10700) |
4.70 |
lotr_graph_window_10/has_node_existing |
139 ns/iter (± 9) |
62 ns/iter (± 11) |
2.24 |
lotr_graph_window_10_materialise/materialize |
3282184 ns/iter (± 16164) |
971980 ns/iter (± 4278) |
3.38 |
lotr_graph_subgraph_10pc_materialise/materialize |
2031221 ns/iter (± 25736) |
334634 ns/iter (± 1287) |
6.07 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
140 ns/iter (± 8) |
62 ns/iter (± 14) |
2.26 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1236749 ns/iter (± 8571) |
230399 ns/iter (± 2617) |
5.37 |
lotr_graph_window_50_layered/num_edges_temporal |
141706 ns/iter (± 4023) |
70121 ns/iter (± 7586) |
2.02 |
lotr_graph_window_50_layered/has_node_existing |
400 ns/iter (± 24) |
129 ns/iter (± 12) |
3.10 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/graph_latest |
84023 ns/iter (± 1434) |
36649 ns/iter (± 916) |
2.29 |
lotr_graph_window_50_layered_materialise/materialize |
28757836 ns/iter (± 478543) |
3488825 ns/iter (± 24948) |
8.24 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
584336 ns/iter (± 6121) |
192686 ns/iter (± 1569) |
3.03 |
lotr_graph_persistent_window_50_layered/has_node_existing |
415 ns/iter (± 373) |
174 ns/iter (± 83) |
2.39 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/graph_latest |
131300 ns/iter (± 2402) |
57549 ns/iter (± 4809) |
2.28 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
49193142 ns/iter (± 455487) |
5298035 ns/iter (± 147912) |
9.29 |
This comment was automatically generated by workflow using github-action-benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add load_nodes_from_parquet and load_edges_from_parquet GraphQL endpoints. It is specifically load_nodes_from_parquet rather than load_nodes, because I only written for this capacity at first.
Note that I am also not allowing shared_metadata, since we'll have to come up with a format for how to interpret types.
Claude mostly wrote the AppConfig changes for the allowed_parquet_paths, but there are tests for negative and positive cases of writing to allowed parquet paths.
Why are the changes needed?
We need endpoints for loading parquet files over GraphQL.
Does this PR introduce any user-facing change? If yes is this documented?
I've copied over the descriptions from the corresponding Python API.
How was this patch tested?
Added tests alongside the other existing GraphQL tests.
Are there any further changes required?
add support for shared_metadata
add support for other input formats like csv and raw blob
(more generally) add support for column name-mapping and casting