Remote py apis - #2675
Draft
shivamka1 wants to merge 236 commits into
Draft
Conversation
…ree read wired end-to-end
…ata, update_metadata, delete_edge) through Transport
…date_metadata) through Transport
…tadata) through Transport
…aph.node/edge to Rust methods (fixes view-chain preservation in Python)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 2c7186b | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
4 ns/iter (± 0) |
1 ns/iter (± 0) |
4 |
lotr_graph/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8193499 ns/iter (± 83355) |
1564816 ns/iter (± 35303) |
5.24 |
lotr_graph_window_100/num_nodes |
14 ns/iter (± 0) |
5 ns/iter (± 0) |
2.80 |
lotr_graph_window_100_materialise/materialize |
8473612 ns/iter (± 84159) |
1669150 ns/iter (± 10700) |
5.08 |
lotr_graph_window_10/has_node_existing |
146 ns/iter (± 10) |
62 ns/iter (± 11) |
2.35 |
lotr_graph_window_10_materialise/materialize |
3430068 ns/iter (± 10169) |
971980 ns/iter (± 4278) |
3.53 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
2020046 ns/iter (± 28225) |
334634 ns/iter (± 1287) |
6.04 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
152 ns/iter (± 10) |
62 ns/iter (± 14) |
2.45 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1217668 ns/iter (± 19805) |
230399 ns/iter (± 2617) |
5.29 |
lotr_graph_window_50_layered/num_edges_temporal |
151553 ns/iter (± 1304) |
70121 ns/iter (± 7586) |
2.16 |
lotr_graph_window_50_layered/has_node_existing |
387 ns/iter (± 25) |
129 ns/iter (± 12) |
3 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/max_neighbour_degree |
847960 ns/iter (± 19297) |
386651 ns/iter (± 3853) |
2.19 |
lotr_graph_window_50_layered/graph_latest |
83486 ns/iter (± 1318) |
36649 ns/iter (± 916) |
2.28 |
lotr_graph_window_50_layered_materialise/materialize |
31815666 ns/iter (± 235486) |
3488825 ns/iter (± 24948) |
9.12 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
648085 ns/iter (± 9447) |
192686 ns/iter (± 1569) |
3.36 |
lotr_graph_persistent_window_50_layered/has_node_existing |
437 ns/iter (± 434) |
174 ns/iter (± 83) |
2.51 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
3642227 ns/iter (± 35164) |
1659940 ns/iter (± 19402) |
2.19 |
lotr_graph_persistent_window_50_layered/graph_latest |
125161 ns/iter (± 1197) |
57549 ns/iter (± 4809) |
2.17 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
54898390 ns/iter (± 223760) |
5298035 ns/iter (± 147912) |
10.36 |
This comment was automatically generated by workflow using github-action-benchmark.
…eld from RemoteGraph
…e impls from Py* types to op types
…Degree/OutDegree/Name) with Python bindings + tests
…)/.dst() navigation
…optional-string machinery
…ypes, ExcludeNodes) + Valid, DefaultLayer, and graph Path/Namespace/Name terminals
…ighbour accessors
… wrong answers under view chains
…est, layer, shrink_*, etc.)
…est, layer, shrink_*, etc.)
…p through base_graph on all Remote types
…verse views to_list/to_list_rev duplicated collect/collect_rev on the history sub-collections, locally and remotely; both sets are gone. __reversed__ on the remote history and its sub-collections now goes through a lazy reverse() view backed by the server's reverse field, so it stays one RPC and composes with any future optimised iterator instead of materializing a reversed list. Interval reversal keeps the local semantics: reversing the history flips the sign of the deltas.
len() and `in` on the remote history and its sub-collections downloaded the entire history to answer a scalar question. The server's History type gains contains (by timestamp, optionally with the event id), and the timestamps, event-id and intervals sub-views gain count and contains; the client asks those instead. Membership takes whatever converts to an EventTime, exactly as locally — a bare int means (t, event_id=0), and a naive datetime is not a member and never reaches the wire.
The columnar views fetched per-member {key, value} entries and pivoted
them by matching the key — after the wire had already been asked for a
specific column. Requested columns now render as aliased single-key gets
and decode straight into columns, so get() takes its column rather than
searching for it, and ColumnarProps, member_value and fetch_with_keys are
gone. The all-columns fetch was dead code, so keys is no longer optional:
one wire shape, one decode shape. A missing alias in the response is a
protocol error, distinct from null, which is an absent value.
Renames GqlNestedEdges.nn to edges and the nested-collection count to len (it is the outer source count, as local NestedEdges.len is), deletes the empty matching module, drops 29 redundant nested json! calls, returns the bound dict and iterator directly instead of erasing and unbinding them, and corrects the reverse-filter-conversion comment: the client sends the unified GqlFilter, and a Layer::All view op is dropped rather than rejected. parse_read now takes the response map itself: the first path segment indexes it directly, so the response is no longer re-wrapped into a JsonValue per read, and everything below the root field borrows.
prop_str, prop_list and prop_map_get join the expect_* family in the transport module, where every other result decoder lives; remote_schema keeps only its own record shapes. The graph model modules and their resolver types are public so plugins can extend them.
PathFromGraph.count is the number of sources, not of neighbours; local spells that len(), so the client does too and the python-only count() wrapper goes. The single-handle server fixture moves to test_utils beside the one it wraps, and a few test nits: the confusing filter docstring says what it means, a redundant hasattr assertion and some list() wrappers are gone.
… enum-argument spelling
…rtyKeys/metadataKeys fields
…d survives rendering
…e cannot silently skip
…it duplicated window
This was referenced Aug 21, 2026
# Conflicts: # python/python/raphtory/__init__.pyi # python/python/raphtory/graphql/__init__.pyi # raphtory-graphql/src/model/graph/edges.rs # raphtory-graphql/src/model/graph/filtering.rs # raphtory-graphql/src/model/graph/nodes.rs # raphtory-graphql/src/model/graph/path_from_node.rs # raphtory/src/db/graph/edges.rs # raphtory/src/db/graph/nodes.rs
…es as a deliberate remote-only extra
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remote Python APIs
Adds a Python
RemoteGraphclient that is a drop-in replacement for the localGraphAPI, backed by a typed transport over GraphQL. Local code migrates byswapping the constructor — method names, signatures, and semantics match the
local API; server-only extras are additive.
Architecture
Transporttrait — the client core isexecute(&Op) -> Result<Option<Prop>>over a typed operation tree (
Op = Read(ReadExpr) | Write(WriteOp)). TheGraphQL rendering lives behind the trait; nothing above it builds query strings.
window,layers,at, filters, …) accumulate in aReadExprtree and fire a single requestwhen a terminal (
degree(),count(),collect(), …) is reached.serde-serialized variables (no string interpolation or templates). The serde
impls on the wire types are the single source of truth for the format.
collect()returns handlesthat replay the parent's op log (views + filters), so members evaluate under
the same view chain as the collection they came from.
event_graph()/persistent_graph()on aremote graph reinterpret the same server graph under the other semantics,
zero-copy: the server converts at load time via
graph(path, graphType:).As locally, they apply to the base graph rather than to views.
Unified filtering
filter(expr: GqlFilter)field on Graph/Node/Edge and all collections.GqlFilteris a oneOf input:nodes,edges,graph,and,or,not,plus flat graph-view spellings (
window,at,layers,snapshotAt, …).andis intersection, matching localg.filter(a & b); cross-kindorproduces a proper graph union.
compositions like
filter.Graph.window(1, 4) & (filter.Node.property("x") > 1))are exported from core as a kind-tagged
FilterTreeand carried to the wirewithout loss.
and/orlists are rejected — previously an emptyormatchedeverything, which mattered for persisted access filters.
Wire fidelity
PropertyTypescalar and
dtypefields; the client decodes stored values type-directed,so numeric widths (
U8staysU8), datetimes, and decimals survive thewire exactly. Older servers without
dtypefall back to shape-based decoding.f64Special/f32Specialinput variants and string sentinels on output(previously: error on write, silent null on read).
Prop::Mapis now insertion-ordered (IndexMap) andthe wire preserves key order end to end, matching local behavior.
Performance
Edge.event(...)pins events via indexed layer/time views instead ofscanning the full exploded history.
Arc; nested columnar reads batch persource; history slicing fetches pages rather than full histories.
Breaking changes
GraphQL schema
filterNodes/filterEdgesare removed. Migration is mechanical andcan preserve response shapes with an alias:
filterNodes(expr: X)→filterNodes: filter(expr: { nodes: X }).{nodes: {name: {where: ...}}}/id:/nodeType:. The enum-argument spelling{node: {field: NODE_NAME, where: ...}}still works but is deprecated.shrinkWindowis removed from all types and the apply-views inputs:internal_windowalready clamps requested bounds to the current view, so itwas observably identical to
window.shrinkStart/shrinkEndremain(not expressible via
window, which needs both bounds).node/edgekeys still load (serdealiases); hand-written GraphQL must use
nodes/edges.PropertyOutputemits"NaN"/"Infinity"/"-Infinity"strings fornon-finite floats where it previously emitted
null.Python API — removals (local and remote)
find_nodes/find_edges: legacy equality-only shortcuts fully subsumedby the filter API.
node[key]/edge[key]property reads: hid the property/metadatadistinction, and node vs edge disagreed on missing keys (
KeyErrorvsNone). Useproperties.get(key)/properties[key].shrink_window: identical towindow(see above).shrink_start/shrink_endremain.count(): a redundant alias; uselen(collection). (Removedfrom local
Edgesand every remote collection.)RemoteTemporalProperty.latest(): a redundant alias ofvalue(), whichlocal never had.
Python API — behaviour changes
and a write with the wrong id type is refused exactly as the local
Graphrefuses it. (Previously ints were silently stringified.)
add_node((5, 3), ...)raises thesame
TypeErroras local instead of silently dropping the event id.keys()returns the graph's registered keys for the entitykind (matching local), instead of the first member's keys — previously
values()/items()/as_dict()could silently drop whole columns.property_typereturns aPropTypeobject instead of a displaystring, so schema comparisons are order-stable.
RemotePathFromGraph.count()removed (nested count was ambiguous; uselen()).Python API — additions (local, closing remote↔local gaps)
filter(expr)onEdge,Edges, andNestedEdges(core supported it;the bindings were missing).
sorted(sort_bys)onNodesandEdges, withNodeSortBy/EdgeSortBy/SortByTime— one shared set of classes used by bothlocal and remote, so drop-in code passes the same key objects to either
side. The sort comparators moved from the GraphQL crate into core; the
server now delegates.
Build fix
pyo3/extension-moduleis now pinned under[tool.maturin]— without it thewheel links libpython and
import raphtorysegfaults on statically-linkedinterpreters (e.g. conda).
Testing
(views, filters, collections, properties, write path), including key-order,
NaN, and dtype round-trips.
Not in this PR (local APIs the remote client does not yet cover)
Rolling / expanding windows —
rolling()andexpanding()(and theWindowSetthey return) are not available on any remote object (Remote expanding() and rolling() window sets #2723).All other time views (
window,at,before,after,snapshot_*,shrink_start,shrink_end) are supported.Collection-level history and node states —
.historyon collections(and
.deletionson edge collections) are not exposed remotely; per-entitynode.history/edge.deletionsare. Same root gap: there is no remoteNodeState, so collection accessors like
nodes.degree()return plain listsrather than the sortable/groupable NodeState views the local API returns
(Remote NodeState subsystem: collection history/deletions, columnar temporal view, history merge #2722, which depends on the server-side Implement the GraphQL NodeState type properly #2730).
Algorithms — the
raphtory.algorithmsmodule (andGraph.largest_connected_component) operate on local graphs only.Vectors / embeddings —
Graph.vectoriseand theraphtory.vectorsmodule (server-side vector endpoints remain reachable via raw
client.query(...)on vectors-enabled servers).Materialization —
materialize,materialize_at: building a newstandalone graph out of a (possibly viewed) graph needs server-side graph
creation, so it is deferred. Flavour conversion is not part of this gap
and is supported — see above.
Minor surface gaps — columnar collection property views lack the
collection-level
.temporalview;History.merge/compose_historiesare unavailable on remote history handles. (Both ledgered as strict xfails
in the parity suite, tracked in Remote NodeState subsystem: collection history/deletions, columnar temporal view, history merge #2722.)
DataFrame / Parquet bulk IO —
load_nodes/load_edges/load_*_metadata,from_parquet/to_parquet. Batch remote writes exist(
RemoteGraph.add_nodes/add_edges) but take update lists, notDataFrames — a deliberate remote-only extra that amortizes network
round-trips, with no local counterpart planned.
Export —
to_networkx,to_pyvis, and collectionto_df.Cross-graph import — the
import_node(s)/import_edge(s)(_as)family, which copies entities from another in-process graph.
Local persistence (not applicable remotely) —
save_to_file/load_from_file/serialise/deserialiseand the disk cache(
cache_view,flush) are in-process concepts; whole-graph transfer iscovered by
RaphtoryClient.send_graph/receive_graph/upload_graph.