feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow)

[zachwinter]

Summary

First icebug (NetworKit) bridge for the algo extension: CALL GDS_PAGE_RANK('G') RETURN node, rank delegates PageRank to libnetworkit instead of the hand-rolled implementation. This is the start of wiring icebug into the CALL GDS_* paths we discussed on Discord (deprecating the hand-rolled algos in favor of icebug's zero-copy Arrow columnar engine).

Per that plan, the icebug-backed functions coexist with the existing algos under GDS_* names for one release cycle, then replace their internals in place. PAGE_RANK/PR are untouched.

How it works

projected graph → InMemGraph CSR (Louvain's fwd/bwd scan idiom)
  → arrow::UInt64Array → NetworKit::GraphR (zero-copy) → NetworKit::PageRank
  → scores streamed back through the existing GDSResultVertexCompute pipeline

The bind/register scaffolding is copied verbatim from page_rank.cpp (GDSFunction::initSharedState/getLogicalPlan/getPhysicalPlan); only the tableFunc body changes — it builds the CSR, hands it to icebug, and maps pr.scores() back to (node, rank) rows. Node offsets are dense in both worlds, so the mapping is identity.

Files

• algo/src/function/gds_page_rank.cpp — the bridge.
• algo/src/include/function/algo_function.h, algo/src/main/algo_extension.cpp — declare + register GDS_PAGE_RANK.
• algo/cmake/download_icebug.cmake — fetches the prebuilt icebug release (icebug-<os>-<arch> tag 12.8) into vendor/. Set -DICEBUG_SOURCE_DIR=<icebug checkout> to link a local source build instead (dev).
• algo/CMakeLists.txt, algo/src/function/CMakeLists.txt — link libnetworkit + Arrow::arrow_shared + OpenMP::OpenMP_CXX, with rpath to the vendored lib.
• algo/test/test_files/gds_page_rank.test.

Build requirements (needs your input)

The build now needs Apache Arrow C++ and OpenMP available (Arrow is a public dependency of libnetworkit). One thing to confirm for CI: the prebuilt icebug 12.8 links Arrow 24.x — the extension build needs a matching Arrow. Happy to adjust the vendoring (pin Arrow, or a newer icebug release built against the Arrow the CI uses) however you prefer. Locally I validated the full path with a source build of icebug against the system Arrow.

Testing

gds_page_rank.test (star graph): the hub ranks highest, and the ordering matches PAGE_RANK. Absolute values differ because NetworKit returns a sum-to-1 probability distribution (0.480 / 0.173…) vs the existing implementation's normalization — expected, and NetworKit's is the standard convention.

Notes / follow-ups

• Single-node-table graphs for now (multi-table is a small follow-up).
• GDS_LOUVAIN, GDS_NODE2VEC (ARRAY output → the vector extension's HNSW index), and personalized PageRank (icebug's built-in personalization/forSources) all follow this same pattern — one bridge, the whole NetworKit catalog.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• I have added tests.
• I have requested a review from a maintainer.
• Documentation — deferred; happy to add once the surface/build approach is settled.

[adsharma]

Pin arrow works on other platforms, but not macosx. Bugscope has the same issue.

We'll likely have to update ladybug, icebug and all platform in sync unless we can figure out how to pin on homebrew.

[zachwinter]

I understand your comment in Discord now!

For now, I -SKIP'd gds_page_rank.test for the OFF build. Would you prefer a proper conditional (maybe a SKIP_ICEBUG_DISABLED token mirroring SKIP_COMPRESSION_DISABLED) so it runs automatically once icebug is enabled? I'll open a PR in lbug core if so.

[adsharma]

@zachwinter - icebug 13.0 out. Can you see if it fixes the problem here?

[zachwinter]

Tested 13.0 — short answer: it fixes macOS, but not this CI. And chasing it down turned up a separate crash that I think matters beyond this PR.

The version arithmetic

	links against
icebug 12.8	libarrow.so.2400 / libarrow.2400.dylib
icebug 13.0	libarrow.so.2500 / libarrow.2500.dylib
ladybug CI (pixi.toml)	libarrow = ">=23.0.1,<24"
Homebrew apache-arrow today	25.0.0_3

So the CI failure here is unchanged, just one major further away — the log is libarrow.so.2400: cannot open shared object file, and since it's the whole algo extension that then fails to dlopen, all ~45 algo tests go red, not only the GDS one.

macOS is genuinely better off, though: 12.8 wanted Arrow 24 after Homebrew had already moved past it, which is exactly the "can't pin on homebrew" problem you hit. 13.0 and Homebrew are both on 25 right now, so they line up.

What I verified on 13.0 (macOS arm64)

Built the extension against the prebuilt tarball (not a source build — the path CI would take), -DENABLE_ICEBUG=ON, system Arrow 25.0.0:

• The bridge compiles unchanged. Despite fts: fix stopword maintenance and transactional statistics #42 removing the std::vector constructors, GraphR(count, bool, shared_ptr<UInt64Array>, shared_ptr<UInt64Array>) is byte-identical, and PageRank's new normalized / personalization args are defaulted.
• 71/71 algo tests pass, gds_page_rank included.

(Also noticed forSources / personalizationFromSources landed in 13.0 — that's the personalized-PR follow-up, I'll pick it up once this settles.)

The crash worth knowing about

While running the suite I hit a non-deterministic SIGSEGV — ~60% of runs, but only when GDS_PAGE_RANK runs after another algo test in the same process:

#1  _mi_thread_init + 391                     libarrow.2500.0.0.dylib
#3  __mi_malloc_generic + 363                 libarrow.2500.0.0.dylib
#5  arrow::MimallocAllocator::AllocateAligned libarrow.2500.0.0.dylib
...
#14 lbug::algo_extension::toU64              gds_page_rank.cpp:145

Arrow's default memory pool is mimalloc-backed, and mimalloc's per-thread bootstrap doesn't survive being reached from ladybug worker threads that already existed before libarrow came in via dlopen. Measured, same 3-test sequence:

• default pool: 5/8 runs crash
• ARROW_DEFAULT_MEMORY_POOL=system: 0/8
• passing arrow::system_memory_pool() to the builder in code: 0/10, full suite green

I observed this on macOS arm64; nothing about it looks macOS-specific, so I'd expect any dlopened extension that links Arrow to be exposed wherever the default pool is mimalloc. Worth flagging for Bugscope too if it links Arrow the same way.

The proposal: get Arrow off the ABI boundary entirely

Both problems — the version lockstep and the mimalloc crash — come from the extension having to link real libarrow just to hand icebug a CSR. Two ways out, in preference order:

1. Accept the Arrow C Data Interface at the GraphR entry point. A GraphR::fromCArrays(const ArrowArray*, const ArrowArray*, ...) taking the plain C structs. That interface is ABI-stable by design, the structs are header-only, and ladybug already ships them (src/common/arrow) — so the extension links zero Arrow, allocates through ladybug's own memory manager, and icebug's prebuilt can be built against whatever Arrow it likes. ladybug, icebug, Homebrew and conda-forge stop needing to move in sync at all.
2. Ship the prebuilt with Arrow statically linked + hidden visibility, so the tarball is self-contained. Fixes the version coupling, doesn't fix mimalloc, and puts a second Arrow in the process.

Happy to prototype (1) against icebug if you're open to it — it's a small surface, and it's the difference between "works today on 25" and "stops mattering".

Meanwhile, to get this PR green

I have the interim commit ready: an ENABLE_ICEBUG CMake option, default OFF, so a stock build links no icebug/Arrow/OpenMP and the extension loads everywhere — existing algos and their tests untouched — with GDS_* compiled and registered only when it's ON. I'll push that unless you'd rather wait and land the Arrow bump + icebug together.

On your Discord question about the test skip: I also have a SKIP_ICEBUG_DISABLED token written for ladybug core (mirrors SKIP_COMPRESSION_DISABLED), so gds_page_rank.test auto-skips when icebug is off and runs when it's on, instead of the blunt -SKIP I have now. Say the word and I'll open that core PR.

[adsharma]

LadybugDB/ladybug#774 has been merged. See if the issues are fixed?

[chiangchenghsin-hash]

Thanks both for the thorough discussion on the Arrow version lockstep. I took this branch for a spin on Windows (MSVC 14.51) to see how the icebug bridge holds up.

Windows build (ENABLE_ICEBUG=ON): compiles and links cleanly

• icebug 13.0 prebuilt (icebug-win-amd64.zip) works out of the box — networkit.lib + headers + runtime DLLs all present and correct. The download_icebug.cmake win-amd64 branch resolves correctly.
• Arrow + OpenMP: resolved via conda-forge. Arrow 13.0.0 is what conda-forge currently ships for Windows — SO mismatch (icebug 13.0 links Arrow SO 2500), so MSVC links fine but crashes at runtime inside NetworKit::GraphR with ACCESS_VIOLATION. Same version-lockstep problem already identified above, confirmed on Windows.
• Hand-rolled PAGE_RANK works correctly against the same projected graph (star graph, hub ranks highest at 0.133), so the graph setup path is solid.

Two small fixes for the ENABLE_ICEBUG gating

1. algo_extension.cpp: the addTableFunc<GDSPageRankFunction>(db) registration needs #if defined(ENABLE_ICEBUG), otherwise an OFF build fails at the final link step with unresolved getFunctionSet.
2. function/CMakeLists.txt: gds_page_rank.cpp should be added via target_sources(... PRIVATE ...) inside the if (ENABLE_ICEBUG) guard — otherwise OFF builds try to compile it without the Arrow/NetworKit include dirs.

Both are tiny — happy to open a PR against your branch with them.

The C Data Interface direction (GraphR::fromCArrays / Arrow C structs at the entry point) sounds like the right call. Once icebug takes Arrow off the ABI boundary, the Windows path goes from "works if you pin the exact Arrow SO" to "works on whatever Arrow is on the machine" — which is the difference between a dev preview and something downstream can depend on.

[adsharma]

@chiangchenghsin-hash thank you for testing. I'm trying to get CI to pass and land this PR. Please open a new PR once this lands.

[adsharma]

Next blocker:

Error: /lib64/libm.so.6: version `GLIBC_2.38' not found

This can only be solved by making an icebug release built in a manylinux 2_28 container - the same one that ladybug uses.

[adsharma]

Rerunning CI with the just released icebug 13.1 binaries which should be compatible with glibc 2.28+

[chiangchenghsin-hash]

One more thought on the multi-table follow-up mentioned in the roadmap — I ran into exactly this while implementing Leiden for the algo extension, and ended up with a working flat-index mapping across node tables (nodeOffsetBase[tableID] + offset). It's not complex, just a loop over maxOffsetMap in the CSR build step.

If it helps, I'd be happy to extract that into a shared utility once the base bridge lands — saves you the follow-up effort and gives GDS_PAGE_RANK multi-table support out of the gate for the next release cycle. The Leiden heterogeneous regression tests (400 nodes across two tables) are already passing, so there's test coverage to borrow.

No rush of course — just wanted to flag it while the CSR path is fresh in my mind.

[adsharma]

I was thinking efficient merging of multiple tables into one could be a cypher query (so users have some flexibility in how they order the nodes in the projected graph).

The base bridge has landed (but not included in the 0.19.1 that just went out).