0.17.0 release: TODO

[aheev]

ladybug

• icebug-memory
• arrow tables extensive testing
  • immutability
  • cross table support
  • read queries similar to current ice_disk test suite
  • load testing
• fix any icebug-disk issues
• storage version backward compatibility wrt small changes
• test disable_hash_index feature, both read and write ops
  • benchmark runs
• update migrate db tool to incorporate new breaking changes

icebug

• demo of full working integration pgembed - ladybug - icebug
• impl ladybug's icebg-disk integration with icebug
• update icebug-format tool and spec to incorporate new changes in ladybug's impl
• add support for parquet tables(non-duckdb) in icebug-format

docs/blogs

• blog post about 0.17.0
• update docs

[aheev]

@adsharma feel free to add anything I missed

[adsharma]

There is no migrate DB tool. When you open the DB with lbug cli tool and checkpoint the DB the first time it writes the new version.

Subsequently the DB can't be opened with older cli.

[aheev]

I was referring to migrate-lbug-db.py. Looks like it just does import and export

[adsharma]

Right - someone asked for it (because the docs said do this) and it was easy enough to do.

But updating in place is what other serious db projects are doing. We should update our docs if this method works well in the field.

[adsharma]

Re: no hash index

There is an in memory data structure that's used to guarantee pk uniqueness. But it doesn't scale to O(disk).

Question is: do we want to implement ART (DuckDB's default index type) or similar in this release cycle?

[aheev]

How long do you think it would take to implement a new index and testing?

[adsharma]

Implementation is done. Let's see how it works in practice and then consider making it the default in one or two release cycles.

Both Hash and ART index do ::load() which means they don't support indexes that are larger than memory.

There is also zone maps, but they can't be used for guaranteeing uniqueness. With 0.17.x we should be at parity with DuckDB in terms of the number/types of indexes, even if we have some limitations.

[adsharma]

pgembed is being pulled into tests. #498

This will improve postgres test coverage and test SQL pushdowns.

[aheev]

For icebug-disk integration in icebug, how about we create a new API fromCSRDisk which converts parquet files to arrow tables internally and call fromCSR with the converted arrow tables?

The only problem I see is once icebug process dies, it has to convert the tables again on re-open

[adsharma]

Not convinced that icebug needs an API to do this.

Read native relationship CSR from disk, then run Icebug section in

This notebook has an example for reading from native table. People can search for "how to read parquet to arrow" and paste the code for simple things like this.

[aheev]

Grafeo benchmark:

It is an in-memory benchmark. Found out that on traversal queries, native is pretty fast compared to arrow-csr. Updated benchmark repo: https://github.com/aheev/grafeo-graph-bench/tree/add-arrow-csr branch: add-arrow-csr

Results attached
bench_20260522_small_arrow.csv

PS: Couldn't go beyond small because others were taking a lot of time. medium didn't complete its run even after 6 hrs

[adsharma]

small is what I was able to run in the past as well. Do you have data for memory usage that grafeo reports? Did it improve relative to 0.16.x?

[adsharma]

Compared 0.16.1 vs 0.17.x-nightly (latest)

bench_20260522_220510.json
bench_20260522_221640.json

Summarized as a table:
https://gist.github.com/adsharma/2d5d93f098b7a27822a97ef88479ceaf

• Memory usage is down (which was an issue flagged in previous benchmark runs)
• Runtime is generally lower (which is probably a side effect of smaller RSS)
• Vector/node insert benchmarks have regressed slightly. Something to look into.

[adsharma]

All of the analytics benchmarks could be rewritten using icebug instead of using ALGO extension. The new fast path conn.query_as_arrow().csr() -> ib.fromCSR(...) should perform significantly better.

[aheev]

on deep traversal queries, latency has certainly improved and beat grafeo
bench_20260523_icebug.csv

[aheev]

@adsharma after the recent commits live-journal benchmark hangs on q6-8. You can try 1 run of a query. Default is 6 runs (1 - warmup, 5 - actual)

prev results: #473 (comment)

[adsharma]

Direct link to benchmark: https://github.com/aheev/live-journal-benchmark

Will look into this later today.

[adsharma]

Why q06-q08 are slow on icebug:

• queries.py:42: Top-10 nodes by out-degree
  Plan is SCAN_NODE_TABLE(user) -> SCAN_REL_TABLE(follows) -> AGGREGATE GROUP BY u.id -> TOP_K. It scans the whole follows relation, groups by every source
  id, then keeps top 10. The LIMIT 10 cannot help early because deg is only known after the full aggregation.
• queries.py:50: Count nodes with at least one out-edge
  Plan is also a full SCAN_REL_TABLE(follows), followed by COUNT(DISTINCT u.id). It does not use the CSR row-pointer metadata / active-row shortcut; it
  materializes source ids from the edge scan and deduplicates them.
• queries.py:58: count(f) with a bound relationship variable
  Plan scans follows and projects f._ID, then aggregates COUNT(f._ID). Because the rel variable is bound, the planner cannot use the relation-count metadata
  fast path.

Control check: q09, MATCH ... RETURN count(*), explains to COUNT_REL_TABLE(follows) directly, which is why it is fast while q08 is slow despite looking nearly equivalent.

[adsharma]

Optimization suggestions:

• Use metadata at two levels: relation cardinality and CSR row-degree summaries.

For these queries specifically:

1. q08: count(f) should use relation cardinality metadata

count(f) over:

MATCH (a:user)-[f:follows]->(b:user)
RETURN count(f)

is semantically just edge count, as long as f is only counted and no rel properties/predicates are used.

Planner rewrite:

SCAN_REL_TABLE + PROJECT f._ID + COUNT(f._ID)

to:

COUNT_REL_TABLE(follows)

Same fast path as q09 count(*).

Metadata needed:

• follows.num_rels, already derivable from indices_follows.parquet row count or final indptr.

Caveat:

• Only valid when the relationship variable is not otherwise used in WHERE, projection, property access, path semantics, etc.

2. q07: count(DISTINCT u.id) should use active-source metadata

For CSR-backed outgoing edges, nodes with at least one out-edge are exactly rows where:

indptr[i + 1] > indptr[i]

So this query:

MATCH (u:user)-[:follows]->(v)
RETURN count(DISTINCT u.id)

can avoid scanning all 69M edges. It only needs either:

• precomputed follows.num_non_empty_src_rows, or
• a scan over indptr only, about 4M entries instead of 69M edges.

Best metadata:

num_src_nodes
num_rels
num_non_empty_src_rows

Planner rewrite:

COUNT_ACTIVE_SRC(follows)

If all nodes have at least one outgoing edge in this dataset, result is just num_non_empty_src_rows = 3997962.

3. q06: top-k out-degree should use per-source degree metadata

This query:

MATCH (u:user)-[:follows]->(v)
RETURN u.id, count(v) AS deg
ORDER BY deg DESC
LIMIT 10

does not need edge expansion. In CSR, out-degree is:

deg[i] = indptr[i + 1] - indptr[i]

So the engine should read degrees per source row, keep a top-k heap, and join/project u.id.

Possible operators:

SCAN_REL_DEGREES(follows, direction=out)
TOP_K(deg)
PROJECT user.id, deg

Even better, persist sorted degree metadata:

top_out_degree_entries: [(src_offset/id, degree), ...]

Then q06 is basically metadata lookup plus id projection.

Pragmatic implementation order:

1. Add/reuse relation cardinality metadata and rewrite count(f) to COUNT_REL_TABLE.
2. Add num_non_empty_src_rows from indptr and rewrite count(DISTINCT src) for full unfiltered outgoing rel scans.
3. Add a degree-scan operator over indptr for grouped count(v) by source.
4. Later, persist top-k degree summaries if this query matters a lot and indptr scanning is still too slow.

The key planner rule is: recognize full unfiltered CSR patterns where the aggregate can be answered from indptr/relation metadata, without touching indices_follows.parquet.

[aheev]

FalkorDB benchmark

FalkorDB: 41m 55s

Ladybug: still running. est ~7 hrs( each worker ran 16000 queries in 25 mins. Each worker should run 250000 queries)

Link: https://github.com/adsharma/falkordb-benchmark/tree/ladybug (not functional. Needs fixes)

[aheev]

Pulling the plug now. Current run stands at 48000 queries

[adsharma]

We discussed some of the queries on #ladybug 4 months ago. I don't recall running the entire suit i.e. this may not be a regression.

[aheev]

@adsharma I was looking into adding a new icebug constructor for IcebugMem object. Seems like icebug expects tables with only 1 uint64 col , but IcebugMem has full indices table. We can add the new constructor and internally call old fromCSR(icemem.indices["target"]). But in the future we might need to add new parameters for weights which defeats the purpose of passing only IcebugMem object

[adsharma]

The reason why things are setup that way is that Icebug is a graph analytics package. The algorithms it runs are purely looking at the graph structure and have no need for the labels or other columns as things stand now.

Perhaps add a helper in the python layer that takes indices and indptr for IcebugMem and passes it to the constructor. More of a convenience func instead of a fundamental change.

[adsharma]

graph.pyx already supports weights. We need to update docs/examples.

[aheev]

Perhaps add a helper in the python layer that takes indices and indptr for IcebugMem and passes it to the constructor. More of a convenience func instead of a fundamental change.

that's what I am doing right now. We might want to add weight arr as a param too

	@classmethod
	def fromIcebugMemGraph(cls, graph, bool_t directed=True):
		"""
		fromIcebugMemGraph(graph, directed=True)

		Create a Graph from an :class:`icebug_format.IcebugMemGraph`.

		Extracts the ``indices`` and ``indptr`` Arrow tables from *graph* and
		delegates to :meth:`fromCSR` for zero-copy construction.

		Parameters
		----------
		graph : icebug_format.IcebugMemGraph
			CSR graph produced by ``icebug_format.convert_arrow_tables_to_csr``.
		directed : bool, optional
			If True, the resulting Graph will be directed. Default: True

		Returns
		-------
		Graph
			A new Graph instance backed by the Arrow arrays from *graph*.
		"""
		import pyarrow as pa
		n = len(graph.src)
		out_indices = graph.indices.column('target').cast(pa.uint64())
		out_indptr = graph.indptr.column('ptr').cast(pa.uint64())
		return cls.fromCSR(n, directed, out_indices, out_indptr)

[adsharma]

  g = ib.Graph.fromCSR(
      n,
      directed=False,
      out_indices=indices,
      out_indptr=indptr,
      out_weights=weights,
  )

  For directed graphs, pass in_indices, in_indptr, and optionally in_weights too. One caveat: isWeighted() is keyed
  off out_weights, so passing only in_weights will not make the graph weighted.

This is already covered in graph.pyx:83

[aheev]

@adsharma icebug_format outputs indices table with "target" column. Can you please add support for target, destination, dest, to in ladybug

[aheev]

icebug-format changes

• PR: fix undirected graphs handling Ladybug-Memory/icebug-format#6
• icebug-format always produces directed graph
• By default undirected = false, If it is passed as True, reverse edges are added to make the graph directed(ladybug only supports directed graphs). Note that undirected graphs are only supported on rel tables with same from and to node tables because neighbour_id col in indices represents node_offset of from_table. So, we can't add reverse edges

icebug

• PR: add a helper method for loading IcebugMem object Ladybug-Memory/icebug#32
• fromIcebugMemGraph constructor is added which takes only IcebugMemGraph as arg. If they want to pass weights or in_indices they can always use already existing fromCSR
• Icebug-formatted graphs are always directed, so, some algos like community detection wouldn't work on them
• To pass undirected arrow csr graphs they have to use fromCSR using custom made indices and indptr tables. Users can't use icebug-format converters
• GraphW has a toUndirected method to convert a directed graph to undirected, this wouldn't work for arrow tables because
  • although ArrowArray supports slice option, we can't use it for our tables because reverse edges are not appended to indices. They are sorted and clustered for an optimized indptr
  • We could create a new ArrowArray with only forward edges which goes against our claim of zero-copy

ladybug

• We need to add support col names like target, destination, dest, to in indices table because icebug-format emits target col for neighbour ids
• no changes reqd w.r.t direction

Verdict

The only way we can support undirected graphs in icebug-format is by making it ouput two indices and two indptrs`

Release protocol

• Test the notebooks locally with these changes
• icebug-format should be merged and released first
• update the icebug-format version in icebug and release icebug before publishing notebooks
• update icebug_format, icebug, and ladybug version in notebooks and publish

[aheev]

Notebooks are not rendering for me. "invalid JSON" when I click on "view file". I'll wait for them to settle.

dem. I don't know what's wrong with github, it failed for me initially, then I downloaded them manually. Lemme try to fix them

[aheev]

I'm not following why the changes to ladybug are needed. It seems to be a column naming, metadata concern?

The problem is c_api / python_api doesn't support custom dstColumnNames. It is default'ed to "to". where as C++ api does accept dstColumnName

def create_arrow_rel_table(
        self,
        table_name: str,
        dataframe: Any,
        src_table_name: str,
        dst_table_name: str,
        layout: ArrowRelTableLayout | str = ArrowRelTableLayout.FLAT,
        indptr_dataframe: Any | None = None,
    )  

ArrowTableCreationResult ArrowTableSupport::createRelTableFromArrowCSR(main::Connection& connection,
    const std::string& tableName, const std::string& srcTableName, const std::string& dstTableName,
    ArrowSchemaWrapper indicesSchema, std::vector<ArrowArrayWrapper> indicesArrays,
    ArrowSchemaWrapper indptrSchema, std::vector<ArrowArrayWrapper> indptrArrays,
    const std::string& dstColumnName) {

static ArrowTableCreationResult createRelTableFromArrowCSR(main::Connection& connection,
        const std::string& tableName, const std::string& srcTableName,
        const std::string& dstTableName, ArrowSchemaWrapper indicesSchema,
        std::vector<ArrowArrayWrapper> indicesArrays, ArrowSchemaWrapper indptrSchema,
        std::vector<ArrowArrayWrapper> indptrArrays, const std::string& dstColumnName = "to");

icebug_format sets dstColName to target

SELECT csr_target::UBIGINT AS target

There seems to be a directionality concern too. At the high level I prefer minimizing disk I/O by reading in only one direction and then duplicating in-memory as needed via a helper method rather than forcing disk I/O in both directions.

changes are not needed on direction front

[adsharma]

Are you making a PR? I didn't see one.

[aheev]

I thought you were working on it 😅

[aheev]

#534
Notebooks are running fine. CI failures in lang_bindings are related to old c_api

[aheev]

@adsharma how do I run buscope? The cmds don't seem to work anymore

[aheev]

I will run ladybug-bugcope on https://github.com/mims-harvard/optimuskg (used by https://www.linkedin.com/posts/wenliang-zhang_drugdiscovery-biomedicalai-knowledgegraph-activity-7464889306998964224-nc5b) in sometime

[aheev]

can you please update the README? None of the steps work for me

[aheev]

cmds in package.json don't work too

[adsharma]

README was updated today. It now says:

cargo tauri dev --features=icebug-analytics -- -- ../test.lbdb

[aheev]

These steps atleast are not included:

• git clone --branch icebug-arrow-graph https://github.com/adsharma/sigma.js .deps/sigma.js
• tauri-cli install
• npm run setup

Even after these, running into issues. Will check back tmrw

Maybe try running package.json cmds on a clean env?

[aheev]

@adsharma found a bug in icebug-disk.

  - MATCH (n:gene) RETURN n.id returns exactly 2,048 rows (hard cap)
   - MATCH (n:gene) RETURN n.id SKIP 2048 LIMIT 10 returns 0 rows — data beyond index 2048 is inaccessible
   - count(n) aggregation is also capped at 2048 (not just display)

  The icebug-disk node loader reads at most 2,048 nodes per type from the parquet files. The correct node count (190,531) can only be obtained by reading the parquet files directly.

[adsharma]

I ran uvx icebug-format --source-db test.duckdb and it produced WITH (storage = '.', format = 'icebug-disk');

[aheev]

yeah it does. But on providing schema.cypher, it should prefer shouldn't it prefer schema.cypher path?

[aheev]

wait, I think it didn't check for files in path(schema.cypher) at all

fact that it failed on storage path
Error: IO exception: Cannot open file ./optimuskg_csr/nodes_anatomy.parquet: No such file or directory

[adsharma]

Try replacing './optimuskg_csr` with '.' - if it passes, let's figure out the CLI options that added the path and try to remove them.

[aheev]

yupp, works

./icebug_disk/lbug -i ./icebug_disk/optimuskg_csr/schema.cypher
-- Processing: ./icebug_disk/optimuskg_csr/schema.cypher
Opening the database under in-memory mode.
Enter ":help" for usage hints.
lbug> CALL show_tables() RETURN *
      ;
┌────────┬───────────────────────────────────────┬────────┬───────────────┬─────────┐
│ id     │ name                                  │ type   │ database name │ comment │
│ UINT64 │ STRING                                │ STRING │ STRING        │ STRING  │
├────────┼───────────────────────────────────────┼────────┼───────────────┼─────────┤
│ 47     │ exposure_gene                         │ REL    │ main(graph)   │         │
│ 41     │ exposure_cellular_component           │ REL    │ main(graph)   │         │
│ 37     │ drug_phenotype                        │ REL    │ main(graph)   │         │
│ 1      │ biological_process                    │ NODE   │ main(graph)   │         │
│ 33     │ drug_drug                             │ REL    │ main(graph)   │         │
│ 45     │ exposure_exposure                     │ REL    │ main(graph)   │         │
│ 29     │ drug_biological_process               │ REL    │ main(graph)   │         │
│ 3      │ disease                               │ NODE   │ main(graph)   │         │
│ 19     │ cellular_component_cellular_component │ REL    │ main(graph)   │         │
│ 6      │ gene                                  │ NODE   │ main(graph)   │         │
│   ·    │                  ·                    │   ·    │      ·        │   ·     │
│   ·    │                  ·                    │   ·    │      ·        │   ·     │
│   ·    │                  ·                    │   ·    │      ·        │   ·     │
│ 39     │ exposure_biological_process           │ REL    │ main(graph)   │         │
│ 43     │ exposure_disease                      │ REL    │ main(graph)   │         │
│ 13     │ anatomy_gene                          │ REL    │ main(graph)   │         │
│ 17     │ biological_process_gene               │ REL    │ main(graph)   │         │
│ 61     │ phenotype_gene                        │ REL    │ main(graph)   │         │
│ 0      │ anatomy                               │ NODE   │ main(graph)   │         │
│ 21     │ cellular_component_gene               │ REL    │ main(graph)   │         │
│ 5      │ exposure                              │ NODE   │ main(graph)   │         │
│ 11     │ anatomy_anatomy                       │ REL    │ main(graph)   │         │
│ 25     │ disease_gene                          │ REL    │ main(graph)   │         │
└────────┴───────────────────────────────────────┴────────┴───────────────┴─────────┘

[aheev]

@adsharma another bug. https://github.com/LadybugDB/ladybug-icebug-notebooks/blob/main/ladybug_icebug_starwars.ipynb is failing at step Read native relationship CSR from disk, then run Icebug after this commit

error: RuntimeError: CSR export is only supported for rowid projections

[aheev]

@adsharma should we update api-server, explorer, bugscope too?

[adsharma]

We should update everything! bugscope needs a new icebug release too.

[aheev]

api-server is blocked on explorer release

[adsharma]

Made a explorer 0.17.0 release