Add PyLucene integration and CPU/GPU end-to-end tests by nvzm123 · Pull Request #174 · NVIDIA/cuvs-lucene

nvzm123 · 2026-07-08T22:50:37Z

Summary

This PR adds PyLucene integration support and a pytest-owned CPU/GPU end-to-end suite for cuVS-Lucene.

The standard cuvs-lucene jar contains only cuVS-Lucene classes and service descriptors. PyLucene supplies Lucene classes, while the base cuvs-java jar remains a separate classpath dependency. The custom assembly descriptor preserves merged service descriptors in the optional dependency jar.

The PR also adds:

SPI-loadable one-layer and three-layer accelerated HNSW codecs
explicit CPU-fallback control for CPU-path testing
writer diagnostics used to verify CPU HNSW, CAGRA-built HNSW, and CAGRA-search execution
version-specific Lucene provider caching and Lucene 102 binary-vector-format handling
compatibility tests for codec discovery and binary-vector formats
current cuVS 26.10 dataset-view handling for serialized CAGRA indexes
filter-bitset caching and per-segment GPU filter-search support from the current main branch

PyLucene test coverage

The suite explicitly covers these execution paths:

CPU HNSW build and HNSW search
GPU CAGRA build followed by HNSW search
GPU CAGRA build and CAGRA search

Coverage includes:

one-layer and three-layer HNSW graphs for CAGRA-built/HNSW-search indexes
a warning-free single-live-document CAGRA-search case
one segment and ten segments
10-to-1 and 100-to-10 force merges
CAGRA searchWidth values 1, 16, and 32
deleted-document exclusion during CAGRA search
selective document filtering through all three execution paths
a selective ten-segment CAGRA filter case exercising per-segment native prefiltering and global top-K merging

Vectors and queries are deterministic. Expected neighbors are computed by brute force. Assertions verify:

rank-one self matches where applicable
exact expected hit counts
no duplicate hits
deleted or filter-rejected documents are not returned
configurable minimum recall
the intended writer, reader, and query implementations were used

The selective-filter cases accept roughly one quarter of each segment and retain more than topK accepted vectors per segment. This prevents Lucene from substituting exact scoring and ensures the approximate/native prefilter path is exercised. Filtered recall is calculated only against accepted vectors.

GPU-required cases fail if cuVS is unavailable or silently falls back to CPU. CPU cases force and report Lucene's CPU HNSW path. HNSW cases independently verify persisted graph structure, including Lucene 10.2's field-number-keyed HNSW metadata.

CAGRA configurations use graphDegree=32 and intermediateGraphDegree=64. Each constructed CAGRA graph contains at least 97 vectors, and the three-layer case starts with 24,832 vectors so its third layer also retains at least 97. The single-document case builds a sufficiently large graph and then deletes all but one document before search. This avoids cuVS graph-parameter clamping warnings while exercising single-live-document behavior.

Test layout

src/test/python/test_pylucene_end_to_end.py owns pytest cases, parametrization, assertions, and reporting.
src/test/python/pylucene_test_support.py contains reusable PyLucene index and search helpers.
src/test/python/conftest.py handles case/group selection.
src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.java provides test-only Java adapters.
ci/run_pylucene_pytests.sh handles Maven artifacts, classpath setup, JVM/native-library inputs, and pytest invocation.
test_pylucene.sh is the repository-root convenience entry point.

The default check runs jar-packaging assertions and cpu-hnsw-1-segment:

./test_pylucene.sh

Run the complete CPU/GPU suite with:

./test_pylucene.sh --full-e2e

Run focused groups with:

./test_pylucene.sh --cases=gpu-cagra-search
./test_pylucene.sh --cases=single-document
./test_pylucene.sh --cases=document-filter

Pytest can also run directly once the documented PyLucene classpath environment variables are available:

python3 -m pytest -q -s src/test/python/test_pylucene_end_to_end.py

Validation

Validated on an NVIDIA A10G against a fresh cuVS 26.10 Java/native build and a matching Lucene/PyLucene 10.2 environment:

shell syntax and Python compilation passed
full pytest collection: 23 tests collected
single-document group: 4 passed, 19 deselected
CAGRA-search group: 12 passed, 11 deselected; displayed recall was 1.000 for every CAGRA-search scenario
complete CPU/GPU PyLucene suite: 23 passed in 33.90 seconds
clean Maven verification: 299 tests, 0 failures, 0 errors, 30 skipped
git diff --check passed

The PyLucene runs emitted no cuVS graph-clamping or CPU-fallback warnings. The only warning was the JVM notice for the incubating vector module.

Follow-up multithreaded concurrency coverage is tracked in NVIDIA/cuvs#2407.

copy-pr-bot · 2026-07-08T22:50:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

and finish updating cuvs version

Convert the PyLucene smoke runner to a pytest-backed case matrix covering GPU CAGRA search, CAGRA-built HNSW, and forced CPU HNSW fallback across segment and force-merge topologies. Add named one-layer and three-layer CAGRA-to-HNSW codecs plus writer-path telemetry so the e2e suite can assert which CPU/GPU path was exercised. Tighten sidecar and Lucene delegate-codec validation to avoid misleading expected probe warnings while still failing on unsupported delegate codecs.

imotov · 2026-07-17T00:14:52Z

  @Override
  public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws IOException {
    var flatWriter = FLAT_VECTORS_FORMAT.fieldsWriter(state);
+    System.setProperty(


These System.setProperty methods make sense in the context of this PR as a way to communicate between the Java and Python layers. However, they introduce unnecessary overhead for most other applications. Moreover, this approach would not work reliably in a multithreaded environment, since the same property is reused across requests. I think it would be better to separate the telemetry concerns and extract them into a dedicated method that is invoked only when telemetry data is actually needed.

Good point! Removed the JVM-global system properties from the indexing path. Telemetry is now computed on demand from the vectors format only when the PyLucene test requests it.

cjnolet · 2026-07-17T14:15:33Z

+To run the PyLucene pytest smoke suite against a local PyLucene environment:
+
+```sh
+./ci/test_pylucene_smoke.sh


I like that we have ci scripts for this , but a user should not have to invoke ci scripts in order to run tests. Rather, we should document how to run these pytests without the need to call scripts inside the CI directory while also providing the scripts in the CI directory for GitHub actions to run automatically.

@cjnolet I moved the user-facing logic to test_pylucene.sh and kept a thin wrapper under ci/. The current GitHub Actions workflows do not invoke it yet. Did you intend for this PR to add a PyLucene Actions job as well, or is providing the CI entry-point sufficient for now?

cjnolet · 2026-07-17T14:16:37Z

+./ci/test_pylucene_smoke.sh --gpu-e2e
+```
+
+The expanded suite runs the `gpu-basic`, `gpu-segments`, `cpu-hnsw`, and


Great description here. How to build and run tests should really be in a separate build and install guide. I think this is okay for now, especially since we are moving cuVS-Lucene to cuVS, but it's something to consider.

makes sense

cjnolet · 2026-07-17T14:24:46Z

+            .withCagraGraphBuildAlgo(CagraGraphBuildAlgo.NN_DESCENT)
+            .withGraphDegree(CAGRA_GRAPH_DEGREE)
+            .withIntermediateGraphDegree(CAGRA_INTERMEDIATE_GRAPH_DEGREE)
+            .withHNSWLayer(1)


Wasn't this an issue that you resolved? Why layer 1?

Yes, that issue was resolved. Layer 1 is intentional test coverage, paired with the three-layer case -- that said, I am not sure that these two particular tests will provide much utility

Move the PyLucene suite into a pytest-owned test tree and add explicit coverage for CPU HNSW, CAGRA-built HNSW, and CAGRA search. Cover segment and force-merge topologies, one- and three-layer HNSW, CAGRA search widths, deletions, vectorless documents, filtering, and single-document behavior. Verify execution paths, persisted graph configuration, and brute-force recall with deterministic vectors. Keep the shell runner focused on environment and classpath setup, and document direct pytest and full GPU execution.

Remove redundant edge cases and reuse canonical topology cases for one-layer HNSW and searchWidth=1 coverage. Add selective brute-force-validated filters across CPU HNSW, CAGRA-built HNSW, and ten-segment CAGRA search, then make cpu-hnsw-1-segment the default documented smoke case.

imotov

I'm not sure how comprehensive we want these cases to be, but they all follow the same pattern - index records under some configuration, close the writer, then search. These are solid, deterministic tests, and I don't want to hold up the PR over this, but there are several important use cases where I think the next round of coverage could pay off.

Everything here is single-threaded: SerialMergeScheduler runs merges inline, IndexSearcher is built without an executor, and numMergeWorkers defaults to 1. So CuVSResources, which is managed per-thread, is only ever exercised in its simplest configuration - one thread, one resource.

In production, indexing and searching frequently overlap: new documents arriving, existing documents being updated, deletes landing, merges running (sometimes on several threads), and queries served against the index the whole time. In my experience that's where the tricky bugs live. A couple of other specific gaps:

No updateDocument/softUpdateDocument at all, so the delete-then-add path is untested even on a single thread.
No near-real-time search — the reader always opens after the writer closes, so a segment is never read while it's still being written, and deletes are always committed before anything searches. That should be covered by Lucene, but it would be nice to ensure that Readers still play by the book, don't leak any resources and do what Lucene expects them to do.
No search concurrent with a merge, so a segment is never dropped from under a live searcher. Same here, Lucene should take care of it, but we should cooperate.

Create flat vector writers only when the accelerated writer consumes them. CPU fallback paths construct their own writers, so eager allocation leaked the unused flat writer for both binary and scalar quantization.

Keep the existing Lucene 102 binary-format initialization while preserving the reminder to replace version-specific partial providers in a separate initiative.

cjnolet

This PR Still needs a lot of work and I'm honestly not sure we should hold up the consolidation of cuVS-lucene to cuVS over it.

@nvzm123 please do make sure when you use AI to make changes that you familiarize yourself enough with our conventions that you can guide the AI to follow them. As it is, this PR is very different than how we do things and having to review changes like this becomes time consuming for everyone else. The pytest infra and related docs do not follow convention. Please make sure to follow all conventions in cuVS and other RAPIDS/cuda-x repos (like RAFT, cuml, etc...).

cjnolet · 2026-08-04T11:45:46Z

+
+cuvs_java_jar = Path(os.environ["CUVS_LUCENE_CUVS_JAVA_JAR"])
+cuvs_lucene_jar = next(
+    jar


These steps still look overly complex for a user that just wants to use pylucene.

Boilerplate like this should ideally be provided in a Python library or scripts somewhere that can be easily included in their code. Ideally they should just have to import the codec and call codec.forCodec call.

Have you looked at other Pylucene extensions on GitHub to see their "usage" docs?

cjnolet · 2026-08-04T11:48:01Z

+
+#### PyLucene end-to-end tests
+
+Pytest cases are under `src/test/python`. The parametrized cases and assertions


If we need to expend this many words on tests then it's too complicated.

The only thing you need to say about the tests, other than a very terse description, is a 1 or 2-liner copy/paste example of how to run them. Please see the cuVS build docs for and example.

Also, the standard for test file naming should "XXX_test.py".

cjnolet · 2026-08-04T11:48:41Z

+`src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.java` are compiled to
+`target/test-classes` and are not included in the published jar.
+
+`ci/run_pylucene_pytests.sh` builds the Maven artifacts when requested, resolves


Please don't reference CI scripts in user docs. These are specifically for CI, not for user consumption.

cjnolet · 2026-08-04T11:49:06Z

+Pytest cases are under `src/test/python`. The parametrized cases and assertions
+are in `test_pylucene_end_to_end.py`; reusable index and search code is in
+`pylucene_test_support.py`. The test-only Java adapters in
+`src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.java` are compiled to


Too much info.. users don't need this.

cjnolet · 2026-08-04T11:51:31Z

+`target/test-classes` and are not included in the published jar.
+
+`ci/run_pylucene_pytests.sh` builds the Maven artifacts when requested, resolves
+the PyLucene classpath inputs, and invokes pytest. `test_pylucene.sh` at the


This is not standard practice in cuVS or other cuda-x repos. Please see cuVS build docs- we use a standard consolidated "build.sh" in the root of the repository.

We are in the process of consolidating cuVs-lucene with cuvs, and the build scripts need to be consolidated as well. We do not provide separate shell scripts for simple 1-liner test run commands. As you can imagine that adds to maintenance overhead for very little benefit. The pytest command should be reference in the build docs for running the tests (as a verification the build succeeded).

cjnolet · 2026-08-04T11:55:16Z

+Select a focused group or set the minimum document count per scenario:
+
+```sh
+./test_pylucene.sh --cases=cagra-search-widths \


Nope. Too complex. User should not require a complex script with flags just to run pytests.

cjnolet · 2026-08-04T11:56:07Z

@@ -0,0 +1,232 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.


Mentioned above. Remove. Please use pytests from cuVS Python package as a guide- the individual test cases should be parameterized.

cjnolet · 2026-08-04T11:56:53Z

@@ -0,0 +1,22 @@
+<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">


Wait- why? Aren't we already building this? cc @imotov

cjnolet · 2026-08-04T11:57:32Z

    this(
        NAME,
-        LuceneProvider.getCodec("101"),
+        LuceneProvider.getDefaultDelegateCodec(),


Not sure we want to change this, but will let Igor weigh in. We need to make sure we dont break anything.

cjnolet · 2026-08-04T11:57:56Z

   */
  public CuVS2510GPUSearchCodec(GPUSearchParams params) throws Exception {
-    this(NAME, LuceneProvider.getCodec("101"), params, FilterBitsetCacheConfig.DEFAULT);
+    this(NAME, LuceneProvider.getDefaultDelegateCodec(), params, FilterBitsetCacheConfig.DEFAULT);


Please make sure this doesn't break anythingz

cjnolet and others added 3 commits May 13, 2026 16:23

Add PyLucene integration support

1c0f648

Fix benchmark version marker

e27a53b

Fix PyLucene sidecar packaging and update cuVS version

3cab384

nvzm123 requested review from a team as code owners July 8, 2026 22:50

nvzm123 requested a review from msarahan July 8, 2026 22:50

Merge remote-tracking branch 'upstream/main' into pr-147

6e3cf20

and finish updating cuvs version

nvzm123 force-pushed the pr-147 branch from 0b028cf to 6e3cf20 Compare July 8, 2026 23:50

nvzm123 marked this pull request as draft July 9, 2026 19:58

cjnolet mentioned this pull request Jul 15, 2026

Add PyLucene integration support #147

Closed

nvzm123 changed the title ~~Fix PyLucene sidecar packaging and add smoke test~~ Fix PyLucene sidecar packaging and add smoke tests Jul 16, 2026

Merge branch 'main' into pr-147

5ade7af

nvzm123 marked this pull request as ready for review July 16, 2026 20:19

imotov reviewed Jul 17, 2026

View reviewed changes

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jul 17, 2026

cjnolet assigned nvzm123 Jul 17, 2026

cjnolet added this to Unstructured Data Processing Jul 17, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Jul 17, 2026

cjnolet requested changes Jul 17, 2026

View reviewed changes

nvzm123 added 6 commits July 21, 2026 03:44

Make writer telemetry on-demand

692c3db

Avoid duplicate binary format initialization

0e6bb43

Cache binary quantized vector formats

eeff876

Expose writer diagnostics through format descriptions

b76e2f5

Use Lucene 102 binary vector formats

bf23e24

Use the standard jar for PyLucene

f7a8c66

nvzm123 added 3 commits July 21, 2026 03:44

Add a public PyLucene test entrypoint

7084157

Move PyLucene tests under examples Python

d13184d

Clarify binary format version handling

df836cc

nvzm123 requested review from cjnolet and imotov July 22, 2026 03:04

imotov approved these changes Jul 22, 2026

View reviewed changes

nvzm123 force-pushed the pr-147 branch from 093028e to df836cc Compare July 25, 2026 02:14

nvzm123 requested a review from imotov July 27, 2026 13:06

nvzm123 added 2 commits July 27, 2026 06:12

Merge branch 'main' into pr-147

4ce56fa

imotov reviewed Jul 27, 2026

View reviewed changes

Comment thread src/main/java/com/nvidia/cuvs/lucene/LuceneAcceleratedHNSWBinaryQuantizedVectorsFormat.java Outdated

Comment thread src/main/java/com/nvidia/cuvs/lucene/LuceneProvider.java

nvzm123 added 2 commits July 27, 2026 23:15

Avoid leaking unused quantized flat writers

9e164ad

Create flat vector writers only when the accelerated writer consumes them. CPU fallback paths construct their own writers, so eager allocation leaked the unused flat writer for both binary and scalar quantization.

Restore Lucene provider follow-up TODO

7d70d2f

Keep the existing Lucene 102 binary-format initialization while preserving the reminder to replace version-specific partial providers in a separate initiative.

nvzm123 changed the title ~~Fix PyLucene sidecar packaging and add smoke tests~~ Add PyLucene integration and CPU/GPU end-to-end tests Jul 27, 2026

nvzm123 mentioned this pull request Jul 31, 2026

Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench NVIDIA/cuvs#2385

Draft

Merge main and finalize PyLucene GPU tests

f0e14f4

nvzm123 mentioned this pull request Aug 4, 2026

Add multithreaded concurrency coverage for cuVS-Lucene PyLucene tests NVIDIA/cuvs#2407

Open

Reference multithreaded test follow-up

1ecc8bb

cjnolet requested changes Aug 4, 2026

View reviewed changes


		#### PyLucene end-to-end tests

		Pytest cases are under `src/test/python`. The parametrized cases and assertions

		@@ -0,0 +1,232 @@
		#!/bin/bash

		# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Uh oh!

Conversation

nvzm123 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

PyLucene test coverage

Test layout

Validation

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imotov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cjnolet left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nvzm123 commented Jul 8, 2026 •

edited

Loading

cjnolet left a comment •

edited

Loading