"Passage not found" errors and "top_k exceeds documents" warnings during Google Drive indexing with categorization

[BuzzClanAI]

DocumentCategorizer.py
Problem Summary
I'm experiencing two critical issues when using LEANN to index Google Drive files with document categorization:

"Passage with ID 0 not found" errors - occurring repeatedly during categorization
"Requested top_k (5) exceeds total documents (1)" warnings - happening on every categorization step

Use Case & Workflow
I'm building a Google Drive indexer that:

• Loops through Google Drive files
• Gets metadata for each file
• Extracts content from files
• Uses LLaMA to categorize the content (this is where errors occur)
• Adds to LEANN index
• Rebuilds index every 5 additions

Error Details
Error 1: "Passage not found"

2025-09-21 14:06:09,255 - ERROR - Passage with ID 0 not found
ERROR:leann.api:   ✗ [ 1] ID: '0' -> ERROR: Passage not found!

Error 2: "top_k exceeds documents"
WARNING:leann.api: ⚠️ Requested top_k (5) exceeds total documents (1) WARNING:leann.api: ✅ Auto-adjusted top_k to 1 to match available documents

Reproduction Steps

• Set up Google Drive API credentials
• Run the indexing process with categorization enabled
• For each file, the categorization step triggers a search with top_k=5
• Since the index only has 1 document at that point, the warning appears
• The search returns ID "0" which doesn't exist in the passage manager
• Error occurs and categorization fails

Expected Behavior

• Categorization should work with existing documents in the index
• No "Passage not found" errors should occur
• The system should handle small index sizes gracefully

Actual Behavior

• Every categorization step fails with "Passage not found" error
• Warnings about top_k exceeding available documents
• Categorization process is unreliable

Environment

LEANN Version : 0.3.3
Python: 3.11.13
Backend: HNSW
Platform: Linux (WSL2)
Log Output Sample

Conversion complete.
🔍 Collecting metadata for BuzzClan OOW13 Lead Reports - 7th file
🔍 Processing file content for BuzzClan OOW13 Lead Reports - 7
📝 Processing Google Drive native file: BuzzClan OOW13 Lead Reports
 gathered content from BuzzClan OOW13 Lead Reports located at None
🔍 Categorizing document for BuzzClan OOW13 Lead Reports - 7
WARNING:leann.api:  ⚠️  Requested top_k (5) exceeds total documents (1)
WARNING:leann.api:  ✅ Auto-adjusted top_k to 1 to match available documents
ZmqDistanceComputer initialized: d=768, metric=0
2025-09-21 14:06:44,499 - ERROR - Passage with ID 0 not found
ERROR:leann.api:   ✗ [ 1] ID: '0' -> ERROR: Passage not found!
🔍 Adding to LEANN index for BuzzClan OOW13 Lead Reports - 7

Questions

• Is there a recommended approach for categorization during incremental indexing?
• Should we use a different search strategy when the index is small?
• Is there a way to ensure passage IDs are consistent between searches?

Additional Context

• This is part of a Google Drive RAG system
• The categorization helps organize documents by type/content
• The incremental indexing approach (every 5 documents) is necessary for memory management
• The errors prevent proper categorization of documents (does it?)

[yichuan-w]

We will try to reproduce this asap, very interesting setting

[yichuan-w]

fixed in #117