[Feature]: Event-driven Incremental RAG Indexing

[jlyoungblood]

Feature Area

Ingestion (document processing, upload, Docling), Connectors (Google Drive, OneDrive, SharePoint)

Problem Description

Current RAG ingestion patterns are commonly batch-oriented:

• Periodic rescans of knowledge sources
• Full document reprocessing
• Full or large-scale re-embedding
• Delayed propagation of updates/deletes/permission changes
• Limited replayability and observability

As enterprise knowledge systems become increasingly dynamic, customers may require:

• Near real-time synchronization
• Incremental indexing
• Event-driven ingestion orchestration
• Replayable/recoverable ingestion pipelines
• Distributed ingestion coordination

Proposed Solution

Introduce support for event-driven ingestion and incremental index synchronization using streaming platforms such as Apache Kafka and Confluent.

The goal is to enable OpenRAG to react to enterprise knowledge changes in near real-time without requiring full re-indexing or periodic batch rescans.

This capability would allow OpenRAG to:

1. Continuously synchronize retrieval indexes with enterprise systems
2. Reprocess only changed chunks/documents
3. Support replayable ingestion pipelines
4. Improve ingestion scalability and freshness
5. Reduce unnecessary embedding generation and indexing costs

Example events:

{
  "event_type": "document.updated",
  "doc_id": "policy-123",
  "source": "sharepoint",
  "version": "v7",
  "modified_at": "2026-05-26T14:00:00Z"
}

OpenRAG ingestion workers consume these events and perform targeted reprocessing.

Example Architecture

Enterprise Systems (SharePoint, GDrive, S3, DBs, APIs)
↓
Kafka / Confluent Topics
↓
OpenRAG Event Consumer
↓
Docling Parsing + Extraction
↓
Chunk Diffing / Hash Comparison
↓
Selective Re-Embedding
↓
OpenSearch / Vector DB Update
↓
RAG Retrieval Layer

Additional Requirements

Incremental Chunk Updates

Support chunk-level change detection:

Compute stable chunk hashes
Compare old vs new chunk states
Re-embed only changed chunks
Preserve embeddings for unchanged chunks
Delete/tombstone removed chunks

Potential metadata:

doc_id
doc_version
chunk_id
chunk_hash
page_number
embedding_model
chunking_config_version
source_modified_time

Replayable Ingestion

Support replaying historical ingestion events to:

• Rebuild indexes
• Re-run improved chunking strategies
• Migrate embedding models
• Recover from ingestion/indexing bugs
• Recompute retrieval metadata

Use Case

• As a platform engineer, I want OpenRAG to consume document and knowledge change events from Kafka/Confluent so that retrieval indexes can stay continuously synchronized without requiring full reindexing.
• As an enterprise AI administrator, I want OpenRAG to incrementally reprocess and re-embed only changed chunks so that I can reduce ingestion costs and keep RAG responses fresh in near real-time.
• As a developer building production RAG systems, I want ingestion pipelines to be event-driven and replayable so that I can recover from failures, rebuild indexes, and migrate embedding/chunking strategies safely at scale.
• As a knowledge platform operator, I want permission, deletion, and document update events to propagate automatically into the retrieval layer so that users only retrieve current and authorized knowledge.

Alternatives Considered

No response

Priority

Would improve my workflow

Additional Context

No response

Contribution

• I would be willing to help implement this feature.
• I can help test this feature once implemented.

Checklist

• I have searched existing issues and discussions to ensure this feature hasn't been requested before.