feat(vcr): Add VCR test system for LLM/embedding recording and playback #18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements a VCR (Video Cassette Recorder) test system that enables recording and replaying LLM and embedding API calls for deterministic, fast, and cost-effective testing. Core components: - VCRMode: 6 modes (PLAYBACK, RECORD, RECORD_NEW, RECORD_FAILED, PLAYBACK_OR_RECORD, OFF) with smart mode selection - VCRExtension: JUnit 5 extension with full lifecycle callbacks - VCRContext: Redis container management with AOF/RDB persistence - VCRRegistry: Test recording status tracking with smart mode logic - @VCRTest, @VCRRecord, @VCRDisabled: Annotations for test configuration Key features: - Redis-based cassette storage with persistence to src/test/resources/vcr-data - Testcontainers integration for isolated Redis instances - Call counter management for deterministic key generation - Statistics tracking (cache hits, misses, API calls) - Configurable data directory and Redis image Also includes: - Comprehensive documentation in docs/experimental section - README section with quick start example - Design documents for VCR system and EmbeddingsCache enhancement All 39 VCR tests passing. LLM/embedding interceptor implementation pending.
Implements VCRCassetteStore for storing and retrieving recorded API
responses in Redis:
- Store cassettes as Redis JSON documents with structured keys
- Support for single and batch embedding cassettes
- Handle both array and object JSON responses from Jedis
- Add VCRCassetteMissingException for strict playback mode
- Unit tests for cassette store operations
Cassette key format: vcr:{type}:{testId}:{callIndex}
Implements VCR wrappers for LangChain4J models: - VCREmbeddingModel: wraps EmbeddingModel for recording/replaying embeddings - VCRChatModel: wraps ChatLanguageModel for recording/replaying chat responses - VCREmbeddingInterceptor: low-level interceptor for embedding operations Features: - Support for all VCR modes (PLAYBACK, RECORD, PLAYBACK_OR_RECORD, OFF) - Redis-backed cassette storage integration - In-memory cassette cache for testing - Call counter tracking for unique cassette keys - Statistics tracking (cache hits, misses, recorded count) Unit tests verify recording and playback behavior.
Implements VCR wrappers for Spring AI models: - VCRSpringAIEmbeddingModel: wraps EmbeddingModel for recording/replaying - VCRSpringAIChatModel: wraps ChatModel for recording/replaying chat Features: - Full Spring AI interface implementation - Support for embedForResponse() and call() methods - Redis-backed cassette storage integration - In-memory cassette cache for unit testing - Statistics tracking for cache hits, misses, and recordings Unit tests verify recording and playback behavior.
Implements declarative VCR support via JUnit 5 annotations:
- @VCRModel: marks model fields for automatic VCR wrapping
- VCRModelWrapper: wraps LangChain4J and Spring AI models automatically
- VCRContext: manages Redis container, cassette store, and test state
- VCRExtension: JUnit 5 extension for lifecycle management
Features:
- VCR_MODE environment variable support for runtime mode override
- Automatic model detection (LangChain4J/Spring AI embedding/chat)
- Cassette store integration for Redis persistence
- Test isolation with per-test call counters
- Statistics tracking across test session
Example usage:
@VCRTest(mode = VCRMode.PLAYBACK_OR_RECORD)
class MyTest {
@VCRModel
private EmbeddingModel model = createModel();
}
Adds a complete demo project showing VCR usage with LangChain4J: - LangChain4JVCRDemoTest: demonstrates embedding and chat model recording - Pre-recorded cassettes in src/test/resources/vcr-data/ - README with usage instructions and best practices Demo tests: - Single and batch text embedding - Chat completion with various prompts - Combined RAG-style workflow (embed + generate) Run without API key using pre-recorded cassettes: ./gradlew :demos:langchain4j-vcr:test
Adds a complete demo project showing VCR usage with Spring AI: - SpringAIVCRDemoTest: demonstrates embedding and chat model recording - Pre-recorded cassettes in src/test/resources/vcr-data/ - README with usage instructions and best practices Demo tests: - Single and batch text embedding - Chat completion with various prompts - Combined RAG-style workflow (embed + generate) Run without API key using pre-recorded cassettes: ./gradlew :demos:spring-ai-vcr:test
Updates build configuration: - settings.gradle.kts: include langchain4j-vcr and spring-ai-vcr demos - build.gradle.kts: configure demo subprojects with dependencies - core/build.gradle.kts: add testcontainers dependency for VCR Demo projects inherit common configuration and add their specific AI framework dependencies (LangChain4J or Spring AI with OpenAI).
Updates the AsciiDoc documentation for VCR testing: - Add VCR_MODE environment variable documentation - Document annotation-based approach with @VCRModel - Add troubleshooting section - Include demo project references
Enhances the README VCR section with: - Quick start with JUnit 5 annotation example - VCR modes table with API key requirements - Environment variable override instructions (VCR_MODE) - Improved code examples for LangChain4J and Spring AI - How it works explanation - Links to demo projects Makes it easier for users to understand and adopt the VCR test system.
Fixes three bugs that caused vcr-data files to be modified even when tests were configured with @VCRTest(mode = VCRMode.PLAYBACK): 1. Fixed @nested test class annotation lookup - The VCRExtension was not finding @VCRTest annotations on enclosing classes for nested tests, causing it to fall back to the default PLAYBACK_OR_RECORD mode. Added findVCRTestAnnotation() to walk up the class hierarchy. 2. Added temp directory copy for playback mode - In PLAYBACK mode, the vcr-data directory is now copied to a temp directory before mounting to Docker, preventing any modifications to the source files. 3. Fixed registry writes in playback mode - testSuccessful() and testFailed() now check the mode before writing to the registry, only updating it when recording. Additional changes: - Disabled Redis AOF persistence in playback mode (--appendonly no) - Disabled Redis RDB saves in playback mode (--save "")
…ar override - Updated Basic Usage section to show declarative @VCRModel annotation approach - Added VCR_MODE environment variable override documentation - Fixed outdated references (VCRCassetteStore now implemented, not 'coming soon') - Changed -Dvcr.mode system property reference to VCR_MODE env var - Added IMPORTANT note about model initialization timing requirement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental VCR (Video Cassette Recorder) test system for recording and replaying LLM and embedding API calls during testing. This enables:
Features
Core VCR Framework
VCRModeenum with 6 modes: PLAYBACK, RECORD, RECORD_NEW, RECORD_FAILED, PLAYBACK_OR_RECORD, OFFVCRTestannotation for declarative test configurationVCRModelannotation for automatic model wrappingVCRContextmanages Redis container lifecycle with AOF/RDB persistenceVCRRegistrytracks recording status per testVCRCassetteStorestores cassettes as Redis JSON documentsVCR_MODEenvironment variable override for CI/CD flexibilityFramework Wrappers
VCREmbeddingModel,VCRChatModelVCRSpringAIEmbeddingModel,VCRSpringAIChatModelDemo Projects
demos/langchain4j-vcr/- LangChain4J VCR demo with pre-recorded cassettesdemos/spring-ai-vcr/- Spring AI VCR demo with pre-recorded cassettesBoth demos can be run without an API key using pre-recorded cassettes:
Documentation
vcr-testing.adocguideTest Plan
./gradlew :core:test --tests "com.redis.vl.test.vcr.*")./gradlew clean build check)