The changes made in this commit include:
1. Adding a `CodeConfluenceGraph` dependency to the `GenericCodebaseProcessingActivity` class constructor. This allows the activity to access the shared graph database instance for managing transactions.
2. Updating the `codebase_processing_envelope` argument passed to the `process_codebase_generic` activity to include the `code_confluence_graph` instance.
3. Removing the `process_codebase_generic` function export and the `generic_codebase_processing_activity` instance, as these are now created in the `main.py` file with proper dependency injection.
4. Updating the `codebase_child_workflow.py` file to use the `GenericCodebaseProcessingActivity` class instead of the `process_codebase_generic` function.
5. Updating the `main.py` file to create the `GenericCodebaseProcessingActivity` instance with the shared `CodeConfluenceGraph` instance, and adding it to the list of registered activities.
These changes ensure that the `GenericCodebaseProcessingActivity` has access to the shared graph database instance, allowing it to manage transactions more effectively during the codebase processing workflow.
User description
The changes made in this commit include:
Adding a
CodeConfluenceGraphdependency to theGenericCodebaseProcessingActivityclass constructor. This allows the activity to access the shared graph database instance for managing transactions.Updating the
codebase_processing_envelopeargument passed to theprocess_codebase_genericactivity to include thecode_confluence_graphinstance.Removing the
process_codebase_genericfunction export and thegeneric_codebase_processing_activityinstance, as these are now created in themain.pyfile with proper dependency injection.Updating the
codebase_child_workflow.pyfile to use theGenericCodebaseProcessingActivityclass instead of theprocess_codebase_genericfunction.Updating the
main.pyfile to create theGenericCodebaseProcessingActivityinstance with the sharedCodeConfluenceGraphinstance, and adding it to the list of registered activities.These changes ensure that the
GenericCodebaseProcessingActivityhas access to the shared graph database instance, allowing it to manage transactions more effectively during the codebase processing workflow.PR Type
Enhancement, Tests, Bug fix
Description
• Major database architecture refactoring: Migrated from neomodel ORM to raw Cypher queries with managed transactions for Neo4j operations, improving performance and reliability
• Enhanced connection management: Implemented per-event-loop AsyncEngine pattern for PostgreSQL to prevent "Future attached to different loop" errors in multi-threaded environments
• Dependency injection pattern: Refactored activities to use constructor-based dependency injection with
CodeConfluenceGraphinstances instead of global singletons• Comprehensive repository deletion: Added managed transaction-based deletion operations with proper relationship cleanup and batch processing
• Improved test infrastructure: Added graph assertion utilities, enhanced integration tests with proper cleanup, and optimized database operations using TRUNCATE CASCADE
• Session management improvements: Implemented session pooling and context managers for both Neo4j and PostgreSQL connections
• Code cleanup: Removed deprecated synchronous methods, simplified module exports, and fixed import errors
• Configuration updates: Updated pytest dependencies and API folder configurations
Diagram Walkthrough
File Walkthrough
11 files
code_confluence_graph_deletion.py
Implement managed transaction-based repository deletion with rawCypherunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph_deletion.py
• Added comprehensive managed transaction methods for repository
deletion using raw Cypher queries
• Implemented batch deletion
operations for files, packages, codebases, and metadata with proper
relationship cleanup
• Added new
delete_repository_by_qualified_name_managedmethod that handlesdeletion in proper dependency order
• Deprecated old neomodel-based
deletion methods while maintaining backward compatibility
generic_codebase_parser.py
Migrate codebase parser to managed transactions with raw Cypherunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/parser/generic_codebase_parser.py
• Added managed transaction methods for creating packages,
hierarchies, and files using raw Cypher MERGE operations
• Implemented
_handle_node_creation_managedmethod for atomic node creation withproper conflict handling
• Updated main processing method to use
shared
CodeConfluenceGraphsession pool instead of global neomodeltransactions
• Added TYPE_CHECKING imports and optional
code_confluence_graphparameter for dependency injectioncode_confluence_graph_ingestion.py
Implement managed transaction-based graph ingestion with raw Cypherunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph_ingestion.py
• Added managed transaction methods for repository and codebase
ingestion using raw Cypher operations
• Implemented transaction
functions for creating repositories, codebases, metadata, and
framework relationships
• Added new
insert_code_confluence_git_repo_managedmethod that uses session-basedmanaged transactions
• Deprecated old neomodel-based ingestion methods
while maintaining backward compatibility
db.py
Refactored PostgreSQL connection management for multi-loopcompatibilityunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/db.py
• Implemented per-event-loop AsyncEngine pattern to prevent "Future
attached to different loop" errors
• Added global engine registry with
thread-safe access using
asyncio.Lock• Enhanced session management
with detailed logging and proper cleanup
• Added
dispose_current_enginefunction for cleanup operationsmain.py
Refactored activity initialization with dependency injection patternunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/main.py
• Updated activity initialization to use dependency injection with
CodeConfluenceGraphinstance• Changed from function-based to
class-based activity registration pattern
• Modified database deletion
operations to use managed sessions
• Updated engine disposal to use
new
dispose_current_enginefunctioncode_confluence_graph.py
Simplified Neo4j connection management and removed singleton patternunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph.py
• Removed singleton pattern and replaced with regular class
instantiation
• Added
get_sessionasync context manager for Neo4jsession management
• Simplified connection management by removing
transaction context manager
• Added direct access to AsyncDriver for
session creation
package_manager_metadata_ingestion.py
Updated package metadata ingestion to use managed Neo4j sessionsunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/package_metadata_activity/package_manager_metadata_ingestion.py
• Added
CodeConfluenceGraphdependency injection in constructor•
Updated to use managed sessions from connection pool instead of global
connection
• Modified graph operations to use session-based
transaction management
• Enhanced logging with contextual information
about graph instance usage
generic_codebase_processing_activity.py
Refactored generic codebase processing with dependency injectionunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/generic_codebase_processing_activity.py
• Added
CodeConfluenceGraphdependency injection in constructor•
Updated parser initialization to include graph instance parameter
•
Removed module-level activity instance and function exports
• Added
note about dependency injection being handled in main.py
confluence_git_graph.py
Updated Git activity to use managed Neo4j sessionsunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/git_activity/confluence_git_graph.py
• Added
CodeConfluenceGraphdependency injection in constructor•
Updated to use managed sessions from shared connection pool
• Modified
graph operations to use session-based transaction management
•
Enhanced logging to reflect shared connection pool usage
sync_db_cleanup.py
Optimized PostgreSQL cleanup using TRUNCATE CASCADEunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/sync_db_cleanup.py
• Changed PostgreSQL cleanup from DELETE to TRUNCATE CASCADE for
better performance
• Added session flush operation after commit
•
Improved cleanup efficiency by using TRUNCATE which removes all rows
faster
codebase_child_workflow.py
Updated workflow to use class-based activity patternunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/codebase_child_workflow.py
• Updated import to use
GenericCodebaseProcessingActivityclassinstead of function
• Changed activity execution to use class method
reference
• Aligned with new dependency injection pattern for
activities
7 files
test_delete_repository.py
Add comprehensive integration tests for repository deletion endpointunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_delete_repository.py
• Added comprehensive integration tests for repository deletion
endpoint functionality
• Implemented test for deleting non-existent
repositories with proper 404 error handling
• Added full workflow test
covering detection, ingestion, completion monitoring, and deletion
verification
• Included graph-level verification using custom
assertion utilities to ensure complete cleanup
test_generic_codebase_parser.py
Update parser tests to use managed transactions and raw Cypherunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/parser/test_generic_codebase_parser.py
• Updated parser integration tests to use managed transactions with
CodeConfluenceGraphinstance• Replaced neomodel graph model imports
with raw Cypher queries for verification
• Modified codebase node
creation to use managed transactions instead of neomodel operations
•
Updated all verification queries to use direct Cypher instead of
neomodel methods
graph_assertions.py
Add graph assertion utilities for comprehensive deletion verificationunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/graph_assertions.py
• Added comprehensive graph assertion utilities for integration test
validation
• Implemented functions for counting nodes by label,
checking repository existence, and relationship validation
• Added
verify_complete_repository_deletionfunction for thorough cleanupverification
• Included utilities for finding residual nodes and
relationships connected to deleted repositories
test_start_ingestion.py
Enhanced integration test database cleanup and async handlingunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_start_ingestion.py
• Added database cleanup using context manager pattern with
get_sync_postgres_sessionanddispose_current_engine• Updated pytest
markers to use
loop_scope="session"for better async test isolation•
Improved workflow termination and cleanup handling with proper
exception management
• Removed one test method and simplified test
structure for better reliability
test_framework_detection_with_postgres.py
Simplified framework detection tests by removing database setupunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_framework_detection_with_postgres.py
• Removed framework loader initialization and database loading
operations from tests
• Simplified test methods by removing sync
database operations
• Updated pytest markers to use
loop_scope="session"• Cleaned up test structure by removing redundant
framework definition loading
sync_db_utils.py
Added isolated PostgreSQL session context manager for testsunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/sync_db_utils.py
• Added
get_sync_postgres_sessioncontext manager for isolateddatabase sessions
• Implemented fresh engine creation per session to
avoid lock issues
• Added proper session lifecycle management with
commit/rollback handling
• Enhanced database session isolation for
testing scenarios
conftest.py
Removed unnecessary sleep delay from test client setupunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/conftest.py
• Removed sleep delay from test client initialization
• Simplified
test client setup by removing unnecessary wait time
10 files
__init__.py
Remove direct async engine export from PostgreSQL moduleunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/init.py
• Removed direct async engine export from PostgreSQL module
• Updated
module documentation to reflect per-event-loop engine management using
contextvars
• Simplified module exports to empty list as engine is now
managed contextually
pyproject.toml
Updated pytest configuration and dependenciesunoplat-code-confluence-ingestion/code-confluence-flow-bridge/pyproject.toml
• Added
pytest-order>=1.2.1dependency for test ordering• Removed
timeout configuration from pytest options
• Updated test configuration
for better test management
yaak.fl_KBoWUpi8x7.yaml
Updated API folder configurationyak/yaak.fl_KBoWUpi8x7.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp
yaak.fl_UkSk9aN56L.yaml
Updated API folder configurationyak/yaak.fl_UkSk9aN56L.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp
yaak.fl_RfTzTxKasf.yaml
Updated API folder configurationyak/yaak.fl_RfTzTxKasf.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp
yaak.fl_TpVgSdpioc.yaml
Updated API folder configurationyak/yaak.fl_TpVgSdpioc.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp
yaak.fl_a9CJGFoCVq.yaml
Added new API folder for code-confluence-flow-bridgeyak/yaak.fl_a9CJGFoCVq.yaml
• Added new API folder configuration for code-confluence-flow-bridge
•
Created folder structure with proper workspace and parent folder
references
yaak.fl_Pgb37JQfSX.yaml
Added new API folder for code-confluence-query-engineyak/yaak.fl_Pgb37JQfSX.yaml
• Added new API folder configuration for code-confluence-query-engine
• Created folder structure with proper workspace and parent folder
references
yaak.fl_pr9oGjGRnn.yaml
Updated API folder configurationyak/yaak.fl_pr9oGjGRnn.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
timestamp for folder configuration
yaak.fl_ADhjPjZJE9.yaml
Updated API folder configurationyak/yaak.fl_ADhjPjZJE9.yaml
• Updated folder hierarchy by changing parent folder ID
• Modified
timestamp for folder configuration
1 files
framework_loader.py
Removed synchronous framework loading methodunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/framework_loader.py
• Removed synchronous version of framework loading method
• Cleaned up
code by removing
load_framework_definitions_at_startup_syncfunction•
Simplified class interface by keeping only async methods
1 files
parent_workflow_db_activity.py
Minor formatting improvementsunoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/parent_workflow_db_activity.py
• Added whitespace formatting improvements
• Minor code formatting
changes for consistency
1 files
test_framework_definitions_ingestion.py
Fixed import error by removing unused async_engine importunoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_framework_definitions_ingestion.py
• Removed unused
async_engineimport that was causing ImportError•
Cleaned up imports after database refactoring
13 files