Skip to content

feat(perf): perf and reliability improvements across neo4j/postgres - #669

Merged
JayGhiya merged 1 commit into
mainfrom
neo4j-fixes
Aug 1, 2025
Merged

feat(perf): perf and reliability improvements across neo4j/postgres#669
JayGhiya merged 1 commit into
mainfrom
neo4j-fixes

Conversation

@JayGhiya

@JayGhiya JayGhiya commented Aug 1, 2025

Copy link
Copy Markdown
Member

User description

The changes made in this commit include:

  1. Adding a CodeConfluenceGraph dependency to the GenericCodebaseProcessingActivity class constructor. This allows the activity to access the shared graph database instance for managing transactions.

  2. Updating the codebase_processing_envelope argument passed to the process_codebase_generic activity to include the code_confluence_graph instance.

  3. Removing the process_codebase_generic function export and the generic_codebase_processing_activity instance, as these are now created in the main.py file with proper dependency injection.

  4. Updating the codebase_child_workflow.py file to use the GenericCodebaseProcessingActivity class instead of the process_codebase_generic function.

  5. Updating the main.py file to create the GenericCodebaseProcessingActivity instance with the shared CodeConfluenceGraph instance, and adding it to the list of registered activities.

These changes ensure that the GenericCodebaseProcessingActivity has access to the shared graph database instance, allowing it to manage transactions more effectively during the codebase processing workflow.


PR Type

Enhancement, Tests, Bug fix


Description

Major database architecture refactoring: Migrated from neomodel ORM to raw Cypher queries with managed transactions for Neo4j operations, improving performance and reliability
Enhanced connection management: Implemented per-event-loop AsyncEngine pattern for PostgreSQL to prevent "Future attached to different loop" errors in multi-threaded environments
Dependency injection pattern: Refactored activities to use constructor-based dependency injection with CodeConfluenceGraph instances instead of global singletons
Comprehensive repository deletion: Added managed transaction-based deletion operations with proper relationship cleanup and batch processing
Improved test infrastructure: Added graph assertion utilities, enhanced integration tests with proper cleanup, and optimized database operations using TRUNCATE CASCADE
Session management improvements: Implemented session pooling and context managers for both Neo4j and PostgreSQL connections
Code cleanup: Removed deprecated synchronous methods, simplified module exports, and fixed import errors
Configuration updates: Updated pytest dependencies and API folder configurations


Diagram Walkthrough

flowchart LR
  A["Old Architecture"] --> B["Neomodel ORM"]
  A --> C["Global Singletons"]
  A --> D["Synchronous Operations"]
  
  E["New Architecture"] --> F["Raw Cypher Queries"]
  E --> G["Managed Transactions"]
  E --> H["Dependency Injection"]
  E --> I["Session Pooling"]
  
  B --> F
  C --> H
  D --> G
  
  F --> J["Better Performance"]
  G --> K["Improved Reliability"]
  H --> L["Better Testability"]
  I --> M["Connection Efficiency"]
Loading

File Walkthrough

Relevant files
Enhancement
11 files
code_confluence_graph_deletion.py
Implement managed transaction-based repository deletion with raw
Cypher

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph_deletion.py

• Added comprehensive managed transaction methods for repository
deletion using raw Cypher queries
• Implemented batch deletion
operations for files, packages, codebases, and metadata with proper
relationship cleanup
• Added new
delete_repository_by_qualified_name_managed method that handles
deletion in proper dependency order
• Deprecated old neomodel-based
deletion methods while maintaining backward compatibility

+598/-15
generic_codebase_parser.py
Migrate codebase parser to managed transactions with raw Cypher

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/parser/generic_codebase_parser.py

• Added managed transaction methods for creating packages,
hierarchies, and files using raw Cypher MERGE operations
• Implemented
_handle_node_creation_managed method for atomic node creation with
proper conflict handling
• Updated main processing method to use
shared CodeConfluenceGraph session pool instead of global neomodel
transactions
• Added TYPE_CHECKING imports and optional
code_confluence_graph parameter for dependency injection

+455/-53
code_confluence_graph_ingestion.py
Implement managed transaction-based graph ingestion with raw Cypher

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph_ingestion.py

• Added managed transaction methods for repository and codebase
ingestion using raw Cypher operations
• Implemented transaction
functions for creating repositories, codebases, metadata, and
framework relationships
• Added new
insert_code_confluence_git_repo_managed method that uses session-based
managed transactions
• Deprecated old neomodel-based ingestion methods
while maintaining backward compatibility

+365/-38
db.py
Refactored PostgreSQL connection management for multi-loop
compatibility

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/db.py

• Implemented per-event-loop AsyncEngine pattern to prevent "Future
attached to different loop" errors
• Added global engine registry with
thread-safe access using asyncio.Lock
• Enhanced session management
with detailed logging and proper cleanup
• Added
dispose_current_engine function for cleanup operations

+135/-86
main.py
Refactored activity initialization with dependency injection pattern

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/main.py

• Updated activity initialization to use dependency injection with
CodeConfluenceGraph instance
• Changed from function-based to
class-based activity registration pattern
• Modified database deletion
operations to use managed sessions
• Updated engine disposal to use
new dispose_current_engine function

+20/-15 
code_confluence_graph.py
Simplified Neo4j connection management and removed singleton pattern

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph.py

• Removed singleton pattern and replaced with regular class
instantiation
• Added get_session async context manager for Neo4j
session management
• Simplified connection management by removing
transaction context manager
• Added direct access to AsyncDriver for
session creation

+31/-28 
package_manager_metadata_ingestion.py
Updated package metadata ingestion to use managed Neo4j sessions

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/package_metadata_activity/package_manager_metadata_ingestion.py

• Added CodeConfluenceGraph dependency injection in constructor

Updated to use managed sessions from connection pool instead of global
connection
• Modified graph operations to use session-based
transaction management
• Enhanced logging with contextual information
about graph instance usage

+19/-14 
generic_codebase_processing_activity.py
Refactored generic codebase processing with dependency injection

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/generic_codebase_processing_activity.py

• Added CodeConfluenceGraph dependency injection in constructor

Updated parser initialization to include graph instance parameter

Removed module-level activity instance and function exports
• Added
note about dependency injection being handled in main.py

+8/-9     
confluence_git_graph.py
Updated Git activity to use managed Neo4j sessions             

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/git_activity/confluence_git_graph.py

• Added CodeConfluenceGraph dependency injection in constructor

Updated to use managed sessions from shared connection pool
• Modified
graph operations to use session-based transaction management

Enhanced logging to reflect shared connection pool usage

+11/-6   
sync_db_cleanup.py
Optimized PostgreSQL cleanup using TRUNCATE CASCADE           

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/sync_db_cleanup.py

• Changed PostgreSQL cleanup from DELETE to TRUNCATE CASCADE for
better performance
• Added session flush operation after commit

Improved cleanup efficiency by using TRUNCATE which removes all rows
faster

+8/-7     
codebase_child_workflow.py
Updated workflow to use class-based activity pattern         

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/codebase_child_workflow.py

• Updated import to use GenericCodebaseProcessingActivity class
instead of function
• Changed activity execution to use class method
reference
• Aligned with new dependency injection pattern for
activities

+2/-2     
Tests
7 files
test_delete_repository.py
Add comprehensive integration tests for repository deletion endpoint

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_delete_repository.py

• Added comprehensive integration tests for repository deletion
endpoint functionality
• Implemented test for deleting non-existent
repositories with proper 404 error handling
• Added full workflow test
covering detection, ingestion, completion monitoring, and deletion
verification
• Included graph-level verification using custom
assertion utilities to ensure complete cleanup

+576/-0 
test_generic_codebase_parser.py
Update parser tests to use managed transactions and raw Cypher

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/parser/test_generic_codebase_parser.py

• Updated parser integration tests to use managed transactions with
CodeConfluenceGraph instance
• Replaced neomodel graph model imports
with raw Cypher queries for verification
• Modified codebase node
creation to use managed transactions instead of neomodel operations

Updated all verification queries to use direct Cypher instead of
neomodel methods

+97/-84 
graph_assertions.py
Add graph assertion utilities for comprehensive deletion verification

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/graph_assertions.py

• Added comprehensive graph assertion utilities for integration test
validation
• Implemented functions for counting nodes by label,
checking repository existence, and relationship validation
• Added
verify_complete_repository_deletion function for thorough cleanup
verification
• Included utilities for finding residual nodes and
relationships connected to deleted repositories

+233/-0 
test_start_ingestion.py
Enhanced integration test database cleanup and async handling

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_start_ingestion.py

• Added database cleanup using context manager pattern with
get_sync_postgres_session and dispose_current_engine
• Updated pytest
markers to use loop_scope="session" for better async test isolation

Improved workflow termination and cleanup handling with proper
exception management
• Removed one test method and simplified test
structure for better reliability

+61/-115
test_framework_detection_with_postgres.py
Simplified framework detection tests by removing database setup

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_framework_detection_with_postgres.py

• Removed framework loader initialization and database loading
operations from tests
• Simplified test methods by removing sync
database operations
• Updated pytest markers to use
loop_scope="session"
• Cleaned up test structure by removing redundant
framework definition loading

+13/-70 
sync_db_utils.py
Added isolated PostgreSQL session context manager for tests

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/utils/sync_db_utils.py

• Added get_sync_postgres_session context manager for isolated
database sessions
• Implemented fresh engine creation per session to
avoid lock issues
• Added proper session lifecycle management with
commit/rollback handling
• Enhanced database session isolation for
testing scenarios

+32/-0   
conftest.py
Removed unnecessary sleep delay from test client setup     

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/conftest.py

• Removed sleep delay from test client initialization
• Simplified
test client setup by removing unnecessary wait time

+1/-1     
Configuration changes
10 files
__init__.py
Remove direct async engine export from PostgreSQL module 

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/init.py

• Removed direct async engine export from PostgreSQL module
• Updated
module documentation to reflect per-event-loop engine management using
contextvars
• Simplified module exports to empty list as engine is now
managed contextually

+3/-4     
pyproject.toml
Updated pytest configuration and dependencies                       

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/pyproject.toml

• Added pytest-order>=1.2.1 dependency for test ordering
• Removed
timeout configuration from pytest options
• Updated test configuration
for better test management

+3/-2     
yaak.fl_KBoWUpi8x7.yaml
Updated API folder configuration                                                 

yak/yaak.fl_KBoWUpi8x7.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp

+3/-3     
yaak.fl_UkSk9aN56L.yaml
Updated API folder configuration                                                 

yak/yaak.fl_UkSk9aN56L.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp

+3/-3     
yaak.fl_RfTzTxKasf.yaml
Updated API folder configuration                                                 

yak/yaak.fl_RfTzTxKasf.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp

+3/-3     
yaak.fl_TpVgSdpioc.yaml
Updated API folder configuration                                                 

yak/yaak.fl_TpVgSdpioc.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
sort priority and updated timestamp

+3/-3     
yaak.fl_a9CJGFoCVq.yaml
Added new API folder for code-confluence-flow-bridge         

yak/yaak.fl_a9CJGFoCVq.yaml

• Added new API folder configuration for code-confluence-flow-bridge

Created folder structure with proper workspace and parent folder
references

+13/-0   
yaak.fl_Pgb37JQfSX.yaml
Added new API folder for code-confluence-query-engine       

yak/yaak.fl_Pgb37JQfSX.yaml

• Added new API folder configuration for code-confluence-query-engine

• Created folder structure with proper workspace and parent folder
references

+13/-0   
yaak.fl_pr9oGjGRnn.yaml
Updated API folder configuration                                                 

yak/yaak.fl_pr9oGjGRnn.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
timestamp for folder configuration

+2/-2     
yaak.fl_ADhjPjZJE9.yaml
Updated API folder configuration                                                 

yak/yaak.fl_ADhjPjZJE9.yaml

• Updated folder hierarchy by changing parent folder ID
• Modified
timestamp for folder configuration

+13/-0   
Miscellaneous
1 files
framework_loader.py
Removed synchronous framework loading method                         

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/framework_loader.py

• Removed synchronous version of framework loading method
• Cleaned up
code by removing load_framework_definitions_at_startup_sync function

Simplified class interface by keeping only async methods

+1/-60   
Formatting
1 files
parent_workflow_db_activity.py
Minor formatting improvements                                                       

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/parent_workflow_db_activity.py

• Added whitespace formatting improvements
• Minor code formatting
changes for consistency

+2/-0     
Bug fix
1 files
test_framework_definitions_ingestion.py
Fixed import error by removing unused async_engine import

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/tests/integration/test_framework_definitions_ingestion.py

• Removed unused async_engine import that was causing ImportError

Cleaned up imports after database refactoring

+1/-3     
Additional files
13 files
txn_context.py +0/-88   
db_cleanup.py +0/-259 
yaak.fl_ZwwFo3jTXL.yaml [link]   
yaak.fl_gqjfAddzF8.yaml [link]   
yaak.rq_844b8rXsh7.yaml [link]   
yaak.rq_DwTkeJP8FJ.yaml [link]   
yaak.rq_EECnrR2aEe.yaml [link]   
yaak.rq_Jp2qmok4Xs.yaml [link]   
yaak.rq_hSEDgHvSMJ.yaml [link]   
yaak.rq_v59Nw6A6i7.yaml [link]   
yaak.rq_v87eKPdXPv.yaml [link]   
yaak.rq_yMsbRmdhHJ.yaml [link]   
yaak.wk_KULjx4bEa8.yaml [link]   

The changes made in this commit include:

1. Adding a `CodeConfluenceGraph` dependency to the `GenericCodebaseProcessingActivity` class constructor. This allows the activity to access the shared graph database instance for managing transactions.

2. Updating the `codebase_processing_envelope` argument passed to the `process_codebase_generic` activity to include the `code_confluence_graph` instance.

3. Removing the `process_codebase_generic` function export and the `generic_codebase_processing_activity` instance, as these are now created in the `main.py` file with proper dependency injection.

4. Updating the `codebase_child_workflow.py` file to use the `GenericCodebaseProcessingActivity` class instead of the `process_codebase_generic` function.

5. Updating the `main.py` file to create the `GenericCodebaseProcessingActivity` instance with the shared `CodeConfluenceGraph` instance, and adding it to the list of registered activities.

These changes ensure that the `GenericCodebaseProcessingActivity` has access to the shared graph database instance, allowing it to manage transactions more effectively during the codebase processing workflow.
@huly-for-github

Copy link
Copy Markdown

Connected to Huly®: UNOPL-806

@claude

claude Bot commented Aug 1, 2025

Copy link
Copy Markdown

Claude finished @JayGhiya's task —— View job


Comprehensive Code Review

I've thoroughly analyzed this PR focusing on performance and reliability improvements. Here's my detailed assessment:

  • Read repository guidelines and understand architecture
  • Review dependency injection changes for CodeConfluenceGraph
  • Analyze transaction management improvements
  • Check database interaction patterns (Neo4j/PostgreSQL)
  • Evaluate test coverage and quality
  • Assess performance and security implications
  • Provide comprehensive feedback

Positive Changes

Architecture Improvements

  1. Excellent Dependency Injection: The move to inject CodeConfluenceGraph into GenericCodebaseProcessingActivity eliminates connection duplication and improves resource management.

  2. Clean Connection Management: Single shared graph connection in main.py:459-461 with proper lifecycle management - connects at startup, closes at shutdown.

  3. Improved PostgreSQL Handling: The per-loop engine pattern in db.py:40-83 elegantly solves the "Future attached to different loop" issue that commonly occurs in async applications.

Performance Enhancements

  1. Connection Pool Optimization: Neo4j connection parameters are properly configured with reasonable defaults (max_connection_pool_size, connection_acquisition_timeout).

  2. Thread Pool Sizing: Smart calculation in main.py:470 - activity slots + buffer ensures adequate thread availability.

  3. Batch Operations: Extensive use of transaction functions in code_confluence_graph_ingestion.py for atomic operations.

Testing Infrastructure

  1. New Test Coverage: The test_delete_repository.py adds crucial integration tests for deletion workflows.

  2. Better Assertions: graph_assertions.py provides comprehensive validation utilities for Neo4j state verification.

⚠️ Areas of Concern

Code Quality Issues

  1. File Size Explosion (generic_codebase_parser.py): +402 lines is concerning. This suggests the class may be taking on too many responsibilities.

  2. Inconsistent Error Handling:

    # main.py:298-304 - Uses ApplicationError for worker init
    raise ApplicationError(error_message, type="WORKER_INITIALIZATION_ERROR")
    
    # vs generic_codebase_processing_activity.py:98-110 - Uses dict details  
    raise ApplicationError(f"Codebase processing failed...", {...})
  3. Resource Management: In main.py:491-500, activities are created but there's no explicit cleanup mechanism for the activity instances themselves.

Transaction Management Concerns

  1. Missing Rollback Logic: The new transaction approach doesn't show explicit rollback handling for partial failures in multi-step operations.

  2. Connection Lifecycle: While the global connection is managed, individual session lifetimes in parsing operations could benefit from more explicit cleanup.

Security Considerations

  1. GitHub Token Handling: The token management in main.py:307-335 looks secure with proper encryption, but the error messages could leak sensitive information.

  2. Input Validation: Neo4j queries use parameterized queries correctly, which prevents injection attacks.

🔧 Specific Recommendations

High Priority

  1. Refactor Large Classes: Consider splitting GenericCodebaseParser into smaller, focused classes:

    # Suggested breakdown:
    - CodebaseAnalyzer (file discovery, package detection)
    - Neo4jIngestionService (database operations)  
    - FrameworkDetectionCoordinator (framework-specific logic)
  2. Standardize Error Handling: Choose one error format for consistency:

    # Recommend this pattern throughout:
    raise ApplicationError(
        message="Brief description",
        details={
            "error_type": "SPECIFIC_ERROR_TYPE",
            "context": {...}
        }
    )

Medium Priority

  1. Add Connection Health Checks: Consider periodic health checks for the shared Neo4j connection, especially for long-running workflows.

  2. Improve Logging Consistency: Some files use structured logging better than others. The trace context binding in generic_codebase_processing_activity.py:66-73 is excellent - apply this pattern consistently.

Low Priority

  1. Documentation: Add inline documentation for complex transaction functions in code_confluence_graph_ingestion.py.

  2. Metrics: Consider adding metrics for connection pool usage and transaction success rates.

📊 Performance Impact Assessment

Positive Impact:

  • Shared connections reduce overhead
  • Transaction batching improves throughput
  • Per-loop engines prevent blocking

Potential Concerns:

  • Large parser class may impact memory usage
  • No explicit connection pool monitoring

🛡️ Security Assessment

Strengths:

  • Parameterized queries prevent injection
  • Token encryption is properly implemented
  • Proper error context without data leaks

Areas to Monitor:

  • Error messages should be reviewed for information disclosure
  • Consider adding rate limiting for expensive graph operations

📋 Test Coverage Evaluation

Excellent Additions:

  • Repository deletion integration tests
  • Graph state validation utilities
  • Comprehensive cleanup verification

Missing Coverage:

  • Transaction rollback scenarios
  • Connection failure recovery
  • Large repository stress tests

Overall Assessment

This is a well-executed architectural improvement that addresses real performance and reliability issues. The dependency injection pattern is properly implemented, and the connection management improvements are sound.

Recommendation: APPROVE with minor refinements

The core changes are solid and move the codebase in the right direction. The identified issues are mostly about code organization and consistency rather than fundamental problems.


Key Files Reviewed: main.py:458-500, generic_codebase_processing_activity.py:32-42, code_confluence_graph.py:78-89, db.py:40-83, code_confluence_graph_ingestion.py:46-150, test files

@qodo-code-review

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Performance Risk

The new managed transaction approach processes packages and files sequentially rather than in batches. The recursive package traversal and individual file processing could be slow for large repositories with thousands of files and nested packages.

while packages_to_process:
    current_pkg_qn = packages_to_process.pop(0)
    if current_pkg_qn in processed_packages:
        continue
    processed_packages.add(current_pkg_qn)
    all_package_qualified_names.append(current_pkg_qn)

    # Get files for this package
    file_records = await session.execute_read(self._get_package_files_txn, current_pkg_qn)
    file_paths = [record["f"]["file_path"] for record in file_records]
    all_file_paths.extend(file_paths)

    # Get subpackages
    subpackage_records = await session.execute_read(self._get_package_subpackages_txn, current_pkg_qn)
    subpackage_qns = [record["sp"]["qualified_name"] for record in subpackage_records]
    packages_to_process.extend(subpackage_qns)
Error Handling

The new managed transaction methods lack proper error handling and rollback mechanisms. If a transaction fails partway through processing multiple files or packages, there's no cleanup or recovery strategy implemented.

    async for file_data in self.extract_files(package_files):
        await self.insert_files_managed(session, [file_data], package_files)

    logger.info(
        f"Processed {self.files_processed} files using managed transactions"
    )

except Exception as e:
    logger.error(f"Failed to process files with managed transactions: {e}")
    raise
Data Consistency

The raw Cypher queries use JSON serialization for complex objects but don't validate the JSON structure or handle deserialization errors. This could lead to data corruption or runtime failures when reading the stored data.

    "repository_name": git_repo.repository_name, 
    "repository_metadata": json.dumps(git_repo.repository_metadata) if git_repo.repository_metadata else "{}", 
    "readme": git_repo.readme
    # Note: github_organization not in domain model, using qualified_name pattern instead
}

@qodo-code-review

Copy link
Copy Markdown

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Fix race condition in engine access

Accessing _engine_per_loop[loop_id] outside the lock creates a race condition.
The dictionary could be modified by another coroutine between releasing the lock
and accessing the value.

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/postgres/db.py [58-83]

 async with _engine_lock:
     if loop_id not in _engine_per_loop:
         ...
         _engine_per_loop[loop_id] = (engine, session_factory)
         ...
     else:
         log_ctx.debug("Reusing cached AsyncEngine for loop {loop_id}")
+    
+    return _engine_per_loop[loop_id]
 
-return _engine_per_loop[loop_id]
-

[To ensure code accuracy, apply this suggestion manually]

Suggestion importance[1-10]: 8

__

Why: This suggestion correctly identifies a potential race condition where the _engine_per_loop dictionary could be modified by another coroutine after the lock is released, and moving the return statement inside the lock resolves this critical issue.

Medium
Fix return type inconsistency

The method returns None when no record is found, but the return type annotation
indicates it should return a str. This could cause runtime errors when the
caller expects a string identifier.

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/parser/generic_codebase_parser.py [265-332]

 async def _handle_node_creation_managed(
     self, session: AsyncSession, node_type: str, node_dict: Dict[str, Any]
 ) -> str:
     """
     Safely create or retrieve a node using managed transactions with raw Cypher MERGE operations.
 
     Args:
         session: Neo4j async session from connection pool
         node_type: Node type (e.g., "CodeConfluencePackage", "CodeConfluenceFile")
         node_dict: Dictionary containing node properties
 
     Returns:
         Node qualified_name or identifier
     """
     try:
         if node_type == "CodeConfluencePackage":
             query = """
             MERGE (n:CodeConfluencePackage {qualified_name: $qualified_name})
             ON CREATE SET 
                 n.name = $name,
                 n.created_at = datetime(),
                 n.updated_at = datetime()
             ON MATCH SET 
                 n.updated_at = datetime()
             RETURN n.qualified_name as identifier
             """
         elif node_type == "CodeConfluenceFile":
             query = """
             MERGE (n:CodeConfluenceFile {file_path: $file_path})
             ON CREATE SET 
                 n.checksum = $checksum,
                 n.structural_signature = $structural_signature,
                 n.imports = $imports,
                 n.created_at = datetime(),
                 n.updated_at = datetime()
             ON MATCH SET 
                 n.checksum = $checksum,
                 n.structural_signature = $structural_signature,
                 n.imports = $imports,
                 n.updated_at = datetime()
             RETURN n.file_path as identifier
             """
         elif node_type == "CodeConfluenceFrameworkFeature":
             query = """
             MERGE (n:CodeConfluenceFrameworkFeature {qualified_name: $qualified_name})
             ON CREATE SET 
                 n.language = $language,
                 n.library = $library,
                 n.feature_key = $feature_key,
                 n.created_at = datetime(),
                 n.updated_at = datetime()
             ON MATCH SET 
                 n.updated_at = datetime()
             RETURN n.qualified_name as identifier
             """
         else:
             raise ValueError(f"Unsupported node type: {node_type}")
 
         result = await session.execute_write(lambda tx: tx.run(query, node_dict))
 
         record = await result.single()
-        return record["identifier"] if record else None
+        if not record:
+            raise RuntimeError(f"Failed to create or retrieve {node_type} node")
+        return record["identifier"]

[To ensure code accuracy, apply this suggestion manually]

Suggestion importance[1-10]: 7

__

Why: The suggestion correctly identifies a return type mismatch where the function is annotated to return str but could return None, which could lead to runtime errors.

Medium
General
Remove unnecessary WITH clause

The WITH clause is unnecessary and could impact performance. The DETACH DELETE
operation already handles relationship cleanup, and counting deleted nodes can
be done more efficiently.

unoplat-code-confluence-ingestion/code-confluence-flow-bridge/src/code_confluence_flow_bridge/processor/db/graph_db/code_confluence_graph_deletion.py [205-224]

 async def _delete_files_batch_txn(self, tx: AsyncManagedTransaction, file_paths: List[str]) -> Optional[Record]:
     """
     Transaction function to delete multiple files in batch
     Note: CodeConfluenceFile uses file_path as unique identifier, not qualified_name
     """
     if not file_paths:
         # Return empty result
         query = "RETURN 0 as count"
         result = await tx.run(query)
         return await result.single()
     
     query = """
     MATCH (f:CodeConfluenceFile)
     WHERE f.file_path IN $file_paths
-    WITH f, f.file_path as path
     DETACH DELETE f
-    RETURN count(path) as count
+    RETURN count(*) as count
     """
     result = await tx.run(query, {"file_paths": file_paths})
     return await result.single()
  • Apply / Chat
Suggestion importance[1-10]: 4

__

Why: The suggestion correctly identifies a redundant WITH clause in the Cypher query, and the proposed change simplifies the query and improves its conciseness without altering functionality.

Low
  • More

@JayGhiya
JayGhiya merged commit 9a93104 into main Aug 1, 2025
3 checks passed
JayGhiya added a commit that referenced this pull request Apr 8, 2026
feat(perf): perf and reliability improvements across neo4j/postgres
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant