Skip to content

Conversation

@gustavobtflores
Copy link
Contributor

@gustavobtflores gustavobtflores commented Dec 1, 2025

Objective

Enhance the ingestion performance testing suite to provide granular metrics and support parameter tuning.

Problem Solved

Existing performance tests were too high-level, making it difficult to identify specific bottlenecks or optimize configuration parameters like worker counts and batch sizes.

Key Changes

  • Added tests for specific pipeline stages: validation, file preparation, and database buffer flushing.
  • Implemented benchmarks for varying worker counts, batch sizes, and file subset sizes.

@gustavobtflores gustavobtflores self-assigned this Dec 2, 2025
@gustavobtflores gustavobtflores added the Backend Most or all of the changes for this issue will be in the backend code. label Dec 2, 2025
@gustavobtflores gustavobtflores force-pushed the feat/improved-ingester-performance-metrics branch from 5128ed3 to 06847ee Compare December 2, 2025 21:28
@gustavobtflores gustavobtflores marked this pull request as ready for review December 2, 2025 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backend Most or all of the changes for this issue will be in the backend code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant