Optimize get_timeseries_metadata() to eliminate N+1 I/O (~94x speedup)

[JackieTien97]

Summary

Optimize TsFile C++ read path in two areas:

1. Optimize get_timeseries_metadata() — eliminate N+1 I/O (~94x speedup for init)

Root Causes:

1. N+1 I/O: get_all_device_ids() traverses the entire device index tree but discards offset information. Then for each device, load_device_index_entry() re-searches the tree from the root — O(N×D) redundant disk reads for N devices and tree depth D.
2. Duplicate read: get_device_timeseries_meta_without_chunk_meta() reads the measurement index node once to check alignment, then load_all_measurement_index_entry() reads the exact same byte range again.
3. Redundant PageArena::init(): Called per-device in a loop despite already being initialized in the constructor.

Changes:

• Add TsFileIOReader::get_device_timeseries_meta_by_offset() — accepts pre-resolved offsets (skips load_device_index_entry), reuses deserialized top node for get_all_leaf() (eliminates duplicate read)
• Add TsFileReader::get_all_device_entries() — single tree traversal collecting device IDs with their (start_offset, end_offset)
• Rewrite get_timeseries_metadata() to combine the above
• Remove redundant PageArena::init() call from get_timeseries_metadata_impl()

Benchmark (ecg_dataset/part_0.tsfile, 634 MB, 53040 devices):

Path	Avg Time	Speedup
Before (get_all_device_ids + get_timeseries_metadata(ids))	~28.8 s	—
After (get_timeseries_metadata() optimized)	~0.30 s	~94x

2. Optimize queryByRow with exact tag filter — B-tree direct lookup (~230x speedup per query)

Root Cause:
When queryByRow is called with a tag filter (e.g. exact match on one device), DeviceMetaIterator still traverses the entire MetaIndexNode tree: reading all 208 internal nodes from disk and scanning all 53040 leaf device entries to check the filter. This takes ~117ms per query even though only 1 device matches.

Changes:

• DeviceMetaIterator::try_setup_direct_lookup() — detects single TagEq filter at construction time and constructs the target device ID
• DeviceMetaIterator::load_results_direct() — uses load_device_index_entry() for O(log N) B-tree binary search instead of full tree traversal
• Move load_device_index_entry() from private to public in TsFileIOReader for reuse
• Add bench_read.cpp for standalone C++ read path benchmarking

Benchmark (ecg_dataset/part_0.tsfile, 53040 devices, offset=1000, limit=3584):

Component	Before	After	Speedup
DeviceTaskIterator::next()	117.4 ms	0.5 ms	~230x
first next() (full lazy init)	128 ms	1.8 ms	~71x
Total queryByRow	128 ms	1.9 ms	~67x

End-to-end Python benchmark (training sample read, ecg_dataset):

Metric	Before	After	Speedup
Average latency	236.8 ms	5.74 ms	41x
Throughput	4.32 windows/sec	174.1 windows/sec	40x

Note: The single-TagEq direct lookup covers the common training data access pattern. Multi-tag AND filters fall back to the existing full-scan path.

Test plan

• All 522 existing tests pass
• Verified result correctness: both paths return identical device count (53040)
• C++ bench_read validates queryByRow returns correct row count (3584)
• Python end-to-end benchmark validates data correctness with training pipeline

[Copilot]

Pull request overview

This PR optimizes the “collect metadata for all devices” path by avoiding repeated device-index traversals and redundant disk reads when building the full device→timeseries-metadata map.

Changes:

• Adds an offset-based metadata loader (TsFileIOReader::get_device_timeseries_meta_by_offset) to skip per-device index-tree searches.
• Adds a single-pass device index traversal that collects (device_id, start_offset, end_offset) entries for all devices.
• Rewrites TsFileReader::get_timeseries_metadata() to use the above and removes a redundant PageArena::init() call.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
cpp/src/reader/tsfile_reader.cc	Adds device-entry collection by offset and rewrites the “all devices metadata” path to use it.
cpp/src/file/tsfile_io_reader.h	Declares new offset-based device metadata API.
cpp/src/file/tsfile_io_reader.cc	Implements new offset-based device metadata loader to reduce redundant reads/traversals.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JackieTien97]

Code Review

Great optimization — the 94x speedup from eliminating the N+1 I/O pattern is impressive. The approach of collecting device entries with offsets in a single traversal and then reusing the deserialized top node is well-designed.

One critical bug found: integer truncation of 64-bit file offsets in get_all_device_entries. See inline comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.00000% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.17%. Comparing base (f7041d8) to head (4b34bf5).

Files with missing lines	Patch %	Lines
cpp/src/reader/tsfile_reader.cc	41.66%	16 Missing and 5 partials ⚠️
cpp/src/reader/device_meta_iterator.cc	74.28%	1 Missing and 8 partials ⚠️
cpp/src/file/tsfile_io_reader.cc	66.66%	0 Missing and 8 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #815      +/-   ##
===========================================
- Coverage    63.18%   63.17%   -0.01%     
===========================================
  Files          717      717              
  Lines        43783    43875      +92     
  Branches      6526     6562      +36     
===========================================
+ Hits         27664    27720      +56     
- Misses       15127    15145      +18     
- Partials       992     1010      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.