Skip to content

in_tail: Add opt-in worker-thread parsing - #5529

Draft
ashie wants to merge 14 commits into
masterfrom
in-tail-multithread
Draft

ashie wants to merge 14 commits into
masterfrom
in-tail-multithread

Conversation

@ashie

@ashie ashie commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Which issue(s) this PR fixes:
None

What this PR does / why we need it:

(This PR depends on #5527 and includes its commits.)

This PR adds opt-in worker-thread parsing to in_tail. The
default remains synchronous (num_threads = 1), setting
num_threads to 2 or more enables parsing on worker threads
when the configured parser and hooks are supported.

Parsing uses worker-specific parser instances. Event emission
and position updates remain on the event loop, and processing
for each file stays serialized. Unsupported parser
configurations and overridden compatibility hooks continue
through the synchronous path. The worker pool is bounded and
handles asynchronous completion, queue saturation, rotation,
and shutdown.

The goal is to reduce delays to other files and watcher
callbacks while a large file is being processed. In an
exploratory local Test Driver measurement, while reading an
80 MiB JSONL file with a 1 MiB/s per-file read limit, median
probe-file latency was about 48 ms with synchronous parsing
and about 1 ms with four workers. The result was similar with
the StatWatcher enabled or disabled. This is a result for
that specific setup, not a general performance guarantee. If
all workers are occupied, tasks can still wait; this change
does not guarantee a latency bound or claim a general
throughput improvement.

Docs Changes:
Updated the num_threads configuration description.

Release Note:
Add opt-in worker-thread parsing to in_tail via
num_threads. The default remains synchronous.

Assisted-by: LLM GPT-6.1 Sol (OpenAI)

ashie added 13 commits October 9, 2026 15:56
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie
ashie force-pushed the in-tail-multithread branch from d384790 to 1501747 Compare October 10, 2026 13:34
…on order

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie

ashie commented Oct 10, 2026

Copy link
Copy Markdown
Member Author
test/plugin/in_tail/test_worker_pool.rb:117:15: C: [Correctable] Performance/TimesMap: Use Array.new(2) with a block instead of .times.map.
    results = 2.times.map { result_from(pool) }
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
test/plugin/in_tail/test_worker_pool.rb:210:15: C: [Correctable] Performance/TimesMap: Use Array.new(3) with a block instead of .times.map.
    results = 3.times.map { result_from(pool) }
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

470 files inspected, 2 offenses detected, 2 offenses autocorrectable
Error: Process completed with exit code 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant