Repository navigation
Conversation
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
ashie
force-pushed
the
in-tail-multithread
branch
from
October 10, 2026 13:34
d384790 to
1501747
Compare
…on order Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Member
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) this PR fixes:
None
What this PR does / why we need it:
(This PR depends on #5527 and includes its commits.)
This PR adds opt-in worker-thread parsing to
in_tail. Thedefault remains synchronous (
num_threads = 1), settingnum_threadsto 2 or more enables parsing on worker threadswhen the configured parser and hooks are supported.
Parsing uses worker-specific parser instances. Event emission
and position updates remain on the event loop, and processing
for each file stays serialized. Unsupported parser
configurations and overridden compatibility hooks continue
through the synchronous path. The worker pool is bounded and
handles asynchronous completion, queue saturation, rotation,
and shutdown.
The goal is to reduce delays to other files and watcher
callbacks while a large file is being processed. In an
exploratory local Test Driver measurement, while reading an
80 MiB JSONL file with a 1 MiB/s per-file read limit, median
probe-file latency was about 48 ms with synchronous parsing
and about 1 ms with four workers. The result was similar with
the StatWatcher enabled or disabled. This is a result for
that specific setup, not a general performance guarantee. If
all workers are occupied, tasks can still wait; this change
does not guarantee a latency bound or claim a general
throughput improvement.
Docs Changes:
Updated the
num_threadsconfiguration description.Release Note:
Add opt-in worker-thread parsing to
in_tailvianum_threads. The default remains synchronous.Assisted-by: LLM GPT-6.1 Sol (OpenAI)