Repository navigation
Conversation
also try to add client details if we can cast it as an rpc client
Previously, blockpoller was polling blocks one by one due to the lock on the clients object (tracking failures and moving to the next rpc client) We now duplicate the client list for every block poll, enabling us to poll multiple blocks in parallel from multiple RPCs. Performance impact is negligible as the main bottleneck here is the RPC call - launching multiple calls in parallel outweighs the penalty of duplicating our clients list. We also try to start with a different rpc for each block poll to avoid spamming a single endpoint with too many requests at once
billettc
approved these changes
Sep 2, 2026
Adds test coverage for parallel block polling and fixes the defects that coverage exposed. Before this the branch did not compile and the existing blockpoller tests failed, including without the race detector. Build: rpc/clients.go imported github.com/streamingfast/eth-go/rpc, which was never added to go.mod. The dependency is now declared, pinned to 00c24bc, the last commit before eth-go bumped itself to Go 1.27, so firehose-core stays on go 1.25.0. Fixes: * Duplicated client lists shared one rolling strategy. DuplicateAndStartAt copied the strategy pointer while giving each duplicate its own mutex, so concurrent polls mutated the same rolling position unguarded: a data race, and a source of spurious "no more clients" errors. Each duplicate now clones the strategy. The rotation was also inverted relative to the method name, so DuplicateAndStartAt(n) now actually starts at index n, and the source list is read under lock. * Two batches could start at once, because the "already fetching" flag was set inside the spawned goroutine. It is now an atomic.Bool claimed by compare-and-swap before spawning. * Stale blocks survived a reorg: a batch already in flight would write blocks from the abandoned chain back into the optimistic cache after it had been cleared. Batches now carry an epoch and drop their results if the cache was reset underneath them. * A failed optimistic fetch killed the poller. Reading ahead is best effort, so those failures are now logged and swallowed. Failures on the demand path stay fatal, since the run loop is blocked on them. * Read-ahead ignored WithDelayBetweenFetch. It is now restricted to batched polling: with a batch size of 1 the poller is explicitly one block at a time, and that is the only mode where the delay applies. * A batch size of 0 or negative panicked on the blockToFetch % batchSize used to pick a starting client. It is clamped to 1. Tests cover rotation semantics, independence between duplicates, concurrent use, fetch concurrency for batched versus sequential polling, in-order firing of out-of-order fetches, endpoint spreading, single-flight batching, reorg cache invalidation and batch-size clamping. WithDelayBetweenFetch is verified under testing/synctest so the rate limit is asserted exactly on a virtual clock. Each fix was mutation checked: reverting it fails at least one test.
billettc
approved these changes
Sep 8, 2026
Contributor
|
It seems most of this was already done via 1a2066d differently. I'm working on a follow up PR for polling for multiple providers when doing backfill, this will be open as another PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR improves the block poller by enabling it to poll multiple blocks in parallel, thus greatly improving throughput.
It also enhances logging to report RPC call failures, which were previously silently failing