LogPoller: process blocks in batches

[dhaidashenko]

Motivation

During prolonged downtime or RPC issues, Chainlink Node may lag behind Solana's latest finalized block. Max lag could be up to 172800 blocks.
LogPoller searches for blocks that contain events defined by Filters and schedules their fetching.
Block fetching is done by async workers. If the operation fails, the job is retried after a delay; it competes with the rest of the pool for a worker.
Fetched blocks are added to another queue, which sorts them and passes them to the events processor. It's guaranteed that the event processor will observe all blocks with subscribed events, in increasing order.
Thus, if we are scheduled to fetch 1k blocks and we fail to fetch the first block for a long time, all 999 blocks will sit in memory.

Our current assumption is that unreliable RPCs and large lag cause OOM for CL Nodes.
As we do not have access to the profile files for LOOP plugins (WIP), there is no guarantee that our assumption is correct.

Changes

• Process slots in batches. Batch size is defined in config. This way, we can limit the maximum number of blocks loaded into the memory.
• Add a metric to monitor the latest slot processed by LogPoller. Low processing rate or lag accumulation between the latest processed and the latest finalized slots can be used for early alerting on LP health.

core ref: 78e0a3fb76940b1aa5d700545e29925d5ad90610

[github-actions]

👋 dhaidashenko, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

⚠️ API Diff Results - Breaking changes detected

📦 Module: github-com-smartcontractkit-chainlink-solana

🔴 Breaking Changes (1)

pkg/solana/config.Config (1)

• LogPollerSlotsBatchSize — ➕ Added

---

📄 View full apidiff report

[github-actions]

⚠️ API Diff Results - github.com/smartcontractkit/chainlink-solana

⚠️ Breaking Changes (1)

pkg/solana/config.Config (1)

• LogPollerSlotsBatchSize — ➕ Added

✅ Compatible Changes (7)

pkg/solana/config.(*TOMLConfig) (1)

• LogPollerSlotsBatchSize — ➕ Added

pkg/solana/config.Chain (1)

• LogPollerSlotsBatchSize — ➕ Added

pkg/solana/config.TOMLConfig (1)

• LogPollerSlotsBatchSize — ➕ Added

pkg/solana/config/mocks (1)

• Config_LogPollerSlotsBatchSize_Call — ➕ Added

pkg/solana/config/mocks.(*Config) (1)

• LogPollerSlotsBatchSize — ➕ Added

pkg/solana/config/mocks.(*Config_Expecter) (1)

• LogPollerSlotsBatchSize — ➕ Added

pkg/solana/logpoller.(*solLpMetrics) (1)

• SetLatestProcessedSlot — ➕ Added

---

📄 View full apidiff report

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
52.4% Coverage on New Code (required ≥ 75%)

See analysis details on SonarQube