Jupiter-Swap-Postgres-Pipeline Example + RpcBlockCrawler & RpcTransactionCrawler Updates #469
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
logs are saved now in examples/jupiter-swap-postgres/logs
custom structured processor, polished table schema and postgres integration for jupiter swap events - Replaced the Jupiter swap example’s JSON processor with a custom structured processor, wired into the entrypoint, and pulled in the dependencies needed for the richer Postgres integration. - Added dedicated migrations and repository logic to create structured route, hop, venue, mint, and envelope tables while providing transactional upserts and normalization tracking for swap data. - Defined strongly typed models plus normalization helpers for route plans and swap events, and implemented a processor that maps every Jupiter route variant and event into the new schema with venue lookup and metrics support.
added rate limiter to work with free tier RPCs and fixed the INSERT error with postgres - Added a shared RateLimiter to ConnectionConfig so both the signature poller and transaction fetchers throttle RPC calls; the limiter is configurable via .with_rate_limit() and guards each get_signatures_for_address/get_transaction call (datasources/rpc-transaction-crawler-datasource/src/lib.rs:98-152,266-309,395-405). - The Jupiter example now builds its crawler with ConnectionConfig::default().with_rate_limit(20) and the new RPC_URL environment variable, so every run respects Alchemy’s 25 req/s ceiling while keeping the existing decoder/processor stack unchanged (examples/jupiter-swap-postgres/src/main.rs (lines 86-107)). - Fixed the swap-hop insert to bind the metadata column explicitly, eliminating the “INSERT has more target columns than expressions” error that kept jupiter_swap_hops empty (examples/jupiter-swap-postgres/src/db.rs (lines 524-534)).
examples/jupiter-swap-postgres/src/main.rs (lines 88-105) now reads RATE_LIMIT (default 20 req/s), logs the chosen value, and applies it via ConnectionConfig::with_rate_limit. Dropping the var (or setting <=0) keeps the previous default.
- if start_slot is left empty it defaults to tailing the head - updated example/block-crawler & example/jupiter-swap-postgres to reflect data source changes - modified log-metrics to log the last procesed slot
Updated... datasources/rpc-block-crawler-datasource/src/lib.rs (lines 185-247) so the fetcher now loops inside each slot request until get_block_with_config actually returns a block. Skippable RPC errors (-32004/-32007/-32009) no longer mark the slot as permanently skipped; we log at debug level, sleep for the configured block_interval, and retry the same slot before proceeding. Successful fetches still record latency/throughput metrics exactly as before, while non-retryable errors continue to be logged and cause the slot to be dropped. Because each future only completes once a block is delivered, the crawler no longer advances past a slot when it’s not ready, eliminating the head-of-chain spin
examples/jupiter-swap-postgres logs now capture... last_processed_slot right before it enters postgres - since capturing last_processed_slot before it enters postgres metrics/log-metrics/src/lib.rs has been reverted it's original state - rate_limit has also been implemented for RpcBlockCrawler since Alchemy showed 429 errors - additional fix to datasources/rpc-block-crawler-datasource/src/lib.rs skipping blocks
TheDeFiQuant
changed the title
Author
|
The Jupiter-Swap-Postgres-Pipeline example shows an entire pipeline, using RpcTransactionCrawler and RpcBlockCrawler as data source, indexing & decoding swap events from Jupiter and persists a structured view of each swap and its route metadata into Postgres. Additionally, rate_limit functionality has been added to both, RpcTransactionCrawler and RpcBlockCrawler, as well as a tailing setting for RpcBlockCrawler by default. |
unbound26z
reviewed
Nov 27, 2025
Contributor
unbound26z
left a comment
unbound26z
left a comment
There was a problem hiding this comment.
Looks good, very in-depth. Great docs.
Let's get CI sorted out
Author
|
Thanks for the review — I’m traveling this week and will fix the CI issues next week @unbound26z |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.