feat: add spkg.go for handling Substreams package resolution and loading#116
Open
maoueh wants to merge 2 commits into
Open
feat: add spkg.go for handling Substreams package resolution and loading#116maoueh wants to merge 2 commits into
maoueh wants to merge 2 commits into
Conversation
Contributor
maoueh
commented
May 4, 2026
maoueh
commented
- Implemented ResolveAndLoadSpkg function to check for existing spkg variants and load them.
- Added LoadSpkgFromURL function to read and parse spkg files, with validation options for older formats.
- Created FindModuleInPackage function to locate modules by name, accommodating prefixed names.
- Introduced SpkgPackageLabel function to generate a "name@version" label for packages.
- Implemented ResolveAndLoadSpkg function to check for existing spkg variants and load them. - Added LoadSpkgFromURL function to read and parse spkg files, with validation options for older formats. - Created FindModuleInPackage function to locate modules by name, accommodating prefixed names. - Introduced SpkgPackageLabel function to generate a "name@version" label for packages.
There was a problem hiding this comment.
Pull request overview
Adds shared utilities and a new CLI subcommand to make it easier to resolve Substreams packages and inspect (or replay) Substreams requests from GCP Cloud Logging.
Changes:
- Added
spkg.gohelpers to resolve/load.spkgvariants, locate output modules, and format package labels. - Introduced
tools substreams logs connection <trace-id>to print request details, stats, optional spkg URL/package label, and inferred endpoint. - Refactored log querying to support TraceID-based searching and expanded parsed log fields; extracted endpoint + date-range parsing into shared helpers.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/tools/substreams/spkg.go | New helpers for resolving/loading spkg files and locating modules within packages. |
| cmd/tools/substreams/logs/types.go | Extends log query options with TraceID and documents intended usage. |
| cmd/tools/substreams/logs/gcp_backend.go | Adds TraceID-based filter mode and parses additional jsonPayload fields into LogEntry. |
| cmd/tools/substreams/logs/backend.go | Extends LogEntry with additional request fields (session, cursor, mode flags). |
| cmd/tools/substreams/logs.go | Registers the new logs connection subcommand. |
| cmd/tools/substreams/logs_reexec.go | Refactors reexec log querying to use the shared logs backend + shared helpers. |
| cmd/tools/substreams/logs_connection.go | New logs connection command implementation and selection logic for request/stats entries. |
| cmd/tools/substreams/logs_connection_test.go | Tests for the new logs connection command and helpers. |
| cmd/tools/substreams/endpoint.go | New shared endpoint resolution/inference helper extracted from reexec. |
| cmd/tools/substreams/daterange.go | New shared date-range parsing helper extracted from reexec. |
| cmd/tools/substreams/daterange_test.go | Unit tests for the new date-range parsing helper. |
| CHANGELOG.md | Documents the new tools substreams logs connection command. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+85
to
+89
| var subjectFilter string | ||
| if opts.TraceID != "" { | ||
| subjectFilter = fmt.Sprintf(`SEARCH("%s")`, opts.TraceID) | ||
| } else { | ||
| subjectFilter = fmt.Sprintf(`jsonPayload.user_id="%s"`, opts.UserID) |
Comment on lines
+180
to
+184
| if request == nil || entry.Timestamp < request.Timestamp { | ||
| request = entry | ||
| } | ||
| case entry.IsRequestStats(): | ||
| if stats == nil || entry.Timestamp > stats.Timestamp { |
Comment on lines
+312
to
+316
| // queryIncomingRequestByTraceID searches GCP logs for the incoming Substreams | ||
| // request matching the given trace ID. Returns the first match (oldest) and | ||
| // the total count of matches found. | ||
| func queryIncomingRequestByTraceID(ctx context.Context, gcpProject, traceID string, startTime, endTime time.Time, logger *zap.Logger) (*reexecRequest, int, error) { | ||
| backend, err := logs.NewGCPBackend(ctx, gcpProject, logger) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.