Skip to content

Commit 3114a80

Browse files
ingest: one ledger walk — ExtractLedgerTxParts + EventsFromTxParts/FeesFromTxParts (supersedes the extractor bundles) (#5966)
* ingest: add ExtractFees, a zero-copy view-based fee extractor ExtractFees(lcm xdr.LedgerCloseMetaView, passphrase) returns one ledger's fee observations (LedgerFees) split into classic per-op fees and Soroban inclusion fees — the buckets stellar-rpc's getFeeStats windows consume — from one TxProcessing walk plus by-hash envelope pairing, decoding nothing beyond FeeCharged, op count/type, and the Soroban resource-fee extension. The classification replicates stellar-rpc v1's FeeWindows.IngestFees bug-for-bug (soroban gate is the op type alone; soroban-shaped txs without SorobanTransactionMetaExtV1 are skipped, not classic; fee-bump reads inner ops with the outer FeeCharged; uint64 subtraction wraps when the charged resource fee exceeds FeeCharged). A verbatim test-only port of that walk is the primary oracle; the parsed LedgerTransaction fee helpers are the secondary oracle, cross-asserted where semantics align and pinned where they deliberately diverge. Part 1 of stellar/stellar-rpc#881. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: link the ExtractFees CHANGELOG entry to its PR Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: harden ExtractFees v1 parity and consolidate fee test fixtures Code-review follow-ups: - Replicate the parsed reader's pre-protocol-10 meta guard (badMetaVersionErr) so ExtractFees rejects the same outdated-stellar-core ledgers v1 does; adds xdr.LedgerCloseMetaView.ProtocolVersion and FeeProcessing on the TxProcessing projection to support it. - Hash the ENTIRE TxSet like v1's reader (no early break, hasher built on the first envelope): an invalid passphrase or malformed envelope anywhere in a non-empty TxSet errors exactly when it does on the parsed path, while an empty TxSet still skips passphrase validation. - Resolve each envelope's fee shape (op count + sole-op type) in the same pairing pass instead of computing the unused isSoroban flag and re-parsing the envelope afterwards. - Tests: reuse the production feeBucket enum, drop the duplicated wrap test, consolidate the LCM builders (shared header/TxProcessing/result helpers, explicit setLedgerVersion override), strip test-function docstrings, and add guard/passphrase parity tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: drop the secondary-oracle divergence tests for ExtractFees The parsed LedgerTransaction fee helpers computed overlapping quantities via independent routes (balance deltas, the protocol inclusion-fee equation), and a block of tests cross-asserted or pinned every agreement and disagreement. Confusing in practice, and the helpers have no production callers — the contract that matters is the one the remaining tests state directly: ExtractFees returns exactly what stellar-rpc v1's IngestFees returns, verified against the verbatim ported walk on every fixture and on a real pubnet ledger (where each transaction's fee is also re-derived from the raw parsed fields). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: drop the pre-protocol-10 meta guard from ExtractFees Fee stats only ever ingest ledgers at the tip (protocol 23+), so the parsed reader's stale-meta check for pre-2019 ledgers can never fire here. Removing it also removes the ProtocolVersion view helper and the FeeProcessing projection field it required — xdr/ledger_close_meta_view.go and ingest/ledger_close_meta_view_nav.go revert to untouched. The deliberate non-goal is noted in the ExtractFees godoc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: drop the ported-IngestFees test oracle for ExtractFees ExtractFees itself is the port of v1's walk; carrying a second copy of that walk in the test file to diff against added 100 lines without adding trust. The tests now assert v1's documented rules directly: hand-computed expected buckets for every fixture in the matrix, and a per-transaction re-derivation from the parsed fields on the real pubnet ledger. The benchmark measures the view path alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: reshape extractors — one ExtractLedgerTxParts walk, products as functions Per tamirms' review on #5966 (spec: stellar/stellar-rpc#912): walking the ledger and reading products off it are different jobs, and only the walk is worth sharing. ExtractLedgerTxParts owns the single TxProcessing walk and returns per-tx handles; EventsFromTxParts and FeesFromTxParts are plain functions of those handles. The ExtractTxHashes / ExtractLedgerEvents / ExtractFees bundles delete, along with the fee path's envelope pairing, whole-TxSet hashing, and passphrase parameter. Fee classification now reads TxProcessing alone: soroban iff SorobanMeta present, opCount = per-operation result count. Matches v1's IngestFees on every ledger a correctly functioning core produces; deliberate deltas: txINTERNAL_ERROR skipped, resource fee above FeeCharged errors instead of wrapping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: validate resource-fee components before summing in FeesFromTxParts A sum-only negativity check can be wrapped through: two huge negative components sum back to non-negative (two MinInt64s make exactly 0) and a negative component can hide behind a larger positive one. Check each component first; with both non-negative, a negative sum can only be int64 overflow, reported as such. Surfaced by Copilot's re-review (its one suppressed comment). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ingest: widen the never-ran-transaction docs; single-pass the V4 meta walker Per tamirms' review: the no-per-op-results family is not just txINTERNAL_ERROR. An account can invalidate its own pending transaction via an operation it signed inside another account's transaction (account merge, signer removal, sequence bump), and pre-protocol-20 tx sets allowed several transactions per account, so old ledgers carry such never-ran failures organically. The docs now say so in plain English with a worked example; classification behavior is unchanged. Adds the no_account_skipped matrix cell (the skip is a family rule, not a txINTERNAL_ERROR special case) and the LedgerTxParts disclosure that FeeBump derives from the result code — a never-ran fee-bump reads as a plain transaction with no inner hash. metaEventRaws' V4 arm now locates all fields in one Fields() pass instead of per-field prefix walks: DiagnosticEvents is the last field, so its accessor re-walked the whole meta interior once more per transaction on the read path (~10% on a synthetic V4 walker, growing with op count; the pubnet fixture is protocol 22/all-V3 where the arm never runs). V3 uses the locate pass only when both event sets are wanted; with one set the single accessor sizes strictly less. mustFields is the local Must twin that Fields() lacks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 3c3872f commit 3114a80

9 files changed

Lines changed: 1253 additions & 211 deletions

‎ingest/CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ All notable changes to this project will be documented in this file. This projec
1414

1515
### New Features
1616
* Add `ledgerbackend.LedgerStream`, a streaming-first, interchangeable ingestion source interface whose single `RawLedgers` method yields the raw XDR bytes of each ledger in a range. Constructors are provided for every backend: `NewBufferedStorageStream` (datastore), `NewCaptiveCoreStream` (captive stellar-core), and `NewRPCStream` (RPC). Each stream owns its backend's full lifecycle — set up on the first pull, torn down when iteration ends — and yields each ledger as a borrowed slice valid only until the next step. Pass `WithStreamMetrics(registry, namespace)` to `RawLedgers` to record `ledger_fetch_duration_seconds` (and, for captive-core, the `captive_stellar_core_*` suite), matching the metrics `WithMetrics` emits on the `GetLedger` path. [5944](https://github.com/stellar/go-stellar-sdk/pull/5944).
17-
* Add zero-copy XDR view extractors (**experimental** — signatures may still change) — the view-based twins of the parsed transaction path, reading directly off the generated `*View` accessors without `UnmarshalBinary`: `ExtractTxHashes` (per-ledger transaction hashes in apply order), `ExtractLedgerEvents` (per-transaction contract events + hash from one `TxProcessing` walk, as raw wire bytes), and the zero-copy `LedgerTransactionView` read path `LedgerTransactionViewByHash` / `LedgerTransactionViewRange` (the view parallel of `LedgerTransaction` / `LedgerTransactionReader`). `network.TransactionViewHasher` computes transaction hashes straight from envelope view bytes. Validated wire-identical against the parsed path across LCM V0/V1/V2 and `TransactionMeta` V0–V4, plus real-ledger equivalence tests and benchmarks against the eager-decode path on `ledger_58752000`. [5949](https://github.com/stellar/go-stellar-sdk/pull/5949).
17+
* Add `ExtractLedgerTxParts` + per-product functions (**experimental** — signatures may still change), the zero-copy per-ledger extraction API: one function owns the single `TxProcessing` walk and returns per-transaction handles (`LedgerTxParts`: hash, inner hash, fee-bump flag, lazy `Result`/`Meta` views), and each product is a plain function of those handles — `EventsFromTxParts` (contract events as raw wire bytes, `TxEvents`) and `FeesFromTxParts` (classic per-op and Soroban inclusion fee buckets, `LedgerFees` — what stellar-rpc's `getFeeStats` windows consume). Consumers compose exactly the products they need from exactly one walk. The fee classification reads only `TxProcessing` (soroban ⟺ `SorobanMeta` present; op count = per-operation result count), so it needs no `TxSet` read and no network passphrase; it matches stellar-rpc v1's `FeeWindows.IngestFees` on organic current-protocol traffic (deliberate deltas: transactions whose operations never ran — `txINTERNAL_ERROR`, or invalidated by an earlier transaction in the same ledger — are skipped since their results carry no per-operation list, and a charged resource fee above `FeeCharged` is an error instead of a uint64 wrap). This supersedes the earlier `ExtractTxHashes` / `ExtractLedgerEvents` bundles from [5949](https://github.com/stellar/go-stellar-sdk/pull/5949), which are removed (unreleased). Covered by table-driven fixtures across LCM V0/V1/V2 × `TransactionMeta` V0–V4 × result shapes and by real-pubnet-ledger equivalence tests; on `ledger_58752000` (249 txs) the walk measures ~0.29ms/17 allocs and all three products together ~0.58ms/154 allocs, vs ~5.7ms/~95k allocs for the parsed path. [5966](https://github.com/stellar/go-stellar-sdk/pull/5966), design: [stellar-rpc#912](https://github.com/stellar/stellar-rpc/issues/912).
18+
* Add the zero-copy `LedgerTransactionView` read path (**experimental** — signatures may still change), reading directly off the generated `*View` accessors without `UnmarshalBinary`: `LedgerTransactionViewByHash` / `LedgerTransactionViewRange` (the view parallel of `LedgerTransaction` / `LedgerTransactionReader`; these pair TxSet envelopes by hash and take the network passphrase). `network.TransactionViewHasher` computes transaction hashes straight from envelope view bytes. Validated wire-identical against the parsed path across LCM V0/V1/V2 and `TransactionMeta` V0–V4, plus real-ledger equivalence tests and benchmarks against the eager-decode path on `ledger_58752000`. [5949](https://github.com/stellar/go-stellar-sdk/pull/5949).
1819

1920

2021
## v23.0.0

‎ingest/extract.go‎

Lines changed: 315 additions & 50 deletions
Large diffs are not rendered by default.

‎ingest/extract_bench_test.go‎

Lines changed: 59 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@ import (
99
"github.com/stellar/go-stellar-sdk/xdr"
1010
)
1111

12-
// Benchmarks for the four public view extractors on a real pubnet ledger,
13-
// each with the eager-decode reference path as the A/B baseline. These
12+
// Benchmarks for the public view extractors on a real pubnet ledger, with
13+
// the eager-decode reference path as the A/B baseline where one exists. These
1414
// supersede the hand-rolled prototypes that lived in xdr
1515
// (extract_tx_bench_test.go, event_extraction_bench_test.go,
1616
// selective_decode_bench_test.go) — same fixture, but measuring the
@@ -40,7 +40,10 @@ func benchRefTransactions(b *testing.B, raw []byte) []ingest.LedgerTransaction {
4040
}
4141
}
4242

43-
func BenchmarkExtractTxHashes(b *testing.B) {
43+
// BenchmarkExtractLedgerTxParts measures the ONE TxProcessing walk every
44+
// per-ledger product reads from, against the full decode that yields the
45+
// same per-tx handles (hash + result + meta) on the parsed path.
46+
func BenchmarkExtractLedgerTxParts(b *testing.B) {
4447
raw := loadRealLedger(b)
4548

4649
b.Run("full_decode", func(b *testing.B) {
@@ -57,16 +60,18 @@ func BenchmarkExtractTxHashes(b *testing.B) {
5760
b.Run("view", func(b *testing.B) {
5861
b.ReportAllocs()
5962
for i := 0; i < b.N; i++ {
60-
hashes, err := ingest.ExtractTxHashes(xdr.LedgerCloseMetaView(raw))
63+
txParts, err := ingest.ExtractLedgerTxParts(xdr.LedgerCloseMetaView(raw))
6164
if err != nil {
6265
b.Fatal(err)
6366
}
64-
_ = hashes
67+
_ = txParts
6568
}
6669
})
6770
}
6871

69-
func BenchmarkExtractLedgerEvents(b *testing.B) {
72+
// BenchmarkEventsFromTxParts measures walk + events — the cold-backfill
73+
// composition (fees never computed).
74+
func BenchmarkEventsFromTxParts(b *testing.B) {
7075
raw := loadRealLedger(b)
7176

7277
b.Run("full_decode", func(b *testing.B) {
@@ -85,15 +90,61 @@ func BenchmarkExtractLedgerEvents(b *testing.B) {
8590
b.Run("view", func(b *testing.B) {
8691
b.ReportAllocs()
8792
for i := 0; i < b.N; i++ {
88-
events, err := ingest.ExtractLedgerEvents(xdr.LedgerCloseMetaView(raw))
93+
txParts, err := ingest.ExtractLedgerTxParts(xdr.LedgerCloseMetaView(raw))
8994
if err != nil {
9095
b.Fatal(err)
9196
}
92-
_ = events
97+
txEvents, err := ingest.EventsFromTxParts(txParts)
98+
if err != nil {
99+
b.Fatal(err)
100+
}
101+
_ = txEvents
93102
}
94103
})
95104
}
96105

106+
// BenchmarkFeesFromTxParts measures walk + fees — the fee-window replay
107+
// composition (events never computed).
108+
func BenchmarkFeesFromTxParts(b *testing.B) {
109+
raw := loadRealLedger(b)
110+
b.ReportAllocs()
111+
for i := 0; i < b.N; i++ {
112+
txParts, err := ingest.ExtractLedgerTxParts(xdr.LedgerCloseMetaView(raw))
113+
if err != nil {
114+
b.Fatal(err)
115+
}
116+
fees, err := ingest.FeesFromTxParts(txParts)
117+
if err != nil {
118+
b.Fatal(err)
119+
}
120+
_ = fees
121+
}
122+
}
123+
124+
// BenchmarkLedgerTxAllProducts measures walk + events + fees — the hot
125+
// ingestion composition, and the whole point of the parts/products split:
126+
// adding the fee product to an events consumer costs a read over the already
127+
// located handles, not a second walk.
128+
func BenchmarkLedgerTxAllProducts(b *testing.B) {
129+
raw := loadRealLedger(b)
130+
b.ReportAllocs()
131+
for i := 0; i < b.N; i++ {
132+
txParts, err := ingest.ExtractLedgerTxParts(xdr.LedgerCloseMetaView(raw))
133+
if err != nil {
134+
b.Fatal(err)
135+
}
136+
txEvents, err := ingest.EventsFromTxParts(txParts)
137+
if err != nil {
138+
b.Fatal(err)
139+
}
140+
fees, err := ingest.FeesFromTxParts(txParts)
141+
if err != nil {
142+
b.Fatal(err)
143+
}
144+
_, _ = txEvents, fees
145+
}
146+
}
147+
97148
func BenchmarkLedgerTransactionViewRange(b *testing.B) {
98149
raw := loadRealLedger(b)
99150

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
package ingest
2+
3+
import (
4+
"fmt"
5+
"testing"
6+
7+
"github.com/stretchr/testify/assert"
8+
"github.com/stretchr/testify/require"
9+
10+
"github.com/stellar/go-stellar-sdk/xdr"
11+
)
12+
13+
// buildMultiPhaseLCM builds an LCM (V1 or V2 — both carry a
14+
// GeneralizedTransactionSet) whose TxSet holds BOTH phase shapes in one
15+
// ledger: txs[:split] sit in a V=0 components phase and txs[split:] in a V=1
16+
// parallel-txs phase (one stage, one cluster per tx). Envelopes are listed in
17+
// reverse within each phase, so nothing about the TxSet ordering matches the
18+
// TxProcessing apply order — pairing is by hash.
19+
func buildMultiPhaseLCM(t testing.TB, version int32, ledgerSeq uint32, closeTime int64, txs []txWithHash, split int) xdr.LedgerCloseMeta {
20+
t.Helper()
21+
require.Less(t, split, len(txs))
22+
23+
v0Envs := make([]xdr.TransactionEnvelope, split)
24+
for i, tx := range txs[:split] {
25+
v0Envs[split-1-i] = tx.env
26+
}
27+
comp := []xdr.TxSetComponent{{
28+
Type: xdr.TxSetComponentTypeTxsetCompTxsMaybeDiscountedFee,
29+
TxsMaybeDiscountedFee: &xdr.TxSetComponentTxsMaybeDiscountedFee{Txs: v0Envs},
30+
}}
31+
32+
parallelTxs := txs[split:]
33+
clusters := make(xdr.ParallelTxExecutionStage, 0, len(parallelTxs))
34+
for i := range parallelTxs {
35+
clusters = append(clusters, xdr.DependentTxCluster{parallelTxs[len(parallelTxs)-1-i].env})
36+
}
37+
38+
phases := []xdr.TransactionPhase{
39+
{V: 0, V0Components: &comp},
40+
{V: 1, ParallelTxsComponent: &xdr.ParallelTxsComponent{
41+
ExecutionStages: []xdr.ParallelTxExecutionStage{clusters},
42+
}},
43+
}
44+
45+
header := fixtureHeader(ledgerSeq, closeTime)
46+
txSet := xdr.GeneralizedTransactionSet{V: 1, V1TxSet: &xdr.TransactionSetV1{Phases: phases}}
47+
48+
switch version {
49+
case 1:
50+
return xdr.LedgerCloseMeta{V: 1, V1: &xdr.LedgerCloseMetaV1{
51+
LedgerHeader: header, TxSet: txSet, TxProcessing: resultMetas(txs),
52+
}}
53+
case 2:
54+
return xdr.LedgerCloseMeta{V: 2, V2: &xdr.LedgerCloseMetaV2{
55+
LedgerHeader: header, TxSet: txSet, TxProcessing: resultMetaV1s(txs),
56+
}}
57+
default:
58+
t.Fatalf("unsupported version %d", version)
59+
return xdr.LedgerCloseMeta{}
60+
}
61+
}
62+
63+
func TestFeesFromTxParts_MultiPhaseTxSet(t *testing.T) {
64+
invoke := feeInvokeHostFunctionOp()
65+
bump := feeBumpSequenceOp()
66+
txs := []txWithHash{
67+
// V0-components phase:
68+
feeTxV1(t, []xdr.Operation{bump, bump}, false, feeMetaV1(), 101), // classic 50
69+
feeTxV1(t, []xdr.Operation{invoke}, true, feeMetaV3Ext1(40, 10), 100), // soroban 50
70+
feeTxV1(t, []xdr.Operation{invoke}, true, feeMetaV3Ext0(), 100), // skipped
71+
// V1 parallel phase:
72+
feeBumpFeeTx(t, []xdr.Operation{invoke}, true, feeMetaV4Ext1(30, 20), 300), // soroban 250
73+
feeTxV1(t, []xdr.Operation{bump}, false, feeMetaV2(), 100), // classic 100
74+
feeTxV0(t, []xdr.Operation{bump, bump, bump}, feeMetaV0(), 100), // classic 33
75+
}
76+
for _, version := range []int32{1, 2} {
77+
t.Run(fmt.Sprintf("lcmV%d", version), func(t *testing.T) {
78+
lcm := buildMultiPhaseLCM(t, version, 8952, 1_700_086_000, txs, 3)
79+
got := feesFromLCM(t, lcm)
80+
assert.Equal(t, []uint64{50, 100, 33}, got.ClassicFeesPerOp)
81+
assert.Equal(t, []uint64{50, 250}, got.SorobanInclusionFees)
82+
})
83+
}
84+
}

0 commit comments

Comments
 (0)