Repository navigation
Commit 3114a80
ingest: one ledger walk — ExtractLedgerTxParts + EventsFromTxParts/FeesFromTxParts (supersedes the extractor bundles) (#5966)
* ingest: add ExtractFees, a zero-copy view-based fee extractor
ExtractFees(lcm xdr.LedgerCloseMetaView, passphrase) returns one ledger's
fee observations (LedgerFees) split into classic per-op fees and Soroban
inclusion fees — the buckets stellar-rpc's getFeeStats windows consume —
from one TxProcessing walk plus by-hash envelope pairing, decoding nothing
beyond FeeCharged, op count/type, and the Soroban resource-fee extension.
The classification replicates stellar-rpc v1's FeeWindows.IngestFees
bug-for-bug (soroban gate is the op type alone; soroban-shaped txs without
SorobanTransactionMetaExtV1 are skipped, not classic; fee-bump reads inner
ops with the outer FeeCharged; uint64 subtraction wraps when the charged
resource fee exceeds FeeCharged). A verbatim test-only port of that walk is
the primary oracle; the parsed LedgerTransaction fee helpers are the
secondary oracle, cross-asserted where semantics align and pinned where
they deliberately diverge.
Part 1 of stellar/stellar-rpc#881.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: link the ExtractFees CHANGELOG entry to its PR
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: harden ExtractFees v1 parity and consolidate fee test fixtures
Code-review follow-ups:
- Replicate the parsed reader's pre-protocol-10 meta guard (badMetaVersionErr)
so ExtractFees rejects the same outdated-stellar-core ledgers v1 does; adds
xdr.LedgerCloseMetaView.ProtocolVersion and FeeProcessing on the
TxProcessing projection to support it.
- Hash the ENTIRE TxSet like v1's reader (no early break, hasher built on the
first envelope): an invalid passphrase or malformed envelope anywhere in a
non-empty TxSet errors exactly when it does on the parsed path, while an
empty TxSet still skips passphrase validation.
- Resolve each envelope's fee shape (op count + sole-op type) in the same
pairing pass instead of computing the unused isSoroban flag and re-parsing
the envelope afterwards.
- Tests: reuse the production feeBucket enum, drop the duplicated wrap test,
consolidate the LCM builders (shared header/TxProcessing/result helpers,
explicit setLedgerVersion override), strip test-function docstrings, and add
guard/passphrase parity tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: drop the secondary-oracle divergence tests for ExtractFees
The parsed LedgerTransaction fee helpers computed overlapping quantities via
independent routes (balance deltas, the protocol inclusion-fee equation), and
a block of tests cross-asserted or pinned every agreement and disagreement.
Confusing in practice, and the helpers have no production callers — the
contract that matters is the one the remaining tests state directly:
ExtractFees returns exactly what stellar-rpc v1's IngestFees returns, verified
against the verbatim ported walk on every fixture and on a real pubnet ledger
(where each transaction's fee is also re-derived from the raw parsed fields).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: drop the pre-protocol-10 meta guard from ExtractFees
Fee stats only ever ingest ledgers at the tip (protocol 23+), so the parsed
reader's stale-meta check for pre-2019 ledgers can never fire here. Removing
it also removes the ProtocolVersion view helper and the FeeProcessing
projection field it required — xdr/ledger_close_meta_view.go and
ingest/ledger_close_meta_view_nav.go revert to untouched. The deliberate
non-goal is noted in the ExtractFees godoc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: drop the ported-IngestFees test oracle for ExtractFees
ExtractFees itself is the port of v1's walk; carrying a second copy of that
walk in the test file to diff against added 100 lines without adding trust.
The tests now assert v1's documented rules directly: hand-computed expected
buckets for every fixture in the matrix, and a per-transaction re-derivation
from the parsed fields on the real pubnet ledger. The benchmark measures the
view path alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: reshape extractors — one ExtractLedgerTxParts walk, products as functions
Per tamirms' review on #5966 (spec: stellar/stellar-rpc#912): walking the
ledger and reading products off it are different jobs, and only the walk
is worth sharing. ExtractLedgerTxParts owns the single TxProcessing walk
and returns per-tx handles; EventsFromTxParts and FeesFromTxParts are
plain functions of those handles. The ExtractTxHashes /
ExtractLedgerEvents / ExtractFees bundles delete, along with the fee
path's envelope pairing, whole-TxSet hashing, and passphrase parameter.
Fee classification now reads TxProcessing alone: soroban iff SorobanMeta
present, opCount = per-operation result count. Matches v1's IngestFees on
every ledger a correctly functioning core produces; deliberate deltas:
txINTERNAL_ERROR skipped, resource fee above FeeCharged errors instead
of wrapping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: validate resource-fee components before summing in FeesFromTxParts
A sum-only negativity check can be wrapped through: two huge negative
components sum back to non-negative (two MinInt64s make exactly 0) and a
negative component can hide behind a larger positive one. Check each
component first; with both non-negative, a negative sum can only be int64
overflow, reported as such.
Surfaced by Copilot's re-review (its one suppressed comment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ingest: widen the never-ran-transaction docs; single-pass the V4 meta walker
Per tamirms' review: the no-per-op-results family is not just
txINTERNAL_ERROR. An account can invalidate its own pending transaction
via an operation it signed inside another account's transaction (account
merge, signer removal, sequence bump), and pre-protocol-20 tx sets
allowed several transactions per account, so old ledgers carry such
never-ran failures organically. The docs now say so in plain English
with a worked example; classification behavior is unchanged. Adds the
no_account_skipped matrix cell (the skip is a family rule, not a
txINTERNAL_ERROR special case) and the LedgerTxParts disclosure that
FeeBump derives from the result code — a never-ran fee-bump reads as a
plain transaction with no inner hash.
metaEventRaws' V4 arm now locates all fields in one Fields() pass
instead of per-field prefix walks: DiagnosticEvents is the last field,
so its accessor re-walked the whole meta interior once more per
transaction on the read path (~10% on a synthetic V4 walker, growing
with op count; the pubnet fixture is protocol 22/all-V3 where the arm
never runs). V3 uses the locate pass only when both event sets are
wanted; with one set the single accessor sizes strictly less. mustFields
is the local Must twin that Fields() lacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>1 parent 3c3872f commit 3114a80
9 files changed
Lines changed: 1253 additions & 211 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
| 12 | + | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
44 | 47 | | |
45 | 48 | | |
46 | 49 | | |
| |||
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
60 | | - | |
| 63 | + | |
61 | 64 | | |
62 | 65 | | |
63 | 66 | | |
64 | | - | |
| 67 | + | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
68 | 71 | | |
69 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
70 | 75 | | |
71 | 76 | | |
72 | 77 | | |
| |||
85 | 90 | | |
86 | 91 | | |
87 | 92 | | |
88 | | - | |
| 93 | + | |
89 | 94 | | |
90 | 95 | | |
91 | 96 | | |
92 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
93 | 102 | | |
94 | 103 | | |
95 | 104 | | |
96 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
97 | 148 | | |
98 | 149 | | |
99 | 150 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
0 commit comments