-
Notifications
You must be signed in to change notification settings - Fork 375
feat(tests): add gas-repricings SLOAD tests #1769
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: forks/osaka
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| # Vector Storage Benchmark | ||
|
|
||
| Single contract design for minimal-overhead storage benchmarking with parametrized operations. | ||
|
|
||
| ## Design Philosophy | ||
|
|
||
| This implementation uses **ONE contract** that: | ||
| - Is pre-deployed with 500 filled storage slots | ||
| - Accepts parameters via calldata for flexible operation | ||
| - Performs storage operations with minimal loop overhead (~30-40 gas per slot) | ||
|
|
||
| ## Contract Architecture | ||
|
|
||
| ### Calldata Layout (32-byte aligned) | ||
| ``` | ||
| Bytes 0-31: Number of slots to write (uint256) | ||
| Bytes 32-63: Starting slot index (uint256) | ||
| Bytes 64-95: Value to write (uint256) | ||
| ``` | ||
|
|
||
| ### Pre-deployed State | ||
| - **500 slots** pre-filled with value `0xDEADBEEF` | ||
| - Enables testing all transition types without redeployment | ||
| - Single contract instance for all benchmarks | ||
|
|
||
| ## Three Test Scenarios | ||
|
|
||
| ### 1. Cold Write (0 β non-zero) | ||
| - **Start**: Slot 500 (beyond prefilled range) | ||
| - **Operation**: Write new value to empty slots | ||
| - **Gas Cost**: ~20,000 gas per slot (most expensive) | ||
| - **Use Case**: Initial storage allocation | ||
|
|
||
| ### 2. Storage Clear (non-zero β 0) | ||
| - **Start**: Slot 0 (within prefilled range) | ||
| - **Operation**: Write zeros to clear existing values | ||
| - **Gas Cost**: ~2,900 gas per slot + refund | ||
| - **Use Case**: Storage cleanup/deletion | ||
|
|
||
| ### 3. Warm Update (non-zero β non-zero) | ||
| - **Start**: Slot 0 (within prefilled range) | ||
| - **Operation**: Update existing values | ||
| - **Gas Cost**: ~2,900 gas per slot | ||
| - **Use Case**: Typical storage updates | ||
|
|
||
| ## Implementation Details | ||
|
|
||
| ### Loop Overhead Breakdown | ||
| ``` | ||
| Per iteration: | ||
| - DUP operations: 12 gas (4 Γ 3 gas) | ||
| - ADD operation: 3 gas | ||
| - LT comparison: 3 gas | ||
| - ISZERO: 3 gas | ||
| - JUMPI/JUMP: 18 gas (8 + 10) | ||
| - SSTORE: Variable (based on transition) | ||
| Total overhead: ~39 gas per slot | ||
| ``` | ||
|
|
||
| ### Contract Size | ||
| - Basic loop contract: ~100 bytes | ||
| - Well under EIP-3860 limit (24,576 bytes) | ||
| - Can handle thousands of operations per call | ||
|
|
||
| ## Gas Cost Summary | ||
|
|
||
| | Transition Type | Gas per Slot | Notes | | ||
| |----------------|--------------|-------| | ||
| | 0 β non-zero | 20,000 | Cold slot, initial write | | ||
| | non-zero β 0 | 2,900 + refund | Max 20% refund of total gas | | ||
| | non-zero β non-zero | 2,900 | Warm slot update | | ||
| | Loop overhead | ~39 | Minimal per-slot overhead | | ||
|
|
||
| ## Running the Benchmarks | ||
|
|
||
| ### Generate Test Fixtures | ||
| ```bash | ||
| # Run benchmarks with stateful marker | ||
| uv run fill tests/benchmark/stateful/vector_storage/test_vector_storage.py \ | ||
| -m stateful --fork Prague | ||
| ``` | ||
|
|
||
| ### Execute Against Client | ||
| ```bash | ||
| # Direct execution | ||
| uv run consume direct --input fixtures/ --client-bin /path/to/geth | ||
|
|
||
| # Via Engine API | ||
| uv run consume engine --input fixtures/ --engine-endpoint http://localhost:8551 | ||
| ``` | ||
|
|
||
| ## Advantages Over Multiple Contracts | ||
|
|
||
| 1. **Single Deployment**: One contract handles all scenarios | ||
| 2. **Consistent Overhead**: Same loop structure for all operations | ||
| 3. **Flexible Testing**: Parametrized via calldata | ||
| 4. **Realistic Patterns**: Mimics real smart contract storage patterns | ||
| 5. **Accurate Benchmarking**: Minimal overhead ensures accurate gas measurements | ||
|
|
||
| ## Test Parameters | ||
|
|
||
| - **num_slots**: [1, 10, 50, 100, 200] - Number of slots per operation | ||
| - **batch_size**: [10, 25, 50] - For batch operation tests | ||
| - **Pre-filled slots**: 500 - Consistent across all tests | ||
|
|
||
| ## Example Usage | ||
|
|
||
| The contract can be called with raw calldata: | ||
| ```python | ||
| # Write 100 values starting at slot 500 | ||
| calldata = ( | ||
| (100).to_bytes(32, 'big') + # num_slots | ||
| (500).to_bytes(32, 'big') + # start_slot | ||
| (0x1234).to_bytes(32, 'big') # value | ||
| ) | ||
| ``` | ||
|
|
||
| This design provides the most accurate storage benchmarking with minimal overhead and maximum flexibility. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| """ | ||
| Vector storage benchmarks for measuring SLOAD/SSTORE operations with | ||
| minimal overhead. | ||
| """ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| """ | ||
| abstract: Vector storage benchmark with single parametrized contract, | ||
| targeting SLOAD. | ||
|
|
||
| This parametrized test takes takes these arguments: | ||
| - The amount of slots to load | ||
| - The slot key incrementer | ||
|
|
||
| The final value is used in the test as boolean: if 0 is used, | ||
| the key is not incremented, and thus the same key is read each time. | ||
|
|
||
| Each test is also tested against these keys in the access list (or not). | ||
| This thus marks if the target slots are warm or cold. | ||
| """ | ||
|
|
||
| import pytest | ||
| from execution_testing import ( | ||
| Account, | ||
| Alloc, | ||
| Block, | ||
| BlockchainTestFiller, | ||
| Bytecode, | ||
| Fork, | ||
| Op, | ||
| Storage, | ||
| Transaction, | ||
| While | ||
| ) | ||
|
|
||
| """ | ||
| abstract: Vector storage benchmark with single parametrized contract, | ||
| targeting SLOAD. | ||
|
|
||
| This parametrized test takes takes these arguments: | ||
| - The amount of slots to load | ||
| - The slot key incrementer | ||
|
|
||
| The final value is used in the test as boolean: if 0 is used, | ||
| the key is not incremented, and thus the same key is read each time. | ||
|
|
||
| Each test is also tested against these keys in the access list (or not). | ||
| This thus marks if the target slots are warm or cold. | ||
| """ | ||
|
|
||
| from typing import List | ||
|
|
||
| import pytest | ||
|
|
||
| from execution_testing import ( | ||
| Account, | ||
| Alloc, | ||
| Block, | ||
| BlockchainTestFiller, | ||
| Bytecode, | ||
| Fork, | ||
| Op, | ||
| Storage, | ||
| Transaction, | ||
| ) | ||
|
|
||
|
|
||
| def create_sload_contract() -> Bytecode: | ||
| """ | ||
| Creates a storage contract. | ||
| """ | ||
| bytecode = Bytecode() | ||
|
|
||
| end_marker = 24 | ||
| start_marker = 4 | ||
|
|
||
| bytecode += Op.PUSH0 | ||
| bytecode += Op.CALLDATALOAD(Op.PUSH1(32)) | ||
|
|
||
| bytecode += Op.JUMPDEST | ||
|
|
||
| bytecode += Op.DUP1 | ||
|
|
||
| bytecode += Op.ISZERO | ||
| bytecode += Op.JUMPI(Op.PUSH1(end_marker)) | ||
|
|
||
| # Loop entry, stack (topmost item first): [entries_left, current_slot] | ||
|
|
||
| bytecode += Op.PUSH1(1) | ||
| bytecode += Op.SWAP1 | ||
| bytecode += Op.SUB | ||
|
|
||
| bytecode += Op.SWAP1 | ||
|
|
||
| # Stack here: [current_slot, entries_left] | ||
|
|
||
| bytecode += Op.DUP1 | ||
|
|
||
| bytecode += Op.SLOAD | ||
| bytecode += Op.POP | ||
|
|
||
| # Stack here: [current_slot, entries_left] | ||
|
|
||
| bytecode += Op.ADD(Op.CALLDATALOAD(Op.PUSH0)) | ||
|
|
||
| bytecode += Op.SWAP1 | ||
|
|
||
| bytecode += Op.PUSH1(start_marker) | ||
|
|
||
| bytecode += Op.JUMPDEST # end_marker | ||
| bytecode += Op.STOP | ||
|
Comment on lines
+98
to
+105
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not sure if i understand here correctly: Initial stack here: PUSH0 # [0, current_slot, entries_left]
CALLDATALOAD # [incrementer, current_slot, entries_left]
ADD # [current_slot + incrementer, entries_left]
SWAP1 # [entires_left, current_slot + incrementer]
PUSH1 marker # [marker, entries_left, current_slot+incrementer]
JUMPDEST #
STOPWhen executing to
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I think the bytecode is missing both correct stack initialization and the JUMP instruction to complete the loop. |
||
|
|
||
| return bytecode | ||
|
|
||
|
|
||
| # TODO: add a pointer to [empty, small, big, XEN] sized contracts | ||
| # See https://github.com/ethereum/execution-specs/issues/1755#issuecomment-3508963411 | ||
| # for a way how I believe we can do this (using 7702 accounts with prefilled | ||
| # storage and then executing code on there, which we can change because its a 7702 account) | ||
| @pytest.mark.valid_from("Prague") | ||
| @pytest.mark.stateful # Mark as stateful instead of benchmark | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All the tests under stateful folder will automatically inherit this label. |
||
| @pytest.mark.parametrize("num_slots", [1, 10, 50, 100, 200]) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we reduce the parameterization slightly and maybe drop the 10/50? |
||
| @pytest.mark.parametrize("warm_slots", [False, True]) | ||
| @pytest.mark.parametrize("storage_keys_set", [False, True]) | ||
| # NOTE: the 0 incrementer will thus SLOAD the same slot again | ||
| @pytest.mark.parametrize("incrementer", [0, 1]) | ||
| def test_storage_sload_benchmark( | ||
| blockchain_test: BlockchainTestFiller, | ||
| pre: Alloc, | ||
| warm_slots: bool, | ||
| storage_keys_set: bool, | ||
| num_slots: int, | ||
| incrementer: int, | ||
| ) -> None: | ||
| """ | ||
| TODO: write docs. | ||
| """ | ||
| sender = pre.fund_eoa() | ||
|
|
||
| initial_storage = Storage() | ||
| slots: set[int] = set() | ||
| if storage_keys_set: | ||
| key = 0 | ||
| for i in range(num_slots): | ||
| initial_storage[key] = 1 | ||
| slots.add(key) | ||
| key += incrementer | ||
|
Comment on lines
+136
to
+141
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When |
||
|
|
||
| storage_contract = pre.deploy_contract( | ||
| code=create_sload_contract(), | ||
| storage=initial_storage, | ||
| ) | ||
|
|
||
| calldata = incrementer.to_bytes(32, "big") + num_slots.to_bytes(32, "big") | ||
|
|
||
| access_lists: List[AccessList] = [] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is a missing import. |
||
|
|
||
| if warm_slots: | ||
| access_lists = [ | ||
| AccessList( | ||
| address=storage_contract, | ||
| storage_keys=list(slots), | ||
| ), | ||
| ] | ||
|
|
||
| # Create transaction to call the contract | ||
| # Use a reasonable gas limit that covers the operation | ||
| gas_limit = 21000 + 10000 + (num_slots * 50000) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we do not need to specify the gas limit for this transaction? By default it would be transaction gas limit cap or block gas limit in our framework |
||
|
|
||
| tx = Transaction( | ||
| to=storage_contract, | ||
| gas_limit=gas_limit, | ||
| access_list=access_lists, | ||
| data=calldata, | ||
| sender=sender, | ||
| ) | ||
|
|
||
| blockchain_test( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These should use the |
||
| pre=pre, | ||
| blocks=[Block(txs=[tx])], | ||
| ) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems duplicate as import & doc string below!