Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions tests/benchmark/stateful/vector_storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Vector Storage Benchmark

Single contract design for minimal-overhead storage benchmarking with parametrized operations.

## Design Philosophy

This implementation uses **ONE contract** that:
- Is pre-deployed with 500 filled storage slots
- Accepts parameters via calldata for flexible operation
- Performs storage operations with minimal loop overhead (~30-40 gas per slot)

## Contract Architecture

### Calldata Layout (32-byte aligned)
```
Bytes 0-31: Number of slots to write (uint256)
Bytes 32-63: Starting slot index (uint256)
Bytes 64-95: Value to write (uint256)
```

### Pre-deployed State
- **500 slots** pre-filled with value `0xDEADBEEF`
- Enables testing all transition types without redeployment
- Single contract instance for all benchmarks

## Three Test Scenarios

### 1. Cold Write (0 β†’ non-zero)
- **Start**: Slot 500 (beyond prefilled range)
- **Operation**: Write new value to empty slots
- **Gas Cost**: ~20,000 gas per slot (most expensive)
- **Use Case**: Initial storage allocation

### 2. Storage Clear (non-zero β†’ 0)
- **Start**: Slot 0 (within prefilled range)
- **Operation**: Write zeros to clear existing values
- **Gas Cost**: ~2,900 gas per slot + refund
- **Use Case**: Storage cleanup/deletion

### 3. Warm Update (non-zero β†’ non-zero)
- **Start**: Slot 0 (within prefilled range)
- **Operation**: Update existing values
- **Gas Cost**: ~2,900 gas per slot
- **Use Case**: Typical storage updates

## Implementation Details

### Loop Overhead Breakdown
```
Per iteration:
- DUP operations: 12 gas (4 Γ— 3 gas)
- ADD operation: 3 gas
- LT comparison: 3 gas
- ISZERO: 3 gas
- JUMPI/JUMP: 18 gas (8 + 10)
- SSTORE: Variable (based on transition)
Total overhead: ~39 gas per slot
```

### Contract Size
- Basic loop contract: ~100 bytes
- Well under EIP-3860 limit (24,576 bytes)
- Can handle thousands of operations per call

## Gas Cost Summary

| Transition Type | Gas per Slot | Notes |
|----------------|--------------|-------|
| 0 β†’ non-zero | 20,000 | Cold slot, initial write |
| non-zero β†’ 0 | 2,900 + refund | Max 20% refund of total gas |
| non-zero β†’ non-zero | 2,900 | Warm slot update |
| Loop overhead | ~39 | Minimal per-slot overhead |

## Running the Benchmarks

### Generate Test Fixtures
```bash
# Run benchmarks with stateful marker
uv run fill tests/benchmark/stateful/vector_storage/test_vector_storage.py \
-m stateful --fork Prague
```

### Execute Against Client
```bash
# Direct execution
uv run consume direct --input fixtures/ --client-bin /path/to/geth

# Via Engine API
uv run consume engine --input fixtures/ --engine-endpoint http://localhost:8551
```

## Advantages Over Multiple Contracts

1. **Single Deployment**: One contract handles all scenarios
2. **Consistent Overhead**: Same loop structure for all operations
3. **Flexible Testing**: Parametrized via calldata
4. **Realistic Patterns**: Mimics real smart contract storage patterns
5. **Accurate Benchmarking**: Minimal overhead ensures accurate gas measurements

## Test Parameters

- **num_slots**: [1, 10, 50, 100, 200] - Number of slots per operation
- **batch_size**: [10, 25, 50] - For batch operation tests
- **Pre-filled slots**: 500 - Consistent across all tests

## Example Usage

The contract can be called with raw calldata:
```python
# Write 100 values starting at slot 500
calldata = (
(100).to_bytes(32, 'big') + # num_slots
(500).to_bytes(32, 'big') + # start_slot
(0x1234).to_bytes(32, 'big') # value
)
```

This design provides the most accurate storage benchmarking with minimal overhead and maximum flexibility.
4 changes: 4 additions & 0 deletions tests/benchmark/stateful/vector_storage/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
"""
Vector storage benchmarks for measuring SLOAD/SSTORE operations with
minimal overhead.
"""
175 changes: 175 additions & 0 deletions tests/benchmark/stateful/vector_storage/test_vector_sload.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
"""
abstract: Vector storage benchmark with single parametrized contract,
targeting SLOAD.

This parametrized test takes takes these arguments:
- The amount of slots to load
- The slot key incrementer

The final value is used in the test as boolean: if 0 is used,
the key is not incremented, and thus the same key is read each time.

Each test is also tested against these keys in the access list (or not).
This thus marks if the target slots are warm or cold.
"""

import pytest
from execution_testing import (
Account,
Alloc,
Block,
BlockchainTestFiller,
Bytecode,
Fork,
Op,
Storage,
Transaction,
While
)
Comment on lines +1 to +28
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems duplicate as import & doc string below!


"""
abstract: Vector storage benchmark with single parametrized contract,
targeting SLOAD.

This parametrized test takes takes these arguments:
- The amount of slots to load
- The slot key incrementer

The final value is used in the test as boolean: if 0 is used,
the key is not incremented, and thus the same key is read each time.

Each test is also tested against these keys in the access list (or not).
This thus marks if the target slots are warm or cold.
"""

from typing import List

import pytest

from execution_testing import (
Account,
Alloc,
Block,
BlockchainTestFiller,
Bytecode,
Fork,
Op,
Storage,
Transaction,
)


def create_sload_contract() -> Bytecode:
"""
Creates a storage contract.
"""
bytecode = Bytecode()

end_marker = 24
start_marker = 4

bytecode += Op.PUSH0
bytecode += Op.CALLDATALOAD(Op.PUSH1(32))

bytecode += Op.JUMPDEST

bytecode += Op.DUP1

bytecode += Op.ISZERO
bytecode += Op.JUMPI(Op.PUSH1(end_marker))

# Loop entry, stack (topmost item first): [entries_left, current_slot]

bytecode += Op.PUSH1(1)
bytecode += Op.SWAP1
bytecode += Op.SUB

bytecode += Op.SWAP1

# Stack here: [current_slot, entries_left]

bytecode += Op.DUP1

bytecode += Op.SLOAD
bytecode += Op.POP

# Stack here: [current_slot, entries_left]

bytecode += Op.ADD(Op.CALLDATALOAD(Op.PUSH0))

bytecode += Op.SWAP1

bytecode += Op.PUSH1(start_marker)

bytecode += Op.JUMPDEST # end_marker
bytecode += Op.STOP
Comment on lines +98 to +105
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if i understand here correctly:

Initial stack here: [current_slot, entries_lest]

PUSH0 # [0, current_slot, entries_left]
CALLDATALOAD # [incrementer, current_slot, entries_left]
ADD # [current_slot + incrementer, entries_left]
SWAP1 # [entires_left, current_slot + incrementer]
PUSH1 marker # [marker, entries_left, current_slot+incrementer]
JUMPDEST # 
STOP

When executing to JUMPDEST, should we jump back to the start marker via JUMPI? But here seems missing a DUP somewhere. Or maybe there is missing some parameter: Op.ADD(Op.CALLDATALOAD(Op.PUSH0)) should be Op.ADD(Op.CALLDATALOAD(Op.PUSH0), N)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think the bytecode is missing both correct stack initialization and the JUMP instruction to complete the loop.


return bytecode


# TODO: add a pointer to [empty, small, big, XEN] sized contracts
# See https://github.com/ethereum/execution-specs/issues/1755#issuecomment-3508963411
# for a way how I believe we can do this (using 7702 accounts with prefilled
# storage and then executing code on there, which we can change because its a 7702 account)
@pytest.mark.valid_from("Prague")
@pytest.mark.stateful # Mark as stateful instead of benchmark
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the tests under stateful folder will automatically inherit this label.

@pytest.mark.parametrize("num_slots", [1, 10, 50, 100, 200])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reduce the parameterization slightly and maybe drop the 10/50?

@pytest.mark.parametrize("warm_slots", [False, True])
@pytest.mark.parametrize("storage_keys_set", [False, True])
# NOTE: the 0 incrementer will thus SLOAD the same slot again
@pytest.mark.parametrize("incrementer", [0, 1])
def test_storage_sload_benchmark(
blockchain_test: BlockchainTestFiller,
pre: Alloc,
warm_slots: bool,
storage_keys_set: bool,
num_slots: int,
incrementer: int,
) -> None:
"""
TODO: write docs.
"""
sender = pre.fund_eoa()

initial_storage = Storage()
slots: set[int] = set()
if storage_keys_set:
key = 0
for i in range(num_slots):
initial_storage[key] = 1
slots.add(key)
key += incrementer
Comment on lines +136 to +141
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When storage_keys_set=False and warm_slots=True, I think the test should make slots warm (via access list) even though they don't exist in storage, but it doesn't because slots is empty. As a result, I think the access list contains no storage keys, so the slots remain cold instead of being warmed.


storage_contract = pre.deploy_contract(
code=create_sload_contract(),
storage=initial_storage,
)

calldata = incrementer.to_bytes(32, "big") + num_slots.to_bytes(32, "big")

access_lists: List[AccessList] = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a missing import.


if warm_slots:
access_lists = [
AccessList(
address=storage_contract,
storage_keys=list(slots),
),
]

# Create transaction to call the contract
# Use a reasonable gas limit that covers the operation
gas_limit = 21000 + 10000 + (num_slots * 50000)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do not need to specify the gas limit for this transaction? By default it would be transaction gas limit cap or block gas limit in our framework


tx = Transaction(
to=storage_contract,
gas_limit=gas_limit,
access_list=access_lists,
data=calldata,
sender=sender,
)

blockchain_test(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should use the benchmark_test format now! Please see here for usage and skip validation field.

pre=pre,
blocks=[Block(txs=[tx])],
)
Loading
Loading