100x faster EVM traces with Python

An introduction to [evm-trace](https://github.com/apeworx/evm-trace), a fast and correct Python library to work with Ethereum Virtual Machine traces.

There are two common trace formats, which we’ll refer to as Geth-style and Parity-style traces. Geth traces provide opcode-level resolution, while the most commonly used Parity trace format provides a call tree.

Our goal today would be to make opcode-resolution traces 100x faster, making them from  a little-known Parity trace format. We’ll also make them compatible with Geth traces so they can be used as a drop-in replacement.

# debug_traceTransaction

[Geth traces](https://geth.ethereum.org/docs/rpc/ns-debug#debug_tracetransaction) return frames containing the full state of the virtual machine at each step of the execution, including the program counter, opcode, remaining gas, gas cost of the operation, call depth, items on the stack, a snapshot of the memory, and a storage diff.

Each subcall starts with a fresh and separate memory, stack, and storage changes.

It’s a powerful but a bit unwieldy format, since you need to do extra work to know things like what’s the current contract being called or what’s the calldata.

I started digging into this when I [stumbled upon](https://github.com/ledgerwatch/erigon/issues/4637) several transactions that crash Erigon if you request a debug trace. It appeared that they generated very large traces, and soon enough I was able to confirm why when Erigon [re-enabled streaming](https://github.com/ledgerwatch/erigon/pull/4647) for heavy RPC endpoints.

In Geth format each frame contains the full copy of the EVM memory, which can lead to traces that don’t fit in the system memory. A contract expanding the memory to 100kb could generate traces [as large as 70 GB](https://gist.github.com/banteg/98dbccbf6e2a3f997199a1b16eb93c5a).

This can be addressed by streaming and [iteratively decoding](https://github.com/ApeWorX/ape/pull/885) the response as you get it from the Ethereum client. This is what ape does by default with Geth traces when you call `receipt.get_transaction_trace()`.

If we look up the memory expansion gas cost formulas at [evm.codes](https://www.evm.codes/about#memoryexpansion), it becomes apparent that EVM memory is not that expensive and we can possibly construct or find transactions with even larger traces.

![memory_cost = (memory_size_word \*\* 2) / 512 + (3 \* memory_size_word)](https://images.mirror-media.xyz/publication-images/jyS9SoVKtriQWElYd3cRY.png?height=1456&width=2892)

After sampling peak memory usage over 10,000 most recent blocks, we find even larger traces. For [one of those](https://etherscan.io/tx/0x9ce97c7e8040002df869fff04caa9b5f176dd11859ab4af317bd86fd4455918d), Erigon hangs up on us after 30 minutes, after returning 460 GB and 898,836 frames.

We can patch Erigon to [increase the write timeout](https://github.com/ledgerwatch/erigon/blob/b6ea28ea806b922617eec3df2ea3ffa663581ad5/rpc/rpccfg/rpccfg.go#L35), and learn that the full trace consists of 960,198 frames, takes up 524 GB and 34m7s to fetch, before deserialization or processing. We can calculate that we got this trace at 275 MB/s and **469 frames/s**.

Keep that number in mind, because with a method presented below, we can recompute the same trace using 137 MB of data (3916x less), which we can fetch in 5.6s, deserialize in 7.1s and **replay in 4.6 seconds**, with **full processing taking 117x less time** than just fetching a Geth trace.

![contracts with peak memory usage](https://images.mirror-media.xyz/publication-images/0EixDUCpLYsVAGrWJ-ow9.png?height=1196&width=2344)

## trace_replayTransaction

[Parity traces](https://openethereum.github.io/JSONRPC-trace-module) are quite different from Geth, the ad hoc traces come in three different flavors: `trace`, `vmTrace`, and `stateDiff`.

> Parity traces are supported in Erigon, Nethermind, Besu, Foundry.
> They are not supported in Geth, Hardhat, Ganache.
> Anything beyond `trace` type requires an archi

Quest for a lost Event

The last time I wrote about [Ethereum logs](https://codeburst.io/deep-dive-into-ethereum-logs-a8d2047c7371) was in 2018. This time we go deeper and try to recover a ghost event that devs have forgotten to add to the Yearn Vaults contract.

```
log Fees(management_fee, performance_fee, strategist_fee)
```

There are several methods to recover values from an internal view method or even add custom views to batch read private storage. One of the more interesting ones is a “[state override](https://www.libevm.com/2022/01/12/advance-geth-pt-2-stateoverrides/)” option of `eth_call`, which allows you to replace the code and storage for the context of the call.

To my deep disappointment, you can’t just set a breakpoint on a line you want, start a debugger to capture the local variables, and call it a day. This may change soon with a [Vyper interpreter](https://github.com/vyperlang/titanoboa) being in the works.

This post will describe a method based on capturing the values from a trace frame. The values we are interested in should appear on the stack or in the memory [around this line](https://github.com/yearn/yearn-vaults/blob/v0.4.3/contracts/Vault.vy#L1621).

## Obtain Vyper metadata

The `debug_traceTransaction` method gives us trace frames with opcode-level resolution. To find the correct trace frame, we need to find the program counter around the end of `_assessFees` method execution. Maybe the metadata Vyper compiler generates could help us do that:

```
vyper -f source_map contracts/Vault.vy > source_map.json
```

The source map contains `pc_jump_map` and `pc_pos_map`, which looks promising. The jump map contains values of `i`, `o`, `-`, which seem to stand for jump in, jump out, and jump umm. Intuitively we want to find the jump out of the function, but it won’t be there since it’s the last internal method in the contract.

The position map is formatted as `start_line`, `start_col`, `end_line`, `end_col` for each program counter. This could help us limit the search window to the desired function. The simplest way to find the line numbers is to just look at the source file, but there are a dozen of Vault releases, and we want to automate the search as much as possible. Getting more metadata from the compiler wouldn’t hurt:

```
vyper -f ast contracts/Vault.vy > ast.json
```

This format outputs the abstract syntax tree of the parsed contract, which looks the same as Python ast. We can find all the definitions in the ast body, and find the function by name:

```
fn = next(item for item in ast["ast"]["body"] if item.get("name") == "_assessFees")
```

The definition would contain `lineno` and `end_lineno`, which is exactly what we need to scale the search to all versions. Now we can output all the program counters which have originated from a line range in the source and exist in the jump table.

They will also come in handy later when we need to deal with edge cases.

## Rewrite in Python

After writing a script that checks out all Vault releases from git tags and compiles them, we have enough metadata to start the search. But we still don’t have the concrete values to match on. Luckily it’s easy to convert Vyper to Python. You just rename the file.

Yearn Vaults also behave according to their `apiVersion`, so we can write a universal `_assessFees` function, which was an interesting enough [exercise on its own](https://github.com/banteg/yearn-fees/blob/master/yearn_fees/assess.py). Even if it doesn’t handle all the edge cases yet, it works well enough to provide us with the calculated values to look for in a trace.

## Look at the trace

> Tip: If you often work with Ethereum traces in Python, consider looking at [evm-trace](https://github.com/apeworx/evm-trace), it can parse both Geth- (opcode-level) and Parity-style (call-level) traces, turn them into a call tree, and display them to your liking via a custom display class.

A trace frame at the program counter we found contains all the values we need both on the stack and in

🔴 How did a hacker steal OP?

Wintermute had their mainnet safe [deployed](https://etherscan.io/tx/0xd705178d68551a6a6f65ca74363264b32150857a26dd62c27f3f96b8ec69ca01) from a Proxy Factory 1.1.1, which is older than the earliest official Optimism [deployment](https://github.com/safe-global/safe-deployments/blob/d41d083ba524848c6e59e073201f7c69fdd44be1/src/assets/v1.3.0/proxy_factory.json#L10) of 1.3.0.

Luckily for our hacker, Safe deliberately utilizes non-EIP-155-compliant deployment transactions, which can be [replayed](https://github.com/5afe/safe-contract-deployment-replay/) on any network which doesn’t enforce [EIP-155](https://eips.ethereum.org/EIPS/eip-155).

They did exactly that, [replaying](https://optimistic.etherscan.io/tx/0x75a42f240d229518979199f56cd7c82e4fc1f1a20ad9a4864c635354b4a34261) the [deployment](https://github.com/safe-global/safe-deployments/blob/d41d083ba524848c6e59e073201f7c69fdd44be1/src/assets/v1.1.1/proxy_factory.json#L7) of Proxy Factory 1.1.1 on Optimism.

Similar to externally owned accounts, contracts on Ethereum maintain a nonce, which is increased when they spawn new contracts with either `CREATE` or `CREATE2`. Both addresses [can be predicted](https://github.com/ethereum/py-evm/blob/e924f63992a35212616b4e20355d161bc4348925/eth/_utils/address.py#L17-L26) from either (address, nonce) or (address, salt, initcode).

This safe was created using `createProxy(masterCopy, data)` which uses `CREATE` that only depends on the Factory address and its nonce value. The attacker can manipulate both the master copy and data parameters, assigning themselves as the owner. This is not possible for safes created with `CREATE2` since they use the [initializer](https://github.com/safe-global/safe-eth-py/blob/master/gnosis/safe/safe_create2_tx.py#L118) (which includes the owner set) as a [salt](https://github.com/safe-global/safe-contracts/blob/c36bcab46578a442862d043e12a83fec41143dec/contracts/proxies/GnosisSafeProxyFactory.sol#L48) which affects the resulting address.

We [can calculate](https://gist.github.com/banteg/a2b4e5fafe51a6b1ef6c54afa773348a) that Wintermute’s safe was created at nonce 8884, and since it’s a fresh Factory, the attacker had to create all the preceding safes first. To pull this off, they created a contract which initializes [162 safes at a time](https://optimistic.etherscan.io/tx/0x068da6525631399e91a2afdd2996853df2ac3173b0f09d85d5abe8f90d6ee4dd), which they called 62 times, increasing the Factory’s nonce to 10,054.

This has allowed them to grab Optimism addresses for a lot of safes existing on mainnet, including a [safe](https://optimistic.etherscan.io/address/0x4f3a120e72c76c22ae802d129f599bfdbc31cb81#tokentxns) matching Wintermute’s address.