15% performance regression in revm-interpreter from 1.0.0 to 1.1.1?

[ziyadedher]

I'm only like 70% confident in these results, but I just wanted to flag for you: ziyadedher/evm-bench#153 (comment)

Let me know if

1. you think the results are wrong, or
2. you think the results are right, or
3. you want some more details.

cc @gakonst

[valo]

I ran the benchmark and compared the latest revm from commit 5ca4741, the one pinned in evm-bench: 3fda52e and evmone. I increased the number of runs of each benchmark to 20, so that I can get more reliable averages. These are the results I get:

|                         | revm    | revm-latest | evmone  |
|-------------------------|---------|-------------|---------|
| **sum**                 | 69.35ms | 70.4ms      | 304.3ms |
| **relative**            | 1.000x  | 1.015x      | 4.388x  |
| erc20.approval-transfer | 7.6ms   | 7.05ms      | 5.8ms   |
| erc20.mint              | 4.1ms   | 4.1ms       | 4.65ms  |
| erc20.transfer          | 8.5ms   | 8.1ms       | 7.25ms  |
| snailtracer             | 46.05ms | 47.35ms     | 284.6ms |
| ten-thousand-hashes     | 3.1ms   | 3.8ms       | 2ms     |

The difference between the revm versions seems negligible. Let me know if you have ideas of other stuff I can try here. I am wondering if there is a point in measuring the std deviation of the run durations.

[rakita]

Few performance improvments were done from this release, So I will close this.