Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix benches for ia_loop, circles #1922

Merged
merged 2 commits into from
Nov 5, 2024
Merged

Fix benches for ia_loop, circles #1922

merged 2 commits into from
Nov 5, 2024

Conversation

mamcx
Copy link
Contributor

@mamcx mamcx commented Oct 30, 2024

Description of Changes

Corrections to how are benched ia_loop & circles.

Basically, circles have auto_inc in their tables definitions, but ia_loop does not.

The consequence is that circles take a lot of time because the init_game_circles get called by each iteration of the benches, resulting in a big table with up to 1_000_000 rows when the setup was meant to be at most 10_000.

Then for the ia_loop the lack of auto_inc causes a unique constraint violation because the table is forbidden to grow.

Now:

  • In both, the database is initialized just once with the maximum amount of rows
  • Is removed group.measurement_time(Duration::from_secs(60 * 2)); that causes the bench to run longer than must
  • Removed the filtering in the ia_loop game loop based on timestamps, it causes the code to run differently between the integration test and benches.
  • Increase the size of ia_loop from [10, 100] to [500, 5_000] so criterion has something to measure:
# BEFORE
special/db_game/ia_loop/load=100
                        time:   [0.0000 ps 0.0000 ps 0.0000 ps]
# AFTER
special/db_game/ia_loop/load=500
                        time:   [386.62 ps 461.53 ps 640.54 ps]

API and ABI breaking changes

Nothing, but this change invalidates the numbers of previous benchmarks.

Expected complexity level and risk

1

Testing

Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!

  • Runing a lot of benches with different parameters

@mamcx mamcx added release-any To be landed in any release window bugfix Fixes something that was expected to work differently release-1.0 labels Oct 30, 2024
@mamcx mamcx self-assigned this Oct 30, 2024
@mamcx mamcx requested a review from RReverser October 30, 2024 19:32
@mamcx
Copy link
Contributor Author

mamcx commented Oct 30, 2024

benchmarks please

Copy link

github-actions bot commented Oct 30, 2024

Benchmarking failed. Please check the workflow run for details.

Copy link

github-actions bot commented Oct 30, 2024

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6890 6890 0.00% 6982 6982 0.00%
sqlite 5579 5579 0.00% 5979 5979 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 77086 77086 0.00% 77646 77622 0.03%
stdb_raw u32_u64_str no_index 64 128 2 string 119585 120674 -0.90% 120363 121464 -0.91%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25592 25577 0.06% 26278 26267 0.04%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24543 24543 0.00% 25023 25027 -0.02%
sqlite u32_u64_str no_index 64 128 2 string 144701 144701 0.00% 146225 146225 0.00%
sqlite u32_u64_str no_index 64 128 1 u64 124050 124050 0.00% 125296 125296 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131361 131361 0.00% 132717 132713 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 134494 134494 0.00% 136160 136164 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 878821 879018 -0.02% 928825 928716 0.01%
stdb_raw u32_u64_str btree_each_column 64 128 1028593 1029081 -0.05% 1055525 1055601 -0.01%
sqlite u32_u64_str unique_0 64 128 398326 398320 0.00% 419024 419018 0.00%
sqlite u32_u64_str btree_each_column 64 128 983649 983643 0.00% 1025367 1025373 -0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 154220 154220 0.00% 154306 154306 0.00%
stdb_raw u32_u64_str unique_0 64 17243 17243 0.00% 17313 17309 0.02%
sqlite u32_u64_str unique_0 1024 1067255 1067255 0.00% 1070575 1070575 0.00%
sqlite u32_u64_str unique_0 64 76201 76201 0.00% 77195 77195 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50180 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9594 9594 0.00%
16 json 12188 12188 0.00% 14092 14092 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 20529210 20337663 0.94% 21036868 20813631 1.07%
stdb_raw u32_u64_str unique_0 64 128 1287422 1288084 -0.05% 1352338 1322444 2.26%
sqlite u32_u64_str unique_0 1024 1024 1802188 1802188 0.00% 1811562 1811562 0.00%
sqlite u32_u64_str unique_0 64 128 128534 128534 0.00% 131426 131426 0.00%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6895 6895 0.00% 7003 7003 0.00%
sqlite 5621 5621 0.00% 6085 6085 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 77091 77091 0.00% 77591 77619 -0.04%
stdb_raw u32_u64_str no_index 64 128 2 string 119590 120679 -0.90% 120320 121493 -0.97%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25582 25581 0.00% 26188 26191 -0.01%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24548 24548 0.00% 24996 24992 0.02%
sqlite u32_u64_str no_index 64 128 1 u64 125965 125965 0.00% 127457 127465 -0.01%
sqlite u32_u64_str no_index 64 128 2 string 146616 146616 0.00% 148370 148366 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 136616 136649 -0.02% 138744 138785 -0.03%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133457 133457 0.00% 135231 135227 0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 828073 828094 -0.00% 878345 878410 -0.01%
stdb_raw u32_u64_str btree_each_column 64 128 977396 978577 -0.12% 1033714 1035067 -0.13%
sqlite u32_u64_str unique_0 64 128 415857 415857 0.00% 435913 435921 -0.00%
sqlite u32_u64_str btree_each_column 64 128 1021908 1021908 0.00% 1063034 1063030 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 154225 154225 0.00% 154291 154295 -0.00%
stdb_raw u32_u64_str unique_0 64 17248 17248 0.00% 17310 17310 0.00%
sqlite u32_u64_str unique_0 1024 1070323 1070323 0.00% 1074093 1074093 0.00%
sqlite u32_u64_str unique_0 64 77973 77973 0.00% 79243 79243 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50180 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9594 9594 0.00%
16 json 12188 12188 0.00% 14092 14092 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19040233 19036056 0.02% 19596037 19596200 -0.00%
stdb_raw u32_u64_str unique_0 64 128 1239742 1240478 -0.06% 1304990 1305964 -0.07%
sqlite u32_u64_str unique_0 1024 1024 1809743 1809743 0.00% 1818387 1818387 0.00%
sqlite u32_u64_str unique_0 64 128 132654 132654 0.00% 135524 135524 0.00%

@RReverser RReverser added this pull request to the merge queue Nov 5, 2024
Merged via the queue into master with commit cb9e017 Nov 5, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix Fixes something that was expected to work differently release-1.0 release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants