Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix forge scheduled tests #3003

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+30 −79
Open

Fix forge scheduled tests #3003

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
db09cf3
Fix forge scheduled tests
ddoktorski Feb 26, 2025
e5f7aba
Fix bactrace tests
ddoktorski Feb 26, 2025
2b83d6d
Fix steps tests
ddoktorski Feb 27, 2025
7368c3a
Merge branch 'master' into fix-scheduled-tests
ddoktorski Mar 6, 2025
b9bf8c4
Adjust steps to be closer to 10m
ddoktorski Mar 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/scheduled.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ jobs:
- name: Install cairo-profiler
run: |
curl -L https://raw.githubusercontent.com/software-mansion/cairo-profiler/main/scripts/install.sh | sh
- name: Install cairo-coverage
run: |
curl -L https://raw.githubusercontent.com/software-mansion/cairo-coverage/main/scripts/install.sh | sh
- uses: taiki-e/install-action@nextest

- run: cargo test --release -p forge e2e
Expand Down
2 changes: 1 addition & 1 deletion crates/forge/test_utils/src/runner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ pub fn assert_gas(result: &[TestTargetSummary], test_case_name: &str, asserted_g
}
AnyTestCaseSummary::Single(case) => match case {
TestCaseSummary::Passed { gas_info: gas, .. } => {
*gas == asserted_gas
asserted_gas.abs_diff(*gas) <= 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a different assert_gas method that uses diff and choose which one to use according to the feature? Then the scheduled tests on version other than scarb from .tool-version could use it.

E.g.

    if cfg!(feature = "non_exact_gas") {
        assert_gas_range(...) 
    else {
        assert_gas(...)
     }

I'd prefer if we still kept exact gas assertions when developing code. I know difference of 1 is not that big but we should assert exact values here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, for now leaving it as it is

&& any_case
.name()
.unwrap()
Expand Down
27 changes: 3 additions & 24 deletions crates/forge/tests/data/steps/src/lib.cairo
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess one thing we are losing with these changes is that we no longer assert if the max steps is sufficiently close to the default of 10m. Maybe with some logic changes we could still retain this check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we could hide this check behind a feature as well, what do you think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has already been changed a few times and it is unmaintainable. Unfortunately, I don't have any idea for a test that will have 10m steps and for sure remain unchanged with future Cairo versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified those tests to get the following steps values:

  • Scarb 2.8.5: 8250085 and 10200085
  • Scarb 2.10.1: 9350088 and 11560088

It seems to be close enough to test the desired behavior (failing for >10m steps - tested in should_default_to_10m test) and hopefully it won't need to be updated with the new Scarb version 🙏

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, we can do that

Original file line number Diff line number Diff line change
@@ -1,42 +1,21 @@
// 75/70 constant cost depending on the Cairo version
// 15 steps per iteration
#[cfg(test)]
mod tests {
#[test]
fn steps_much_less_than_10000000() {
let mut i = 0;

while i != 37_997 {
i = i + 1;
assert(1 + 1 == 2, 'who knows?');
}
}

#[test]
fn steps_less_than_10000000() {
let mut i = 0;

while i != 666_661 {
while i != 10_000 {
i = i + 1;
assert(1 + 1 == 2, 'who knows?');
}
}

#[test]
fn steps_more_than_10000000() {
let mut i = 0;

while i != 666_663 {
i = i + 1;
assert(1 + 1 == 2, 'who knows?');
}
}

#[test]
fn steps_much_more_than_10000000() {
fn steps_more_than_10000000() {
let mut i = 0;

while i != 750_000 {
while i != 800_000 {
i = i + 1;
assert(1 + 1 == 2, 'who knows?');
}
Expand Down
8 changes: 4 additions & 4 deletions crates/forge/tests/e2e/backtrace.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ fn test_backtrace() {
indoc! {
"Failure data:
(0x454e545259504f494e545f4e4f545f464f554e44 ('ENTRYPOINT_NOT_FOUND'), 0x454e545259504f494e545f4641494c4544 ('ENTRYPOINT_FAILED'))
error occurred in contract 'InnerContract' at pc: '72'
error occurred in contract 'InnerContract' at pc: '[..]'
stack backtrace:
0: backtrace_vm_error::InnerContract::inner_call
at [..]lib.cairo:47:9
Expand All @@ -44,7 +44,7 @@ fn test_backtrace() {
2: backtrace_vm_error::InnerContract::__wrapper__InnerContract__inner
at [..]lib.cairo:37:9

error occurred in contract 'OuterContract' at pc: '107'
error occurred in contract 'OuterContract' at pc: '[..]'
stack backtrace:
0: backtrace_vm_error::IInnerContractDispatcherImpl::inner
at [..]lib.cairo:22:1
Expand Down Expand Up @@ -104,12 +104,12 @@ fn test_backtrace_panic() {
indoc! {
"Success data:
0x61616161 ('aaaa')
error occurred in contract 'InnerContract' at pc: '70'
error occurred in contract 'InnerContract' at pc: '[..]'
stack backtrace:
0: backtrace_panic::InnerContract::__wrapper__InnerContract__inner
at [..]lib.cairo:34:9

error occurred in contract 'OuterContract' at pc: '107'
error occurred in contract 'OuterContract' at pc: '[..]'
stack backtrace:
0: backtrace_panic::IInnerContractDispatcherImpl::inner
at [..]lib.cairo:22:1
Expand Down
17 changes: 4 additions & 13 deletions crates/forge/tests/e2e/running.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,28 +1047,19 @@ fn incompatible_snforge_std_version_warning() {
[..]Compiling[..]
[..]Finished[..]


Collected 4 test(s) from steps package
Running 4 test(s) from src/
[PASS] steps::tests::steps_much_less_than_10000000 [..]
Collected 2 test(s) from steps package
Running 2 test(s) from src/
[PASS] steps::tests::steps_less_than_10000000 [..]
[FAIL] steps::tests::steps_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

[FAIL] steps::tests::steps_much_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

[PASS] steps::tests::steps_less_than_10000000 [..]
Tests: 2 passed, 2 failed, 0 skipped, 0 ignored, 0 filtered out
Tests: 1 passed, 1 failed, 0 skipped, 0 ignored, 0 filtered out

Failures:
steps::tests::steps_more_than_10000000
steps::tests::steps_much_more_than_10000000
"},
);
}
Expand Down
48 changes: 13 additions & 35 deletions crates/forge/tests/e2e/steps.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,39 +20,25 @@ fn should_allow_less_than_default() {
[..]Compiling[..]
[..]Finished[..]

Collected 4 test(s) from steps package
Running 4 test(s) from src/
[FAIL] steps::tests::steps_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

Collected 2 test(s) from steps package
Running 2 test(s) from src/
[FAIL] steps::tests::steps_less_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

[FAIL] steps::tests::steps_much_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

[FAIL] steps::tests::steps_much_less_than_10000000
[FAIL] steps::tests::steps_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

Tests: 0 passed, 4 failed, 0 skipped, 0 ignored, 0 filtered out
Tests: 0 passed, 2 failed, 0 skipped, 0 ignored, 0 filtered out

Failures:
steps::tests::steps_more_than_10000000
steps::tests::steps_less_than_10000000
steps::tests::steps_much_more_than_10000000
steps::tests::steps_much_less_than_10000000
steps::tests::steps_more_than_10000000
"
),
);
Expand All @@ -74,13 +60,11 @@ fn should_allow_more_than_10m() {
[..]Compiling[..]
[..]Finished[..]

Collected 4 test(s) from steps package
Running 4 test(s) from src/
[PASS] steps::tests::steps_much_less_than_10000000 [..]
Collected 2 test(s) from steps package
Running 2 test(s) from src/
[PASS] steps::tests::steps_more_than_10000000 [..]
[PASS] steps::tests::steps_less_than_10000000 [..]
[PASS] steps::tests::steps_much_more_than_10000000 [..]
Tests: 4 passed, 0 failed, 0 skipped, 0 ignored, 0 filtered out
Tests: 2 passed, 0 failed, 0 skipped, 0 ignored, 0 filtered out
"
),
);
Expand All @@ -98,24 +82,18 @@ fn should_default_to_10m() {
[..]Compiling[..]
[..]Finished[..]

Collected 4 test(s) from steps package
Running 4 test(s) from src/
[PASS] steps::tests::steps_much_less_than_10000000 [..]
[FAIL] steps::tests::steps_much_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.

Collected 2 test(s) from steps package
Running 2 test(s) from src/
[PASS] steps::tests::steps_less_than_10000000 (gas: ~[..])
[FAIL] steps::tests::steps_more_than_10000000

Failure data:
Could not reach the end of the program. RunResources has no remaining steps.
Suggestion: Consider using the flag `--max-n-steps` to increase allowed limit of steps

[PASS] steps::tests::steps_less_than_10000000 [..]
Tests: 2 passed, 2 failed, 0 skipped, 0 ignored, 0 filtered out
Tests: 1 passed, 1 failed, 0 skipped, 0 ignored, 0 filtered out

Failures:
steps::tests::steps_much_more_than_10000000
steps::tests::steps_more_than_10000000
"
),
Expand Down
4 changes: 2 additions & 2 deletions crates/forge/tests/integration/fuzzing.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ fn fuzzed_while_loop() {

// TODO (#2926)
assert_eq!(gas_info.min, 2);
assert_eq!(gas_info.max, 23);
assert!(gas_info.max.abs_diff(23) <= 1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

assert!((gas_info.mean - 12.).abs() < f64::EPSILON);
assert!((gas_info.std_deviation - 6.24).abs() < 0.01);
assert!((gas_info.std_deviation - 6.24).abs() < 0.05);
}
Loading