Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: Improve integration tests to reduce flakyness #928

Merged
merged 4 commits into from
Feb 12, 2021
Merged

test: Improve integration tests to reduce flakyness #928

merged 4 commits into from
Feb 12, 2021

Conversation

jan-auer
Copy link
Member

@jan-auer jan-auer commented Feb 12, 2021 •
edited
Loading

  • Fixes python deprecation warnings, including an upgrade of confluent-kafka
  • Removes consumer config attributes from Kafka producer
  • Send a test message on Kafka consumer creation to wait for the connection
  • Use the same test message to assert a topic is empty instead of polling with timeout
  • Reduce default poll timeouts to speed up tests
  • Increase poll timeout where UA parsing occurs. Relay needs to lazy initialize the UA parser, which takes a few seconds in debug mode.
  • Remove user agents from tests that don't need them. This speeds up tests.
  • For tests that expect errors in mini_sentry, clear them in a finally block so that the original assertion is not hidden.
  • Increase the rate limiting window in test_processing_quotas to 1d, reducing the chance of wrap-around during the test.
  • Only check for one error in test_forced_shutdown, the other is only raised if the event is sending at time of shutdown, which is not deterministic.

#skip-changelog

@jan-auer jan-auer self-assigned this Feb 12, 2021
@jan-auer jan-auer requested a review from a team February 12, 2021 12:29
Swatinem
Swatinem approved these changes Feb 12, 2021
View reviewed changes
Copy link
Member

@Swatinem Swatinem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Admittedly, I know very little about the test setup here, but changes look good also considering the description

@jan-auer
Copy link
Member Author

jan-auer commented Feb 12, 2021 •
edited
Loading

The shutdown test is still flaky, both in the integration test suite and the unit test. It seems to be failing more often on CI now, but I cannot reproduce the failure locally as easily.

@jan-auer jan-auer merged commit c287218 into master Feb 12, 2021
@jan-auer jan-auer deleted the test/fix-flakey branch February 12, 2021 15:46
jan-auer added a commit that referenced this pull request Feb 26, 2021
@jan-auer
Merge branch 'master' into release/docker-openssl-vendored
ae36f35 
* master: (23 commits)
  feat(server): Add inbound filters functionality to dynamic sampling (#920)
  fix rule generation to new format (in tests) (#939)
  ref: Remove actix-web client, make reqwest the new http client (#938)
  fix(server): Reject empty envelopes with an explicit error (#937)
  fix(server): Ensure outcome data categories are numeric (#936)
  feat(stats): Add data category to TrackOutcome (#931)
  fix(pii): Make username pii-strippable (#935)
  build: Bump symbolic to 8.0.4 (#899)
  fix: Update to tokio 1.0 (#909)
  fix(pii): Make and/or work correctly with pii=maybe (#932)
  release: 21.2.0
  release: 0.8.3
  fix(build): Fix path to 3.7 (#930)
  build: drop python 2.7 support (#929)
  feat(protocol): Add snapshot to the stack trace interface (#927)
  test: Improve integration tests to reduce flakyness (#928)
  fix: Ignore false-positive lints from Clippy 1.50 (#926)
  feat(protocol): Add NSError to mechanism (#925)
  meta: Fix changelog
  meta(readme): Document development config for processing mode (#921)
  ...
jan-auer added a commit that referenced this pull request Feb 26, 2021
@jan-auer
Merge branch 'master' into fix/reject-release-backslash
6715e92 
* master: (42 commits)
  build: Switch to vendored openssl build (#914)
  feat(server): Add inbound filters functionality to dynamic sampling (#920)
  fix rule generation to new format (in tests) (#939)
  ref: Remove actix-web client, make reqwest the new http client (#938)
  fix(server): Reject empty envelopes with an explicit error (#937)
  fix(server): Ensure outcome data categories are numeric (#936)
  feat(stats): Add data category to TrackOutcome (#931)
  fix(pii): Make username pii-strippable (#935)
  build: Bump symbolic to 8.0.4 (#899)
  fix: Update to tokio 1.0 (#909)
  fix(pii): Make and/or work correctly with pii=maybe (#932)
  release: 21.2.0
  release: 0.8.3
  fix(build): Fix path to 3.7 (#930)
  build: drop python 2.7 support (#929)
  feat(protocol): Add snapshot to the stack trace interface (#927)
  test: Improve integration tests to reduce flakyness (#928)
  fix: Ignore false-positive lints from Clippy 1.50 (#926)
  feat(protocol): Add NSError to mechanism (#925)
  meta: Fix changelog
  ...
jan-auer added a commit that referenced this pull request Feb 26, 2021
@jan-auer
Merge branch 'master' into ref/get-value-macro
e4c3e7d 
* master: (120 commits)
  fix(protocol): Deny backslashes in release names (#904)
  ref(server): Extract dynamic sampling functionality into own crate (#940)
  build: Switch to vendored openssl build (#914)
  feat(server): Add inbound filters functionality to dynamic sampling (#920)
  fix rule generation to new format (in tests) (#939)
  ref: Remove actix-web client, make reqwest the new http client (#938)
  fix(server): Reject empty envelopes with an explicit error (#937)
  fix(server): Ensure outcome data categories are numeric (#936)
  feat(stats): Add data category to TrackOutcome (#931)
  fix(pii): Make username pii-strippable (#935)
  build: Bump symbolic to 8.0.4 (#899)
  fix: Update to tokio 1.0 (#909)
  fix(pii): Make and/or work correctly with pii=maybe (#932)
  release: 21.2.0
  release: 0.8.3
  fix(build): Fix path to 3.7 (#930)
  build: drop python 2.7 support (#929)
  feat(protocol): Add snapshot to the stack trace interface (#927)
  test: Improve integration tests to reduce flakyness (#928)
  fix: Ignore false-positive lints from Clippy 1.50 (#926)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Swatinem Swatinem Swatinem approved these changes

@RaduW RaduW RaduW approved these changes

Assignees

@jan-auer jan-auer

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jan-auer @Swatinem @RaduW