Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky smoketest due to race condition in subscribe #1656

Merged
merged 3 commits into from
Aug 29, 2024

Conversation

coolreader18
Copy link
Collaborator

This should fix it - now it'll only error if the initial update was never printed, i.e. the subscription never started. I'm about 90% sure that this should fix any race condition issues, but if there's any flakiness we can just go back to the fix from before.

@coolreader18 coolreader18 requested a review from bfops August 29, 2024 18:00
bfops added a commit that referenced this pull request Aug 29, 2024
…ert the always-wait behavior of smoketest from #1536
@bfops bfops force-pushed the noa/flaky-smoketest-subscribe branch from 13975d1 to fe7f18e Compare August 29, 2024 18:06
@@ -72,7 +72,7 @@ def test_add_then_remove_index(self):
# There are no indices, resulting in an unsupported unindexed join.
self.publish_module(name, clear = False)
with self.assertRaises(Exception):
self.subscribe(self.JOIN_QUERY, n = 0)()
self.subscribe(self.JOIN_QUERY, n = 0)
Copy link
Collaborator

@bfops bfops Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(if we've fixed the race condition, these changes from #1536 are no longer necessary)

code = proc.poll()
if code:
raise subprocess.CalledProcessError(code, fake_args)
print("no inital update, but no error code either")
Copy link
Collaborator

@bfops bfops Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to double-check - is it intended that we trigger this both if the code is None and if the code is 0?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Maybe it should check for 0, but it doesn't make much of a difference anyway.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair, I guess it just falls through to a thread that immediately returns below

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh - it should wait. I see the potential race condition here.

Copy link
Collaborator

@bfops bfops Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that seems right. I suspect it's basically when the process hasn't quite exited yet, but I was unable to reproduce locally so I couldn't really narrow done.

Copy link
Collaborator

@bfops bfops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this! Seems worth trying. It's definitely nicer UX if it's free of race conditions.

@coolreader18 coolreader18 enabled auto-merge August 29, 2024 18:50
@coolreader18
Copy link
Collaborator Author

Ran it test_add_then_remove_index 20 times with no failure, so it seems fine now, hopefully.

Copy link
Collaborator

@bfops bfops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM let's give it a shot

@coolreader18 coolreader18 added this pull request to the merge queue Aug 29, 2024
Merged via the queue into master with commit f7ae920 Aug 29, 2024
8 checks passed
@coolreader18 coolreader18 deleted the noa/flaky-smoketest-subscribe branch September 23, 2024 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants