fix(election): retry participating if the lease is improperly lost

[jlevesy]

What Does This PR do?

This PR fixes an issue where a prometheus-elector instance could stop participating into the election indefinitely if it loses (ie: without releasing it explicitly) the election lease.

In that case, the LeaderElector.Run method (see here) exits, this is actually documented 🤦 .

But the previous implementation didn't account for that and it was leading to the actual member not taking part into the election anymore.

This PR fixes this issue by checking if the given runCtx is done or not after Run exits.

• If the context is not done, it means that we're still supposed to participate to the election, so we're recalling elector.Run to join back the election again.
• If the context is done, then it means that we're in a proper termination. So it signals its termination and exits.

How to Test This PR?

# Run the agent example
make run_agent_example

# In another terminal monitor the logs
stern .

# In a third terminal, set the `spec.holderIdentity` field to  ` prometheus-elector-dev-1`
k edit lease prometheus-elector-lease

# You should see elector-1 take the ownership, and elector-0 drop it
# after a while and then start grabbing again the lease.
# Then do the opposite operation, and you'll see the roles reversing. On the current main, prometheus-elector-0 would have been dead in the water.

Good PR Checklist

• Addresses one issue
• Adds/Updates unit tests
• Adds/Updates the documentation
• Opened against the right branch
• Correctly Labeled

Additional Notes