Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug fix] tls reload: enable for client cert #42

Merged
merged 1 commit into from
Jan 2, 2025

Conversation

huikang
Copy link
Contributor

@huikang huikang commented Dec 16, 2024

This PR fixes a bug in reloading the tls certificate for the gRPC client. Gubernator instance reports messages remote error: tls: expired certificate after running for a couple of days.

What caused the problem

in the current implementation, the client certificate of the key reload refers to a slice of tls.Certificate struct. During reload, there could be a chance the certificate is not loaded correctly, leading to the tls expiration error.

How the PR fixes the issue

We use the tls.GetClientCertificates callback function to load the certificate for gRPC client to avoid any race condition. Additional improvements are that hot-reload of client cert is supported as well in this PR.

Test

We deployed an image with the fix in 3-node cluster and let it run for 5 days with tls ttl=5m. There has been no error of tls expiration.

@huikang huikang requested a review from Baliedge as a code owner December 16, 2024 15:37
- remove unused fields in TLSConfig struct
@huikang huikang force-pushed the hk/fix-tls-client-cert branch from 82762bf to b7ea1f6 Compare December 16, 2024 16:03
@huikang huikang changed the title tls reload: enable for client cert [Bug] tls reload: enable for client cert Dec 16, 2024
@huikang huikang changed the title [Bug] tls reload: enable for client cert [Bug fix] tls reload: enable for client cert Dec 16, 2024
@huikang
Copy link
Contributor Author

huikang commented Jan 2, 2025

@thrawn01 , do you mind taking a look at this PR?
We are certain that the issue can be quickly reproduced with ttl=5m. The version with the PR has been deployed over 3 weeks and has not issue so far.

@@ -109,12 +109,6 @@ type TLSConfig struct {
// (Optional) The client auth CA Certificate in PEM format. Used if ClientAuthCaFile is unset.
ClientAuthCaPEM *bytes.Buffer

// (Optional) The client auth private key in PEM format. Used if ClientAuthKeyFile is unset.
ClientAuthKeyPEM *bytes.Buffer
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fields are not used throughout the code

@thrawn01 thrawn01 self-assigned this Jan 2, 2025
Copy link
Collaborator

@thrawn01 thrawn01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry this took so long, thank you for mentioning me on this PR. I'll check notifications for this repo, some how I didn't see this PR until now.

@thrawn01 thrawn01 merged commit e3b65fc into gubernator-io:master Jan 2, 2025
4 checks passed
@huikang
Copy link
Contributor Author

huikang commented Jan 2, 2025

@thrawn01 , thanks for merging. I guess we need a new minor release for the fix.

@thrawn01
Copy link
Collaborator

thrawn01 commented Jan 2, 2025

https://github.com/gubernator-io/gubernator/releases/tag/v2.10.0

@huikang huikang deleted the hk/fix-tls-client-cert branch January 8, 2025 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants