proxy.golang.org: reports of failing requests in the southeastern US area

[heschi]

Edit: We believe the issue is resolved as of 8AM US Eastern time.

proxy.golang.org's monitoring, confirmed by widespread user reports, shows that Google is having an infrastructure failure that causes requests to proxy.golang.org to fail. Unfortunately there's nothing the Go team can directly do to fix this problem, but we are monitoring it and escalating internally.

[heschi]

As a workaround, users can set GOPROXY=direct to bypass the mirror. However, that won't help for modules that happen to be hosted on Google's infrastructure, e.g. golang.org/x modules.

[Raffo]

@heschi thank you for opening this issue and for reaching out internally to get it fixed. I have one question: would it be possible to have a status page up for proxy.golang.org for the community to check in case an outage is suspected? I spent quite some time figuring out what was wrong with some of our builds and a status page would have been handy.

[heschi]

@Raffo That's fair. We should definitely consider it. I believe this is the first really major outage we've had in the ~4 years since the proxy launched, so we don't have a lot of practice managing them. Thanks for bearing with us.

[or-shachar]

Is there a a status page for the Google infrastructure service that is malfunctioning ?

[OrBin]

Thanks for the report, just want to note that it's happening not only in southeastern US, but also in Europe and the Middle East (Israel).

[xymon2]

Asia(Korea) has same problem too.

[avanier]

I'll add another report from AWS US-East-1.

[heschi]

@or-shachar not directly. If it affected a major Google product you might see the status page for those products updated, but it appears that that's not the case. proxy.golang.org's use case is different from most Google services and it's possible that a common outage is affecting us worse than others.

@OrBin @xymon2 That's quite surprising based on my current understanding of the issue. Thank you.

[heschi]

All: please do not post +1s for the eastern US. I appreciate new information, but we are well aware of the issues in the eastern US :)

[heschi]

The proxy.golang.org front page should start showing a message pointing to this issue soon. Unfortunately it's cached for 30 minutes, so it'll take a while to be visible everywhere, but it is better than nothing. Thanks again for bearing with us.