cmd/go: toolchain directive in go.mod being updated unnecessarily

[dprotaso]

There doesn't seem to be a way to stop the toolchain directive from being updated when running various go commands

In our OSS project we rely on the go directive in the go.mod to ensure a min go version. We do not want to use the toolchain directive as we always expect it to match the go directive.

some user feedback

• https://gophers.slack.com/archives/C0VPK4Z5E/p1704471476388119
• https://gophers.slack.com/archives/C9BMAAFFB/p1708525315637839

Go Version

go version go1.22.0 darwin/amd64

Go ENV

expand

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/dprotasowski/Library/Caches/go-build'
GOENV='/Users/dprotasowski/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/dprotasowski/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/dprotasowski/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.22.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/Cellar/go/1.22.0/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/dprotasowski/work/test/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xs/xfp1z4cn643c46w69lv8bv8w0000gp/T/go-build4163821748=/tmp/go-build -gno-record-gcc-switches -fno-common'

Scenario 1 - creating a new module and downgrading go

What did you do

$ mkdir test && cd test
$ go mod init blah
$ go get [email protected]

What did you see happen?

• Tool chain directive of 1.22.0 was added - but this isn't necessary
• min go version set to 1.21.7

What did you expect to happen?

• No toolchain directive to be added
• Min go version set to 1.21 - since I didn't specify the point release

Scenario 2 - pulling a new dependency that upgraded the go version

What did you do

Project structure

--- go.mod
module blah

go 1.19

---- main.go
package main

import "fmt"

func main() {
	fmt.Println("vim-go")
}

Fetch a dependency that updates the go min version

$ go get knative.dev/pkg@9f033a7

What did you see happen?

console output

go: upgraded go 1.19 => 1.21
go: added toolchain go1.22.0
go: added knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7

go.mod output

module blah

go 1.21

toolchain go1.22.0

require knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7 // indirect

What did you expect to happen?

toolchain directive shouldn't appear. We want it to always default to the same as the go directive

[bcmills]

Please fill out the complete issue template.

In particular: what version of Go are you using, what was your go directive set to initially, and what commands did you run that added the toolchain directive?

[dprotaso]

@bcmills updated - can you take a look again and see if I'm missing anything?

[dylan-bourque]

Adding my own experiences for reference.

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain). We have many modules with the go directive in their go.mod files targeting Go versions from 1.16 to 1.21. We also do not proactively update those directives across the codebase when a new Go version is released.

This feature started causing issues for us when one engineer using Go 1.21.x locally updated some dependency and ran go mod tidy, which silently updated the go directive in that module and added the new toolchain one. We were all surprised when this change failed to build in CI, which was still using Go 1.20.x.

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine. While that's not an unreasonable stance in the general case, it is a new restriction/requirement that, as far as I've seen, is not called out in the release notes.

We have tried setting GOTOOLCHAIN to local but that doesn't seem to affect the behavior of go mod tidy. It will still upgrade both the go and toolchain directives.

One could also argue that forcing more conservative development shops to always stay on the most current Go release, despite the previous minor release being explicitly supported, is a bad situation.

[bcmills]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.

• go get [email protected] is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

• I don't understand why go get [email protected] is not writing out a toolchain line. That is at the very least a mismatch between the documentation and the implementation.

[bcmills]

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain).

You do have that ability: you can set GOTOOLCHAIN=local in your process environment or GOENV file. (You can even edit $GOROOT/go.env to make that the default for your Go toolchain installation, if you are so inclined.)

You can also force go get to stick to a particular go version, the same way you would for any other dependency that you want to avoid upgrading, by passing that version as an explicit argument to go get: go get [email protected] will downgrade to go 1.20, and go get knative.dev/pkg@9f033a7 [email protected] will correctly error out (and report that those versions are not mutually compatible).

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine.

That is exactly the problem that GOTOOLCHAIN=auto is supposed to mitigate: when everyone is on Go 1.21.0 or above, the developers' machines should automatically download whatever toolchain is needed to work in the module.

[dylan-bourque]

You do have that ability: you can set GOTOOLCHAIN=local in your process environment

This works, but it also means that as soon as anyone moves to a newer Go release then everyone else in the organization must also move to that same newer version immediately. This is a problem in tightly regulated environments where, as an example Go 1.22.x may not be "certified" but Go 1.21.5 is. As it stands today it's not possible to both fix CI to 1.21.x and also allow developers to use a newer 1.21.y or 1.22.

You can also force go get to stick to a particular go version ... by passing that version as an explicit argument to go get: go get [email protected]

TIL. Thanks for this.

[cevich]

you can set GOTOOLCHAIN=local in your process environment

There are environments where setting an env. var. is not possible. For example, the public Renovate service does not support this. There are other ways to pin the golang version in Renovate, but they're undesirable for maintainability reasons. Devcontainers could be another example, where the tooling env. is shared, and must remain static, for consistency across a whole team of developers. In any case, my point is there are places where setting env. vars. isn't a viable workaround. IMHO such major behavior changes should almost always default to the previous or "least disruptive" option.

[gaby]

I just ran into this same issue, go1.21 is supported but my box has go1.22. I can't run any commands because golang keeps forcing the toolchain directive into the go.mod file.

This is for a library used by a lot of users and it should support go1.21 and go1.22

[XANi]

Still a problem, it just makes a complete mess any time common CI server is used by few people

[dmitshur]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command? For example:

$ cat go.mod
module example.com

go 1.21.0

$ go get [email protected] toolchain@none
go: upgraded go 1.21.0 => 1.22.6

That comes with the disadvantages of not having the toolchain version written down, but that's no different from the outcome where it's manually removed after go get adds it.

[prymitive]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

A while ago one would set the go field in go.mod to specify the language version the project requires at minimum. The change to use full go release as the value of go, rather than just the major/minor release numbers as before, means that every single module might specify a different point release when in reality all patch releases of Go X.Y are guaranteed to be compatible - and if they are not that would be for some very good reason and is rare.
So every library out there putting a different point release in their go.mod combined with this automatic bumping of go / toolchain means there's a lot of noise in PR diffs that does not (in my opinion) adds any value at all. It's just churn.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.
• go get [email protected] is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

A while ago go.mod would be the files that's maintained / validated by the code author to write down requirements and constrains of the project. The go.sum file would be the lock file that holds all the low level details of how to resolve this constrains / requirements.
With this automatic bumping of go and insertion/updates of toolchain go.mod starts to also be a mini-lock file that encodes some of that information, which I think is unfortunate.
The decision for all of that happens automatically, unless the use puts in place enough env variables or files that stops go command from doing all of this means that there's by default a magic happening where it wasn't before, and maybe all of that will get easier and more obvious over time, but at present (for me personally) it took away simplicity of go.mod away while not giving me anything in return.

[dnwe]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command?

Thanks for posting this. That's exactly the behaviour I (and probably others) had been looking for and was originally the behaviour that I thought that GOTOOLCHAIN=local would provide for me — i.e., always use the toolchain I have installed locally and never add a toolchain directive in the go.mod file. I always have the most recent release kept up-to-date locally, but I don't want that to keep getting stamped into the go.mod when its unnecessary.

Certainly for my own individual modules I'd likely move to a model of always doing:
go get toolchain@none go@{{- .GoLangVersion }}.0 <otherpkg> [otherpkgs...] to retain the behaviour of the go directive declaring the minimum language and stdlib version required to build the module, and then I'm happy for whatever toolchain >= to that version to be used at build time.

Ideally I'd want to bake that client behaviour into the go.mod file so that when anyone else updates a dependency, it retains the go directive and doesn't add an arbitrary toolchain directive based on whatever version they happened to have installed locally.

[matthewhartstonge]

I would argue that by adding toolchain as a non opt-in feature breaks user expectations around Go's release policy due to shadow forcing people on to whatever latest version is injected into go.mod.

Release Policy¶
Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release. We fix critical problems, including critical security problems, in supported releases as needed by issuing minor revisions (for example, Go 1.6.1, Go 1.6.2, and so on).

I have found this really painful for library developers as well. I've had to ask maintainers to lower the versioning or remove the toolchain directive in a popular OSS lib to support said release policy.

As with others, I'm finding I'm constantly wasting time having to unwind/remove the toolchain directive as we run a n-1 Go versioning policy at our place of work.

[anacrolix]

This is a massive nuisance, and GOTOOLCHAIN settings don't seem to support a mode that doesn't update it.

[dprotaso]

This is a massive nuisance, and GOTOOLCHAIN settings don't seem to support a mode that doesn't update it.

We've been using GOTOOLCHAIN=local

[anacrolix]

So have I but it still likes to blast the go.mod file.

[matloob]

@matthewhartstonge I'd like to understand the issue with library modules better. Is the issue that when a library module has a toolchain directive, it may not be tested with older Go versions in CI and may develop incompatibilities?

[dnwe]

@matloob i think he was conflating the go and toolchain directives there. Where the go directive implicitly also becomes the minimum toolchain version for all consuming modules

See sigstore/rekor#2323 and similar PRs that aim to rectify this. As soon as one module bumps their go directive up then it’s a cascading ripple across chains of modules, forcing everyone to bump to match.

Ideally the Go directive would never have become a toolchain version and would have remained as purely major language level

[matthewhartstonge]

@matloob yeah, it is a conflation - and [my understanding] is that most Go developers won't/don't know the difference. Even massive projects have been affected by this (k8s, gRPC and Google's own API SDKs). The conflation comes because the recognisable pattern developers hook on to is that a new toolchain directive appears whenever this occurs.

As @dnwe mentioned, the change to the go directive/toolchain has become a pervasive change that has affected the whole dependency ecosystem, forcing dependants to bump project go versions.

Personally, I've ended up spending many hours with CI breaking and having to request/informing upstream library authors that they've broken go version compatibility since this change has come in - for example if an external dependency/library happens to set the latest version of go that dropped that week.

Due to our business' specific requirements with government, security and our ISO response to CVEs, It's easier to forefront these changes. When a dependency creates a new release, dependabot will pick up and explode anytime the go directive is changed even if the API is compatible. We've locked our lint, build, and SAST tools to the latest "n-1" version of Go.

If the go directive would only allow specifying minor versions i.e. 1.22, 1.24 and the toolchain could be used to specify a specific patch version (unless overridden by GOTOOLCHAIN=local) I think it may help some of us to sleep better at night.

@cevich explains it well in relation to CI: #65847 (comment) (RenovateBot being the cause of the recent go directive updates across the Google Cloud Go SDKs)

Hope that helps bring some understanding?

[matloob]

@matthewhartstonge It would be very helpful to have a video or phone chat about this if you're able.

[matthewhartstonge]

@matloob Hit me up on matt <at> mykro dot co dot nz :)

[anacrolix]

Yeah my reading from the docs and the announcement was the toolchain directive would for users that have it configured, pick the best toolchain to use automatically. It is independent of the go directive. But in practice, it just seems to additionally replace the go directive which becomes unusable as the toolchain is used for everything.

[nyetwurk]

golangci-lint makes this even more annoying... suddenly it will fail if you don't keep it updated with your toolchain directive.

golangci-lint run --timeout 10m
Error: can't load config: the Go language version (go1.23) used to build golangci-lint is lower than the targeted Go version (1.24.0)
Failed executing command with error: can't load config: the Go language version (go1.23) used to build golangci-lint is lower than the targeted Go version (1.24.0)

This means you cannot pin golangci-lint in your CI either, because you never know what toolchain renovate will pick for your project.

[gopherbot]

Change https://go.dev/cl/656835 mentions this issue: cmd/go: don't always write toolchain line when updating go line