cmd/go: allow 'go mod tidy' to proceed despite errors

[bcmills]

When adding module definitions to existing repositories, we may (often?) find that one or more existing packages do not actually build, or that some import cannot be resolved due to a bug (e.g., #26602).

Failure to build is especially likely for files with rarely-used build constraints, but rarely-used constraints are exactly the space that go mod -sync scans.

If the user needs to intervene to fix something, it would be nice if they could generate the rest of the module definition before they start trying to troubleshoot one or two packages that still won't build. For example, here is the output of go mod -sync for golang.org/x/build: it's a huge dependency graph, and seems to be held back by one missing import.

~/src/golang.org/x/build$ go mod -sync
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/time/rate latest
go: finding google.golang.org/api/googleapi latest
go: finding golang.org/x/time latest
go: finding google.golang.org/api latest
go: finding golang.org/x/sync/errgroup latest
go: finding google.golang.org/api/oauth2/v2 latest
go: finding golang.org/x/sync latest
go: finding google.golang.org/api/oauth2 latest
go: finding github.com/golang/protobuf/ptypes/timestamp latest
go: finding github.com/golang/protobuf/ptypes latest
go: finding google.golang.org/api/option latest
go: finding google.golang.org/genproto/googleapis/api/label latest
go: finding google.golang.org/genproto/googleapis/api latest
go: finding google.golang.org/genproto/googleapis latest
go: finding google.golang.org/genproto latest
go: finding grpc.go4.org latest
go: finding google.golang.org/appengine/urlfetch latest
go: finding google.golang.org/appengine/datastore latest
go: finding go4.org/types latest
go: finding go4.org/strutil latest
go: finding google.golang.org/api/deploymentmanager/v2 latest
go: finding go4.org latest
go: finding google.golang.org/api/deploymentmanager latest
go: finding github.com/google/go-github/github latest
go: finding google.golang.org/appengine/memcache latest
go: finding golang.org/x/perf/storage latest
go: finding github.com/gregjones/httpcache latest
go: finding golang.org/x/perf latest
go: finding google.golang.org/genproto/googleapis/monitoring/v3 latest
go: finding google.golang.org/appengine/delay latest
go: finding google.golang.org/genproto/googleapis/monitoring latest
go: finding golang.org/x/crypto/ssh latest
go: finding golang.org/x/crypto latest
go: finding golang.org/x/crypto/acme/autocert latest
go: finding golang.org/x/net/http2 latest
go: finding golang.org/x/crypto/acme latest
go: finding golang.org/x/net latest
go: finding github.com/davecgh/go-spew/spew latest
go: finding github.com/tarm/serial latest
go: finding google.golang.org/appengine/log latest
go: finding github.com/shurcooL/httpgzip latest
go: finding golang.org/x/sys/unix latest
go: finding google.golang.org/api/iterator latest
go: finding golang.org/x/sys latest
go: finding github.com/golang/protobuf/proto latest
go: finding github.com/shurcooL/issues latest
go: finding github.com/shurcooL/issues/maintner latest
go: finding go4.org/syncutil latest
go: finding google.golang.org/api/compute latest
go: finding github.com/jellevandenhooff/dkim latest
go: finding golang.org/x/tools/go/vcs latest
go: finding github.com/google/go-cmp/cmp latest
go: finding golang.org/x/tools/go latest
go: finding github.com/bradfitz/go-smtpd/smtpd latest
go: finding golang.org/x/oauth2/google latest
go: finding github.com/bradfitz/go-smtpd latest
go: finding golang.org/x/tools latest
go: finding golang.org/x/net/context/ctxhttp latest
go: finding google.golang.org/api/container latest
go: finding golang.org/x/net/context latest
go: finding golang.org/x/net/nettest latest
go: finding google.golang.org/genproto/googleapis/api/metric latest
go: finding github.com/shurcooL/issuesapp latest
go: finding github.com/coreos/go-systemd/activation latest
go: finding github.com/coreos/go-systemd/daemon latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/shurcooL/gofontwoff latest
go: finding cloud.google.com/go/monitoring/apiv3 latest
go: finding cloud.google.com/go/storage latest
go: finding cloud.google.com/go/bigquery latest
go: finding cloud.google.com/go/errorreporting latest
go: finding cloud.google.com/go/datastore latest
go: finding cloud.google.com/go/monitoring latest
go: finding cloud.google.com/go/compute/metadata latest
go: finding cloud.google.com/go/compute latest
go: finding github.com/shurcooL/github_flavored_markdown latest
go: finding github.com/shurcooL/htmlg latest
go: finding github.com/shurcooL/users latest
go: finding github.com/shurcooL/octicon latest
go: finding github.com/shurcooL/reactions/component latest
go: finding github.com/shurcooL/notifications latest
go: finding github.com/dustin/go-humanize latest
go: finding github.com/shurcooL/reactions latest
go: finding google.golang.org/grpc/metadata latest
go: finding google.golang.org/grpc/codes latest
go: finding github.com/shurcooL/httpfs/html/vfstemplate latest
go: finding github.com/shurcooL/httpfs/html latest
go: finding github.com/google/go-querystring/query latest
go: finding github.com/shurcooL/httpfs latest
go: finding google.golang.org/grpc/status latest
go: finding github.com/shurcooL/httpfs/union latest
go: finding github.com/google/go-querystring latest
go: finding go.opencensus.io/plugin/ocgrpc latest
go: finding go.opencensus.io/stats/view latest
go: finding go.opencensus.io/trace latest
go: finding golang.org/x/text/unicode/norm latest
go: finding go.opencensus.io/plugin latest
go: finding go.opencensus.io/stats latest
go: finding github.com/shurcooL/github_flavored_markdown/gfmstyle latest
go: finding golang.org/x/text/unicode latest
go: finding google.golang.org/grpc/credentials latest
go: finding google.golang.org/grpc/credentials/oauth latest
go: finding google.golang.org/grpc/naming latest
go: finding github.com/shurcooL/events latest
go: finding github.com/shurcooL/events/event latest
go: finding golang.org/x/text/unicode/bidi latest
go: finding go.opencensus.io/plugin/ochttp latest
go: finding golang.org/x/text/secure/bidirule latest
go: finding github.com/anmitsu/go-shlex latest
go: finding github.com/shurcooL/webdavfs/vfsutil latest
go: finding go.opencensus.io/exporter/stackdriver/propagation latest
go: finding go.opencensus.io/exporter/stackdriver latest
go: finding github.com/shurcooL/webdavfs latest
go: finding go.opencensus.io/exporter latest
go: finding golang.org/x/text/secure latest
go: finding github.com/shurcooL/httperror latest
go: finding github.com/shurcooL/go/gopherjs_http latest
go: finding dmitri.shuralyov.com/state latest
go: finding github.com/shurcooL/go latest
go: finding github.com/google/martian/httpspec latest
go: finding dmitri.shuralyov.com/html/belt latest
go: import "golang.org/x/build/autocertcache" ->
	import "cloud.google.com/go/storage" ->
	test ->
	import "cloud.google.com/go/httpreplay" ->
	import "cloud.google.com/go/httpreplay/internal/proxy" ->
	import "github.com/google/martian/httpspec": cannot find module providing package github.com/google/martian/httpspec

[rogpeppe]

There's a significant example of this that I encountered recently when using TinyGo. TinyGo has some additional packages in its standard library (for example, the "machine" package). Because those packages don't exist in the regular Go standard library, go mod tidy just gives up, which means that it's not possible to use it to tidy up, despite that other Go module-related commands can be used OK.

Here's a testscript example to demonstrate the issue:

go mod tidy

-- go.mod --
module example

go 1.15
-- main.go --
package main

import _ "machine"

func main() {
}

The go command fails with this error:

example imports
	machine: package machine is not in GOROOT (/home/rog/go/src/machine)

[bcmills]

@rogpeppe, do the TinyGo source files have a particular build constraint? If so, the narrower fix described in #40067 (comment) might be more appropriate.

[rogpeppe]

I showed a complete example above. No build constraints involved, I'm afraid.

[bcmills]

Then I would argue the defect in that example is in TinyGo, not cmd/go. A program that expects a different version of the standard library should have a distinguished build tag (as we do for mainline Go releases) or a distinguished go version in the go.mod file.

If the program by all accounts indicates that it expects to be compiled in a standard go 1.15 environment, then the standard go command should diagnose issues that would prevent its dependencies from being loaded in a standard go 1.15 environment. (Build constraints exist for a reason.)

[rogpeppe]

FWIW there's no problem with using the standard go command for this code (I can happily go get etc) but only go mod tidy as far as I can tell.

Since one of the aims of go mod tidy is to evaluate all Go code regardless of build tag, I don't really see why a build tag should affect things much (and it doesn't).

For my particular problem, it seems to me that go mod tidy could see the stdlib import (easy to tell because the first element of the path has no dots) and ignore the error because no stdlib import can ever imply an external dependency.

[bcmills]

no stdlib import can ever imply an external dependency

Unfortunately, that does not hold in the presence of replace directives. (In particular, the missing import could be one that the user intended — but forgot — to shim in with a replace.)

[rogpeppe]

Unfortunately, that does not hold in the presence of replace directives

You can use a replace directive to replace a stdlib package? Interesting. I had no idea.

[bcmills]

You can't use it to replace a package that actually exists in the standard library, but you can use it to slot in a module whose path would normally be reserved for the standard library (just like you can do by manually poking directories into a GOPATH/src tree).

[gopherbot]

Change https://golang.org/cl/255960 mentions this issue: cmd/go: add a '-e' flag to 'mod tidy' and 'mod vendor'

[gopherbot]

Change https://golang.org/cl/287413 mentions this issue: content/static/doc: document -e flag for tidy and vendor