20.10.22-dind image failing with runtime error: index out of range [0] with length 0 or fatal error: concurrent map writes

[gaslitbytech]

Using docker/20.10.22-dind-alpine3.17 we seem to be mostly getting one of the following in our gitlab ci runners.

time="2023-01-06T06:18:51Z" level=warning msg="The \"MAX_ROWS\" variable is not set. Defaulting to a blank string."
time="2023-01-06T06:18:51Z" level=warning msg="`scale` is deprecated. Use the `deploy.replicas` element"
panic: runtime error: index out of range [0] with length 0

goroutine 662509 [running]:
github.com/docker/buildx/build.BuildWithResultHandler.func4.1()
	github.com/docker/[email protected]/build/build.go:897 +0x685
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

or on a different pipeline

time="2023-01-07T00:32:18Z" level=warning msg="`scale` is deprecated. Use the `deploy.replicas` element"
fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 32 [running]:
github.com/docker/compose/v2/pkg/compose.(*composeService).build.func1({0x20a02e0, 0xc0006d4ec0}, {0xc0001dde88?, 0x7?})
    github.com/docker/compose/v2/pkg/compose/build.go:95 +0x652
github.com/docker/compose/v2/pkg/compose.(*graphTraversal).run.func1()
    github.com/docker/compose/v2/pkg/compose/dependencies.go:127 +0x63
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

Once in a while one of our pipelines will pass though it is quite rare.

I noticed that the dockerhub image was updated recently and used this as a reasoning to downgrade to 20.10.21 that hasn't been updated for over a month and this seems better so far.

[asmithdt]

Having the same issue here

[tianon]

I poked some folks internally -- looks like the docker buildx issue is probably related to docker/buildx#1497 (still under active investigation) and the docker compose issue is docker/compose#10150 which is already fixed via docker/compose#10151 and a new Compose release is apparently imminent. 😄

[tianon]

The docker compose half should be fixed by 95414fb (thus docker-library/official-images#13859).

[jedevc]

@tourdownunder @asmithdt for the issue panic: runtime error: index out of range [0] with length 0:

• Could you share the results of your docker buildx ls, as well as docker buildx inspect?
• Is the issue relatively reproducible, and if so, could you share the command you're using?

@tianon on an initial glance, this doesn't seem the same as docker/buildx#1497, though it's a related section of code 😢

[gaslitbytech]

I tried to reproduce it by upgrading back to 20.10.22-dind-alpine3.17 though it passed. Could it be that this version has been updated on dockerhub 21 hours ago?

I found the sha version that failed in the logs: docker@sha256:5ff506edc91614933ae78748222c5e50a0daba30d9f932edd606f789f538e497 though when I try to use that I get another error while the pipeline is setting up.

unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory

[gaslitbytech]

Just reporting back that 20.10.23-dind-alpine3.17 has been fine.

[jedevc]

@tianon I think this should be resolved now 🎉

Still not sure about panic: runtime error: index out of range [0] with length 0, I can't see how we get into that state... No one else has reported it though, but I'll keep an eye out on the buildx repo for it in the future.

[tianon]

Nice, thank you! 🙇 ❤️