Description
Go version
go version go1.22.1 linux/arm64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/timer/.cache/go-build'
GOENV='/home/timer/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/timer/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/timer/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go1.22.1.linux-arm64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go1.22.1.linux-arm64/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/timer/Development/proxy/https-terminator/go.mod'
GOWORK='/home/timer/Development/proxy/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1654315752=/tmp/go-build -gno-record-gcc-switches'What did you do?
We run a HTTP/2 server that flushes an early headers map (status 103 - http.StatusEarlyHints) using func (http.ResponseWriter) WriteHeader(statusCode int).
After calling WriteHeader, we may need to mutate the Header map for the final 2xx response based on information we discover. In our application, we simply clear the Headers after flushing the early hint:
r.W.Header().Set("Server", "...")
r.W.Header().Set("Link", "...")
r.W.WriteHeader(http.StatusEarlyHints)
// remove the headers after sending the early hints to avoid
// sending duplicate headers when the final response is sent:
r.W.Header().Del("Server")
r.W.Header().Del("Link")
What did you see happen?
After >60 minutes of runtime and millions of requests (this is hard to reproduce), we were faced with this crash:
fatal error: concurrent map read and map write
goroutine 44382232 [running]:
golang.org/x/net/http2.encodeHeaders(0xc09549f798?, 0x1412ef5?, {0x0?, 0xc0416e72b9?, 0xc07fbd2f18?})
golang.org/x/[email protected]/http2/write.go:343 +0x156
golang.org/x/net/http2.(*writeResHeaders).writeFrame(0xc04d077c00, {0x15edb08, 0xc08528a000})
golang.org/x/[email protected]/http2/write.go:217 +0xe9
golang.org/x/net/http2.(*serverConn).writeFrameAsync(0xc08528a000, {{0x15e9140?, 0xc04d077c00?}, 0xc01a73d860?, 0xc0326e0c60?}, 0xc070d666a8?)
golang.org/x/[email protected]/http2/server.go:852 +0x74
created by golang.org/x/net/http2.(*serverConn).startFrameWrite in goroutine 44380821
golang.org/x/[email protected]/http2/server.go:1266 +0x390
goroutine 1 [semacquire, 87 minutes]:
This appears to be due to writeFrameAsync running on a separate goroutine and the encodeHeaders function doing a naive iteration over the values (vv := h[k], line 343 from above).
What did you expect to see?
Encoding headers needs to be concurrent-safe now that a server can send and mutate its headers multiple times before sending the final response (e.g. 103, 103, 103, 200).