Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: update OpenBSD builders to 7.6 #69823

Open
4a6f656c opened this issue Oct 9, 2024 · 15 comments
Open

x/build: update OpenBSD builders to 7.6 #69823

4a6f656c opened this issue Oct 9, 2024 · 15 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-OpenBSD
Milestone

Comments

@4a6f656c
Copy link
Contributor

4a6f656c commented Oct 9, 2024

OpenBSD 7.6 has just been released (https://www.openbsd.org/76.html) - this means that the only two supported OpenBSD releases are now 7.5 and 7.6. The current openbsd-amd64 and openbsd-386 builders are running 7.2 which is unsupported. As such, these should be updated to run 7.6.

@4a6f656c 4a6f656c added OS-OpenBSD Builders x/build issues (builders, bots, dashboards) labels Oct 9, 2024
@gopherbot gopherbot added this to the Unreleased milestone Oct 9, 2024
@4a6f656c
Copy link
Contributor Author

4a6f656c commented Oct 9, 2024

Some useful details also in #63568 (comment)

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/618895 mentions this issue: env/openbsd-amd64: update build image script to OpenBSD 7.6

@cherrymui cherrymui added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 9, 2024
gopherbot pushed a commit to golang/build that referenced this issue Oct 22, 2024
OpenBSD 7.6 has been released, meaning that the only two supported
versions are 7.5 and 7.6. Update the image build script for 7.6.

Additionally, bump the maxproc limit for the swarming user, which
will hopefully reduce the number of failures due to fork/exec
returning EAGAIN.

Updates golang/go#66474
Updates golang/go#69823

Change-Id: Ie2667de2defefbb6a56adca3dec92d7663c615e4
Reviewed-on: https://go-review.googlesource.com/c/build/+/618895
Reviewed-by: Cherry Mui <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@4a6f656c
Copy link
Contributor Author

4a6f656c commented Jan 1, 2025

@dmitshur any idea when/how we can get this rolled out?

@dmitshur
Copy link
Contributor

dmitshur commented Jan 4, 2025

For the 386/amd64 builders that we run as VMs in GCE, this will first need to go through a process similar to what was recently done in #61095 (e.g., see CL 616836 which added the ability to bootstrap the swarming bot for LUCI), which is tracked in #61122. I'll increase the visibility of that issue within our list of available builder-related work, but I can't say how quickly someone will be able to get to it.

For reverse builders owned by @golang/openbsd builder owners, it's a matter of updating the OS version where the swarming bot runs.

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Jan 4, 2025

For the 386/amd64 builders that we run as VMs in GCE, this will first need to go through a process similar to what was recently done in #61095 (e.g., see CL 616836 which added the ability to bootstrap the swarming bot for LUCI), which is tracked in #61122. I'll increase the visibility of that issue within our list of available builder-related work, but I can't say how quickly someone will be able to get to it.

I may be missing something, however the openbsd/amd64 builder is already running with LUCI - https://go.dev/cl/618895 updated the image build script to produce a 7.6 image and attempts to fix the current flakiness, however AFAIK I have no ability to deploy that. This is the one that most urgently needs attention since it is flakey and has let to issues being missed (e.g. #70880).

The openbsd/386 builder is not yet on LUCI.

For reverse builders owned by @golang/openbsd builder owners, it's a matter of updating the OS version where the swarming bot runs.

They're all already on OpenBSD 7.6, with the exception of openbsd/arm that is still on 7.5
and openbsd/ppc64 that is running snapshots that are newer than 7.6.

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Jan 4, 2025

Additionally, as I just noted on #61122, as far as I can see the openbsd/386 image should already be LUCI capable (since openbsd/amd64 and they share the same build script).

@dmitshur dmitshur moved this to Planned in Go Release Feb 27, 2025
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/653496 mentions this issue: main.star: add openbsd-amd64_7.6

gopherbot pushed a commit to golang/build that referenced this issue Feb 28, 2025
For golang/go#69823.

Change-Id: I9b9aed05dd12cdc0c1a409054f5ca0e041f4c4e1
Reviewed-on: https://go-review.googlesource.com/c/build/+/653496
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Michael Pratt <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
Auto-Submit: Dmitri Shuralyov <[email protected]>
@dmitshur dmitshur moved this from Planned to In Progress in Go Release Feb 28, 2025
@dmitshur
Copy link
Contributor

dmitshur commented Feb 28, 2025

The image and configuration for openbsd/amd64 with OpenBSD 7.6 are in place, but the swarming bot is currently failing to start up. I commented here with the details.

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Mar 4, 2025

@dmitshur - that's rather strange - the python package installs /usr/local/bin/python3 and /usr/local/bin is included in the PATH that is exported before the su invocation (none of this really changed with the make.bash update). If I build the image and boot it everything appears to be correct.

Is there a chance that the image failed to build correctly and that the python package did not actually get installed?

The build output should include:

Adding python-3.11.10p1:libb2-0.98.1v0
python-3.11.10p1 (processing)
python-3.11.10p1 (extracting)
python-3.11.10p1 (skipping)
python-3.11.10p1 (installing)

Are you able to boot the image and see if /usr/local/bin/python3 exists (or provide a way to access the image so I can check)?

@dmitshur
Copy link
Contributor

dmitshur commented Mar 4, 2025

@4a6f656c Thanks for the hint, it was right on. The output from my run of VERSION=7.6 ARCH=amd64 ./make.bash indeed included a line "pkg_add: Installation of python-3.11.10p1 failed, partial installation recorded as partial-python-3.11.10p1", I just hadn't noticed it earlier:

[…]
+ usermod -L moreres swarming
+ syspatch
Installing patch 001_unbound
Installing patch 005_expat
Installing patch 006_wg
Installing patch 007_pffrag
Installing patch 008_ssh
Relinking to create unique kernel... done; reboot to load the new kernel
Errata can be reviewed under /var/syspatch
+ syspatch
+ pkg_add -iv -I bash curl git python%3 sudo--gettext
quirks-7.50 (processing)
quirks-7.50:updatedb-0p0 (processing)
quirks-7.50:updatedb-0p0 (extracting)
quirks-7.50:updatedb-0p0 (skipping)
quirks-7.50:updatedb-0p0 (installing)
Adding quirks-7.50:updatedb-0p0
quirks-7.50 (processing)
[…]
git-2.46.1 (installing)
Shell /usr/local/bin/git-shell appended to /etc/shells
Adding git-2.46.1
python-3.11.10p1 (processing)
python-3.11.10p1:sqlite3-3.44.2 (processing)
python-3.11.10p1:sqlite3-3.44.2 (extracting)
python-3.11.10p1:sqlite3-3.44.2 (skipping)
python-3.11.10p1:sqlite3-3.44.2 (installing)
Adding python-3.11.10p1:sqlite3-3.44.2
python-3.11.10p1:bzip2-1.0.8p0 (processing)
python-3.11.10p1:bzip2-1.0.8p0 (extracting)
python-3.11.10p1:bzip2-1.0.8p0 (skipping)
python-3.11.10p1:bzip2-1.0.8p0 (installing)
Adding python-3.11.10p1:bzip2-1.0.8p0
python-3.11.10p1:xz-5.6.2 (processing)
python-3.11.10p1:xz-5.6.2 (extracting)
python-3.11.10p1:xz-5.6.2 (skipping)
python-3.11.10p1:xz-5.6.2 (installing)
Adding python-3.11.10p1:xz-5.6.2
python-3.11.10p1:libffi-3.4.6 (processing)
python-3.11.10p1:libffi-3.4.6 (extracting)
python-3.11.10p1:libffi-3.4.6 (skipping)
python-3.11.10p1:libffi-3.4.6 (installing)
Adding python-3.11.10p1:libffi-3.4.6
python-3.11.10p1:libb2-0.98.1v0 (processing)
python-3.11.10p1:libb2-0.98.1v0 (extracting)
python-3.11.10p1:libb2-0.98.1v0 (skipping)
python-3.11.10p1:libb2-0.98.1v0 (installing)
Adding python-3.11.10p1:libb2-0.98.1v0
python-3.11.10p1 (processing)
Ustar [https://cdn.openbsd.org/pub/OpenBSD/7.6/packages-stable/amd64/python-3.11.10p1.tgz][lib/python3.11/__v
python-3.11.10p1 (extracting)
Adjusting sha for /usr/local/lib/python3.11/__pycache__/zipfile.cpython-311.opt-1.pyc from n4zNJvUfshBuOLt043
The following new rcscripts were installed: /etc/rc.d/gitdaemon
See rcctl(8) for details.
New and changed readme(s):
	/usr/local/share/doc/pkg-readmes/git
pkg_add: Installation of python-3.11.10p1 failed, partial installation recorded as partial-python-3.11.10p1
+ chown root:wheel /etc/sudoers
+ halt -p
syncing disks... done
Archiving disk.raw... (this may take a while)
Done. GCE image is openbsd-7.6-amd64-gce.tar.gz.

I'll start the process a second time and see if the problem is reproducible or one-off. (It would also be good if such a failure were to produce an error and stop the process, if it's possible to adjust make.bash to be able to catch it.)

@dmitshur
Copy link
Contributor

dmitshur commented Mar 4, 2025

Trying it again without any changes resulted in no errors this time:

[…]
+ pkg_add -iv -I bash curl git python%3 sudo--gettext
[…]
git-2.46.1 (installing)
Shell /usr/local/bin/git-shell appended to /etc/shells
Adding git-2.46.1
python-3.11.10p1 (processing)
python-3.11.10p1:sqlite3-3.44.2 (processing)
python-3.11.10p1:sqlite3-3.44.2 (extracting)
python-3.11.10p1:sqlite3-3.44.2 (skipping)
python-3.11.10p1:sqlite3-3.44.2 (installing)
Adding python-3.11.10p1:sqlite3-3.44.2
python-3.11.10p1:libb2-0.98.1v0 (processing)
python-3.11.10p1:libb2-0.98.1v0 (extracting)
python-3.11.10p1:libb2-0.98.1v0 (skipping)
python-3.11.10p1:libb2-0.98.1v0 (installing)
Adding python-3.11.10p1:libb2-0.98.1v0
python-3.11.10p1:libffi-3.4.6 (processing)
python-3.11.10p1:libffi-3.4.6 (extracting)
python-3.11.10p1:libffi-3.4.6 (skipping)
python-3.11.10p1:libffi-3.4.6 (installing)
Adding python-3.11.10p1:libffi-3.4.6
python-3.11.10p1:xz-5.6.2 (processing)
python-3.11.10p1:xz-5.6.2 (extracting)
python-3.11.10p1:xz-5.6.2 (skipping)
python-3.11.10p1:xz-5.6.2 (installing)
Adding python-3.11.10p1:xz-5.6.2
python-3.11.10p1:bzip2-1.0.8p0 (processing)
python-3.11.10p1:bzip2-1.0.8p0 (extracting)
python-3.11.10p1:bzip2-1.0.8p0 (skipping)
python-3.11.10p1:bzip2-1.0.8p0 (installing)
Adding python-3.11.10p1:bzip2-1.0.8p0
python-3.11.10p1 (processing)
python-3.11.10p1 (extracting)
python-3.11.10p1 (skipping)
python-3.11.10p1 (installing)
Adding python-3.11.10p1
sudo-1.9.15.5p0-gettext (processing)
sudo-1.9.15.5p0-gettext (extracting)
sudo-1.9.15.5p0-gettext (skipping)
sudo-1.9.15.5p0-gettext (installing)
The existing file /etc/sudoers has NOT been changed
It does NOT match the sample file /usr/local/share/examples/sudo/sudoers.OpenBSD
You may wish to update it manually
Adding sudo-1.9.15.5p0-gettext
The following new rcscripts were installed: /etc/rc.d/gitdaemon /etc/rc.d/sudo_logsrvd
See rcctl(8) for details.
New and changed readme(s):
	/usr/local/share/doc/pkg-readmes/git
+ chown root:wheel /etc/sudoers
+ halt -p
syncing disks... done
Archiving disk.raw... (this may take a while)
Done. GCE image is openbsd-7.6-amd64-gce.tar.gz.
$ echo $?
0

The resulting image this time successfully started a swarming bot instance, which connected okay and we have a first build at https://ci.chromium.org/b/8721307255879001089. Thanks for the help. I'll work on the next steps of turning it up for x/ repos.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/654675 mentions this issue: main.star: enable openbsd-amd64_7.6 for x/ repos

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/654676 mentions this issue: main.star: finish upgrading openbsd-amd64 to OpenBSD 7.6

gopherbot pushed a commit to golang/build that referenced this issue Mar 6, 2025
The image was built successfully (on second try) and has proven to work
in the main Go repo (https://ci.chromium.org/b/8721307255879001089), so
take the next step of enabling this builder for golang.org/x repos.

For golang/go#69823.

Change-Id: Ic5c8aee36f3d2f0f3ff57525f996a3da600c25a0
Reviewed-on: https://go-review.googlesource.com/c/build/+/654675
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: Junyang Shao <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Dmitri Shuralyov <[email protected]>
@dmitshur
Copy link
Contributor

dmitshur commented Mar 6, 2025

From looking at https://ci.chromium.org/ui/p/golang/builders?q=openbsd-amd64_7.6, the openbsd/amd64 builder with OpenBSD 7.6 is still running into what looks like #66474, possibly less frequently after the :maxproc-{cur,max} doubling. It's working okay in almost all x/ repos, but x/tools needs some action. Filed #72145 for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-OpenBSD
Projects
Status: In Progress
Development

No branches or pull requests

5 participants