Osbuild cache uses ~4.5x the rootfs size when doing cosa build

[mtalexan]

Bug Report

The cache/cache2.qcow2 seems to consistently use about 4.5x the size of the reported rootfs size when building the buildextend-qemu (default build).

Environment

What operating system is being used to run coreos-assembler?

Linux with official podman/docker container.

What operating system is being assembled?

Custom derivative of fedora-coreos-config (Fedora 41).

Is coreos-assembler running in Podman or Docker?

Yes, podamn.

If Podman, is coreos-assembler running privileged or unprivileged?

Privileged, rootless, daemonless.

Expected Behavior

The RUNVM_CACHE_SIZE doesn't need to be set to 4.5x the size of the rootfs being assembled.

Actual Behavior

A reported rootfs of 16 GB requires a RUNVM_CACHE_SIZE of ~75 GB in order to do cosa build/cosa buildextend-qemu.

Reproduction Steps

1. Remove the cache/cache2.qcow2 file
2. cosa build (default qemu)
3. Examine the reported rootfs size passed to the osbuild-mpp command in the output
4. Examine the size of the cache/cache2.qcow2 file.
5. Notice the cache2.qcow2 file is ~4.5x the size of the rootfs

Other Information

I'm guessing this has to do with the hardcoded list of checkpoints passed to the osbuild command in src/runvm-osbuild, and the list of matching stages in the mpp file. Even though only the qemu target is being exported as the default cosa build output, it appears a long list of checkpoints is being added to the osbuild store and is probably what bloats the cache file. E.g.

manifest /tmp/osbuild-J4Qg.json finished successfully
oci-archive:	005cbff73ffa67116c903fba8921e11ef3e4562079ef1a72c0331b59d316b04b
deployed-tree:	e98ad89c59001305087d57fbb7e233be88fece8bf70e4d962ce3b1bddf581f74
tree:     	788b0f67ee9eb62952e9c9bc6c839081b5a1f605d3fb136e4e3893098c812866
raw-image:	9c3d934c9af584b7b489e3a0909e43b0bdeb58926c26eccd9db6987cc06b923b
raw-4k-image:	aebc4d405ff166be620312e5285f992c78b70d32d4212da5b5b8acee0eff49d5
raw-applehv-image:	b9644f6a6eb8c05bc8edac32617d68c4af749a7b7e95e016aa1dab33cba4ba7d
applehv:  	1fd2ee7a7886a7f2bbb89a97ee63256c609354638af3360158ddcd23038769e2
raw-gcp-image:	abe15eb7faf76d01b423d06e3455ccc853c9a65f862e5010e67e32527637f930
raw-gcp-image-tar:	cedb95b3e0269d590b10ba8faf31df9d7990c5c5f7785d205cb0cf75fe8b7f3c
gcp:      	9ae9a2c76377f33f4b4e8d2abe07358eeb9eb8bda61c61307d76559a754f67df
raw-hyperv-image:	de7c1fcc24a88951534d7957fb1a6c84fcf00439ab965c634368b478200a1d34
raw-hyperv-image-vhdx:	b54eb3ca6d78b4290b632ec46cc4331d19c406160452767dc7f1a98409472413
hyperv:   	68b9b6023ed97b608dda91130109be5f7170fa90297b65e60459a46b3adc6cb1
raw-metal-image:	4923913b3aaf60e51ddbe0903198f5df7a0a117d7ca206ad2881a1cf254d6d49
metal:    	e970710dccebd5b621d784d879ebbe064501f5cfdc9cc9f407ccdb75dcc03dbe
raw-metal4k-image:	7a717e52525a436cf1ad431490adf9a8f9228a998297d3e5a445e1022dbd21ab
metal4k:  	49f4ffe1296ad436e3e50994e75019e72bf1e62a4e013954c8bfbbadb8c1b990
raw-qemu-image:	2be680f76cce848cbe8fcbe16df1122f8438b290f7890df36ea0dc835f139de5
qemu:     	25017676b28f2d11677e983b75f6e03ef7a350e8b9b58045b1c9c4e83b0b9dc8
live:     	82df15017eec8b8d16548613161715711a6fb9e0cea177d549657fbafa5f51d6

Presumably a subsequent buildextend-gcp, buildextend-applehv, etc wouldn't increase the size of the cache2.qcow2 file anymore since they're already generated and included in the osbuild store, but it would be preferrable if these were conditionally generated on request rather than unconditionally as part of the main build. Maybe the cosa build interface should have a single build command and the complete list of outputs should be passed via image.yaml (or something) instead so only the desired outputs are created and written to the osbuild store?

That said though, even ignoring some of the raw formats, it looks like the oci-archive, deployed-tree, and tree would each be a copy of the ostree that already lives in the tmp/repo folder, and would end up with ~3x the rootfs size alone. I do wonder whether it's strictly necessary to checkpoint the tree stage since it's just an ostree pull of the primary ostree ref from the tmp/repo ostree that's independently stored persistently (I think). Though maybe there's internal COW support that makes this near-duplication negligible and the extreme bloat is coming from something else.

[dustymabe]

when building the qemu platform any pipelines that are needed by the qemu pipeline will be built. None that aren't needed will be built. The list of checkpoints that we pass in, we do it to optimize later invocations because we know those particular pipelines are depended on by other platforms.

For qemu it would checkpoint deployed-tree tree raw-image, and yes osbuild-mpp would also store a copy of the ociarchive in the osbuild store too. So what you came up with sounds right.

I'm not sure there is much to do about this.

[mtalexan]

If it's really only what's needed, then yeah the issue can be closed. It's just a huge change from pre-osbuild when the qemu was generated from only the ostree and some config files, and all other images were generated from it. Now there's many copies in various formats generated and preserved, increasing the disk usage of a build from starting at ~2x the rootfs to starting at ~6.5x of the rootfs (adding 4.5x of the rootfs as a new cache storage)

[dustymabe]

yep. we've decided to fork most of our stuff over into OSBuild and there are some tradeoffs.