Allow forcing overlayfs

[pschichtel]

Currently podman/buildah won't attempt to mount native overlay filesystems for images, by first probing for the backing filesystem. This is probably very reasonable in most setups, however when using sysbox as the container runtime, this leads to podman being unable to use sysbox' overlayfs support. If I understand their description correctly, they intercept syscalls and make sure the mount goes to a dynamic per-container path that is not on an overlayfs.

Sysbox' documentation on the topic: https://github.com/nestybox/sysbox/blob/46ba726e8e894aa22e20465a32d22dfa2863ec12/docs/user-guide/design.md#overlayfs-mounts-inside-the-sysbox-container

This issue came up when running podman via gitlab-ci runners using the docker executor with the sysbox runtime.

I propose 2 solutions:

1. Change the storage driver to actually attempt to mount using the native overlayfs and only when that fails, fallback to fuse and eventually fail completely
2. Add a configuration option to force the use of native overlay driver.

[pschichtel]

this can easily be reproduced:

1. install and start sysbox
2. configure docker to explicitly use overlayfs (might default to e.g. btrfs depending on the /var/lib filesystem) and configure the sysbox-runc runtime as documented by sysbox
3. run docker run --runtime sysbox-runc --rm -it quay.io/containers/podman:latest
4. run podman run --rm -it alpine:latest inside that container

[giuseppe]

why does the check for overlay fails when running inside sysbox?

[pschichtel]

@giuseppe it doesn't. The problem isn't that the check doesn't work, but that it shouldn't be done in the first place. Sysbox uses native overlayfs and so podman detects that and immediately falls back to alternatives. However: Sysbox intercepts the mount operation to work regardless: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/design.md#overlayfs-mounts-inside-the-sysbox-container, at least that's what they claim.

[giuseppe]

could you please share the output of podman --log-level debug run ...?

[pschichtel]

# podman --log-level debug run alpine:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run alpine:latest) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: imagestore=/usr/lib/containers/storage 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Initialized SHM lock manager at path /libpod_lock 
DEBU[0000] Podman detected system restart - performing state refresh 
INFO[0000] Setting parallel job count to 43             
DEBU[0000] Pulling image alpine:latest (policy: missing) 
DEBU[0000] Looking up image "alpine:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/library/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "localhost/alpine:latest" ...         
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]localhost/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "registry.fedoraproject.org/alpine:latest" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]registry.fedoraproject.org/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "registry.access.redhat.com/alpine:latest" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]registry.access.redhat.com/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/library/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/library/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "alpine:latest" ...                   
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine:latest 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/library/alpine:latest" 
DEBU[0000] Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) 
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]docker.io/library/alpine:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/alpine:latest" 
DEBU[0000] No credentials matching docker.io/library/alpine found in /run/containers/0/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.dockercfg 
DEBU[0000] No credentials for docker.io/library/alpine found 
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///var/lib/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0001] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474 for current system 
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
Getting image source signatures
DEBU[0001] Reading /var/lib/containers/sigstore/library/alpine@sha256=1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474/signature-1 
DEBU[0001] Not looking for sigstore attachments: disabled by configuration 
DEBU[0001] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0001] ... will first try using the original manifest unmodified 
DEBU[0001] Checking if we can reuse blob sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0001] Failed to retrieve partial blob: no TOC found and convert_images is not configured 
DEBU[0001] Downloading /v2/library/alpine/blobs/sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 
DEBU[0001] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 
DEBU[0001] Detected compression format gzip             
DEBU[0001] Using original blob without modification     
Copying blob f18232174bc9 [--------------------------------------] 0.0b / 3.5MiB
Copying blob f18232174bc9 done   | 
DEBU[0001] No compression detected                      
DEBU[0001] Compression change for blob sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b ("application/vnd.oci.image.config.v1+json") not supported 
DEBU[0001] Using original blob without modification     
Copying config aded1e1a5b done   | 
Writing manifest to image destination
DEBU[0001] setting image creation date to 2025-02-14 03:28:36 +0000 UTC 
DEBU[0001] created new image ID "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" with metadata "{}" 
DEBU[0001] added name "docker.io/library/alpine:latest" to image "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Pulled candidate docker.io/library/alpine:latest successfully 
DEBU[0001] Looking up image "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" in local containers storage 
DEBU[0001] Trying "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" ... 
DEBU[0001] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Found image "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" as "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" in local containers storage 
DEBU[0001] Found image "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" as "aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b) 
DEBU[0001] exporting opaque data as blob "sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Looking up image "alpine:latest" in local containers storage 
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0001] Trying "docker.io/library/alpine:latest" ... 
DEBU[0001] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Found image "alpine:latest" as "docker.io/library/alpine:latest" in local containers storage 
DEBU[0001] Found image "alpine:latest" as "docker.io/library/alpine:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.imagestore=/usr/lib/containers/storage,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b) 
DEBU[0001] exporting opaque data as blob "sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Inspecting image aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
DEBU[0001] exporting opaque data as blob "sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Inspecting image aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
DEBU[0001] Inspecting image aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
DEBU[0001] Inspecting image aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b 
DEBU[0001] using systemd mode: false                    
DEBU[0001] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
INFO[0001] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host 
DEBU[0001] Allocated lock 0 for container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 
DEBU[0001] exporting opaque data as blob "sha256:aded1e1a5b3705116fa0a92ba074a5e0b0031647d9c315983ccba2ee5428ec8b" 
DEBU[0001] Created container "06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151" 
DEBU[0001] Container "06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151" has work directory "/var/lib/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata" 
DEBU[0001] Container "06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151" has run directory "/run/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata" 
DEBU[0001] Not attaching to stdin                       
INFO[0001] Received shutdown.Stop(), terminating!        PID=78
DEBU[0001] Enabling signal proxying                     
DEBU[0001] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/ECPT5JT2XA2TLQU7CUHFA2TDND,upperdir=/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/diff,workdir=/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/work,nodev,fsync=0 
DEBU[0001] Mounted container "06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151" at "/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged" 
DEBU[0001] Created root filesystem for container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 at /var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged 
INFO[0001] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
INFO[0001] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
DEBU[0001] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0001] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged" 
DEBU[0001] Created OCI spec for container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 at /var/lib/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata/config.json 
DEBU[0001] /usr/bin/conmon messages will be logged to syslog 
DEBU[0001] Running with no Cgroups                      
DEBU[0001] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 -u 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata -p /run/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata/pidfile -n peaceful_carson --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 --full-attach -l k8s-file:/var/lib/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata/ctr.log --log-level debug --syslog --runtime-arg --cgroup-manager --runtime-arg disabled --conmon-pidfile /run/containers/storage/overlay-containers/06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/var/lib/shared --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/usr/lib/containers/storage --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --stopped-only --exit-command-arg 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0001] Received: -1                                 
DEBU[0001] Cleaning up container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 
DEBU[0001] Network is already cleaned up, skipping...   
DEBU[0001] Error unmounting /var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged with fusermount3 - exit status 1 
DEBU[0001] Failed to replace mountpoint 52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5 overlay: /var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged: device or resource busy 
DEBU[0001] Failed to unmount container "06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151": replacing mount point "/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged": device or resource busy 
ERRO[0001] Cleaning up container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151: unmounting container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 storage: cleaning up container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 storage: unmounting container 06a1b81d9a72a2fbad8728460ac10468ff84eece1a134466efe61535f8c6d151 root filesystem: replacing mount point "/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged": device or resource busy 
DEBU[0001] ExitCode msg: "crun: mount `/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged` to `/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged`: no such file or directory: oci runtime attempted to invoke a command that was not found" 
Error: crun: mount `/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged` to `/var/lib/containers/storage/overlay/52b55e0bbba9de636d372bfbb48bd72ce5f02e2e62680902c6c3f0eb3c45c8b5/merged`: No such file or directory: OCI runtime attempted to invoke a command that was not found
DEBU[0001] Shutting down engines                        
[root@ac663585d031 /]# podman run alpine:latest
ERRO[0000] Cleaning up container 7a527b4a9a7c5565a32d2a878d1a8b15586e68969aa18f636dd62c280645fd32: unmounting container 7a527b4a9a7c5565a32d2a878d1a8b15586e68969aa18f636dd62c280645fd32 storage: cleaning up container 7a527b4a9a7c5565a32d2a878d1a8b15586e68969aa18f636dd62c280645fd32 storage: unmounting container 7a527b4a9a7c5565a32d2a878d1a8b15586e68969aa18f636dd62c280645fd32 root filesystem: replacing mount point "/var/lib/containers/storage/overlay/6ca3d003199159435a6d9822e288e51a076531e63ec443ce4ae445df2ee0687d/merged": device or resource busy 
Error: crun: mount `/var/lib/containers/storage/overlay/6ca3d003199159435a6d9822e288e51a076531e63ec443ce4ae445df2ee0687d/merged` to `/var/lib/containers/storage/overlay/6ca3d003199159435a6d9822e288e51a076531e63ec443ce4ae445df2ee0687d/merged`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Weirdly I don't think this is the same behavior I had when I reported this issue, not sure what changed in the meantime. But it fails non the less.

[giuseppe]

the podman image has some preconfigured settings in /etc/containers. You may want to get rid of (or customize) them

[pschichtel]

Can you provide some examples?