[Discussion] Consider removing --mount type=devpts,destination=/dev/pts

[alexpdp7]

When using distrobox with Podman nested inside an LXC container, distrobox enter does not work, although in general podman works.

If I patch distrobox not to use --mount type=devpts,destination=/dev/pts, then it seems to work OK.

I traced the introduction of this option to toolbx, which introduced this in containers/toolbox#568 (comment) . However, it seems that this change does not address the underlying problem- the issue is still open.

Given that distrobox supports Docker, which does not even support this option... perhaps we could remove it? It doesn't seem to serve its original purpose, and it causes issues in some (admittedly "unsupported"- and I think it's fine not to support nesting in LXC) environments.

(I suspect there might be an underlying "bug" in podman/crun that's triggered by using this flag, but it's just a hunch.)

[alexpdp7]

This seems to be related to #671 ? The actual error I get is this alexpdp7/alexpdp7#10 (comment) , which looks somewhat similar.

There seems to be something interesting lurking around, but I don't know enough to pin it down :(

[ericcurtin]

Hit the same issue, trying to run CentOS Stream 9 QT-based UI apps using distrobox + podman + crun on debian bookworm lxc environment on a aarch64 Chromebook running Chrome OS (hope that makes sense), but I ran into many issues to be honest, this is just one.

[89luca89]

As of commit f1913c2 the init is now doing the devpts internally
can anyone test if this still creates problems for LXC?

[ericcurtin]

Different bug now, on latest version of git main branch:

$ distrobox enter -r fedora-38
Container fedora-38 is not running.
Starting container fedora-38
run this command to follow along:

 sudo podman logs -f fedora-38

 Error: could not start entrypoint.
You must run  entrypoint inside a container!
distrobox-init should only be used as an entrypoint for a distrobox!

This is not intended to be used manually, but instead used by distrobox-enter
to set up the container's entrypoint.
Error: An error occurred

[ericcurtin]

This would be a significant feature for ChromeOS if we could fix this, it would mean you could run almost any Linux distro's UI app on ChromeOS

[89luca89]

How is your setup?

Lxc with inside rootless podman?

[ericcurtin]

Yeah (well I'm trying rootful rather than rootless)... That's what ChromeOS gives you out of the box (well you can choose to install either docker or podman)... You are given an LXC Debian environment that can run UI apps with sound, etc.

But there's no easy way to switch from Debian to other distros in this environment, that's why distrobox is super handy here.

I've gotten distrobox working with non-UI apps on it very well, by more of less deleting offending lines if memory serves me right on how I hacked it.

But UI apps don't work there at present.

[ericcurtin]

Just tried rootless, same result

[ericcurtin]

And just to confirm UI apps work fine of course outside the distrobox environment, in the Debian environment provided by ChromeOS

[89luca89]

Not sure what is different in Chromeos, but on ubuntu, using LXC (via lxd) + Distrobox with nested rootless podman seems to work as intended:

EDIT:

This is the config dump of my LXC test:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20230815)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20230815"
  image.type: squashfs
  image.version: "22.04"
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: ebed156c664de5e9f86c1a2eee7b61031af072af35e90048d2944b3e485f8303
  volatile.cloud-init.instance-id: 38365859-f6da-4c31-a6ba-c8664ca73ede
  volatile.eth0.host_name: veth6dd1fa82
  volatile.eth0.hwaddr: 00:16:3e:14:f3:24
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: e34e7328-cd6f-4a52-9d42-285fdd93f439
  volatile.uuid.generation: e34e7328-cd6f-4a52-9d42-285fdd93f439
devices:
  rundir:
    path: /run/user/1000
    source: /run/user/1000
    type: disk
  tmpdir:
    path: /tmp
    source: /tmp
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

[ericcurtin]

Maybe we could close this issue if it solves @alexpdp7 's case and open a separate issue called "ChromeOS support" or something like that?

[89luca89]

We could yes
But anyway it seems undoable for me to debug or support something that I don't own nor I can easily reproduce (in VM or such)

[ericcurtin]

You probably can run this in a VM or such, see below. Admittedly I run an aarch64 Chromebook, but if it was working in a x86 vm, it likely would fix aarch64 also:

See "How to try google:debian/stretch on our own LXD installation"

https://blog.simos.info/a-closer-look-at-chrome-os-using-lxd-to-run-linux-gui-apps-project-crostini/

https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master/project-termina/chromeos-base/termina-lxd-scripts/files/lxd_setup.sh

[alexpdp7]

Hmmm,

I just did:

[alex@ws ~]$ curl -s https://raw.githubusercontent.com/89luca89/distrobox/main/install | sudo sh -s -- --next
[alex@ws ~]$ distrobox create
[alex@ws ~]$ distrobox enter my-distrobox
Container my-distrobox is not running.
Starting container my-distrobox
run this command to follow along:

 podman logs -f my-distrobox

 Error: could not start entrypoint.
You must run  entrypoint inside a container!
distrobox-init should only be used as an entrypoint for a distrobox!

This is not intended to be used manually, but instead used by distrobox-enter
to set up the container's entrypoint.
Error: An error occurred

This is in a Rocky Linux 9 LXC Container in Proxmox. I'll do more digging.

[alexpdp7]

OK, distrobox create starts a container, but dies. I ran podman inspect on the container to extract the command used to create the container:

'podman create --hostname my-distrobox.ws.h1.int.pdp7.net --name my-distrobox --privileged --security-opt label=disable --user root:root --ipc host --network host --pid host --label manager=distrobox --env SHELL=bash --env HOME=/home/alex --mount type=tmpfs,destination=/run --volume /:/run/host:rslave --volume /dev:/dev:rslave --volume /sys:/sys:rslave --volume /tmp:/tmp:rslave --volume /usr/local/bin/distrobox-init:/usr/bin/entrypoint:ro --volume /usr/local/bin/distrobox-export:/usr/bin/distrobox-export:ro --volume /usr/local/bin/distrobox-host-exec:/usr/bin/distrobox-host-exec:ro --volume /home/alex:/home/alex:rslave --volume /dev/pts --volume /dev/null:/dev/ptmx --volume /var/log/journal --volume /run/user/1284000001:/run/user/1284000001:rslave --volume /etc/hosts:/etc/hosts:ro --volume /etc/resolv.conf:/etc/resolv.conf:ro --ulimit host --annotation run.oci.keep_original_groups=1 --userns keep-id --entrypoint /usr/bin/entrypoint registry.fedoraproject.org/fedora-toolbox:38 --verbose --name alex --user 1284000001 --group 1284000001 --home /home/alex --init 0 --nvidia 0 --pre-init-hooks  --additional-packages  --'

I tried:

[alex@ws ~]$ podman run -it --rm --hostname my-distrobox.ws.h1.int.pdp7.net --name my-distrobox --privileged --security-opt label=disable --user root:root --ipc host --network host --pid host --label manager=distrobox --env SHELL=bash --env HOME=/home/alex --mount type=tmpfs,destination=/run --volume /:/run/host:rslave --volume /dev:/dev:rslave --volume /sys:/sys:rslave --volume /tmp:/tmp:rslave --volume /usr/local/bin/distrobox-init:/usr/bin/entrypoint:ro --volume /usr/local/bin/distrobox-export:/usr/bin/distrobox-export:ro --volume /usr/local/bin/distrobox-host-exec:/usr/bin/distrobox-host-exec:ro --volume /home/alex:/home/alex:rslave --volume /dev/pts --volume /dev/null:/dev/ptmx --volume /var/log/journal --volume /run/user/1284000001:/run/user/1284000001:rslave --volume /etc/hosts:/etc/hosts:ro --volume /etc/resolv.conf:/etc/resolv.conf:ro --ulimit host --annotation run.oci.keep_original_groups=1 --userns keep-id --entrypoint /usr/bin/entrypoint registry.fedoraproject.org/fedora-toolbox:38 --verbose --name alex --user 1284000001 --group 1284000001 --home /home/alex --init 0 --nvidia 0 --pre-init-hooks  --additional-packages  --
Error: OCI runtime error: crun: unlockpt: Invalid argument

If I remove --volume /dev/null:/dev/ptmx, then:

[alex@ws ~]$ podman run -it --rm --hostname my-distrobox.ws.h1.int.pdp7.net --name my-distrobox --privileged --security-opt label=disable --user root:root --ipc host --network host --pid host --label manager=distrobox --env SHELL=bash --env HOME=/home/alex --mount type=tmpfs,destination=/run --volume /:/run/host:rslave --volume /dev:/dev:rslave --volume /sys:/sys:rslave --volume /tmp:/tmp:rslave --volume /usr/local/bin/distrobox-init:/usr/bin/entrypoint:ro --volume /usr/local/bin/distrobox-export:/usr/bin/distrobox-export:ro --volume /usr/local/bin/distrobox-host-exec:/usr/bin/distrobox-host-exec:ro --volume /home/alex:/home/alex:rslave --volume /dev/pts --volume /var/log/journal --volume /run/user/1284000001:/run/user/1284000001:rslave --volume /etc/hosts:/etc/hosts:ro --volume /etc/resolv.conf:/etc/resolv.conf:ro --ulimit host --annotation run.oci.keep_original_groups=1 --userns keep-id --entrypoint /usr/bin/entrypoint registry.fedoraproject.org/fedora-toolbox:38 --verbose --name alex --user 1284000001 --group 1284000001 --home /home/alex --init 0 --nvidia 0 --pre-init-hooks  --additional-packages  --
Error: crun: open /dev/pts/4: No such file or directory: OCI runtime attempted to invoke a command that was not found

If I remove --volume /dev/pts on top, then:

[alex@ws ~]$ podman run -it --rm --hostname my-distrobox.ws.h1.int.pdp7.net --name my-distrobox --privileged --security-opt label=disable --user root:root --ipc host --network host --pid host --label manager=distrobox --env SHELL=bash --env HOME=/home/alex --mount type=tmpfs,destination=/run --volume /:/run/host:rslave --volume /dev:/dev:rslave --volume /sys:/sys:rslave --volume /tmp:/tmp:rslave --volume /usr/local/bin/distrobox-init:/usr/bin/entrypoint:ro --volume /usr/local/bin/distrobox-export:/usr/bin/distrobox-export:ro --volume /usr/local/bin/distrobox-host-exec:/usr/bin/distrobox-host-exec:ro --volume /home/alex:/home/alex:rslave --volume /var/log/journal --volume /run/user/1284000001:/run/user/1284000001:rslave --volume /etc/hosts:/etc/hosts:ro --volume /etc/resolv.conf:/etc/resolv.conf:ro --ulimit host --annotation run.oci.keep_original_groups=1 --userns keep-id --entrypoint /usr/bin/entrypoint registry.fedoraproject.org/fedora-toolbox:38 --verbose --name alex --user 1284000001 --group 1284000001 --home /home/alex --init 0 --nvidia 0 --pre-init-hooks  --additional-packages  --
You must run  entrypoint inside a container!
distrobox-init should only be used as an entrypoint for a distrobox!

This is not intended to be used manually, but instead used by distrobox-enter
to set up the container's entrypoint.
Error: An error occurred

, which means that the container works, but the entrypoint doesn't...

[alex@ws ~]$ podman run -it --rm --hostname my-distrobox.ws.h1.int.pdp7.net --name my-distrobox --privileged --security-opt label=disable --user root:root --ipc host --network host --pid host --label manager=distrobox --env SHELL=bash --env HOME=/home/alex --mount type=tmpfs,destination=/run --volume /:/run/host:rslave --volume /dev:/dev:rslave --volume /sys:/sys:rslave --volume /tmp:/tmp:rslave --volume /usr/local/bin/distrobox-init:/usr/bin/entrypoint:ro --volume /usr/local/bin/distrobox-export:/usr/bin/distrobox-export:ro --volume /usr/local/bin/distrobox-host-exec:/usr/bin/distrobox-host-exec:ro --volume /home/alex:/home/alex:rslave --volume /var/log/journal --volume /run/user/1284000001:/run/user/1284000001:rslave --volume /etc/hosts:/etc/hosts:ro --volume /etc/resolv.conf:/etc/resolv.conf:ro --ulimit host --annotation run.oci.keep_original_groups=1 --userns keep-id registry.fedoraproject.org/fedora-toolbox:38
[root@my-distrobox /]# /usr/bin/entrypoint
You must run  entrypoint inside a container!
distrobox-init should only be used as an entrypoint for a distrobox!

This is not intended to be used manually, but instead used by distrobox-enter
to set up the container's entrypoint.
Error: An error occurred

What I don't understand is the probing for running inside a container. It seems to check for /run/.containerenv and /.dockerenv. But I suspect podman does not create /.dockerenv and /run is supposed to be empty because of --mount type=tmpfs,destination=/run?

[alexpdp7]

Yup, I think 1894e4f introduced a regression by adding --mount type=tmpfs,destination=/run. If I edit /usr/local/bin/distrobox-create and just remove that mount, distrobox enter works.

[alexpdp7]

I spotted #916 , checked that out, and installed from Git. I can confirm that also works OOB for me.

[ericcurtin]

I now have working UI apps on distrobox on ChromeOS, cool 😄

Fully working now.

[ericcurtin]

After #916 it works

[89luca89]

Great stuff thanks everyone for checking and testing!

Set the @ericcurtin PR as solver for this issue 👍