Certain binaries fail to start with "cannot execute binary file" error since UEK 5.4.17-2136.304.4.5

[LenzGr]

There seems to be a regression between UEK 5.4.17-2136.304.4.5 and the previous 5.4.17-2102.201.3 version on OL7 with regards to running third-party binaries that are linked against different glibc versions than the one included in the OS.

After updating to this version, running certain binaries fails with a cannot execute binary file error. I was able to reproduce this with the binary tarball distribution of Percona Server for MySQL, as well as a snap install of MicroK8S (as reported on the K8S forum).

I initially reported this issue here, but wanted to make sure this is getting sufficient visibility. Here's a quick example on how to reproduce this:

Boot OL7 using linux-uek-5.4.17-2102.201.3, download Percona Server for MySQL and note that the mysql binary works as expected:

[vagrant@localhost ~]$ cat /etc/oracle-release 
Oracle Linux Server release 7.9
[vagrant@localhost ~]$ uname -r
5.4.17-2102.201.3.el7uek.x86_64
[vagrant@localhost ~]$ wget -q https://downloads.percona.com/downloads/Percona-Server-5.7/Percona-Server-5.7.36-39/binary/tarball/Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz.sha256sum
[vagrant@localhost ~]$ wget -q https://downloads.percona.com/downloads/Percona-Server-5.7/Percona-Server-5.7.36-39/binary/tarball/Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz
[vagrant@localhost ~]$ sha256sum -c Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz.sha256sum 
Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz: OK
[vagrant@localhost ~]$ tar xf Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz
[vagrant@localhost ~]$ ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql --version
./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql Ver 14.14 Distrib 5.7.36-39, for Linux (x86_64) using 6.0

Now perform a yum update and reboot into the 5.4.17-2136.304.4.5 kernel:

[vagrant@localhost ~]$ uname -r
5.4.17-2136.304.4.5.el7uek.x86_64
[vagrant@localhost ~]$ ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql --version
-bash: ./Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12/bin/mysql: cannot execute binary file

Note that this error only seems to occur with third-party binaries that are linked against a different glibc version (as it's the case for the Percona Server binary and the MicroK8S snap). The rest of the OS seems to be working fine.

[totalamateurhour]

@LenzGr 👋 Thanks for reporting. We are going to take a look

[totalamateurhour]

Root cause identified. Fix in release process.

[NicolasPrivat]

Hello @totalamateurhour , we experienced the same issue on OL8 while installing microk8S with at least grep and gawk binaries. Your fix will target both OL7 and 8 versions ?

Works: 5.4.17-2102.201.3.el8uek
Fail: 5.4.17-2136.302.7.2.2.el8uek

Thanks for the fix.

[totalamateurhour]

@NicolasPrivat Yes, the fix will be for UEK6 (5.4.x), which we provide for both OL7 and OL8.

[totalamateurhour]

@LenzGr @NicolasPrivat these are now on yum.oracle.com for x86_64

kernel-uek-5.4.17-2136.305.5.4.el7uek.x86_64
kernel-uek-5.4.17-2136.305.5.4.el8uek.x86_64

[bourretp]

Fix confirmed for microk8s (1.22 & 1.23) on kernel-uek-5.4.17-2136.306.1.3.el8uek.x86_64: it now installs (and run) like a charm.
Thanks!

[lgatellier]

Hi,
Got the same error, upgraded the kernel-uek package to 5.4.17-2136.306.1.3.el8uek.x86_64, and it works fine 😺

OS : Oracle Linux 8.5
microk8s : 1.24

Thanks !

[maxsatula]

Still having an issue, although it is arm64 not x86_64

# uname -a
Linux oci-arm-dev 5.4.17-2136.307.3.5.el8uek.aarch64 #2 SMP Thu May 26 18:21:01 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux
# grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Oracle Linux Server 8.6"
# snap install microk8s --classic
error: cannot perform the following tasks:
- Run install hook of "microk8s" snap if present (run hook "install":
-----
++ /snap/microk8s/3278/bin/uname -m
+ ARCH=' aarch64'
+ export 'LD_LIBRARY_PATH=/snap/microk8s/3278/lib:/snap/microk8s/3278/usr/lib:/snap/microk8s/3278/lib/ aarch64-linux-gnu:/snap/microk8s/3278/usr/lib/ aarch64-linux-gnu'
+ LD_LIBRARY_PATH='/snap/microk8s/3278/lib:/snap/microk8s/3278/usr/lib:/snap/microk8s/3278/lib/ aarch64-linux-gnu:/snap/microk8s/3278/usr/lib/ aarch64-linux-gnu'
+ export PATH=/snap/microk8s/3278/usr/sbin:/snap/microk8s/3278/usr/bin:/snap/microk8s/3278/sbin:/snap/microk8s/3278/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ PATH=/snap/microk8s/3278/usr/sbin:/snap/microk8s/3278/usr/bin:/snap/microk8s/3278/sbin:/snap/microk8s/3278/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/bin:/usr/local/bin
+ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
+ source /snap/microk8s/3278/actions/common/utils.sh
++ [[ /var/lib/snapd/snap/microk8s/3278/meta/hooks/install == \/\s\n\a\p\/\m\i\c\r\o\k\8\s\/\3\2\7\8\/\a\c\t\i\o\n\s\/\c\o\m\m\o\n\/\u\t\i\l\s\.\s\h ]]
+ cp -r /snap/microk8s/3278/default-args /var/snap/microk8s/3278/args
cp: with --parents, the destination must be a directory
Try 'cp --help' for more information.
-----)

[totalamateurhour]

@maxsatula this looks to be a different issue than reported by @LenzGr. His error message was cannot execute binary file. Do you have reason to believe it's the same issue?

[maxsatula]

@maxsatula this looks to be a different issue than reported by @LenzGr. His error message was cannot execute binary file. Do you have reason to believe it's the same issue?

I don't have 100% proof right now, but it seems to be a different manifestation of the same root cause. Note a leading space in the architecture name, looks like an uname executable runs but internally messes up with something.