Debian Bookworm Kernel with Docker/Podman Support #4
Comments
The serial console is not required and networking will come up on its own. HDMI video is not supported on the rk3588 or that I have been able to get working. You will need to ssh into the device. |
|
It looks like there are a number of kernel settings that are going to be required: |
No. Maybe with the kernel YOU built it works. With the Kernel I built (even scratch - not even with Podman/Docker support) I get dropped to an emergency shell ("Please enter root password or Ctrl +D"). Networking does NOT come up. NVMe does not get detected. IF Ethernet comes up then of course you can use SSH. But if it doesn't (and it does NOT with the kernel I built, neither with your older Debian Bookworm image) then you are screwed essentially. Could this be due to a hardware difference ? I got the "early bird" Rock 5B (the one where you paid a 5$ to get a 50$ [?] coupon). Microcode / firmware update needed maybe ? |
Yeah I mentioned those in the related thread. And implemented those in my repository to test. But unless the kernel boots and Ethernet & NVMe work, it's still useless. |
I see, it can't mount the rootfs for some reason. This can be challenging to work out. I would recommend to get the build fully working on removable mmc media before trying to get nvme working. Check the kernel command line for correctness. Rather than mount by uuid, try the device name: |
|
It can also fail to mount the rootfs due to a bad device tree: When the kernel builds, it puts it here: |
|
Yeah, I don't know what's going on to be honest. It's not your fault. But it also doesn't feel "scientific" if I cannot replicate the result, you know what I mean ? The Device tree is present in both places Tried with |
|
For this specific issue, the EXT4 options required by Docker/Podman (e.g. CONFIG_EXT4_FS_SECURITY) might have caused this issue. But as I said, previously even without those options, I couldn't access via Ethernet (thus I assume it's a common issue). |
|
/boot/extlinux/extlinux.conf right now |
|
I assume you do have a serial console and are selecting boot option l0? I see the default is set to l1. The uuid of the boot partition is found this way: |
|
Also, when you regenerate the boot menu by running |
Yes. I set the default to something (the only kernel) where Ethernet works. I manually select the first item from the serial console. (NVMe is not detected since PCIe [and Ethernet] don't work ....) |
I manually changed /boot/extlinux/extlinux.conf. Not much to do around that right now to be honest ... |
|
I see, so you are booting one kernel successfully with basically the same settings as the one you compiled. Can you send a listing of the files in |
I do not see any major kernel / initramfs size issue Kernel config I cannot send right now (cannot scp into the Rock 5B since Ethernet not working). I posted it in the other thread though previously IIRC. |
|
Did you install the kernel using Check to be sure your modules exist too: |
|
EDIT: the config I posted was for 6.5.9 here I started serial debug logging again for .config for 6.6.0-1 where I just do a |
I installed all .deb files that were generated using your script using |
|
Maybe I should set these as built-in (actually whatever option contains "ROCKCHIP") ? |
|
This script? This installs the kernel into an image file. From a booted system you would want to install it from the .deb: |
This is from a working kernel: |
I just used ./make_kernel.sh and then dpkg -i since this is a running system. Seing that ./install_kernel.sh installs to a SD card / NVMe / .img file, I didn't see a reason to run that from a live system. |
|
If you want to put your |
Unfortunately I don't seem to have the generated binaries anymore. I guess after a lot of frustration I tried to do a "clean start" (with 6.5.9 though), probably related to the multiple overrides of defconfig if you call ./make_kernel.sh several times. I'll reboot into the working Ethernet kernel (6.6 rc4) and try to rebuild everything for 6.6.0-1. |
|
Check the |
That's what i meant by clean start. I deleted it ... |
|
OK rebuilt .deb files, reinstalled them, same problem. Where should I put them so that they can be shared ? |
|
Hej,
Initially started with the same Then tried another approach, took the config from the latest Debian 6.5.x aarch64 kernel and tried to make a new defconfig from it. Finally ported all networking/netfilter options from Debian's config to the defconfig and finally have a working 6.6.0 kernel with docker working ! Still would like to investigate what is missing in Debian's config to allow it to boot, or what is added that prevents it from booting. |
|
Nice work! I have not used Docker in a while, and am not really setup to test it. I built a kernel based on your config above. Would you be able to test it to see if I have it right? linux-image-6.6.1-1-arm64_6.6.1-1_arm64.deb |
|
@MartinHerren Not sure what you mean by "Finally ported all networking/netfilter options from Debian's config to the defconfig". Can you explain your procedure in full? Which Debian's config and which defconfig (the one used by @inindev or just a new We seem to be 3 people getting 3 different results so far. You @MartinHerren could NOT boot at all, I could boot but NVMe/PCIe/Ethernet didn't work (thus serial access required), @inindev could boot with NVMe/PCIe/Ethernet working. There must be something off ... Either toolchain version (gcc 13 vs 12 etc), Hardware differences, firmware/microcode, ... |
|
@inindev I could give your build a try, but could you please share ALL .deb files generated (4 files in total) ? I usually like to have also the kernel headers installed. @MartinHerren Did you also try with Kernel 6.5.x version ? I'd also like to have ZFS working, and that is not supported on Kernel 6.6.x (there is a patch on the development GitHub repo though). |
Thanks, your kernel installs and boots fine from NVMe (bootloader on SPI... still the one from your 6.6.0-rc4 release, will need to update it).
The procedure:
So for now everything works fine. There are still some options in the config used by Debian that prevent it from booting at all, some others from systemd to start properly. Will still be interesting to identify which ones as the ultimate goal is to run a vanilla Debian with the official kernel. There is still some patching needing to be mainlined and then to trickle down to the Debian kernels. Which i would know more to help in this process. |
|
Well I tried @MartinHerren defconfig with Kernel 6.6.2 compile (with makedefconfig accepting all defaults changes) and it fails to recognize PCIe, Ethernet, etc. |
|
@inindev I'd be OK to install the image you built, if you can provide ALL .deb files (including linux-headers). I'd particularly like to get ZFS working. Right now they just released OpenZFS 2.2.1 which should support Linux 6.6 series. Is there a particular procedure to update bootloader, SPI, EEPROM, firmware, etc that I might have missed ? How did you handle that ? |
|
Here is everything from the build I did on 11/18:
I got busy with the holiday, but I wanted to study the config and test to be sure all of my use cases work before I update the build on the main page. My current thoughts are to provide a bookworm build with a 6.5.x or a 6.6.x kernel and a trixie build with a 6.7-rcx build. |
|
Alright, thank you. I can confirm that it boots, Ethernet works, NVME gets also detected, and @MartinHerren: after applying your defconfig, did you still call confix_fixups (and add the ROCKCHIP SBC kernel module) ? I removed that part as I assumed that it was part of your defconfig file now ... |
|
And of course I tried to recompile everything to try to replicate the success/issue. Partly due to a WARNING during ZFS module compile (kernel provided by @inindev is built using GCC 13, I am using GCC 12 since I am on Debian Bookworm). Tried with same Kernel 6.6.1 with your config file. Also tried to call config_fixups on top of that, since the first attempt resulted in failure. Nothing worked when I do the compile. Might be a GCC 12 or other related Toolchain issue ... |
|
I managed to compile the ZFS kernel modules & utils. They should work correctly. However, since I also want to encrypt the data (BEFORE creating the zpool on it), I tried to use cryptsetup / DM-Crypt / LUKS, but this fails with This seems to be due to some potentially missing kernel options/modules. Specifically: CONFIG_CRYPTO_AES_ARM64 is probably a good idea for AES Acceleration on ARM64 devices such as the Rock 5B. If somebody wants other ciphers than AES, it is currently not possible, since Kernel support is not enabled While these seems to be OK I just tried to import all modules by "brute force", this is the result and the complaints @inindev Would it be possible for you to rebuild a kernel with these options enabled ? Also v 6.6.2 if possible ... Then I can give it a try and provide some feedback. I find it weird that these dm-crypt and ciphers are not enabled by default on the kernel / Debian's default ARM64 which @MartinHerren used to generate his defconfig. They are defaults on AMD64 / x86_64 platforms. |
|
Just to make sure. What did you reply ? I replied with the defaults (n & n). |
|
When I built with the config you linked, I did the following: |
|
I always used your make_kernel.sh script, of course modified to handle a custom config file. I also tried on a second Rock 5B SBC with the Debian Trixie image you provided (had to use the 6.5 image as the 6.6 latest trixie didn't have ethernet working). Anyway... I also tried to compile there with gcc 13 and same config/set of scripts. Got same result. Pcie, ethernet, NVMe NOT working. Maybe back to the most obvious ... Maybe I need to pull theatest changes you uploaded to your git repository (I saw some commits related to kernel 6.6). I still have my own separate repo to try a more modular approach, as I dont / didn't want to fiddle too much with the full kernel configure file directly. By the way, using the commands you mentioned in your last post and my config file, did you get it to boot with ethernet etc working? |
|
@inindev: alright, actually it's my own fault for not synchronizing your latest commits to my forked repo. Now I could compile the Kernel and it boots with Ethernet/PCIe/NVMe support as well as DM_CRYPT. I tried 6.6.1 but I believe I should also be able to upgrade to later versions LUKS+Cryptsetup+DM Crypt as well as ZFS Works (ZFS Requires you to disable SIMD acceleration support for AARCH64/ARM64 though - openzfs/zfs#15401) |
|
@inindev Alright, I now also compiled Kernel 6.6.2. Seems to work fine. I also uploaded the kernel binaries here https://github.com/luckylinux/rock-5b |
|
I have published a release with an enhanced array of kernel modules which should look a lot more like a stock debian kernel. The config should also include a superset of the modules you found to make docker work in your use case as well as requests from others. I would be interested in your feedback if you have time. |
It will take a few days, I just got sick. If you can prepare 6.6.x I can for sure give it a try. Also please upload the linux-libc and linux-image-dbg packages though. |
Hi There, I would like to give some feedback regarding podman: since all things I need seems to work (also I have enough backups) I created a small script to migrate all my pod/containers to your RC7.7. It all worked pretty much flawlessly. Currently USB3 is not working so my appdata is using USB2 with some slowdown but it's enough for me until I can test RC8. Thank you & best Regards |
|
Thanks for the feedback. I am in the process of spliting the linux compile out of my images as it is really just a duplicated effort. The most current kernel is based off of the 6.7 release and should have usb3 working: https://github.com/inindev/linux-rockchip/releases I also published all four .debs which is something you had asked about as well. |
You are my hero! .. until I disconnect..: [ 2493.534779] usb 3-1: USB disconnect, device number 7 Also one different USB3.0 hub will only be recognized as USB2. I can however now connect my Touch T7 2TB external drive and it's working in USB3 mode without errors in dmesg. It is not as fast as it should be, only getting around ~300-350MB/s with the official kernel I've got fullspeed on all devices (protocol limited) samba 3.1.1 multichannel is also a bit slower I've used to have full 5Gbps throughput with multichannel enabled (2x2,5gbe) but now only around ~3Gbps. It could be related to power management... cpu usage is also higher than before. Neverthenless, I'm very happy with this image. Thanks again ! |
fwiw: I noticed the linux 6.7 compatibility changes for openzfs have been merged and are in staging: It looks difficult to predict their next release though. |
I also saw one issue a few weeks back. IIRC it was a minor change that was needed. But I'm holding on until they will have something released and tested. Too many issues already with ZFS 2.2.1 for instance ... |
|
I have updated the debian images to sid kernel 6.8.9: |
|
I'm still running Kernel 6.6.x (updated to the latest available) in order to mimimize Breakages. I saw you included some Patches for Kernel 6.7.x a few months ago, but I didn't want to "change everything" just yet. According to https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md, we might finally get USB 3 Support in Kernel 6.10 👍. |
|
I have rolled-up support for the various platforms into a single repo: I am testing video for the rock-5b and nanopc-t6 with the latest 6.9.2 patched kernel. Collabra had various usb and usb-c patches but I have not tested with them yet. Moving forward, when I post new releases to these repos:
I will back-link them to the original repos. |
|
How was it that you are configuring these Kernels now ? Are you using "Stock" Generic Kernel Config ? I had to manually enable LUKS/Dmcrypt/Cryptsetup and, lately, also drivers for Mediatek mt7921au USB-Wireless Adapter. |
|
I have a few updates I would imagine many people would be interested in reading. I have enabled discussions on this repo and will move this question over there: https://github.com/inindev/rock-5b/discussions see post #10 |
Hi,
As already discussed quite a long way in the other Issue
#3
I would like you to share the Debian Bookworm Kernel with Docker/Podman Support.
I tried to build the kernel on my side but it does NOT boot (correction: it DOES boot but you require SERIAL access, since you drop to an emergency shell - PCIe/Ethernet/NVMe does NOT work)
https://github.com/luckylinux/rock-5b
And previously when I tried to build the complete Debian Bookworm image, it also does NOT boot (correction: it PROBABLY DOES boot but you require SERIAL access, since you drop to an emergency shell - PCIe/Ethernet/NVMe does NOT work)
#2
So there is something weird going on. Possibly related to the build toolchain (I am using GCC 12, you are using GCC 13), but apart from that I cannot really say what's wrong.
I am booting from SD card, so that should theoretically minimize some issues (e.g. broken NVMe) - I have a DATA NVMe installed, but the Root partition (/) is on SD Card.
Attached some logs
rock5_6.6.0_no_ethernet_no_nvme.log
rock5_6.6rc4_ethernet_nvme.log
The text was updated successfully, but these errors were encountered: