Kernel Dev Tools

This is a repo I use to support my own Linux kernel development. I put scripts/tools here instead of inside the actual kernel tree. It is expected that this repo will live on my machine, and any kernel trees will live under this directory, but won't be committed to this repo.

Usage

Here is a typical workflow with bells and whistles:

1. (Optional) Update the kernel source tree or get a new one

   $ git clone <some-linux-tree>
   $ cd linux && git pull upstream master
   # or
   $ ./scripts/fetch-kernel-tarball.sh 5.18.6

2. Configure the kernel, using a QEMU configuration with ./scripts/minimal-qemu-kernel-config.sh linux

3. Create a QEMU image using ./scripts/create-qemu-image.sh <debian|nixos>

4. Compile the kernel with cd linux/ && make -j32

5. Run the kernel with QEMU image using something like:

   $ ./scripts/run-qemu-kernel.sh linux nixos.img /path/to/shared-files

   /path/to/shared-files here is a directory that will be packaged up into the QEMU .img file and mounted at /shared in the VM. This arg is optional, but it is useful for e.g. adding compiled, out-of-tree kernel modules.

Cross compilation

Start a cross-compilation dev shell with ./cross-compile.sh <arch>, reconfigure the kernel, and then run a build using the $LINUX_CROSS_COMPILE env var ($ARCH) is set in the shell:

$ ./cross-compile.sh arm64
$ ./scripts/minimal-qemu-kernel-config.sh linux
$ cd linux
$ make CROSS_COMPILE=$LINUX_CROSS_COMPILE -j$(nproc)

Linux Kernel Language Server Protocol (LSP) Configuration

The kernel has a script to generate a compile-commands.json usable with LSP:

$ cd linux/
# Make a config and build the kernel
$ make mrproper
$ make defconfig
# Build just for good measure
$ make -j32
# Generate compile-commands.json
$ make compile_commands.json -j32

Buildroot

Raspberry Pi using external directory

1. Set up some stupid symlinks that buildroot expects (well, really the problem is libtool and autotools I think. buildroot can't use host libraries and pretty much compiles everything from scratch).

   $ sudo ln -s /run/current-system/sw/bin/file /usr/bin/file
   $ sudo ln -s /run/current-system/sw/bin/true /bin/true
   $ sudo ln -s /run/current-system/sw/bin/awk /usr/bin/awk
   $ sudo ln -s /run/current-system/sw/bin/bash /bin/bash
   $ sudo ln -s /run/current-system/sw/bin/install /usr/bin/install
   $ sudo ln -s /run/current-system/sw/bin/openssl /usr/bin/openssl

2. Use the external config

   $ cd buildroot
   $ make BR2_EXTERNAL=../buildroot-rpi reaver_rpi_defconfig menuconfig

3. Build $ make (no need for -j since buildroot uses parallelism internally, not for top level targets)

4. Copy image to SD card (see below for generic instructions)

Raspberry Pi Generic

(These were the instructions I used before I set up the buildroot-rpi external infra)

Getting buildroot working is not too onerous, at least since I figured out some nix quirks. Here is a general procedure:

1. Set up some stupid symlinks that buildroot expects (well, really the problem is libtool and autotools I think. buildroot can't use host libraries and pretty much compiles everything from scratch).

   $ sudo ln -s (which file) /usr/bin/file
   $ sudo ln -s (which true) /bin/true
   $ sudo ln -s (which awk) /usr/bin/awk
   $ sudo ln -s (which bash) /bin/bash

2. git clone git://git.buildroot.net/buildroot

3. cd buildroot

4. make raspberrypi4_64_defconfig

5. make -j32

6. Copy output/images/sdcard.img to an SD card (MAKE SURE TO VERIFY DEVICE NAME, REPLACE sdz)

   $ sudo dd if=output/images/sdcard.img of=/dev/sdz status=progress

7. Connect Pi serial port to USB serial port connector, use dmesg to find tty device name

   dmesg | grep 'cp210x converter now attached'
   [48407.800462] usb 1-4: cp210x converter now attached to ttyUSB0
   

8. Connect with GNU screen

   $ sudo screen /dev/ttyUSB0 115200

9. Boot the Pi! Log in with root and no password.

Cool links:

• https://www.thirtythreeforty.net/series/mastering-embedded-linux/
• Serial console
  • https://learn.adafruit.com/adafruits-raspberry-pi-lesson-5-using-a-console-cable/test-and-configure
  • https://www.jeffgeerling.com/blog/2021/attaching-raspberry-pis-serial-console-uart-debugging

Misc resources

TODO Check out this repo for kernel dev with nix https://github.com/jordanisaacs/kernel-module-flake

QEMU dev env:

• https://nixos.wiki/wiki/Kernel_Debugging_with_QEMU
  • https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix
  • Alternate NixOS QEMU image https://gist.github.com/tarnacious/f9674436fff0efeb4bb6585c79a3b9ff
  • NixOS qcow2 build https://gist.github.com/jahkeup/14c0f35383bf949fdd92fbfa20184b4f
  • https://discourse.nixos.org/t/how-to-build-a-standalone-nixos-qemu-vm/5688/4
• https://vccolombo.github.io/cybersecurity/linux-kernel-qemu-setup/
• https://kernel-recipes.org/en/2015/talks/speed-up-your-kernel-development-cycle-with-qemu/
  • https://www.youtube.com/watch?v=PBY9l97-lto
• https://medium.com/@daeseok.youn/prepare-the-environment-for-developing-linux-kernel-with-qemu-c55e37ba8ade
• https://www.collabora.com/news-and-blog/blog/2017/01/16/setting-up-qemu-kvm-for-kernel-development/
• Nice explanation of using -nographic with available options https://web.archive.org/web/20180104171638/http://nairobi-embedded.org/qemu_monitor_console.html
• ClangBuiltLinux has a repo super similar to this repo, including QEMU and buildroot scripts https://github.com/ClangBuiltLinux/boot-utils

Email:

• https://www.kernel.org/doc/html/latest/process/email-clients.html
• https://offlinemark.com/2020/09/26/tips-for-submitting-your-first-linux-kernel-patch/
• https://ane.iki.fi/emacs/patches.html
• https://git-send-email.io
• https://devtut.github.io/git/git-send-email.html#sending-patches-by-mail

First time contributions:

• https://kernelnewbies.org/
  • https://kernelnewbies.org/KernelHacking
    • Suggests running checkpatch.pl on drivers/staging directories
    • Also suggests running Coccinelle
• Very informative patch set with revisions, followups, responses from Greg K-H's autobot, etc https://lore.kernel.org/linux-staging/[email protected]/T/
  • Versioning patch revisions https://kernelnewbies.org/FirstKernelPatch#Versioning_one_patch_revision
• https://www.linux.com/news/three-ways-beginners-contribute-linux-kernel/
• https://williamdurand.fr/2021/02/22/first-patch-in-the-linux-kernel/
• tpiekarski's comment from discussion "Is reading Linux kernel development helpful in 2020? Is it outdated?"
• How to become a Kernel Developer?
• How should I start kernel development?
• What (not how) to contribute to the kernel
• Recommendations for newer books on kernel development?

Getting started, things to do

• Kernel dev process
• https://github.com/agelastic/eudyptula
• https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project

TODO

• 32 bit HIGHMEM setup with QEMU. Probably 32 bit ARM, maybe with some Debian distribution?
  • Or get BeagleBone or STM32 working