Joe Barbere’s Blog

Linux Kernel Development in 2026

In my last post I mentioned a few deeper rabbit holes that didn't make the cut. This is the deepest one: turning that Fedora 43 workstation into a proper Linux kernel development machine. The goal is a tight loop — edit, build, boot in a VM in seconds, break in a debugger, navigate 30-million-plus lines of source without getting lost, and, when you find something, report it upstream the way the kernel community actually works: over email.

Compiling the kernel

Everything starts with a clone of the tree. I add the stable tree as a second remote so I can pull in point releases and backports:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git remote add -f stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

Fedora needs a handful of build dependencies beyond the usual C toolchain — dwarves (pahole) is the one people forget, and you'll need it for BTF debug info:

sudo dnf install gcc make flex bison openssl-devel elfutils-libelf-devel ncurses-devel bc dwarves

Then configure and build. x86_64_defconfig is a sane starting point; allnoconfig when you want the smallest possible kernel to iterate on:

make x86_64_defconfig      # or: make allnoconfig
make menuconfig            # toggle what you need
make -j$(nproc) bzImage    # add 'modules' if you're building any

Applying a patch from the list is a three-way git am:

git am -3 mypatch.patch

Tag your builds so you can tell them apart at runtime. The top of the Makefile carries the version, and EXTRAVERSION (or CONFIG_LOCALVERSION) is the clean place to stamp your own:

VERSION = 6
PATCHLEVEL = 12
SUBLEVEL = 0
EXTRAVERSION = -joe-hack1

When the build starts behaving irrationally — stale objects, config drift, a mysterious "bzImage does not exist" — the reset button is make mrproper. It clears everything, including your .config, so save that first.

Booting fast: QEMU and mkosi

A kernel with nowhere to boot is just a big binary. You need a root filesystem, and hand-rolling one gets old. mkosi builds a bootable OS image from a short declarative config and can launch it straight into QEMU:

sudo dnf install qemu mkosi mkosi-initrd

A minimal mkosi.conf that produces a bootable Fedora disk image:

[Distribution]
Distribution=fedora

[Output]
Format=disk

[Content]
Packages=systemd-udev,systemd-boot,kernel-core
Autologin=yes

Then it's a two-verb loop — build the image, boot it:

mkosi                                 # build
mkosi qemu                            # boot in QEMU
mkosi vm                              # (newer verb, same idea)
mkosi qemu -kernel ~/src/linux/vmlinux   # boot YOUR freshly built kernel

mkosi can also boot the image in a container via systemd-nspawn instead of a full VM, which is dramatically faster when you don't need real hardware emulation — great for exercising userspace against a new syscall.

Two gotchas from my own notes: if boot dies in a "timed out waiting for device… dependency failed for sysroot.mount" cascade, your root partition isn't where the initrd expects it — check the image's partition type/UUID wiring. And if you're driving the excellent mkosi-kernel helper and it insists bzImage does not exist, the fix is almost always a make mrproper back in the kernel source tree. Pin mkosi-kernel to a known-good commit and set its Fedora version to match your host to avoid surprises.

Debugging: gdb and lldb

This is where a VM earns its keep. QEMU exposes a gdb stub, so you can single-step the kernel from instruction zero. Build with CONFIG_DEBUG_INFO=y and CONFIG_GDB_SCRIPTS=y, then start QEMU frozen (-S) with the stub open (-s is shorthand for -gdb tcp::1234). Disable KASLR so symbol addresses line up:

qemu-system-x86_64 -s -S -kernel arch/x86/boot/bzImage \
  -append "console=ttyS0 nokaslr" -nographic -initrd initramfs.img

In a second terminal, point gdb at the uncompressed vmlinux (not bzImage) and connect:

gdb vmlinux
(gdb) target remote :1234
(gdb) lx-symbols          # load module symbols + kernel helpers
(gdb) break start_kernel
(gdb) continue

lx-symbols, lx-dmesg, lx-ps, and friends come from the kernel's own scripts/gdb Python helpers — add the tree to your ~/.gdbinit with add-auto-load-safe-path so gdb will source vmlinux-gdb.py automatically.

Prefer lldb? It can attach to the same stub with gdb-remote 1234 and is perfectly capable for stepping and breakpoints. The catch is that the lx-* convenience scripts are gdb-specific, so you lose the kernel-aware helpers — I keep gdb as the default for kernel work and reach for lldb mostly on the userspace side.

Reading the source: Neovim, clangd, and nvim-dap

The kernel is too big to navigate by memory. I do it in Neovim with a language server plus old-school tag databases as a fallback.

clangd needs a compilation database. The kernel ships a generator that reads the build you just ran:

./scripts/clang-tools/gen_compile_commands.py

That drops a compile_commands.json at the tree root, and nvim-lspconfig's clangd setup takes it from there — cross-references, go-to-definition, and hover across the whole tree. clangd occasionally chokes on the kernel's macro-heavy, non-standard build; a small .clangd file to drop the offending flags smooths most of it over.

For the places the LSP gives up — deep macro expansions, assembly, generated code — nothing beats ctags and cscope:

make ARCH=x86_64 tags cscope

cscope in particular is unbeatable for "who calls this?" and "where is this symbol assigned?" across the entire kernel.

To close the loop, nvim-dap drives the debugger from inside the editor: point a gdb/lldb adapter at that same QEMU stub on :1234 and you get breakpoints, stepping, and variable inspection without leaving your buffers.

The mailing-list workflow: mutt and Thunderbird

Kernel development still runs on email, and that trips up newcomers more than any build error. Patches are sent as inline plaintext to LKML and the relevant subsystem lists — no attachments, no HTML, no whitespace-mangling. Your mail client has to cooperate.

Sending is git send-email once you've configured an SMTP server:

git send-email --to=maintainer@example.org --cc=linux-kernel@vger.kernel.org *.patch

For pulling a series down off the list — applying someone else's patches or a whole thread — b4 is the modern tool:

pipx install b4
b4 shazam <message-id>     # fetch + apply a series in one shot

On the client itself, I lean on mutt — it's terminal-native, handles LKML's firehose volume gracefully, shows patches as the plaintext they are, and piping a message straight into git am is trivial. Thunderbird is the friendlier GUI option, but you have to tame it first: disable HTML composing, turn off format=flowed, and use a fixed-width font so patches don't get reflowed into garbage. The kernel's own Documentation/process/email-clients.rst has the exact settings for both — read it before you send your first patch.

Finding bugs: syzkaller

Once you can build, boot, and debug, you can fuzz. syzkaller is Google's coverage-guided kernel fuzzer, and it is a bug-finding machine. It needs a kernel built with the right instrumentation — CONFIG_KCOV for coverage, CONFIG_KASAN to catch memory corruption, plus CONFIG_DEBUG_INFO and CONFIG_KALLSYMS_ALL so crash reports are legible.

syzkaller is written in Go (already on the workstation), so building it is quick:

git clone https://github.com/google/syzkaller
cd syzkaller && make

You point syz-manager at a config describing your kernel image and a QEMU VM pool, turn it loose, and it spins up VMs, fuzzes syscalls, and reports crashes — each with a C reproducer you can replay under gdb using everything above:

./bin/syz-manager -config my.cfg

When it finds something real, that reproducer plus a clear write-up is exactly what goes back out over email. Report responsibly — give maintainers time before anything goes public.

Wrapping up

Put together, it's a genuinely fast loop: edit in Neovim, make -j$(nproc), mkosi qemu into a fresh VM in seconds, break in gdb when it misbehaves, and mail the fix upstream when it doesn't. The kernel can feel impenetrable from the outside, but the tooling in 2026 makes the door a lot easier to find.


AI icon by Flaticon.

#debugging #development #fedora #kernel #linux