FreeBSD offers several approaches to kernel debugging: DDB, live kernel debugging, and core dump analysis. DDB is an interactive debugger built directly into the FreeBSD kernel, with syntax inspired by GDB — making it immediately familiar to most developers. Live debugging leverages FreeBSD's GDB stub (defined under sys/gdb); on the LLDB side, the gdb-remote plugin connects an LLDB client to a FreeBSD host via the GDB protocol, keeping the client platform-independent. Core dump debugging, meanwhile, can be performed with either KGDB or LLDB's FreeBSDKernel plugin.

KGDB remains the preferred tool for many developers, largely because widely used debugger scripts and utilities like crashinfo depend on it. However, KGDB is GPL-licensed and therefore not shipped with FreeBSD. To address this, Moritz Systems ported KGDB's core functionality into LLDB several years ago under contract with the FreeBSD Foundation. One important distinction: KGDB is only available on FreeBSD because it depends on FreeBSD's kvm library — a dependency that also enables partial live kernel debugging through /dev/mem. LLDB's FreeBSDKernel plugin, by contrast, supports two backends: kvm and libfbsdvmcore.

The Problem with the Dual-Provider Model

Why does the FreeBSDKernel plugin offer two separate interfaces? The choice between them isn't made at runtime — it's determined at compile time. The root issue is that kvm is only available on FreeBSD. To enable cross-platform support, LLDB needed an alternative, which led to the creation of libfbsdvmcore (commonly referred to as fvc).

This dual-provider approach introduces several problems.

First, fvc is effectively a clone of kvm with FreeBSD-specific code stripped out and cross-platform scaffolding added in. As a result, it inherits kvm's bugs and limitations. Patches applied to kvm on the FreeBSD side need to be backported to fvc — and when they aren't, things break. For example, if a new minidump format version lands in FreeBSD but isn't reflected in fvc, cross-platform users will be unable to process core dumps generated under the new format.

Second, fvc is a third-party library that isn't bundled with LLDB. The original goal of making FreeBSDKernel cross-platform was to allow developers on macOS or Linux to debug FreeBSD core dumps. In practice, however, distribution-supplied LLDB packages don't include libraries like fvc, meaning those developers would need to build LLDB from source — a barrier high enough that virtually no one does it. This has left fvc largely unmaintained and undertested. I recently discovered, for instance, that it was missing minidump version 3 support on arm64. My ongoing work to bring the FreeBSDKernel plugin to feature parity with KGDB is made considerably harder by having to validate against two separate providers.

Solving the Dual-Provider Problem

The solution, originally proposed by the developer of the FreeBSDKernel plugin, is to integrate kvm's functionality directly into LLDB. I agree with this approach: it enables cross-platform debugging without relying on external libraries, and it allows us to retire both the kvm and fvc interfaces in favor of a single, unified backend — significantly reducing the maintenance burden.

The one trade-off is that changes to the internalized implementation will need to be backported to FreeBSD's in-tree LLDB and upstreamed accordingly. In practice, though, this is far more manageable than maintaining two parallel interfaces.

That said, this is a substantial undertaking involving significant research, implementation, and testing. Since I already have ongoing LLDB improvement work in progress, it isn't the immediate priority. My current target is to complete the in-flight work by LLVM 23, with the unified interface aimed for LLVM 24. In the interim, the most practical step is to remove fvc from the FreeBSDKernel plugin and make it kvm-only. A pull request for that change is currently under review.

Other Improvements to FreeBSDKernel

My broader work on the FreeBSDKernel plugin focuses on improving LLDB's overall FreeBSD kernel debugging support. This was originally conceived as a GSoC project idea, but I took it on as a work item during my co-op with the FreeBSD Foundation. The scope includes adding support for additional architectures, implementing commands and features where FreeBSDKernel currently falls short of KGDB, and other enhancements. Progress is being tracked publicly on LLVM's GitHub issue.

Minsoo Choo

Future of the FreeBSD Kernel LLDB Plugin