Under a sponsorship from the FreeBSD Foundation, I’ve been working on FreeBSD userspace/kernel debugging since February 2026. I stopped working on it from May to August due to a heavy course load at my university, but I have now come back to the Foundation and am finishing parts of the project, aiming for the LLVM 24 release in early 2027.
Besides bug fixes, minor feature additions, and additional architecture support, only trapframe unwinding support has been left. This was the hardest part because it required deep knowledge of both FreeBSD kernel and LLDB internals. Unlike trap or signal frames, normal function calls usually follow the platform ABI and conventional call-frame structure, which debugger unwinders and compiler-generated unwind metadata are designed around. Exceptions, system calls, and interrupts, however, enter the kernel through special entry paths. The processor saves part of the interrupted state, while the kernel’s low-level assembly entry code saves additional registers and constructs a trapframe. Because a trapframe represents an interrupted machine state rather than a conventional function call, the debugger’s normal unwinding assumptions may not apply. To handle this, FreeBSD’s kgdb(1) has used custom trapframe sniffers to identify these special frames and unwind them according to the trapframe structure, sometimes with additional architecture-specific logic.
So I tried this frame-sniffer approach first in LLDB, but it didn’t work out well because LLDB didn’t support a GDB-like mechanism for dynamically selecting a custom frame unwinder. LLDB can identify trap-handler functions and apply special unwind behavior to them, but this mechanism relies on a list of function names rather than a GDB-style architecture-specific sniffer. This was a huge disadvantage for platforms like amd64, where the X* prefix is prevalent for IDT vectors and would be much easier to describe using a pattern. I considered introducing a GDB-like frame-sniffer infrastructure to LLDB, but it required too many changes to the existing codebase, so I abandoned that approach.
I was at a dead end at that point and seriously considered delaying trapframe unwinding support. Then another LLDB contributor said he uses DWARF CFI for Apple’s Darwin kernel and suggested using it instead. I hadn’t heard of DWARF CFI before, so I did a few hours of research on it and concluded that it would be a perfect fit for the FreeBSD kernel.
So what is DWARF CFI? CFI (Call Frame Information) describes how a debugger or unwinder can recover the caller’s register state while walking the stack. In assembly source, this information can be described using .cfi_* directives, which are then encoded by the assembler into sections such as .eh_frame or .debug_frame.
Unlike kgdb’s frame sniffers, there is almost no work required on the debugger side. This means that there is no need to update debugger code whenever the trapframe ABI changes because the call-frame unwinding information is embedded into the kernel binary or debug information.
For example:
# From https://github.com/llvm/llvm-project/blob/main/lldb/test/Shell/Unwind/Inputs/trap_frame_sym_ctx.s
.att_syntax
.text
.globl bar
bar:
.cfi_startproc
leal (%edi, %edi), %eax
ret
.cfi_endproc
.globl asm_main
asm_main:
.cfi_startproc
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
movq %rsp, %rbp
.cfi_def_cfa_register %rbp
movl $47, %edi
# install tramp as return address
# (similar to signal return trampolines on some platforms)
leaq tramp(%rip), %rax
pushq %rax
jmp bar # call, with return address pointing to tramp
popq %rbp
.cfi_def_cfa %rsp, 8
ret
.cfi_endproc
.globl tramp
tramp:
.cfi_startproc
.cfi_signal_frame
# Emit cfi to line up with the frame created by asm_main
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
.cfi_def_cfa_register %rbp
# copy asm_main's epilog to clean up the frame
popq %rbp
.cfi_def_cfa %rsp, 8
ret
.cfi_endprocThe CFI directives tell LLDB how to reconstruct the caller’s state across this unusual frame layout, allowing it to unwind correctly from bar through tramp. According to the test case, LLDB will produce the following backtrace:
thread backtrace -u
# CHECK: frame #0: {{.*}}`bar
# CHECK: frame #1: {{.*}}`tramp
# CHECK: frame #2: {{.*}}`mainFor the FreeBSD kernel, I chose to emit the CFI into .eh_frame rather than .debug_frame using .cfi_sections .eh_frame. This is because Clang drops .cfi_signal_frame when CFI is emitted into .debug_frame. I’m not sure whether this is a bug or intended behavior, though.
.cfi_signal_frame is required here because signal or trap frames have different return-address semantics from normal function calls. With a normal call instruction, the return address saved on the stack points to the instruction after the call, so an unwinder may use an address such as PC - 1 when determining which call site or unwind information applies. On the other hand, when execution is interrupted by a trap handler, the saved PC points directly to the instruction that was executing. Marking the frame with .cfi_signal_frame tells the debugger to use the saved PC directly instead of applying the normal call-frame adjustment.
LLDB already has support for specially handling trap or signal frames, but traditionally those frames need to be identified through a list of known trap-handler symbol names. This is problematic for the FreeBSD kernel because there are dozens of trap handlers, and maintaining and distributing a complete list of those handlers for kernel debugging is cumbersome. I have opened a pull request to fix this by prioritizing signal-frame CFI when evaluating unwind information, allowing the CFI itself to describe the frame as a signal frame without requiring a separate list of handler names.
Thus, this marks the end of the journey for the LLDB/FreeBSD project, although I still have some trivial stuff left, mostly architecture support. The CFI will be supported in upcoming FreeBSD releases, including stable/14 and stable/15, on all platforms except i386, since FreeBSD’s i386 trapframe unwinding logic cannot be expressed using DWARF CFI. To see why, take a look at kgdb’s i386 unwinding logic.
I’m aiming to finish all LLDB/FreeBSD features and fixes by the LLVM 24 release, which I hope will be MFVed to the FreeBSD base system as early as possible so users can have an out-of-the-box kernel debugging experience. If you find any issues or want to open a PR related to LLDB/FreeBSD, please tag me (@mchoo7) on the pull request. I’m willing to help you anytime.