12 Dec 2016

Using debugfs to study kernel code

Author: derek0883@gmail.com

There are lots pr_debug()/dev_dbg() in kernel code, if CONFIG_DYNAMIC_DEBUG is set, we can enable/disable those print at running time through debugfs. So we can observe kernel behavior without change/recompile/reboot.

If you are using popular distributions, such as ubuntu, fedora, debugfs is already set up on your machine. If you’re compiling the kernel from scratch, you should enable debugfs in the kernel configuration. Once you reboot to your newly compiled kernel, check if debugfs is already mounted, with the following command:

# mount | grep  debugfs
none on /sys/kernel/debug type debugfs (rw)

Above output shows debugfs already mounted. If not, you can mount it (as root) with the command shown below:

# mount -t debugfs nodev /sys/kernel/debug

If you are study some kernel source code, e.g. if you are trying to learn KVM related souce code, you can use this command

echo -n 'file *kvm* +plf' > dynamic_debug/control

It will do a wild match for all source file, if the name matched “kvm”, then the pr_dbg or dev_dbg will print out. It will print(p) line number(l) and function name(f). To disable it, just change ‘+’ to ‘-‘. with above command, I got some debug print like this.

/sys/kernel/debug# dmesg 
[ 5001.727157] kvm: zapping shadow pages for mmio generation wraparound
[ 5001.727385] kvm: zapping shadow pages for mmio generation wraparound
[ 5004.030902] kvm [3596]: vcpu0 disabled perfctr wrmsr: 0xc2 data 0xffff

you can get more information from Documentation/dynamic-debug-howto.txt or here is the online version.

Some code only get run during boot stage, so we can enable the debug print by pass command to kernel boot command line. e.g.

$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.4.0-53-generic root=UUID=12db2ad9-0ae7-4a09-9894-7271bcb91bb3 \
        ro quiet splash vt.handoff=7 apic=debug  \
        "dyndbg=file *pci* +pf; file *irq* +pf"

above example, I added “dyndbg=file pci +pf; file irq +pf” to kernel command line, it will print all pci and irq related pr_dbg or dev_dbg.

You will get output like this from dmesg after reboot.

[    0.163949] pci_scan_child_bus: pci_bus 0000:00: scanning bus
[    0.163956] pci 0000:00:00.0: [8086:0154] type 00 class 0x060000
[    0.163967] fixup_debug_start: pci 0000:00:00.0: calling quirk_mmio_always_on+0x0/0x10
[    0.164034] pci 0000:00:02.0: [8086:0166] type 00 class 0x030000
[    0.164047] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf03fffff 64bit]
[    0.164053] pci 0000:00:02.0: reg 0x18: [mem 0xe0000000-0xefffffff 64bit pref]
[    0.164058] pci 0000:00:02.0: reg 0x20: [io  0x5000-0x503f]
[    0.164141] pci 0000:00:14.0: [8086:1e31] type 00 class 0x0c0330
[    0.164172] pci 0000:00:14.0: reg 0x10: [mem 0xf2520000-0xf252ffff 64bit]
[    0.164231] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.164236] pci_pme_active: pci 0000:00:14.0: PME# disabled

The following example will show all registered print

/sys/kernel/debug# cat dynamic_debug/control

# grep fixup_debug_start, currenlly eanbled flag "pfl"
/sys/kernel/debug# cat dynamic_debug/control | grep fixup_debug_start
drivers/pci/quirks.c:3047 [quirks]fixup_debug_start =pfl "calling  %pF @ %i for %s\012"
drivers/pci/quirks.c:3044 [quirks]fixup_debug_start =pfl "calling %pF\012"

you can also add you own file entry to debugfs in kernel, IRQ domain debug is a good example. you can follow this to add you own print.

static int virq_debug_show(struct seq_file *m, void *private)
    unsigned long flags;
    struct irq_desc *desc;
    struct irq_domain *domain;
    struct radix_tree_iter iter;
    void *data, **slot;
    int i;

    seq_printf(m, " %-16s  %-6s  %-10s  %-10s  %s\n",
           "name", "mapped", "linear-max", "direct-max", "devtree-node");
    return 0;

static int virq_debug_open(struct inode *inode, struct file *file)
    return single_open(file, virq_debug_show, inode->i_private);

static const struct file_operations virq_debug_fops = {
    .open = virq_debug_open,
    .read = seq_read,
    .llseek = seq_lseek,
    .release = single_release,

static int __init irq_debugfs_init(void)
    if (debugfs_create_file("irq_domain_mapping", S_IRUGO, NULL,
                 NULL, &virq_debug_fops) == NULL)
        return -ENOMEM;

    return 0;

Above code will create a file under debugfs, name is irq_domain_mapping

/sys/kernel/debug# cat irq_domain_mapping 
 name              mapped  linear-max  direct-max  devtree-node
 IR-IO-APIC            24          24           0  
 DMAR-MSI               2           0           0  
 IR-PCI-MSI             6           0           0  
 (null)             65563       65536           0  
 IR-PCI-MSI             1           0           0  
 (null)             65536       65536           0  
 (null)                 0           0           0  
 (null)                 0           0           0  
*(null)                31           0           0  
irq    hwirq    chip name        chip data           active  type            domain
    1  0x00001  IR-IO-APIC       0xffff8802158a6f80     *    LINEAR          IR-IO-APIC
    3  0x00003  IR-IO-APIC       0xffff880214972180          LINEAR          IR-IO-APIC
    4  0x00004  IR-IO-APIC       0xffff880214972280          LINEAR          IR-IO-APIC
    5  0x00005  IR-IO-APIC       0xffff880214972380          LINEAR          IR-IO-APIC
    6  0x00006  IR-IO-APIC       0xffff880214972480          LINEAR          IR-IO-APIC
    7  0x00007  IR-IO-APIC       0xffff880214972580          LINEAR          IR-IO-APIC
    8  0x00008  IR-IO-APIC       0xffff880214972680     *    LINEAR          IR-IO-APIC
    9  0x00009  IR-IO-APIC       0xffff880214972780     *    LINEAR          IR-IO-APIC
   10  0x0000a  IR-IO-APIC       0xffff880214972880          LINEAR          IR-IO-APIC
   11  0x0000b  IR-IO-APIC       0xffff880214972980          LINEAR          IR-IO-APIC
   12  0x0000c  IR-IO-APIC       0xffff880214972a80     *    LINEAR          IR-IO-APIC
   13  0x0000d  IR-IO-APIC       0xffff880214972b80          LINEAR          IR-IO-APIC
   14  0x0000e  IR-IO-APIC       0xffff880214972c80          LINEAR          IR-IO-APIC
   15  0x0000f  IR-IO-APIC       0xffff880214972d80          LINEAR          IR-IO-APIC
   16  0x00010  IR-IO-APIC       0xffff8800d3e4d940     *    LINEAR          IR-IO-APIC
   17  0x00011  IR-IO-APIC       0xffff8800d3e4da40          LINEAR          IR-IO-APIC
   18  0x00012  IR-IO-APIC       0xffff8800d3e4db40          LINEAR          IR-IO-APIC
   19  0x00013  IR-IO-APIC       0xffff8802147d3bc0          LINEAR          IR-IO-APIC
   20  0x00014  IR-IO-APIC       0xffff8800350411c0          LINEAR          IR-IO-APIC
   22  0x00016  IR-IO-APIC       0xffff88020f9e8e40          LINEAR          IR-IO-APIC
   23  0x00017  IR-IO-APIC       0xffff880035670000     *    LINEAR          IR-IO-APIC
   24  0x00000  DMAR-MSI                     (null)     *     RADIX          DMAR-MSI
   25  0x00001  DMAR-MSI                     (null)     *     RADIX          DMAR-MSI
   26  0x50000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   27  0x7d000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   28  0x58000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   29  0x08000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   30  0x6c000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   31  0x64000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI
   32  0x180000  IR-PCI-MSI                   (null)     *     RADIX          IR-PCI-MSI