24 Dec 2016

PCI interrupt routing II

Author: derek0883@gmail.com

In PCI interrupt routing I, I figured out the how PCI interrupt pin connect to a PCI interrupt router, how system software configure the router. and I also figured out the relationship between device IRQ number and interrupt vector number. This post I’m going to do some experiment to verify my understanding. You need read PCI interrupt routing I first.

QEMU PCI config write tracing

In order to observe how system software configure interrupt pin, interrupt line and interrupt router. I will trace PCI configuration read/write, for how QEMU tracing works, check my previous post Tracing mechanism of QEMU.

Copy following text from trace-events-all to trace-events file.

pci_cfg_read(const char *dev, unsigned devid, unsigned fnid, unsigned offs, unsigned val) "%s %02u:%u @0x%x -> 0x%x"
pci_cfg_write(const char *dev, unsigned devid, unsigned fnid, unsigned offs, unsigned val) "%s %02u:%u @0x%x <- 0x%x"

Start QEMU, refer to my post Build minimal kernel for QEMU for how to build your own kernel for QEMU.

$ echo pci_cfg_read > /tmp/events
$ echo pci_cfg_write >> /tmp/events

$ ./x86_64-softmmu/qemu-system-x86_64 -trace events=/tmp/events \
	-kernel images/bzImage -initrd images/cirros-x86_64-initramfs \
	-append "console=ttyS0 apic=debug loglevel=8" -monitor stdio 

Now use simpletrace.py to parse event log, replace 5159 to PID of your QEMU instance. If you got empty output, because QEMU has not flush event buffer yet. use commnad “trace-file flush” to flush event buffer.

# flush trace file in QEMU console
(qemu) trace-file flush

# go to another shell console
$ ./scripts/simpletrace.py trace-events trace-5159 | grep e1000 

As you can see, at very beginning BIOS write 0xb to Interrupt Line(offset 0x3c), No write to interrupt pin(offset 0x3d), the default value is 1. So it use Pin A to signal interrupt. And also 0x3c, interrupt line register was write to 0xb, IRQ number is 11.

pci_cfg_write 4.302 pid=5159 dev=e1000 devid=0x3 fnid=0x0 offs=0x3c val=0xb
pci_cfg_read 1.480 pid=5159 dev=e1000 devid=0x3 fnid=0x0 offs=0x3d val=0x1

In PCI interrupt routing I, I already figured out Interrupt Pin A is connect to PIIX PIRQ C, So offset 0x62 will be set to IRQ 0xb. We can verify this by checking pci_cfg_write to offset 0x62 of PIIX3.

$ ./scripts/simpletrace.py trace-events trace-5159 | grep PIIX3 | grep offs=0x62

pci_cfg_write 2190071.800 pid=26298 dev=PIIX3 devid=0x1 fnid=0x0 offs=0x62 val=0xb

Next I’m going to trace how kernel get throse information.

kernel ACPI tracing

First step need to build kernel with CONFIG_ACPI_DEBUG=y. Refer to my post Build minimal kernel for QEMU for how to build a kernel for QEMU.

Boot kernel with following command, you can check ACPI debug for more information about ACPI debug command.

$ ./x86_64-softmmu/qemu-system-x86_64 --enable-kvm \
	-kernel ./images/bzImage -hda ./images/xenial-server-cloudimg-amd64-disk.img \
	-nographic -append "console=ttyS0 root=/dev/sda1 apic=debug loglevel=8 \
	acpi.debug_layer=0x400000 acpi.debug_level=0x4"

After login, you can find the following output from dmesg.

[    0.366147] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    0.368264] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.396293]       0000:00:03[A] -> \_SB_.LNKC[0]
[    0.397247]   pci_irq-0324 pci_irq_lookup        : Found 0000:00:03.0[A] _PRT entry
[    0.399256]  pci_link-0286 pci_link_get_current  : Link at IRQ 11
[    0.400193]  pci_link-0400 pci_link_set          : Set IRQ 11
[    0.400978] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[    0.401698]  pci_link-0676 pci_link_allocate_irq : Link LNKC is referenced

_PRT is PCI interrupt routing table defined by ACPI spec. BIOS pass it to kernel. kernel function acpi_pci_irq_lookup decode this entry, it means device 0000:00:03(eth0) interrupt pin A connect to South Bridge(PIIX3) PIRQ link 3. _PRT entry also defined IRQ number used by eth0, it is 11. Then kernel call pci_link_allocate_irq to allocate IRQ 11, and write 0xb to PIIX3 PIRQ register 0x62.

ACPI dump

ACPI is a very complex specification, we can dump ACPI data then decode it to human readable text file.

Now boot QEMU with Ubuntu root filesystem.

$ sudo apt install acpidump
$ mkdir ACPI
$ cd ACPI
$ sudo acpidump > acpidata.dat
$ acpixtract -a acpidata.dat
$ ls 
acpidata.dat  dsdt.dat  facp.dat  hpet.dat
apic.dat      facs.dat  ssdt.dat

# decode dsdt.dat
$ iasl -d dsdt.dat 

Then you will find _PRT entry like this. I don’t fully understand the meanning of ACPI yet. You can download ACPI spec here.

    Scope (_SB)
        Scope (PCI0)
            Method (_PRT, 0, NotSerialized)  // _PRT: PCI Routing Table
                Local0 = Package (0x80) {}
                Local1 = Zero
                While ((Local1 < 0x80))