26 Nov 2016

QEMU user network study

Author: derek0883@gmail.com

If no network options are specified, QEMU will default to emulating a single Intel e1000 PCI card with a user-mode network stack. Read this for more information. I’m going to explore how it was implemented in this post.

User mode networking experiment

In this mode, outside will not “see” guest OS, in order to access guest from outside, need QEUM setup port forwarding. The following figure from here

QEMU User mode networking

“-redir tcp:5555::5556” will redirect host tcp port 5555 to guest port 5556. Then outside access guest by using host’s IP at tcp port 5555.

$ sudo gdb ./x86_64-softmmu/qemu-system-x86_64
(gdb) r images/cirros-d161007-x86_64-disk.img -netdev user,id=user.0,hostfwd=tcp::5555-:5556 -device e1000,netdev=user.0

On host, QEMU listen on host’s tcp port 5555.

$ sudo netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0  *               LISTEN      29642/qemu-system-x

In guest OS, start NC

$ sudo nc -lvp 5556 > hello.txt

On host(or another PC), send “Hello qemu Geust” to tcp port 5555 with nc, then guest will display “Hello qemu Geust”. with port forwarding, you can run any sevice on guest, such as HTTP/SSH Server, then outside can access guest.

$ echo "Hello qemu Geust" | nc localhost 5555
# Or another PC
$ echo "Hello qemu Geust" | nc $hostIP 5555

QEMU’s builtin DHCP serve assigned to guest’s eth0. When connect to hostIP:5555, QEMU listen on this port, and it will forward the packet to guest with destination address

$ ifconfig 
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:  Bcast:  Mask:

Inside user mode networking code

In net/net.c, defined initilization function net_init_slirp. It will open and listen on host socket. listen on 5555 in this example.

static int (* const net_client_init_fun[NET_CLIENT_DRIVER__MAX])(
    const Netdev *netdev,
    const char *name,
    NetClientState *peer, Error **errp) = {
        [NET_CLIENT_DRIVER_NIC]       = net_init_nic,
        [NET_CLIENT_DRIVER_USER]      = net_init_slirp,

I’m pretty familar with device driver, and from source code I know virtual device e1000’s Rx handler is e1000_receive_iov. Go back to GDB, set a break point there:

(gdb) b e1000_receive_iov 
Breakpoint 1 at 0x555555a84ccb: file net/net.c, line 583.
(gdb) c

Thread 1 "qemu-system-x86" hit Breakpoint 1, filter_receive at net/net.c:583
(gdb) bt
#0  e1000_receive_iov  at hw/net/e1000.c:845
#1  0x0000555555a85130 in qemu_deliver_packet_iov  at net/net.c:730
#2  0x0000555555a879fa in qemu_net_queue_deliver_iov  at net/queue.c:179
#3  0x0000555555a87b69 in qemu_net_queue_send_iov  at net/queue.c:224
#4  0x0000555555a85275 in qemu_sendv_packet_async  at net/net.c:768
#5  0x0000555555a852a2 in qemu_sendv_packet  at net/net.c:776
#6  0x0000555555a88538 in net_hub_receive_iov  at net/hub.c:72
#7  0x0000555555a88732 in net_hub_port_receive_iov  at net/hub.c:123
#8  0x0000555555a85130 in qemu_deliver_packet_iov  at net/net.c:730
#9  0x0000555555a8797e in qemu_net_queue_deliver  at net/queue.c:164
#10 0x0000555555a87a9a in qemu_net_queue_send  at net/queue.c:199
#11 0x0000555555a84ef0 in qemu_send_packet_async_with_flags  at net/net.c:659
#12 0x0000555555a84f28 in qemu_send_packet_async  at net/net.c:666
#13 0x0000555555a84f55 in qemu_send_packet  at net/net.c:672

#14 0x0000555555a9238a in slirp_output  at net/slirp.c:112
#15 0x0000555555aa40ba in if_encap  at slirp/slirp.c:1008
#16 0x0000555555a9c983 in if_start  at slirp/if.c:197
#17 0x0000555555a9c85f in if_output  at slirp/if.c:141
#18 0x0000555555aa0c3d in ip_output  at slirp/ip_output.c:85
#19 0x0000555555aadd8d in tcp_output  at slirp/tcp_output.c:469
#20 0x0000555555aaf7e6 in tcp_connect  at slirp/tcp_subr.c:512
#21 0x0000555555aa3391 in slirp_pollfds_poll  at slirp/slirp.c:624
#22 0x0000555555af1ba2 in main_loop_wait  at main-loop.c:510
#23 0x00005555558d3a00 in main_loop  at vl.c:1966
#24 0x00005555558db15f in main  at vl.c:4684

ip_output will do some TCP protocol related jobs, e.g. QEMU need change destination IP to guest’s IP, e1000_receive_iov will put the packet into Rx ring buffer, and call set_ics.

static void
set_ics(E1000State *s, int index, uint32_t val)
    DBGOUT(INTERRUPT, "set_ics %x, ICR %x, IMR %x\n", val, s->mac_reg[ICR],
    set_interrupt_cause(s, 0, val | s->mac_reg[ICR]);

set_intterup_cause will trigger an interrupt, eventually Guest OS kernel will call Rx handler of e1000 driver. After that the packet was delivered to guest OS.