Debugging ARM programs inside QEMU

Posted on 2010/08/17


Recently I wanted to debug a Linux program running inside an ARM system emulated with QEMU. I went into some troubles, so I’m going to write here the procedure that worked for me. I wanted to use gdbserver to run a program inside QEMU, and then connect to it from a GDB instance running on my PC, using a TCP link. gdbserver is a piece of software that implements some of the GDB functionalities (the debugging stubs) and then offers the possibility to connect to a full GDB instance through the network (or through a serial port). What I wanted to obtain is illustrated in the following figure.

Using gdbserver inside QEMU

Using gdbserver inside QEMU

The color blue indicates software compiled to run on my Ubuntu PC (32-bit x86) while the color green indicates software compiled to run on ARM. qemu-system-arm is the software that emulates a VersatilePB platform; I tried to use the one that can be installed through Ubuntu repositories (the package is qemu-kvm-extras) but it froze running the last version of Linux (2.6.35). For this reason I decided to compile the upstream version and use that one. The GDB server and “client” come from the CodeSourcery compiler collection for ARM, as well as the compiler used to cross-compile the software for ARM. The program I’ll debug in this example is the simple GNU Hello, that doesn’t do very much beyond printing “Hello World”, but is a nice example of cross-compilation with GNU Autotools.


In order to follow this procedure, I installed:

  • CodeSourcery GNU/Linux toolchain for ARM
  • The native x86 toolchain (build-essentials package in Ubuntu) to compile QEMU
  • DDD as a graphic interface to the debugger
  • cpio, an utility to create the Linux filesystem image.
  • the package libncurses5-dev to run the menu configurations of both Linux kernel and Busybox
  • libsdl-dev and zlib1g-dev to compile QEMU
$ sudo apt-get install build-essential ddd cpio libncurses5-dev libsdl-dev zlib1g-dev
$ wget
$ chmod +x arm-2010q1-202-arm-none-linux-gnueabi.bin
$ ./arm-2010q1-202-arm-none-linux-gnueabi.bin

I installed the toolchain in the default directory “~/CodeSourcery”. The gdbserver executable in my case can be found at the path “/home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/usr/bin/gdbserver”.

Note that the following procedure is run in a dedicated folder, and no root access is required from now on. At the end of the procedure about 1 Gigabyte of hard disk space was used.

Linux Kernel

First of all, I took the new kernel version from the official repositories.

$ wget
$ tar xjf linux-2.6.35.tar.bz2
$ cd linux-2.6.35/
$ make ARCH=arm versatile_defconfig
$ make ARCH=arm menuconfig

When the menu appears, I go into the “Kernel Features” section and enable EABI support; then I exit (saving the configuration) and compile:

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- all
$ cd ..

The result is a compressed kernel image in “./linux-2.6.35/arch/arm/boot/zImage”.


Next, I take the latest version of Busybox; in a previous tutorial I compiled it statically, but this time I will not, because gdbserver (that I plan to use) needs shared libraries anyway.

$ wget
$ tar xjf busybox-1.17.1.tar.bz2
$ cd busybox-1.17.1
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- install
$ cd ..

The result is the folder “busybox-1.17.1/_install”, containing a minimal root filesystem that lacks shared libraries.


I recompiled from source only the QEMU binaries needed to emulate an ARM system.

$ wget
$ tar xzf qemu-0.12.5.tar.gz
$ cd qemu-0.12.5
$ ./configure --enable-sdl --disable-kvm --enable-debug --target-list=arm-softmmu
$ ./make
$ cd ..

The relevant result is the program “./qemu-0.12.5/arm-softmmu/qemu-system-arm” that will be used to emulate the VersatilePB platform.

GNU Hello

This package needs to be configured for cross-compilation; turns out it’s very easy to do: it needs just the prefix of the cross-compiler.

$ wget
$ tar xzf hello-2.6.tar.gz
$ cd hello-2.6
$ ./configure --host=arm-none-linux-gnueabi
$ make
$ cd ..

The result is the ARM-executable “hello-2.6/src/hello”.

Complete the filesystem

All the ARM binaries involved (busybox, gdbserver, hello) need shared libraries. The Codesourcery toolchain offers these libraries in a subfolder of its installation. In my case it’s “/home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/”. In order to discover what are the needed libraries I used the readelf tool distributed in the CodeSourcery toolchain. In particular, I ran:

$ arm-none-linux-gnueabi-readelf hello-2.6/src/hello -a |grep lib
 [Requesting program interpreter: /lib/]
 0x00000001 (NEEDED)                     Shared library: []
 0x00000001 (NEEDED)                     Shared library: []
00010694  00000216 R_ARM_JUMP_SLOT   0000835c   __libc_start_main
 2: 0000835c     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.4 (2)
 89: 0000844c     4 FUNC    GLOBAL DEFAULT   12 __libc_csu_fini
 91: 0000835c     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
 101: 00008450   204 FUNC    GLOBAL DEFAULT   12 __libc_csu_init
 000000: Version: 1  File:  Cnt: 1
 0x0020: Version: 1  File:  Cnt: 1

The hello binary requires three shared libraries: “”, “” and “”. I executed this command for all three binaries, and I copied the required libraries into the Busybox filesystem, together with the gdbserver executable and the hello executable.

$ cd busybox-1.17.1/_install
$ mkdir -p lib
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/ lib/
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/ lib/
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/ lib/
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/ lib/
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/ lib/
$ cp /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/usr/bin/gdbserver usr/bin/
$ cp ../../hello-2.6/src/hello usr/bin/
$ cd ../../

For my experiment I need a working network from the guest ARM system side, so I prepared an initialization script to enable it. Extending from my previous tutorial, here is the script “rcS” that i used.

mount -t proc none /proc
mount -t sysfs none /sys
/sbin/mdev -s
ifconfig lo up
ifconfig eth0 netmask
route add default gw

Next, I copy rcS it inside the “etc/init.d” directory of the Busybox filesystem and create a compressed filesystem image:

$ cd busybox-1.17.1/_install
$ mkdir -p proc sys dev etc etc/init.d
$ cp ../../rcS etc/init.d
$ chmod +x etc/init.d/rcS
$ find . | cpio -o --format=newc | gzip > ../../rootfs.img.gz
$ cd ../../

Running and debugging

Now I have put everything in place:

  • A compressed kernel image
  • QEMU
  • A compressed filesystem image containing:
    • Busybox
    • rcS initialization script
    • GNU Hello binary compiled for ARM
    • gdbserver for ARM
    • Required shared libraries

To run the plaform the command line is:

$ ./qemu-0.12.5/arm-softmmu/qemu-system-arm -M versatilepb -m 128M -kernel ./linux-2.6.35/arch/arm/boot/zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init" -redir tcp:1234::1234

The “redir” option will redirect any TCP communication from port 1234 of my Ubuntu PC to port 1234 of the guest ARM system. The system will boot,  and a console can be opened with root access. Inside the bash prompt, I run the debugging server:

# gdbserver --multi

This command will launch a server that waits for a GDB connection from port 1234. On my PC I open the debugger:

$ ddd --debugger arm-none-linux-gnueabi-gdb

It is also possible to run the arm-none-linux-gnueabi-gdb command alone or connected to another front-end. In order to debug the remote program, I need to tell GDB to consider the ARM shared libraries instead of the local ones (that are for 32-bit x86); otherwise on execution the debugger will complain that the libraries don’t match.

set solib-absolute-prefix nonexistantpath
set solib-search-path /home/francesco/CodeSourcery/Sourcery_G++_Lite/arm-none-linux-gnueabi/libc/lib/
file ./hello-2.6/src/hello
target extended-remote localhost:1234
set remote exec-file /usr/bin/hello
break main

At this point the debugging goes on as usual.

Posted in: Embedded