Distcc
Contents |
Distributed Remote-Distributed-Compile from the n900 to native armel targets
For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!
This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.
Where possible I distinguish the different system root-prompts as:
n900-chroot# The easydebian chroot prompt (you can ssh in too if you like) host-root# The high-power desktop system to run the ARM VM's under qemu guest-root# The ARM VM's themselves.
1. Install Easy Debian
Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the n900. Once it is installed, install ccache, distcc, and build-essential:
n900-chroot# apt-get install ccache distcc build-essential
Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.
2. Install a few QEMU VMs
On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:
I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.
Install qemu-arm on your system (usually apt-get install qemu and boot the VM:
host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \ -initrd initrd.img-2.6.26-1-versatile \ -hda debian_lenny_armel_small.qcow2 \ -append "root=/dev/sda1"
3. Install the required packages on the VM
First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:
guest-root# apt-get update guest-root# apt-get upgrade guest-root# apt-get install build-essential ccache distcc
4. Configure distcc and ccache
a. Add the hosts to /etc/distcc/hosts
b. Configure /etc/default/distcc with at least:
STARTDISTCC="true" # configure your subnet ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
c. create a ccache/distcc wrapper
root# mv /usr/bin/gcc /usr/bin/gcc-orig root# cat<<EOT > /usr/bin/gcc-ccache #!/bin/sh export CCACHE_PREFIX=distcc # List the distcc hosts here; I had trouble with /etc/distcc/hosts working export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103' exec ccache gcc-orig "$@" EOT root# chmod 755 /usr/bin/gcc-ccache
Be sure to do this on the n900 too!
d. link the wrapper as the new gcc:
root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
e. test the distcc-ccache-gcc wrapper:
root# gcc -version gcc-orig: no input files distcc[2486] ERROR: compile (null) on localhost failed
The distcc error is expected
5. Configure the network to DHCP
guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces
6. Halt the vm and launch multiple instances
a. Halt the guest and close the window:
guest-root# shutdown -h now
b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).
root# cat<<EOT > /usr/local/bin/qemu-ifup-br0 #!/bin/sh iface="$1" bridgeName=`basename "$0" | cut -f3 -d-` echo "bridgeName=$bridgeName iface=$1" /sbin/ifconfig "$iface" 0.0.0.0 up /usr/sbin/brctl addif "$bridgeName" "$iface" EOT chmod 755 /usr/local/bin/qemu-ifup-br0
host-root# brctl addbr br0 host-root# brctl addif br0 eth0 host-root# ifconfig br0 192.168.99.222 host-root# ip route replace default via 192.168.99.1
c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:
host arm-vm1 {
hardware ethernet 00:16:3e:00:00:01;
fixed-address 192.168.99.101; }
host arm-vm2 {
hardware ethernet 00:16:3e:00:00:02;
fixed-address 192.168.99.102; }
host arm-vm3 {
hardware ethernet 00:16:3e:00:00:03;
fixed-address 192.168.99.103; }
d. Launch the vm's:
host-root# for i in 1 2 3; do qemu-system-arm -name armvm$i \ -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \ -initrd initrd.img-2.6.26-1-versatile \ -hda debian_lenny_armel_small.qcow2 \ -append "root=/dev/sda1" \ -net nic,macaddr=00:16:3e:00:00:0$arm \ -net tap,script=/usr/local/bin/qemu-ifup-br0 \ -daemonize \ -snapshot ; done
e. The bridge listing should look something like this:
host-root# brctl show br0 8000.001cc0b1dadf no eth0 tap1 tap2 tap3
Run a distributed build on your n900
a. Download a recent vanilla kernel:
cd /usr/src wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv cd linux-2.6.34 [...] make -j3 defconfig *** Default configuration is based on 'versatile_defconfig' # # configuration written to .config # make -j4 [...]
b. Open top and watch your ARM VM's spin at 100% cpu:
USER PR NI VIRT RES SHR S %CPU %MEM COMMAND root 20 0 556m 362m 1492 R 101 6.0 qemu-system-arm -name armel-vm1 ... root 20 0 539m 278m 1472 R 101 4.6 qemu-system-arm -name armel-vm2 ... root 20 0 530m 216m 1472 R 99 3.6 qemu-system-arm -name armel-vm3 ... root 20 0 538m 208m 1472 R 101 3.5 qemu-system-arm -name armel-vm4 ...
As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.
You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.
--ewheelerinc 03:27, 20 August 2010 (UTC)
About the Author
Eric Wheeler is a Linux consultant and security professional available for hire at http://www.GlobalLinuxSecurity.com. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under Professor Jonathan Walpole.