Distcc

For those who have tried to compile on the N900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, is often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the N900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as: 

n900-chroot#  The easydebian chroot prompt (you can ssh in too if you like)
host-root#    The high-power desktop system to run the ARM VM's under qemu
guest-root#   The ARM VM's themselves.

Contents

[edit] Install Easy Debian

Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the N900. Once it is installed, install ccache, distcc, and build-essential: 

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as below in the easydebian chroot.

[edit] Install a few QEMU VMs

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

Lenny armel qemu images

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system, usually: 

apt-get install qemu

and boot the VM: 

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
   -initrd initrd.img-2.6.26-1-versatile \
   -hda debian_lenny_armel_small.qcow2 \
   -append "root=/dev/sda1"

[edit] Install the required packages on the VM

First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages: 

guest-root# apt-get update
guest-root# apt-get upgrade
guest-root# apt-get install build-essential ccache distcc

[edit] Configure distcc and ccache

  1. Add the hosts to /etc/distcc/hosts
  2. Configure /etc/default/distcc with at least: 
    STARTDISTCC="true"
# configure your subnet
ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
  3. create a ccache/distcc wrapper 
    root# mv /usr/bin/gcc /usr/bin/gcc-orig
root# cat<<EOT > /usr/bin/gcc-ccache
#!/bin/sh
export CCACHE_PREFIX=distcc
# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
exec ccache gcc-orig "$@"
EOT
root# chmod 755 /usr/bin/gcc-ccache

    Be sure to do this on the N900 too!

  4. link the wrapper as the new gcc: 
    root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
  5. test the distcc-ccache-gcc wrapper: 
    root# gcc -version
gcc-orig: no input files
distcc[2486] ERROR: compile (null) on localhost failed

    The distcc error is expected

[edit] Configure the network to DHCP 

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

[edit] Halt the vm and launch multiple instances

  1. Halt the guest and close the window: guest-root# shutdown -h now
  2. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the N900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common). 
     root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
 #!/bin/sh
 iface="$1"
 bridgeName=`basename "$0" | cut -f3 -d-`
 echo "bridgeName=$bridgeName iface=$1"
 /sbin/ifconfig "$iface" 0.0.0.0 up
 /usr/sbin/brctl addif "$bridgeName" "$iface"
 EOT
 chmod 755 /usr/local/bin/qemu-ifup-br0

 host-root# brctl addbr br0
 host-root# brctl addif br0 eth0
 host-root# ifconfig br0 192.168.99.222
 host-root# ip route replace default via 192.168.99.1
  3. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this: 
           host arm-vm1 {
               hardware ethernet 00:16:3e:00:00:01;
               fixed-address 192.168.99.101;        }
       host arm-vm2 {
               hardware ethernet 00:16:3e:00:00:02;
               fixed-address 192.168.99.102;        }
       host arm-vm3 {
               hardware ethernet 00:16:3e:00:00:03;
               fixed-address 192.168.99.103;        }
  4. Launch the vm's: 
     host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
    -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
    -initrd initrd.img-2.6.26-1-versatile \
    -hda debian_lenny_armel_small.qcow2 \
    -append "root=/dev/sda1" \
    -net nic,macaddr=00:16:3e:00:00:0$arm \
    -net tap,script=/usr/local/bin/qemu-ifup-br0 \
    -daemonize \
    -snapshot ; done
  5. The bridge listing should look something like this: 
     host-root# brctl show
 br0		8000.001cc0b1dadf	no		eth0
 							tap1
 							tap2
 							tap3

[edit] Run a distributed build on your N900

  1. Download a recent vanilla kernel: 
     cd /usr/src
 wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
 cd linux-2.6.34
    [...]
 make -j3 defconfig
 *** Default configuration is based on 'versatile_defconfig'
 #
 # configuration written to .config
 #
 make -j4
    [...]
  2. Open top and watch your ARM VM's spin at 100% CPU: 
     USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                          
 root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
 root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ...
 root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
 root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...

As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the N900's localhost to spin the native 600 MHz CPU too---and I leave this as an exercise for the reader.

[edit] References

Retrieved from "http://bugs.maemo.org/Distcc"