• Intro
• Downloads
• Community
• Development
• News
• TALK

Distcc

Latest revision as of 08:38, 20 August 2010

For those who have tried to compile on the N900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, is often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the N900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as:

n900-chroot#  The easydebian chroot prompt (you can ssh in too if you like)
host-root#    The high-power desktop system to run the ARM VM's under qemu
guest-root#   The ARM VM's themselves.

[edit] Install Easy Debian

Before continuing, you will need to install Easy Debian on your phone. This is the fastest way to get a working native gcc environment on the N900. Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as below in the easydebian chroot.

[edit] Install a few QEMU VMs

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

Lenny armel qemu images

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system, usually:

apt-get install qemu

and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
   -initrd initrd.img-2.6.26-1-versatile \
   -hda debian_lenny_armel_small.qcow2 \
   -append "root=/dev/sda1"

[edit] Install the required packages on the VM

First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it. Then add the packages:

guest-root# apt-get update
guest-root# apt-get upgrade
guest-root# apt-get install build-essential ccache distcc

[edit] Configure distcc and ccache

1. Add the hosts to /etc/distcc/hosts
2. Configure /etc/default/distcc with at least:

   STARTDISTCC="true"
   # configure your subnet
   ALLOWEDNETS="127.0.0.1 192.168.0.0/16"
   

3. create a ccache/distcc wrapper

   root# mv /usr/bin/gcc /usr/bin/gcc-orig
   root# cat<<EOT > /usr/bin/gcc-ccache
   #!/bin/sh
   export CCACHE_PREFIX=distcc
   # List the distcc hosts here; I had trouble with /etc/distcc/hosts working
   export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
   exec ccache gcc-orig "$@"
   EOT
   root# chmod 755 /usr/bin/gcc-ccache
   

   Be sure to do this on the N900 too!

4. link the wrapper as the new gcc:

   root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc
   

5. test the distcc-ccache-gcc wrapper:

   root# gcc -version
   gcc-orig: no input files
   distcc[2486] ERROR: compile (null) on localhost failed
   

   The distcc error is expected

[edit] Configure the network to DHCP

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

[edit] Halt the vm and launch multiple instances

1. Halt the guest and close the window: guest-root# shutdown -h now
2. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the N900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).

    root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
    #!/bin/sh
    iface="$1"
    bridgeName=`basename "$0" | cut -f3 -d-`
    echo "bridgeName=$bridgeName iface=$1"
    /sbin/ifconfig "$iface" 0.0.0.0 up
    /usr/sbin/brctl addif "$bridgeName" "$iface"
    EOT
    chmod 755 /usr/local/bin/qemu-ifup-br0
   
    host-root# brctl addbr br0
    host-root# brctl addif br0 eth0
    host-root# ifconfig br0 192.168.99.222
    host-root# ip route replace default via 192.168.99.1
   

3. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:

          host arm-vm1 {
                  hardware ethernet 00:16:3e:00:00:01;
                  fixed-address 192.168.99.101;        }
          host arm-vm2 {
                  hardware ethernet 00:16:3e:00:00:02;
                  fixed-address 192.168.99.102;        }
          host arm-vm3 {
                  hardware ethernet 00:16:3e:00:00:03;
                  fixed-address 192.168.99.103;        }
   

4. Launch the vm's:

    host-root#  for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
       -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
       -initrd initrd.img-2.6.26-1-versatile \
       -hda debian_lenny_armel_small.qcow2 \
       -append "root=/dev/sda1" \
       -net nic,macaddr=00:16:3e:00:00:0$arm \
       -net tap,script=/usr/local/bin/qemu-ifup-br0 \
       -daemonize \
       -snapshot ; done
   

5. The bridge listing should look something like this:

    host-root# brctl show
    br0		8000.001cc0b1dadf	no		eth0
    							tap1
    							tap2
    							tap3
   

[edit] Run a distributed build on your N900

1. Download a recent vanilla kernel:

    cd /usr/src
    wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
    cd linux-2.6.34
       [...]
    make -j3 defconfig
    *** Default configuration is based on 'versatile_defconfig'
    #
    # configuration written to .config
    #
    make -j4
       [...]
   

2. Open top and watch your ARM VM's spin at 100% CPU:

    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM COMMAND                          
    root      20   0  556m 362m 1492 R  101  6.0 qemu-system-arm -name armel-vm1 ...
    root      20   0  539m 278m 1472 R  101  4.6 qemu-system-arm -name armel-vm2 ...
    root      20   0  530m 216m 1472 R   99  3.6 qemu-system-arm -name armel-vm3 ...
    root      20   0  538m 208m 1472 R  101  3.5 qemu-system-arm -name armel-vm4 ...
   

As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the N900's localhost to spin the native 600 MHz CPU too---and I leave this as an exercise for the reader.

[edit] References

• Click here for the forum thread on this subject
• Original author