Maemo.org Infrastructure/Autobuilder

Contents

[edit] Autobuilder and friends

maemo autobuilder setup

autobuilder consists of multiple VMs

[edit] drop VM

this VM has /etc/passwd synchronised with garage and ~ folders mounted via NFS from garage

account synchronisation is handled by scripts running on garage VM and then sync is triggered using ssh and scripts in /usr/local/bin

packages are uploaded to /mnt/incoming-builder via SCP

[edit] garage VM

this is the VM where stuff happens

password/account sync to gforge/postgresql is done using 

*/10 *   * * *   root    /usr/local/bin/add_groups_users_git_ssh.sh > /tmp/add_groups_users_git_ssh.log dev/null 2>&1

this also updates ~/.ssh/authorized_keys

garage also handles web extras-uploader (/var/lib/extras-assistant/) - package is uploaded and then moved to the same folder as packages uploaded to drop and then chowned using 

/var/lib/extras-assistant/bin/copy_package_files_to_autobuilder.sh

A lot of jobs on garage VM is done using local root crontab (/var/spool/cron/crontabs/root)

after package is uploaded it's processed by buildME

buildME runs as builder user and it's started from cron every minute 

* * * * * builder /home/builder/buildme

buildme is configured using /etc/buildme.conf

buildme takes care of couple things

  • verify that .tar.gz and other files are correct (checked using checksum from .dsc file)
  • select free destination (buildme can handle parallel builds on multiple hosts/users)
  • scp all required files to selected destination
  • start sbdmock on the destination
  • copy results back and resulting .deb to repository incoming folder (result_dir = /mnt/builder/%(product)s and repo_queue = /mnt/incoming/extras-devel/%(product)s/)
  • send emails to list and user uploading package

[edit] builder VM

this VM has standard installation of scratchbox with no targets configured (it's not required for sbdmock)

when sbdmock is started it cleans up old build folder, creates new target and prepares build enviroment and then runs dpkg-buildpackage

sbdmock also generates logfiles that are parsed by buildme

[edit] repository/stage VM

this is where repository management happens 

*/2 * * * *     repository      /home/repository/queue-manage-extras-devel.sh
*/5 * * * *     repository      /home/repository/queue-manage-extras.sh
*/5 * * * *     repository      /home/repository/queue-manage-community-testing.sh
*/5 * * * *     repository      /home/repository/queue-manage-community.sh

those scripts (and scripts inside /home/repository/queue-manager-extras) check for new packages in repository incoming folder and then move those to /var/repository/staging, regenerate Packages

(using sums that were previously cached) and sign it if required and then if any changes happened 

#touch .changed file, so we know that we need to sync to live
touch /var/repository/staging/community/.$dist.changed

this file is then checked by 

1003     10634     1  0 Mar18 ?        00:00:00 /bin/sh /usr/local/bin/packages/rqp.sh

started by /etc/init.d/repository-qp

this script starts rsync when required to sync to live repository

this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)

[edit] maintenance 

builder@garage:~$ ./buildme
>>>> /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3-1.dsc
[2013-11-23 18:22:39] REJECTED: "Can't stat /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3.orig.tar.gz: No such file or directory"
[2013-11-23 18:22:39] Unexpected error:
   Error'>: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/buildlib/app.py", line 81, in run
    mainfunc(argv, options, self._logger, self.conf)
  File "./buildme", line 641, in main
    'conf'    : conf
  File "/usr/lib/python2.7/dist-packages/buildlib/fsm.py", line 72, in run
    code = handler(self)
  File "/usr/lib/python2.7/dist-packages/buildlib/helpers.py", line 82, in wrapper
    rc = func(*args)
  File "./buildme", line 341, in setup_build
    reject_fname(env, exobj)
  File "./buildme", line 193, in reject_fname
    shutil.move(name, rejected_dir)
  File "/usr/lib/python2.7/shutil.py", line 289, in move
    raise Error, "Destination path '%s' already exists" % real_dst
Error: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
builder@garage:~$ ls -al /mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc
Retrieved from "http://bugs.maemo.org/Maemo.org_Infrastructure/Autobuilder"