Maemo Reconstructed

Image:Ambox_notice.png
 Maemo Reconstructed was a proposal that eventually led to the Mer project. You can find out more about Mer on its Wiki page: Mer, where you can also download flashable images and get involved in the project.

Our goal is to bring together Nokia and community developers who have an interest in the Maemo platform and reach a vision of the future Maemo system architecture that's satisfying to both Nokia and the community.

These are the low level technical system goals (use of booting process, busybox/coreutils, kernel features etc.), and not high level stuff like "Ogg support" or "open source everything". The process is called Maemo "Reconstructed" to indicate we can remix parts from and disassemble the current Maemo platform and augment it until we get a vision that would be fitting as the future of the Maemo platform.

If you have anything to contribute that is constructive criticism, or alternate proposals for parts of the proposal, please add it to the talk page. Again, this is not the place for discussion on high-level topics like A2DP support and finger-friendliness. These are low-level platform issues. If you want to shoot down part of the proposal, document your findings on why it is not possible or should not be done and discuss it on the talk page.

This proposal was initially written by Stskeeps (Carsten V. Munk), but is intended to be a unified vision that Nokia and the community will agree upon.

This discussion was sparked by:

Thanks to Ryan Abel for proofreading and to other #maemo members for comments.

A small bunch of individuals are working on a proof of concept of this platform, can be followed on http://jaiku.com/channel/reconstructedPOC and on irc.freenode.net #maemo.

Contents

[edit] The Maemo "Reconstructed" platform

The purposes of Maemo "Reconstructed" are to make Maemo a general platform for tablet devices (including Nokia devices), to make it more developer-friendly, and to bring it into alignment with standard Linux distributions. It is also within the philosophy that the platform should be "hackable", a main selling point with open-source friendly hardware.

We should stop seeing the tablets as strictly under-powered embedded systems, and see them for what they really are—powerful, power-efficient, economical handheld computers. Not simply hopped-up electronic calendars, but real computers that are capable of almost as much computing as a laptop.

For distribution, the methods, categories, and distributions outlined in the Maemo System Model - Part II could come to good use in Maemo "R". Maybe a part about Maemo vs Maemo + Nokia closed source components and seperation in archives?

There should exist scripts and documented methods for creating a Maemo "R" image that can be adapted to any setup or device. It should be easy to build your own jffs2 image.

Development of the Maemo "R" platform (non-device-specific and non-vendor-specfic differentiation) should be done in the open, with public SCM, bugtrackers and blueprints. This way, developers can adjust their applications to later 'stable' releases and know what is coming regarding the Maemo API.

One of the goals of the Maemo "R" platform is that it should be easy to port an existing desktop application by hildonizing it and adjusting to the form factor of a tablet, without having to deal with platform particularities like what features the busybox version of a tool uses or other factors. A secondary goal is to have a platform that is flexible enough to encourage experimentation and development for tablets.

[edit] Kernel

For the Maemo "R" platform specific kernels do not matter, but it may be relevant to discuss certain features that would be nice to have for the platform to work correctly.

[edit] Boot process

The basic boot process of a Maemo "R" platform consists of loading a kernel from a flash area, or from a file on a partition either on flash, MMC or USB, which then mounts the chosen root partition. This could be done with uBoot for instance (as seen on Beagle Boards), but the bootloader choice is up to the vendor, with the stipulation that it should support some of the following abilities:

In the boot process of the Maemo "R" platform, there should be a way to use alternate boot configurations. This menu of alternate boot configurations should be possible to enter through a key combination while turning on the device (as with the current bootmenu and holding down the menu key). The primary use for this would be to get the device into a rescue menu for recovering the rootfs after a reboot-loop or other crash.

The behaviour of the boot process should be modelled on grub's menu.lst, as to allow different boot configurations. Possible ways to deal with this can be alike with a /etc/bootmenu.d like setting which updates the boot configuration with refresh_bootmenu.d (inspiration from grub menu.lst, http://internettablettalk.com/forums/showthread.php?t=23051).

Cloning to and booting from a flash card is a popular hack, as it offers significantly more storage space and some performance improvements. A bundled boot menu system will streamline the setup of multi-boot systems, and reduce the impact of SSU on these setups. The boot menu should be unintrusive and invisible to the "normal" user, except when activated with the boot menu key combination (but may be activated to show the menu always at boot in a control panel).

There should exist a rescue partition in standard Maemo "R" image, possibly like http://www.youtube.com/watch?v=h24f2YjzWBE (or in text form). This can both activate certain services from the rescue partition or set boot settings such as framebuffer console or other settings for the Maemo system (think F8-while-boot in Windows.)

Many users often run into the reboot loops. A rescue partition will help users recover from these loops by offering a remote shell, dumping the user's files from /home/user to a connected PC, and allowing for the creation and restoration of backups on a flash card or a connected PC. These abilities should be modelled with a "plugin" structure, to encourage innovation in this area.

Busybox+uclibc should be used to ensure minimal rescue partition size. Toolkit for the right combination should be as easy to get as Maemo SDK. Busybox shouldn't be stuck at 1.0 like it is in initfs right now.

The rescue partition should be possible to update without having to reflash, and may employ a package system, to encourage innovation in this area (referencing to the case of initfs getting reflashed completely, removing boot menus and other initfs modifications, in Diablo).

[edit] Base system

Maemo needs to be brought into alignment with "major" distributions like Ubuntu and Debian, and the base system is a good place to begin. Maemo is its own distribution, targeted toward embedded systems, but on the other hand Maemo runs on systems that are as powerful as the computers these distributions were initially developed on, and many of the typical embedded tricks may simply not be needed. It is helpful seeing Maemo targeted at power-saving mobile devices where it's fine to be able to run a traditional "desktop" base system.

Nokia should embrace the Debian heritage of Maemo fully and put processes and practices into place that make Maemo a Debian derivative like Ubuntu, only better.

– Marius Volmer

This proposal tries to see Maemo as a deriative of Ubuntu. The reasons for this direction is:

  • Handhelds Mojo, which is sponsored by Nokia.
  • Debian, for now, targets armv4t, and Mojo has already done the hard work to target typical Maemo device processors.
  • Taking Ubuntu over Debian (for now), gives us upstart along (part of Task:Maemo roadmap/Fremantle), which has not been adapted properly (except in Debian experimental) in Debian just yet.
  • A remark however should be that Debian and Ubuntu is not very different, at base system level. The choice of Ubuntu here has been done to illustrate a base Linux system optimized for the Internet Tablets, with some of the technologies such as Upstart pre-existing in the distribution. armv5 was chosen for no particular reason except that the Hasty armv6 build has not yet been completed.

Maemo base system includes:

  • 'minbase' Mojo handhelds Ubuntu 'Hasty' (based on Hardy, or later)
    • Motivation: This is a proof of concept. How functional and how well performing is a standard minimal ubuntu base image on a N800? Our restrictions is about 250mb flash availiable to us, with ideally space to spare.
    • Details:
      • # cp /usr/share/debootstrap/scripts/hardy /usr/share/debootstrap/scripts/hasty
      • # debootstrap --arch=arm --variant=minbase hasty /target http://repository.handhelds.org/hasty-armv5el-vfp/
      • # chroot /target mount -t proc proc /proc
      • # chroot /target mount -t devpts devpts /dev/pts
      • # apt-get install apt-utils udev
      • # cd /dev; /sbin/MAKEDEV fb0 fb1 and such.
      • # apt-get autoclean; apt-get clean
      • add in a couple of boot scripts from and framebuffer console kernel drivers from Deblet, and docpurge-1.6 from Maemo.
      • Now you have a bootable minimal ubuntu image (tested on N800 with bootmenu with linuxrc ability), with full coreutils and bsdutils, etc.
    • Performance: From selecting in boot menu to boot, 12 seconds to get to framebuffer console login, with ext3 mount, udev and all.
    • Size: uncompressed 89.5mb but, mkfs.jffs2 -r /target -o maemo-r.jffs2 -e 128 -l -n , 37.0mb in JFFS2.
    • Contents: http://bsd.tspre.org/~stskeeps/minboot.txt + docpurge
    • Other examples, http://bsd.tspre.org/~stskeeps/ubuntumin.txt (69m jffs2, may have forgotten docpurge in this one.)
    • Advantages: We now have a full ubuntu base system on our tablet. We've got Upstart for for free without any development. We have a base we can easily work with upstream on, as we base on typical packages. The mojo project also gets to have instant value to the Nokia/Maemo development and Nokia gets a return from the research investment in it.
    • Discussion: Is busybox really needed on a tablet? In initfs, sure, but in the actual system? Having sane coreutils and bsdutils (and debconf) -really- helps porting. Now that dash exists, is there a reason for busybox as sh? This avoids any conflicts with coreutils, bsdutils etc if they are going to be used.
      • Following experiment was done:
        • These packages dpkg-deb -x'ed into a path: findutils_4.2.32-1ubuntu2_arm.deb gzip_1.3.12-3.2_arm.deb hostname_2.94_arm.deb ifupdown_0.6.8ubuntu8_arm.deb module-init-tools_3.3-pre11-4ubuntu5_arm.deb net-tools_1.60-19ubuntu1_arm.deb procps_3.2.7-9ubuntu1-mojo0_arm.deb sed_4.1.5-5_arm.deb tar_1.19-3_arm.deb coreutils_6.10-3ubuntu2_arm.deb grep_2.5.3~dfsg-3_arm.deb mount_2.13.1-5ubuntu1_arm.deb bsdutils_2.13.1-5ubuntu1_arm.deb dash_0.5.4-8ubuntu1_arm.deb debianutils_2.28.2-0ubuntu1_arm.deb vim-tiny_7.1-138+1ubuntu3_arm.deb mawk_1.3.3-11ubuntu2_arm.deb mktemp_1.5-5ubuntu2_arm.deb sysvutils_2.86.ds1-14.1ubuntu45_arm.deb iputils-ping_20071127-1_arm.deb psmisc_22.6-1_arm.deb time_1.7-21build1_arm.deb bash_3.2-0ubuntu16_arm.deb login_4.0.18.2-1ubuntu2_arm.deb passwd_4.0.18.2-1ubuntu2_arm.deb
        • Stripping man pages, locales and info pages. 7.6mb uncompressed 3.8mb jffs2 flash. And with busybox getting same treatment, 300k flash.
      • Is 3.5mb flash so much to sacrifice for making porting for Maemo a matter of hildonization and platform-centric things, instead of hunting debconf, busybox incompatibilities and such for weird libraries you never heard of?
      • Are there any direct and noticable differences in performance that justifies the use of busybox, and do they justify the lessened ability of being able to port to Maemo without hassle?
      • Result: It should be possible to have busybox in Maemo images instead of coreutils and others, but the busybox package should then be able to co-exist with coreutils. If a package needs coreutils, it should depend on coreutils which will then take over key functions. Maybe a system of depending on coreutils | busybox-coreutils, could do the trick, for the packages that work fine with busybox coreutils.
    • Problems: For some reason the mojo people decided to divert from Debian arch notation and call their ARM EL distribution arch type 'arm', not 'armel', which is in wide use with Maemo.
  • Upstart
    • Part of Fremantle roadmap, and with the integration of alarmd functionality (Events generated at timed intervals or scheduled times), Upstart could replace alarmd.
  • DSME (Device State Management)
    • Basic device/Mobile service DSME should be open sourced, but may load modules of hardware-specific closed source modules (CAL access, watchdog kicker, etc), and DSME protocol should be documented and the modules associated with it (CAL access seems to be a touchy subject though).
    • DSME is a good part of a mobile platform (process watchdog, etc), but needs documentation of the communication protocol (even though most of it has been reverse engineered) for community OS developers to work properly with it. Can it really be true things like Advanced Backlight (http://adv-backlight.garage.maemo.org/) has to use undocumented DSME protocol to do what it should?
  • OHM (Open Hardware Manager)
    • Part of Task:Maemo roadmap/Fremantle. http://ohm.freedesktop.org/wiki/UseCases OHM may apply device-specific plugins but these are not included in the Maemo "R" base. Developers should strive towards not using sysvinit init scripts, but go for upstart job descriptions instead, which can be controlled by OHM (disable RSS feed reader download attempts while obviously offline, for instance).
  • GUPnP
  • Meta Tracker (trackerd)
  • D-Bus
    • Part of the Maemo platform as of now, - is ideal for embedded systems as it works with signals and events.
  • HAL
    • Part of the Maemo platform as of now, needed for OHM. HAL should be patched towards mobile systems but should not contain device specific patches. These should be published as HAL addons instead (and may be closed source, example could be hald-addon-bme as the principle done right, and hald-addon-omap-gpio not), as to keep as close to HAL upstream as possible.
  • Generic API for connectivity (libconic comes to mind), using D-Bus. It should not be needed to include a ICD as some vendors may vary on that (Nokia), but libconic should be generic and usable in both Nokia ICD and NetworkManager based setups. Ideally a NM-based ICD or a stub ICD should be provided along with platform, but should be replaceable. The ICD would traditionally depend on software like iptables, udhcpc, dnsmasq etc.
    • Many Maemo applications depend on libconic and this could be a showstopper for portability.
  • Udev
    • Part of Maemo platform already, and useful for desktop and embedded systems.
  • Ke-recv
    • Some daemon representing this (Kernel event handling + scripts, maybe OHM?)
  • GConf
    • Central repository for system-wide configuration.

Other things like supporting libraries such as OpenSSL and such are not directly mentioned but obviously needed as dependancies.

[edit] Hardware Support

A Maemo "R" image may contain closed-source hardware-specific firmware, drivers (cx3110x), daemons (BME, hw watchdog kicker). This will probably be common on commercial platforms since there are trade secrets to protect and NDAs to consider. Vendors for the Maemo platform should strive towards allowing redistribution of the binary blobs (restricting to use only on the hardware they are intended for should be acceptable), to encourage remixing and experimentation with the system and the device.

In case of kernel drivers these should ideally be open source, but this may not always be possible. Nokia, for instance is going in the right direction with open source wifi driver. In case of newer kernels for "old" devices, vendors should aim for giving a community developer access to kernel module source for recompiling (with possible NDA) for later versions or provide build services (insert kernel tar.gz, get out .ko).

In case of closed source daemons (BME, DSME modules) the protocols should ideally be documented (it should be possible to get remaining battery time and bars to show and such from BME, maybe even current power usage of the device). This will satisfy most OSS OS developers, but the problem about closed source daemons is that there should optimally be updates of the software, even for old devices. Backwards compatible daemons could be an idea, maybe automated build services again/community/head contributors access to recompile, so Nokia wouldn't have to invest time to support "old" devices.

In case of generic devices that will exist on multiple platforms (battery for instance), it is encouraged to work on standardization of the interface for the daemons (HAL battery status comes to mind).

[edit] User Experience Base

  • Xorg
    • Part of Fremantle roadmap.
    • Device-specific Xorg drivers (omap for N8x0's, other devices has other drivers, etc.). It should not be a requirement which Xorg driver is in use, but the capabilities of this.
    • In the case of OMAP framebuffers, work has been done at http://gitweb.pingu.fi/?p=xf86-video-omapfb.git;a=tree for a Xorg driver for both BeagleBoards and N8x0, packaged for Debian at http://nchipin.kos.to/deblet/
  • GStreamer
    • Part of Maemo platform as of now, ideal for codecs, include vendor-specific proprietary ones.
  • PulseAudio
  • BlueZ
    • Part of Maemo platform as of now. Optional part of platform?
  • Hildon desktop & Matchbox WM
    • A basic part of the Maemo platform as of now. However, the initialization of Hildon should be done through an alternatives system like in Debian (update-alternatives --config) x-session-manager, so having Hildon on top of a Maemo system is one use interface. The Maemo system should not depend on it being Hildon running on the device.
  • GTK/GDK/etc.
    • The base graphical framework of Maemo. This GTK version should be Maemo 'GTK', but the following things should hold true for the Maemo GTK.
      • Maemo GTK is designed for power-saving low-freq devices. It is designed for touchscreens and with memory conservation in mind
      • It should not alter its behaviour in such a way that a certain window manager or desktop environment is needed to run applications using this toolkit. If this change of behaviour is needed, the toolkit should detect the WM/DE and enable the behaviour if it's Hildon+Matchbox.
      • It should not change upstream API (but may add or cause breakage that would make normal GTK applications not work with it.
  • GTKMM
    • There should be possibility to use two versions of GTKMM, one with vfuncs removed for embedded usage and one with them enabled for running GTK applications that require these to make porting easier (or simply enabling vfuncs in GTKMM for Maemo.). The replacement GTKMM may be a package that is downloadable which replaces the non-vfunc GTKMM.
  • Hildon application manager
    • A mobile system must have ability to add/remove applications..
  • Hildon control panel
    • The Maemo "R" platform can be adjusted from in here, but the configuration settings used in various applets should be fully documented on how to do programmatically.
  • Hildon input method
    • Input method for touchscreens.
  • Basic Hildon theme
    • Plankton or the likes to start building Maemo systems that look "good" from the start.
  • Other open source components currently in Maemo.
    • There are plenty other open source components currently in Maemo, which is not listed here, but considered part of the platform.

The goal of this user experience base is to have a solid open-source base to build Maemo platform devices, both for Nokia and for community to remix, or use on other devices. This user experience base should be usable, but should be augmented with vendor-specific differentiations (ICD, Applications, Browser support, drivers, hardware support, home applets, etc.), but documenting the interfaces developed for these.

Developing for the Maemo platform, and not for the actual devices, should be encouraged except in cases where the applications require certain non-standardized (yet) interfaces. This will be useful for hardware vendors as well as development and change in hardware contents will be easier to achieve, with the applications still working on new hardware as they target the Maemo platform, not the device.

[edit] SDK (developer kit for the platform)

The SDK is a basic part of developing for Maemo.

Maemo "R" SDK should be easy to install. It should be a matter of apt-get'ing a package or installing a VMware appliance for non-Linux users. It's alright for device-specific dev packages & emulator images to require a license to download, but the base Maemo "R" SDK should be easily installable and allow you to start developing for the base Maemo "R" platform immediately. You shouldn't have to go through hoops to get Scratchbox + Rootstrap + Generic Maemo device emulator image and such, - this attracts developers.

A prime example is Maemo SDK+, http://maemo-sdk.garage.maemo.org/

Maemo "R" SDK should not only focus on native application development, but also embrace developer environments for non-native languages such as Python, Java, etc, to show the versatile nature of the Maemo "R" platform. Maemo SDK+ would be able to contain these and could also allow for Eclipse-based development environments etc.

Maemo "R" SDK should not be the only way to develop for the tablet - it should be just as possible to get the needed compilers and dev packages and be able to compile on tablet - which is easy as the rootstrap is really the Maemo "R" OS with build-essentials packages on top.

Maemo "R" SDK, should consist of four parts:

[edit] Scratchbox 2 (cross-compilation environment)

Build programs for another architecture/target than on the one compiling.

Packages for Maemo should not expect Scratchbox to be where they're compiled (There's been seen examples of this amongst Nokia code.). This increases portability of Maemo platform. A package that builds in Scratchbox should also build on a tablet.

Scratchbox 2 is obviously the way to go, as it provides a more seamless way to deal with cross-compilation.

Cross-compiliation certainly speeds up compilation and allows a proper developer environment for developing for Maemo "R".

[edit] Maemo "R" rootstraps

The SDK should contain at least one rootstrap - the Maemo "R" base OS, targeted towards both cross-compilation towards the reference platform, for instance targeted towards ARMv5EL and the QEMU versatile-pb emulator.

Developers should be encouraged to develop for this platform API - the reference Maemo "R" platform API. Though, there will be device-specific features that vendors would like to make developers able to develop for. The vendors would then either provide hardware support packages to include on top of the base OS or provide additional rootstraps which represent the device and device-specific APIs shipped on the actual tablets with build-essentials and such on top. These rootstraps or packages would probably be provided under some kind of licensing scheme.

A rootstrap may be either device-specific (processor types/etc) or have device-specific packages in it. Vendors providing device-specific interfaces for the platform should strive towards standardizing similar types of hardware interfaces (HAL battery templates come to mind, GPS or Bluetooth interfacing), but the vendor may provide subtypes of the APIs - extended APIs for component-specific things.

Maemo has great possibility to become a general platform for tablet like devices. The possibility of additional device platforms which also embrace the Maemo "R" base direction will cause every Maemo-using device to have additional applications as well when they appear for other devices.

[edit] QEMU, able to emulate target device(s) emulated

Vendors should strive for contributing hardware emulation for QEMU for their devices or provide stubs inside the hardware support packages but should act as close to the actual device as possible.

There should exist a reference generic Maemo "R" platform to develop for, with associated emulator - in the proof of concept this will be the n800 machine and arm1136-r2 CPU emulated in qemu.

Vendors should provide emulator images (in the form of flash images, disk images) associated to the device rootstraps to make users able to test on their platforms.

N8x0 emulation is already seen (http://blog.haerwu.biz/2008/04/11/nokia-n800-emulation/) and in qemu SVN HEAD. Being able to boot their development environments will allow the user to instantly test how their applications work on the actual device or an emulated version.

Maemo SDK+ already provides the "maemo-runtime" scripts which could be adapted to include features like this.

[edit] Deployment and debugging support scripts

It's much better when you can test on an emulated device instead of dealing with a rootstrap trying to act like it's a device. Easy development ability is needed - it's a question that's often seen when people start developing for Maemo (how do we transfer to the device? Try see Visual Studio's deployment ways)

Work could be done on easy deployment of developed packages (# sb-copy-to-tablet -i usb, -i emu, etc) or running in similar manner to sbrsh with NFS and the likes.

Again, Maemo SDK+ seems to provide much of this and is definately the way to go regarding SDK plugins (GUIs, Anjuta as an example). Through a simple interface it could be extended to provide deploying and remote debugging features. GDB supports remote debugging for instance.

Retrieved from "http://bugs.maemo.org/Maemo_Reconstructed"