Comment 1 Eero Tamminen 2008-07-02 17:47:11 UTC

Updating summary; initfs doesn't have rootstrap, it's built with system similar
to Garnome (I guess you could just use copy of initfs contents as your
rootstrap
if really needed) and toolchain would also need to be eabi, scratchbox.org 
toolchains seem all to be oabi based ones.

Comment 2 Frantisek Dufka (reporter) 2008-07-03 09:56:35 UTC

I'm not sure what is proper solution. Maybe it should be part of SDK with all
the overhead it brings and I can imagine this could take quite long to decide
and accomplish. Would it be also possible to simply look for correct eabi
toolchain inside Nokia (like
scratchbox-toolchain-cs2005q3.2-uclibc-arm-eabi_0.9.8.5-3_i386.deb mentioned in
http://maemo.org/community/wiki/os2007hackereditionarchives/ or newer) and put
it to scratchbox.org?
http://www.scratchbox.org/download/files/sbox-releases/legacy/contrib/ may be
appropriate. Thanks.

Comment 3 Andre Klapper 2008-11-11 15:17:23 UTC

Also note: Fremantle doesn't have initfs according to bug 3745 comment 2.

Comment 4 Frantisek Dufka (reporter) 2008-11-11 16:14:16 UTC

(In reply to comment #3)
> Also note: Fremantle doesn't have initfs according to bug 3745 comment 2.
> 

This request is mainly about getting _current_ toolchain (used for building all
OS2007 and OS2008 initfs images so far) before it is too late and it gets lost.
It is great that Fremantle changes boot procedure so we won't have this problem
in future but this is about hacking initfs of current N8x0 devices.

Since gcc and uclibc is GPLed, there are hopefully no secrets so it should be
possible to ask  Nokia people in charge of building the initfs and dump the
toolchain somewhere as suggested in comment #2

Comment 5 Quim Gil 2008-12-29 16:11:41 UTC

Setting HIGH visibility.

Question, would this feature be relevant for any Fremantle Community Edition
work?

Comment 6 Carsten Munk 2008-12-29 16:34:40 UTC

Yes, it definately has relevance for community editions. The reasons for these
are already listed above in fanoush's posts. It will also make it possible for
the community to maintain initfs for 770, N8x0 when Nokia no longer supplies
software for the devices, and will aid in intended community edition work with
rescue capabilities, etc.

Comment 7 Quim Gil 2008-12-29 17:56:09 UTC

(In reply to comment #6)
> Yes, it definately has relevance for community editions.

An additional reason to support this.

Comment 8 Faheem Pervez 2008-12-29 18:34:27 UTC

(In reply to comment #7)
> (In reply to comment #6)
> > Yes, it definately has relevance for community editions.
> 
> An additional reason to support this.
> 

Fremantle may not have an initfs but the 770, N800 and the N810 do and the
community do not have enough information about modifying the partition tables
etc to remove an initfs so I guess we are stuck with it :-).

Being able to compile for initfs would be a massive advantage. At the moment,
we have to use a uclibc toolchain that, by coincidence it seems, has the same
version number of uclibc in initfs but is not compatible with it (undefined
symbols when trying to run the binary from initfs) so I have to compile
statically. We don't exactly have the biggest amount of space in initfs so
static compiling is not the best option.

Here is some examples of binaries I've compiled statically using that toolchain
that run under initfs:
http://qwerty12.maemobox.org/initfs/

(the binaries are stripped)

Comment 9 Eero Tamminen 2008-12-30 10:53:16 UTC

> and the community do not have enough information about modifying the partition tables etc to remove an initfs so I guess we are stuck with it :-).

You could file an enhancement request for that information.

Comment 10 Andre Klapper 2009-05-27 23:27:44 UTC

Assuming that this is still an issue for Mer:
Can any Nokian CC'ed here please try to find out whether this is possible?

Comment 11 Frantisek Dufka (reporter) 2009-05-28 11:09:34 UTC

(In reply to comment #10)
>  whether this is possible?

Just a comment - 'possible' may not be the right word for this :-) At least
gcc, uclibc, busybox is (L)GPLed.  The core of this issue is the very same
thing that recently FSF sued CISCO/Linksys for and won or what is the whole
http://gpl-violations.org/ site about (Harald Welte chasing companies that ship
busybox etc. in embedded devices without shipping sources).

Nokia did its very own combination of uclibc 0.9.28 and EABI patches (AFAIK
upstream 0.9.28 does not support EABI without patching) and built gcc toolchain
and shipped binaries of uclibc and busybox with every tablet. It is Nokia's
obligation to provide exact sources they used so we can rebuild the
toolchain/uclibc/busybox. It may be a bit easier for Nokia to just provide the
toolchain binary as a start which is what original request and suggestion in
comment #2 was about. And frankly I expected this to be resolved in days/weeks
not years :-)

So sorry for quibbling over the 'possible' word but I felt this needs to be
said.

Comment 12 Andre Klapper 2009-06-16 14:40:54 UTC

(In reply to comment #11)
> gcc, uclibc, busybox is (L)GPLed. It is Nokia's obligation to provide exact 
> sources they used so we can rebuild the toolchain/uclibc/busybox. It may be 
> a bit easier for Nokia to just provide the toolchain binary as a start which 
> is what original request and suggestion in comment #2 was about.

Soumya / Quim: Sounds a lot like a legal requirement to fix this issue.
Can you find out and ping the responsible people on this?

Comment 13 Mohammed Hassan 2009-07-02 15:06:42 UTC

My understanding is that Nokia should provide the sources for uclibc and
busybox upon request but is not forced to provide the sources for gcc unless we
release binaries for it.

Of course I'd love to see all the sources being published but let's be fair :)

Comment 14 Frantisek Dufka (reporter) 2009-07-02 16:05:26 UTC

(In reply to comment #13)
> My understanding is that Nokia should provide the sources for uclibc 

Yes. If there is no better solution, getting exact uclibc sources used for
building uclibc binary in initfs should be enough to both close this bug and
(IMO, IANAL) to comply with GPL.

gcc/binutils source is not needed but could help a lot, especially if there are
any patches to it 

busybox is good test case for verification of the binary built with the uclibc
based toolchain  - if everything is ok, we should get same of very similar
binary as is currently distributed with initfs

Comment 15 Carsten Munk 2009-07-12 16:33:08 UTC

While this does not correspond to the legal issue, it is a possible solution to
the problem we're trying to solve:

One of the Gentoo/N8x0 people set out to make an experiment: is it possible to
make 'busybox' from initfs run inside a uclibc 0.9.30.1 compiled by gcc4.4
chroot  (libgcc is needed too for busybox)

Turns out it actually works - as uclibc is probably backwards compatible. So if
we replace the uclibc in initfs, we can now build non-static binaries that work
using our uclibc, and there's a big chance old-uclibc blobs (BME,  DSME, etc)
work, since busybox does.

Test case:

1. install gentoo. 
2. emerge crossdev
3. build crosstoolchain: crossdev armv6j-gentoo-linux-uclibceabi
4. emerge crossdev-wrappers
5. emerge armv6j-gentoo-linux-uclibceabi -v1 =sys-devel/gcc-4.4*
6. copy /usr/armv6j-gentoo-linux-uclibceabi/ to tablet. 
7. copy busybox from initfs into it
8. chroot.

Additional test done was text2screen from initfs.

Thanks to slonopotamus for taking the time to test this hypothesis out.

Comment 16 Quim Gil 2009-10-14 08:33:20 UTC

What is the status of this?

Comment 17 Carsten Munk 2009-11-04 16:30:46 UTC

An educated guess: The uclibc version in use may be r14638 from
svn://uclibc.org/trunk/uClibc which seems to match the idea of 0.9.28 + EABI
support. Can someone check if this one matches the initfs?

Comment 18 Frantisek Dufka (reporter) 2009-11-05 14:04:26 UTC

(In reply to comment #17)
> An educated guess: The uclibc version in use may be r14638 from
> svn://uclibc.org/trunk/uClibc which seems to match the idea of 0.9.28 + EABI
> support.

Why r14638?

$ svn log -r r14638 svn://uclibc.org/trunk/uClibc
------------------------------------------------------------------------
r14638 | vapier | 2006-03-24 01:16:17 +0100 (Fri, 24 Mar 2006) | 8 lines

Richard Sandiford writes:
m68k is one of those targets that does not accept -fPIC across the board.
It is supported for 68020 and above, but not for 68000, 68010, or Coldfire,
where using -fPIC leads to a compiler error.  (Note that the error is flagged
by cc1 proper, so it probably wouldn't have mattered with older compilers.
It only matters now because cc1 is used to preprocess assembly code.)
The attached patch makes m68k targets use -fpic instead of -fPIC.

------------------------------------------------------------------------

Nothing EABI specific. When checking uclib site they moved to git so I am not
sure how to find r14638 there or how to browse old svn repository or make
proper svn diff.

So, why do you think this revision is important and what are those tarballs
mentioned in post here http://talk.maemo.org/showthread.php?p=366085#post366085

> Can someone check if this one matches the initfs?
> 

What does match means exactly? Busybox and initfs stuff runs with new library?
newly built random binary (busybox?) works with old library? binary diff of
library?

I wonder which toolchain and kernel headers one should use for building and
what uclibc configuration and compiler flags one should use. At least the
hacker edition wiki page mentioned it is build with vfp (i.e. it did not run on
770 because of that).

So before trying random stuff can any Nokian here confirm what is the status
here? There is no trace of how initfs was built for Diablo inside Nokia? No
debian changelogs? No sources of "We used rx-34-initfs from apt-repository as a
base?" mentioned in [1]? No binaries of that
scratchbox-toolchain-cs2005q3.2-uclibc-arm-eabi_0.9.8.5-3_i386.deb mentioned in
[1] too? No backups?

1. currently available here
http://maemo.org/community/wiki-closed/os2007hackereditionarchives/

Comment 19 Frantisek Dufka (reporter) 2009-11-06 13:45:20 UTC

Created an attachment (id=1545) [details]
TortoiseSVN screenshot browsing uclibc svn

I have tried TortoiseSVN which can browse remote svn: repository and show nice
revison graphs and allows to search logs and it looks like 
- revision 11186 is 0.9.28 tagging
- revision 13690 adds ARM EABI support
- revision 18564 branches 0.9.29

revision 13690 is touching a lot of files and adds core eabi support but there
appears to be other newer revisions touching eabi (bugfixes), see the
screenshot with list of revisions that contains eabi string (in comments of
file path)

Comment 20 Carsten Munk 2009-11-17 13:08:13 UTC

After poking a bunch of people - basis source code for initfs toolchain and
uclibc can be found at
http://scratchbox.org/download/files/sbox-releases/apophis/src/scratchbox-toolchain-cs2005q3-2-20060524.tar.bz2
(been there since 2007). This must be compiled within a HOST target on SB1.

Patch to enable uclibc-eabi in the scratchbox toolchain can be found at
http://stskeeps.subnetmask.net/maemo.org/uclibc-eabi.patch.

Binary package that I've compiled myself based on these sources can be found at
http://stskeeps.subnetmask.net/maemo.org/scratchbox-toolchain-cs2005q3.2-uclibc-arm-eabi_0.9.8.5-7_i386.deb

I have tested a smaller dynamically linked binary (utelnetd) on N8x0 initfs and
this seemed to work without hassle.

Let me know if there's any problems with this toolchain and I hope that this
will help initfs development on N8x0. I haven't had a chance to test it towards
770 initfs.

Comment 21 Frantisek Dufka (reporter) 2009-11-17 14:52:29 UTC

(In reply to comment #20)
> After poking a bunch of people 

Excellent. Thanks. Too bad nobody poked them one and half year ago :-( 

>- basis source code for initfs toolchain and
> uclibc can be found at
> http://scratchbox.org/download/files/sbox-releases/apophis/src/scratchbox-toolchain-cs2005q3-2-20060524.tar.bz2
> (been there since 2007). This must be compiled within a HOST target on SB1.

Oh, who would have thought about this source. I've tried two other binary ones
at that time.

> 
> Patch to enable uclibc-eabi in the scratchbox toolchain can be found at
> http://stskeeps.subnetmask.net/maemo.org/uclibc-eabi.patch.
> 
> Binary package that I've compiled myself based on these sources can be found at
> http://stskeeps.subnetmask.net/maemo.org/scratchbox-toolchain-cs2005q3.2-uclibc-arm-eabi_0.9.8.5-7_i386.deb

Thanks a lot. What is the history of uclibc-eabi.patch? Is this something you
figured out semi-randomly or it is the patch Nokia used? Jut to know if we
should expect any surprise when compiling and running random stuff.

> Let me know if there's any problems with this toolchain and I hope that this
> will help initfs development on N8x0. I haven't had a chance to test it towards
> 770 initfs.

Will try to at least recompile bootmenu stuff which is currently static for
N8x0 (dropbear).

I no longer have the time I had 1.5 years ago but still this makes some stuff
possible. I am thinking about having e2fsutils, mtd utils, full GNU tar, rsync
, 0xFFFF etc. so one can do full rootfs backup and restore when ssh-ed into
initfs. Even clean rootfs reflash directly from firmware stored on mmc card
would be cool (0xFFFF can unpack it).

770 uses older uclibc, 0.9.27, I have toolchain that seems to work for 770, it
is called "arm-linux-gcc3.4.cs-uclibc0.9.27", not sure if it is still available
on scratchbox site.

Comment 22 Frantisek Dufka (reporter) 2009-11-17 23:41:36 UTC

Created an attachment (id=1594) [details]
libuClibc-0.9.28.so exported symbols difference

OK, first the good news. I have installed this toolchain, compiled dropbear
with it and it apperas to runs just fine inside diablo's initfs (I have ssh-ed
in). Static dropbear currently used in bootmenu has ~200KB, new dynamic one
from same source has ~100KB. That's nice.

Now the bad news. The uclibc is not the same. Scratchbox toolchain binaries in
/lib/ are significantly larger than on device. Even if both are stripped,
libuClibc-0.9.28.so on device has 306168 bytes and scratchbox one has 580724
(i.e almost twice the size). Statically linked dropbear with this uclibc has
~400KB (so it links in most of the uclibc !). Also when listing exported
dynamic symbols (objdump -T libuClibc-0.9.28.so | cut -d ' ' -f 8 | sort) there
are some differences see attachement. Looks like there is extra stuff in
scratchbox one related to iconv and localization and there are also some
symbols missing like xxx_r - i.e. multithreaded variants of some calls.

So it looks like it is quite similar but not the same, most likely it differs
at least in compiler flags (size? maybe the one on device is compiled as
thumb?) and uclibc configuration (locale, iconv,
program_invocation_short_name).

Comment 23 Carsten Munk 2009-11-18 00:09:24 UTC

Right, this seems to be a matter of uclibc configuration, quoting freely from
the generated .config

UCLIBC_HAS_PROGRAM_INVOCATION_NAME=y
UCLIBC_HAS___PROGNAME=y

UCLIBC_HAS_WCHAR=y
UCLIBC_HAS_LOCALE=y
UCLIBC_PREGENERATED_LOCALE_DATA=y
UCLIBC_DOWNLOAD_PREGENERATED_LOCALE_DATA=y
# UCLIBC_HAS_REENTRANT_RPC is not set (explains _r)
UCLIBC_SUSV3_LEGACY=y (explains rindex, bcmp etc)

Could I ask if you used .deb or built your own toolchain? I just noticed a
slight disprenancy in my source set that might have to deal with me juggling
two very identical uclibc versions:

One does: sed -i -e
's,.*UCLIBC_HAS_WCHAR.*,UCLIBC_HAS_WCHAR=y\nUCLIBC_HAS_LOCALE=
n,g' \               .config

and the other one doesn't. The other has UCLIBC_SUSV3_LEGACY=y as well in
debian packaging and the other doesn't.

I'll release a new patch in the morning when I've determined which one is
correct. They are both the same uclibc SVN version as far as I can tell.

Comment 24 Faheem Pervez 2009-11-18 00:17:29 UTC

Carsten, you've been working as the maemo.org Distmaster for, what, a couple of
weeks? And, yet, you've managed to do what Nokia couldn't in a year (although,
that suggests that they were trying).

Your role is a brilliant one, and you're just the right person for it. Thank
you.

Comment 25 Frantisek Dufka (reporter) 2009-11-18 00:30:00 UTC

(In reply to comment #23)
> Could I ask if you used .deb or built your own toolchain? 

.deb, had no time and also no disk space to rebuild it, need to upgrade my
system :-)


> Right, this seems to be a matter of uclibc configuration

Good news :-)

Comment 26 Carsten Munk 2009-11-19 14:26:26 UTC

Right, an update:

http://stskeeps.subnetmask.net/maemo.org/2/scratchbox-toolchain-cs2005q3.2-uclibc-arm-eabi_0.9.8.5-7_i386.deb

http://stskeeps.subnetmask.net/maemo.org/2/uclibc-eabi-2.patch

Changes:
* Fixed the mixup of uclibc sources
* Disable UCLIBC_HAS_LOCALE (brings size to 300kb as it should be)
* Disable UCLIBC_SUSV3_LEGACY (fixes bcmp, index, etc symbols - the use of
defines instead of functions broke stdc++. libstdc++ is not provided on
initfs.)
* Disable UCLIBC_HAS_PROGRAM_INVOCATION_NAME
* Enable UCLIBC_HAS_REENTRANT_RPC

Diff of symbols has only 'sigqueue' missing, which I'm not entirely sure why is
missing in my local copy. Especially since sigqueue came much later, in
revision 15284 (see
http://lists.busybox.net/pipermail/uclibc/2006-June/036419.html). Which is
still between 0.9.28 and 0.9.29.

Comment 27 Carsten Munk 2009-11-19 14:55:46 UTC

For what it's worth, sigqueue isn't in use at the initfs (grep -r'ed /bin,
/sbin, /usr/bin, /usr/sbin, /usr/lib and /lib only had libuClibc).

Let me know how this toolchain is for you. I'm personally satisfied with the
fact I can build pretty precise binaries for initfs right now.

I don't believe there was any modifications done to uclibc beyond configuration
and taking a certain SVN release.

If need be, we can try out the other revision in my previous comment, but I'm
not sure if it's worth the time if the current one generates proper binaries
(and hence satisfying the original bug report).

Comment 28 Carsten Munk 2009-11-23 23:15:38 UTC

So, what's the feeling about this bug? Can I mark it as FIXED or should we
investigate further (as mentioned in former post)?

I plan to follow up on the busybox issue but I can't promise it will happen
this month.

Comment 29 Frantisek Dufka (reporter) 2009-11-24 10:29:34 UTC

(In reply to comment #28)
> So, what's the feeling about this bug?

Warm and fuzzy? It is great. Many thanks Carsten :-)

> Can I mark it as FIXED 

I guess so.

> or should we
> investigate further (as mentioned in former post)?

Well, until we hit some problems I see further investigation as waste of time.
I'm sure there are more important things needing your attention :-)

> 
> I plan to follow up on the busybox issue but I can't promise it will happen
> this month.
> 

Which issue? You mean getting exact busybox sources and configuration used?
That's IMO low priority too.


(In reply to comment #27)
> Let me know how this toolchain is for you. I'm personally satisfied with the
> fact I can build pretty precise binaries for initfs right now.

Even the previous toolchain was satisfactory. So far I had no free evening but
sure I'll try some other stuff. I have only rebuilt dropbear with this new
toolchain and again it works inside initfs and I can log in. 

>  I'm
> not sure if it's worth the time if the current one generates proper binaries
> (and hence satisfying the original bug report).

Agreed.

Comment 30 Carsten Munk 2009-11-24 10:31:18 UTC

Thank you, and let me know if additional things are needed in this regard.

(In reply to comment #24)
> Carsten, you've been working as the maemo.org Distmaster for, what, a couple of
> weeks? And, yet, you've managed to do what Nokia couldn't in a year (although,
> that suggests that they were trying).
> 
> Your role is a brilliant one, and you're just the right person for it. Thank
> you. 
> 

Faheem, do you just read random bug reports? You are Mr. All Things Maemo.