Bug 978 - metalayer-crawler high CPU consuming
: metalayer-crawler high CPU consuming
Status: RESOLVED FIXED
Product: Data
metalayer-crawler
: unspecified
: N800 Maemo
: Medium major (vote)
: ---
Assigned To: Pekka Marjola
: metatracker-bugs
:
:
:
:
  Show dependency tree
 
Reported: 2007-01-27 15:31 UTC by William Maddler
Modified: 2008-12-06 18:23 UTC (History)
3 users (show)

See Also:


Attachments
daemon start/stop script with added CRAWLPATH variable (1.78 KB, text/plain)
2007-01-27 15:34 UTC, William Maddler
Details


Note

You need to log in before you can comment on or make changes to this bug.


Description William Maddler (reporter) 2007-01-27 15:31:51 UTC
metalayer-crawler daemon is consuming not less that 50% CPU (stats have been
collected toward a 12hrs time frame). The problem could be caused by the crawler
trying to explore the whole file system (no change by removing the external 1GB
MMC card anyway).

Restricting the search only to "media" folders solved the poroblem:

/home/user/MyDocs/.videos
/home/user/MyDocs/.sounds
/home/user/MyDocs/.images

adding to /etc/init.d/metalayer-crawler0 a new varialbe:

CRAWLPATH="/home/user/MyDocs/.video,/home/user/MyDocs/.sounds,/home/user/MyDocs/.images"

and modifying adding "-c CRAWLPATH" to the start/restart cases fixed the issue.
Comment 1 William Maddler (reporter) 2007-01-27 15:34:22 UTC
Created an attachment (id=172) [details]
daemon start/stop script with addned CRAWLPATH variable
Comment 2 Laurent MARTIN 2007-02-04 12:32:12 UTC
I've also encountered the same bug and my system becomed to be unusable due to
metalayer-crawler taking more than 80% of the CPU.
I've applied the "patch" (thank you!) which has worked like a charm but I think
that Nokia should resolve this: what about N800 owner's who are not confident
enough with the system to edit and change the proper file in /etc/init.d?
Comment 3 Pekka Marjola nokia 2007-02-05 10:01:02 UTC
Corrupted filesystem on memory card often leads to this problem. Note that
removing MMC while it is being scanned does not kill crawler, and it still takes
CPU. This is being worked on. 

For now, you should re-format your MMC and re-copy the contents back there. In
addition, you may lower the priority of metalayer-crawler:

/etc/init.d/metalayer-crawler0:
    dsmetool -f "$DAEMON -F" --nice=19 -U $USER
Comment 4 Bob Fillmore 2007-03-04 17:32:27 UTC
(In reply to comment #0)
> metalayer-crawler daemon is consuming not less that 50% CPU (stats have been
> collected toward a 12hrs time frame). The problem could be caused by the crawler
> trying to explore the whole file system (no change by removing the external 1GB
> MMC card anyway).
> 
> Restricting the search only to "media" folders solved the poroblem:
> 
> /home/user/MyDocs/.videos
> /home/user/MyDocs/.sounds
> /home/user/MyDocs/.images
> 
> adding to /etc/init.d/metalayer-crawler0 a new varialbe:
> 
>
CRAWLPATH="/home/user/MyDocs/.video,/home/user/MyDocs/.sounds,/home/user/MyDocs/.images"
> 
> and modifying adding "-c CRAWLPATH" to the start/restart cases fixed the issue.

(In reply to comment #3)
> Corrupted filesystem on memory card often leads to this problem. Note that
> removing MMC while it is being scanned does not kill crawler, and it still takes
> CPU. This is being worked on. 
> 
> For now, you should re-format your MMC and re-copy the contents back there. In
> addition, you may lower the priority of metalayer-crawler:
> 
> /etc/init.d/metalayer-crawler0:
>     dsmetool -f "$DAEMON -F" --nice=19 -U $USER
> 
> 

I think I found another situation that can cause this.  I wanted to be able to
browse the root filesystem using file manager, so I created a symlink with:
  ln -s / /home/user/MyDocs/rootfs
Not long after I noticed that metalayer-crawler hit high CPU.  I was curious
about this because I only have a few hundred media files on the memory card.  I
then remembered the symlink... experience tells me that many programs that walk
the filesystem tree don't check for symlink loops.  Maybe metalayer-crawler
should not follow symlinks?
Comment 5 coba 2007-03-18 20:28:33 UTC
using patch of adding CRAWLPATH parameter also have problem:

after system boot up, it also have a high load period of appr. 3-5 minuts on my
n800.

but if i mask the metalayer-crawler0 by adding a "exit 0" to directly escape
this script, system will not have such a long high load period.
Comment 6 Benjamí Villoslada 2007-04-03 18:17:14 UTC
In my N800 metalayer-crawl process consumption is high too.

After boot, metalayer-crawl process:

- More than 90% CPU consumption (45% and 85% in some moments).
- 12 min scanning,
- when crawl stops:

~ $ uptime
 17:01:56 up 12 min, load average: 1.27, 1.21, 0.76

- metalayer-crawl retains 11.7% memory.  First 5 in 'top' command with M option
(memory sort)

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
 1120 user     S N      14M   319  0.0 11.7 metalayer-crawl
 1127 user     S N      14M  1120  0.0 11.7 metalayer-crawl
 1007 user     S <      11M   931  0.0  9.4 maemo-launcher
 1018 user     S <     6332   931  0.0  4.9 maemo-launcher
 1014 user     S <     6084   931  0.0  4.7 maemo-launcher

I've two 4 GB SD cards:

$ df -h
[...]
/dev/mmcblk1p1            3.7G      2.8G    994.4M  74% /media/mmc1
/dev/mmcblk0p1            3.7G      1.7G      2.0G  47% /media/mmc2

With this file extensions:

mmc1:

      1 c
      1 css
      2 xbm
      3 txt
      4 xml
      9 jpg
     20 ogg
     29 pdf
    154 gif
    242 html
    393 mp3
   2012 no-extension-in-file
   3497 htm

mmc2:

      1 31_armel
      1 metadata
      1 swap
      5 avi
      5 no-extension-in-file
      7 zip
     10 deb
 178623 jpg (775.8 Google Maps jpg files)

OS version:

~ $ cat /etc/osso_software_version
RX-34_2007SE_3.2007.10-7_PR_MR0

Thanks!

Regards
Comment 7 Eero Tamminen nokia 2007-08-02 12:21:22 UTC
A serious bug in the (open source) libid3 library was fixed for the latest
N800 RX-34_2007SE_4.2007.26-8_PR_MR0 release.  If mp3 file was missing
metainformation, libid3 scanned through the whole file, this is now fixed
(handling single file could take nearly 1/2 min, so it really "crawled").

There are also several improvements to the metalayer crawler present
in the original N800 release:
- it reacts much faster/better to MMC eject events
  (no more "card is in use" messages because of this)
- it's fixed to work robustly when a corrupted MMC with
  recursive directory entries is inserted (no more out of memory
  issues and resulting neverending CPU usage from corrupt MMCs contents)
- symlinks and hidden directories & files (starting with ".") are ignored
- it's niced so that it doesn't take CPU from the other processes,
  it uses only slice of CPU that is unused


Although the crawler can still take time when there's a huge number of
media files available (and they are re-scanned when device is plugged
into USB cable and off because the MMC card gets unmounted&mounted),
I consider this bug to be fixed.  Pekka?

If there are still issues with crawler, please create a separate bug
with information about what file types cause the problem and whether
fsck reported any errors for the memory card.
Comment 8 Pekka Marjola nokia 2007-08-02 12:31:06 UTC
Yes, I agree. I'm resolving this bug, if there are specific other problems,
please submit new bugs.
Comment 9 Benoît HERVIER 2007-12-12 17:33:00 UTC
Seems to be still here on OS2008 Beta on n800.
Comment 10 Eduard Urbach 2007-12-23 19:32:04 UTC
> I think I found another situation that can cause this.  I wanted to be able to
> browse the root filesystem using file manager, so I created a symlink with:
>   ln -s / /home/user/MyDocs/rootfs
> Not long after I noticed that metalayer-crawler hit high CPU.  I was curious
> about this because I only have a few hundred media files on the memory card.  I
> then remembered the symlink... experience tells me that many programs that walk
> the filesystem tree don't check for symlink loops.  Maybe metalayer-crawler
> should not follow symlinks?
> 
I have the same problem with the latest OS 2008 release.
Version: RX-34 2.2007.50-2
Comment 11 Eero Tamminen nokia 2007-12-27 14:01:14 UTC
(In reply to comment #10)
> >   ln -s / /home/user/MyDocs/rootfs
...
> I have the same problem with the latest OS 2008 release.

The bug for the symlink thing is bug 1760.