Comment 1 Andre Klapper 2008-10-02 13:27:54 UTC

Probably no easy workaround except to set a retrieval limit (which didn't work
for you).
Do you download the entire messages or only the headers?

I very much assume that Modest is using CPU?
What is the output if you strace it?
    strace -p $(pidof modest|cut -d' ' -f1)
See http://maemo.org/development/tools/ and
http://maemo.org/development/tools/doc/diablo/strace/ for more information.

I set Modest to download only headers. Modest was in fact using CPU during this
process. The easy workaround is to use Claws-mail :-) But Claws' user interface
is a desktop application squished onto a handheld, and I'd prefer something
that was better designed. I have even given half a thought to porting Balsa, as
it loads my inbox nearly instantly on any platform I've tried.

I'll install strace and capture a log to my SD card. It will probably take me
until tomorrow to do this.

This is a known issue, and at least two reasons were discussed in
maemo-developers a while ago:

http://lists.maemo.org/pipermail/maemo-developers/2008-June/033963.html
http://lists.maemo.org/pipermail/maemo-developers/2008-June/034059.html

Comment 4 Andre Klapper 2008-10-07 15:58:58 UTC

As the links show, this has been a low priority - might be closed as WONTFIX.

A couple of additional notes:

- This bug manifests if any large folder exists in the IMAP account's personal
namespace.  There is no workaround as modest doesn't obey subscriptions and has
no other means of excluding folders from indexing (bug 3484).

- This patch:
<http://mail.gnome.org/archives/tinymail-devel-list/2008-October/msg00016.html>
looks like it may alleviate the I/O side of the performance issue.

Still an issue in Maemo 4.1.3 (5.2008.43-7)

Tested in my gmail mail account.

Comment 7 Andre Klapper 2009-05-26 14:08:57 UTC

Anybody tested the performance again with Modest from trunk?
https://git.maemo.org/projects/modest/gitweb?p=modest

It's a bit hard to gauge performance in scratchbox, but it doesn't look very
promising.  A "quick" test of starting modest, opening a folder with ~60K
messages, waiting for it to settle and quiting took

real    5m13.058s
user    4m8.544s
sys    0m7.852s

the first time, and that was running @2.4GHz with plenty of RAM and folder
summary stored on ext3 on disk.  Most of the CPU cycles were spent sorting the
folder list and updating the display.  Subsequent runs (after modest has cached
the folder summary) take about a minute less.

For comparison, mutt on the same box does the same in:

real    0m14.190s
user    0m7.552s
sys    0m1.404s

(the idle time was just typing the IMAP password).

Comment 9 Andre Klapper 2009-06-05 14:49:14 UTC

I had the chance to test this in Fremantle on a test device and it worked way
better than compared with Diablo on an N810. Basically no lag at all, and my
IMAP test account has more than 10000 messages in its Inbox.
Hence I consider this issue to be FIXED in the Fremantle.

Unfortunately this is a WONTFIX for Diablo as Diablo is in maintenance mode and
Nokia will only provide bugfixes for critical issues if at all.
For your interest the Mer project aims to provide a community backport of
Fremantle for N8x0 devices. See http://wiki.maemo.org/Mer for more information.

Comment 10 Andre Klapper 2009-07-14 13:38:22 UTC

Fix should be included in Fremantle SDK beta 2 hence updating Target Milestone.
If you are the reporter of this bug: Feel free to verify the fix if possible.

I would like to report that using 1.2009.41-10 and having just below 10000
messages using IMAP with Gmail, the view of the Inbox is absolutely unusable
for me. The messeges load, but the view does not react to my trying to scroll,
at all. I eventually get a "program unresponsive, would you like to end it"
messege. I have to either tap the back arrow to unfreeze the program (go back
in the UI) or just choose Yes to kill the e-mail application.

Seconded.  Testing on 1.2009.41-10 with a folder containing 7.5K messages,
hosted on dovecot 1.2 on the local LAN, folder previously opened with another
client so it's cached and the index is up to date on the server side:

- The first modest saw the folder it took 65 seconds to load with low to
(occasionally) medium CPU usage while displaying a black message list.

- Subsequent visits to the folder take about 40-45 seconds with 100% CPU usage,
while displaying a message list but completely unresponsive to input.  The
culprit seems to be sorting the index: changing the sort order takes roughly
the same amount of time & cpu cycles.

The second case could clearly use some love, reopening.

I also have this exact same problem on the Nokia N900 using Gmail. My
workaround was to use Nokia Messaging instead. I think it only grabs the last 3
days of emails instead of all the 10,000

*** Bug 5354 has been marked as a duplicate of this bug. ***

I'll try to clarify things a little bit. Basically Modest becomes very slow and
it even seems to hang because the code that performs the sorting is the same
code that shows the contents in the screen. As these two things cannot be done
at the same time, if sorting takes too much time, then showing the application
or attending user requests will be delayed.

The technical explanation is that the list of headers is like this:
(GtkTreeSortable (GtkTreeModelFilter (TnyHeaderListModel)))

We need filtering to remove deleted but not expunged headers among some other
reasons, and sorting, well it's obvious. Thing is that the sorting and
filtering happen inside gtk+. Being single-threaded code, gtk+ either sorts or
attend events or draw widgets, but it cannot do everything at the same time.

Now the good news :). We're working in an optimization for the
TnyHeaderListModel that will expose to the upper levels only a small subset of
messages to drastically reduce sorting and filtering times. Stay tuned.

I use Gmail. It was suggested to me to archive my emails. Never used it before,
but it looks like a good workaround for this problem. Just tried it and it cut
down my inbox of 24,000+ emails to now 50.

IMAP on the Nokia N900 is now snappy.

Confirmed for 42-11 as well.

I have a small INBOX, but I've got plenty of fairly large IMAP folders.
Unfortunately, this means that any access to my IMAP account takes a long time,
and worst of all, *even if I close the UI*, modest stays in the background,
drives load up to four, eats all my battery, and makes the system overall quite
unresponsive (e.g. video becomes really choppy). The main CPU consumption goes
to IO (top claims 80%).  Took me a while to figure out the culprit.

The workaround is to "killall modest", but that only postpones the inevitable -
the next time email is started the whole shebang starts again.

Comment 18 Andre Klapper 2009-11-17 15:16:51 UTC

This has been fixed in package
modest 3.1.12+0m5
which is part of the internal build version
2.2009.46-16
(Note that 2009 is the year and the number after is the week.)

Any public update released with or after this build version will include the
fix.
Please verify that the new version fixes the bug by marking this bug report as
VERIFIED after the public update has been released and if you have some time.

(In reply to comment #17)
> Confirmed for 42-11 as well.
> 
> I have a small INBOX, but I've got plenty of fairly large IMAP folders.
> Unfortunately, this means that any access to my IMAP account takes a long time,
> and worst of all, *even if I close the UI*, modest stays in the background,
> drives load up to four, eats all my battery, and makes the system overall quite
> unresponsive (e.g. video becomes really choppy). The main CPU consumption goes
> to IO (top claims 80%).  Took me a while to figure out the culprit.
> 
> The workaround is to "killall modest", but that only postpones the inevitable -
> the next time email is started the whole shebang starts again.
> 

I already explained the reasons of that CPU consumption. In the case high IO
consumption it's because saving a summary file (which could be very big with
huge inboxes) is a terribly slow operation in a flash storage. That's why we
also reduced the amounts of writes.

(In reply to comment #17)
> I have a small INBOX, but I've got plenty of fairly large IMAP folders.

FYI, as a workaround you can unsubscribe those to stop modest from looking at
them (unless your account is on a cyrus server in which case it will go into
everything anyway).

(In reply to comment #20)
> (In reply to comment #17)
> > I have a small INBOX, but I've got plenty of fairly large IMAP folders.
> 
> FYI, as a workaround you can unsubscribe those to stop modest from looking at
> them (unless your account is on a cyrus server in which case it will go into
> everything anyway).

HMhm, the thing is, subscription is a global setting, which will then apply to
all IMAP clients on that account, so it might not be the desired behavior.
--
Yves-Alexis

*** Bug 6251 has been marked as a duplicate of this bug. ***

This bug seems to be a good example for what happens if we mix up bugs for
maemo 4 and 5.

I also can't use my hi-tech internet tablet N900 to read emails! And I am not
the only one:
https://bugs.maemo.org/show_bug.cgi?id=6251
I only have 500 mail in the inbox but much larger number in other folders.

So why on earth a bug is closed as FIXED without telling us when and if it will
be fixed some day?

An internet tablet where I can't use one of the major applications is a shame.

Is somebody working on that issue or will this be a "maybe fixed in maemo6"?

Please take a look at https://bugs.maemo.org/show_bug.cgi?id=3762#c18

Comment 25 Andre Klapper 2009-11-19 13:18:54 UTC

(In reply to comment #23)
> So why on earth a bug is closed as FIXED without telling us when and if it will
> be fixed some day?

See comment 18 why it is FIXED for Maemo5 - will be in the next update.
For Diablo/Maemo4 this is WONTFIX.
And bug 6251 is a dup of this, yeah. :-)

Sorry, I overlooked this comment ;-) 
Well to know that everything will be okay.

Maybe it'a good idea to have an field to show when bug will be fixed. Target
Milestone seems to be somewhat unspecific.

Again: Sorry for the noise,
ciao Uwe

Comment 27 Andre Klapper 2009-11-19 13:33:32 UTC

(In reply to comment #26)
> Sorry, I overlooked this comment ;-) 
> Well to know that everything will be okay.
> 
> Maybe it'a good idea to have an field to show when bug will be fixed. Target
> Milestone seems to be somewhat unspecific.

But for that, Nokia needs to know themselves which internal release the next
public update will be based on, and that can change quickly. :-P
I try to work around this by publishing the week number that includes the fix
(see comment 18) because otherwise I'd also run into problems not knowing if a
certain fix is already part of a public update or not. :-/
Yeah, it's complicated. :-D

VERIFIED FIXED on 46-16 (yeah, I work for the company, it's just that I
registered with my private email address to bugs.maemo.org). 

Email is now quite usable even when I've got over 2000 unread emails in
multiple folders. Thank you :-)

Comment 29 Andre Klapper 2009-11-19 14:14:06 UTC

Oh cool. Thanks. :-)

(In reply to comment #27)
> But for that, Nokia needs to know themselves which internal release the next
> public update will be based on, and that can change quickly. :-P
> I try to work around this by publishing the week number that includes the fix

For openly developed projects such as this, a reference to the relevant SCM
commit(s) could be useful (don't worry, no one is expecting you to dig through
commit logs to provide those ;-) ).

In this case <http://svn.tinymail.org/trac/tinymail/changeset/4046> looks like
the tinymail part, but I don't see any references to
tny_gtk_header_list_model_[gs]et_show_latest in the modest tree yet.

Comment 31 Eero Tamminen 2009-11-20 11:45:01 UTC

(In reply to comment #30)
> For openly developed projects such as this, a reference to the relevant SCM
> commit(s) could be useful

Unless you're interested in building the software yourself, I doubt that.

You would need to know whether corresponding branch is merged to master, which
version of master is in which version of the source package, that the fix
doesn't get reverted or otherwise need improvements, which version of the
resulting binary package is in the release, and which release will be the
public one.

And the last one is decided only after it passes multiple layers of testing,
it's not decided beforehand but after verification.

(In reply to comment #31)
> Unless you're interested in building the software yourself, I doubt that.

Well, sometimes just looking at the patch helps to understand the fix.  Also 
some fixes could be backported to things like the Diablo community updates
project if we ever manage to get it off the ground (though some fixes are less
trivial than others and some code bases move faster than others so perhaps this
one isn't a good example) or Mer (again, not a good example as it doesn't ship
modest at the moment).

Of course I'm not suggesting that developers or bugmasters should feel obliged
to do it, and the relevant commits can always be dug up by anyone interested
with a little bit of effort.

(In reply to comment #15)
> I'll try to clarify things a little bit. Basically Modest becomes very slow and
> it even seems to hang because the code that performs the sorting is the same
> code that shows the contents in the screen. As these two things cannot be done
> at the same time, if sorting takes too much time, then showing the application
> or attending user requests will be delayed.
> 
> The technical explanation is that the list of headers is like this:
> (GtkTreeSortable (GtkTreeModelFilter (TnyHeaderListModel)))
> 
> We need filtering to remove deleted but not expunged headers among some other
> reasons, and sorting, well it's obvious. Thing is that the sorting and
> filtering happen inside gtk+. Being single-threaded code, gtk+ either sorts or
> attend events or draw widgets, but it cannot do everything at the same time.
> 
> Now the good news :). We're working in an optimization for the
> TnyHeaderListModel that will expose to the upper levels only a small subset of
> messages to drastically reduce sorting and filtering times. Stay tuned.
> 

Hi sergio, but in this moment, what is the solution??

I can not use gmail!

Any suggestions?

(In reply to comment #33)

> Hi sergio, but in this moment, what is the solution??
> 
> I can not use gmail!
> 
> Any suggestions?

Already applied the patch to tinymail and Modest. You will eventually get an
update from Nokia with that performance improvement. Thing is that the first
retrieval will take some time as now but once that retrieval is made you'll
only see the most recent 250 messages. If you want to see more you'll have a
"Show more" button.

This patch can't come soon enough! Mail is unusable without it

Comment 36 Andre Klapper 2009-11-27 18:13:13 UTC

(In reply to comment #35)
> This patch can't come soon enough! Mail is unusable without it

Please avoid such unneeded comments as they trigger bugmail.
This is not a forum. Thanks.

Can (In reply to comment #34)
> Already applied the patch to tinymail and Modest. You will eventually get an
> update from Nokia with that performance improvement. Thing is that the first
> retrieval will take some time as now but once that retrieval is made you'll
> only see the most recent 250 messages. If you want to see more you'll have a
> "Show more" button.  

Can confirm this behaviour for 1.2009.51-1

Comment 38 Andre Klapper 2009-12-28 14:07:25 UTC

(In reply to comment #37)
> Can confirm this behaviour for 1.2009.51-1 

That's the "fix" - hence the original bug does not occur in that version.
Resetting version. ;-)

Comment 39 Andre Klapper 2010-01-14 12:27:51 UTC

The problem reported here should be fixed in the update released today for
public: The Maemo5 update version 2.2009.51-1 (also called "PR1.1" sometimes).
Please leave a comment if the problem is not fixed for you in this update
version.

It is improved thats true,
Weird thing is that sometimes my gmail inbox (imap) is fast it is there almost
instant but other times it is 3-4 seconds.  Can i configure the "250 More" so
that it it only 100? Does modest only show the last 250 of all mail or does it
not download more then 250?

Weird thing is that with the gmail through imap i cant seem to show any other
map then inbox, its spinning forever, showing nothing.

I wouldn't consider it fully fixed. It took almost 5 minutes to open my GMail
inbox on fast WiFi after it already took 2 or 3 minutes for Modest to list all
IMAP folders. I suppose accessing it on EDGE would be almost impossible.
But at least the UI doesn't lock up anymore.

I can confirm the previous comment :-(

I subscribed a folder containing 17.5K messages and tried to access it with
modest.  The first time took ~45" with pretty much full CPU utilisation @600MHz
(top showed ~75% user / 20% system) at the end of which it had retrieved all
messages' headers and displayed.

It then took just over another 10 minutes at >95% user CPU utilisation (most
likely sorting the headers, more below) until it settled and the device became
responsive again. At this point tapping the menu bar shows that 250 messages
are displayed.

Subsequent accesses or changing the sort order take between 5-8' (variable,
tested 3 times) with similar resource usage.

Note that this is all CPU-bound, ie it's not writing the summary that's slowing
things down.

Running it under oprofile for a minute or so shows most of the CPU cycles going
to:

samples  %        image name               symbol name
23513    28.2157  libgtk-x11-2.0.so.0.1400.7
gtk_tree_model_filter_get_nth_visible
10776    12.9313  libgtk-x11-2.0.so.0.1400.7 gtk_tree_model_filter_row_inserted
9355     11.2260  libgtk-x11-2.0.so.0.1400.7 gtk_tree_model_sort_row_inserted
7085      8.5020  libglib-2.0.so.0.2000.3  /usr/lib/libglib-2.0.so.0.2000.3
5971      7.1652  libgobject-2.0.so.0.2000.3
/usr/lib/libgobject-2.0.so.0.2000.3
3462      4.1544  no-vmlinux               /no-vmlinux

So my guess is that the full header list is still reaching the gtk+ layer.

(In reply to comment #42)
> I can confirm the previous comment :-(
> 
> I subscribed a folder containing 17.5K messages and tried to access it with
> modest.  The first time took ~45" with pretty much full CPU utilisation @600MHz
> (top showed ~75% user / 20% system) at the end of which it had retrieved all
> messages' headers and displayed.
> 
> It then took just over another 10 minutes at >95% user CPU utilisation (most
> likely sorting the headers, more below)

If you have access to the mail server config, you might want to activate
server-side sorting.

(In reply to comment #43)
> If you have access to the mail server config, you might want to activate
> server-side sorting.

AFAICT it's on:

CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT
SEARCHRES WITHIN CONTEXT=SEARCH

so the biggest problem is guess still that it downloads everything?
It shouldnt do that that new window of 250 should i guess also be what it keeps
on the N900..

because as far as i know imap4 also support searching on the server
so when we also have global search then for imap that means just search on the
server then modest is nothing more then just playing the last results the
server gives us (and can cache X number of messages)

(In reply to comment #45)
> so the biggest problem is guess still that it downloads everything?
> It shouldnt do that that new window of 250 should i guess also be what it keeps
> on the N900..

Yes, it still download all the headers. Tinymail still needs more changes

*** Bug 7855 has been marked as a duplicate of this bug. ***

(In reply to comment #46)
> Yes, it still download all the headers. Tinymail still needs more changes

I think that's fine.  It's reasonable to expect the initial download (but only
that) to take some time and all headers are needed locally if you want
meaningful results with any sort order apart from "Date (most recent first)".

I can't help thinking that this bug is really on the GTK+ side of things. 10
minutes to sort 17.5K records seems quite pathological.  The CPU is certainly
capable of doing a lot better than that:

Nokia-N900-51-1:~# od -x < /dev/urandom | head -17500 | time sort > /dev/null
real    0m 2.32s
user    0m 0.52s
sys    0m 0.02s

(and yes, I realise this is hardly measuring the same thing, but a 3 order of
magnitude difference seems a bit excessive).

Just guessing wildly: is it possible the headers are added to the data
structure one (or just a few) at a time and the whole list is re-sorted
continuously?

(In reply to comment #48)
> (In reply to comment #46)
> > Yes, it still download all the headers. Tinymail still needs more changes
> 
> I think that's fine.  It's reasonable to expect the initial download (but only
> that) to take some time and all headers are needed locally if you want
> meaningful results with any sort order apart from "Date (most recent first)".
> 
> I can't help thinking that this bug is really on the GTK+ side of things. 10
> minutes to sort 17.5K records seems quite pathological.  The CPU is certainly
> capable of doing a lot better than that:

Well Lucas maybe you don't believe it but we have verified it :). I don't know
if you have the tools, but we run several tests using oprofile and top 10
functions during that time were gtk functions related to treeview filtering and
sorting. As you might now gtk is single threaded, so whether it draws on the
screen or it sorts.

> Just guessing wildly: is it possible the headers are added to the data
> structure one (or just a few) at a time and the whole list is re-sorted
> continuously?

That's true. Headers are loaded progressively so it's very likely that extra
sorting happens.

*** Bug 8213 has been marked as a duplicate of this bug. ***

(In reply to comment #49)
> I don't know
> if you have the tools, but we run several tests using oprofile and top 10
> functions during that time were gtk functions related to treeview filtering and
> sorting.

Sounds like we had the same hunch, see comment 42 :-)

> That's true. Headers are loaded progressively so it's very likely that extra
> sorting happens.

Is the size of the header batches loaded tunable?  It would seem a big win to
adjust according to folder size so it never sort more than n times.  Or maybe
load just a screenfull or two and then the rest in one go (it's not like the
user can scroll while the sorting takes place anyway).

(In reply to comment #51)
> (In reply to comment #49)
> > I don't know
> > if you have the tools, but we run several tests using oprofile and top 10
> > functions during that time were gtk functions related to treeview filtering and
> > sorting.
> 
> Sounds like we had the same hunch, see comment 42 :-)
> 
> > That's true. Headers are loaded progressively so it's very likely that extra
> > sorting happens.
> 
> Is the size of the header batches loaded tunable?  It would seem a big win to
> adjust according to folder size so it never sort more than n times.  Or maybe
> load just a screenfull or two and then the rest in one go (it's not like the
> user can scroll while the sorting takes place anyway).

That's what we do right now :) since PR1.1. We created a special tree model
that only exposes X items. By default X is 250 so we're now only sorting and
filtering 250 headers.

Created an attachment (id=2175) [details]
tinymail debug output

I compiled the current modest-3-2 and was able to reproduce my imap performance
problems.
it looks like the headers are redownloaded all the time due to the problem you
can see in the log.
after moving a lot of mails from the inbox into a seperate directory the
problem disappeared.

Comment 54 Andre Klapper 2010-02-02 16:52:24 UTC

Philipp, can you please start an X Terminal window and enter
   export "CAMEL_DEBUG"="all"
to enable the debug mode of the mail library, and after that
   /usr/bin/maemo-summoner /usr/bin/modest.launch -s >
/home/user/MyDocs/.documents/log 2>&1

to start the email application. Then try to connect to the mailserver. Attach
the logfile named "log" here which will be in your Home directory (please check
for confidential data before attaching).

(In reply to comment #54)
I'll try to reproduce it again. As I said, I already moved the mails out of the
INBOX and the problem disappered.

Another interesting thing I found is that the mail library always downloads all
headers in a folder even if the mail client only shows the latest mails. This
takes a lot of time but also uses more bandwidth than necessary.

Comment 56 Andre Klapper 2010-08-26 16:14:55 UTC

Closing this bug report as the issue could not be reproduced. Please feel free
to reopen this report if you can provide better steps to reproduce this and if
this is still a problem in 10.2010.19-1.

Using GMail IMAP inbox I still very often have to wait > 20 seconds to see new
email headers.  While this is much much better than it used to be its still
fairly unacceptable compared to any mail client worth its salt.

Reproduction steps:
Sign up for gmail
Send yourself a thousand or two mails
Load the inbox on modest (just to build some initial cache)
reboot your phone
Load modest

Expected Results:
Can browse all gmail imap folders quickly and have headers appear within 1-2
seconds

Actual Results:
IMAP folders often take > 20-30 seconds to show headers.

Details:
Running latest firmware.  One thing I noticed is that perhaps it is
re-downloading the headers or checking for updates?  Modest should at least
display the mail it has previously downloaded so I can browse them while it
fetches new mail.

Zach: did you try the patch for modest
http://talk.maemo.org/showthread.php?t=56634&highlight=modest+patch

that one does speed it up.

I think the problem is that modest download constantly way way to much. I
noticed this when i was not in my home country, within no time i got the
warning 10MB are already roamed.. continue? Thats just crazy.

Comment 59 Neil MacLeod 2010-08-26 21:20:47 UTC

Also see bug 11139, probably related.

Comment 60 Andre Klapper 2010-10-25 23:33:21 UTC

This problem still happens in 20.2010.36-2, I assume?

Unfortunately I'm not seeing much performance improvement in PR1.3.  There may
be a slight speedup, but that could be just the UI in general.

hi dear i have some problems with email application modest....i have many email
on the server but modest import only the first 500 (about) email ....i don't
understand why...any issue???

and if you say through the menu, show more?

Comment 64 Andre Klapper 2010-11-20 12:15:22 UTC

(In reply to comment #62)
> hi dear i have some problems with email application modest....

Please ask in http://talk.maemo.org as bugs.maemo.org is NOT a support forum.