Bug 2536 (int-119011)

Summary: Retrieving a message from IMAP servers removes message flags
Product: [Maemo Official Applications] Email Reporter: Lucas Maneos <maemo>
Component: GeneralAssignee: modest-bugs
Status: RESOLVED WONTFIX QA Contact: modest-bugs
Severity: major    
Priority: Low CC: andre_klapper, arjan, Axel.Thimm, blondius, joseph, list-ener, maemo, pvanhoof, svillar
Version: 5.0/(1.2009.41-10)   
Target Milestone: ---   
Hardware: N810   
OS: Maemo   
Attachments: Perdition log showing modest setting $Label1

Description Lucas Maneos (reporter) 2007-12-13 11:56:13 UTC
STEPS TO REPRODUCE THE PROBLEM:

1. Configure modest to access an IMAP account, connect to the account and
retrieve some messages.

2. Connect to the same account with another IMAP client such as Thunderbird or
Evolution.  The messages retrieved by modest are now marked as important.  I
think this is implemented using IMAP keywords (thunderbird calls them 'tags'
and evolution 'labels'), specifically $Label1.

EXPECTED OUTCOME:

Modest shouldn't set message keywords on retrieving a message, or if it really
has to it should use a keyword that doesn't collide with other MUAs' common
usage.

ACTUAL OUTCOME:

As above.

REPRODUCIBILITY:
(always/sometimes/once)

Always.

EXTRA SOFTWARE INSTALLED:

Nothing really relevant on the device.  The IMAP server is Cyrus imapd 2.3.8.

OTHER COMMENTS:

I haven't tested interactions in the other direction, ie marking messages
'important' in another MUA and seeing how modest handles that.  Not sure what
modest is using this keyword for (haven't looked at the source yet) but there
may be interoperability issues there as well.
Comment 1 Dirk-Jan C. Binnema nokia 2007-12-13 12:13:33 UTC
hmmm... cannot reproduce this (but not using Cyrus here).

do you have a protocol dump / pcap file (ie. something you can
get from the e.g. wireshark program) that shows this
behaviour? thanks.
Comment 2 Lucas Maneos (reporter) 2007-12-13 13:35:34 UTC
The server enforces TLS but I managed to set up privoxy on a box on the local
LAN instead and look at the traffic more closely.  It appears that the flags
are set when modest is closed and not when the messages are read as I reported
initially.  Will attach session log in a minute.
Comment 3 Lucas Maneos (reporter) 2007-12-13 13:43:40 UTC
Created an attachment (id=664) [details]
Perdition log showing modest setting $Label1
Comment 4 Lucas Maneos (reporter) 2007-12-13 13:45:38 UTC
Sorry I got my proxies mixed up.  I meant perdition, not privoxy in the
previous comment.
Comment 5 Philip Van Hoof 2007-12-13 21:53:05 UTC
This should be fixed in Tinymail's trunk. Please retest and close if fixed (I
don't have sufficient rights to mark bugs as fixed on this bugzilla Dirk-Jan.
Can you fix this for me?)
Comment 6 Dirk-Jan C. Binnema nokia 2007-12-13 22:03:29 UTC
should be fixed, please reopen (with a new trace please!) if you can still
reproduce.

coming soon...
Comment 7 Lucas Maneos (reporter) 2007-12-15 19:32:08 UTC
The fix (http://tinymail.org/trac/tinymail/changeset/3116) actually creates the
opposite problem: if any of $Label1 .. $Label5 are set on a message then modest
removes them.  I think it's obvious what's happening, but do let me know if you
want me to produce a trace anyway.

If I understand http://tools.ietf.org/html/rfc2060#section-6.4.6 correctly, it
should be fixable by changing the "UID STORE ... FLAGS.SILENT" command to
+FLAGS.SILENT instead.
Comment 8 Philip Van Hoof 2007-12-17 21:50:17 UTC
Comment #7 makes sense. I'll take a look at this issue.
Comment 9 Philip Van Hoof 2007-12-17 22:16:44 UTC
Now it should be fixed. A bitfield was being abused by recently changed code. I
reverted last change and cleared the bit from being abused (that bit, in the
flags, #18, held the $Label1 setting on IMAP specific providers).

Please mark as closed once verified that trunk (tinymail) indeed fixes this.
Comment 10 Dirk-Jan C. Binnema nokia 2007-12-19 10:17:51 UTC
ok, fixed in svn -- should be in the next modest update as well.
please reopen if you reproduce it after that, thanks.
Comment 11 Lucas Maneos (reporter) 2009-03-14 10:43:43 UTC
Keywords set by other MUAs are again cleared by modest 1.0-2008.43-1:

C: A00010 UID FETCH
44248,44272,44275,44308,44391,44438,44463,44472,44477:44479,44487,44489,44496:44505,44507:44508,44510:44521,44524,44528:44535
(FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (DATE FROM TO CC
SUBJECT MESSAGE-ID X-PRIORITY X-MSMAIL-PRIORITY IMPORTANCE X-MS-HAS-ATTACH
CONTENT-TYPE)])
S: [...]
   31 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44515 [...]
   32 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44516 [...]
   33 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44517 [...]
   [...]
[...]
C: A00112 UID STORE 44515:44528 FLAGS.SILENT (\\Seen)
Comment 12 Andre Klapper maemo.org 2009-05-20 15:50:33 UTC
Anybody brave and willing to test this again with Modest from trunk?
https://git.maemo.org/projects/modest/gitweb?p=modest
Comment 13 Lucas Maneos (reporter) 2009-05-25 17:55:16 UTC
The existing-label-removal bug is still reproducible in modest 3.0.15-rc3 /
tinymail r3911 on Fremantle beta scratchbox.  This is slightly out of date now
(retrieved and built soon after the beta SDK release) - if anything has been
committed specifically to address this problem since let me know and I will try
to build a fresher version.
Comment 14 Lucas Maneos (reporter) 2009-05-25 20:10:51 UTC
Built today's modest/tinymail trunks for testing bug 3369 earlier, no change
regarding this.
Comment 15 Sergio Villar Senin 2009-05-27 10:49:32 UTC
We do this intentionally, tinymail only retrieves the minimum possible data in
order to reduce the amount of disk space for summaries and thus for loading
them into memory.

So this is a wontfix I think
Comment 16 Andre Klapper maemo.org 2009-05-27 11:12:12 UTC
Thanks for explaining.
Comment 17 Lucas Maneos (reporter) 2009-05-27 12:17:20 UTC
(In reply to comment #15)
> We do this intentionally, tinymail only retrieves the minimum possible data in
> order to reduce the amount of disk space for summaries and thus for loading
> them into memory.

I'm not sure I understand you correctly, but $Labeln keywords do seem to be
retrieved (see comment 9 & comment 11).  Also, even if they were not, wouldn't
"relative" STOREs (comment 7) solve the problem?
Comment 18 Andre Klapper maemo.org 2009-06-02 14:56:34 UTC
Sergio, can you respond to Lucas' last comment?
Is using STOREs worth a separate issue?
Comment 19 Sergio Villar Senin 2009-06-16 14:58:22 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > We do this intentionally, tinymail only retrieves the minimum possible data in
> > order to reduce the amount of disk space for summaries and thus for loading
> > them into memory.
> 
> I'm not sure I understand you correctly, but $Labeln keywords do seem to be
> retrieved (see comment 9 & comment 11).  Also, even if they were not, wouldn't
> "relative" STOREs (comment 7) solve the problem?
> 
Ah sorry I haven't understood you at first sight. Regarding using "relative"
stores you might be right. We would take a look at it (or if you have time just
feel free...)
Comment 20 Lucas Maneos (reporter) 2009-06-18 19:32:43 UTC
(In reply to comment #19)
> We would take a look at it (or if you have time just feel free...)

Ok, I did have a look but I'm not at all familiar with the codebase so take the
following with several large pinches of salt:

AFAICT it looks like the flags have already been lost from the local storage
when imap_sync_online() is called.  The change that originally broke this has
been reverted and I'm not sure where else to look, any pointers appreciated.

While testing various things I came across a related bug: all flags also get
reset when marking a message unread (or even leaving an unread message
unopened), due to the old cyrus empty flags workaround.  There is an upstream
bug (and patch) for this at <http://bugzilla.gnome.org/show_bug.cgi?id=530241>,
but obviously it won't do much good until the lost flags issue is fixed.  I
could file a separate bug for this if you prefer.

> Regarding using "relative" stores you might be right.

Maybe not... For some strange reason I had assumed that tinymail/modest keeps
track of flags modified in the current session, but this doesn't seem to be the
case.  So doing differential UID STOREs isn't really possible, unless they are
done in realtime when messages get read/unread/deleted.  This is going into bug
2987 territory, but it might be the best way to fix this for all cases (for
example it seems that imap_sync_online() will happily remove any flags that
imap_parse_flag_list() doesn't know about).
Comment 21 Lucas Maneos (reporter) 2009-06-19 10:39:22 UTC
(In reply to comment #20)
> So doing differential UID STOREs isn't really possible, unless they are
> done in realtime when messages get read/unread/deleted.

Scratch that, there's still the case of modifying messages when offline, then
syncing later.  (Question: how does one go about testing offline mode operation
in scratchbox?)

I can see two possible ways out of this:

A. Do a "UID STORE ... +FLAGS.SILENT" and a "UID STORE ... -FLAGS.SILENT"
instead of a single unqualified one, or

B. FETCH message flags from the server and merge them with the locally stored
ones before STORE

but both would require extra round trips.  In the case of A the worst case is
two STOREs per set of messages with identical modest-side flags, while B would
need one extra FETCH to retrieve all message flags and depending on the
response up to one STORE per message.

Any other ideas?
Comment 22 Lucas Maneos (reporter) 2009-10-22 05:52:12 UTC
Reproducible also in 1.2009.41-10.  Bumping severity a notch - I probably
wouldn't go as far as "critical", but corrupting users' message flags does
count as data loss in my book.
Comment 23 Andre Klapper maemo.org 2009-10-22 21:45:42 UTC
Does this change comment 15 here?
Comment 24 Andre Klapper maemo.org 2010-01-07 15:40:18 UTC
This is currently a WONTFIX:
We do this intentionally, tinymail only retrieves the minimum possible data in
order to reduce the amount of disk space for summaries and thus for loading
them into memory.
Comment 25 Lucas Maneos (reporter) 2010-01-07 18:06:04 UTC
Right, so it's OK to eat the users' message flags as long as it does it
efficiently :-b
Comment 26 Joseph Heenan 2010-02-17 11:16:11 UTC
Is there any known workaround for this? 

(I'm curious as to why it seems to not be affecting a lot of people, but seems
like a very visible/annoying issue.)
Comment 27 Joseph Heenan 2010-03-28 05:45:09 UTC
Further to my last comment, it seems using nokia messaging instead is a
"workaround" as the then the flags don't get cleared...
Comment 28 Lucas Maneos (reporter) 2010-03-28 13:32:47 UTC
(In reply to comment #27)
> Further to my last comment, it seems using nokia messaging instead is a
> "workaround" as the then the flags don't get cleared...

For small values of "workaround", since Nokia Messaging is only an option for a
small subset of email accounts and even for those some people are not
comfortable handing their email contents and account passwords over to Nokia.
Comment 29 strank 2010-04-26 13:26:14 UTC
I realise that this is marked as WONTFIX, but it is severely hindering me in
using Modest: I shy away from it since I know that all my Thunderbird tags will
be lost... and there is no (practical) workaround or even a comparable
alternative client that does not destroy user data, right?

What about an option that would enable the extra network roundtrips necessary?
This would give people that value data integrity more than limiting network
usage a choice?

According to comment #21 method A, there could be a solution that only requires
extra network usage and no increase in memory usage, right?
I would be grateful if someone could give me a hint on where the relevant
sections in the code are.

cheers
Comment 30 Joseph Heenan 2010-04-26 13:37:23 UTC
I believe one simple fix would be just to record the thunderbird flags (I think
they're called $Label<1-5> or something like that) in the same way other flags
are recorded, and then pass them back when the server sets flags. Other flags
are handled like this, and there's space in the bitfield flags are stored in to
do this without requiring any extra memory from what I remember. (I've
forgotten where the code is, sorry)

The patch here possibly points you in roughly the right area of code:
http://bugzilla-attachments.gnome.org/attachment.cgi?id=110100
Comment 31 Arjan Schrijver 2010-05-07 15:21:52 UTC
Hi, I just want to say 'me too'.
I'm using Thunderbird message flags/tags quite frequently, and my new n900 just
removed all tags from my messages.
Please fix this, as it really is a very annoying bug.
Comment 32 Lucas Maneos (reporter) 2010-07-24 13:53:07 UTC
*** Bug 10995 has been marked as a duplicate of this bug. ***
Comment 33 Elisa 2010-07-25 14:47:18 UTC
I agree that this is an extremely annoying issue and as I rely on setting these
message flags in how I go about using my e-mail client on a regular basis I see
this as also as a form of data corruption and disappointed that this is an
issue that won't/cannot be resolved.