Comment 1 Dirk-Jan C. Binnema 2007-12-13 12:13:33 UTC

hmmm... cannot reproduce this (but not using Cyrus here).

do you have a protocol dump / pcap file (ie. something you can
get from the e.g. wireshark program) that shows this
behaviour? thanks.

The server enforces TLS but I managed to set up privoxy on a box on the local
LAN instead and look at the traffic more closely.  It appears that the flags
are set when modest is closed and not when the messages are read as I reported
initially.  Will attach session log in a minute.

Created an attachment (id=664) [details]
Perdition log showing modest setting $Label1

Sorry I got my proxies mixed up.  I meant perdition, not privoxy in the
previous comment.

This should be fixed in Tinymail's trunk. Please retest and close if fixed (I
don't have sufficient rights to mark bugs as fixed on this bugzilla Dirk-Jan.
Can you fix this for me?)

Comment 6 Dirk-Jan C. Binnema 2007-12-13 22:03:29 UTC

should be fixed, please reopen (with a new trace please!) if you can still
reproduce.

coming soon...

The fix (http://tinymail.org/trac/tinymail/changeset/3116) actually creates the
opposite problem: if any of $Label1 .. $Label5 are set on a message then modest
removes them.  I think it's obvious what's happening, but do let me know if you
want me to produce a trace anyway.

If I understand http://tools.ietf.org/html/rfc2060#section-6.4.6 correctly, it
should be fixable by changing the "UID STORE ... FLAGS.SILENT" command to
+FLAGS.SILENT instead.

Comment #7 makes sense. I'll take a look at this issue.

Now it should be fixed. A bitfield was being abused by recently changed code. I
reverted last change and cleared the bit from being abused (that bit, in the
flags, #18, held the $Label1 setting on IMAP specific providers).

Please mark as closed once verified that trunk (tinymail) indeed fixes this.

Comment 10 Dirk-Jan C. Binnema 2007-12-19 10:17:51 UTC

ok, fixed in svn -- should be in the next modest update as well.
please reopen if you reproduce it after that, thanks.

Keywords set by other MUAs are again cleared by modest 1.0-2008.43-1:

C: A00010 UID FETCH
44248,44272,44275,44308,44391,44438,44463,44472,44477:44479,44487,44489,44496:44505,44507:44508,44510:44521,44524,44528:44535
(FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (DATE FROM TO CC
SUBJECT MESSAGE-ID X-PRIORITY X-MSMAIL-PRIORITY IMPORTANCE X-MS-HAS-ATTACH
CONTENT-TYPE)])
S: [...]
   31 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44515 [...]
   32 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44516 [...]
   33 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44517 [...]
   [...]
[...]
C: A00112 UID STORE 44515:44528 FLAGS.SILENT (\\Seen)

Comment 12 Andre Klapper 2009-05-20 15:50:33 UTC

Anybody brave and willing to test this again with Modest from trunk?
https://git.maemo.org/projects/modest/gitweb?p=modest

The existing-label-removal bug is still reproducible in modest 3.0.15-rc3 /
tinymail r3911 on Fremantle beta scratchbox.  This is slightly out of date now
(retrieved and built soon after the beta SDK release) - if anything has been
committed specifically to address this problem since let me know and I will try
to build a fresher version.

Built today's modest/tinymail trunks for testing bug 3369 earlier, no change
regarding this.

We do this intentionally, tinymail only retrieves the minimum possible data in
order to reduce the amount of disk space for summaries and thus for loading
them into memory.

So this is a wontfix I think

Comment 16 Andre Klapper 2009-05-27 11:12:12 UTC

Thanks for explaining.

(In reply to comment #15)
> We do this intentionally, tinymail only retrieves the minimum possible data in
> order to reduce the amount of disk space for summaries and thus for loading
> them into memory.

I'm not sure I understand you correctly, but $Labeln keywords do seem to be
retrieved (see comment 9 & comment 11).  Also, even if they were not, wouldn't
"relative" STOREs (comment 7) solve the problem?

Comment 18 Andre Klapper 2009-06-02 14:56:34 UTC

Sergio, can you respond to Lucas' last comment?
Is using STOREs worth a separate issue?

(In reply to comment #17)
> (In reply to comment #15)
> > We do this intentionally, tinymail only retrieves the minimum possible data in
> > order to reduce the amount of disk space for summaries and thus for loading
> > them into memory.
> 
> I'm not sure I understand you correctly, but $Labeln keywords do seem to be
> retrieved (see comment 9 & comment 11).  Also, even if they were not, wouldn't
> "relative" STOREs (comment 7) solve the problem?
> 
Ah sorry I haven't understood you at first sight. Regarding using "relative"
stores you might be right. We would take a look at it (or if you have time just
feel free...)

(In reply to comment #19)
> We would take a look at it (or if you have time just feel free...)

Ok, I did have a look but I'm not at all familiar with the codebase so take the
following with several large pinches of salt:

AFAICT it looks like the flags have already been lost from the local storage
when imap_sync_online() is called.  The change that originally broke this has
been reverted and I'm not sure where else to look, any pointers appreciated.

While testing various things I came across a related bug: all flags also get
reset when marking a message unread (or even leaving an unread message
unopened), due to the old cyrus empty flags workaround.  There is an upstream
bug (and patch) for this at <http://bugzilla.gnome.org/show_bug.cgi?id=530241>,
but obviously it won't do much good until the lost flags issue is fixed.  I
could file a separate bug for this if you prefer.

> Regarding using "relative" stores you might be right.

Maybe not... For some strange reason I had assumed that tinymail/modest keeps
track of flags modified in the current session, but this doesn't seem to be the
case.  So doing differential UID STOREs isn't really possible, unless they are
done in realtime when messages get read/unread/deleted.  This is going into bug
2987 territory, but it might be the best way to fix this for all cases (for
example it seems that imap_sync_online() will happily remove any flags that
imap_parse_flag_list() doesn't know about).

(In reply to comment #20)
> So doing differential UID STOREs isn't really possible, unless they are
> done in realtime when messages get read/unread/deleted.

Scratch that, there's still the case of modifying messages when offline, then
syncing later.  (Question: how does one go about testing offline mode operation
in scratchbox?)

I can see two possible ways out of this:

A. Do a "UID STORE ... +FLAGS.SILENT" and a "UID STORE ... -FLAGS.SILENT"
instead of a single unqualified one, or

B. FETCH message flags from the server and merge them with the locally stored
ones before STORE

but both would require extra round trips.  In the case of A the worst case is
two STOREs per set of messages with identical modest-side flags, while B would
need one extra FETCH to retrieve all message flags and depending on the
response up to one STORE per message.

Any other ideas?

Reproducible also in 1.2009.41-10.  Bumping severity a notch - I probably
wouldn't go as far as "critical", but corrupting users' message flags does
count as data loss in my book.

Comment 23 Andre Klapper 2009-10-22 21:45:42 UTC

Does this change comment 15 here?

Comment 24 Andre Klapper 2010-01-07 15:40:18 UTC

This is currently a WONTFIX:
We do this intentionally, tinymail only retrieves the minimum possible data in
order to reduce the amount of disk space for summaries and thus for loading
them into memory.

Right, so it's OK to eat the users' message flags as long as it does it
efficiently :-b

Is there any known workaround for this? 

(I'm curious as to why it seems to not be affecting a lot of people, but seems
like a very visible/annoying issue.)

Further to my last comment, it seems using nokia messaging instead is a
"workaround" as the then the flags don't get cleared...

(In reply to comment #27)
> Further to my last comment, it seems using nokia messaging instead is a
> "workaround" as the then the flags don't get cleared...

For small values of "workaround", since Nokia Messaging is only an option for a
small subset of email accounts and even for those some people are not
comfortable handing their email contents and account passwords over to Nokia.

I realise that this is marked as WONTFIX, but it is severely hindering me in
using Modest: I shy away from it since I know that all my Thunderbird tags will
be lost... and there is no (practical) workaround or even a comparable
alternative client that does not destroy user data, right?

What about an option that would enable the extra network roundtrips necessary?
This would give people that value data integrity more than limiting network
usage a choice?

According to comment #21 method A, there could be a solution that only requires
extra network usage and no increase in memory usage, right?
I would be grateful if someone could give me a hint on where the relevant
sections in the code are.

cheers

I believe one simple fix would be just to record the thunderbird flags (I think
they're called $Label<1-5> or something like that) in the same way other flags
are recorded, and then pass them back when the server sets flags. Other flags
are handled like this, and there's space in the bitfield flags are stored in to
do this without requiring any extra memory from what I remember. (I've
forgotten where the code is, sorry)

The patch here possibly points you in roughly the right area of code:
http://bugzilla-attachments.gnome.org/attachment.cgi?id=110100

Hi, I just want to say 'me too'.
I'm using Thunderbird message flags/tags quite frequently, and my new n900 just
removed all tags from my messages.
Please fix this, as it really is a very annoying bug.

*** Bug 10995 has been marked as a duplicate of this bug. ***

I agree that this is an extremely annoying issue and as I rely on setting these
message flags in how I go about using my e-mail client on a regular basis I see
this as also as a form of data corruption and disappointed that this is an
issue that won't/cannot be resolved.