maemo.org Bugzilla – Bug 2536
Retrieving a message from IMAP servers removes message flags
Last modified: 2010-07-31 01:02:49 UTC
You need to log in before you can comment on or make changes to this bug.
STEPS TO REPRODUCE THE PROBLEM: 1. Configure modest to access an IMAP account, connect to the account and retrieve some messages. 2. Connect to the same account with another IMAP client such as Thunderbird or Evolution. The messages retrieved by modest are now marked as important. I think this is implemented using IMAP keywords (thunderbird calls them 'tags' and evolution 'labels'), specifically $Label1. EXPECTED OUTCOME: Modest shouldn't set message keywords on retrieving a message, or if it really has to it should use a keyword that doesn't collide with other MUAs' common usage. ACTUAL OUTCOME: As above. REPRODUCIBILITY: (always/sometimes/once) Always. EXTRA SOFTWARE INSTALLED: Nothing really relevant on the device. The IMAP server is Cyrus imapd 2.3.8. OTHER COMMENTS: I haven't tested interactions in the other direction, ie marking messages 'important' in another MUA and seeing how modest handles that. Not sure what modest is using this keyword for (haven't looked at the source yet) but there may be interoperability issues there as well.
hmmm... cannot reproduce this (but not using Cyrus here). do you have a protocol dump / pcap file (ie. something you can get from the e.g. wireshark program) that shows this behaviour? thanks.
The server enforces TLS but I managed to set up privoxy on a box on the local LAN instead and look at the traffic more closely. It appears that the flags are set when modest is closed and not when the messages are read as I reported initially. Will attach session log in a minute.
Created an attachment (id=664) [details] Perdition log showing modest setting $Label1
Sorry I got my proxies mixed up. I meant perdition, not privoxy in the previous comment.
This should be fixed in Tinymail's trunk. Please retest and close if fixed (I don't have sufficient rights to mark bugs as fixed on this bugzilla Dirk-Jan. Can you fix this for me?)
should be fixed, please reopen (with a new trace please!) if you can still reproduce. coming soon...
The fix (http://tinymail.org/trac/tinymail/changeset/3116) actually creates the opposite problem: if any of $Label1 .. $Label5 are set on a message then modest removes them. I think it's obvious what's happening, but do let me know if you want me to produce a trace anyway. If I understand http://tools.ietf.org/html/rfc2060#section-6.4.6 correctly, it should be fixable by changing the "UID STORE ... FLAGS.SILENT" command to +FLAGS.SILENT instead.
Comment #7 makes sense. I'll take a look at this issue.
Now it should be fixed. A bitfield was being abused by recently changed code. I reverted last change and cleared the bit from being abused (that bit, in the flags, #18, held the $Label1 setting on IMAP specific providers). Please mark as closed once verified that trunk (tinymail) indeed fixes this.
ok, fixed in svn -- should be in the next modest update as well. please reopen if you reproduce it after that, thanks.
Keywords set by other MUAs are again cleared by modest 1.0-2008.43-1: C: A00010 UID FETCH 44248,44272,44275,44308,44391,44438,44463,44472,44477:44479,44487,44489,44496:44505,44507:44508,44510:44521,44524,44528:44535 (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (DATE FROM TO CC SUBJECT MESSAGE-ID X-PRIORITY X-MSMAIL-PRIORITY IMPORTANCE X-MS-HAS-ATTACH CONTENT-TYPE)]) S: [...] 31 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44515 [...] 32 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44516 [...] 33 FETCH (FLAGS (\\Seen NonJunk $Label2) UID 44517 [...] [...] [...] C: A00112 UID STORE 44515:44528 FLAGS.SILENT (\\Seen)
Anybody brave and willing to test this again with Modest from trunk? https://git.maemo.org/projects/modest/gitweb?p=modest
The existing-label-removal bug is still reproducible in modest 3.0.15-rc3 / tinymail r3911 on Fremantle beta scratchbox. This is slightly out of date now (retrieved and built soon after the beta SDK release) - if anything has been committed specifically to address this problem since let me know and I will try to build a fresher version.
Built today's modest/tinymail trunks for testing bug 3369 earlier, no change regarding this.
We do this intentionally, tinymail only retrieves the minimum possible data in order to reduce the amount of disk space for summaries and thus for loading them into memory. So this is a wontfix I think
Thanks for explaining.
(In reply to comment #15) > We do this intentionally, tinymail only retrieves the minimum possible data in > order to reduce the amount of disk space for summaries and thus for loading > them into memory. I'm not sure I understand you correctly, but $Labeln keywords do seem to be retrieved (see comment 9 & comment 11). Also, even if they were not, wouldn't "relative" STOREs (comment 7) solve the problem?
Sergio, can you respond to Lucas' last comment? Is using STOREs worth a separate issue?
(In reply to comment #17) > (In reply to comment #15) > > We do this intentionally, tinymail only retrieves the minimum possible data in > > order to reduce the amount of disk space for summaries and thus for loading > > them into memory. > > I'm not sure I understand you correctly, but $Labeln keywords do seem to be > retrieved (see comment 9 & comment 11). Also, even if they were not, wouldn't > "relative" STOREs (comment 7) solve the problem? > Ah sorry I haven't understood you at first sight. Regarding using "relative" stores you might be right. We would take a look at it (or if you have time just feel free...)
(In reply to comment #19) > We would take a look at it (or if you have time just feel free...) Ok, I did have a look but I'm not at all familiar with the codebase so take the following with several large pinches of salt: AFAICT it looks like the flags have already been lost from the local storage when imap_sync_online() is called. The change that originally broke this has been reverted and I'm not sure where else to look, any pointers appreciated. While testing various things I came across a related bug: all flags also get reset when marking a message unread (or even leaving an unread message unopened), due to the old cyrus empty flags workaround. There is an upstream bug (and patch) for this at <http://bugzilla.gnome.org/show_bug.cgi?id=530241>, but obviously it won't do much good until the lost flags issue is fixed. I could file a separate bug for this if you prefer. > Regarding using "relative" stores you might be right. Maybe not... For some strange reason I had assumed that tinymail/modest keeps track of flags modified in the current session, but this doesn't seem to be the case. So doing differential UID STOREs isn't really possible, unless they are done in realtime when messages get read/unread/deleted. This is going into bug 2987 territory, but it might be the best way to fix this for all cases (for example it seems that imap_sync_online() will happily remove any flags that imap_parse_flag_list() doesn't know about).
(In reply to comment #20) > So doing differential UID STOREs isn't really possible, unless they are > done in realtime when messages get read/unread/deleted. Scratch that, there's still the case of modifying messages when offline, then syncing later. (Question: how does one go about testing offline mode operation in scratchbox?) I can see two possible ways out of this: A. Do a "UID STORE ... +FLAGS.SILENT" and a "UID STORE ... -FLAGS.SILENT" instead of a single unqualified one, or B. FETCH message flags from the server and merge them with the locally stored ones before STORE but both would require extra round trips. In the case of A the worst case is two STOREs per set of messages with identical modest-side flags, while B would need one extra FETCH to retrieve all message flags and depending on the response up to one STORE per message. Any other ideas?
Reproducible also in 1.2009.41-10. Bumping severity a notch - I probably wouldn't go as far as "critical", but corrupting users' message flags does count as data loss in my book.
Does this change comment 15 here?
This is currently a WONTFIX: We do this intentionally, tinymail only retrieves the minimum possible data in order to reduce the amount of disk space for summaries and thus for loading them into memory.
Right, so it's OK to eat the users' message flags as long as it does it efficiently :-b
Is there any known workaround for this? (I'm curious as to why it seems to not be affecting a lot of people, but seems like a very visible/annoying issue.)
Further to my last comment, it seems using nokia messaging instead is a "workaround" as the then the flags don't get cleared...
(In reply to comment #27) > Further to my last comment, it seems using nokia messaging instead is a > "workaround" as the then the flags don't get cleared... For small values of "workaround", since Nokia Messaging is only an option for a small subset of email accounts and even for those some people are not comfortable handing their email contents and account passwords over to Nokia.
I realise that this is marked as WONTFIX, but it is severely hindering me in using Modest: I shy away from it since I know that all my Thunderbird tags will be lost... and there is no (practical) workaround or even a comparable alternative client that does not destroy user data, right? What about an option that would enable the extra network roundtrips necessary? This would give people that value data integrity more than limiting network usage a choice? According to comment #21 method A, there could be a solution that only requires extra network usage and no increase in memory usage, right? I would be grateful if someone could give me a hint on where the relevant sections in the code are. cheers
I believe one simple fix would be just to record the thunderbird flags (I think they're called $Label<1-5> or something like that) in the same way other flags are recorded, and then pass them back when the server sets flags. Other flags are handled like this, and there's space in the bitfield flags are stored in to do this without requiring any extra memory from what I remember. (I've forgotten where the code is, sorry) The patch here possibly points you in roughly the right area of code: http://bugzilla-attachments.gnome.org/attachment.cgi?id=110100
Hi, I just want to say 'me too'. I'm using Thunderbird message flags/tags quite frequently, and my new n900 just removed all tags from my messages. Please fix this, as it really is a very annoying bug.
*** Bug 10995 has been marked as a duplicate of this bug. ***
I agree that this is an extremely annoying issue and as I rely on setting these message flags in how I go about using my e-mail client on a regular basis I see this as also as a form of data corruption and disappointed that this is an issue that won't/cannot be resolved.