Bug 8574 – Subfolders of Zimbra IMAP server not appearing/not available

Description Bill Arlofski (reporter) 2010-01-27 14:49:00 UTC

SOFTWARE VERSION:
- 2.2009.51-1.002

- Email server is Zimbra Collaboration Server v6.0.4GA

--[snip]--
* OK zimbra.example.com Zimbra IMAP4rev1 service ready
A1 CAPABILITY
* CAPABILITY IMAP4rev1 LOGINDISABLED STARTTLS AUTH=X-ZIMBRA ACL BINARY CATENATE
CHILDREN CONDSTORE ENABLE ESEARCH ESORT I18NLEVEL=1 ID IDLE LIST-EXTENDED
LITERAL+ LOGIN-REFERRALS MULTIAPPEND NAMESPACE QRESYNC QUOTA RIGHTS=ektx
SASL-IR SEARCHRES SORT THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN
X-DRAFT-I00-LIST-STATUS
A1 OK CAPABILITY completed
--[snip]--


EXACT STEPS LEADING TO PROBLEM: 
1. Open EMAIL app, choose IMAP4 email account
2. Open INBOX (or any other 'top level' folder of IMAP server)
3. No sub folders are listed

EXPECTED OUTCOME:
I would expect to see the same tree structure as in my other clients do with 
this same account (Thunderbird 2.x and 3.x on Linux for example, the web ajax
Zimbra interface, my iPhone, Zimbra Desktop Client)


ACTUAL OUTCOME:
- No sub folders show up of INBOX
- Other folders at same level as INBOX show up 
- Subfolders in other folders also do not show up
- When attempting to MOVE a message from INBOX, my list of top-level folders is
presented, and when choosing one, I am not presented with a list of its
sub-folders
- In this current state, I simply can not use the email app without subfolder
support. Perhaps Zimbra is telling the email app something it does not
understand/accept?


REPRODUCIBILITY:
Always

EXTRA SOFTWARE INSTALLED:
N/A - Was seeing this same behavior before first app was installed, and in the
two previous firmwares released.

OTHER COMMENTS:

- ALL folders and sub-folders are visible and functional in other email/IMAP
clients
- ALL folders and sub-folders are subscribed (verified with Thunderbird bird)

I would be happy to help in any way needed including offering a login to my
Zimbra server if that would help.  I really need email on my N900. :)


User-Agent: N/A

Comment 1 Andre Klapper maemo.org

2010-01-27 15:40:30 UTC

Thanks for reporting this.
This is direct IMAP access, not over Nokia Messaging Service?
Also see bug 8191 for a related.

Comment 2 Lucas Maneos 2010-01-27 20:33:33 UTC

Sounds similar to bug 6702 also.  A log (see
<https://wiki.maemo.org/Bugs:Stock_answers#Debugging_the_connection_to_the_mail_server>)
would be helpful here.

Comment 3 Bill Arlofski (reporter) 2010-01-28 01:38:35 UTC

(In reply to comment #1)
> Thanks for reporting this.
> This is direct IMAP access, not over Nokia Messaging Service?

Hi Andre.

Yes, direct IMAP access using the built-in email client on Maemo5 on N900

> Also see bug 8191 for a related.

I reviewed that bug and this is not the same. That bug refers to MFE (syncing
calendars, email contacts etc)   I am just using plain IMAP against my Zimbra
server as it is not currently licensed for their "push" capabilities via
Exchange compatible methods.

Comment 4 Bill Arlofski (reporter) 2010-01-28 02:12:51 UTC

(In reply to comment #2)
> Sounds similar to bug 6702 also.

Hi Lucas. YES, the symptoms seem exact, just a different IMAP server. 

> A log (see
> <https://wiki.maemo.org/Bugs:Stock_answers#Debugging_the_connection_to_the_mail_server>)
> would be helpful here.

See new attachment.

Please let me know if there is anything else you need to help

Comment 5 Bill Arlofski (reporter) 2010-01-28 02:14:56 UTC

Created an attachment (id=2151) [details]
Per Lucas:  Debugging_the_connection_to_the_mail_server Output

Comment 6 Andre Klapper maemo.org

2010-01-28 19:14:28 UTC

Thanks!

Comment 7 Sergio Villar Senin 2010-01-28 19:17:06 UTC

Could it be possible to get a test account in the server?

Comment 8 Bill Arlofski (reporter) 2010-01-28 19:25:10 UTC

(In reply to comment #7)
> Could it be possible to get a test account in the server?
> 

ABSOLUTELY.

I will email your credentials shortly.

If you could just respond to this ticket when you get them and successfully log
in I'd appreciate it.

Comment 9 Sergio Villar Senin 2010-01-28 20:23:04 UTC

(In reply to comment #8)
> (In reply to comment #7)
> > Could it be possible to get a test account in the server?
> > 
> 
> ABSOLUTELY.
> 
> I will email your credentials shortly.
> 
> If you could just respond to this ticket when you get them and successfully log
> in I'd appreciate it. 

I got credentials. It seems there is something wrong with your server as it is
rejecting a LIST command from tinymail. I've just verified that it does the
same in evolution. And with Thunderbird I cannot even see any folder.

Do you have access to the server? Have you configured the namespaces?

Comment 10 Lucas Maneos 2010-01-28 20:40:19 UTC

(In reply to comment #5)
> Created an attachment (id=2151) [details] [details]

Thanks for the log, enough info there to confirm I think.

> sending : A00005 NAMESPACE
> received: * NAMESPACE (("" "/")) (("/home/" "/")) NIL
> received: A00005 OK NAMESPACE completed
> sending : A00006 LIST (SUBSCRIBED) "" %

Hm, the empty namespace causing tinymail to list folders with a "%" wildcard
(ie, non-recursively, top-level only) reminds me of bug 2537.

> received: * LIST (\Subscribed \HasChildren) "/" "Archives"
> [...]
> received: * LIST (\Subscribed \HasChildren) "/" "INBOX"
> [...]
> received: A00006 OK LIST completed
> sending : A00007 LIST (SUBSCRIBED) "Archives/" *
> received: * LIST (\Subscribed \HasNoChildren) "/" "Archives/2008"
> received: * LIST (\Subscribed \HasNoChildren) "/" "Archives/2009"
> received: A00007 OK LIST completed
> sending : A00008 LSUB "/home/" {1+}

I don't quite understand the logic behind it, but get_folders_sync_ns() decides
to recurse into the first folder found in the namespace (and only that).  IMHO
it should LIST (SUBSCRIBED) / LSUB all folders (except the ones matching other
namespaces obviously).

Comment 11 Sergio Villar Senin 2010-01-28 20:46:55 UTC

(In reply to comment #10)
> (In reply to comment #5)
> > Created an attachment (id=2151) [details] [details] [details]
> 
> Thanks for the log, enough info there to confirm I think.
> 
> > sending : A00005 NAMESPACE
> > received: * NAMESPACE (("" "/")) (("/home/" "/")) NIL
> > received: A00005 OK NAMESPACE completed
> > sending : A00006 LIST (SUBSCRIBED) "" %
> 
> Hm, the empty namespace causing tinymail to list folders with a "%" wildcard
> (ie, non-recursively, top-level only) reminds me of bug 2537.
> 
> > received: * LIST (\Subscribed \HasChildren) "/" "Archives"
> > [...]
> > received: * LIST (\Subscribed \HasChildren) "/" "INBOX"
> > [...]
> > received: A00006 OK LIST completed
> > sending : A00007 LIST (SUBSCRIBED) "Archives/" *
> > received: * LIST (\Subscribed \HasNoChildren) "/" "Archives/2008"
> > received: * LIST (\Subscribed \HasNoChildren) "/" "Archives/2009"
> > received: A00007 OK LIST completed
> > sending : A00008 LSUB "/home/" {1+}
> 
> I don't quite understand the logic behind it, but get_folders_sync_ns() decides
> to recurse into the first folder found in the namespace (and only that).  IMHO
> it should LIST (SUBSCRIBED) / LSUB all folders (except the ones matching other
> namespaces obviously).

Exactly, I was taking a look at that

Comment 12 Bill Arlofski (reporter) 2010-01-28 20:50:40 UTC

(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > Could it be possible to get a test account in the server?
> > > 
> > 
> > ABSOLUTELY.
> > 
> > I will email your credentials shortly.
> > 
> > If you could just respond to this ticket when you get them and successfully log
> > in I'd appreciate it. 
> 
> I got credentials. It seems there is something wrong with your server as it is
> rejecting a LIST command from tinymail. I've just verified that it does the
> same in evolution. And with Thunderbird I cannot even see any folder.
> 
> Do you have access to the server? Have you configured the namespaces?

Sergio... I was afraid that some of the limits I put on that account might
cause us some issues. I have assigned that account full "normal" rights for all
features (calendar, briefcase etc etc etc - all stuff you don't need, but I
don't want any restrictions hindering your troubleshooting)

I just tested with Thunderbird and it sees your two subfolders fine (and I sent
a test email and that is in INbox too) 

I will attach Thunderbird image next


I have not done anything with "namespaces"

Comment 13 Bill Arlofski (reporter) 2010-01-28 20:51:36 UTC

Created an attachment (id=2158) [details]
Thunderbird connected to test account, showing subfolders properly

Comment 14 Lucas Maneos 2010-01-28 20:54:36 UTC

BTW, test accounts can also be obtained at
http://www.zimbra.com/products/hosted_demo.php.

Comment 15 Bill Arlofski (reporter) 2010-01-28 20:57:33 UTC

(In reply to comment #14)
> BTW, test accounts can also be obtained at
> http://www.zimbra.com/products/hosted_demo.php.

heh. Good point I hadn't thought of that. 

Thanks for jumping on this so quickly Sergio, Andre & Lucas!

Comment 16 Sergio Villar Senin 2010-01-29 10:29:46 UTC

The code Lucas mentioned makes no sense to me.

I've reviewed the rationale of the functions that get the folder information
for servers with more than 1 namespace, and it basically does not get all the
folders recursively in order to avoid massive LIST commands caused by IMAP
servers that wrap NNTP newsgroups.

IMHO it's better just to show all the folders as I don't think it's the more
common case. The fix is not complicated.

Comment 17 Sergio Villar Senin 2010-02-01 13:25:33 UTC

I have finally committed the fix that allows Modest to show subfolder of IMAP
in Zimbra servers, and in general fixes namespace handling in tinymail.

Fixed in
tinymail-1-2: r4163 - r4165
trunk: r4166 - r4168

Comment 18 Sergio Villar Senin 2010-02-01 13:55:26 UTC

(In reply to comment #17)
> I have finally committed the fix that allows Modest to show subfolder of IMAP
> in Zimbra servers, and in general fixes namespace handling in tinymail.
> 
> Fixed in
> tinymail-1-2: r4163 - r4165
> trunk: r4166 - r4168
> 

Actually I had to add another commit more to have it completely fixed

Comment 19 Bill Arlofski (reporter) 2010-02-01 14:50:15 UTC

(In reply to comment #17)
> I have finally committed the fix that allows Modest to show subfolder of IMAP
> in Zimbra servers, and in general fixes namespace handling in tinymail.
> 
> Fixed in
> tinymail-1-2: r4163 - r4165
> trunk: r4166 - r4168


Hi Sergio!   This sounds great!   How do I go about applying this fix to test
on my N900? 

Or do I need to wait for the next Maemo official update?

Thanks!

Comment 20 Sergio Villar Senin 2010-02-01 14:53:32 UTC

(In reply to comment #19)
> (In reply to comment #17)
> > I have finally committed the fix that allows Modest to show subfolder of IMAP
> > in Zimbra servers, and in general fixes namespace handling in tinymail.
> > 
> > Fixed in
> > tinymail-1-2: r4163 - r4165
> > trunk: r4166 - r4168
> 
> 
> Hi Sergio!   This sounds great!   How do I go about applying this fix to test
> on my N900? 
> 
> Or do I need to wait for the next Maemo official update?

Yes, my advice is to wait for the next official update. Otherwise you'd have to
do tricky things in order not to break the system update.

Comment 21 Bill Arlofski (reporter) 2010-02-01 18:12:37 UTC

(In reply to comment #20)
> (In reply to comment #19)
> > (In reply to comment #17)
> > > I have finally committed the fix that allows Modest to show subfolder of IMAP
> > > in Zimbra servers, and in general fixes namespace handling in tinymail.
> > > 
> > > Fixed in
> > > tinymail-1-2: r4163 - r4165
> > > trunk: r4166 - r4168
> > 
> > 
> > Hi Sergio!   This sounds great!   How do I go about applying this fix to test
> > on my N900? 
> > 
> > Or do I need to wait for the next Maemo official update?
> 
> Yes, my advice is to wait for the next official update. Otherwise you'd have to
> do tricky things in order not to break the system update.


Sergio... I failed to mention it earlier, but I am an experienced Unix/Linux
admin and am not afraid of making minor modifications to source, and compiling
etc - Gentoo being my Linux of choice even. :)

I was told in #maemo on freenode to ask you the following with regards to your
last comment:

'..if it's anything other than "the modest data store format has changed" I'd
say patch and build.'


So if the change is not in the mail data store format, and if there is a URL
that shows a quick how-to for me to get the latest patched modest mail app
installed to my N900 I would be really happy to take the steps - and report
back even. 

Thanks!

Comment 22 Lucas Maneos 2010-02-02 19:31:00 UTC

If you are feeling brave and have some time to kill you could apt-get source
libtinymail-1.0-0, grab changesets 4148, 4149, 4163-4165 & 4170 from upstream
(<http://tinymail.org/trac/tinymail>), apply manually and rebuild the package.
Leaving the package version intact is Wrong(TM) on many levels, but avoids
breaking dependencies and therefore the next SSU.

I think you only need to install the libtinymail-camel deb.

Of course if it breaks you get to keep both pieces etc :-)

Comment 23 Sergio Villar Senin 2010-02-02 20:46:39 UTC

(In reply to comment #21)
> (In reply to comment #20)
> > (In reply to comment #19)
> > > (In reply to comment #17)
> > > > I have finally committed the fix that allows Modest to show subfolder of IMAP
> > > > in Zimbra servers, and in general fixes namespace handling in tinymail.
> > > > 
> > > > Fixed in
> > > > tinymail-1-2: r4163 - r4165
> > > > trunk: r4166 - r4168
> > > 
> > > 
> > > Hi Sergio!   This sounds great!   How do I go about applying this fix to test
> > > on my N900? 
> > > 
> > > Or do I need to wait for the next Maemo official update?
> > 
> > Yes, my advice is to wait for the next official update. Otherwise you'd have to
> > do tricky things in order not to break the system update.
> 
> 
> Sergio... I failed to mention it earlier, but I am an experienced Unix/Linux
> admin and am not afraid of making minor modifications to source, and compiling
> etc - Gentoo being my Linux of choice even. :)

Ok, then use the 

> 
> I was told in #maemo on freenode to ask you the following with regards to your
> last comment:
> 
> '..if it's anything other than "the modest data store format has changed" I'd
> say patch and build.'
> 
> 
> So if the change is not in the mail data store format, and if there is a URL
> that shows a quick how-to for me to get the latest patched modest mail app
> installed to my N900 I would be really happy to take the steps - and report
> back even. 
> 
> Thanks!
> 

(In reply to comment #22)
> If you are feeling brave and have some time to kill you could apt-get source
> libtinymail-1.0-0, grab changesets 4148, 4149, 4163-4165 & 4170 from upstream
> (<http://tinymail.org/trac/tinymail>), apply manually and rebuild the package.
> Leaving the package version intact is Wrong(TM) on many levels, but avoids
> breaking dependencies and therefore the next SSU.
> 
> I think you only need to install the libtinymail-camel deb.
> 
> Of course if it breaks you get to keep both pieces etc :-)

As usual Lucas is completely right. The other option is to download the
tinymail-1-2 branch which is the one used for the next release and build debian
packages. Note that in this case you should replace the version in
debian/control with the same version installed in your device in order not to
break SSU.

Comment 24 Andre Klapper maemo.org

2010-03-15 20:56:33 UTC

Setting explicit PR1.2 milestone (so it's clearer in which public release the
fix will be available to users).

Sorry for the bugmail noise (you can filter on this message).

Comment 25 Bart Van Kerckhove 2010-09-09 20:34:25 UTC

(In reply to comment #24)
> Setting explicit PR1.2 milestone (so it's clearer in which public release the
> fix will be available to users).
> 
> Sorry for the bugmail noise (you can filter on this message).
> 

I still seem to hit this bug, with Zimbra 5.x.
Has this been included in the 1.2 release?

Comment 26 Andre Klapper maemo.org

2010-09-10 23:53:49 UTC

(In reply to comment #25)
> I still seem to hit this bug, with Zimbra 5.x.
> Has this been included in the 1.2 release?

(In reply to comment #17)
> Fixed in
> tinymail-1-2: r4163 - r4165
> trunk: r4166 - r4168

Latest Maemo5 release still ships tinymail 1.1.* so it's not shipped it seems.
:-/
Reverting FIXED state as fix is not integrated.

Comment 27 Bart Van Kerckhove 2010-09-11 22:09:01 UTC

(In reply to comment #26)
> (In reply to comment #25)
> > I still seem to hit this bug, with Zimbra 5.x.
> > Has this been included in the 1.2 release?
> 
> (In reply to comment #17)
> > Fixed in
> > tinymail-1-2: r4163 - r4165
> > trunk: r4166 - r4168
> 
> Latest Maemo5 release still ships tinymail 1.1.* so it's not shipped it seems.
> :-/
> Reverting FIXED state as fix is not integrated.
> 

Is there any quick way to get this fix installed? Any binary builds available?
Having done no n900 development so far, setting up sandbox for one library
seems a little daunting.. Anyone?

Comment 28 Andre Klapper maemo.org

2010-09-15 23:26:55 UTC

(In reply to comment #27)
> Is there any quick way to get this fix installed? 

Compiling tinymail from master in Scratchbox...

Comment 29 Andre Klapper maemo.org

2010-09-20 13:13:14 UTC

The commits r4163 - r4165 are included in the next release:
Repository location: http://git.gitorious.org/tinymail/tinymail.git
Commits info:
- r4163: df977755fd561be59650bcb7d391492ef5f396b2
- r4164: efce3d1a62a1cb9c266a8cc67697ecdd70c9e756
- r4165: fad57ae4ae9f7da79b129e64d0c31494cd69f1d7

Comment 30 Andre Klapper maemo.org

2010-10-25 17:12:28 UTC

The problem reported here should be fixed in the update that was released today
for public: The Maemo5 update version 20.2010.36-2 (also called "PR1.3"
sometimes). Please leave a comment if the problem is not fixed for you in this
update version.

Comment 31 Bart Van Kerckhove 2010-10-25 20:30:46 UTC

(In reply to comment #30)
> The problem reported here should be fixed in the update that was released today
> for public: The Maemo5 update version 20.2010.36-2 (also called "PR1.3"
> sometimes). Please leave a comment if the problem is not fixed for you in this
> update version.
> 

I tested this first thing after the PR1.3 release.
No dice - still does not work.
I'm afraid this will mean it will never get fixed (or at least properly
committed).
Perhaps it can be marked as such.

Comment 32 Andre Klapper maemo.org

2010-10-25 22:28:36 UTC

(In reply to comment #31)
> I tested this first thing after the PR1.3 release.
> No dice - still does not work.

I wonder if it's because you use Zimbra 5. Feel free to file a new separate bug
report and link to it from here.


Bill: Does it work for you with Zimbra 6 in 20.2010.36-2?

Comment 33 stoffel.010170 2010-10-26 16:31:19 UTC

after update to zimbra server 6.0X imap sync don't work anymore on n900 under
firmware pr 1.2 and pr 1.3. 

with zimbra server 5.X works everything fine under firmware pr 1.2.

Comment 34 Andre Klapper maemo.org

2010-10-26 16:37:03 UTC

(In reply to comment #33)
> after update to zimbra server 6.0X imap sync don't work anymore on n900 under
> firmware pr 1.2 and pr 1.3. 

That is covered in bug 11428.

Comment 35 Bill Arlofski (reporter) 2010-11-01 23:14:18 UTC

> Bill: Does it work for you with Zimbra 6 in 20.2010.36-2?

Except for subfolders of my INBOX folder.  Those still just appear in the main
list in alphabetical order so there is no knowing which  parent folder they
reside under.

For other top-level folders (same level as INBOX) It looks like they do now.
These folders do show sub folders beneath them, tabbed in slightly under their
parent folder and have a dot (bullet point) next to them on the left. 



Sorry to say that this does not fix sub folders of the INBOX.

Attachments
Per Lucas: Debugging_the_connection_to_the_mail_server Output (33.54 KB, text/plain) 2010-01-28 02:14 UTC, Bill Arlofski	Details
Thunderbird connected to test account, showing subfolders properly (38.33 KB, image/png) 2010-01-28 20:51 UTC, Bill Arlofski	Details
Add an attachment (proposed patch, testcase, etc.)

(int-154858)
Summary:	Subfolders of Zimbra IMAP server not appearing/not available