Bug 11993 - Mangling email addresses containing accented characters when replying
: Mangling email addresses containing accented characters when replying
Status: NEW
Product: Maemo 5 Community SSU
modest
: unspecified
: N900 Maemo
: Unspecified normal with 3 votes (vote)
: ---
Assigned To: unassigned
: general
:
:
:
:
  Show dependency tree
 
Reported: 2011-02-28 00:49 UTC by Petr Vanek
Modified: 2014-05-18 23:08 UTC (History)
8 users (show)

See Also:


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description Petr Vanek (reporter) 2011-02-28 00:49:18 UTC
SOFTWARE VERSION: 
mp-fremantle-community-pr 20.2010.36-2maemo12.5
modest 3.90.7-7

If the senders name contains accented characters, the name gets either
shortened or the address is completely missing upon hitting reply.


EXACT STEPS LEADING TO PROBLEM: 
(Explain in detail what you do (e.g. tap on OK) and what you see (e.g. message
Connection Failed appears))
1. on incoming email, hit reply or reply all 

EXPECTED OUTCOME: 

To: address field contains original sender address and name, Cc: address field
contains ccied addresses and names

ACTUAL OUTCOME: 

If the senders name contains accented characters, it gets either shortened or
the address is completely missing upon hitting reply (so the To: field is
empty):

"Halserová Eva" <email@address> gets shortened to: "Halserová Ev"
<email@address>;

"Adéla Horáková" <email@address> is shortened to "Adéla Horáko" <email@address>

but for example "Pavla Vlkovská" <email@address> is not included at all. it
seems to be related to other Cc: addresses too.



REPRODUCIBILITY: 
always

EXTRA SOFTWARE INSTALLED: 

OTHER COMMENTS:
Comment 1 Andre Klapper maemo.org 2011-02-28 13:50:01 UTC
I sent an email from GNOME Evolution 2.32.1 with a gmx.net POP account with
sender "André Klapperček <foo@bar.com>" to my GMail IMAP account configured in
Modest on Maemo5.
When clicking "Reply" in Modest, name is correctly displayed.
After sending it and receiving the answer in Evolution again, name is still
correct.

Hence very likely a bug in the email client of the sender and nothing to be
fixed in Modest.
Can you attach a complete email header (but feel free to change domain names to
xxxxxxx for privacy reasons as bugs.maemo.org is a public place) here?
Comment 2 Petr Vanek (reporter) 2011-02-28 14:44:32 UTC
Thank you for checking.

Very likely a modest/system issue as i have this on two email accounts, and
with any sender with accented characters (outlook clients, freemail clients).

n900 Settings -> Language & Region -> 

Device Language: English (USA) , tested also Czech but no luck.
Regional Settings: Czech republic



With the following received email header, when hitting Reply the result is TO:
"Radim Žlebe" <REMOVED>, when hitting Reply ALL, the result is TO: "Radim
Žlebe" <REMOVED>; Pavel Závork <REMOVED>, addresses in CC are missing
altogether. In reality i am noticing now, that "Pavel Závorka" <REMOVED> should
have been in CC, not in TO.


Received: from REMOVED ([192.168.2.101]) by
 REMOVED ([192.168.2.101]) with mapi; Mon, 28 Feb 2011 13:08:49
 +0100
From: Radim Žlebek <REMOVED>
To: Pavel Závorka <REMOVED>
CC: Petr Vaněk <REMOVED>, Ivana Holubová <REMOVED>
Date: Mon, 28 Feb 2011 13:08:47 +0100
Subject: RE: REMOVED
Thread-Topic: REMOVED
Thread-Index: AcvEYWjuZldnyFdjThSAq6wj9NgcHQPAGTTgADBo+xAAxxh/AAAAFOGg
Message-ID: 
 <80087F1E22EC0245BEB8706A5F553F39197DE186A2@REMOVED>
References: 
 <80087F1E22EC0245BEB8706A5F553F39197DE186A1@REMOVED>
In-Reply-To: 
 <80087F1E22EC0245BEB8706A5F553F39197DE186A1@REMOVED>
Accept-Language: cs-CZ
Content-Language: cs-CZ
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: REMOVED
X-MS-Has-Attach: yes
X-MS-Exchange-Organization-SCL: -1
X-MS-TNEF-Correlator: 
acceptlanguage: cs-CZ
Content-Type: multipart/related;
 boundary="_006_80087F1E22EC0245BEB8706A5F553F39197DE186A2exchange2REMOVED_";
 type="multipart/alternative"
MIME-Version: 1.0 


With the following header i get on Reply: TO: Hauserová Ev
<Eva.Hauserova@REMOVED>, on Reply ALL i get TO: Hauserová Ev
<Eva.Hauserova@REMOVED>; max@REMOVED; michal.krat@REMOVED; Mgr. Jiří Va
<Jiri.Vaca@REMOVED>;vanous@REMOVED
Please notice shortened names and again, this is all in TO, rather then in TO
and CC.



Delivered-To: REMOVED
Received: by 10.REMOVED.128.6 with SMTP id i6cs496876wbs;
        Wed, 23 Feb 2011 05:09:44 -0800 (PST)
Received: by 10.REMOVED.153.197 with SMTP id l5mr3540490wbw.75.1298466578914;
        Wed, 23 Feb 2011 05:09:38 -0800 (PST)
Received-SPF: softfail (google.com: best guess record for domain of
transitioning Eva.Hauserova@REMOVED does not designate REMOVED as permitted
sender) client-ip=REMOVED;
Received: by 10.REMOVED.241.103 with POP3 id 39mf524148wwb.6;
        Wed, 23 Feb 2011 05:09:38 -0800 (PST)
X-Gmail-Fetch-Info: REMOVED 1 penguin.cz 995 vanous
Return-Path: <Eva.Hauserova@REMOVED>
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on penguin
X-Spam-Level: *
X-Spam-Status: No, score=1.2 required=5.0 tests=BAYES_50,DNS_FROM_OPENWHOIS,
 HTML_MESSAGE,RDNS_NONE autolearn=no version=3.2.3
X-Original-To: REMOVED
Delivered-To: REMOVED
Received: from localhost (localhost [127.0.0.1])
 by ns.penguin.cz (Postfix) with ESMTP id E175D1416BC5
 for <vanous@REMOVED>; Wed, 23 Feb 2011 13:56:22 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at ns.penguin.cz
Received: from ns.penguin.cz ([127.0.0.1])
 by localhost (ns.penguin.cz [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id bHCGeoB1ZvxD for <vanous@REMOVED>;
 Wed, 23 Feb 2011 13:56:13 +0100 (CET)
Received: from REMOVED (unknown [213.151.95.3])
 by ns.penguin.cz (Postfix) with ESMTP id A61521410276
 for <vanous@REMOVED>; Wed, 23 Feb 2011 13:56:13 +0100 (CET)
Received: from eim.IM.local ([192.168.5.10]) by eim.IM.local ([192.168.5.10])
 with mapi; Wed, 23 Feb 2011 13:54:15 +0100
From: Hauserová Eva <Eva.Hauserova@REMOVED>
To: "max@REMOVED" <max@REMOVED>, "michal.krat@REMOVED" <michal.krat@REMOVED>,
Mgr. Jiří Vaca <Jiri.Vaca@REMOVED>, "vanous@REMOVED" <vanous@REMOVED>
Date: Wed, 23 Feb 2011 13:54:15 +0100
Subject: FW: Re: ekoobchudek - CZ verze
Thread-Topic: Re: ekoobchudek - CZ verze
Thread-Index: AcvSuIrI0nJQhV42Q16dLCH+HrFLXAAnTOSwAACqkzA=
Message-ID: <42C9323534EBBB4184E3160046DA870B2EA536CD15@eim.IM.local>
Accept-Language: cs-CZ, en-US
Content-Language: cs-CZ
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: cs-CZ, en-US
Content-Type: multipart/alternative;
 boundary="_000_42C9323534EBBB4184E3160046DA870B2EA536CD15eimIMlocal_"
MIME-Version: 1.0
Comment 3 Andre Klapper maemo.org 2011-02-28 15:14:40 UTC
(In reply to comment #2)
> From: Radim Žlebek <REMOVED>
> To: Pavel Závorka <REMOVED>
> CC: Petr Vaněk <REMOVED>, Ivana Holubová <REMOVED>

Accented characters are outside of the 7-bit ASCII character code and hence
must be encoded according to RFC (don't remember the exact one, my guesses are
either 2822 or 2047).
Closing as INVALID as the bug is in the sender's mail application.
Comment 4 Petr Vanek (reporter) 2011-02-28 20:03:32 UTC
Dear Andre,

apparently this issue has been trashed completely in many email clients'
bugtrackers. Please read for example here [1]. It is apparent, that the issue
is not in modest but it also is very clear that most freemail services (from
what i see is all the major here in CZ), hotmail and also Windows clients are
handling it in such a way, that the n900 email app is simply impossible to use
if you are receiving emails from such an address. This is rendering it unusable
for any corporate or even private use and i would like to advocate that you
please read the bugreport below to get a complete understanding. I also would
assume, that support for this _incorrect_ parsing has already been done in
tinymail or modest before. (I have searched, but there seems to be no
bugtracker for tinymail?) Is there a way to find out if the issue is handled
correctly in the nokia provided version of modest? Can we perhaps use the code
or technique from thunderbird to make this easier?

thank you
Petr


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=254519
Comment 5 Andre Klapper maemo.org 2011-02-28 20:17:25 UTC
(In reply to comment #4)
> Is there a way to find out if the issue is handled
> correctly in the nokia provided version of modest?

Any community update versions are on top of Nokia's versions. CSSU ChangeLog is
at http://wiki.maemo.org/Community_SSU#Changelog

> Can we perhaps use the code
> or technique from thunderbird to make this easier?

Thunderbird, Modest and Tinymail code is public. Maybe you can try to guess an
interpretation of non-7bit bytes by looking for charset info in other places of
the message, but there will always be some ambiguousness left, (plus generally
adding a workaround for every broken mailer out there will make code more
fragile and unreadable).
Without provided patches it's very unlikely that anything will happen though.
Comment 6 Andrew Flegg maemo.org 2011-02-28 20:44:26 UTC
(In reply to comment #4)
> 
> apparently this issue has been trashed completely in many email clients'
> bugtrackers. Please read for example here [1].
> 
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=254519

I'm not sure why this is pertinent, neither of the headers you've pasted above
include charset information; the handling of which (when they contain commas)
is the Mozilla bug in question.

If there is a bug around Modest handling commas inside encoded strings, we can
look at that, but your headers seem to contain top-bit set characters, and I
suspect the relevant RFCs do not state whether or not these should be
considered UTF-8, ISO-8859-15, ....

Copied from Mozilla's report:

  From: =?iso-8859-1?Q?Martin=2C_Andr=E9?= <andre.martin@example.com>
Comment 7 Petr Vanek (reporter) 2011-02-28 20:58:06 UTC
Hi, 

i will reply more in detail later, but in fact i can replicate this issue by
sending an email to myself. I have Vaněk in surname, when my own modest created
email arrives and i hit reply, i only get to see "Petr Vaně"<email>.

The issue is probably somewhere else and i am reopening it as UNCONFIRMED for
now.

Petr
Comment 8 Petr Vanek (reporter) 2011-02-28 23:16:32 UTC
OK, so i can 100% replicate by sending an email to myself fvia two different
servers - exchange and gmail.

On mail for exchange account, where i could only change the Display name in the
Exchange i guess, when sending an email, by looking through imap access into
the sent folder by Claws Mail, i can see the sent header as this:

From: Petr Vaněk <petr.vanek@REMOVED.cz>
To: Petr Vaněk <petr.vanek@REMOVED.cz>
Date: Mon, 28 Feb 2011 19:42:15 +0100
Subject: Test
Thread-Topic: Test
Thread-Index: AcvXdzHd3Hypw1+jRIO77izzctMgyA==
Message-ID: <1298918535.1708.8.camel@Nokia-N900>
Reply-To: Petr Vaněk <petr.vanek@REMOVED.cz>
Accept-Language: en-US, cs-CZ
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-Exchange-Organization-SCL: -1
X-MS-TNEF-Correlator: 
Content-Type: multipart/alternative;
 boundary="_000_129891853517088camelNokiaN900_"
MIME-Version: 1.0


The received email looks like this:


Received: from exchange2.REMOVED.cz ([192.168.2.101]) by
 exchange2.REMOVED.cz ([192.168.2.101]) with mapi; Mon, 28 Feb 2011 19:42:05
 +0100
From: Petr Vaněk <petr.vanek@REMOVED.cz>
To: Petr Vaněk <petr.vanek@REMOVED.cz>
Date: Mon, 28 Feb 2011 19:42:15 +0100
Subject: Test
Thread-Topic: Test
Thread-Index: AcvXdzHd3Hypw1+jRIO77izzctMgyA==
Message-ID: <1298918535.1708.8.camel@Nokia-N900>
Reply-To: Petr Vaněk <petr.vanek@REMOVED.cz>
Accept-Language: en-US, cs-CZ
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: exchange2.REMOVED.cz
X-MS-Has-Attach: 
X-MS-Exchange-Organization-SCL: -1
X-MS-TNEF-Correlator: 
acceptlanguage: en-US, cs-CZ
Content-Type: multipart/alternative;
 boundary="_000_129891853517088camelNokiaN900_"
MIME-Version: 1.0

And when hitting reply i get TO: "Petr Vaně" <petr.vanek@REMOVED.cz>

I did the same thing on my gmail imap account, setting the username to Petr
Vaněk, sending an email to myself, looking at it via Claws Mail via imap, here
is the result and below see the same composing Reply behavior:

Return-Path: <petrnek@REMOVED.com>
Received: from [192.168.3.13] ([194.REMOVED.223.3])
        by mx.google.com with ESMTPS id 5sm1775536fak.23.2011.02.28.12.12.50
        (version=SSLv3 cipher=OTHER);
        Mon, 28 Feb 2011 12:12:51 -0800 (PST)
From: Petr Vaněk <petrnek@REMOVED.com>
Reply-To: Petr Vaněk <petrnek@REMOVED.com>
To: petrnek@REMOVED.com
Subject: Test
X-Mailer: Modest 3.90.7
Content-Type: multipart/alternative; boundary="=-qVJrbJddDnzq+QJzDAzE"
Date: Mon, 28 Feb 2011 21:12:47 +0100
Message-Id: <1298923967.1821.17.camel@Nokia-N900>
Mime-Version: 1.0

When hitting reply i get TO: "Petr Vaně" <petrnek@REMOVED.com>

I can do more tests, if you can hint me what to check

thank you
Petr
Comment 9 Andrew Flegg maemo.org 2011-03-01 10:59:02 UTC
Confirming, but I haven't yet checked if this bug was introduced in the CSSU.

STEPS TO REPRODUCE.
0. Configure a Gmail account. Gmail's IMAP & SMTP servers seem to implement the
relevant standards well and support "Show original" so you can see the raw
headers/SMTP traffic.
1. Open Contacts
2. Select "New contact"
3. Enter last name: "Tëst, Må"
4. Add your Gmail account email address
5. Send an email to that account from Modest.
6. Press "Send & Receive"
7. Press "Reply" to the message received

EXPECTED OUTCOME
"To" line shows "Tëst, Må <foo@example.com>"

ACTUAL OUTCOME
"To line shows "Tëst, M <foo@example.com>"

OTHER COMMENTS
The original line when received by Gmail is:

To: =?UTF-8?Q?T=C4=9Bst=2C_M=C3=A5?= <example@gmail.com>
Comment 10 Petr Vanek (reporter) 2011-03-01 12:57:52 UTC
Andrew, thank you.

i don't use the gmail web interface but the Show original is a great help, in
my case the address is displayed as:

From: Petr =?UTF-8?Q?Van=C4=9Bk?= <email@address>

I am not sure how many packages do affect modest behavior, dpkg
--get-selections shows quite a few installed that have modest in their name,
would we know which ones to downgrade to see where this comes from?
Comment 11 Petr Vanek (reporter) 2011-03-04 15:16:45 UTC
I have reflashed rootfs and on this new system i set a new gmail acount. This
fixes the issue - when making a Reply, i now see correctly full name:

To: "Petr Vaněk" <email>

The header original (in gmail web) though, still looks like this:

From: Petr =?UTF-8?Q?Van=C4=9Bk?= <email>
Reply-To: Petr =?UTF-8?Q?Van=C4=9Bk?= <email>
Comment 12 Petr Vanek (reporter) 2011-03-06 19:06:58 UTC
After clean rootfs flash and installing CSSU, the bug is back.
Comment 13 Petr Vanek (reporter) 2011-11-21 23:19:45 UTC
I tried to update CSSU today (see below explanation only today) which reupdated
modest to Version: 3.90.7-10 which doesn't solve this issue - reply addresses
are unfortunately still being mangled and in some cases, the accent containing
addresses are not included at all in the To: or CC: after hitting Reply button.

The only way to work around this issue for me is to install older version of
modest ( apt-get install modest=3.4.8+0m5 ) which breaks CSSU and no further
auto updates are possible, leading to (for me) not applied security fixes of
CSSU :/

Details of current modest:

Depends: calendar-ui-widgets-0, gtkhtml3.14, libalarm2, libatk1.0-0 (>=
1.24.0), libc6 (>= 2.5.0-1), libcairo2, libcanberra0 (>= 0.10),
         libconic0, libdbus-1-3 (>= 1.1.4), libdbus-glib-1-2 (>= 0.76),
libebook (>= 1.4.2.1), libedataserver (>= 1.4.2.1), libfontconfig1
         (>= 2.6.0), libfreetype6 (>= 2.3.9), libgconf2-6 (>= 2.13.5),
libglib2.0-0 (>= 2.20.0), libgtk2.0-0 (>= 2:2.12.9-0osso1),
         libhildon1 (>= 2.2.22), libhildonfm2 (>= 1:2.28+0m5), libhildonmime0
(>= 2.1.5), libhildonnotify0 (>= 0.0.2-1-recomp1), libiphb0,
         libmcclient6, libnotify1, libosso-abook, libosso-gnomevfs2-0, libosso1
(>= 2.23), libpango1.0-0, libprofile0, libtelepathy-glib0
         (>= 0.7.37), libtime0, libtinymail-1.0-0, libtinymail-camel-1.0-0,
libtinymail-gnomevfs-1.0-0, libtinymail-maemo-1.0-0,
         libtinymailui-1.0-0, libtinymailui-gtk-1.0-0, libxml2 (>= 2.6.32),
wpeditor0, maemo-launcher, libmodest-dbus-client (>= 3.2.0),
         modest-providers-data, modest-l10n-mr | modest-l10n-mr0,
osso-icons-theme, operator-wizard-settings, osso-countries-l10n-mr |
         osso-countries-l10n-mr0, microb-engine


Here is why CSSU gets broken after downgrading modest:

Nokia-N900:~# apt-get install modest=3.4.8+0m5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libcpaboutcssu
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  mp-fremantle-community-pr
The following packages will be DOWNGRADED:
  modest
0 upgraded, 0 newly installed, 1 downgraded, 1 to remove and 6 not upgraded.
Need to get 435kB of archives.
After this operation, 250kB of additional disk space will be used.
Do you want to continue [Y/n]? y


Trying to keep CSSU intact will cause upgrade of modest:

Nokia-N900:~# apt-get install mp-fremantle-community-pr
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  modest
The following NEW packages will be installed:
  mp-fremantle-community-pr
The following packages will be upgraded:
  modest
1 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 0B/405kB of archives.
After this operation, 250kB disk space will be freed.

Is there an upstream bugtracker for modest/libtinymail somewhere?
Comment 14 Christian Ratzenhofer 2012-02-17 14:08:44 UTC
(In reply to comment #13)
> I tried to update CSSU today (see below explanation only today) which reupdated
> modest to Version: 3.90.7-10 which doesn't solve this issue

Could you check again with current cssu modest (3.90.7-11)
Comment 15 Petr Vanek (reporter) 2012-02-17 15:14:10 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > I tried to update CSSU today (see below explanation only today) which reupdated
> > modest to Version: 3.90.7-10 which doesn't solve this issue
> 
> Could you check again with current cssu modest (3.90.7-11)

Just tested, issue still there... :(

Nokia-N900:~# aptitude update
...
Nokia-N900:~# aptitude install modest
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Reading extended state information       
Initializing package states... Done
Building tag database... Done      
The following packages will be upgraded:
  modest 
1 packages upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 422kB of archives. After unpacking 45,1kB will be freed.
Do you want to continue? [Y/n/?] 
Writing extended state information... Done
Get:1 http://repository.maemo.org fremantle/free modest 3.90.7-11 [422kB]
Fetched 422kB in 2s (189kB/s)  
(Reading database ... 47917 files and directories currently installed.)
Preparing to replace modest 3.4.8+0m5 (using .../modest_3.90.7-11_armel.deb)
...
Unpacking replacement modest ...
Setting up modest (3.90.7-11) ...
Reading package lists... Done             
Building dependency tree       
Reading state information... Done
Reading extended state information       
Initializing package states... Done
Building tag database... Done      

Current status: 9 updates [-1].

Nokia-N900:~# killall -9 modest

Nokia-N900:~# ps ax | grep modest
 4115 root      2088 S    grep modest 

hmmmmmm :(

apt-get install modest=3.4.8+0m5
Comment 16 Pali Rohár 2012-03-03 14:17:05 UTC
I can confirm same problem.
Comment 17 Peter Pichler 2013-06-24 20:36:34 UTC
I can confirm same problem too.

It seems that names containing a single accented character at the end (in my
case "á", like the OP's "Pavla Vlkovská") are dropped altogether.

Modest version 3.90.7-12.1, last updated 2013-05-13.
Comment 18 Pali Rohár 2014-05-18 23:08:26 UTC
I commited patch from
https://gitorious.org/modest/modest/commit/8ced6118ac86c3307ca676a5a705d6144e3349b7
to CSSU modest git repo.