Bug 2918 - Shell line editing doesn't support UTF-8, non-ASCII letters followed by Ctrl+a in XTerminal move cursor too far to the left
: Shell line editing doesn't support UTF-8, non-ASCII letters followed by Ctrl+...
Status: RESOLVED FIXED
Product: Core
Busybox
: 5.0/(2.2009.51-1)
: All Maemo
: Low minor with 3 votes (vote)
: Harmattan
Assigned To: unassigned
: busybox-bugs
: https://bugs.busybox.net/show_bug.cgi...
: upstream
:
:
  Show dependency tree
 
Reported: 2008-02-10 18:37 UTC by Koos Vriezen
Modified: 2010-05-05 02:48 UTC (History)
6 users (show)

See Also:


Attachments
Problematic chars (82.83 KB, image/png)
2008-08-19 05:56 UTC, Alsor Zhou
Details
Problematic scene while using backspace to delete the special chars (93.60 KB, image/png)
2008-08-19 05:57 UTC, Alsor Zhou
Details


Note

You need to log in before you can comment on or make changes to this bug.


Description Koos Vriezen (reporter) 2008-02-10 18:37:11 UTC
SOFTWARE VERSION:
(Control Panel > General > About product)
2.2007.50-2

STEPS TO REPRODUCE THE PROBLEM:
Start the X terminal, type some text and then via the sliding keyboard Ctrl+a

EXPECTED OUTCOME:
The cursor goes to the first character on the line

ACTUAL OUTCOME:
It goes one character beyond the first. Also the line has a mysterious
character at the end, as it seems.

REPRODUCIBILITY:
(always/sometimes/once)
always
EXTRA SOFTWARE INSTALLED:

OTHER COMMENTS:

User-Agent:       Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.8
(like Gecko) (Debian)
Comment 1 Alsor Zhou 2008-08-14 06:30:53 UTC
>type some text and then via the sliding keyboard Ctrl+a
What kind of `text'? I can confirm this bug while typing `special character'
(non-ascii) in Xterm.

It's not existed in bash2.05b.0(1), sounds like the problem caused by SHELL
(ash).

Is that possible to mask (Non-ASCII) in ASH?

SOFTWARE VERSION:
(Control Panel > General > About product)
4.2008.23-14
Comment 2 Koos Vriezen (reporter) 2008-08-17 13:56:46 UTC
The original report was about any text. And this seem to work now in Diablo's
upgrade.
Alsor: Can you post the text symbols where this still occurs?
Comment 3 Alsor Zhou 2008-08-19 05:56:06 UTC
Created an attachment (id=884) [details]
Problematic chars
Comment 4 Alsor Zhou 2008-08-19 05:57:10 UTC
Created an attachment (id=885) [details]
Problematic scene while using backspace to delete the special chars
Comment 5 Alsor Zhou 2008-08-19 06:04:46 UTC
See attachment for details. AYK, the special h/w keys are not in standard PC
kbd layout. I think the xterm can not recognize the position and size.

There is a workaround, set the current encoding to "Latin (ISO 8859 - 1)",
ASCII ?.  (osso-xterm menu -> Tools -> Current Encoding -> Latin (ISO 8859 -
1))
http://www.utoronto.ca/webdocs/HTMLdocs/NewHTML/iso_table.html

Is that possible to set Latin encoding as the default method of OSSO-XTERM?
Comment 6 Alsor Zhou 2008-08-28 12:31:46 UTC
File bug to upstream
http://bugs.busybox.net/view.php?id=4784

It's a common bug of busybox
Comment 7 Andre Klapper maemo.org 2008-11-26 18:24:23 UTC
Does this only happen for the special h/w keys?
Comment 8 Alsor Zhou 2008-11-27 06:28:15 UTC
yes, the only way I knew to type in the european umlauts under osso-xterm
shell, is use those umlauts h/w key.  Not? 

seems to be a problem of cursor position and the edit line buffer pointer not
matched, actually, the busybox shell (msh) is ANSI, not UTF-8 fully compatible. 

That might be better to wait the upstream developer update the msh to support
UTF-8, which is on their schedule.

http://busybox.net/cgi-bin/viewcvs.cgi/trunk/busybox/TODO?rev=23494&view=markup
Comment 9 Eero Tamminen nokia 2008-12-01 14:41:21 UTC
So is this is a Busybox bug (not supporting UTF-8) or Xterm bug (no setting
Busybox supported char encoding)?   I think using other than UTF-8 would break
things for other software than Busybox, so I guess this would be Busybox
issue...
Comment 10 Andre Klapper maemo.org 2008-12-30 16:18:58 UTC
(In reply to comment #8)
> That might be better to wait the upstream developer update the msh to support
> UTF-8, which is on their schedule.
> http://busybox.net/cgi-bin/viewcvs.cgi/trunk/busybox/TODO?rev=23494&view=markup

This is now
http://sources.busybox.net/index.py/trunk/busybox/TODO?revision=23494&view=markup

Alsor, can you answer Eero's last question?
Comment 11 Alsor Zhou 2008-12-31 04:35:12 UTC
(In reply to comment #9)
> So is this is a Busybox bug (not supporting UTF-8) or Xterm bug (no setting
> Busybox supported char encoding)?   I think using other than UTF-8 would break
> things for other software than Busybox, so I guess this would be Busybox
> issue...

Agree. It's better to implement a UTF-8 shell than shrink the Xterm to be an
ANSI Term.
Comment 12 Eero Tamminen nokia 2008-12-31 10:29:23 UTC
Thanks, confirming, setting as minor (comment 5 has easy workaround) and moving
to Busybox.
Comment 13 Andre Klapper maemo.org 2009-01-13 18:21:50 UTC
> File bug to upstream
> http://bugs.busybox.net/view.php?id=4784

Busybox maintainers set up a new bug database and erased all existing bug
reports. Awesome.
Filed https://bugs.busybox.net/show_bug.cgi?id=41 upstream.
Comment 14 Andre Klapper maemo.org 2009-03-03 14:25:27 UTC
Forwarding comment from upstream:

"msh is not likely to be fixed in that regard, it abuses 7th bit in characters
for it's evil purposes.
There was no active work on msh in months. ash and hush are more active. It's
tentatively planned to improve hush to the point where it surpasses msh.
Of course, if someone will start hacking on msh and fixing its problems, it
will be gladly accepted."


Hence sounds a lot like WONTFIX.
Comment 15 Lucas Maneos 2009-03-03 22:28:02 UTC
There seems to be some confusion here.  Maemo ships ash, not msh.

Diablo:

CONFIG_FEATURE_SH_IS_ASH=y
# CONFIG_FEATURE_SH_IS_HUSH is not set
# CONFIG_FEATURE_SH_IS_LASH is not set
# CONFIG_FEATURE_SH_IS_MSH is not set
# CONFIG_FEATURE_SH_IS_NONE is not set
CONFIG_ASH=y

Fremantle:

CONFIG_FEATURE_SH_IS_ASH=y
# CONFIG_FEATURE_SH_IS_HUSH is not set
# CONFIG_FEATURE_SH_IS_MSH is not set
# CONFIG_FEATURE_SH_IS_NONE is not set
CONFIG_ASH=y
Comment 16 Lucas Maneos 2009-03-04 10:57:39 UTC
Furthermore, the issue is in libbb/lineedit.c which is linked into all four sh
variants (and fdisk, although maemo doesn't ship that and it doesn't need to
handle non-ASCII input anyway).  Fix one, fix all.
Comment 17 Lucas Maneos 2009-03-06 12:31:19 UTC
FYI, this is from the busybox TODO file:
> The low hanging fruit is UTF-8 character set support.  We should do this.
> (Vodz pointed out the shell's cmdedit as needing work here.  What else?)

and there is some work along those lines in ftp://ftp.simtreas.ru/pub/my/bb/
(although it's based on an ancient bb version).
Comment 18 Andre Klapper maemo.org 2009-07-16 15:18:37 UTC
This is now fixed upstream in 1.15.x.
As Fremantle ships 1.10.2 it's unlikely to get this in for Fremantle.
Comment 19 Lucas Maneos 2010-01-03 07:04:02 UTC
*** Bug 7592 has been marked as a duplicate of this bug. ***
Comment 20 Vincent Lefevre 2010-01-03 12:09:17 UTC
The hardware should be changed to "All" since it also affects the N900, and the
OS should be changed to Maemo.
Comment 21 Lucas Maneos 2010-01-03 12:59:49 UTC
True, thanks for noticing.  Note also that the comment 5 workaround no longer
applies.

Andre/Eero: I assume this is WONTFIX for Fremantle, but do we know what busybox
version is going into Harmattan?  If it's >= 1.15.0 we can set TM, if not is it
worth trying to backport the fix?
Comment 22 Andre Klapper maemo.org 2010-01-04 18:40:36 UTC
Maemo6/Harmattan will definitely use Busybox >= 1.15 - just checked it myself.
Comment 23 Andre Klapper maemo.org 2010-05-05 02:48:43 UTC
Closing as FIXED in Harmattan.