[g11n-ko-discuss] Re: [i18n-discuss] [Fwd: Re: ksh93 i18n problems on Solaris ? / was:Re: [ksh93-integration-discuss] comments on ksh93 migration plan]

[Ienup Sung]

Roland Mainz wrote:
>>I also briefly tested the ksh with en_US.UTF-8 locale and found the result
>>a bit of mixed bag. I was able to input most of
> 
> 
> What do you mean with "most of" ? Were there any failures with
> characters in these languages?

Hello Roland,

I have not tested input of all possible CJK and such languages but briefly
typed in some notable characters. And so my test wasn't covering all possible
characters or byte patterns in them. I hoped to make sure to inform you
about that.


>>CJK, Arabic, Greek, Cyrillic
>>characters and also view all Unicode characters (for that, by doing
>>"more /usr/pub/UTF-8") but I wasn't able to input some accented characters
>>such as ä, ç, and so on.
> 
> 
> Which terminal emulator did you use ? The Gnome terminal ? Does it work
> with other terminal emulators (e.g. "dtterm" or KDE's "konsole") ?
> How does normal Solaris ksh behave for such characters (or bash3) ?
> BTW: Did you enable "emacs" or "vi" editing mode (e.g. % set -o emacs #)
> before entering the chars ?

I used gnome-terminal, dtterm, and xterm at Solaris 10. If I use /usr/bin/csh
or /usr/bin/ksh, then, I don't have any problem. With the ksh93, I wasn't
able to input some of the accented characters as I noted:

	ä (0xc3 0xa4, a with diaeresis)
	ç (0xc3 0xa7, c with cedilla)
	¡ (0xc2 0xa1, inverted !)

I haven't tested all Latin-1 characters but it appears I cannot input
Latin-1 characters of Unicode (U+0080 - U+00FF). I also just re-did some of
the tests that I did yesterday and attached the screen shots in this email.
Hope everyone doesn't mind me sending these JPEG files (abuot 110KB) in
this email...

I didn't use any editing software but just the shell as shown in the log
file I sent out.


>>I was, however, able to input Å,, Å?, and so on.
> 
> 
> Weired. Any idea what is going wrong here (except the list of "usual
> suspects" listed above) ?

At this point and personally I don't have any clue since I've not looked at
the source code.

Things that are sure are there appears at least two problems:

1. I cannot input Latin-1 characters of UTF-8 locales.
2. Error messages from the ksh93 appears went through either not 8-bit clean
    or some form of a conversion.

As I mentioned though, I was having no problem what so ever concatenate or
do "more" on the entire Unicode characters especially for all printable
characters in the range of U+0000 and U+10FFFF.

I'd like to ask other language experts also try it out in your favorite
locales and report back. Once some more data are collected, I think I could
or anyone in this group could debug and also code review the patches and
find out why there are problems and suggest possible fix if it is a bug in
the ksh93 source.


> BTW: You build the 64bit version on SPARC... did you get the same issues
> with accented characters when you run the 32bit version ?

The x86 version is an i386 binary (32-bit). As I reported, yes, I saw the same
problems with the ksh-i386 binary.


>>Would I also
>>need to add -Bstatic before linking with the -lcmd?
> 
> 
> The libcmd issue is slightly different. ksh93 has it's libcmd and
> Solaris has it's own libcmd.
> - For "buildksh93.ksh" I am simply linking libcmd.a (due lack of a
> shared library this should work (or not... I really didn't check this
> but I think you're right with the -Bstatic thing... ;-/ ) - and for the
> non-shared version of ksh93 (e.g. without seperate libast, libdll,
> libshell etc. libraries) it works definately as this is the normal way
> how the kornshell.com people build it).
> - For OS/Net I have a merged version of libcmd which contains both
> versions (see
> http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2006-March/000172.html).
> IMO it's IMO better to ship the non-shared version for testing purposes
> which does not depend on all the other libraries...

Apparently the ksh built with your build script recorded and thinks that it
needs shared object libraries which I think not good unless the libcmd between
the two versions are 100% identical. Please let me know if the two versions of
libraries are 100% compatible and shouldn't matter which one the ksh93 binaries
use or not. (Based on your answer, I may need to rebuild the ksh93 binaries
before people do the test.)


> 
> BTW: Does Sun have any automated i18n test scripts for shells ?

I do not know. I hope people in this mailing list can answer that question.

Hi Ales, would you know or know someone who can answer on this question please?

Ienup
-------------- next part --------------
???? ?? ??? ?? ?? ?????...
??: ksh-test-gnome-terminal.jpg
??: image/jpeg
??: 46762 bytes
??: ??? ? ????.
Url : http://oss-beta1.opensolaris.org/pipermail/g11n-ko-discuss/attachments/20060413/aa7a45be/attachment.jpg 
-------------- next part --------------
???? ?? ??? ?? ?? ?????...
??: ksh-test-dtterm.jpg
??: image/jpeg
??: 67413 bytes
??: ??? ? ????.
Url : http://oss-beta1.opensolaris.org/pipermail/g11n-ko-discuss/attachments/20060413/aa7a45be/attachment-0001.jpg