The evolving Story of Locale Support, part 4 (working beyond one's bugs, and the case for an MSKLC update)

by Michael S. Kaplan, published on 2011/11/04 07:01 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2011/11/04/10233999.aspx

Previous blogs from this series:

• part 0 (The introduction)
• part 1 (Some people don't want to double-d's)
• part 2 (raising the roof on keyboards) 
• part 3 (working beyond one's bugs, the setup) 

Now the net effect of Part 3 was a whole lot of nothing -- I mean, I pointed out a bug, I claimed to be the person who essentially introduced it, and I apologized for it. I didn't explain what was going on or even give hints as to what the next step would be.

Sorry about that....

Anyway, we'll start over.

That mail Frank Grießhammer of Adobe sent me?

Michael, Maybe you remember – I asked you some questions once before – most of which I could actually deal with myself.

I have been creating a bunch of keyboard layouts for Windows and Mac, to be shipped with the Adobe Pi fonts. The motivation behind this project is providing the user with a method to ‘key’ their symbol glyphs. The layouts were created for the Mac first, the XML was converted to a .klc file using a Python script, the final layout being compiled with MS Keyboard Layout Creator. In the process, I made the following observation, which might be interesting material for your blog:

As of Unicode 5, Unicode values with 5 digits exist. Both Mac and Windows have (at least) four possible shift states for keyboards, I kept them unified across the platforms: ‘alt’ on the Mac would become ‘altGr’ on Windows (e.g. the right ‘alt’-key).

The observation I made has to do with the altGr/altGr+Shift states: If there is a 5-digit unicode value mapped to a key in either of those states, the respective key just won’t return anything on Windows. I filed a bug with your colleagues, and in a long email conversation we came down to the point that unicode support in the OS is – to say the least – leaving a lot to desire. It’s broken. We could rule out MSKLC being the culprit, it really came down to the OS. By the way: I tested my example layout on a developer version of Windows 8 as well, and the same problem exists.

Now here’s my question: Do you maybe have an explanation for that? I see your latest post is about keyboards, so that might fit in just nicely.

Best greetings,
Frank Grießhammer
Type Design & Font Production
Adobe Systems

I could make some minor corrections -- like supplementary characters were first introduced in Unicode 3.1, not 5.0. But that's not the point.

I could point out that as bad as Microsoft may be for not properly supporting a Unicode 5.03.1 feature, but Adobe was not properly supporting a Unicode 2.0 feature (ref: Beauty isn't only glyph deep), but that's not the point either. :-)

The point is one of the main feature additions in KBDUTOOL.EXE is the ability to define supplementary characters in the LAYOUT table of the .KLC file, which the tool then converts to surrogate pairs and adds them to the LIGATURE table. Which is where supplementary characters work - by defining surrogate pairs in the LIGATURE table.

Here is where the bug comes in.

When the .KLC file defines supplementary characters in the LAYOUT table in any state other than the BASE or SHIFT states, the supplementary characters are not properly converted to surrogate pairs. So those keys won't work properly.

This is the bug that Frank was reporting.

Now MSKLC.EXE does not have UI for a ligature table -- it just lets you define up to four UTF-16 code units on a key and any time 2, 3, or 4 are defined on a key, it adds them to the LIGATURE table.

Now before you decide to just convert the supplementary characters to surrogate pairs yourself, there is an MSKLC feature that automatically converts surrogate pairs to supplementary characters.

So you can't ever save the file in MSKLC.EXE, because it is designed to do something that exacerbates the problem in KBDUTOOL.EXE.

The only way to define these characters is to calculate the surrogate pairs and add them to the LIGATURE table directly, and then run KBDUTOOL.EXE from the command line to generate the various layout DLLs for the different platform architectures.

This is pretty awful, and I'm a little embarrassed no one found this bug.

I'm even more embarrassed that I never found this bug.

For Frank (who is generating his .KLC files from a Python script, the problem is slightly less onerous to deal with, though it is still ugly.

For a look the syntax, you can create some actual LIGATURE entries and see the syntax MSKLC.EXE uses for the table:

SHIFTSTATE

0 //Column 4
1 //Column 5 : Shft
2 //Column 6 :       Ctrl
6 //Column 7 :       Ctrl Alt
7 //Column 8 : Shft  Ctrl Alt

LAYOUT  ;an extra @ at the end is a dead key

//SC VK_  Cap 0 1 2 6 7
//-- ----  ---- ---- ---- ---- ---- ----

29    OEM_3   0  %%   %%   -1  %%  %%  // <null>, <null>, <none>, <null>, <null>
39    SPACE   0  0020 0020 -1  -1  -1  // SPACE, SPACE, <none>, <none>, <none>
53    DECIMAL 0  002e 002e -1  -1  -1  // FULL STOP, FULL STOP, , ,

LIGATURE

//VK_ Mod# Char0 Char1 Char2 Char3
//----  ---- ---- ---- ---- ----

OEM_3  0 0041 0041  // LATIN CAPITAL LETTER A + LATIN CAPITAL LETTER A
OEM_3  1 0042 0042  // LATIN CAPITAL LETTER B + LATIN CAPITAL LETTER B
OEM_3  3 0043 0043  // LATIN CAPITAL LETTER C + LATIN CAPITAL LETTER C
OEM_3  4 0044 0044  // LATIN CAPITAL LETTER D + LATIN CAPITAL LETTER D

 Obviously only the ALTGR and SHIFT+ALTGR states will need this done, but I defined a single ligature in all four states so you can see the syntax.

For the syntax of KBDUTOOL.EXE, you can see other blogs I've done before like In case you have a yen to extend your keyboard (or at least want a yen?).

For the record, I consider this a must fix bug in MSKLC, or more specifically the KBDUTOOL.EXE binary that is within the MSKLC package.

In fact, I think between this bug and the need to remove enough of the IA64 support to shrink the download and created setup package size (discussed in Still with the Itanium?), and the bug discussed in A picture that *still* can't be easily described with words, someone really ought to do a little dev work and release a tiny update -- MSKLC 1.5!

Now in the interests of turning this blog post into a mini "IStartedSomething.com" type issue, does anyone agree with me that these three bugs are worh fixing? Yes they can all be worked around, but they all suck IMNSHO.

Okay, and now with that said this series is officially off track (assuming I am not given the job to fix MSKLC!), so in the next part, I'll return to topic....

Doug Ewell on 4 Nov 2011 7:22 AM:

Michael S. Kaplan on 4 Nov 2011 8:27 AM:

Michael S. Kaplan on 4 Nov 2011 8:57 AM:

Van on 5 Nov 2011 1:55 AM:

Michael S. Kaplan on 5 Nov 2011 12:45 PM:

Van on 5 Nov 2011 10:48 PM:

Michael S. Kaplan on 6 Nov 2011 12:45 AM:

Andrew_Cunningham on 6 Nov 2011 3:41 PM:

Andrew West on 7 Nov 2011 7:06 AM:

Michael S. Kaplan on 7 Nov 2011 7:21 AM:

Matthew Slyman on 16 Apr 2013 11:09 PM:

Matthew Slyman on 16 Apr 2013 11:28 PM:

referenced by

go to newer or older post, or back to index or month or day