by Michael S. Kaplan, published on 2005/09/12 03:31 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2005/09/12/463560.aspx
Back in February, I talked about theory vs. practice for Korean text collation, comparing Microsoft's implementation of old Hangul versus the one specified by Unicode (which uses normalization to move between the Korean Jamo and the Korean Hangul syllables).
I thought I should maybe say a little more about what is happening here.
After all, I spent some time explaining that the effort is not made to give the two different Unicode Normalization forms identical weights. But I did not mention that the relative weight difference between any two strings in one Normalization form versus the other, so that the fact that the Form "C" character order
가 < 각 < 힢 < 힣
means that the Form "D" character order will also be the same.
가 < 각 < 힢 < 힣
1100 1161 < 1100 1161 11a8 < 1112 1175 11c1 < 1112 1175 11c2
Now as I said the actual absolute weights will be different if you are looking at the sort keys, but this is okay since in practice the two forms are not really compared against each other. The fact that the relative weights will be the same is good enough for the real world data that would exist for Korean customers.
(The typographic support for Korean Jamo is not yet there anyway!)
By the way, for those who need it to work for them, the Unicode solution will work as well! As I discussed in the Mitigation tools for IDN security problems post, the Unicode normalization functions that have been added to the NLS API have been provided for customers running downlevel of Vista. Simply install the tools and both the NormalizeString and IsNormalizedString funtions will be available to use. And since they are completely conformant with Unicode 4.1, you can use them to convert strings if you truly need to do so.
For those of you using managed code, the .NET Framework 2.0 (a.k.a. Whidbey) also has the Unicode Normalization functionality, in the System.String.Normalize and System.String.IsNormalized methods. They are also completely conformant to the Unicode 4.1 definition.
This post brought to you by "ᇂ" (U+11c2, a.k.a. HANGUL JONGSEONG HIEUH)
(a character that looks a lot like HANGUL JUNGSEONG I (U+1175), though since one is a trailing jamo and the other is a vowel jamo, they will never sort as exactly the same!)
# kurakuraninja on 14 Sep 2005 10:16 AM:
referenced by