I swear the Romanian bug is fixed; it was fixed 4.5 years ago!

by Michael S. Kaplan, published on 2010/12/13 07:01 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2010/12/13/10103924.aspx

In its latest incarnation, the mails that came in the other day, first this one:

Dear Michael Kaplan,

I am writing to this email address because I've seen it mentioned in a previous blog entry<http://blogs.msdn.com/b/michkap/archive/2010/11/30/10098113.aspx>
.

My question is mentioned in the subject of this email: *How to do accent insensitive comparison under Windows*?

My test case would be:
voila == voilá
garcon == garçon
kase == käse, or better kaese == käse
arsita == arșiță (Romanian s and t with comma bellow, and a with breve)
arsita == arşiţă (Romanian s and t with cedilla, and a with breve)
arşiţă == arșiță

I've hacked a MSDN C++ sample <http://pastebin.com/fG9APFDt>, but it doesn't quite work as expected.

After reading the documentation of Boost.Locale<http://cppcms.sourceforge.net/boost_locale/html/tutorial.html> I was left with the impression that C++ could work as expected (at least "garcon" should be equal with "garçon")

If it is possible in C++ how about the Romanian s and t with comma bellow characters? The C++ locale for Romanian on Windows is equivalent with ISO-8859-2.

I've been informed that Microsoft doesn't want to add support for ISO-8859-16 in their products, UTF-8 and Unicode being the future, but how can I use C++ to have "arsita" and "arșiță" as equal?

Thank you in advance.

Sincerely,
Cristian Adam

Dear Michael Kaplan,

I have found the CompareString Win32 function and I've made a test application <http://pastebin.com/6mvkpzxx>.

The results on Windows XP are:

---------------------------
Comparison
---------------------------
voila == voilá
garcon == garçon
kase == käse
kase > kaese
arsita == arşiţă
arsita > arșiță
arşiţă > arșiță

---------------------------
OK
---------------------------

CompareString on Windows XP doesn't recognize the s and t comma bellow for Romanian language!!!

Should I file a bug report through Microsoft Connect?

Sincerely,
Cristian Adam.

First of all, you do not need to use that weird way to contact me, use the Contact link on THIS page in THIS blog!

XP is not going to be updated to support the Romanian S and T with comma below letters.

There are updates to fonts to display them, you can use MSKLC to create keyboards for them.

No XP solution to this problem is going to materialize that will give the right answer for accent insensitive comparisons.

You can search on this site for the Romanian issue with these letters because I have talked about it often in the past.

No, it is not worth the time of putting in a Connect bug -- because it can't be fixed downlevel. The fix already exists, and has existed for over half a decade....

But with all that said, the real problem here is not the one I have been talking about, either here or in that other blog about Latvian.

The "problem" is that there are lots of people who still love XP, and that every day those people notice a problem that they want fixed.

Hell, I have one machine (a Dell Latitude D820) that I use for many things. It runs like crap on Vista, and Dell refuses to support with Windows 7 on it -- no drivers, occasional blue screens, etc. Though Vista is supported by Dell on it. So believe me, that one machine will be running XP for years to come.

But if your language is Latvian or Romanian, you have an even better reason to upgrade:

That one machine of mine that @#%&*! Dell won't support on Windows 7? It is doing specific tasks at home that involve neither language. And I have lower expectations because I know it is running something that was mostly developed a decade ago.

By the way, that XP machine is running IE8 (I am not part of the 45% of China that is using @#%&*! IE6).

But as I said earlier in this blog and many times in this Blog, XP is not going to get updated collation, or casing....

Actually, it does look like LINGUISTIC_IGNOREDIACRITIC has problems with Romanian.

Try:

wchar_t const szNoAccents[] = { 0x0061, 0x0072, 0x0073, 0x0069, 0x0074, 0x0061, 0x0000 };

wchar_t const szWithComma[] = { 0x0061, 0x0072, 0x0219, 0x0069, 0x021B, 0x0103, 0x0000 };

int result = CompareStringEx(

L"ro-RO",

LINGUISTIC_IGNOREDIACRITIC,

szNoAccents, -1,

szWithComma, -1,

NULL, NULL, 0

);

Result: CSTR_LESS_THAN, should be CSTR_EQUAL.

Tried it on Windows 7.

If you look at the Romanian data, we don't consider these to be diacritic differences -- the letters are given independant weight.

If you choose anything other than Romanian, you can get the comma below ignored....

Hmmm...

I agree that these are technically different, stand-alone characters.

But the lack of support was missing for such a long time (and the ANSI applications on non-Romanian systems did not allow even for the cedilla form) that a lot of people find it handy to just do the searches without "accents"

Very handy (try searching for "resita" in Google or Bing and you will find "Reşiţa", nice)

But we can't have it both ways if search/string compare and SORT use the same function -- otherwise the order will be wrong, and that is what the Romanian collation is built for. These letters must have primary weight (like the cedilla below characters used to), or dictionarioes will be wrong.

Until they add an EqualString function, that is. :-)

I agree think that sort and search are different beasts, and most people don't even grok that.

Adding an extra API would make it quite confusing.

But by default sorting in most languages will be messed up if one uses any kind of *_IGNORE* flag.