by Michael S. Kaplan, published on 2007/03/20 03:01 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2007/03/20/1917161.aspx
So when I first posted Double Secret ANSI, part 1 (Somewhere between ANSI and Unicode), I was only a little bit surprised when regular reader Mihai knew where I was heading....
(Mihai works for a company that produces some of those Double Secret ANSI applications I was talking about!)
You see, we kind of broke some of those Double Secret ANSI applications in Vista.
I was a bit slow posting the follow-up as I waiting to see what the plans were. And Shawn has just posted about that in Some Keyboards fail with ANSI applications on Windows Vista RTM, and about how a fix is being planned.
This is very cool, and I am sure we'll both be posting about the fix when it is available. It is a pretty bad bug.
How the bug happens is interesting though, so I thought I'd talk about that for a bit....
You see, at the point where a keyboard layout is loaded, the kernel mode keyboarding code thunks to a function that uses TranslateCharsetInfo and the LANGID that is inside of the KLID (Keyboard Layout Identifier) of the keyboard. It gets the lsCsbDefault piece of the LOCALESIGNATURE and uses it for three purposes:
It is that last bullet point that is where the power of double secret ANSI applications comes from. :-)
And it is that bug in a bunch of the LOCALESIGNATURE data in Vista that leads to the problems in those same applications for some keyboards :-(
Which is not to say that there aren't analogous bugs in the LOCALESIGNATURE data in prior versions.
Because there are. Remember this post?
It is just that the values are much more broken in Vista. Did I mention that I am glad we are working to get this fixed?
Though to be honest, in a future version I am going to try to look into simplifying this work a bit. I mean, rather than having user/k thunk to uer mode to call gdi32.dll so it can call kernel32.dll to retrieve what was originally loaded in kernel mode anyway to retrieve what has (historically speaking) been the single least reliable piece of locale data we ship, there are several links there that are simply not needed.
Talk about needless dependencies!
Especially when you consider the fact that it is incredibly difficult to ever retrieve the code page value that is being used in the keyboarding code at any later point in the use of the keyboard layout. I mean, what's a nice, honest double plus ANSI application supposed to do? :-)
There is probably a whole bunch more here that I can talk about another time, so if the topic is interesting to you then stay tuned....
About the only mildly reassuring point here is that most apps that don't support Unicode never do much beyond the default system code page. Which is not saying much.
For all of those double plus ANSI applications, sorry about that. This was not some kind of stunt to get people to write Unicode applications.
1 - More on the weirdnesses here beyond these ones another day.
This post brought to you by А (U+0410, a.k.a. CYRILLIC CAPITAL LETTER A)
# ANSI is the bug on 20 Mar 2007 4:27 AM:
It's always nice to fix bugs.
However, for this one in particular I almost feel it serves them (ANSI developers) right. While initially painful, I think a lot of good could come from depreciating ANSI in a very visual way. Whenever an ANSI application is executed, put up a really annoying message box saying something like "This is an ANSI application. Some functionality will be reduced. Strongly suggest to the manufacturer to get in touch with the last decade."
Of course some applications can not be updated anymore. So leave the existing support. I think the with a significant user base complaining about this many companies would eventually upgrade to Unicode.
# Michael S. Kaplan on 20 Mar 2007 7:16 AM:
Well, it's one thing to make decisions that make a feature like "ANSI" support less desirable (e.g. Unicode-only locales, new functions that are Unicode only), but it is quite another to literally break something that used to work. That's just a line that I can't get comfortable crossing....
Yuhong Bao on 31 Mar 2012 2:44 PM:
"Though to be honest, in a future version I am going to try to look into simplifying this work a bit. I mean, rather than having user/k thunk to uer mode to call gdi32.dll so it can call kernel32.dll to retrieve what was originally loaded in kernel mode anyway to retrieve what has (historically speaking) been the single least reliable piece of locale data we ship, there are several links there that are simply not needed."
There was a use-after-free in this code (see Keyboard Layout Object Use-After-Free (CVE-2011-1241)):
referenced by
2012/01/16 I'm reasonably certain that those who disagree with me here are wrong!
2011/04/04 UXTheme in the non-Unicode world isn't "Double Secret ANSI"
2010/08/25 Yet another cost to not supporting Unicode?
2008/10/15 What is the freaking point of LOCALE_USE_CP_ACP?
2008/08/15 Yet another time that UTF-8 can't be the ACP
2008/06/19 How do[es what] the common controls [call ]convert between ANSI and Unicode?
2007/11/21 Abobe knows, but someone needs to tell Skype....
2007/10/18 Trying to get people to use Unicode? Lock and load, baby!
2007/08/13 The font cold war?
2007/06/25 Sometimes when you say 'the fix is in' you mean it in a good way
2007/05/07 The Unicode train? It left the station....