If you just don't think you can hold it (64-bit style!)
by Michael S. Kaplan, published on 2006/09/28 06:01 -07:00, original URI: http://blogs.msdn.com/michkap/archive/2006/09/28/774957.aspx
I can actually say again and again and again and again that although 64-bit keyboards are important to Microsoft and to the GIFT team and how there will be an update at some point that it has not happened just yet....
The truth is that sometimes, some people just don't want to wait.
I just got email from such a person yesterday, in fact! :-)
Igor Levicki is that person, and in his post (How to build keyboard layouts for Windows x64?) he describes his spluenking efforts to find the 64-bit support that was partially completed but not yet finished in the Microsoft Keyboard Layout Creator (and as he noticed we did not ship the cross-compilers since we weren't supporting the platforms yet anyway!).
Now of course the real support plan will include the real setup, and will also include the WOW64 dll as well (Mr Levicki's does not, which will cause some headaches for 32-bit applications running on 64-bit). Also the version of the tools we would have shipped would have worked okay with the original settings, meaning that Igor figured out the diffs needed for the later tools, as well.
Note to anyone from Microsoft who is reading this: someone may want to try and find out if Mr. Levicki might want to look into interviewing with Microsoft.
And note to Igor Levicki -- if we ever find ourselves in the same (or a similar) place, let me know. I'd love to buy you a drink and we maybe talk about some of the cooler aspects your efforts! :-)
This post brought to you by ♔ (U+2654, a.k.a. WHITE CHESS KING)
# Centaur on Thursday, September 28, 2006 10:35 AM:
That’s a pretty impressive bit of reverse engineering. You might want to include this into the MSKLC documentation, as an “unofficial guide”, until there is a universal 32/64-bit version.
# Michael S. Kaplan on Thursday, September 28, 2006 11:04 AM:
Hi Centaur,
The docs would not be updated until the tool is, and when that happens the functionality will be built in....
# Igor on Thursday, September 28, 2006 11:02 PM:
Michael,
I am glad you liked my article. Feel free to send me an email if you have some spare time and we can disscuss whatever details you want. I would like to hear about wow64.dll and those "headaches for 32-bit applications" if it is not classified of course.
About the possible interview, I certainly wouldn't mind.
I am a software engineer and I am sort of a low-level guy. I work with C, considerably less with C++, and quite a lot with x86 assembler. I am familiar with all SIMD extensions up to SSE4 and with 64-bit ABI and register calling convention.
Code optimization is usually the kind of work I do for hire over Internet:
http://www.intel.com/cd/ids/developer/asmo-na/eng/dc/code/languages/194751.htm
I also love reverse-engineering (you learned about that already) so disassembler and debugger are some of my favorite tools.
About that drink, I will remember your kind offer if we ever meet in person. In the meantime, have fun reading this article:
http://levicki.blogspot.com/2006/01/annoying-windows-xp-feature-aka-make.html
Have a nice day.
# Michael S. Kaplan on Friday, September 29, 2006 12:30 AM:
Ah, the WOW64 problem is briefly discussed here in
this post -- it is the need for keyboard DLLs (just like kernel32.dll and a few select other binaries) that need a 64-bit pointer representation in a 32-bit binary to work properly....
# igor_levicki on Friday, September 29, 2006 12:45 AM:
Ah but I solved it unintentionaly!!!
I compiled a 32-bit DLL and put it there but when I was disassembling one of the shipped layouts to find the correct base address for linker I noticed that the exported function differs by one instruction:
.data:5FFF2188 ; __stdcall KbdLayerDescriptor()
.data:5FFF2188 public _KbdLayerDescriptor@0
.data:5FFF2188 _KbdLayerDescriptor@0 proc near
.data:5FFF2188 mov eax, offset off_5FFF2090
.data:5FFF218D cdq
.data:5FFF218E retn
.data:5FFF218E _KbdLayerDescriptor@0 endp
It was the CDQ (Convert Double to Quad) and I figured out that mine should have it too so I just patched the DLL I made with a hex editor and fixed its checksum by a tool I wrote for that purpose some time ago.
In the end it would probably work without patching depending on the contents of the EDX register on entry to exported function. CDQ just sign extends EAX into EDX:EAX thus returning 64-bit pointer whose high part is always zero in this case.
I guess that now I will have to update the post.
# Michael S. Kaplan on Friday, September 29, 2006 2:32 AM:
That does indeed take care of the issue. :-)
Though of course it will not scale as a generic solution, at least not as effectively as doing it as part of the compilation/link process....
You can actually see the same types of differences (only many more of them) if you compare system32\kernel32.dll with syswow64\kernel32.dll. Thankfully keyboards are much simpler to deal with, though.
# igor_levicki on Friday, September 29, 2006 11:47 AM:
Well, because I hate to leave things half-finished I dug out the right way to deal with that issue during compilation :)
It is just a matter of compiling using 32-bit compiler with -DBUILD_WOW6432.
So now I have the complete solution.
# Michael S. Kaplan on Friday, September 29, 2006 1:25 PM:
Yep, that's the change. Perfect!
# igor_levicki on Friday, September 29, 2006 1:46 PM:
# Michael S. Kaplan on Friday, September 29, 2006 1:54 PM:
That last part likely needs the MSKLC update itself, I think (the logic in the setup custom action that adds the reg. keys and such). But the hard part of the dev. end of this is covered in your solution....
The setup solution is surmountable but takes a little thought (building a setup that can install both 64-bit and 32-bit components is a bit of a challenge, but it is not impossible to do!).
# igor_levicki on Friday, September 29, 2006 2:21 PM:
I know how to disable System folder redirection using API from 32-bit APP so I could even write a setup but I don't know which keys I need to add (depends on the layout language id I guess).
# Michael S. Kaplan on Friday, September 29, 2006 2:33 PM:
I was actually thinking more about doing it in the Windows Installer... creating a single setup that would include all for keyboard layout DLLs, install the correct ones based on platform and add the reg keys. :-)
# igor_levicki on Friday, September 29, 2006 2:42 PM:
Well, I would leave that up to you Michael. I haven't dealt with Windows Installer yet (although I am familiar with Orca).
# Michael S. Kaplan on Sunday, October 01, 2006 12:43 AM:
And Igor, if you were looking for info on why defining BUILD_WOW6432 changes what it does, you can see the following at the top of kbd.h:
#if defined(BUILD_WOW6432)
#define KBD_LONG_POINTER __ptr64
#else
#define KBD_LONG_POINTER
#endif
And it all comes together.... :-)
# igor_levicki on Sunday, October 01, 2006 11:55 AM:
Yes and you could have read that in my post two days ago ;-)
# Michael S. Kaplan on Sunday, October 01, 2006 3:22 PM:
Heh, that is true -- I had not read the update yet. You did have the info there already! :-)
# igor_levicki on Sunday, October 01, 2006 8:47 PM:
There is one more thing here completely unrelated to the layouts but nevertheless very interesting.
As far as I know any 32-bit user mode code usually doesn't get base address above 2GB so I guess it would it be safe to assume (for those thunking DLLs in SYSWOW64 at least) that the __ptr64 is unsigned?
It means that for user mode code simple XOR EDX, EDX would suffice instead of a bit slower CDQ which does sign instead of zero extension to 64 bits.
Anyway, memory pointers should be treated as unsigned, right? I believe that maybe Kevin Frei and his team would like to think about this for a while. If not, then at least I would like to find out the logic behind it. Can't help myself with this inquiring mind :)
# Michael S. Kaplan on Sunday, October 01, 2006 8:54 PM:
Another excellent question that I remember asking back in the day. :-)
Actually, these DLLs are loaded in kernel mode by the userk subsystem, so they can indeed end up with base addresses above the user mode boundary....
In theory, since all they do is export a function that takes a struct, all of this could be re-archirected to be handled in a very different way that would not have such semi-unique requirements. But that would break backcompat with all existing layouts (which has never been explicitly promised but without a convincing reason I doubt they would ever be willing to break, especially with MSKLC out there).
# igor_levicki on Sunday, October 01, 2006 9:07 PM:
Hmm...
But there is no point in sign-extending a pointer?
If you have 0x8123ABCD by sign-extending it you will get 0xFFFFFFFF8123ABCD which is not exactly a valid address anywhere, right?
# Michael S. Kaplan on Sunday, October 01, 2006 9:17 PM:
Well, we have to assume that the code on both sides knows how to handle the sign extension as needed.
I wish that they had never made the pointer type signed, myself. It makes everything much more complicated....
# Michael S. Kaplan on Sunday, October 01, 2006 9:24 PM:
# igor_levicki on Sunday, October 01, 2006 9:26 PM:
Having signed pointers is downright crazy and I must admit that I can't understand a logic (if any?) behind that.
I really must remember to ask Kevin why __ptr64 is signed.
# Michael S. Kaplan on Sunday, October 01, 2006 9:34 PM:
Probably to be consistent with the 32 bit pointer (which is also signed).
Which leads to the real question -- why did they make the 32-bit pointer signed? That just leads to later insanity!
# igor_levicki on Sunday, October 01, 2006 9:37 PM:
I checked that out. It is crazy and even though it is not a bug it is a bad design which fosters errors.
For example, if you have some API which returns -1 on failure and other valid return values get sign extended all applications which did this:
rv = SomeAPI();
if (rv < 0) { // instead of explicitly comparing rv to -1
fail();
} else {
pass();
}
will start to fail randomly if they get sign extended values.
# Michael S. Kaplan on Sunday, October 01, 2006 9:41 PM:
Agree++ on that. Not to mention that sign extension is not well understood enough to doc in a non-confusing way for most developers -- so there is no easy way to educate people out of it....
# igor_levicki on Sunday, October 01, 2006 9:49 PM:
# igor_levicki on Sunday, October 01, 2006 10:30 PM:
# Michael S. Kaplan on Sunday, October 01, 2006 11:48 PM:
Hey, no argument from me. :-)
It seems slightly worse from managed code (to me) mainly because it is even less well-documented there. But it is easy to end up with bugs all the way around, I agree.
# igor_levicki on Monday, October 02, 2006 12:57 AM:
Sorry for "arguing" with you about this when you said you agree but I just can't believe that someone decided to make pointers signed instead of changing HWND_TOPMOST to something other than -1. That is what I would call a serious value system disorder.
# Michael S. Kaplan on Monday, October 02, 2006 1:05 AM:
I understand, and I agree with you. But there isn't much I personally can do about it. :-)
FWIW, I'm also not 100% convinced that this was the only reason. I think it was an example of a more widespread tendency that predated 64-bit computing. It is still a bad thing in my opinion, but oit's a bit late to do anything about it now.
# Igor on Monday, October 02, 2006 10:39 AM:
Michael,
When we are at the subject of 64 bits, I have few questions because documentation is vague:
Wow64DisableWow64FsRedirection()
1. Does this API really disable redirection just for the calling thread or for the process (or for the calling threads and all the threads created after the call to the above API)?
2. Does calling the above API affect LoadLibrary() in any way?
This is really important to know.
# Michael S. Kaplan on Monday, October 02, 2006 1:09 PM:
Not my area, but as far as I know Wow64DisableWow64FsRedirection is per thread only.
My understanding is that it will affect all file system stuff run in the thread, though I haven't tested it extensively....
# Igor on Tuesday, October 03, 2006 11:48 AM:
Houston, we have a problem.
I have tested it and here is what happens.
If you have an application such as file manager which uses plugins and it is 32-bit and it doesn't have any benefits from going 64-bit, it would be nice to turn the filesystem redirection off at the process start so that user can manipulate files in real system32 folder and not in syswow64.
Now suppose I load the plugin dll from the file manager and then call an exported function from the plugin.
If the plugin has a code snippet like this one:
HMODULE my_dll;
TCHAR szPath[MAX_PATH];
GetSystemDirectory(szPath, MAX_PATH);
_tcscat(szPath, _T("\\my_dll.dll"));
my_dll = LoadLibrary(szPath);
IT WILL FAIL!!!
Why? Because GetSystemDirectory() doesn't honor the redirection state.
Now you can argue that:
1. Redirection is supposed to be restored immediately after you obtain a handle
2. GetSystemDirectory() is obsolete
But in reality:
1. It would be too complicated to turn it on and off all the time before you call any plugin.
2. Is not in the code you have any influence over.
So, I suggest a fix for this:
Make GetSystemDirectory() honor redirection state:
If in 64-bit process:
- always return \windows\System32
If in 32-bit process, redirection on:
- return \windows\System32
If in 32-bit process, redirection off:
- return \windows\Syswow64
# Michael S. Kaplan on Tuesday, October 03, 2006 12:09 PM:
Of course, we are getting more than a little bit off-topic, but anyone who is writing a 32-bit file manager that they expect to run on 64-bit windows with naught but minimal changes actually knows exactly where the source of problems is, if they put their mind to it. :-)
But I am not an expert in 64-bit programming issues, so its just my [limited] opinion....
# Igor on Thursday, October 05, 2006 1:25 AM:
[EDITED to add some line breaks to the continuous paragraph submitted --michkap]
Sorry for that offtopci, I have contacted Raymond and he promised to forward this to the right team. I just hope they won't turn the blind eye on it.
There are five obvious solutions I could come up on such a short notice:
- Turn redirection off, get a handle to system32, then turn it back on. Use the handle as needed later. But what if the \Windows is on a network share and the connection is lost?
- Call Wow64DisableWow64FsRedirection() before each plugin call and re-enable it afterwards. Even though Microsoft originally intended the API to be used that way (e.g. disable/get handle/enable) that is not good solution performance wise. What if you need to call plugin function numerous times for example to enumerate files in a large folder? You will waste cycles by turning redirection on and off. You will also have to keep track from which thread you called it and if user attempts to change into system32 while plugin is doing say background archiving he may end up either in system32 or in syswow64.
- Use separate thread to call plugins. Although better, it is not practical to implement and it may require quite a lot of synchronisation code to be added.
- Ask plugin authors to remove GetSystemDirectory references. Not practical and in some cases not possible.
- Have Microsoft fix the GetSystemDirectory() to honor redirection state and when redirection is _off_ return \Windows\SysWow64 instead of \windows\System32.
IMO #5 would be the best and even righteous solution because it seems like they forgot about GetSystemDirectory() when they invented redirection. I believe someone should be thanking me for pointing that out ;)
# Michael S. Kaplan on Thursday, October 05, 2006 1:38 AM:
I certainly wouldn't mind forwarding it on myself, but I honestly doubt they could change the behavior without breaking apps that are relying on what happens now. Once you have established a function's behavior, changing it is guaranteed to break someone....
# Igor on Thursday, October 05, 2006 2:20 PM:
>EDITED to add some line breaks to the continuous paragraph submitted --michkap
Not my fault Michael, I use paragraph breaks but this box is ignoring them somehow.
# Partybwal on Wednesday, October 18, 2006 11:05 AM:
So, when can I expect to be able to use my custom keyboard layout in x64? It's a variant of Dvorak with swedish letters.
I could fix it myself, if only MSKLC would build 64-bit compatible layouts. But since x64 ships with lots of layouts, howcome there is no such tool, or even just a converter? How did MS create them?
# Michael S. Kaplan on Wednesday, October 18, 2006 11:31 AM:
Hi Partybawl,
I have already the answered the 'when' question in this blog more times than one could really count, at least as much as I can. It is important, and as Igor proved the underlying tools can and do support the notion, but the integrated experience is not available yet....
Please consider a
donation to keep this archive running, maintained and free of advertising.
Donate €20 or more to receive an offline copy of the whole archive including all images.
referenced by
go to newer or older post, or back to index or month or day