The so-called Ultimate Keyboard

by Michael S. Kaplan, published on 2005/07/20 22:14 -07:00, original URI: http://blogs.msdn.com/michkap/archive/2005/07/20/441227.aspx


It all started on Friday, July 15th, at 2:43am. I received the following mail:

Hi Michael

I read your blog everyday, and I really enjoy it. Normally I would never contact someone who's blog I read (nor any other celebrity) but I saw this and I thought "I wonder if Michael Kaplan has seen this" I didn't act on it for 24hrs but now I still think you would like to know about it. Anyway enough waffling here is the link

http://www.artlebedev.com/portfolio/optimus/

If you have already seen this or you don't care then I'm sorry to have wasted your time.

Andy

Well, I'm not a celebrity or anything like that. I had not yet at that point seen the info on this keyboard.

Roughly 53 minutes later, someone had posted it to the Unicode List -- with the link and not much else.

I'll give a smattering of the feedback after that....

JC Helary asked:

And what about languages that work with syllable input ?

First letter input: the keyboard displays something like qwerty, second letter input: the keyboard changes to what is possible in combination with the first letter ?!?!?

JC Helary

ps: the concept itself is very interesting, also the fact that the keyboard was a Mac one.

And then Don Osborne stated:

... and all of this without footpedals or levers.

The implications are significant and not only for extended and non-Lestern scripts. Once you are no longer constrained by what is painted on the keys at the factory, a lot of possibilities are opened up in addition to facilitating multilingual use of any given keyboard.

Ultimately a post-QWERTY world? Well if the keyboard is not dedicated to one layout, even users of one language are not obliged to learn and stay with the legacy system. I.e., it could facilitate learning and use of alternative layouts such as Dvorak for English without requiring a hardware change away from the legacy layout.

Don Osborn
Bisharat.net

I weighed in at this point, myself, with a point I have mentioned here:

All of this implies that one cannot have a keyboard layout that does not match the letters painted on the keys. Which is of course false (although some people have trouble with the concept, even though they have no probem with shift states that may vary from those letters....),

To which Don replied:

Hi Michael. I agree the problem with the painted keys is not what's under them of course, but what one is in the user's head (or isn't, or can't be kept in it...). Remembering which key was reassigned or what such and such key means with Alt/Ctrl, etc., may be easy when it's your own setup. But if you have more than one keyboard, or one with a lot of modifications, or an unfamiliar layout, or are a person altogether new to keyboards, it may be difficult to remember or keep track of. Hence the LED keys can be a great aid.

Re Dvorak, yes one can select it easily. But if you are learning typing, it's a whole lot easier if you can check the keys (or at least don't have the "wrong" layout looking back at you). One thing going against movement to Dvorak - not that I have a thing about that but some do - is that it's not ideal to learn/use a layout different than what you see, and no one will buy a Dvorak over a QWERTY since so many others have learned the latter. The LED key keyboard in effect levels the field because the keyboard can visually represent any layout you select.

The changing visual key sequences idea that Jean-Christophe asked about is another interesting dimension.

Don

JC Helary then responded:

I was specifically thinking about Japanese input. And I am curious how that would be implemented. Of course, it is not so much the sillabic part of the input that is a problem (there are Japanese kb layouts that assign one syllable to one key, even most Japanese users type with combinations of qwerty  letters.) It is rather the fact that the qwerty input part is converted 2 times: first time when the second letter is associated to the first to produce a Japanese kana, second when the kana (on its own or associated with other kanas) is converted to a kanji.

JC Helary

If you understand how the IME works, then you will understand how at this point, the wheels have started to come off the wagon. Yes, it is true that you enter Kana and then a candidate list comes up. But the list expects you to use arrow keys or similar methods to select items. There is never a time that the changing keyboard will have an opportunity to show the candidates because the various letter keys on the keyboard are not where one selects the Kanji -- that is where one types more kana. I am not going to knock this keyboard just yet, but let's not give it powers it cannot possess!

Don then put the conversation back on track:

Another possibility that came to mind was Ethiopic.

In any event the potential to readily view - on the keys themselves - alternative layouts, extended and non-Latin characters, and to have it respond to input in cases like you describe could be a great equalizer in perceptions of scripts and their value/utility. Of course it's a practical thing, first and foremost, with specific advantages to many users, but it's also a matter of psychology (visual information being so critical in our perceptions). And it's in the latter that the impact could be revolutionary. (Of course it's also possible that other technologies will overtake it in terms of reshaping how we input text in various scripts, but I won't take that tangent...)

Anyway, this is the last I'll post on this topic until we hear more from the keyboard designers or someone who has a chance to evaluate the prototype.

Don

MJ then added an interesting tidbit but gave no explanation:

Dear all,
We are already using such keyboard for Bangla for 18 years.
Thanks
MJ

George W. Gerrity then weighed in:

I have been waiting for such a keyboard for over 20 years. I even tried to get one with LED matrices on keys, manufactured to my specification about 15 years ago, with no luck (at least, at a cost I could afford).

I am surprised that there is even some uncertainty as to how such a keyboard might be used for non-latin scripts. Consider, for instance, its use with Simplified (mainland) and Traditional (Taiwan) Chinese and Japanese: In all three cases, there exist four or five commonly-used input methods designed to be used with a standard QWERTY keyboard, that depend on remapping keys. The only alternatives are expensive, huge (largely unstandardised) keyboards originally designed for typesetters.

Users who normally enter text in Chinese and Japanese have no trouble with the remapping because usage patterns develop for touch typing just as for users of QWERTY keyboards for Latin characters. The problem arises for users like myself, who occasionally want to enter Chinese, Hebrew, Greek, etc, and simply can't remember which keys go with which method.

Having such a keyboard, whose visual indicators can be programmed to change with input method and with shift, opt and alt keys, would be a real boon.

George

Ken Whistler had some additional thoughts about problems with trying to use this keyboard, especially with IMEs:

The whole concept of moving your hands *off* the keys to *read* the keys, and then back on the keys to key your input -- particularly for complicated IME's that change the state of input as you go -- strikes me as at best un-ergonomic and at worst frustrating and inefficient.

And Raymond Mercier extended this thought:

We are more bound by the tactile experience of the keys than we imagine, even those of us who are not real touch typists. I tried a new keyboard recently (mis-designed by Belkin) that had the unfortunate characteristic that the character appeared on the screen only after the finger was lifted from the key, instead of appearing on the downstroke of the key. It was quite impossible to type at normal speed. That experience rather suggests to me that normal typing would be very frustrated by having to pay so much visual attention to the keys.

And Gregg Reynolds responded to these thoughts:

Depends on how often you need to do it.  I can touch type on an Arabic keyboard, but there are always those rarely-used keys that I can't quite remember.  In that case, it would be a big time-saver if the keyboard could change dynamically from English to Arabic.

I think it would be hugely useful for learning e.g. Vi.  Assuming that is that it can be programmed to do something like:  hold down shft-ctl-z, and little arrows pop up on the h/j/k/etc keys.  Or in emacs, hold down some key combo and some kind of mnemonics show up indicating e.g. ctl-f for forward-char.  Much easier than trying to remember where such info is in online help.

If they can keep the price point somewhere in the vicinity of ordinary keyboards they will own the corporate marketplace, or at least that segment whose workers use multiple languages.  I have the Arabic keyboard memorized, so I don't generally need the letters printed, but
most of my colleagues covet such a keyboard, even if it only uses stickers.  They could just print out an image of an Arabic keyboard and pin it next to the monitor, but they want the info on the keys.

I would think a dynamic keyboard display could do the same (in principle).

Now in the past I have mentioned the cool soft keyboard that the Tablet PC version of Windows XP has, which repaints its "keys" based on the chosen keyboard layout. It works quite well with the idea Gregg is talking about.

It really is impossible to know how a keyboard will work here, though I suspect that many of the problems related to fingers having to be lifted to see the characters are really just the tip of the iceberg. But before I get into that, Suzanne E. McCarthy covered some thoughts on this new keyboard in her blog (abecedaria):

The Optimus Keyboard

Optimus Update

A Post QWERTY World (contains many of the same comments I quoted above)

Now, where was I? Oh yes, the tip of the iceberg.

The code that one must write to interrogate a keyboard layout using Win32 keyboarding APIs in even a single shift state is non-trivial. Now add to that a need to keep running that code on a per-thread basis (since the input language is indeed a per-thread setting across the whole operating system), and the fact that this would have to be user-mode code (the kernel APIs to do all of these interrogations do not really exist). Therefore, to work reliably on Windows, one would probably require a low level keyboard hook that would grab any keystroke made (to know when to change for shift states or dead keys).

Yikes, the performance implications of having to do all of that on a per-thread basis any time one switches to different UI threads is somewhat frightening. Though the code complexity issues also seem rather formidble. I actually know a few things about the code needed here (I had to write it for MSKLC's File|Load from existing... functionality. How else to interrogate a layout? But the notion of having to do this as often as the hardware would have to (and having to integrate inyerim states like dead keys and possibly even typograophical ligatures if one were truly adventurous!) is quite a challenging one.

One day I might go through the full process here to help give some appreciation of the level of work. :-)

And this is just for Windows. The work for any supported platform would likely not match other platforms.

Perhaps there are plans to do this differently -- perhaps interrogating the keyboard layout DLL directly. Not impossible but also not trivial. And hard to manage across all of the possible keyboard layouts.

My prediction (well, my guess) about the future of the keyboard would be that it will not end up being completely dynamic. It will just show the base state (like the hardware keyboard does now), and it will switch when the regular keyboard layout does. A lot like what the Tablet PC keyboard does now, which gives a good balance of the performance versus functionality issues.

 

This post brought to you by "" (U+1127, a.k.a. HANGUL CHOSEONG PIEUP-CIEUC)


# Dean Harding on Thursday, July 21, 2005 1:41 AM:

I saw that keyboard - I thought the "Quake" layout was pretty cool. Especially for games with lots of key mappings, it can be hard to remember what maps to what...

Another way they might implement it is that it's all completely custom - i.e. not based on IME or anything, and it's got it own driver to support switching shift states and doing dead letters and stuff. Not ideal from a usability point of view, but much simpler from an engineering point of view :)

It does say on the site "at least it’s going to be able to work in some default state with any OS" which means that on "unsupported" OSes, it's probably just a regular QWERTY keyboard.

# Dean Harding on Thursday, July 21, 2005 1:45 AM:

Oh yeah, it also says on the site "It will cost less than a good mobile phone," which seems like an awful lot for a keyboard... even an LED-key based one. And what's a "good" mobile phone anyway? They range from $100 to $1,000 (at least in Australia) so that's still quite an open price-range.

(I think that "Answers" page is new - at least it wasn't there when I looked)

# Rosyna on Thursday, July 21, 2005 2:05 AM:

Well, one possible benefit of this keyboard is that it could bring an entirely new class of keyboard APIs to the different OSes. But how.. You register a callback for key images and provide a png (or gif) image back for what the key should be? But then it'd be called fairly often and for every key. Hmm. Could have it call back when only the modifier state is changed. But then handling dead keys would be a bare.

But I don't see any way to make this fast even with HUGE caching.

Then again, if the OS supported this, it could create an entirely new class of HID devices. Imagine game controllers that use these same APIs to change what is shown.

# Michael S. Kaplan on Thursday, July 21, 2005 2:10 AM:

The strangest part about the whole thing (to me) is that no one even considered the real technical difficulties here (a list thats gets bigger every time I think about it!)....

# Rosyna on Thursday, July 21, 2005 2:49 AM:

Yes, the issues are huge and just keep getting bigger. But none of them are insurmountable. And the changes required might benefit the future quite a bit. Kind of like the change to Win32 from Win16 it created an entirely new class of devices and APIs just because it was nor far easier to do in some situations. Whatever the case, I'm glad I won't have to be the one to write the code to support one of these things.

I do hope that it comes with deep OS X support and it looks like it might since the designers are clearly mac users. And it would give the designers a limited test case of annoyingly vocal users so they could get feedback for HUGE design flaws in the keyboard before the world at large gets it. But how to handle return and \ since some keyboards join these keys and some (like the US) do not.

On a different note, it annoys me that people are calling this LED based. It is going to be (so they say) OLED based which is hugely different and MUCH cooler.

# Martin Bohring on Thursday, July 21, 2005 2:54 AM:

Well the candidate list list entry is selected with the numerical keyboard. Why not displaying the candidate list chars directly on the numeric keyboard numbers, if a candidate list entry is selected ?

# robdoyle on Thursday, July 21, 2005 3:05 AM:

The first reaction by most in the office was a geeky 'wow, imagine what you could do with that...' - however we quickly spotted a problem.
Various keyboard layouts not only produce different outputs, but they can have keys in physically different places.
For example, many European layouts have an extra key between Left Shift and Z - for UK / Irish it's the |\ pipe and backslash key. This "Universal" keyboard does not. :s

# Centaur on Thursday, July 21, 2005 11:06 AM:

Oh, that’s all well and good, micro-displays on keys and whatever. This keyboard goes where no other keyboard manufacturer has gone before: They add one key to each row of the alphabetic area.

Why is it so important? Because the Cyrillic alphabet has 33 letters while Latin has 26. This makes it impossible to fit all the Cyrillic letters on a standard 104-key keyboard without sacrificing something. And, due to the longstanding typewriter tradition, the characters that are sacrificed are: ` ' [ ] { } < > ~ |. I would be much pleased to see them back where they belong.

And that block of 10 keys on the left. The pictures show they intend to use them for application switching; but as far as I understand, they will be programmable and that’s just what I miss, a bunch of programmable keys with no default functions to conflict with.

All in all, I want a keyboard like that.

# Michael S. Kaplan on Thursday, July 21, 2005 11:42 AM:

At least on Windows, it is not up to the the hardware what will show up, it is up to the keyboard layout DLL, no matter what kind of hardware you have....

# gigi on Thursday, July 21, 2005 5:54 PM:

the next step would be to make the individual keys detachable so you can put them wherever you want, although i think it would be quite annoying to try using a keyboard with a unique layout that somebody else made up

# Dean Harding on Thursday, July 21, 2005 8:06 PM:

> At least on Windows, it is not up to the the hardware what will show up, it is up to the keyboard layout DLL, no matter what kind of hardware you have....

But that's assuming they install it as a regular "keyboard" device. They could just install their own custom driver which bypasses all the keyboard stuff and just does something equivalent to what that "Unicode Input" program from a couple of days ago does. Of course, that's not a very good solution, since it's not going to work for every application.

Or maybe they install their own layout DLL, which has custom hooks into the keyboard so that it knows when to switch layout or whatever.

All in all, my guess is that there'd have to be something "custom" to make it automatically display the right thing, because like you said, the architecture isn't right for doing it "out of the box"

Of course, I don't really know anything about how keyboards and their drivers work, so I could be (am?) making stuff up :)

# Dean Harding on Thursday, July 21, 2005 8:46 PM:

gigi: Check this out: http://www.canesta.com/products.htm :)

Surely this could be customized to display keys wherever you want!

# gigi on Friday, July 22, 2005 2:49 AM:

i was thinking more on the lines of http://www.ergodex.com/ cause i can't imagine typing on a flat surface is so great (not the same tactile feeling (and i really love the click-claks :)))

referenced by

2008/08/16 Optimus: from science fiction to fiction to frustration to geek porn, in just 24 months

2006/02/20 Mirroring and Keyboards are complicated

2005/07/22 Is the Optimus keyboard just a myth?

go to newer or older post, or back to index or month or day