BACKSPACE vs. DELETE

by Michael S. Kaplan, published on 2005/12/21 04:01 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2005/12/21/506248.aspx

I have often talked about how awesome I think the MSDN Product Feedback Center is. But I do have to admit that sometimes bugs can kind of get lost....

No one intentionally meant to be unresponsive, but this particular bug had been reported several different times, by many different people (I saw over a dozen bug reports put in across several product bug databases!). And unfortunately at some point when the bug report was resolved as a duplicate of a duplicate and so on, the follow-up information was not transferred to the active bug report (so the old bug report kept getting your updates but no one knew to look at it anymore since it was not active).

For the primary bug itself, the actual problem is an issue in the core OS control (you can reproduce the issue in Notepad, for example) and the managed controls are really wrappers around the core controls in this case. So the active bug in the "Whidbey" bug database was resolved as an 'external issue' while the bug report was put into the Windows bug database.

You see, in situations where one is typing digraphs or other compounds that are considered to be a single character to the user even when they are not represented by a single keystroke on the keyboard, the decision to have the DELETE key get rid of the entire text element yet have the BACKSPACE key only remove the last one is an intentional one. A decision that allows you to avoid the frustration of having to retype 2-4 or more characters if you only mis-typed the last one.

However, while this is a reasonable plan of action in most cases, it is obviously not so reasonable in the case of a supplementary character (since it is in most cases unlikely that individual keystrokes are the high and low surrogate code units). And in this case it probably would make more sense to delete both the high and low surrogate rather than just the trailing low surrogate, especially considering the fact that there really is no recovery other than hitting the BACKSPACE again (since the keyboard is unlikely to be made up of individual high and low surrogate code units with one per key). And all of that is ignoring the very real concerns that Ben pointed out about having unpaired surrogates in data streams -- a situation that is definitely best avoided.

The bug in the Windows database was reported fixed a few months ago, but was never marked as resolved (and does not appear to be fixed in the latest build I have installed on my machine, from a few days ago, so it is probably not fixed in the December CTP build, either).

I will keep an eye on this bug in any case, not only to make sure it is fixed but also to make sure the desired behavior for the non-supplementary character scenario is not unintentionally broken.

In any case, I am sorry about how letting you know kind of fell through the cracks. The MSDN Product Feedback Center is still a very awesome resource and at the very least the issue itself was still being tracked. We've got your back on this one!

This post brought to you by "𐂀" (U+10080, a.k.a. LINEAR B IDEOGRAM B100 MAN)

Wow - I actually came across this behaviour while testing my product with digraphs and after I thought about it, I was convinced this was by design.
But of course, as you pointed out, the supplementary char scenario doesn't make much sense!

Another bug caused by having a "wide character" that is not wide enough for storing characters. It's more or less inevitable that coders trying to use 16-bit integers as a "native" Unicode character will get these cases wrong over and over again and again.

If I were to go look in MSDN right now, how many examples would I find where there's an implicit assumption that characters fit in a 16-bit Win32 wchar_t ?

Nick, this is really not the point, since even in UTF-32, what a user sees as a single character may be in fact many Unicode code points.

For most purposes, buffer lengths are not based on a character count but on a code point count. Supplementary characters being two Uniocde code units is no more of an issue than them being 4 bytes in UTF-8.

Thank you for the response.
I had a feeling that the report got lost somewhere along the lines.

I am glad that Microsoft is aware of the issue. I understand and agree with your comments on diagraphs. However, surrogates are a result of the chosen encoding (UTF-16) even when they map to a single Unicode scalar value. Unpaired surrogates are a serious concern to data integrity and conformance. I think this may warrant special handling, and I hope to see a resolution of some type in the near future.

How about defining Backspace to remove a single Unicode scalar value? This should preserve the current behavior designed for diagraphs as well as handling surrogates. It is also independent of the encoding.

Ben Monroe
Tokyo, Japan

Hi Ben,

It is unfortunately not that simple -- at the level where these things are determined, there is no knowledge of Unicode scalar values.

But like I said, the fix should be available in Vista soon (and although Vista is not the first version to support supplementary characters, it is the first version to ship with font and input support for them in the box!).

"Nick, this is really not the point, since even in UTF-32, what a user sees as a single character may be in fact many Unicode code points. "

Just so that we're clear, is it your contention that this bug was /not/ caused by the issue I'm describing ? If so, can you show your readers the affected code ?

Or is it instead that you believe it doesn't matter, that while a different API would avoid this mistake it would make some other type of error more common ?

My point is that the issue of the "wide character that is not wide enough for storing characters" being treated as a bug in and of itself is a fundamental misunderstanding of the Unicode standard and how it is formulated.

The issue I am describing is a side effect of the design decision that was made to have the DELETE key and th BACKSPACE key show different behaviors -- a decision that is still obviously not fixed for anyone who creates a Form 'D' keyboard that combined multiple code points on single keystrokes.

The fix of the supplementary character issue is obviously more critical than the other issue (which is good since of the two it is the one that is more fixable!).

But worrying about it as if it were a 'size of UTF-16 code units' issue kind of distracts from the actual problem....

Then you guys mixed two unrelated things together.

Deleting part of a character by backspace is legitimate and may have a meaning.

It should have nothing to do with how many bytes does it take in UTF-16 or other encoding of unicode.

Especially given that UTF-16 will hopefully not survive in Windows - the UI should be designed regardless of underlying text encoding.

Please consider a donation to keep this archive running, maintained and free of advertising.
Donate €20 or more to receive an offline copy of the whole archive including all images.