Short-sighted text processing #1: Uniscribe filters nothing

by Michael S. Kaplan, published on 2010/12/18 07:01 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2010/12/18/10106789.aspx

It was just the other day that regular reader Random832 commented to my I think MaxLength needs protection to assure safer text:

Who decided that cutting off your string and beeping at you was good UI, anyway? Surely it'd be better to just not let the user submit the form until they delete enough stuff to fit [and provide interactive feedback about the limit, a la twitter]

Well, I'm only half-serious - i'm sure this was easier with the limitations programmers had to work with in 1983 - but surely no new programs should be using this "feature".

Now there's a question.

It is in fact an excellent point, though.

I mean, for all the bold talk about one of the possible indicatons for "complex script support" is indeed Filters out illegal character combinations -- and this is called out in Thai, specifically. GoGlobal goes a bit further in its Complex Scripts FAQ:

Why do we need to filter illegal character combinations?
Since Thai syllables consist of a consonant optionally followed by one vowel and/or one tone mark, some character combinations (e.g., two vowel marks in succession) are nonsensical. Thus, one of the tasks of complex script enabling is to filter out or disallow illegal character combinations.

Interestingly, it makes me think of the prototypical example of this behavior:

1. Open Notepad
2. Switch to the That Kedmanee keyboard
3. Type the "J" key, which tries to type U+0e48, aka THAI CHARACTER MAI EK

Every time you hit the key, the computer will beep and insert nothing.

But can I tell you a secret?

You have to promise that you won't tell anyone. It is kind of embarrassing.

Uniscribe isn't doing that.

Seriously.

I can put a bunch of those characters in a row just fine, in text, in an applicaion other than Notepad. And Uniscribe will display them.

I can even paste lines of them into Notepad:

or here:

่่่่่่่่่่่่่่่่่่่่่่่่
่่่่่่่่่่่่่่่่่่่่่่่่
่่่่่่่่่่่่่่่่่่่่่่่่
่่่่่่่่่่่่่่่่่่่่่่่่

and guess what? There's no problem with doing it.

The code that "filters" these characters sits in code called by the EDIT control that checks for two things:

• Does the script of the given text "disallow illegal sequences" as described by the SCRIPT_PROPERTIES->fRejectInvalid from GetScriptProperties TRUE, and
• is the SCRIPT_LOGATTR->fInvalid of the given character also TRUE?

If both are, while you are typing, this code that is not in Uniscribe itself will fail the attempt to insert the text, and it will beep.

Obviously that doesn't work so well for text that is already present (how do you scold someone for illegal text alreay typed?), so in that case Uniscribe will just do as it is told. And it will of course include the 'empty circle" that implies a missing base character.

Now there are several problems inherent in this direction for the text processin engine to go, and I am going to get into that more tomorrow.

But I wanted to start by saying that it is a limited number of dumb controls that screw with the input stream while you are typing that is doing the work here -- Uniscribe filters nothing.

There are some folks who will like upcoming parts to this series, so I hope that (for example) Andrew West and Martin Hosken are around. Because both of them and a few folks like them, are gonna like this one....

Andrew West on 18 Dec 2010 4:48 PM:

Michael S. Kaplan on 18 Dec 2010 11:31 PM:

Doug Ewell on 20 Dec 2010 8:00 AM:

Michael S. Kaplan on 20 Dec 2010 9:13 AM:

Doug Ewell on 20 Dec 2010 10:50 AM:

Michael S. Kaplan on 20 Dec 2010 10:57 AM:

Michael S. Kaplan on 20 Dec 2010 11:10 AM:

referenced by

go to newer or older post, or back to index or month or day