More on cursor support: the rest of the answer
by Michael S. Kaplan, published on 2006/01/10 06:31 -08:00, original URI: http://blogs.msdn.com/michkap/archive/2006/01/10/511194.aspx
Ok, several people took up the challenge I suggested in More on cursor support: in Bidi, intuition can be a four letter word once I gave a hint in More on cursor support: half of the answer!
Before that, all of the guesses had been by email and none of them found five non-contiguous regions. I'll let that result speak for itself. :-)
Here is the answer, graphically.
Start with a left to right reading order, at the numbers on the left:
Hold down the shift key and then start hitting the right arrow key:
As soon as you cross the border from numbers to letters something interesting starts to happen:
And then the next transition:
You keep holding down the SHIFT key and continue hitting the right arrow key:
And finally you will hit that last bit:
As Mihai suggested, this will not work if you have an RTL reading order set (though I think the appropriate string to work in the RTL context can be left as an exercise at this point!).
And as Doug suggested, the secret here to get up to five is the "Bidi neutral" status of numbers (without the numbers, you can only get to three regions, I believe).
The fact that this is the work of Uniscribe's "automatic" support in Windows is evident from Ben Bryant's observation that if you do not have extra language support installed, it will still work in Explorer but it will not work in Notepad. The reason for this is that clicking the checkbox to Install files for complex script and right-ro-left languages (including Thai) will enable the layer that causes all text to be evaluated by LPK.DLL to see if it should go through Uniscribe. It is actually an easy way to detect whether an application is calling Uniscribe directly or relying on the LPK.DLL support to have Uniscribe called automatically....
Of course this is a rather contrived string, but the results, while seeming unintuitive to someone whose native language is not Hebrew, are really not entirely unexpected to a native speaker of a bidirectional language.
As several people noted, this is not just in Notepad -- it is also happening in Windows Explorer, in IE, and in FireFox -- including on Linux. It is not due to developers trying to emulate MS platforms, but rather due to developers trying to work in the way that people who use the language would expect things to work. Even if it is confusing to people who do not work with bidirectional languages....
As to the issue of why this is the expectation, I think it would be interesting to study the phenomenon here. When I think back to my past life as a neurodiagnostic technician, I think it would fascinating to actually use an electronystagmogram (ENG) machine to monitor eye movement while native and non-native speakers are reading bidirectional text; the results would likely be quite cool.
(don't worry people, I really don't have the equipment to start running those types of experiments; I did mention it was a past life!)
The fact that the string is contrived does not mean that the behavior is unexpected -- a native speaker of English would have no problem describing how the yexpected the cursor to work in a document full of nonsense words, even if they were gibberish. Though it is true that Microsoft Word has options to alter both cursor movement and selection behavior that can make both more intuitive for non-native speakers:
The description of these options is just an F1 away from this dialog:
Cursor control
Movement Specifies Logical or Visual cursor movement.
- Logical Sets insertion point movement to progress within bidirectional text according to the direction of the text encountered. For example, when using the arrow keys to move through Arabic and then English text in the same sentence, the arrow key moves from right to left through the Arabic text and then starts at the leftmost character in the English word and progresses from left to right.
- Visual Sets insertion point movement to progress within bidirectional text by moving to the next visually adjacent character. For example, when using the arrow keys to move from right to left through Arabic and then English text in the same sentence, the arrow key moves from right to left regardless of the direction of the text.
Visual selection Specifies cursor movement.
- Block Wraps text from line to line as you select downward, with all selected lines being the same width.
- Continuous Wraps text from line to line as you select downward and lets you vary the width of the final line of the block.
If you have Word you can actually try the "Movement" settings out to see what they do to your expectations, and whether the results make more sense or less to you. This way everyone gets to win, right? :-)
Addeddum 12 Jan 2006, sent by Mihai Nita:
Think that all lower-case characters are Hebrew characters, and this is the logical order (how they are stored in memory and how one would read them). RED shows the selection at each step.
abc123defABCghi456jkl
To display, we split this in text-runs with the same direction:
abc123def | ABC | ghi456jkl
123 and 456 are numbers weak types and do not break the run.
Then we render each run according to it's rules. 1st one is RTL, 2nd LTR, 3rd RTL again:
fed321cba | ABC | lkj654igh
The numbers in a RTL text are still rendered LTR, so we fix this:
fed123cba | ABC | lkj456igh
So here we have it, 5 selections, and all makes sense:
fed123cbaABClkj456igh
This post brought to you by "ף" (U+05e3, a.k.a. HEBREW LETTER FINAL PE)
# Michael S. Kaplan on Tuesday, January 10, 2006 9:42 AM:
Special thanks to Yaniv Feinberg for his help in coming up with the test string! :-)
# Ben Bryant on Tuesday, January 10, 2006 1:15 PM:
Oh I was hoping you would explain why the 5 non-contiguous selection areas would make sense to anyone at all! Or is that something that takes too long to explain i.e. requires reading a book about the subject? I gather it is due to there being 3 different bidi status chars involved and the way the switching of direction can cause it to go and include from the beginning of the line, be it the "beginning" that is at the right or the one that is at the left... :) Actually I intend to implement bidi editing (hopefully this year sometime!) so I guess I will find out soon enough.
# Mihai on Tuesday, January 10, 2006 1:53 PM:
"(though I think the appropriate string to work in the RTL context can be left as an exercise at this point!)"
My first intuition (the "four letter word") is that it cannot be done.
# Shoshannah Forbes on Tuesday, January 10, 2006 3:35 PM:
The experiment you are thinking about sounds really interesting- too bad it is your past life peeking :-)
I would like to note that as a native speaker, reading BIDI text like the example you gave is intuitive, even though it seems complex here. On the other hand, EDITING such text can be tricky sometimes- it isn't always obvious where the insertion point should be in order to add characters to the desired location, esp. if you start adding parenthesis into the mix.
In fact, I don't know anyone here (Israel) who doesn't use the "trial and error" method on occasion to find the right location for the insertion point when editing complex bidi text (most common example: editing the source code of a Hebrew web page in a text editor. Arrrrg).
Anyway- great posts. Nice to see i18n people talking about BiDi. And don't get me started ranting about RTL UIs and the use of "Left" and "Right" instead of "Before" and "After" (or - systems where some open windows have RTL UI and some have LTR UI).
# Michael S. Kaplan on Tuesday, January 10, 2006 4:56 PM:
Mihai --
My knowledge source originally thought there was a solution that could provide *seven* non-contiguous areas! But he scaled back eventuaslly....
An RTL sentence that will do the same thing is indeed supposed to be possible.
# Mihai on Tuesday, January 10, 2006 9:20 PM:
-- MisKa --
I guess it takes some years of really using RTL languages to get the intuition work right, so I would not be sourprised :-)
-- For Ben --
I will try a short explanation for the why part, very-very simplified. You can check the Unicode BiDi Algorithm for the real deal (
http://www.unicode.org/reports/tr9)
This will be more visible with fixed width fonts, so copy/paste in Notepad/Word and select a fixed font.
Think that all lower-case characters are hebrew characters, and this is the logical order (how they are stored in memory and how one would read them). ~ shows the selection
abc123defABCghi123jkl
~~~~~~~~~~~~~
To display, we split this in text-runs with the same direction:
abc123def | ABC | ghi123jkl
~~~~~ ~~~ ~~~~~
123 are numbers weak types and do not break the run.
Then we render each run according to it's rules. 1st one is RTL, 2nd LTR, 3rd RTL again:
fed321cba | ABC | lkj321igh
~~~~~ ~~~ ~~~~~
The numbers in a RTL text are still rendered LTR, so we fix this:
fed123cba | ABC | lkj123igh
~~~ ~~ ~~~ ~~ ~~~
So here it is, 5 selections, and all makes sense:
fed123cbaABClkj123igh
~~~ ~~ ~~~ ~~ ~~~
# Mihai on Tuesday, January 10, 2006 11:35 PM:
Sorry, the explanation is totally messed up. The submission form collapsed the multiple spaces to one space only, so the alignment is wrong even with fixed width fonts :-(
I will try to post this somewhere else, with a link to and from here.
# Michael S. Kaplan on Wednesday, January 11, 2006 1:10 AM:
Ah, no worries -- you can also mail it to me if you want and I'll post it....
# Nick Lamb on Wednesday, January 11, 2006 10:14 AM:
It's easier to understand what's going on with a better example. Begin the string with English text (so that people implementing UAX #9 P3 detect it as an LTR paragraph) then use 123/456 rather than 123/123 and use nine different Hebrew characters rather than the same three repeated.
This makes it obvious that your selection is just a contiguous series of characters throughout, and re-ordering is done for display as required by UAX #9. As observed by one of your posters, this selection exists regardless of the cursor movement rules, but the means to achieve it differs if you use the keyboard.
Unfortunately this series of posts doesn't really seem to address my earlier comments, except the mention of Word having support for visual cursor movement hidden away. You also mentioned Firefox and Linux, apparently without realising that they both implement visual cursor movement.
For clarity: The Unicode BiDi Algorithm does not specify cursor movement. Specifications which seem more comprehensive than Unicode (while not contradicting it) such as IBM's, specify visual cursor movement as the preferred or only mechanism as well as a secondary hint cursor and other things not (yet) implemented by Windows.
# Universalis on Wednesday, January 11, 2006 10:36 AM:
Thank you for explaining what LPK.DLL is - I was wondering why DrawTextW(,L"Crash&",6,,) sometimes crashed and sometimes didn't. It's down to LPK intercepting DrawTextW and passing the string on to Uniscribe and Uniscribe (USP10.DLL) scanning the string and trying to look beyond the final ampersand (since DT_NOPREFIX was not specified). If the string happens to be allocated at the end of a page then the result is a crash as USP10 reads unallocated memory.
On non-complex-script installations LPK doesn't interfere, and there's no crash because the real live Windows DrawTextW doesn't have this bug.
I don't think Unicows has the bug either. Perhaps it should be added, for compatibility with the latest Windows implementation...
# Michael S. Kaplan on Wednesday, January 11, 2006 12:20 PM:
Hi Nick --
Your suggested improvment would actually cause there to not be those five non-contiguous areas, though perhaps a modified version of it could be used. I am pretty sure that this example is clear enough for the point to be understood.
I understand that you believe the MS implementation is a legacy thing and that the IBM visual solution is preferred. At this point I guess we will have to agree to disagree, though it is worth noting that generally speasking I think you will find that native speakers do not agree with you.
I am always willing to be wrong and admit I am, but when native speakers agree with me, I will assume that I am right? :-)
# Nick Lamb on Wednesday, January 11, 2006 2:15 PM:
Hmm, it works fine here, the sample string I'm using is
This אבג123דהוABCזחט456יכל works fine for me
Since blogs.msdn.com seems to have various problems I'll also include instructions on how to enter this yourself (but I assume you know how to enter Hebrew characters even if only by typing their Unicode codepoint)
1. Type "This" followed by a space
2, Then type Alef, Bet, Gimel, the first three characters of the Hebrew alphabet
3. Type "123"
4. Now the next three Hebrew characters
5. And now "ABC"
6. Then the next three Hebrew characters
7. And "456"
8. Finally type Yod, Kaf, Lamed
9. Type a space followed by "works fine for me"
You can select the range from between 2-3 and 4-5 to see the five non-contiguous areas as before.
It's still nonsense, but now anyone can recognise the re-ordering rules in operation.
Michael, if this doesn't work for you it may be because of a bug, please describe the problem further.
# Michael S. Kaplan on Wednesday, January 11, 2006 7:04 PM:
Ah, this would work,I had something different in mind here....
# Uri Bernstein on Friday, January 20, 2006 2:05 PM:
I'm a bit late to discover this post (and this blog), but for the record: I'm a native speaker (and writer) of Hebrew, and I strongly prefer "visual" caret movement (and selection).
The catch is that visual caret movement is much more difficult to specify and implement (due to the interaction with logical typing and deleting), so some of the existing implementations are still buggy in this way or another. I believe that this, along with the fact that most people became used to working with Microsoft products (which use logical caret movement), is the reason that many people claim to prefer logical movement.
Please consider a
donation to keep this archive running, maintained and free of advertising.
Donate €20 or more to receive an offline copy of the whole archive including all images.
referenced by
go to newer or older post, or back to index or month or day