Why international test is an art (and why there are few fine artists)

by Michael S. Kaplan, published on 2005/01/04 02:04 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2005/01/04/346116.aspx

Today someone had sent mail around noting that a toString() call on a date in 1911 threw an ArgumentOutOfRangeException when the Taiwan calendar was used and asking if it was a known issue. It is, because the calendar started in 1912. One of a zillion issues that someone who lives in an area will probably know, but others may or may not.

Dinesh Bhat talked yesterday about What is Testing? and it reminded me about how I was talking about how Culture names are not region names (and vice versa) the other day how after I talked about the problem with culture/region names I called it a symptom of a larger problem with international testing. I figured that there is no better time than the present to explain what I meant....

Dinesh pointed out that Testing is the unregulated art¹ of evaluating the invisible² against the ambiguous³ so as to avoid the unthinkable.⁴

In large part I think that is true, although international test adds new wrinkles to that equation -- it increases both the invisibility and the ambiguity, while sometimes decreasing the hunches and thereby sometimes decreasing the skill that the tester brings to the art.

It does this because the job of international functionality in an OS or a framework or an application is to make things act intuitively with a user's cultural/language/lingustic expectations. No matter what the culture or language. Since a tester cannot speak every possible language and even a George Campbell⁵ cannot know about the cultural expectations of every language they speak.

To give a specific example -- Kanya Jiaranaipreeda is one of the excellent testers who works in GIFT. She owns testing for MSKLC and due in no small part to her native speaking knowledge of the Thai language she is single-handedly responsible for making sure that MSKLC does not suck for Thai. And that is not an exageration -- if she did not point out the problems, then large sections of her language would not be able to be entered into keyboard layouts.

What is interesting is that she walked in knowing that some of the issues would possibly exist, mainly because software often messes them up. And both the problems she reported and the solutions I came up with covered a wider range of issues that helped other languages. Furthermore, she found other bugs related to other languages with which she did not have as much knowledge because she has good instincts.

But she is also a dedicated tester for international functionality, as are all of the testers on our team. For most teams there is only one such tester. And when they introduce themselves the conversation goes like this.

Jason: Hi, my name is Jason, I am a tester here.

Elissa: What area do you test?

Jason: International functionality.

Elissa: Oh, you're the new tester.

Its hard to develop those instincts when you move to a new area after not too long and then someone else gets to start all over as the tester. Especially when a lot of what are one is testing (cultural/linguistic expectations) is not covered in a spec, except obliquely. And it takes time to really develop instincts beyond just covering the functionality in the spec.

About four years ago, I wrote a book called Internationalization with Visual Basic. Its chapter 16 entitled Testing International Software (available for free online in English, Farsi, Thai, Chinese, and Romanian) has nothing to do with Visual Basic of any version. It is a small piece that was added with the help of one of the best testers ever, Suzanne Ezell (neé Goebel) at the very last minute (which is why it was only eight pages). It points out a lot of the issues of the minimal areas wants to cover when one is trying to cover international testing when one has almost no knowledge of international functionality.

I highly recommend the chapter, which is one of the reasons it is available for free online.

I have gotten so much fan mail about those eight pages that I am convinced I could do a whole book on the topic that would sell and be appreciated, if only I knew enough about the topic (which I do not). I do know I'd probably buy such a book and read it (or maybe get a free copy after being asked to write the foreward? <grin>). And this may be a hint if there are any testers-who-wants-to-be-writers out there....

Though note that that there is no replacement for those instincts -- those must be developed.

Usually after a product ships and a bug is found, both the developer and tester who were responsible for causing the bug and not finding it (respectively) will do a mental triage where they will either understand how that bug could have gotten by them or they will be kicking themselves because they should have found it. Usually they go through this separately which is unfortunate since it might have been an interesting conversation.

Though it's probably just as well since it might be too easy for the tone of the conversation to sound like blame. It's better when they do it separately because the good ones will blame themselves -- the dev will think "how the hell did I let that case get by without checking for it?" while the tester will think "I screwed up, why didn't I find this one?". And then they can move on and fix it at the next opportunity.

But anyway, the thing about international test is that it's like testing, but harder.

Put another way -- in testing one can become expert in large parts of an area over time (and I have met many testers who have a better grasp of the full scope of a feature and its behavior than the developers who wrote the code). But in international testing even though one can never be an actual expert on all the specific languages and cultures, one must beome a meta-expert on the forces that guide the proper enabling of them. They must develop good instincts to sense when things may not be right so that they can work with experts on the language level when their "spidey sense" is tingling abot an issue.

The testers who are good at this are worth their weight in gold, as they are the best chance to make sure one's development efforts are not shipped in the form of crap, due to one or more showstopping bugs.

1 - an art because nobody can teach testing. One can teach testing methodologies process but at the end you need to tester’s smartness to unearth a bug.
2 - invisible because when you get a hunch that there is might be a bug, you write repro code, try out user action many times to evaluate your guess.
3 - ambiguous because it is not always clear what the behavior is supposed to be -- testers are the ones who fill in the blanks that the specs do not cover.
4 - Unthinkable #1: Customer hitting the bug which you couldn’t find. Unthinkable #2: Being fired because you shipped product with a showstopper.
5 - Linguist who passed away in the middle of December 2004. Was fluent in 44 languages and had a working knowledge of 20+ others.

This post brought to you by "ཪ" (U+0f6a, a.k.a. TIBETAN LETTER FIXED-FORM RA).

referenced by

go to newer or older post, or back to index or month or day