For me, SQM is pronounced SQuirM

by Michael S. Kaplan, published on 2007/09/07 03:01 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2007/09/07/4800323.aspx

I am pretty sure it is obvious in this post, but just in case: I am speaking for only three people here: me, myself, and I. Although some others may agree with me, I know for fact some people do not. I am not speaking for Microsoft....

I'll explain what the title is about but not until I blather on for a while, so you might be needing to either be patient or skipping this post.:-) 

The backstory is important as the first version of this post had no backstory and therefore no context, and was even more confusing than this one!

So today I went to the Microsoft company meeting (I took the bus there and made it there and back just fine).

Kind of a typical one, perhaps a few more silly things said then I remember from previously.

But when Ray Ozzie was talking it reminded me of something I had been meaning to write about....

It started back a while before I came out to Redmond, so we're talking maybe a little under 12 years ago.

I happened to be out in Redmond taking the alpha version of the Access 95 certification exam.

Have you ever taken an alpha of one of the exams? I don't know if they still do it the same way, but the process was quite fascinating!

You see, they invite out some real experts for the product, both inside and outside of Microsoft.

You only take the parts of the exam that you are an expert in, because you are not the one being tested -- the test is. The assumption is that you would get a perfect score, so afterward each answer that someone got wrong is reviewed to determine whether the question can be altered in such a way as to be salvaged or whether it should just be discarded.

Like I said, a very unique experience.

The author of the exam was MikeG1, a.k.a. Mike Gunderloy (this was back when he liked Microsoft, obviously!) which is I think why I was invited. Or maybe because there was no one else to do the replication section. Whichever....

Anyway, in between sections of the test I was talking to Steve from Certification Systems, the group that was managing the whole process. There were interesting statistical issues about building [provably] valid exams so that you can have the exam be slightly different (pulling in different questions) yet have comparable results between them, and I was dealing with similar issue myself related to the validating of comparisons between different medical studies and I had just written something that did this, so many of the issues were kind of fresh in my mind.

I was actually just passing the time, and it is a problem I find fascinating, so it was interesting (I am a fast test taker so if not for this I might have been bored a lot!).

Anyway, he was apparently really impressed, because he contacted me a few months later when I was back in Columbus, and asked if I would be interested in coming out to Redmond and working the folks in Certification Systems with the goal of writing an application to build exams with all of the right validity checking, etc.

The pay rate was sufficient for me to take a nice little vacation from all of the doctors I was doing work for, so soon I was heading out to Redmond for a short term contract with Microsoft. :-)

Now the engine was going to be working with the beta exams, which leads to a whole different set of interesting problems....

Have you ever taken a beta of one of these exams? I don't know if they still do it the same way, but the process was interesting.

You see, people who take the beta exam were not being charged for taking it, and the exam was gong to be a bit longer than the actual exam would be. But for the extra effort, you still got to be certified.

And HERE is where those interesting differences come up, ones that made for some fascinating conversations with the psychometrician types....

Because the initial validity of the test (at least until as baseline of actual exam result became available) was largely being determined by these beta results, and it seemed obvious that there were quite possibly going to be some potentially significant differences between the two groups, with beta testers possibly being influenced by factors such as:

• the kind of people who would prefer not to pay for a test
• the kind of people who like knowing they making the test better by being involved
• the kind of people who do not mind more questions on their exam
• the kind of people who realize that they can have a lower percentage and still pass since not all of the questions will count
• etc.

Now the differences may or may not have been statistically significant, but the differences had to at least be considered.

Okay, so that is the backstory.

Anyway, Ray Ozzie was talking about thing and one of the themes was about how PCs when they came out empowered people individually, kind of in a very different direction than the more institutional, big computer world that really wasn't leaving room for individuality.

I got to thinking about how these days every product wants metrics -- they want to get people to check "YES" on the checkbox that says you're okay with sending Microsoft information about your use of the product, like this one in Windows Media Player 9.0:

And this whole freaking dialog in Visual Studio 2005:

I personally do not like to send this data.

There are many reasons for this, and in the end it is my choice so choosing NO is valid for anyone who does not wish to.

But now here is the thing:

• Some products default this option to YES and some default it to NO;
• Some customers always go with defaults;
• Some customers really want to give feedback and will go out if their way to say YES;
• Some customers really hate the whole notion of this "giving MS information" and will go out of their way to say NO;
• Some customers actually act differently in their use of the product knowing that way it is used is reported, even if it is anonymous;
• Some customers really want to screw with Microsoft and therefore use the product in stupid ways hoping that the info is reported sand random people at MS work to make something more intuitive since there is "proof" that the UI has usability problems.

Some of these may be less common but I actually know of specific customers who have made each of these claims in the past.

You can probably see where I am going with this.

First, there is the fact that it does kind of violate some of the whole individual empowerment thing Ray Ozzie was talking about when products phone home and report to the man what people are doing.

And second there is the fact that I myself do not like data being sent, and I am not the only person who feels that way.

(I'm the sort of person who spends less time on microsoft.com now that I have to log in to look at KB articles -- that kind of boldfaced "monitoring" that is clearly non-anonymous really annoys me, almost as much as the whole Google My Search History stuff!)

Third, the data that is being collected, the information that is gathered, is fundamentally limited by the fact that in all cases the set of people who are sending feedback to Microsoft about a product are a specific subset of those people who buy and use the product. And the difference may very well skew the results, meaning groups that do not get feedback in other ways will not be using the best possible data about product usage as they plan for new features and bug fixes in future versions.

The title of the post (For me, SQM is pronounced SQuirM) was inspired for me as I thought about all of this and remembered a comment from Matt Rhoten to my post There is no 'I' in MUI... errr, never mind! from the other day that went:

I was originally taught (by PaulC, my first boss at the collective) that SQL was pronounced 'squeal'.

I personally recommend that for every novice database programmer. You will definitely squeal the first time you forget the WHERE in an UPDATE while doing maintenance on a production system.

In the end, SQM (Software Quality Monitoring) makes me squirm uncomfortably, which is why pronouncing SQL as 'squirm' makes a lot of sense to me....

Now at this point huge fans of SQM who can't believe I feel this way as they do find it very useful, and that this is like voting (i.e. the people who do not choose to give feedback simply don't get a voice in some things).

To those people, I will say that (a) we will just have to agree to disagree and (b) these two additional factors should be considered:

• Some customers have no Internet connection to send such info through;
• Some customers are not allowed to send the feedback by the choice of their administrators.

Clearly some of those scenarios represent not only large numbers of customers, but also customers who Microsoft wants to be writing product for.

I'd rather find out stuff the hard way than having products report back on their usage, even at the risk of not finding everything out.

It is hardly a theoretical issue even for internationalization; I believe Locale Builder has SQM support and I am pretty sure the group that owns MSKLC now would probably add it to future versions. I do not know what the plans are for Regional and Language Options but everyone seems to like SQM these days, so who can say for sure....

Even with a privacy statement, even if the opt-out is tasken seriously, it still feels less than transparent....

And in the end, I don't like feeling like I am spying on people, anonymously or not. It just makes me squirm.

Spelled SQM? :-)

This post brought to you by ⺫ and ⽬ (U+2eab and U+2f6c, a.k.a. CJK RADICAL EYE and KANGXI RADICAL EYE)

Jan Kučera on 7 Sep 2007 5:18 AM:

Skip on 7 Sep 2007 10:15 AM:

Dean Harding on 7 Sep 2007 9:22 PM:

Jan Kučera on 8 Sep 2007 7:14 AM:

Kalle Olavi Niemitalo on 9 Sep 2007 4:02 AM:

Michael S. Kaplan on 9 Sep 2007 8:12 AM:

go to newer or older post, or back to index or month or day