For me, SQM is pronounced SQuirM

by Michael S. Kaplan, published on 2007/09/07 03:01 -04:00, original URI: http://blogs.msdn.com/b/michkap/archive/2007/09/07/4800323.aspx


I am pretty sure it is obvious in this post, but just in case: I am speaking for only three people here: me, myself, and I. Although some others may agree with me, I know for fact some people do not. I am not speaking for Microsoft....

I'll explain what the title is about but not until I blather on for a while, so you might be needing to either be patient or skipping this post.:-) 

The backstory is important as the first version of this post had no backstory and therefore no context, and was even more confusing than this one!

So today I went to the Microsoft company meeting (I took the bus there and made it there and back just fine).

Kind of a typical one, perhaps a few more silly things said then I remember from previously.

But when Ray Ozzie was talking it reminded me of something I had been meaning to write about....

It started back a while before I came out to Redmond, so we're talking maybe a little under 12 years ago.

I happened to be out in Redmond taking the alpha version of the Access 95 certification exam.

Have you ever taken an alpha of one of the exams? I don't know if they still do it the same way, but the process was quite fascinating!

You see, they invite out some real experts for the product, both inside and outside of Microsoft.

You only take the parts of the exam that you are an expert in, because you are not the one being tested -- the test is. The assumption is that you would get a perfect score, so afterward each answer that someone got wrong is reviewed to determine whether the question can be altered in such a way as to be salvaged or whether it should just be discarded.

Like I said, a very unique experience.

The author of the exam was MikeG1, a.k.a. Mike Gunderloy (this was back when he liked Microsoft, obviously!) which is I think why I was invited. Or maybe because there was no one else to do the replication section. Whichever....

Anyway, in between sections of the test I was talking to Steve from Certification Systems, the group that was managing the whole process. There were interesting statistical issues about building [provably] valid exams so that you can have the exam be slightly different (pulling in different questions) yet have comparable results between them, and I was dealing with similar issue myself related to the validating of comparisons between different medical studies and I had just written something that did this, so many of the issues were kind of fresh in my mind.

I was actually just passing the time, and it is a problem I find fascinating, so it was interesting (I am a fast test taker so if not for this I might have been bored a lot!).

Anyway, he was apparently really impressed, because he contacted me a few months later when I was back in Columbus, and asked if I would be interested in coming out to Redmond and working the folks in Certification Systems with the goal of writing an application to build exams with all of the right validity checking, etc.

The pay rate was sufficient for me to take a nice little vacation from all of the doctors I was doing work for, so soon I was heading out to Redmond for a short term contract with Microsoft. :-)

Now the engine was going to be working with the beta exams, which leads to a whole different set of interesting problems....

Have you ever taken a beta of one of these exams? I don't know if they still do it the same way, but the process was interesting.

You see, people who take the beta exam were not being charged for taking it, and the exam was gong to be a bit longer than the actual exam would be. But for the extra effort, you still got to be certified.

And HERE is where those interesting differences come up, ones that made for some fascinating conversations with the psychometrician types....

Because the initial validity of the test (at least until as baseline of actual exam result became available) was largely being determined by these beta results, and it seemed obvious that there were quite possibly going to be some potentially significant differences between the two groups, with beta testers possibly being influenced by factors such as:

Now the differences may or may not have been statistically significant, but the differences had to at least be considered.

Okay, so that is the backstory.

Anyway, Ray Ozzie was talking about thing and one of the themes was about how PCs when they came out empowered people individually, kind of in a very different direction than the more institutional, big computer world that really wasn't leaving room for individuality.

I got to thinking about how these days every product wants metrics -- they want to get people to check "YES" on the checkbox that says you're okay with sending Microsoft information about your use of the product, like this one in Windows Media Player 9.0:

And this whole freaking dialog in Visual Studio 2005:

I personally do not like to send this data.

There are many reasons for this, and in the end it is my choice so choosing NO is valid for anyone who does not wish to.

But now here is the thing:

Some of these may be less common but I actually know of specific customers who have made each of these claims in the past.

You can probably see where I am going with this.

First, there is the fact that it does kind of violate some of the whole individual empowerment thing Ray Ozzie was talking about when products phone home and report to the man what people are doing.

And second there is the fact that I myself do not like data being sent, and I am not the only person who feels that way.

(I'm the sort of person who spends less time on microsoft.com now that I have to log in to look at KB articles -- that kind of boldfaced "monitoring" that is clearly non-anonymous really annoys me, almost as much as the whole Google My Search History stuff!)

Third, the data that is being collected, the information that is gathered, is fundamentally limited by the fact that in all cases the set of people who are sending feedback to Microsoft about a product are a specific subset of those people who buy and use the product. And the difference may very well skew the results, meaning groups that do not get feedback in other ways will not be using the best possible data about product usage as they plan for new features and bug fixes in future versions.

The title of the post (For me, SQM is pronounced SQuirM) was inspired for me as I thought about all of this and remembered a comment from Matt Rhoten to my post There is no 'I' in MUI... errr, never mind! from the other day that went:

I was originally taught (by PaulC, my first boss at the collective) that SQL was pronounced 'squeal'.

I personally recommend that for every novice database programmer. You will definitely squeal the first time you forget the WHERE in an UPDATE while doing maintenance on a production system.

In the end, SQM (Software Quality Monitoring) makes me squirm uncomfortably, which is why pronouncing SQL as 'squirm' makes a lot of sense to me....

Now at this point huge fans of SQM who can't believe I feel this way as they do find it very useful, and that this is like voting (i.e. the people who do not choose to give feedback simply don't get a voice in some things).

To those people, I will say that (a) we will just have to agree to disagree and (b) these two additional factors should be considered:

Clearly some of those scenarios represent not only large numbers of customers, but also customers who Microsoft wants to be writing product for.

I'd rather find out stuff the hard way than having products report back on their usage, even at the risk of not finding everything out.

It is hardly a theoretical issue even for internationalization; I believe Locale Builder has SQM support and I am pretty sure the group that owns MSKLC now would probably add it to future versions. I do not know what the plans are for Regional and Language Options but everyone seems to like SQM these days, so who can say for sure....

Even with a privacy statement, even if the opt-out is tasken seriously, it still feels less than transparent....

And in the end, I don't like feeling like I am spying on people, anonymously or not. It just makes me squirm.

Spelled SQM? :-)

 

This post brought to you by and (U+2eab and U+2f6c, a.k.a. CJK RADICAL EYE and KANGXI RADICAL EYE)


Jan Kučera on 7 Sep 2007 5:18 AM:

Well I actually agree to send this data. I simple believe it could make the product better. I also belong to the customers who usually finds their way if they want to provide feedback anyway.

And I think that customers who don't have the ability - either physical or by means of some superior force - to send feedback are lost in the statistic value of results.

On the other side, I do think that it has flaws. First, I think that it would be, say at least fair, to publish the results, which I'm really interested in.

But the most important point I think is that the feedback comes from different users. As a developer (or IT pro or whatever) I see that I use regular products in different way than usual users. Some people believe this should be distinguished, some not. However, I am afraid (and it's only my personal feeling) that the group who actively chooses to provide the feedback significally tend to be the more experienced ones in opposite for example of those who just dread any dialog and go with the default settings using the red close button.

If this was true, than habits of experienced users are involving the products in negative way from the point of ordinary users.

That's not the case of Visual Studio feedback, though. :)

Skip on 7 Sep 2007 10:15 AM:

I'm in the 'no way will I ever enable sending this data if given a choice' camp.   It all boils down to a trust issue.   The dialog says, basically 'we mean you no harm, trust us.'

On the other hand, if there was some way to know what was actually being sent, the decision might be different.   Let's look at the two things you listed above?   On the media player one it just says 'Player usage data'.   What the heck does that mean?   The songs I'm listening to?  The websites they come from?  How much time I'm using the media player?   Nope, sorry, you haven't even begun to justify needing that info.  On the other hand, the dialog for visual studio is better.  It gives me an idea of the data being sent.   But I'll still likely say no, because I don't believe for a minute that it's really anonymous.    It can't be, otherwise it can't keep tying the info from me into the same records.   Sure, it may be using the CPUID or something that's not easily identifiable as me by name, but all that requires is a leak somewhere else.

But Microsoft isn't the worst here, by a long shot.   One of my dell boxes has a craplet on it by default that phones home and looks for updates of things I have installed.   It's not a work box, strickly a gaming one, so I let it stay installed.  This is useful, because it lets me know about things like flash updates and such that I should track down for my other boxes.  About once a month it will pop up asking me to ok sending usage info to them, with the default set to 'yes'.   That really annoys me, because if you ask me, and I say no, no means no.   Don't ever ask me again.

Dean Harding on 7 Sep 2007 9:22 PM:

While the Office 2007 betas were doing the rounds, there were a couple of blog posts by people on the Office team about how the used their "usage data" to help drive the design of the office ribbon.

For example, they found out that "superscript" and "subscript" were very common operations that people performed; even though they were buried in the Format->Font menu item somewhere. So now superscript and subscript are right there on the "Home" tab!

I thought that kind of thing was pretty cool, and as a result, I'm quite happy to provide my "usage data" to the office team, because I know they're actually paying attention (not that I'm a big user of Office).

On the other hand, I *won't* send my usage data to the windows media player team... so I guess I live somewhere in between the "NEVER send" and "ALWAYS send" :-)

Jan Kučera on 8 Sep 2007 7:14 AM:

Yeas Dean, I really like how the Office team used their data. This is what makes me believe this kind of feedback can make the product better. A very little of results is also published at http://blogs.msdn.com/jensenh/archive/2006/04/07/570798.aspx for example.

And for the privacy? Well, I just trust it. If the dialog wasn't done by Microsoft I'd probably choose No, in very most of cases. But I believe that Microsoft uses the data for what they say. Maybe dewy, maybe not.

Kalle Olavi Niemitalo on 9 Sep 2007 4:02 AM:

"I'm the sort of person who spends less time on microsoft.com now that I have to log in to look at KB articles"

http://support.microsoft.com/kb/941761/ seems to have been published last week, and I can read it without logging in.  I don't have any cookies whatsoever in my browser.  So is this login requirement something only imposed on Microsoft employees?

Michael S. Kaplan on 9 Sep 2007 8:12 AM:

Maybe only on people who are already logged in? I was on messenger at the time....


go to newer or older post, or back to index or month or day