The Software Tester's Axioms

by Michael S. Kaplan, published on 2005/11/12 01:01 -08:00, original URI: http://blogs.msdn.com/michkap/archive/2005/11/12/491519.aspx

Like any science (and please do not for a moment doubt that Test is an engineering discipline, and at heart a good engineer is an applied scientist!), software testing has some basic principles which cannot be proven but are simply universal truths upon which other theorems can be based.

(The original version of these was put up years ago by me in a newsgroup, but after many reformulations, the current ones have stayed stable for some time. There have been other corollaries and theorems proposed but I have not captured all of them here)

Here are the Tester's axioms:

Axiom I: If you have not tested it, assume it is broken.

Axiom II: If someone changes it, they probably broke it.

Axiom III: Sometimes when it is not changed, it breaks too.

Axiom IV: What is broken on the test machine often works fine on the dev machine.

Axiom V:There is always another bug that a user can find.

And there is of course the first theorem of test, which is largely based on applying on the developer's doubt of Axioms 2, 3, and 5 created by the circumstances of Axiom 4:

THEOREM I: Developers will often not believe there is a bug until it is proven to them, sometimes with extreme prejudice.

And of course the corollary to Axiom V:

There are at least as many bugs left to be found as there are users who haven't yet run your code.

 

(Special thanks to David Fenton for pointing out the corollary to Axiom V, years ago!)

# Maurits on Saturday, November 12, 2005 5:50 AM:

Theorem II: Developers believe (and sometimes even assume) that bugs "fix themselves" between builds

# Jonathan Wilson on Saturday, November 12, 2005 6:36 AM:

Theorem III: No developer can ever fix all the bugs in their own code on their own. Having one other experienced person go over your code once is significantly better than you reading your code many times over looking for bugs.

# Michael S. Kaplan on Saturday, November 12, 2005 9:30 AM:

Hi Maurits --

Ok, I agree with the principle of Theorem II -- though it may be a corollary of Axioms II, III, and IV? :-)

But then it does also depend on Theorem I, so perhaps it is a theorem.....

# Michael S. Kaplan on Saturday, November 12, 2005 9:32 AM:

Hi Jonathan --

I agree with the statement, but I am not sure we can call it a theorem of Test without working on the specific language a bit (as currently worded it seems more like a dev-focused item).

# Serge Wautier on Saturday, November 12, 2005 9:52 AM:

The more a bug is difficult to identify, the more valuable it is. Because the more other unknown bugs it will help uncover during its debugging.

[Speaking of bug, can somebody pleeease help the 'Remember me' checkbox actually remember me...]

# Nicholas Allen on Saturday, November 12, 2005 10:52 AM:

I don't like the third axiom at all.

If something breaks and nothing changed, you just don't KNOW what changed.

# Marvin on Saturday, November 12, 2005 1:14 PM:

Theorem VI: The more bugs you have found in a module, the more bugs there are left undetected.

Corrolary: If you have two modules: one that had a few bugs in the past and another that had a lot, spend more time testing the second one.

# Michael S. Kaplan on Saturday, November 12, 2005 1:29 PM:

Hi Nick -- the implication of the third axiom (which perhaps should be recast to include include this info) is that sometimes external changes cause a break -- so even if the code is not changed, something can break it.

# Michael S. Kaplan on Saturday, November 12, 2005 1:30 PM:

Marvin, I love it! And I get your Theorem IV (we aee only up theorem IV so far!) pointed out by example all the time. :-)

# Nicholas Allen on Saturday, November 12, 2005 2:21 PM:

Michael- Yeah, that's totally right. I'm just being pedagogical. A lot of software bugs get papered over instead of fixed because programmers don't think about what's going on outside their code.

# Michael S. Kaplan on Saturday, November 12, 2005 4:39 PM:

I think your point is a good one though. How about changing from this:

"Sometimes when it is not changed, it breaks too."

to this:

"Sometimes when it is not changed, external factors can break it, anyway."

# Marvin on Saturday, November 12, 2005 9:30 PM:

It was supposed to be Axiom VI. No idea why I typed theorem ;-)

Seriously this fact is supported by statistics and is often mentioned in books about software testing (I once read a few for general education). I also had many chances to see how true it is.

# Michael S. Kaplan on Saturday, November 12, 2005 9:40 PM:

Ah, that may take more thought (it is not that I disagree, it is that we have to be cautious about adding new axioms to a system!).

# Jerry Pisk on Sunday, November 13, 2005 3:27 AM:

I don't think the third axiom talks about external factors. I think it simply points out the fact that even code that did not change has bugs that you didn't find earlier. An external factor might have exposed the bug instead of causing it.
Please consider a donation to keep this archive running, maintained and free of advertising.
Donate €20 or more to receive an offline copy of the whole archive including all images.

referenced by

2012/05/02 Who owns keyboard testing?

2011/04/16 Chain Chain Chain, Chain of Dead Keys

2010/04/09 A difference that makes no difference, makes no difference (well, other than the bugs you cause "fixing" it, that is!)

2008/06/29 Did someone break GetTimeZoneInformation in XP?

2008/06/18 How Do You Solve a Problem Like Meiryo? (Windows XP edition)

2008/01/17 Expertise isn't always everything (aka When the one who is learning teaches us something important)

2007/07/13 GetLocaleInfo for other languages?

2007/01/01 Converting a Project to Unicode: Part 5 (Are we there yet? Well, not *just* yet)

2006/07/02 Support of 'Cross Language Support'

2006/06/11 Death of a Data Access Page Wizard

2006/04/12 Getting all you can out of a keyboard layout, Part #9a

2006/03/30 If at first you don't succeed, there's probably still a bug

2006/02/22 Keep off the grass

go to newer or older post, or back to index or month or day