Musing about Software Testing

I' ve been negligent on my blog due to a busy summer.  I spent 4 weeks in Beijing training our Protocol Testers and 2 weeks in Hyderabad training our Protocol Reviewers in Protocol document Quality Assurance Process (PQAP) which was an Honorable Mention at the Microsoft internal Engineering Excellence Awards. 

I’ve developed and submitted a paper for the Pacific Northwest Software Quality Conference (PNSQC) this year:

Too much automation or not enough?
When to automate testing.

Fundamentally test automation is about Return On Investment (ROI).  Do we get better quality for less money by automating or not automating?  The obvious and famous consultant answer is “it depends”.  This paper explores those factors that influence when to choose automation and when to shun it.

Three major factors you must consider:

1)     Rate of change of what you are testing.  The less stable, the more automation maintenance costs.

2)     Frequency of test execution.  How important is each test result and how expensive to get it?

I presented a rough version to Seattle Area Software Quality Assurance Group (SASQAG) last week.  

I highly recommend PNSQC as a great conference and a real value.  This year you can get an even better value by using promotional code of “FOA – Keith Stobie”.  This will give you a $50 discount off of the normal conference attendance fee.

Recently, several aspects of my teams work has become public.   One aspect is the demonstration of scenarios listed in System Overview Documents.    The System Overview Documents describe a family of protocols or task(s) and connect them together.  A part of the document gives actual scenarios.   Our test team demonstrates the accuracy of the scenarios by reproducing them on real windows systems and using Netmon 3.3 captures of the traffic.   We then use a new Netmon 3.3 feature called comments to annotate the messages captured that relate to the scenario.   You can learn more at http://codeplex.com/SysDocCap

Another aspect is the verification of Technical Documents using test suites (frequently model-based).  The test suites are designed to be implementation agnostic and with simple implementations of an interface portable across different implementations.   This has been demonstrated at periodic plugfests that Microsoft holds for licensees that want to verify interoperability of their implementations with Microsoft and other implementations.   Classic examples of this are the SMB2 protocol with Samba (snia plugfest)  or Kerberos Protocol Extensions with MIT Kerberos.    A set of presentations from a System Overview Document perspective was the Active Directory Plugfest.

As last night’s SASQAG meeting, James Whittaker again presented his Future of Testing talk (GTAC version still not on YouTube).

A statement near the end piqued me. 

            "Users will be genuinely surprised when it fails."

I think James is tainted by his past experiences.  There are past and current companies that already work like that.  My first job out of college at Tandem Computers warped my expectations forever.   As a fault tolerant computer company, not failing (hardware or software) was important.   In my first week there I found a trivial bug in their editor with a recursive macro called that crashed.  I was told to report the bug and get it fixed.  That was the attitude in general, all bugs reported and generally all bugs fixed.  There was of course triage and minor bugs deferred until next release, but I never saw bug tracking systems for tens of thousands of bugs until I came to Microsoft.   Long ago Tandem saw the criticality of avoiding data corruption and a fail fast philosophy.  Code reviews of the kernel were keen on what could go wrong.  Many others from that crucible ended up at Microsoft with an expectation for quality in their soul.  Many worked on Microsoft SQL with an expectation for outstanding quality.

The Agile community with their “Test Driven Development” and automated regression runs every check in help instill a quality expectation in developers.  But many already had it.  I have my Software Quality Engineering big red buttons from the mid 1980’s – “TEST then CODE”.  My favorite was a developer who bet me a dinner I couldn’t find a bug in his code.   Why the challenge?  Mutual respect and improvement.   I didn’t take the challenge.  Why?  The developer was good; they had done the right things.  I was not currently on their project (but had been months previously).    I could probably find a bug, but it would probably cost me my entire weekend and that wasn’t worth a dinner to me.   But the real point is the proper confidence of the developer.  Not arrogance or bravado.  He knew if I worked hard enough I would find a bug and he would be happy to receive it and improve his product.  To him, a dinner was a small price to pay for a better product.

My expectation is always that when I am asked to test a developer's work product it is because they believe it is ready to ship to the best of their ability.  They’ve code reviewed (peer review, pair programming, etc.) it and unit tested and run the standard tests available to them.  The only bugs I should be finding are those requiring resources outside the scope of a single developer, such as large scale stress testing or full system (integration) testing.

I recently acquired the “Software Testing at a Glance – or two” poster  (A3 PDF here) from Delta Axiom. From  http://www.deltaaxiom.com/C12571E70035EE18/0/835FB67AB3628800C12573E1002A9EAC you can order poster for Free.

The poster (1mx70cm), a gift from Specialisterne, is hanging on my wall.

Naturally I have a few issues with it, but it is the widest smorgasbord on testing I’ve ever seen.   I like it and agree with probably 90% of it.

“Based on ISTQB Certified Tester Advanced Syllabus 2007 and references therein (BS-7925-1:1998, IEEE Std. 610-1999; IEEE Std 829-1998).”

I had no idea what the (SPICE) ISO 15504 – result profile was telling me   Has anybody used SPICE or now what this diagram says?  [See http://www.westfallteam.com/Papers/Making_Sense_of_15504.pdf page 6 for color version of diagram] [I would replace with something more meaningful].

I don’t agree that all of their “Useful testing measurements” are useful.   Number of test cases isn’t very meaningful to me.

The PDF is a picture and doesn’t scale up well to read bottom left corner.  The circle reads “Management and Support” with (Left to Right): Blue: Idea & Requirements, Green: Design, Lime: Implementation; Orange: Test, Red: Manufacturing & Operations

Grigori’s interview of me on Spec Explorer for Visual Studio is online at TechEd Online Tech talks (Search for “stobie”).  Although the score for my TechEd talk, , and Q&A Session, , rate only just below and above average, they generated some excitement.   Several found the talk interesting and one person said “this session really was revolutionary for me.”  The video of the talk won’t be publicly available for about 6 months.  We also get several people interested and apply for the Spec Explorer for Visual Studio Early Adopter Program (EAP).

I just visited Microsoft’s Development Center Copenhagen (MDCC) and presented Model-Based Testing and Spec Explorer to many or our Dynamics testers.  Several of them have already explored MBT and some have dabbled with Spec Explorer already.  

Today I met with Thorkil Sonne from Specialisterne whom I heard speak and met at StarWest 2008.   I first heard about his company many months ago and sniggered at their use of “software testers” being almost the opposite of what I and most of Microsoft looks for in software testers.  However, my narrow view was, of course, misguided.   There will always remain mundane manual tasks in software testing (e.g. verifying the installation instructions work correctly – you must very carefully and precisely follow the instructions – not an automation task).  Everything about the company, from the name through logo has been carefully chosen.   I can only hope he succeeds in franchising one million of his testers worldwide.