Recently, several aspects of my teams work has become public.   One aspect is the demonstration of scenarios listed in System Overview Documents.    The System Overview Documents describe a family of protocols or task(s) and connect them together.  A part of the document gives actual scenarios.   Our test team demonstrates the accuracy of the scenarios by reproducing them on real windows systems and using Netmon 3.3 captures of the traffic.   We then use a new Netmon 3.3 feature called comments to annotate the messages captured that relate to the scenario.   You can learn more at http://codeplex.com/SysDocCap

Another aspect is the verification of Technical Documents using test suites (frequently model-based).  The test suites are designed to be implementation agnostic and with simple implementations of an interface portable across different implementations.   This has been demonstrated at periodic plugfests that Microsoft holds for licensees that want to verify interoperability of their implementations with Microsoft and other implementations.   Classic examples of this are the SMB2 protocol with Samba (snia plugfest)  or Kerberos Protocol Extensions with MIT Kerberos.    A set of presentations from a System Overview Document perspective was the Active Directory Plugfest.

As last night’s SASQAG meeting, James Whittaker again presented his Future of Testing talk (GTAC version still not on YouTube).

A statement near the end piqued me. 

            "Users will be genuinely surprised when it fails."

I think James is tainted by his past experiences.  There are past and current companies that already work like that.  My first job out of college at Tandem Computers warped my expectations forever.   As a fault tolerant computer company, not failing (hardware or software) was important.   In my first week there I found a trivial bug in their editor with a recursive macro called that crashed.  I was told to report the bug and get it fixed.  That was the attitude in general, all bugs reported and generally all bugs fixed.  There was of course triage and minor bugs deferred until next release, but I never saw bug tracking systems for tens of thousands of bugs until I came to Microsoft.   Long ago Tandem saw the criticality of avoiding data corruption and a fail fast philosophy.  Code reviews of the kernel were keen on what could go wrong.  Many others from that crucible ended up at Microsoft with an expectation for quality in their soul.  Many worked on Microsoft SQL with an expectation for outstanding quality.

The Agile community with their “Test Driven Development” and automated regression runs every check in help instill a quality expectation in developers.  But many already had it.  I have my Software Quality Engineering big red buttons from the mid 1980’s – “TEST then CODE”.  My favorite was a developer who bet me a dinner I couldn’t find a bug in his code.   Why the challenge?  Mutual respect and improvement.   I didn’t take the challenge.  Why?  The developer was good; they had done the right things.  I was not currently on their project (but had been months previously).    I could probably find a bug, but it would probably cost me my entire weekend and that wasn’t worth a dinner to me.   But the real point is the proper confidence of the developer.  Not arrogance or bravado.  He knew if I worked hard enough I would find a bug and he would be happy to receive it and improve his product.  To him, a dinner was a small price to pay for a better product.

My expectation is always that when I am asked to test a developer's work product it is because they believe it is ready to ship to the best of their ability.  They’ve code reviewed (peer review, pair programming, etc.) it and unit tested and run the standard tests available to them.  The only bugs I should be finding are those requiring resources outside the scope of a single developer, such as large scale stress testing or full system (integration) testing.

I recently acquired the “Software Testing at a Glance – or two” poster  (A3 PDF here) from Delta Axiom. From  http://www.deltaaxiom.com/C12571E70035EE18/0/835FB67AB3628800C12573E1002A9EAC you can order poster for Free.

The poster (1mx70cm), a gift from Specialisterne, is hanging on my wall.

Naturally I have a few issues with it, but it is the widest smorgasbord on testing I’ve ever seen.   I like it and agree with probably 90% of it.

“Based on ISTQB Certified Tester Advanced Syllabus 2007 and references therein (BS-7925-1:1998, IEEE Std. 610-1999; IEEE Std 829-1998).”

I had no idea what the (SPICE) ISO 15504 – result profile was telling me   Has anybody used SPICE or now what this diagram says?  [See http://www.westfallteam.com/Papers/Making_Sense_of_15504.pdf page 6 for color version of diagram] [I would replace with something more meaningful].

I don’t agree that all of their “Useful testing measurements” are useful.   Number of test cases isn’t very meaningful to me.

The PDF is a picture and doesn’t scale up well to read bottom left corner.  The circle reads “Management and Support” with (Left to Right): Blue: Idea & Requirements, Green: Design, Lime: Implementation; Orange: Test, Red: Manufacturing & Operations

Grigori’s interview of me on Spec Explorer for Visual Studio is online at TechEd Online Tech talks (Search for “stobie”).  Although the score for my TechEd talk, , and Q&A Session, , rate only just below and above average, they generated some excitement.   Several found the talk interesting and one person said “this session really was revolutionary for me.”  The video of the talk won’t be publicly available for about 6 months.  We also get several people interested and apply for the Spec Explorer for Visual Studio Early Adopter Program (EAP).

I just visited Microsoft’s Development Center Copenhagen (MDCC) and presented Model-Based Testing and Spec Explorer to many or our Dynamics testers.  Several of them have already explored MBT and some have dabbled with Spec Explorer already.  

Today I met with Thorkil Sonne from Specialisterne whom I heard speak and met at StarWest 2008.   I first heard about his company many months ago and sniggered at their use of “software testers” being almost the opposite of what I and most of Microsoft looks for in software testers.  However, my narrow view was, of course, misguided.   There will always remain mundane manual tasks in software testing (e.g. verifying the installation instructions work correctly – you must very carefully and precisely follow the instructions – not an automation task).  Everything about the company, from the name through logo has been carefully chosen.   I can only hope he succeeds in franchising one million of his testers worldwide.

I will be presenting at TechEd EMEA 2008 Developers this year in Barcelona, Spain.

Grigori Melnik will interview me about Spec Explorer 2007 in the Fish Bowl for a recorded TechTalk on Tuesday, 11 November 16:00 - 16:30.  We will be discussing What’s new about Spec Explorer for Visual Studio vs older versions (SE 2004)? How have people been successful using it?

Thursday, 13 November 9-10:15 in DVP302:  Automating Test Creation I’ll be introducing Spec Explorer 2007.  Here is the abstract:

Most unit tests, functional tests, etc. are hand crafted by people anticipating expected results. With model-based testing (MBT), developers use their skills to create a model that generates both the sequences of calls and their expected results. Models also allow analysis for expected properties and behaviors before the production code is written. Just as test first development can help expose testability issues, model-based approaches can show design issues up front. This talk will show the Spec Explorer plug in for Visual Studio that has been used to verify over 50 of the Windows Protocols and the experiences of product teams using MBT. Come to this session to learn how to find unexpected consequences up front.

I follow that up on Thursday afternoon, 13Nov 15:15-16:30 with an interactive session DVP01-IS Model Based Testing with Spec Explorer

Building on the Automating Test Creation break out session, this session will go into more depth about the Visual Studio Team System plug-in, Spec Explorer. Spec Explorer can be used for both model exploration (e.g. checking Liveness or Safety properties with anit-scenarios) and for automatic test generation. Details about parameter generation, state comparison and debugging, and best patterns and practices for adapter design will be covered. Pitfalls new modelers frequently stumble over will be demonstrated with plenty of time for Q&A. In this session, discover through theory, demonstration and best practices discussion, how to enrich the design of any model.

The Visual Studio plug in, Spec Explorer 2007, is using an Early Adopter Program (EAP) to get feedback.  For more information about the EAP you can mail to speccon at microsoft.com.