Recently, several aspects of my teams work has become public.   One aspect is the demonstration of scenarios listed in System Overview Documents.    The System Overview Documents describe a family of protocols or task(s) and connect them together.  A part of the document gives actual scenarios.   Our test team demonstrates the accuracy of the scenarios by reproducing them on real windows systems and using Netmon 3.3 captures of the traffic.   We then use a new Netmon 3.3 feature called comments to annotate the messages captured that relate to the scenario.   You can learn more at http://codeplex.com/SysDocCap

Another aspect is the verification of Technical Documents using test suites (frequently model-based).  The test suites are designed to be implementation agnostic and with simple implementations of an interface portable across different implementations.   This has been demonstrated at periodic plugfests that Microsoft holds for licensees that want to verify interoperability of their implementations with Microsoft and other implementations.   Classic examples of this are the SMB2 protocol with Samba (snia plugfest)  or Kerberos Protocol Extensions with MIT Kerberos.    A set of presentations from a System Overview Document perspective was the Active Directory Plugfest.

As last night’s SASQAG meeting, James Whittaker again presented his Future of Testing talk (GTAC version still not on YouTube).

A statement near the end piqued me. 

            "Users will be genuinely surprised when it fails."

I think James is tainted by his past experiences.  There are past and current companies that already work like that.  My first job out of college at Tandem Computers warped my expectations forever.   As a fault tolerant computer company, not failing (hardware or software) was important.   In my first week there I found a trivial bug in their editor with a recursive macro called that crashed.  I was told to report the bug and get it fixed.  That was the attitude in general, all bugs reported and generally all bugs fixed.  There was of course triage and minor bugs deferred until next release, but I never saw bug tracking systems for tens of thousands of bugs until I came to Microsoft.   Long ago Tandem saw the criticality of avoiding data corruption and a fail fast philosophy.  Code reviews of the kernel were keen on what could go wrong.  Many others from that crucible ended up at Microsoft with an expectation for quality in their soul.  Many worked on Microsoft SQL with an expectation for outstanding quality.

The Agile community with their “Test Driven Development” and automated regression runs every check in help instill a quality expectation in developers.  But many already had it.  I have my Software Quality Engineering big red buttons from the mid 1980’s – “TEST then CODE”.  My favorite was a developer who bet me a dinner I couldn’t find a bug in his code.   Why the challenge?  Mutual respect and improvement.   I didn’t take the challenge.  Why?  The developer was good; they had done the right things.  I was not currently on their project (but had been months previously).    I could probably find a bug, but it would probably cost me my entire weekend and that wasn’t worth a dinner to me.   But the real point is the proper confidence of the developer.  Not arrogance or bravado.  He knew if I worked hard enough I would find a bug and he would be happy to receive it and improve his product.  To him, a dinner was a small price to pay for a better product.

 

My expectation is always that when I am asked to test a developer's work product it is because they believe it is ready to ship to the best of their ability.  They’ve code reviewed (peer review, pair programming, etc.) it and unit tested and run the standard tests available to them.  The only bugs I should be finding are those requiring resources outside the scope of a single developer, such as large scale stress testing or full system (integration) testing.

 

I recently acquired the “Software Testing at a Glance – or two” poster  (A3 PDF here) from Delta Axiom. From  http://www.deltaaxiom.com/C12571E70035EE18/0/835FB67AB3628800C12573E1002A9EAC you can order poster for Free.

The poster (1mx70cm), a gift from Specialisterne, is hanging on my wall.

 

Naturally I have a few issues with it, but it is the widest smorgasbord on testing I’ve ever seen.   I like it and agree with probably 90% of it.

“Based on ISTQB Certified Tester Advanced Syllabus 2007 and references therein (BS-7925-1:1998, IEEE Std. 610-1999; IEEE Std 829-1998).”

 

I had no idea what the (SPICE) ISO 15504 – result profile was telling me   Has anybody used SPICE or now what this diagram says?  [See http://www.westfallteam.com/Papers/Making_Sense_of_15504.pdf page 6 for color version of diagram] [I would replace with something more meaningful].

 

I don’t agree that all of their “Useful testing measurements” are useful.   Number of test cases isn’t very meaningful to me.

 

The PDF is a picture and doesn’t scale up well to read bottom left corner.  The circle reads “Management and Support” with (Left to Right): Blue: Idea & Requirements, Green: Design, Lime: Implementation; Orange: Test, Red: Manufacturing & Operations

Grigori’s interview of me on Spec Explorer for Visual Studio is online at TechEd Online Tech talks (Search for “stobie”).  Although the score for my TechEd talk, , and Q&A Session, , rate only just below and above average, they generated some excitement.   Several found the talk interesting and one person said “this session really was revolutionary for me.”  The video of the talk won’t be publicly available for about 6 months.  We also get several people interested and apply for the Spec Explorer for Visual Studio Early Adopter Program (EAP).

I just visited Microsoft’s Development Center Copenhagen (MDCC) and presented Model-Based Testing and Spec Explorer to many or our Dynamics testers.  Several of them have already explored MBT and some have dabbled with Spec Explorer already.  

Today I met with Thorkil Sonne from Specialisterne whom I heard speak and met at StarWest 2008.   I first heard about his company many months ago and sniggered at their use of “software testers” being almost the opposite of what I and most of Microsoft looks for in software testers.  However, my narrow view was, of course, misguided.   There will always remain mundane manual tasks in software testing (e.g. verifying the installation instructions work correctly – you must very carefully and precisely follow the instructions – not an automation task).  Everything about the company, from the name through logo has been carefully chosen.   I can only hope he succeeds in franchising one million of his testers worldwide.

I will be presenting at TechEd EMEA 2008 Developers this year in Barcelona, Spain.

 

Grigori Melnik will interview me about Spec Explorer 2007 in the Fish Bowl for a recorded TechTalk on Tuesday, 11 November 16:00 - 16:30.  We will be discussing What’s new about Spec Explorer for Visual Studio vs older versions (SE 2004)? How have people been successful using it?

 

Thursday, 13 November 9-10:15 in DVP302:  Automating Test Creation I’ll be introducing Spec Explorer 2007.  Here is the abstract:

Most unit tests, functional tests, etc. are hand crafted by people anticipating expected results. With model-based testing (MBT), developers use their skills to create a model that generates both the sequences of calls and their expected results. Models also allow analysis for expected properties and behaviors before the production code is written. Just as test first development can help expose testability issues, model-based approaches can show design issues up front. This talk will show the Spec Explorer plug in for Visual Studio that has been used to verify over 50 of the Windows Protocols and the experiences of product teams using MBT. Come to this session to learn how to find unexpected consequences up front.

 

I follow that up on Thursday afternoon, 13Nov 15:15-16:30 with an interactive session DVP01-IS Model Based Testing with Spec Explorer

Building on the Automating Test Creation break out session, this session will go into more depth about the Visual Studio Team System plug-in, Spec Explorer. Spec Explorer can be used for both model exploration (e.g. checking Liveness or Safety properties with anit-scenarios) and for automatic test generation. Details about parameter generation, state comparison and debugging, and best patterns and practices for adapter design will be covered. Pitfalls new modelers frequently stumble over will be demonstrated with plenty of time for Q&A. In this session, discover through theory, demonstration and best practices discussion, how to enrich the design of any model.

The Visual Studio plug in, Spec Explorer 2007, is using an Early Adopter Program (EAP) to get feedback.  For more information about the EAP you can mail to speccon at microsoft.com.

 

In Two Mistakes and Error-Free Software: A Confession, Robert Glass asks:

how often do missing logic and combinatoric considerations actually happen in the software real world?

and answers with his own observation:

35% of the errors involved omission of required logic

40% could have been found only if the right combination of segments had been executed.

In other words, in this case, the Test Coverage Analyzer could have helped find only 25% of the errors!

In another study of his:

60% were cases of omitted logic. Another 23% were cases of failure to reset data.

Concluding with

the most persistent and therefore most troublesome kinds of errors are the same ones the Test Coverage Analyzer couldn’t help us with!

 

This is some of the better support I’ve found for constantly questioning managers or others that blindly push for higher code coverage via more testing, and especially automated testing.  

 

Always ask:  Where are our customers finding bugs? 

          In the already covered code or the uncovered code?

With good defect tracking and source control systems, check in of fixes can be linked and you can see how many fixes are in areas not covered by tests versus those covered by tests.

The few times I’ve seen people do the analysis (even if just informally), they (not surprisingly to me), find that most of the customer reported defects are in already covered code!   So why, would a test professional think that spending their time covering more code will help their customers find fewer defects?  

 

Testers don’t spend enough time choosing the most appropriate method and technique for their problem and situation.  Too often untrained testers knowing only one (or perhaps a handful) of techniques and tools, use what they know which may not be all that applicable.

 

My current focus is on state coverage, as opposed to code coverage.  States can involve the combinatorics and with negative testing give indications of missing logic.

I’ve spent the last 3 days at International Symposium on Software Testing and Analysis (ISSTA).

Always interesting to catch up on the researchers of software testing.  The details of static analysis can be deep and beyond me, but most of it is quite approachable.  For example, A Metric for Software Readability abstract is only at the junior college level of readability J .

My own groups work was summarized in the keynote (The Real Value of Testing) as the only bright light in the failure of specification based development including models.   As I suspected, electronic voting for elections is still a hoax (details: Are Your Votes Really Counted? Testing the Security of Real-world Electronic Voting Systems). 

The big new term I learned that I’m behind the times on is “Concolic Testing”.   From Concolic Testing tutorial (2007):

Concolic testing automates test input generation by combining the concrete and symbolic (concolic) execution of the code under test. Traditional test input generation techniques use either (1) concrete execution or (2) symbolic execution that builds constraints and is followed by a generation of concrete test inputs from these constraints. In contrast, concolic testing tightly couples both concrete and symbolic executions: they run simultaneously, and each gets feedback from the other.

Concolic appears to be the future for automated test input generation with numerous papers on this topic showing variations.  Concolic is a catchier name than Directed Assisted Random Testing (DART) or Dynamic Symbolic Execution (DSE).

 

An intern in our group, Xiao Qu, presented her paper Configuration-Aware Regression Testing: An Empirical Study of Sampling and Prioritization which use combinatorial interaction testing techniques to model and generate configuration samples for use in regression testing.   She has been helping to introduce combinatorial testing for parameters into Spec Explorer.   Configuration testing is my favorite use of PICT.   There result:

By running the same set of test cases using additional configurations, we were able to uncover faults previously undetected.

 

Several object oriented software measures for Java can be measured with varying outputs for the same measure from different tools.  Moral of Comparing Software Metrics Tools is you can’t trust a single tool as it may mis-guide you if you don’t fully and completely understand deep, detailed little assumptions.

 

Researchers hunger for real industrial data.  AFID: An Automated Fault Identification Tool was written to automatically gather test cases and bugs from evolving software for empirical study by researchers.  Many researchers compared based on the 7 programs in the standard “Siemens Benchmark Suite”.   Small programs listed in numerous papers at this and other conferences.

[See Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques for more details]

At the follow on State-space Exploration for Automated Testing (SSEAT) workshop

I spoke with Myra Cohen who works on combinatorics and would like information about typical application of pair-wise testing in industry.  What type of domain to you use pair-wise testing in?  (e.g. API, GUI, Configurations),   How many factors do you consider in a single pair-wise problem?   How many values for the factor with the most values?     How many constraints?   My own experience using it for configuration testing of Indigo configurations was about 20 factors with no more than 10 values per factor.

 

I found one article in the poster session of great interest.  I’m a big believer in test set minimization for regression testing.  Most methods up to now are heuristic (e.g. greedy algorithm).    MINTS: A General Framework and Tool for Supporting Test-suite Minimization (and paper) attempts to find the optimal solution (using constraint solving of binary Integer linear programming) including multiple weighted and prioritized criteria like code coverage, time to execute, defect finding history.   Their data indicates

MINTS was able to generate optimal test suites in only a few seconds, a time comparable to that of heuristic (i.e., suboptimal) approaches.

For small test suites (the Siemens 7) it works as fast as the heuristic techniques.

 

From http://hbswk.hbs.edu/item/5869.html  (Harvard Business school - Published: April 14, 2008, Author: Martha Lagace)

Software analysts and programmers live to innovate—but hate to run tests. Yet top-notch testing saves many a company money when bugs are caught early.

The majority of a Danish consultancy's testers have Asperger syndrome or a form of autism spectrum disorder.
• Software testing requires superb powers of concentration combined with tolerance (even preference) for routine tasks.

I hear complaints all the time from testers who have to run routine tests over and over again.  If that is the task a company gives testers then the testers need to change something.  

 

If the tests are repetitive and needed -- automate them.  Humans shouldn't repeat what computers can do.

If the tests are repetitive and not needed -- convince management that they are not needed and stop running them!

If management can't be convinced to avoid repetitive, unnecessary, and especially difficult to automate tests -- suggest they use the Danish consultancy in the article.

 

Testers should be creative - either in their automation or in their exploration of the system under test.  Repeatedly run tests typically have low bug finding ability.  Management likes the idea of avoiding regressions by repeatedly running the same tests.  Regression tests should be automated.  

 

Many products get released with low regressions (from running the same tests), but still with high bug rates!  Because they spend too much time avoiding regressions and not enough time finding bugs.

 

 

 

Click here for the best of 30,000 pages of never-before-seen insight into the Redmond lifestyle.

News Flash -- my semi-secret project has just been outed!    You can read the press announcement etc.

I work at verifying the usability and accuracy of protocol documents which are now all publicly available on MSDN.

Some of the things my team facilitates:

Sorry, my blog entries have gotten behind.   Last November I was re-elected to the PNSQC board (by the board - since they forgot to ask me to be on the ballot) and then elected me Vice President!   That got me heading a sorbanes-oxley derived committee on document retention.

The PNSQC board, chairs, and voluntees did have an interesting and productive strategic offsite at the Heathman lodge in late January.  You should see an updated mission statement soon.

 

Finally, in case you missed it, the Call for Papers for this October's Pacific Northwest Software Quality Conference (PNSQC) is open.  The theme is Collaborative Quality.

PNSQC 2007 conference has the 2007 [10.5 MB]proceedings posted now.  
You can also see my  slide deck about Model Based Testing For Protocols (PDF),using Spec Explorer that I presented.  The talk got very mixed reviews.   A few astute people noted that my Web Services testing paper (the talk got good reviews) had the following reference:

6] Keith Stobie, Service Compatibility

<to appear on msdn Tester Center site>

 

What is Tester Center?

The Microsoft Tester Center showcases the test discipline as an integral part of the application lifecycle, describes test roles and responsibilities, and promotes the test investments required to deliver high quality software.  With the new Microsoft Tester Center, Microsoft can not only share its own experiences and lessons learned, but also provide the testing community a new way to engage with one another.

This is in addition to the MSDN forums set up for discussing software testing issues.

 

I recently updated the Service Compatibility paper responding to reviewers (AlanPa) comments and it should be posted soon under Technical Articles or in the Library. 

 

After PNSQC and QSIC, I went to Hyderabad India for 2 weeks.  One of the days there I visited the Visual Studio Test Team working on VSTS2008 and beyond.  I gave them a demo of Spec Explorer for Visual Studio and they liked it.  We discussed making it a power tool or otherwise distributed.  The wheels are in motion to get the licensing worked out and publicly released.

I was in Beijing for 3 weeks and negligent in updating by blog.

 

I presented at PNSQC.  My testing web services talk was well received.  The paper should appear soon in the proceedings.  I also stepped in to give a talk in place of a speaker who withdrew.  The talk on Model Based Testing of Protocol Specifications was less well received.  The slides should be on the PNSQC site shortly.

I attended the SPIN meeting with an excellent talk by Niels Malotaux about estimation with a simple, but great exercise. 

At the lunch panel, Dick Hamlet indicated to be on the look out for two phrases:  "Adaptive Random Testig" (ART) and "Partial Oracles".  I persued them a bit at QSIC07 and don't think they are ready for prime time.  ART has mostly been researched only for numerical domains and the major question (which perplexed the last WHET 4 Workshop on boundary testing also) is how to measure distance or closeness of inputs in a non-numerical domain.

Partial Oracles is related to metamorphic testing which is perhaps more problematic than I realized.  Since there are so many possible partial oracles, how do you choose which ones?

 

I found QSIC07 far more useful to me than I expected.  I particularly like Gordon Fraser's talk (Improving Model-Checkers for Software
Testing) on generating tests from Models.  He introduced several concepts I would like to follow up on.  Mainly he took ideas I've seen applied to implementations and applied them to models.  In particular for example, as given by Jean Hartman's talk at PNSQC, is prioritizing tests.  Code coverage is a traditional way to do this for implementations.  We can do the same for tests of models.  Which tests cover the most states, transitions, etc. 

Even more exciting was the thought of determining the most powerful tests.  Kaner defines test B as more powerful than test A if test B can detect all the defects of test A and other defects besides.  Using the old implementation concept of mutation testing, we can mutate our models and see which tests detect this.   Mutation of implementations suffers from the mutation possibly causing just another legal implementation of the behavior expressed differently.  It is an undecidable computer problem to determine if two programs have the same behavior.  However with model mutation, Gordon claims it is decidable, and thus an even more powerful technique.