Musing about Software Testing

I’ve spent the last 3 days at International Symposium on Software Testing and Analysis (ISSTA).

Always interesting to catch up on the researchers of software testing.  The details of static analysis can be deep and beyond me, but most of it is quite approachable.  For example, A Metric for Software Readability abstract is only at the junior college level of readability J .

My own groups work was summarized in the keynote (The Real Value of Testing) as the only bright light in the failure of specification based development including models.   As I suspected, electronic voting for elections is still a hoax (details: Are Your Votes Really Counted? Testing the Security of Real-world Electronic Voting Systems). 

The big new term I learned that I’m behind the times on is “Concolic Testing”.   From Concolic Testing tutorial (2007):

Concolic testing automates test input generation by combining the concrete and symbolic (concolic) execution of the code under test. Traditional test input generation techniques use either (1) concrete execution or (2) symbolic execution that builds constraints and is followed by a generation of concrete test inputs from these constraints. In contrast, concolic testing tightly couples both concrete and symbolic executions: they run simultaneously, and each gets feedback from the other.

Concolic appears to be the future for automated test input generation with numerous papers on this topic showing variations.  Concolic is a catchier name than Directed Assisted Random Testing (DART) or Dynamic Symbolic Execution (DSE).

An intern in our group, Xiao Qu, presented her paper Configuration-Aware Regression Testing: An Empirical Study of Sampling and Prioritization which use combinatorial interaction testing techniques to model and generate configuration samples for use in regression testing.   She has been helping to introduce combinatorial testing for parameters into Spec Explorer.   Configuration testing is my favorite use of PICT.   There result:

By running the same set of test cases using additional configurations, we were able to uncover faults previously undetected.

Several object oriented software measures for Java can be measured with varying outputs for the same measure from different tools.  Moral of Comparing Software Metrics Tools is you can’t trust a single tool as it may mis-guide you if you don’t fully and completely understand deep, detailed little assumptions.

Researchers hunger for real industrial data.  AFID: An Automated Fault Identification Tool was written to automatically gather test cases and bugs from evolving software for empirical study by researchers.  Many researchers compared based on the 7 programs in the standard “Siemens Benchmark Suite”.   Small programs listed in numerous papers at this and other conferences.

[See Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques for more details]

At the follow on State-space Exploration for Automated Testing (SSEAT) workshop

I spoke with Myra Cohen who works on combinatorics and would like information about typical application of pair-wise testing in industry.  What type of domain to you use pair-wise testing in?  (e.g. API, GUI, Configurations),   How many factors do you consider in a single pair-wise problem?   How many values for the factor with the most values?     How many constraints?   My own experience using it for configuration testing of Indigo configurations was about 20 factors with no more than 10 values per factor.

I found one article in the poster session of great interest.  I’m a big believer in test set minimization for regression testing.  Most methods up to now are heuristic (e.g. greedy algorithm).    MINTS: A General Framework and Tool for Supporting Test-suite Minimization (and paper) attempts to find the optimal solution (using constraint solving of binary Integer linear programming) including multiple weighted and prioritized criteria like code coverage, time to execute, defect finding history.   Their data indicates

MINTS was able to generate optimal test suites in only a few seconds, a time comparable to that of heuristic (i.e., suboptimal) approaches.

For small test suites (the Siemens 7) it works as fast as the heuristic techniques.

From http://hbswk.hbs.edu/item/5869.html  (Harvard Business school - Published: April 14, 2008, Author: Martha Lagace)

Software analysts and programmers live to innovate—but hate to run tests. Yet top-notch testing saves many a company money when bugs are caught early.

I hear complaints all the time from testers who have to run routine tests over and over again.  If that is the task a company gives testers then the testers need to change something.  

If the tests are repetitive and needed -- automate them.  Humans shouldn't repeat what computers can do.

If the tests are repetitive and not needed -- convince management that they are not needed and stop running them!

If management can't be convinced to avoid repetitive, unnecessary, and especially difficult to automate tests -- suggest they use the Danish consultancy in the article.

Testers should be creative - either in their automation or in their exploration of the system under test.  Repeatedly run tests typically have low bug finding ability.  Management likes the idea of avoiding regressions by repeatedly running the same tests.  Regression tests should be automated.  

Many products get released with low regressions (from running the same tests), but still with high bug rates!  Because they spend too much time avoiding regressions and not enough time finding bugs.

Click here for the best of 30,000 pages of never-before-seen insight into the Redmond lifestyle.