Pairwise TestCase

From Wikipedia...
  • All-pairs testing or pairwise testing is a combinatorial software testing method that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters.
  • The reasoning behind all-pairs testing is justified: the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing.[1] Bugs involving interactions between three or more parameters are progressively less common, whilst at the same time being progressively more expensive to find by exhaustive testing, which has as its limit the exhaustive testing of all possible inputs.
  • Many testing methods regard all-pairs testing of a system or subsystem as a reasonable cost-benefit compromise between often computationally infeasible higher-order combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters.

A sample scenario
Let us say, you are testing whether the printout of your target web-page from the browsers is perfect or not, and let us assume following are the Test Parameters across which you need to cover the tests-

OS:Win XP,Redhat Linux,Macintosh
Browser:IE 6,IE 7,Firefox,Opera
Orientation:Portrait,Landscape
Size:A3,A4,A5
Color:Mono,Multicolor

The total combinations that need to be tested are 3 operating systems X 4 browsers X 2 types of Orientation X 3 sizes X 2 types of Color, that is, 144.

Our Pairwise Test Case Generator takes the above input, prepares a list of "all pairs" of values from the Test Parameters and packages them in the least possible number of test cases/combinations. In this specific case we are able to create 13 test cases that cover all the pairs. Check it in the below list.

OS,Browser,Mode,Size,Color
Win XP,IE 6,Portrait,A3,Mono
Redhat Linux,IE 6,Landscape,A4,Multicolor
Macintosh,IE 6,Portrait,A5,Multicolor
Win XP,IE 7,Landscape,A4,Mono
Redhat Linux,IE 7,Portrait,A3,Multicolor
Macintosh,IE 7,Landscape,A5,Mono
Win XP,Firefox,Portrait,A5,Multicolor
Redhat Linux,Firefox,Landscape,A3,Mono
Macintosh,Firefox,Portrait,A4,any_value_of_Color
Win XP,Opera,Landscape,A3,Mono
Redhat Linux,Opera,Portrait,A5,Multicolor
Macintosh,Opera,any_value_of_Mode,A4,any_value_of_Color
Macintosh,any_value_of_Browser,any_value_of_Mode,A3,any_value_of_Color

That is the power of pairwise testing, and we have a working solution freely available online. Several individuals and organizations around the world use it on a regular basis.

It has reportedly reduced the testing efforts in many organizations and also technically achieved more code coverage than usual. The practical approach is to perform testing on a feature's pairwise Test Configurations as well as knowledge-based configurations. Knowledge-based configurations would mean selecting (or deselecting) the values of the Test Parameters based on your context.

Right now we accept a maximum of 20 Test Parameters. Each of them can have any number of values as long as the average across parameters does not cross 10.

Capability metrics

Following is a list of test metrics we have gathered to demonstrate the capability of pairwise generator:

Please read, for example, 324521 as "two parameters with three values each", "five parameters with four values each" and "one parameter with two values".

S.No Parameters and Values 100% coverage reduced to
1 322211 36 9
2 104 74 83 62 32 52 42 91 14338695168000000 144
3 1015 1000000000000000 199
4 105563525 2430000000000 142
5 33425262718210191 15676416000 96
Share