What is N-Wise Testing?

N-wise testing has the aim of testing all the possibilities of any random combination of N factors.
The maximum value for N is equal to the number of parameters. In that case, the result is equal to the testing of the complete decision table: all the combinations of all the values of all the parameters. In practice, a value of 4 or higher is seldom applied. In order to apply N-wise testing tools are required.

For the given table all combinations or 4-wise would result in 4 x 4 x 4 = 64 combinations. Having in mind that we are talking about test environments this is significant number because it will be mutiplier for our tests. To cover all the environments for lets say 1000 tests, this will require 64 000 tests to be executed.

What is Pairwise Testing?

The most common application of N-wise testing is pairwise testing. Pairwise testing is based on the phenomenon that most faults in software are the consequence of one particular factor or the combination of 2 factors. The number of faults that are caused by a specific combination of more than 2 factors becomes exponentially smaller. Instead of testing all the possible combinations of all the factors, it is very effective if every combination of 2 factors is tested.

The aim of pairwise testing is to test all the possibilities of any combination of 2 factors.

There are many pairwise tools available on the market. For the current demo I have picked Microsoft PICT. PICT is a command line tool, there is no UI. Can be installed on any OS but installation is different for MacOs/Linux.

Download and Install Microsoft PICT

Microsoft PICT and many other pairwise tools can be found at Pairwise tools

PICT is number 20 in the list. Download the MSI installer on your local machine. Installing PICT requires just a few clicks. If everything is OK, when running pict command in the console the following response should be displayed: Pairwise Independent Combinatorial Testing

This means that pict was properly installed and environment variable is properly added to the path variable. If not, we have to do it manually in system environment variables.

We can either create new PICT environment variable and then add it in Path environment variable with %PICT% or just add new variable in Path holding the path to PICT.exe.

Creating Input Model for PICT

In order to be able to use PICT for 2-wise combinations, we need to first create an input model in .txt file. PICT uses this model as an input for the combinations.

For the following table of environments:

the input file will look like this:

We use colon(:) to separate parameters and their real values. All the parameter values are split by comma(,)

Executing the Model in PICT

Navigate to the folder where env.txt file is located using: cd path_to_env.txt

Execute the following command in console to get the combinations: pict env.txt

In my case, env.txt file is located on the Desktop so I will navigate to there first and then execute the command.

The result is:

We can redirect the output from pict to xls file so we can manipulate with it easily.

The command for this is: pict env.txt > env-combinations.xls

This will create new xls file with name env-combinations in your current folder. In my case this is Desktop. If you want you can of course provide another path after the redirect > command.

We have reduced the test environment combinations from 64 to 17 without serious impact on the bug detection rate(less than 10%). If we have 1000 test cases for the system, we can save 64000 – 17000 = 47000 unnecessary test cases executions.

 

Excluding the Conflict Sets

If we take a closer look at the combinations we will notice that there are some rows which are impossible(with conflict). The good news is that PICT provides us with an ability to write statements and remove those conflict sets(combinations). For example Edge + MacOS Mojave or Safari + Windows 7/10. Instead of removing them from the combinations which will ruin the idea of pairwise testing, we should find a way to exclude them from the input model somehow.

When we execute the model again, the result will be different.

As you can see from the picture, now MacOS is combined only with Safari. If we see further conflicts we can write more statements to remove them. For example Windows 7 + Edge.

The condition will be something like this: IF[Browser]=”Edge” then [OS]= “Windows 10”

Be aware that PICT is case sensitive about parameters and options. So you either use only small letters or make sure your statement values are exactly the same as they are described in the model above. For example: IF[browser]=”edge” will lead to error message when the model is executed because Edge and Browser are starting with capitals in the model. The keywords are not case sensitive IF, THEN, IN are the same as if, then, in for the tool.