Model-Based Testing with Graph Walker
Functional testing takes huge amount of our time as testers. We try to utilize different approaches to reduce this time as much as possible.
How can model-based testing help us be more effective?
In order to answer this question lets first see what is model-based testing in general.
Model-based testing is an application of model-based design for designing and optionally also executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a system under test (SUT), or to represent testing strategies and a test environment. The picture on the right depicts the former approach.
Why Testing Must Be Model-Based?
Testing can be viewed as search problem. We are looking for those few input and state combinations that will reach, trigger and propagate bugs out of trillions and trillions which wont. Brute force is impotent at this scale. Testing by poking around is a waste of time that leads to unwarranted confidence.
Instead, our search must be systematic, focused and automated.
Revealing defects in the source code can sometimes be very tricky. In order to increase our success we should first understand the link between human mistakes, software defects and software failures. Every successful strategy should take into account so called – ideal fault conditions.
What is a Model?
- A model is description of system behavior
- Models are simpler than the system they describe
- Models help us understand and predict the system’s behavior
Elements of Model
- A model must have a well-defined subject. Example: building, bridge
- Point of view
- A model must be based on a frame of reference and principles that can guide identification of relevant issues and information
- A modeling technique must have a means to express a model (equations, CAD model)
- Models are complex artifacts and skills of modeler matters
What is a State Machine?
A state machine is a system whose output is determined by both current and past input. The effect of previous inputs is represented by a state.
State Machine Building Blocks
- An abstraction that summarizes the information concerning past inputs that is needed to determine the behavior of the system on subsequent inputs
- An allowable two-state sequence. A transition is caused by event.
- An input or an interval of time
- The result or output that follows an event
What is a Directed Graph?
Directed graph(or digraph) is a graph that is a set of vertices connected by edges, where the edges have a direction associated with them.
GraphWalker is an open-source Model-based testing tool for test automation.
It is designed to make it easy to design your tests using directed graphs.
How to use GraphWalker to automate “walking” of directed graphs?
First you need to create models of your system(directed graphs)
Good software for the purpose is yEd editor. You can download it from here: https://www.yworks.com/
Getting familiar with the software takes some time so I suggest you start simple. Try to describe something you are very familiar with.
I use simple invoicing software as an example for this post but functionality is pretty common in most web based applications.
Reset Password Model
Think of all possible events/states that might happen
Save the graphml file from the above model in yEd (File > Save as > ResetPassword.graphml)
The graphml is meta language describing your model. This will be later used by GraphWalker in order to generate implementable interface.
Generate Maven Project from Archetype
Now it is time to generate maven project from archetype. The command is pretty simple but in order to be able to execute it, you have to make sure all the preconditions are met/installed.
- JDK 8 + (Java Developmen Kit)
- Apache Maven 3.5+
- IDE like InteliJ or just a simple console
Execute the following command in IDE terminal or in your console:
mvn archetype:generate -B DarchetypeGroupId org.graphwalker DarchetypeArtifactId graphwalker maven archetype DgroupId com.company DartifactId myProject
This will generate a valid maven project with everything neccessary to start using GraphWalker. All dependencies will be properly set in the pom file. Of course everything else you need in order to implement your steps like external dependencies for browser/api communication should be added by you.
Generate the Interface from the Model
Normally we store all graphml file under src/main/resources
To generate implementable interface execute the following command in the terminal:
The result of this command is brand new interface which we need to implement.
Implement the Interface
Execute Model-based Tests
GraphWalker takes care of walking the graph which basically generates unique automated tests every time you run it. While creating the model takes some time, support of the tests is highly reduced because your concern is only to support edges/verticies logic working or at max to update the directed graph if something is missed or changed.
Comparison with Traditional Way of Writting Tests
Normally we describe every step in the test case and the procedure is fixed. If we want to cover different path/procedure, we have to create brand new tests.
When using GraphWalker, all tests are automatically generated. We need to only define vertecies(expected results) and edges (events).
Benefits of Model-Based Testing
- Easy test case maintenance
- Reduced cost
- Early bug detection
- Increased bug count
- Time savings
- Can be used for Web/Mobile/Web Services
Obstacles to Model-Based Testing
- Comfort factor
- This is not your parents test automation
- Skill sets
- Need testers who can design
- Models can be significant upfront investment
- Will never catch all the bugs
- Different metrics
- In model-based testing we use different metrics (coverage)
You can find the source code for the demo here: https://github.com/karamfilovs/devbg
Have in mind that browsers evolve and you might need to update webdrivers in order to be able to run the examples.