What is software testing? There are many ways to answer this question:
- ISTQB: “Process of executing a program or application with the intent of finding the software bugs.”
- Wikipedia (Cem Kaner would probably agree): “Investigation conducted to provide stakeholders with information about the quality of the product or service under test.”
- James Bach: “Lighting the way of the project by evaluating the product.”
I give you here another approach on what software testing can be: a risk-reducing activity.
When you use it, any unknown in the project will show up as potential risk, and testing-related activities will reduce the risk.
This approach can help you answer the “hard” questions of testing: what to test, and when you have tested enough?
Let’s start with classifying risk. The classic approach is to estimate the impact and likelihood of a given risk. For testing purposes it may be enough to asses both on a 3-point scale: low / medium / high. It is up to you to define what these mean for your organization. A loss of a single customer is a low impact in some cases, or a high one if that is your organization’s only client. Same for likelihood: you can try giving specific percentage value, or just do an estimate.
This classification allows you to map risks to a heat map like this one (after OWASP):
|Overall risk severity|
What to test?
Risk identification, a process of calling out risks, will help you with the “what to test” question. It does not have to be hard.
Based on my experience you may want to:
- Learn from the past: look through past work and see what others have called out as risks in their projects. Also any issues in similar work missed before? These are worth calling out as risks that you need to deal with in the project.
- Cast a wide net: your organization: developers, product owners, fellow testers…, are great in coming up with potential risks.
- Search for the unknowns: Socratic “I know that I know nothing” is a good start to other risk-identifying tasks. Assumptions may have to be broken…
After you have identified the risks you can assess them. I would encourage you to do it as group exercise: in-person, offline, using delphi method etc.
The assessment will allow you to map the risks to a heat map and then prioritize the testing. You are most likely to start with items identified as critical risks.
A note: you will never be able to identify all the risks. But you will get better over time. Testing and test planning are risk-reducing on meta-level, too.
When have I tested enough?
Risk management helps to answer this question. There are generally 5 things you can do with the risk:
Reduce it. For a team this typically means “test it”. An unknown becomes known, and risk is lowered.
Eliminate it. It could be through change of scope, requirements change, etc.
Share it. This may not be obvious, but maybe you can have a third-party certify your product, and take responsibility for issues found later? Transfer can be also achieved through insurance. (yes, there seems to be an insurance that covers bugs)
Even if you identified and assessed a risk it does not always have to mean you need to do anything about it. Low risks can be sometimes simply ignored.
- Exploit (after DBP Management)
Turn it into an advantage. Maybe you have identified a risk of handling huge load by your website? But that is something you actually want, so you exploit the risk by making your website even more attractive, useful, easy to reach.
When do you know you have tested enough then? You lower the risk to an acceptable level. And what is an acceptable level? The answer to this question may be a risk you can transfer to your stakeholders :).
Let me know what your take on risk-reducing in software testing is.
Risk Assessment in Practice – easy to follow guide
Risk/Impact Probability Chart – risk heatmap example
Perceptions of Risk – people are bad at estimating risk by Bruce Schneier, this should not discourage you though from trying to reduce risks
10 Ways We Get the Odds Wrong – people are bad et estimating risk by Psychology Today to learn more what your brain does with risks
This post is also available in: English