I did not come up with this term. We started using it back in 2009/2010 at Symantec inspired by Michael Bolton‘s When do we stop testing? presentation. In it he mentions a Piñata heuristic as one of the ways to decide when you tested enough.
The idea is to beat on the Piñata (software) until candy (bugs) starts falling out. And stop when you hit the first dramatic problem.
This term also describes well a testing technique we were using at Symantec. Its basic principle is to use the Piñata beating approach to identify weak, or lower quality, parts of the software. You start testing the application, and when issues start showing up in certain area, you concentrate on it even more. Keep hammering as long as candy/bugs are falling out. Do it as long as the ratio of new issues is not slowing down. When that happens you take a pause.
Now comes the time to assess the damage made to the Piñata (our test subject):
- Have you found something that may require a fix invalidating your further tests? If yes, you probably should stop here.
- Do the issues found so far have something in common? Patterns showing up in bugs may indicate that there are repeated issues in the code under test. This might be a good time to chat with your developers(*). Maybe they are using a buggy library? Patterns may also show up in code coming from specific developer only. Maybe that person is not following coding practices others are using? If it seems like a more general problem, it might be worth testing other areas of your software for similar issues, not only the module that exposed them.
- Looking at the issues, can you predict other issues showing up? For example: lack of good input handling may also signal potential SQL injection exposure. Think about scenarios like that one, and check the suspicious ones.
- How does this specific area compare to the rest of your software? Is it significantly worse? Significantly better quality? Can you tell why is it better or worse?
Depending on the results of the assessment described above you may decide to stop testing this specific area, or keep hitting it. If you keep testing, you will again stop and assess at some time, and decide you were done. You may eventually ask yourself one of two questions:
- Have I found all the issues in given area? Am I sure there is not even more critical issue still uncovered?
- Have I correctly identified the risky/weak part of the application?
The answer to both questions is “maybe not”. You may have not identified all the issues, there still may be severe bug hiding, and the other areas of your software may be even worse than this one. But when you look at testing as risk-reducing activity I think Piñata testing, when combined with other testing techniques, is a fun way to find good issues and great help in identifying weak spots.
(*) Actually any time is good to chat with developers. They are smart people and you can both learn from each other.
This post is also available in: English