Image may be NSFW.
Clik here to view.
As uTest’s CTO, Fumi Matsumoto spends a lot of time immersed in the world of software development, testing and releases. He’s also a guest editor for Dr. Dobbs, where he recently wrote about beta testing’s role in today’s software development life cycle. Here’s a excerpt.
Many QA engineers I’ve known have had a love/hate relationship with beta testing. “It’s expensive to administer and doesn’t give me useful information,” said one. “Too many bugs slip through this so-called ‘beta testing,’ so I have to test the whole app in the lab anyway,” said another, “but if Marketing wants it, I can’t stop ‘em.”
On the other hand, some organizations continue to see value in beta testing: “We need to plan for three weeks of beta phase to make sure we get better coverage on use cases and environments that we can’t test against ourselves.”
Used judiciously, beta testing programs can be valuable, but modern software development practices challenges the whole notion of beta testing. How do you fit beta testing cycles in already compressed release cycles or with frequent releases? And as user reviews and ratings of applications become more transparent through social and app store review channels, the definition of “app quality” is slowly morphing from functional correctness to user-perceived value. Doesn’t that change the entire premise of a beta testing program?
There are many problems associated with traditional beta testing:
- Beta testing often generates too much noise (that is, feedback) that is not accurate and not actionable.
- Inconsistent participation — too much or too little — often administered with poor processes for collecting and analyzing feedback. Not all use cases get covered, so bugs slip through.
- Good catches but insufficient information: Even when bugs are identified, the reports are often not useful because they lack sufficient information to reproduce the defect.
- Delay: Beta testing slows the release cycle by having a dedicated phase before the production release.
In addition to these problems, several modern deployment practices are making beta testing less attractive.
Read the rest at Dr. Dobbs >>>
