There’s a certain idea making the rounds in the world of software development. Some people have implicitly accepted it, others have eagerly embraced it and some people hate it with the passion. So what is the big idea? It’s that software testing is essentially dead.
This argument has been made on numerous blogs, articles and presentations – each with its own variations – and so I won’t list or reference any one in particular. Instead, I’ll use this post to explain how I view the idea and why I consider it to have one glaring flaw.
So what does it mean to say that software testing is dead? Here is how I understand it:
Today, software applications have an enormous base of users, with an equal number of unique use cases. Rather than hire a traditional QA team to cover this increasingly complex matrix, it’s more effective/practical to find bugs through big data (i.e. real time metrics), real users (i.e. customers or beta users) and through the programmers themselves.
My thoughts:
The premise is spot on. The testing matrix is indeed more complicated than it has ever been, with an endless combination of hardware, browsers, carriers, languages, APIs and other criteria. And it’s also true that a traditional test teams can’t possibly account for all of these scenarios – and that a whole class of bugs are likely to go undiscovered until the app is released into the real world. For this reason, measuring performance in real-time (often referred to as Testing in Production or “TiP”) and beta programs are incredibly valuable, but they should not, in my opinion, be considered as a replacement for professional testing. Therefore…
The conclusion is wrong. The challenges of the new testing matrix are well documented and indeed quite daunting, but if one concludes that the solution is to abandon formal testing (and testers) altogether, then they can expect an entirely new set of problems.
Mainly, that if everybody is considered a tester, then nobody is really a tester. Rest assured, users WILL find the bugs, but from there, it’s anyone’s guess as to what happens. uTest CTO Fumi Matsumoto covered the many weaknesses of beta testing in a recent blog post, they included:
- Beta testing often generates too much noise (that is, feedback) that is not accurate and not actionable.
- Inconsistent participation — too much or too little — often administered with poor processes for collecting and analyzing feedback. Not all use cases get covered, so bugs slip through.
- Good catches but insufficient information: Even when bugs are identified, the reports are often not useful because they lack sufficient information to reproduce the defect.
- Delay: Beta testing slows the release cycle by having a dedicated phase before the production release.
Like any consumer of apps, I’ve encountered hundreds of bugs in my day. But how many of those have I reported to the developer or company? I could probably count them on one hand. Apart from the frustration of not being able to complete a task, I really had no reason to.
That’s obviously not the case for testers; they approach an application with a much different mindset.
So how can a company “keep testing alive” but still get feedback from a diverse, dispersed set of users? Through crowdsourcing, or as we we like to call it, in-the-wild testing. Fumi continued in his post:
With the advent of crowdsourced testing, or what is often referred to as “expert-sourcing” because it often utilizes vetted and trained QA professionals, development organizations can now get the benefit of in-the-wild testing without the downside of beta testing’s high noise level. This option offers companies the ability to test pre-deployment under real-world conditions and, in particular, address the difficult problem of mobile device fragmentation: OS versions, mobile carriers, memory and other mobile device configurations, or location diversity. Typically, the vendors will hire a test company’s members in specific locales to beta test the software and report defects via agreed upon forms and channels.
So is traditional testing dead? For some that might be true. Is software testing altogether dead? Not by a long shot.
What do you think? Is there a case to be made that testing is dead, or perhaps a dying profession? Please share your thoughts in the comment section below.