Image may be NSFW.
Clik here to view.
Software testing has changed. At first, you only had to worry about a person’s (limited) hardware and any software they got off of actual, physical discs. Then came the internet, which added a few more kinks to the testing puzzle – connection speed, the ability to download plugins, easier, more frequent updates. But still, all of this was hard-lined. The testing matrix was still relatively simple and there wasn’t much happening in the real world that you couldn’t feasible replicat in a lab. WiFi and laptops made it a bit harder, but still, nothing to panic about. Then came the mobile revolution. You can panic now.
The explosion of mobile means that the way you test doesn’t have to be incrementally adjusted like with past advances, this time around it needs to be rethought entirely. The way users consume applications and software is drastically different these days and old lab-based testing isn’t going to cut it. In the simplest terms, lab testing doesn’t cover enough of the variables mobile users encounter. This infograpic by InMobi shows how mobile consumption differs from traditional software usage. Here’s an idea of the places people use mobile that they likely wouldn’t have used other forms of software:
- 77% of people use mobile devices while lying in bed
- 70% use mobile while watching TV
- 65% use it while waiting for something
- 43% use it while spending time with their family
- 41% use it in the bathroom
- 36% use mobile while commuting
- 35% use mobile devices during social gatherings
- 34% use it while shopping
Developers and testers now have to account for a collection of mobile devices with different screen sizes, resolutions and controls; different operating systems and their versions (sometimes six or more still-functioning); different network carriers, connections and connection strengths (which change with location); myriad apps that might be functioning on any given device, taking up space, vying for attention and bandwidth. The demands of current software use dictate a change to the way software is tested. Here’s Dave Berg, Senior Director of Product Management at Shunra Software, outlining one of the biggest shifts in a guest piece for Dr. Dobbs:
Which do you usually complain about more when it comes to performance: your traditional Internet connection or your mobile data connection? I’ll put my money on your mobile connection. The reason is the network. From city to city, and even sometimes block to block, connection and performance is unreliable and constantly shifting. The simple physics of mobile technology result in greater latency variation than a hard-lined connection, making slower, unreliable speeds inherent in a mobile connection.
This was not something you had to account for when testing apps across traditional network conditions. A hard-lined connection, even if it was at a crawling 28.8 Kbps, could be counted on to remain constant. You knew what network conditions your app would encounter and this made performance testing (relatively) easy.
Assuming a constant level of network performance is a risky misstep with mobile that can leave major issues undetected. Testing apps in a traditional manner without accounting for such mobile network issues as increased and variable latency, jitter, and packet loss means your testing will yield unreliable results and put your app in a position for failure.
“Unreliable results” and incomplete results are the two overarching reasons mobile app testing will be forced outside of the traditional test setting and into the wild. You simply cannot adequately recreate all the different network connections users will experience if you’re testing in a lab. Even if you could, you’d have to replicate that connection on literally hundreds (if not thousands) of devices and dozens of OS versions if you’re looking to release a far-reaching app. Then just hope for the best that any third party apps a user downloaded won’t negatively affect your app’s performance. The requirements for testing mobile apps are staggering.
“Any company with device specific apps knows the complexity of the testing matrix,” said Chad Taylor, Co-Founder of Thrillcall.
The bottom line is that testing entirely in a lab just doesn’t cut it anymore. Don’t abandon in-house testing, it can save you a lot of time and money if obvious bugs are found in house. But if you expect to release a well performing, user-friendly, highly rated app, you’re going to have to move a portion of the testing into the real world. It will save you money (by cutting down on the number of testing scenarios you have to pay to house internally) and it will save your brand reputation. Software testing was never simple. But the way we use software these days – taking it everywhere with us on devices that developers have no control over – is forcing a drastic change.
Adjusting testing practices to the issues and challenges at hand is not a new concept for testers. QA professionals all have that epiphany where they realize their favorite test or their favorite method just isn’t going to cut it all the time. The best testers are able to adapt.
“It took me about ten years before I reluctantly accepted the idea that my favorite test techniques, attitudes, and life-cycle models were appropriate in situations similar to the ones where I developed my preferences, but not so appropriate in situations that were quite different.” – Cem Kaner
“Let’s not lock in a set of context-free definitions and practices and make them a standard. Such standards will hurt the quality of software, not improve it.” – Ben Simo
“I think that the only way to perform good testing is to adapt the approach to your testing mission to your own specific environment or context. You will be missing some major testing areas without considering how your current situation is different to other situations in which you have been involved.” – Paul Holland
In-the-wild testing is just another technique you can use to address the particular and peculiar challenges of mobile app testing. Embrace it.