You may be familiar with our Testing the Limits series where we interview thought leaders in software development and testing. These experts answer questions, give advice and share nuggets of wisdom that will help readers succeed in their careers.
How do you know they’re not just full of hot air? Well, for one, they probably wouldn’t be well respected members of the QA community if they were just lying all the time. But also, real life backs them up.
In this series, we’ll take excerpts from Testing the Limits interviews and pair them up with real life situations that prove these experts know what they’re talking about. There’s our first expert excerpt:
“Real testing, to me, should be based on investigating how the software allows people to deal with what we call ‘exceptions’ or ‘corner cases.’ That’s what we call them, but if we bothered to look, we’d find out that they were a lot more common than we realize; routine, even.” – Michael Bolton
This is what in-the-wild testing is all about. Ultimately, real life people will be using your software in real life situations. Sometimes, those real life situations are scenarios you couldn’t test for in the lab, didn’t occur to you or that you didn’t even dream of. But odds are your users will find them. While a crazy fringe use case will pop up from time to time no matter what you do, it’s important to take a step back, think through your testing and make sure you’ve come up with as many use cases as possible.
One of the best, most obvious and repetitive examples of “corner cases not really being that uncommon” can be found in video game development. Gamers are dedicated people and will try to make your software do things you wouldn’t believe. The biggest problems occur when they’re trying to do something reasonably within scope – uncommon maybe, but not unusual or unheard of. For instance, when Mass Effect 3 came out early last year the game encountered a bug that wouldn’t let players import customized avatars from earlier editions. If players set the appearance of their Commander Shepard avatar in the original Mass Effect and didn’t tweak it at all in Mass Effect 2, the character would revert to factory appearance when imported into game 3. From CNet:
The bug means some characters created in the original Mass Effect fail to import their beautiful custom faces. The choices you made in the first two games are unaffected, but gamers particularly attached to their Commander Shepards are up in arms.
“We have determined that faces importing incorrectly is an error with how codes were detected when transferred from Mass Effect 1 into Mass Effect 2 and then on into Mass Effect 3,” BioWare’s Chris Priestly wrote on the game’s community forum.
This is a particularly bad miss because it’s not that much of a fringe case. It stands to logic that avid fans of the game would want to import their customer characters for this new adventure and that certainly should have been tested. A more nitty-gritty instance of a fringe bug comes up in another video game – Diablo III. (If you’re releasing a third installment of something you might want to dedicate a little more time to testing, it seems like the third time isn’t the charm).
In Diablo III, a very specific set of actions ended up crashing the game, throwing an error message and locking players out. From CNet:
The bug, which only affects the Demon Hunter class, supposedly occurs when one equips a Templar follower with a shield early in the game. Performing this action reportedly causes the game to disconnect the player immediately, and then follows up with a disastrous repetitive “Error 75″ that continuously prevents the user from further log-in.
Testers are human, they can’t possible think of every use case. But Michael Bolton is right, more time needs to be dedicated to identifying these “corner cases” so they can be fixed before they disrupt your users, especially if they’re not really that unusual.