In an ideal world, testing would be done within the comfy confines of a staging environment, where software bugs – no matter how big or small – had zero chance of making it in front of real users. In this environment, testers and developers would have ample amount of time (and practically no pressure) to identify and fix issues. They could then launch with no surprises and sing songs together while their bosses showered them with praise.
We don’t live in that world. Instead, we live in a world that sometimes requires us to test our applications in production, in front of real users. Sometimes, there’s simply no other option.
But what are some of those instances? When does it make sense to test in production vs. a staging environment? I don’t presume to know all of the times when this might be the case, but a recent article from the insurance industry (ironically enough) highlighted a few times when and why testing in production is necessary.
Here is their broad overview, followed by a few examples:
In fact, we test in production all the time. Even well-disciplined, well-managed application development teams must rely on a production release to fully vet and test their code. I am not talking about unit testing, or functionality testing, or integration testing, or system testing or even performance testing.
We still need to do all those things and if we do them correctly we won’t have any bugs or code defects when we go to production. But it is impossible to validate actual performance of a complex system in a lower life cycle. The key word here is test. You can test and retest, but all you are really doing is validating the performance and functionality of the application in a testing environment.
There are a few specific examples mentioned by the author – and they are some of the same reasons cited by TiP expert Seth Eliot when he was interviewed on our blog. The first of which involves data. The author writes:
Then there is data. You simply don’t have the same data in your lower life cycles that you do in production. Test data is not real data. It may look like real data but it isn’t. Production data is sometimes replicated for pre-production or staging, but never for testing. Scary IT campfire stories abound of the fate that befell organizations (or CIO’s) that mixed production data in test systems. Too many things can go wrong to ever consider that option.
Couldn’t agree more. Most of the time, if you need actionable data it will need to come from production. This is particularly true of performance and security testing, something that was acknowledged by the author. Integration is another area where it makes sense to test in production:
How many times have you seen a team stumble when all the unit testing and integration is done and it’s time for production? Little things like developer accounts embedded in connection strings creep out of the woodwork. All those little tricks that the development team hacked together to get the application running come back to haunt them.
Now this should not be viewed as an endorsement of testing only in production. Though we specialize in in-the-wild testing, we are fully aware of the value that staging environments provide. So too is the author:
So do we test in production? Of course we do, but we are looking for different defects in our testing. You still need to do the necessary work in lower life cycles and only promote code that has been rigorously tested. Code that is promoted to production must be fully functional and proven to satisfy business requirements. We aren’t expecting surprises or to discover hidden defects, but we do need this final validation that it runs just as well on busy interstates and crowded city streets as it did on the test track.
To close, a question for all of the developers and testers out there: Do you test in production? Be sure to let us know your thoughts in the comment section below.