Is Testing Overrated?

Posted August 5th, 2008 by Dennis Powell

Luke Franci and Matt Heuser think that TESTING IS OVERRATED (gasp!), at least according to recent blog posts from both under that title. Matt’s August 1st post at Creative Chaos provided an intro to Luke’s July 11 post at Rail Spikes .

Upon further review it’s clear that developer/QA code testing is their focus, and the “overrated” assessment points to an over-confidence by developers who test their own code during development (TDD) at the expense of separate QA efforts. Although both posts focus on application software testing and the dev-to-QA cycle, both posts provide a segue to the broader IT challenge of establishing readiness of the production environment to absorb change.

Matt notes that “testing” means “checking the software to find out if it works or not, right?” Yep, particularly if you’re the developer or QA professional at whom Matt’s blog is aimed. But what about IT operators and administrators, who have to ensure that all of this “tested” software works with the reality that is the data center today, meaning the vendor’s “tested” software, the partner’s “tested” software, the “tested” legacy software, the “tested” database software, the operating system configurations, the hardware configurations, the network permutations…not to mention the growth in hybrid physical and virtual environments? Oh by the way, not only does all this “tested” stuff need to work, IT needs to make sure that it runs 24×7, is always at peak, and meets all SLAs.

When there is a “we don’t need no stinkin’ QA people” mindset at the software development and unit test level, what happens to availability testing, capacity testing, performance testing, security testing, system testing, UAT…? What happens when this code makes its way “over the wall” to IT to be released into a production environment that’s about much more than just the code? This is why 25% of all changes to production cause problems, and why 10% of changes must be rolled back because they can’t be fixed.

Developers that “don’t need QA” might not test everything or test as deeply. QA groups that aren’t prepared to validate software readiness to run in the organization’s infrastructure in turn won’t be able to test permutations. Most importantly, pre-production IT personnel who rely on “component testing” or “patch-and-pray”, tend to introduce unnecessary risk and potential impact to production.

This is why, regardless of how briefly or thoroughly an application is tested, regardless of how broadly an application environment is QA’d, there needs to be that final line of pre-production defense based on sound change and release management principles. Build and maintain the representative staging environment, test every new component and every change (software, configuration, patch, upgrade, etc), test all changes end-to-end, schedule change testing and deployment, and follow change and release best practice guidelines.

Even the most effectively-tested software will invariably cause some type of problem when it finally gets deployed into the production infrastructure. Although it requires more time, resources and budget to make the proper service transition from code to production, treat testing at every level as an underrated service.