The Dirty Half-Dozen: Testing Environment Shortcuts for IT Operations

We are always trying to understand the way that companies test changes at IT’s About Uptime. How often are changes tested? Who does the testing? How many changes get made in a year? How many get rolled back? You get the picture.

One key component that also needs to be considered is where (i.e. in what environment) do changes get tested. This is a very difficult problem for many companies – especially those stuck in the “unfortunate middle” who need to make large number of changes to stay responsive to the business, but often lack Fortune 500-level resources for testing changes as completely as they would like.

And the question of where a change is tested is very important because it relates directly to the insight you will have on how a given change will impact an application. Testing in QA is one thing, because these tests typically focus on whether an application will do what is was designed to do. Testing for functionality in an environment that represents actual production is quite another.

We were recently in a room of IT Operations professionals at a trade show where the depth of this problem became obvious. The discussion was about the ITIL best practice of building and maintaining a staging environment that is identical to the production environment. When the question was asked “who actually does this?” you could heard chuckles of laughter in the room from people who knew they had nowhere near the budget to implement the best practice.

The only gentleman who claimed to actually have a full staging environment was practically pelted with tomatoes by the rest of the envious audience.

So where do you test your changes? Do any of these sound familiar? We call them the “dirty half-dozen” of testing environment shortcuts. They are all fairly common and all have some obvious (and major) downsides, that we will discuss in a follow on post:

• Patch and Pray - bypassing testing altogether and deploy changes directly into production.
• “Non-Customer-Facing” Tests - deploying new configurations and patches on live systems that are low-priority or non-customer facing applications.
• Scheduled Downtime – testing the change in production during scheduled downtime
• Borrowing “Redundant” Resources – taking disaster recovery systems offline for testing purposes.
• Component Testing – limiting testing to an individual software component or configuration change rather than testing the change as part of a full end-to-end distributed stack test.
• Table-Top Testing – evaluating changes using documents, spreadsheets and diagrams on a conference table.

Popularity: 10% [?]

Filed Under: Change Impact Analysis, Testing