Hitting a Wall: Testing Environments for Application Development and IT Operations

There is a wall between application development teams and IT Operations teams when it comes to testing. The wall isn’t a result of turf or expertise. It exists because these two groups usually have very different responsibilities and perspectives with regard to preparing a component for production deployment and, as a result, the focus of the testing activities they conduct differs greatly.

Unfortunately, sometimes the wall can hide an organization’s view of the impact that a change, upgrade or introduction can cause. In the worst scenario, this can cause unplanned downtime.

Testing in Application Development vs. Testing in IT Operations

Application development and quality assurance (QA) teams typically focus on:

• Managing sophisticated sandboxes to support their best practices for code testing, unit/component testing and change impact analysis
• Maximizing performance, stress, load, and response times within the confines of the application-centric environment
• The application first – they are less likely to understand, much less test, all aspects and dependencies within the broader IT production environment
• Utilizing application-focused testing tools, cycles, and environments

IT Operations teams have typically been focused on:

• The production infrastructure and implementation of a much broader and diverse set of software infrastructure configuration changes
• Ensuring availability and resiliency of multi-tier software infrastructure stacks
• Preventing and fixing problems that arise due to changes – not the more traditionally focused allocation of time or resources on organized and comprehensive testing

The importance of testing as a part of application development is a well established best practice. In fact, formal testing is viewed as an application development requirement–– with organizations making significant investments in specialty tools and environments to support application-centric testing as a part of the software development life cycle, development and QA.

The problem is that this testing occurs inside development and QA lab environments that are optimized for the application development test process. These environments are not typically representative of the actual production infrastructure and, once testing is completed, software is considered “production ready” and is handed off to IT operations to deploy and manage.

Outside of the small number of organizations that maintain representative staging environments of mission critical applications, the first time changes are ever tested against the entire “live” software infrastructure stack is when they are released into the production environment.

The result of this disconnect is the “wall” between development and IT Operations. Except in rare cases where end-to-end configuration, change and release management have been achieved, testing activities in Development, QA and Operations are isolated from one another.

Jason Grimes put the problem of testing differences between development and operations in very direct terms on his blog earlier this month:

“Every company who has a website or web application should have a disciplined Release Management Process. There are so many benefits from getting this right and I’m sure everyone is familiar with not getting this one right. The results of a poor process usually entail, Development throwing code over the fence to Test and subsequently throwing it over to the Technical Operations team. The end result usually ends in configuration, integration issues or last minute bug fixes that do not get thoroughly tested for regressions.”

This wall occurs because of the difference in point of view between development/QA and operations. But in a business environment where the expectation for uptime levels for all applications are nearing 100%, IT Operations teams are beginning to place more emphasis on improving their own testing capabilities – to better understand the impact of changes to the production environment prior to release. Slowly, IT Operations is beginning to break down the wall between application development teams and IT operations groups.

There are real quantifiable business benefits to this approach. Based on our IT Operations Research Report: Testing Maturity, we found that IT Operations organizations that have invested in a more mature approach to testing see some significant business benefits, including:

• 25% fewer emergency changes
• 15% greater confidence that changes won’t impact the production environment
• 20% more likely to view the change process as smooth

As more IT Operations teams realize the benefits of conducting their own testing of all changes to the production environment before they go live, we can expect to see the wall start to come crumbling down.

Popularity: 4% [?]

Filed Under: Change Impact Analysis, IT Operations, IT Operations Research, Testing