Tuesday 27 November 2007

Risk Based Regression Test Strategy

Using a risk based strategy to develop a regression test plan

Designing a regression test plan can be a very laborious task. Whilst we are trying to incorporate into the test plan all those test tests that we feel cover the functionality or requirement of the test, we very often miss out important tests. If we are aware of a particular tests impact within the test plan and the risk that we have by not including that test, we can create a much more thorough test plan. Assigning a risk index to each and every test that we have, allowing the ability to change the risk weighting of a particular test, allows us to have a greater control over the selection process. Using a risk based test selection strategy we can test thoroughly, the required parts of an application. In this article I’ll discuss a method that applies risk analysis to the creation of a regression test plan.

Risk based selection

The basic principle behind this strategy is to apply a risk index to a test case and then select those test cases that have reached a sufficiently high enough level of risk to be involved in the test plan.

A simple risk model

To calculate risk we need to identify measurable elements within a test that can be calculated to provide the risk index. Two key elements that we can consider are

The probability of error occurring with the test functionality or of the scenario occurring. (Probability)

The consequence of a fault occurring in a production environment. (Cost)

These two elements are described in Amaland’s [1] , and form the elements that we will use to calculate risk exposure.

Risk exposure is calculated mathematically using the following formula:

RE(f) = P(f) * C(f)

Risk Exposure = Probability * Cost

The calculation is based on a scale of importance that is applied to each of the functions within the risk exposure calculation. To facilitate the ease of weighting probability and cost we use a five point scale with one being low and five being high.

Selection

Selecting tests for the regression test plan consists of a two part process: The selection of test cases and the selection of end to end scenario coverage.

Test case selection

1. Cost estimation of each test case – It is important that we, as tests analysts, take an objective viewpoint on the cost of an error. We should consider ourselves as the client when making these decisions. What would they consider to be costly in terms of errors? Losing the client perspective can very often cause incorrect test case weighting.

2. Extract an error probability weighting. Error probability is a calculation based on various metrics. The easiest way to calculate probability is to count the number and identify the severity of defects associated with a particular functionality. More precise methods also include information garnered from code analysis, although, this type of metric can be time consuming to obtain.

3. Once the two functions have been calculated we can then calculate the Risk Exposure. We now have a means to extract tests based on there risk exposure.

Scenario selection for regression and system testing

A collection of scenarios is what forms the basic test plan. Scenarios are designed to cover a multitude of test cases and application modules working together. When selecting scenarios we use two principle strategies.

1. Selection of scenarios to cover the critical modules of an application.

2. Selection of scenarios to cover the greatest number of tests.

When scenarios have been created we can use traceability a traceability matrix to show the relation between test cases and scenarios. Using this matrix we can construct a risk based scenario test plan using the following approach:

1. Calculate the risk exposure for each scenario: Using the traceability matrix, sum the risk exposure of all tests covered by the scenario.

2. Select the scenarios with the highest risk exposure.

This is a very basic approach that allows a greater coverage of functionality. This is a practice that we would normally associate with regression testing or cyclic system testing but can also be applied to other areas of testing. It is a method that greatly improves error detection by focussing on critical components within an application.


This article is adapted from the following:
Stale Amland, “Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Vol. 53, 2000, pp. 287 - 295.

Yanping Chen, Robert L. Probert A Risk-based Regression Test Selection Strategy, University of Ottawa, Ontario, Canada, Fast Abstract ISSRE 2003, Chillarege Press