What is regression testing? Why do it? What value does it have? My short answer to these questions is regression testing as a concept is about searching for new faults introduced through changes to the code base, and to learn more about the system you are building.
Usually in modern development, through an automated build pipeline, you are constantly executing checks that help detect whether the changes made to your code have resulted in new faults being introduced. Couple this with the interactive testing that we do throughout the development of any story then we have a combined effort that you could describe as regression testing.
In my mind, thats pretty much it. Its main purpose is to find faults, and learn more about the system.
Many people become confused about the term regression testing, and rightly question its worth when they judge either automated only solutions, typically through a UI or API, or look to have regression testing as a phase that occurs at some point on the critical path to delivery. Both of these approaches fall into the realm of delivery anti patterns.
If you think of your entire test and development approach as a way to minimise the risk of regression and to deliver the right thing in the most effective and safe way, you very rarely have to think about or understand the need for regression testing.
Thats my rather simplistic view. For more detail, have a read of Michael Bolton’s presentation "Things Could Get Worse: Ideas About Regression Testing"
Showing posts with label regression. Show all posts
Showing posts with label regression. Show all posts
Wednesday, 10 September 2014
Tuesday, 27 November 2007
Risk Based Regression Test Strategy
Using a risk based strategy to develop a regression test plan
Designing a regression test plan can be a very laborious task. Whilst we are trying to incorporate into the test plan all those test tests that we feel cover the functionality or requirement of the test, we very often miss out important tests. If we are aware of a particular tests impact within the test plan and the risk that we have by not including that test, we can create a much more thorough test plan. Assigning a risk index to each and every test that we have, allowing the ability to change the risk weighting of a particular test, allows us to have a greater control over the selection process. Using a risk based test selection strategy we can test thoroughly, the required parts of an application. In this article I’ll discuss a method that applies risk analysis to the creation of a regression test plan.
Risk based selection
The basic principle behind this strategy is to apply a risk index to a test case and then select those test cases that have reached a sufficiently high enough level of risk to be involved in the test plan.
A simple risk model
To calculate risk we need to identify measurable elements within a test that can be calculated to provide the risk index. Two key elements that we can consider are
The probability of error occurring with the test functionality or of the scenario occurring. (Probability)
The consequence of a fault occurring in a production environment. (Cost)
These two elements are described in Amaland’s [1] , and form the elements that we will use to calculate risk exposure.
Risk exposure is calculated mathematically using the following formula:
RE(f) = P(f) * C(f)
Risk Exposure = Probability * Cost
The calculation is based on a scale of importance that is applied to each of the functions within the risk exposure calculation. To facilitate the ease of weighting probability and cost we use a five point scale with one being low and five being high.
Selection
Selecting tests for the regression test plan consists of a two part process: The selection of test cases and the selection of end to end scenario coverage.
Test case selection
1. Cost estimation of each test case – It is important that we, as tests analysts, take an objective viewpoint on the cost of an error. We should consider ourselves as the client when making these decisions. What would they consider to be costly in terms of errors? Losing the client perspective can very often cause incorrect test case weighting.
2. Extract an error probability weighting. Error probability is a calculation based on various metrics. The easiest way to calculate probability is to count the number and identify the severity of defects associated with a particular functionality. More precise methods also include information garnered from code analysis, although, this type of metric can be time consuming to obtain.
3. Once the two functions have been calculated we can then calculate the Risk Exposure. We now have a means to extract tests based on there risk exposure.
Scenario selection for regression and system testing
A collection of scenarios is what forms the basic test plan. Scenarios are designed to cover a multitude of test cases and application modules working together. When selecting scenarios we use two principle strategies.
1. Selection of scenarios to cover the critical modules of an application.
2. Selection of scenarios to cover the greatest number of tests.
When scenarios have been created we can use traceability a traceability matrix to show the relation between test cases and scenarios. Using this matrix we can construct a risk based scenario test plan using the following approach:
1. Calculate the risk exposure for each scenario: Using the traceability matrix, sum the risk exposure of all tests covered by the scenario.
Designing a regression test plan can be a very laborious task. Whilst we are trying to incorporate into the test plan all those test tests that we feel cover the functionality or requirement of the test, we very often miss out important tests. If we are aware of a particular tests impact within the test plan and the risk that we have by not including that test, we can create a much more thorough test plan. Assigning a risk index to each and every test that we have, allowing the ability to change the risk weighting of a particular test, allows us to have a greater control over the selection process. Using a risk based test selection strategy we can test thoroughly, the required parts of an application. In this article I’ll discuss a method that applies risk analysis to the creation of a regression test plan.
Risk based selection
The basic principle behind this strategy is to apply a risk index to a test case and then select those test cases that have reached a sufficiently high enough level of risk to be involved in the test plan.
A simple risk model
To calculate risk we need to identify measurable elements within a test that can be calculated to provide the risk index. Two key elements that we can consider are
The probability of error occurring with the test functionality or of the scenario occurring. (Probability)
The consequence of a fault occurring in a production environment. (Cost)
These two elements are described in Amaland’s [1] , and form the elements that we will use to calculate risk exposure.
Risk exposure is calculated mathematically using the following formula:
RE(f) = P(f) * C(f)
Risk Exposure = Probability * Cost
The calculation is based on a scale of importance that is applied to each of the functions within the risk exposure calculation. To facilitate the ease of weighting probability and cost we use a five point scale with one being low and five being high.
Selection
Selecting tests for the regression test plan consists of a two part process: The selection of test cases and the selection of end to end scenario coverage.
Test case selection
1. Cost estimation of each test case – It is important that we, as tests analysts, take an objective viewpoint on the cost of an error. We should consider ourselves as the client when making these decisions. What would they consider to be costly in terms of errors? Losing the client perspective can very often cause incorrect test case weighting.
2. Extract an error probability weighting. Error probability is a calculation based on various metrics. The easiest way to calculate probability is to count the number and identify the severity of defects associated with a particular functionality. More precise methods also include information garnered from code analysis, although, this type of metric can be time consuming to obtain.
3. Once the two functions have been calculated we can then calculate the Risk Exposure. We now have a means to extract tests based on there risk exposure.
Scenario selection for regression and system testing
A collection of scenarios is what forms the basic test plan. Scenarios are designed to cover a multitude of test cases and application modules working together. When selecting scenarios we use two principle strategies.
1. Selection of scenarios to cover the critical modules of an application.
2. Selection of scenarios to cover the greatest number of tests.
When scenarios have been created we can use traceability a traceability matrix to show the relation between test cases and scenarios. Using this matrix we can construct a risk based scenario test plan using the following approach:
1. Calculate the risk exposure for each scenario: Using the traceability matrix, sum the risk exposure of all tests covered by the scenario.
2. Select the scenarios with the highest risk exposure.
This is a very basic approach that allows a greater coverage of functionality. This is a practice that we would normally associate with regression testing or cyclic system testing but can also be applied to other areas of testing. It is a method that greatly improves error detection by focussing on critical components within an application.
This article is adapted from the following:
Stale Amland, “Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Vol. 53, 2000, pp. 287 - 295.
Yanping Chen, Robert L. Probert A Risk-based Regression Test Selection Strategy, University of Ottawa, Ontario, Canada, Fast Abstract ISSRE 2003, Chillarege Press
Subscribe to:
Posts (Atom)