Risk Based Regression Test Strategy

Using a risk based strategy to develop a regression test plan

Designing a regression test plan can be a very laborious task. Whilst we are trying to incorporate into the test plan all those test tests that we feel cover the functionality or requirement of the test, we very often miss out important tests. If we are aware of a particular tests impact within the test plan and the risk that we have by not including that test, we can create a much more thorough test plan. Assigning a risk index to each and every test that we have, allowing the ability to change the risk weighting of a particular test, allows us to have a greater control over the selection process. Using a risk based test selection strategy we can test thoroughly, the required parts of an application. In this article I’ll discuss a method that applies risk analysis to the creation of a regression test plan.

Risk based selection

The basic principle behind this strategy is to apply a risk index to a test case and then select those test cases that have reached a sufficiently high enough level of risk to be involved in the test plan.

A simple risk model

To calculate risk we need to identify measurable elements within a test that can be calculated to provide the risk index. Two key elements that we can consider are

The probability of error occurring with the test functionality or of the scenario occurring. (Probability)

The consequence of a fault occurring in a production environment. (Cost)

These two elements are described in Amaland’s [1] , and form the elements that we will use to calculate risk exposure.

Risk exposure is calculated mathematically using the following formula:

RE(f) = P(f) * C(f)

Risk Exposure = Probability * Cost

The calculation is based on a scale of importance that is applied to each of the functions within the risk exposure calculation. To facilitate the ease of weighting probability and cost we use a five point scale with one being low and five being high.

Selection

Selecting tests for the regression test plan consists of a two part process: The selection of test cases and the selection of end to end scenario coverage.

Test case selection

1. Cost estimation of each test case – It is important that we, as tests analysts, take an objective viewpoint on the cost of an error. We should consider ourselves as the client when making these decisions. What would they consider to be costly in terms of errors? Losing the client perspective can very often cause incorrect test case weighting.

2. Extract an error probability weighting. Error probability is a calculation based on various metrics. The easiest way to calculate probability is to count the number and identify the severity of defects associated with a particular functionality. More precise methods also include information garnered from code analysis, although, this type of metric can be time consuming to obtain.

3. Once the two functions have been calculated we can then calculate the Risk Exposure. We now have a means to extract tests based on there risk exposure.

Scenario selection for regression and system testing

A collection of scenarios is what forms the basic test plan. Scenarios are designed to cover a multitude of test cases and application modules working together. When selecting scenarios we use two principle strategies.

1. Selection of scenarios to cover the critical modules of an application.

2. Selection of scenarios to cover the greatest number of tests.

When scenarios have been created we can use traceability a traceability matrix to show the relation between test cases and scenarios. Using this matrix we can construct a risk based scenario test plan using the following approach:

1. Calculate the risk exposure for each scenario: Using the traceability matrix, sum the risk exposure of all tests covered by the scenario.

2. Select the scenarios with the highest risk exposure.

This is a very basic approach that allows a greater coverage of functionality. This is a practice that we would normally associate with regression testing or cyclic system testing but can also be applied to other areas of testing. It is a method that greatly improves error detection by focussing on critical components within an application.


This article is adapted from the following:
Stale Amland, “Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Vol. 53, 2000, pp. 287 - 295.

Yanping Chen, Robert L. Probert A Risk-based Regression Test Selection Strategy, University of Ottawa, Ontario, Canada, Fast Abstract ISSRE 2003, Chillarege Press

Test Automation - Visual Basic Web Automation with IE - Part 3

Finally... Automation

The final part of this series on web application automation will focus on some examples using the concepts discussed in the previous two parts. We shall also discuss some of the limitations of using the DOM and the alternatives available to automate web application testing.

Automating a Web Page

The basic automation of web pages is simple and requires only the combination of the concepts discussed in the first two parts of this series. Before looking in more detail at the examples of automation we need to take into consideration the following DOM limitations mean that not all web pages can be easily automated.

• 
  Only HTML/DHTML web content can be automated. Web applications based on Java or other development frameworks such as .NET can not always be automated.
• 
  Web pages are not always developed with automation in mind. It is often necessary to create coding procedures within a development team to include the required properties within the elements to facilitate automation.

Locating Elements

There are various ways in which we can identify elements with the document.

• Viewing the source code directly.
  
• Using a DOM inspector
  

DOM inspectors are the preferred method as they usually allow the identification of elements by means of clicking on them or hover over them. Source code viewing is effective and can often uncover elements that have not been recognised by a DOM inspector. The only draw back with source code viewing is that it can be time consuming and not always accurate.

Automation

The following examples use Visual Basic Script to automate browser operations.

This function opens a browser and will then navigate to a web page.

Function OpenBrowser(url, sleepTime)
Dim IE
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate2 url
Do
WScript.Sleep(sleepTime)
Loop while IE.Busy = TRUE
Set OpenBrowser = IE
End Function

A function to click on link.

Function ClickLink( byval oIE, strLnk)
Dim Link
For Each Link In oIE.Document.links
If Instr(strLnk, Link.href) > 0 Then
ClickLink = Link.href
Link.Click
Exit Function
End If
Next
ClickLink = ""
End Function

A basic script that uses the functions OpenBrowser and ClickLink to open up a web page and click on a link.

Set objBrowser = OpenBrowser ("www.google.com", 1000)

Call ClickLink(objBrowser, http://images.google.com/imghp?tab=wi)

More Methods to access objects in a web page.

Searching on google using the getElementsByName method

Dim IE
Set IE = CreateObject(“InternetExplorer.Application”)
IE.Visible = True
IE.Navigate2 “http://www.google.com/”
Do
WScript.Sleep(1000)
Loop while IE.Busy = TRUE
IE.document.getElementsByName(“q”)(0).value=“Classical Guitar”
IE.document.getElementsByName(“btnG”)(0).click
Set IE=Nothing

Frames - Problems within the DOM

In many cases we find that with an application that contains frames, very often we can’t access the elements inside some of the frames. This is caused by the frame having a different domain to the principal web page, causing access to the DOM of that frame to be denied. This is a security feature that stems from cross frame scripting. It basically stops unwanted access and changes occurring from outside of the domain of the frame; you can read more about it here http://msdn2.microsoft.com/en-us/library/ms533028.aspx. In a development environment developers can be made aware of this and can include the document.domain property to reference the principal domain inside of the frame. This allows access to the frames DOM.

Even though there are limitations to the DOM, there are many other objects such as xml based objects, and methods that can be used to assist us when DOM limitations are reached. A series on test automation frameworks is planned in the near future which will expand on the concepts learned in this series. You will learn how to develop a test automation framework using just VBScript and the API of an application such as Excel.

Conclusion
In this series we have looked at the various concepts involved in the automation of Internet Explorer. We have also described some very basic methods used in web application automation. We now how have a solid base to develop a test harness that takes advantage of these concepts.

For more information on TEST AUTOMATION FRAMEWORKS using VB and Excel. Please contact me through insideSQA.

Test Automation - VBA Web Automation with IE - Part 2

Introduction

In the previous entry of this series we looked at ways of creating the Internet Explorer object and how to interact with the elements contained within that particular object using the DOM. In this entry we will look at the various types of HTML element and some examples of how we can access and manipulate these particular elements.

HTML Elements

How do we interact with the different elements that are described within the DOM? The following examples take the most common elements and demonstrate some of the methods that we can use to access and manipulate them.

Links and Buttons

The following link can be accessed through a wide array of differing methods.

<a href=http://www.brownswood.co.uk name="brownslink" id="brlk" >brownswood</a >

document.Links("brlk").click
document.Links("brownslink").click
document.getElementById("brlk").click
document.getElementsByName("brownslink")(0).click
document.all("brownslink").click
document.all("brlk").click

A button can be accessed using some of the methods above. The properties of the button element below are almost identical to those of the link above.

<INPUT type="button" id=brownsBtn name=brownswood value="Click">

document.getElementById("brownsBtn").click
document.getElementsByName("brownswood")(0).click

Text Boxes

A textbox can have both its value read and set.

<INPUT userProp = "Company" type="text" id="Company Name" name="Company Name" value="Test this text box">

There are various ways to change the value of text box.

document.getElementById(“Company Name”).value=“insideSQA”
document.getElementsByName(“name”)(0).value=“insideSQA”

If the properties name and id are not available then the code below can be used to change its value.

Set Elems= document.getElementsByTagName(“INPUT”)

For each elem in Elems
if elem.userProp=“company” then
elem.value=“insideSQA”
Exit For
end if
Next

List Box

A list box usually contains a number of options that a user can select.

<SELECT size="1" name=“demo_ComboBox">
<option value=“Value 1">Value 1</option>
<option value=“Value 2" >Value 2</option>
<option value=“Value 3" >Value 4</option>
</SELECT>

Set oList=document.getElementsByName(“company type”).item(0)

optionsCount=oList.Options.length
optionOneValue=oList.Options(0).value
optionTxt=oList.Options(0).text

To select one of the options the code below can be used

oList.Options(0).Selected = true
oList.value="value 2"

Checkbox

A checkbox is either checked or unchecked.

<input type="checkbox" name="Company Checkbox">

Set oChkBox=document.getElementsByName("Company Checkbox").item(0)
oChkBox.Checked=True

Radio Button

The radio button element is an array of options where only one option can be selected.

Radio button elements are grouped using the same name.

<input type="radio" name="Company Type" value="Public" checked="checked" />
<input type="radio" name="Company Type" value="Private" />

A radio button is selected by assigning the required value to the object .

Set oRadio=document.getElementsByName(“Company Type”).item(0)
oRadio.checked=True 'This will select Public
oRadio.value="Private" 'This will select Private even if the Publci object node is being pointed to.

Tables

Tables are amongst the most difficult but useful of elements to access. When tables become nested it becomes necessary to work with the child objects of these tables, and sometimes its difficult to identify how many table structures a document contains. A great tool for element identification is the DOM inspector from Mozilla http://www.mozilla.org/projects/inspector/. With this tool it is possible to view all elements in a document, rather like an object spy that you find with a tool like HP's Quick Test Professional.

The table object provides two collections

cells – Access the cells of the table
rows – Access the rows of the table. The rows also allow access to a cells collection to access particular cells present in the row.


Row1 cell1 Row1 cell1
Row2 cell1 Row2 cell1


Access first row and the first column.

Set oTable=document.getElementById(“companyTable”)
oTable.rows(0).cells(0).outerText
oTable.cells(0).outerText

Access second row and the first column.

oTable.rows(1).cells(0).outerText
oTable.cells(2).outerText

Counting Cells and rows

oTable.rows.length
oTable.cells.length
oTable.rows(0).cells.length

Coming up..
Now that the basic concepts are covered we can now look at putting together the various parts to automate a web application. The final part of this series will demonstrate the possibilities of automating html objects within the Internet Explorer Object.

Test Automation - VB Web Automation with IE - Part 1

Internet Explorer

Internet Explorer is developed using the component object model (COM). This simple fact allows us to write code that can automate many web based applications without the need to invest in expensive automation test tools. Many market leading web automation test tools use the principles that I am going to describe here. In this article, although I am going to describe the use of Visual Basic (VB) in automating many aspects of Internet Explorer, you should note that all these working practices can be easily transcribed to other languages.
VB Development Environments

Throughout this article we will be using visual studio as our VB development environment but you can use any VB development environment. I have at times used the VB editor of MS office to get the work done!

Creating the Internet Explorer COM Object
There are two ways in which we can create the Internet Explorer COM object

- By referring to the Internet Explorer ActiveX library

Dim IE as InternetExplorer.Application
Set IE = New InternetExplorer.Application
IE.Visible = True

- By creating the object using the program identifier (progID) of the COM application.

DIM IE as object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True

Methods and Properties Available in Internet Explorer

The following list is some of the methods available in Internet Explorer. These methods can be viewed using the object browser that comes integrated with most development environments.

* Navigate2 - Navigates to a specified URL.
* Stop - Aborts the navigation.
* Refresh - Refresh the present URL
* Quit - Closes the browser. Closing a browser while its navigating to some URL is not recommended.
* GoBack - Navigates one page back.
* GoHome - Navigates to home page.
* GoForward - Navigates one page forward.

There are various properties available for use in Internet Explorer. Here is a list of some of the more commonly used properties used when automating web application.

* AddressBar - Controls whether address bar is shown. (Boolean)
* FullScreen - Maximizes window and turns off statusbar, toolbar, menubar, and titlebar. (Boolean)
* LocationName - Gets the short (UI-friendly) name of the URL/file currently viewed. (String)
* LocationURL - Gets the full URL/path currently viewed. (String)
* Path - Returns the path to the application. (String)
* Resizable - Controls whether the window is resizable. (Boolean)
* Silent - Controls if any dialog boxes can be shown. (Boolean)
* Type - Returns the type of the contained document object. (String)
* Visible - Determines whether the application is visible or hidden. (Boolean)
* Busy - Query to see if something is still in progress. (Boolean)
* ReadyState - Describes the current loading state of a page/Object. (Long)

Fundamental Actions

The following sections will demonstrate the use of these methods and properties.
- Browsing to a Web Page or site and waiting for the page to finish loading.

Dim IE

Dim urlStr
urlStr = "www.insideSQA.blogspot.com"
Set IE = CreateObject("InternetExplorer.Application")
IE.Visble = True
IE.Navigate2 urlStr
Do
DoEvents
Loop while IE.Busy = True
Msgbox urlStr & " has now loaded."
IE.Quit
Set IE = Nothing

- Enumerating all open Internet Explorer objects. The following code will form the basis of many of the object recognition techniques used on IE web applications.

Dim IEWins As SHDocVw.ShellWindows
Set IEWin = New SHDocVw.ShellWindows
Dim IEWin
For Each IEWin In IEWins
Msgbox "Name: " & IEWin.Name & vbcr & "LocationURL: " & IEWin.LocationURL
Next
For i = 0 To IEWins.Count - 1
Set IEWin = IEWins.Item(i)
Next

- Closing all open instances of Internet Explorer.

Dim IEWins As SHDocVw.ShellWindows
Set IEWins = New SHDocVw.ShellWindows
Dim IEWin()
ReDim IEWin(1 To IEWins.Count)
Dim i
For i = 1 To IEWins.Count
Set IEWin(i) = IEWins.Item(i - 1)
Next
For i = LBound(IEWin) To UBound(IEWin)
If InStr(1, IEWin(i).FullName, "iexplore.exe", vbTextCompare) Then
IEWin(i).Quit
End If
Set IEWin(i) = Nothing
Next

- Referencing an existing instance of Internet Explorer. This is a useful function as it allows you to work with instances of Internet Explorer that are already existent.

Public Function GetOpenIE(ByVal strUrl As String) As InternetExplorer
Dim insIE As InternetExplorer
Dim shWins As New ShellWindows
For Each insIE In shWins
If insIE.LocationURL Like strUrl Then
Set GetOpenIE = insIE
Exit Function
End If
Next
Set GetOpenIE = Nothing
End Function

Document Object Model

Once the Internet Explorer object is created, we need to access and interact with the objects that the web application consists of, for example, clicking on buttons, selecting web radio buttons, selecting list items, etc... These objects are defined by the html document object model or DOM.
The html DOM is a standard set of objects and includes a standard ways to access and manipulate these objects. All web applications, regardless of the technology or language that they are constructed with, make use of the DOM.
DOM Structure

The html DOM views html documents as a tree structure of elements. It is possible to access and manipulate all elements and their attributes through the DOM tree structure.
Every tag within the html structure of a document is represented as a node. When the tag is accessed or opened, all descendant tags become child nodes of the original starting node.
A tag is a construct that can have various attributes, some maybe pre defined and others may e user defined. In the following example type, value, and name are predefined attributes, and txtVal is a user defined attribute.
INPUT type="textbox" value="Chomsky" name="txtNombre" txtVal="Indepth"

Document

The upper most node in the DOM is the document, this represents the whole document. The document node has no dependant nodes, but provides various collections for different types of html element. It also provides various functions to interact with these elements.
Elements

An element is an object contained within the DOM. Each element has various means of accessing the methods and properties relating to that particular object. Al elements make use of the properties - outerText, innerText, innerHtml, tagName etc...
This piece of VBscript shows a way of accessing a particular text box in a document and changing its value.

Set txtBoxes = documents.GetElementsByName("chomskyTxt")
For i = 0 to txtBoxes.Length - 1
txtBoxes(i).value = "Sky"
next

This is the most common approach we can take when accessing elements of a web page. Using logic and programming structures, we can usually locate all elements within the document and interact with them.

Part 2 of Test Automation - Web Automation with IE will detail the different elements available in Internet Explorer, giving practical examples as how the DOM and VB can be used to access and control automatically a web application.