Friday, 30 January 2009

Agile Metrics

As a tester you will always find yourself absorbed in metrics creation, reporting on anything and everything that your management team wishes to report on. However, in agile development environments, sometimes its very difficult to assess the importance of metrics. I've been looking at some ways to provide management teams with interesting and useful information to measure our success.

One of the most fundamental approaches we can take towards metrics collection in an agile environment is the Return On Investment that a collection of metrics can give us. That is, how can our metrics help us provide more business value.

Using measurement goals such as only having 5 serious defects found in production after release give us more incentive to produce less buggy code.

Measuring the time it takes from the conception of a user story to seeing that user story providing business value also gives us an idea of how we are performing as a team. This is known as cycle time.

These metrics give us valuable information that provide us with the correct means to motivate a team.

Using the following principles, we can develop metrics that give us a decent ROI from our metrics:
Achieving SLA's relating to service and availability of products and systems.
  1. Low cycle time, that is, a high throughput of change.
  2. Low amount of unplanned work.
  3. Effective process improvement practices.
  4. Effective change management.
We can build metrics around these principles that give us usable metrics that bring value and motivation to a team.

Sunday, 28 December 2008

Quick Tips: Wiki - Internal Team Information Sharing

When I'm working to get new processes published, or writing informal tests that need to be accessed by other team members, I sometimes feel restricted by corporate intranet and share point services. These tools are great for publishing formal information, or storing common access templates, but they limit your ability to express ideas and communicate effectively within smaller teams.

Using a hosted wiki service such as Google Sites within test or development environments can give you the freedom to develop processes, applications, and test documentation without formally committing information to clients or end users. You can freely communicate with each other using these very professional tools.

If you are creating processes for a test environment, it is far more productive to get practices that are being designed out to your team as soon as possible. This allows your process implementation to be a continuous movement, taking away the pain of implementing a multitude of practices in one go.

Wiki's provides the ability to publish work and forward relevant links or information to the team as and when you feel. Often, with an intranet, the content has to be managed and controlled and adhere to standards. With your internal team wiki, you can remove yourself from these restrictions and have a open shared information portal.

At some point some of the information that we draft out within the wiki needs to be formalised and perhaps published to a Sharepoint serves or intranet. When you finally do publish you at least know that the information is refined and in common use.

A Wiki is also a great tool for constructing adhoc test documentation that can be shared freely between team users. If I have to develop tests in an agile environment, I set up a page within a wiki and create tables that allow the input of values. I can share these tests with all team members very easily without having to formally publish them. This methodology can be used to construct tests within other test process environments.

Sunday, 2 November 2008

Agile Testing

I've just started work in a development team that has recently adopted scrum processes. Part of my initial tasks were centred around the implementation of a test process that works within scrum.

Here are a few links that will help you on your way to discover the delights of agile testing:
Even if you're not moving towards agile development practices, some of the techniques, tools, and practices referenced in the above links can greatly improve the productivity and reputation of the tester as valuable member of the development team. Take a look.

Sunday, 28 September 2008

Selenium

Selenium is an automated web testing application that provides a real alternative to developing solid test scripts without investing in heavy weight test tools such as QuickTest Professional.

An open source project developed by ThoughtWorks, Selenium has been growing in popularity amongst some of the worlds biggest development centres. Google has deployed a selenium test framework to assist in many of its projects, and has also provided some of the principle contributors to the selenium project.

Selenium provides an IDE and a remote control serverto allow complete control over the development and implementation of test frameworks. How Selenium Works is a great article that provides an overview of the general functioning of a selenium framework.

Selenium is unique in that it provides support for a multitude of platforms, OS, and programming languages, making it a tool that can be integrated into many different test environments.

Selenium integrates well with existing test frameworks. The following article on thinkvitamin describes an innovative regression test framework using Selenium and and another open source project, Hudson, a build server designed for Java applications.

Selenium & Hudson

Selenium has a lot of support and some very interesting contributors. It is definitely a tool that should be considered when looking for a strong feature packed web testing application. The ease of use, integration, and its status as an open source project with innovative backers such as Google means that there is no doubt that this will become one of the standard web test tools in the near future.

Selenium:

http://selenium.openqa.org/

Tuesday, 27 November 2007

Risk Based Regression Test Strategy

Using a risk based strategy to develop a regression test plan

Designing a regression test plan can be a very laborious task. Whilst we are trying to incorporate into the test plan all those test tests that we feel cover the functionality or requirement of the test, we very often miss out important tests. If we are aware of a particular tests impact within the test plan and the risk that we have by not including that test, we can create a much more thorough test plan. Assigning a risk index to each and every test that we have, allowing the ability to change the risk weighting of a particular test, allows us to have a greater control over the selection process. Using a risk based test selection strategy we can test thoroughly, the required parts of an application. In this article I’ll discuss a method that applies risk analysis to the creation of a regression test plan.

Risk based selection

The basic principle behind this strategy is to apply a risk index to a test case and then select those test cases that have reached a sufficiently high enough level of risk to be involved in the test plan.

A simple risk model

To calculate risk we need to identify measurable elements within a test that can be calculated to provide the risk index. Two key elements that we can consider are

The probability of error occurring with the test functionality or of the scenario occurring. (Probability)

The consequence of a fault occurring in a production environment. (Cost)

These two elements are described in Amaland’s [1] , and form the elements that we will use to calculate risk exposure.

Risk exposure is calculated mathematically using the following formula:

RE(f) = P(f) * C(f)

Risk Exposure = Probability * Cost

The calculation is based on a scale of importance that is applied to each of the functions within the risk exposure calculation. To facilitate the ease of weighting probability and cost we use a five point scale with one being low and five being high.

Selection

Selecting tests for the regression test plan consists of a two part process: The selection of test cases and the selection of end to end scenario coverage.

Test case selection

1. Cost estimation of each test case – It is important that we, as tests analysts, take an objective viewpoint on the cost of an error. We should consider ourselves as the client when making these decisions. What would they consider to be costly in terms of errors? Losing the client perspective can very often cause incorrect test case weighting.

2. Extract an error probability weighting. Error probability is a calculation based on various metrics. The easiest way to calculate probability is to count the number and identify the severity of defects associated with a particular functionality. More precise methods also include information garnered from code analysis, although, this type of metric can be time consuming to obtain.

3. Once the two functions have been calculated we can then calculate the Risk Exposure. We now have a means to extract tests based on there risk exposure.

Scenario selection for regression and system testing

A collection of scenarios is what forms the basic test plan. Scenarios are designed to cover a multitude of test cases and application modules working together. When selecting scenarios we use two principle strategies.

1. Selection of scenarios to cover the critical modules of an application.

2. Selection of scenarios to cover the greatest number of tests.

When scenarios have been created we can use traceability a traceability matrix to show the relation between test cases and scenarios. Using this matrix we can construct a risk based scenario test plan using the following approach:

1. Calculate the risk exposure for each scenario: Using the traceability matrix, sum the risk exposure of all tests covered by the scenario.

2. Select the scenarios with the highest risk exposure.

This is a very basic approach that allows a greater coverage of functionality. This is a practice that we would normally associate with regression testing or cyclic system testing but can also be applied to other areas of testing. It is a method that greatly improves error detection by focussing on critical components within an application.


This article is adapted from the following:
Stale Amland, “Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Vol. 53, 2000, pp. 287 - 295.

Yanping Chen, Robert L. Probert A Risk-based Regression Test Selection Strategy, University of Ottawa, Ontario, Canada, Fast Abstract ISSRE 2003, Chillarege Press

Friday, 26 October 2007

Test Automation - Visual Basic Web Automation with IE - Part 3

Finally... Automation

The final part of this series on web application automation will focus on some examples using the concepts discussed in the previous two parts. We shall also discuss some of the limitations of using the DOM and the alternatives available to automate web application testing.

Automating a Web Page

The basic automation of web pages is simple and requires only the combination of the concepts discussed in the first two parts of this series. Before looking in more detail at the examples of automation we need to take into consideration the following DOM limitations mean that not all web pages can be easily automated.

  • Only HTML/DHTML web content can be automated. Web applications based on Java or other development frameworks such as .NET can not always be automated.

  • Web pages are not always developed with automation in mind. It is often necessary to create coding procedures within a development team to include the required properties within the elements to facilitate automation.

Locating Elements

There are various ways in which we can identify elements with the document.

  • Viewing the source code directly.
  • Using a DOM inspector
DOM inspectors are the preferred method as they usually allow the identification of elements by means of clicking on them or hover over them. Source code viewing is effective and can often uncover elements that have not been recognised by a DOM inspector. The only draw back with source code viewing is that it can be time consuming and not always accurate.

Automation

The following examples use Visual Basic Script to automate browser operations.

This function opens a browser and will then navigate to a web page.

Function OpenBrowser(url, sleepTime)
Dim IE
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate2 url
Do
WScript.Sleep(sleepTime)
Loop while IE.Busy = TRUE
Set OpenBrowser = IE
End Function


A function to click on link.

Function ClickLink( byval oIE, strLnk)
Dim Link
For Each Link In oIE.Document.links
If Instr(strLnk, Link.href) > 0 Then
ClickLink = Link.href
Link.Click
Exit Function
End If
Next
ClickLink = ""
End Function


A basic script that uses the functions OpenBrowser and ClickLink to open up a web page and click on a link.

Set objBrowser = OpenBrowser ("www.google.com", 1000)

Call ClickLink(objBrowser,
http://images.google.com/imghp?tab=wi)

More Methods to access objects in a web page.

Searching on google using the getElementsByName method

Dim IE
Set IE = CreateObject(“InternetExplorer.Application”)
IE.Visible = True
IE.Navigate2 “
http://www.google.com/
Do
WScript.Sleep(1000)
Loop while IE.Busy = TRUE
IE.document.getElementsByName(“q”)(0).value=“Classical Guitar”
IE.document.getElementsByName(“btnG”)(0).click
Set IE=Nothing


Frames - Problems within the DOM

In many cases we find that with an application that contains frames, very often we can’t access the elements inside some of the frames. This is caused by the frame having a different domain to the principal web page, causing access to the DOM of that frame to be denied. This is a security feature that stems from cross frame scripting. It basically stops unwanted access and changes occurring from outside of the domain of the frame; you can read more about it here http://msdn2.microsoft.com/en-us/library/ms533028.aspx. In a development environment developers can be made aware of this and can include the document.domain property to reference the principal domain inside of the frame. This allows access to the frames DOM.

Even though there are limitations to the DOM, there are many other objects such as xml based objects, and methods that can be used to assist us when DOM limitations are reached. A series on test automation frameworks is planned in the near future which will expand on the concepts learned in this series. You will learn how to develop a test automation framework using just VBScript and the API of an application such as Excel.

Conclusion
In this series we have looked at the various concepts involved in the automation of Internet Explorer. We have also described some very basic methods used in web application automation. We now how have a solid base to develop a test harness that takes advantage of these concepts.

For more information on TEST AUTOMATION FRAMEWORKS using VB and Excel. Please contact me through insideSQA.

Test Automation - VBA Web Automation with IE - Part 2

Introduction

In the previous entry of this series we looked at ways of creating the Internet Explorer object and how to interact with the elements contained within that particular object using the DOM. In this entry we will look at the various types of HTML element and some examples of how we can access and manipulate these particular elements.

HTML Elements

How do we interact with the different elements that are described within the DOM? The following examples take the most common elements and demonstrate some of the methods that we can use to access and manipulate them.

Links and Buttons

The following link can be accessed through a wide array of differing methods.

<a href=http://www.brownswood.co.uk name="brownslink" id="brlk" >brownswood</a >

document.Links("brlk").click
document.Links("brownslink").click
document.getElementById("brlk").click
document.getElementsByName("brownslink")(0).click
document.all("brownslink").click
document.all("brlk").click

A button can be accessed using some of the methods above. The properties of the button element below are almost identical to those of the link above.

<INPUT type="button" id=brownsBtn name=brownswood value="Click">

document.getElementById("brownsBtn").click
document.getElementsByName("brownswood")(0).click

Text Boxes

A textbox can have both its value read and set.

<INPUT userProp = "Company" type="text" id="Company Name" name="Company Name" value="Test this text box">

There are various ways to change the value of text box.

document.getElementById(“Company Name”).value=“insideSQA”
document.getElementsByName(“name”)(0).value=“insideSQA”

If the properties name and id are not available then the code below can be used to change its value.

Set Elems= document.getElementsByTagName(“INPUT”)

For each elem in Elems
if elem.userProp=“company” then
elem.value=“insideSQA”
Exit For
end if
Next

List Box

A list box usually contains a number of options that a user can select.

<SELECT size="1" name=“demo_ComboBox">
<option value=“Value 1">Value 1</option>
<option value=“Value 2" >Value 2</option>
<option value=“Value 3" >Value 4</option>
</SELECT>

Set oList=document.getElementsByName(“company type”).item(0)

optionsCount=oList.Options.length
optionOneValue=oList.Options(0).value
optionTxt=oList.Options(0).text

To select one of the options the code below can be used

oList.Options(0).Selected = true
oList.value="value 2"

Checkbox

A checkbox is either checked or unchecked.

<input type="checkbox" name="Company Checkbox">

Set oChkBox=document.getElementsByName("Company Checkbox").item(0)
oChkBox.Checked=True

Radio Button

The radio button element is an array of options where only one option can be selected.

Radio button elements are grouped using the same name.

<input type="radio" name="Company Type" value="Public" checked="checked" />
<input type="radio" name="Company Type" value="Private" />

A radio button is selected by assigning the required value to the object .

Set oRadio=document.getElementsByName(“Company Type”).item(0)
oRadio.checked=True 'This will select Public
oRadio.value="Private" 'This will select Private even if the Publci object node is being pointed to.

Tables

Tables are amongst the most difficult but useful of elements to access. When tables become nested it becomes necessary to work with the child objects of these tables, and sometimes its difficult to identify how many table structures a document contains. A great tool for element identification is the DOM inspector from Mozilla http://www.mozilla.org/projects/inspector/. With this tool it is possible to view all elements in a document, rather like an object spy that you find with a tool like HP's Quick Test Professional.

The table object provides two collections

cells – Access the cells of the table
rows – Access the rows of the table. The rows also allow access to a cells collection to access particular cells present in the row.


Row1 cell1 Row1 cell1
Row2 cell1 Row2 cell1


Access first row and the first column.

Set oTable=document.getElementById(“companyTable”)
oTable.rows(0).cells(0).outerText
oTable.cells(0).outerText

Access second row and the first column.

oTable.rows(1).cells(0).outerText
oTable.cells(2).outerText

Counting Cells and rows

oTable.rows.length
oTable.cells.length
oTable.rows(0).cells.length

Coming up..
Now that the basic concepts are covered we can now look at putting together the various parts to automate a web application. The final part of this series will demonstrate the possibilities of automating html objects within the Internet Explorer Object.