Showing posts with label selenium. Show all posts
Showing posts with label selenium. Show all posts

Monday, 16 January 2012

Data driven tests



Filehelpers is a great .Net library that allows you to easily import data into your code from a number of different data sources.

I use it to data drive selenium tests, reading in all the iterations in a setup method and then using the data either through iteration or targeting rows for specific data sets.

This quick start guide is pretty all you need to get data driving your tests.

http://www.filehelpers.com/quick_start.html

Thursday, 17 November 2011

Using Selenium WebDriver with C#, NUnit and the Page Object Pattern

In this quick tutorial I am going to show you how to do the following:

  • Set up Selenium Webdriver test project using the C# implementation
  • Use C# and NUnit to build a test framework
  • Use the page object model to build a simple automation test suit
  • Simple data management


Part One - Setting up the project

  • Create a C# .net class library project using the .Net 3.5 framework

    ..\ NUnit-2.5.10.11092\bin\net-2.0\framework\nunit.framework.dll

     ..\net35\*.dll
  • Add a reference to System.Configuration to the project, and the add an application  configuration file

Your solution should like this:



Part Two – Set up a simple test framework using the page object model

Design patterns such as the page object model take the pain out of planning how to manage and set up tests, they also greatly reduce maintenance. I have chosen the page object model as I believe that it is the easiest to understand and gives a robust framework that requires very little, in UI automation terms, maintenance. Read more about automation design patterns here.

For this example I am using a web application I’m working on at the moment, it’s a simple device to manage restaurant table availability. It consists of an account login page and several account utility pages. In this test I have decided to automate the account login page, and the account home page.

Set up some base classes for the project

Create a class called base.cs which will contain all your common variables, objects, methods, etc. For the moment, we just create a new WebDriver instance


using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Configuration;
using System.Linq;
using System.Text;
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

namespace Automated
{

    public class Base
    {
        public static IWebDriver driver;
        public static string _baseUrl;
        static Base()
        {
            _baseUrl = ConfigurationManager.AppSettings["baseUrl"];
        }
       
        public void NavigateTo(string url)
        {
            var navigateToThisUrl = _baseUrl + url;
            driver.Navigate().GoToUrl(navigateToThisUrl);
        }
        public void GetDriver()
        {
            //driver = new ChromeDriver();
            driver = new FirefoxDriver();
        }
    }
}

You will notice a reference to the application configuration file we created earlier. In here we are going to put the target url of the application under test. It doesn’t need to go here, but having it here allows us to make this value easily configurable without building a data management framework.

Add the following contents:

  <appSettings>
    <add key="baseUrl" value="http://www.mynewapplication.com" />
  appSettings>


There is also a “navigate to” function. This takes our application base url and tacks on any required paths that we may need to visit directly. We call this method inside the page objects.

For this example I need to create two page object models so I create a class called AccountLogin.cs and AccountHome.cs in a solution folder called Pages. We then model the pages in the classes.

By model, what we are doing is identifying actions and elements on a page and placing them inside a method that can be called on that page object. We may even group together these actions as a part of or full work flow on a page. We may also include in the page object any elements that may serve as assertions.

My two page classes look like this

Pages\AccountLogin.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.IE;
using OpenQA.Selenium;

namespace Automated
{
    public class AccountLogin : Base
    {

        public AccountLogin NavigateToLogin()
        {
            NavigateTo("/account/login");
            return new AccountLogin();
        }
      
        public AccountHome LoginAs(string username, string password)
        {
            driver.FindElement(By.Id("email")).SendKeys(username);
            driver.FindElement(By.Id("password")).SendKeys(password);
            driver.FindElement(By.Id("submit")).Submit();
            return new AccountHome();
        }
    }
}

In the accountlogin page I have modelled the login action (LoginAs). This action fills the login fields and then submits the login form, finally returning the proceeding page, account home. I am using the FindElement method with the field element and submit Ids being used as the identifiers.

Pages\AccountHome

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenQA.Selenium;

namespace Automated
{
    class AccountHome:Base
    {

       
        public bool User_Welcome_Name(String userFullName)
        {
            IWebElement userWelcome = driver.FindElement(By.ClassName("Welcome");
            bool result = userWelcome.Text.Contains(userFullName);
            return result;
        }
    }
}

In the account home page model, for this example I have identified an object on the page that will tell me if the login has been successful for the user. I have modelled this object as a method that returns a Boolean value, it will return true if the users details that it finds match those that are expected. I am using the FindElement method of Webdriver to locate the text by finding the css classname “Welcome”, I then look in the text to see if the username is contained within.

Bringing the two page objects together as a test

Where possible I try to marry up a test class to a page object, and the actions within that object, for example, the login test class only contains tests that test the login page. It’s similar to functional decomposition only we are breaking an application down into pages. This keeps the testing simple. Sometimes we may also need to introduce workflow testing where we are testing a flow through several pages.

A test class usually contains a Set Up method, one or more tests, and a Tear Down method. The set up method will usually do something like get the webdriver object and make it available to all tests within the class. The Tear Down will usually tidy up the test environment or reset values for the next set of tests. Either of these can be run globally as either a SetUpFixture or TearDownFeature for all tests, or equally, within each test class to be run before and after each test.

A test procedure using selenium requires the following to happen

  1. Instantiate the WebDriver for the browser you wish to automate. This is what we do in the Set Up method, calling the GetDriver method from the base class. This opens up browser specified.
  2. Navigate to a start page. We call the NavigateTo function in the base class to do this. This Method grabs our base url from the app.config file and appends whatever route that we give to the method. See data management below for how to manage different data types such as URLs.
  3. Execute one or more actions, and assert the outcome of those actions.
  4. Tear Down the WebDriver if required, and run clean up commands.

The first test I have written for this example is a successful login test. It navigates to the account login page, fills in the login fields, and submits the login form; it then waits for the account home page to display before executing a checkpoint action to make sure we are on the right page.

The test class looks like this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NUnit.Framework;

namespace Automated
{
    [TestFixture]
    class LoginTests : AccountLogin
    {

        [SetUp]
        public void Setup()
        {
            GetDriver();
            driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
        }

        [Test]
        public void Login_Successful()
        {
            AccountLogin accountLogin = NavigateToLogin();
            // Assert.IsTrue(accountLogin.Sign_Button_Visible());
            AccountHome accountHome = LoginAs("tamgus.bultgreb@yahoo.com", "itsasecret");
            Assert.IsTrue(accountHome.User_Welcome_Name("Tamgus Bultgreb"));

        }

        [TearDown]
        public void TearDown()
        {
            // Some funky stuff here..
        }

    }
}

As you can see, there are some nice hard coded values in the test such as user name and password. These should be extracted out of the test and placed in a test data repository, more about that later.

Running the tests

There are several ways to run the tests, from the command line, with NUnits own UI, or with a test add tool such as test driven. Whilst developing I go for Test Driven as it allows me to run tests quickly from visual studio, but during test runtime, with tests being triggered from a CI server, you need to go down the command line route.

Managing the test data

Your tests will almost always need to have some kind of data to drive them. In the example above we needed to provide a username, a password, and some URLs, we also needed to provide data for some checkpoints.

There are several ways of managing data, each with its own merits, from experience the main mechanisms I have seen successfully used are constant files and datasheets. Constant files are basically static classes that contain all the data that your tests will use assigned to variables. This is a nice way to keep all your test artefacts manageable inside the project. It does mean, though, that data can’t be easily manipulated close to runtime, and if you wish to change data, the test project must be rebuilt. It also stops you from using dynamic data at runtime.

Datasheets give you a lot of flexibility when feeding data into tests, you have the ability to implement different scenarios with the same tests, you can also feed data into your datasheets dynamically, and you can update data at any moment up to the test runtime. However, using datasheets tends to remove some of the robustness of your tests. When you are working with constant files, you are immediately aware of any break in your data, datasheets tend to promote a more “detached” approach to data management which can often lead to incorrect data being used within tests.

When choosing which approach to take you can look at the following

Test flexibility
Data reliability

Test flexibility refers to some of the objectives of your automation efforts. If you wish to have flexibility with the data going into your tests, or you wish a business representative or such to feed in data to your tests, the datasheet route is more favourable. If your data is simple and unchanging, constant files usually give a more stable and robust test project, which should always be one of your main objectives.

Data reliability refers to how “safe” is your data from changing. This is one point that I push more than any in automation. Apart from bad programming, data management is the biggest killer of automation projects. Uncontrollable data will mean lots of failed tests. A good example would not be having complete control over a user in an application, and someone changes the password for that user, or the language that the user sees an application in. This would break your test where you are using either one of those items in the test. In our example above, we have a test called successful login, if someone else has access to the user being used and the data, they could change the password and that successful login test will now fail for data reasons, and not a code bug. This adds to the maintenance debt.

Here is an example of how I can add data values into the base class in the form of constants and use that to drive the tests.

In the base class add test data values to the constructor

public class Base
{
    public static IWebDriver driver;

    public static string _baseUrl, userName, password, userFullName;

    static Base()
    {
        _baseUrl = ConfigurationManager.AppSettings["baseUrl"];
        userName = "livetest999@gmail.com";
        password = "testtest";
        userFullName = "Live Test";        
    }

In the test, you can now reference these values as follows
   
[Test]
public void Login_Successful()
{

    AccountLogin accountLogin = NavigateToLogin();
    Assert.IsTrue(accountLogin.Sign_Button_Visible());
    AccountHome accountHome = LoginAs(userName, password);
    Assert.IsTrue(accountHome.UserWelcomeName(userFullName));

}

If you do go down the datasheet route be prepared to invest time in setting up a data management mechanism in your test suite that can read in and parse data for the tests. Libraries such as FileHelpers can greatly facilitate this.

Ok, that’s it for now. Hopefully you can now set up a test project using the .NET version of WebDriver, implement tests using a common design pattern and implement a simple data management technique. We have not touched upon a lot of UI automation here, just the basics. In the coming months I would like to explore cross browser testing, and localisation testing in more detail.

For more information on NUnit visit their site.

For indepth information on WebDriver visit the Selenium WebDriver Project Wiki.

Saturday, 24 September 2011

UI Automation: Avoiding Failure


I have developed and worked on many UI automation frameworks using both commercial and open source tools, some of these have been more successful than others. Regardless of the tool you use there are some common indicators that you can help you identify if you are going down the wrong path.

No business backed strategy to begin UI automation

UI automation is expensive; it requires a lot of time and effort. Make sure you have enough secured resource to carry out your plans. If you don’t, that resource will be pulled somewhere else and you will end up with several unfinished automation projects that have cost the business money but bring no value.

No clear reason to automate

Why are we automating?  If you have no clear objectives, what’s the point?

Some of the reasons why we should automate
  • to reduce the feedback loop between code implementation and possible defect detection
  • to reduce the amount of time we spend in resource intensive activities such as regression testing
  • to remove human error from test execution

       …and we should automate when:
  • we have stable functionality
  • we have the skill necessary to automate

 No design pattern is being applied to the test design

There is no need to reinvent the wheel with UI automation. Using a recognised design pattern like the page object model will reduce the time it takes to automate. It will also give you cleaner and easier to understand tests. Other engineers will be able to pick up the project and understand it.

Tests are brittle

If small changes to the AUT cause the tests to break then your tests are too brittle and not easily maintainable. 

Things to look out for:

  • Too many checkpoints per test. Always ask yourself what the value of a checkpoint is before implementing it. The more checkpoints we have the greater the chance of test failing. In automation I only test for critical information. Many people will put checkpoints in for page layout, sizing, or styles applied. This is useful but is something that can become hazardous when you start cross browser testing, or testing with different resolutions.
  • Hard coded data. Tests should be driven by a driver class or datasheet if data is required. If you don’t have complete control over your test data then it could change and your tests will break. Being able to feed data in gives you much greater control over the tests and makes them much more useful.
  • Over engineered. Is your test code more complex than the code it is testing? Make sure that your tests are not failing due to over complicated code. I've seen this happen so many times!
  • Tests that depend on the test results of another test. This is a very common mistake. If one test with dependencies fails all dependent tests will also fail.

 No control over the data that drives the tests. When the data changes the tests fail.

Data control is one of the most common reasons why UI automation project fail. Any automation project should have its own environment where data is guaranteed. If the environment is shared then you must have a way of protecting the data your tests use. It’s very frustrating when your data is tampered with and the result is failed tests.

Tests fail because the identification of page or application objects is using ambiguous names, or auto generated identifiers.

UI automation requires us to identify page objects through their properties or location. Many applications will generate unique changing names for objects. Using these names is not secure. Try to get developers to add decent identifiers to objects. It will help keep the tests from becoming brittle and save time during test creation.

Large xpath identifiers are also a common cause of test failure. Try to avoid!

More time is spent on maintaining tests than testing

This is a sure sign you have problems. 

Things to look out for
  • A large number of tests fail on each test run
  • Engineers are “Baby sitting” test runs. Helping the tests complete is not automation!
  • You seem to be spending all your resource on one application. You never get to automate anything else
  • New test creation drops

Nobody knows what the tests coverage is

One thing I’ve found in agile environments is that although UI automation is high on the agenda, there is little understanding of what the test coverage is. Developers and testers churn out tests, get excited, but there is no understanding of what really has been tested. This means that you still have to invest in manual regression runs to assure yourself. UI automation is a safety harness, so we need to know how much protection it is giving us.


Friday, 27 March 2009

When Should We Automate the User Interface in an Agile Environment?

Many misunderstand automation in the context of agile. Picking up a user interface test tool and automating from the word go is never going to work. Automation is often associated directly to UI tests but in an agile environment, automation refers to the whole test strategy. One of our primary goals in agile automation is to develop automated unit tests that will help us monitor the pulse of our project. From there we can look at developing integration tests, coupling together groups of unit tests, or we can look straight to UI automation. UI automation allows us to do both integration and system testing.

In an agile environment we have to look at the real return on investment when deciding what and how we automate in terms of the UI.

Examining the following factors may make this task easier:

1. Risk - A business or safety critical application is always a candidate. Use the other factors to assess the correct moment to automate. Low business impact applications should really be avoided unless work flows are simple.

2. Maturity/Stability - If the web application is still in primary development stages then there will be lots of changes to the UI. Sometimes it is better to wait for a Beta release before beginning UI automation. At this point there will usually be a lower frequency of change or less impacting changes. Waiting until this point saves a lot of time, and reduces the maintenance overhead.

3. Resource - Automation is labour intensive. If you can only devote a small amount of time to UI automation then you are probably not going to have success. Time block an automation project if necessary, it will bring benefit.

4. Change Index - Web applications with a high change index usually require the largest amount of maintenance. Keeping up with a site that has constantly changing layout or content can kill an automation project.

5. Complexity - Large complex systems that use a host of differing technologies, and contain work flows that cross these technologies, should be avoided. Unless, that is, you have the necessary resources and tools to combat this.

6. Technology - If you don't have well supported UI automation tools such as QuickTest Pro or Selenium, or the expertise to use the ones you have got, then your automation project could extend far beyond the original plan. The problem of not having expertise in your automation applications, or even in automation, is that you could find yourself with a very brittle framework that requires constant maintenance, increasing the cost and burden of automation.

Friday, 30 January 2009

Agile Metrics

As a tester you will always find yourself absorbed in metrics creation, reporting on anything and everything that your management team wishes to report on. However, in agile development environments, sometimes its very difficult to assess the importance of metrics. I've been looking at some ways to provide management teams with interesting and useful information to measure our success.

One of the most fundamental approaches we can take towards metrics collection in an agile environment is the Return On Investment that a collection of metrics can give us. That is, how can our metrics help us provide more business value.

Using measurement goals such as only having 5 serious defects found in production after release give us more incentive to produce less buggy code.

Measuring the time it takes from the conception of a user story to seeing that user story providing business value also gives us an idea of how we are performing as a team. This is known as cycle time.

These metrics give us valuable information that provide us with the correct means to motivate a team.

Using the following principles, we can develop metrics that give us a decent ROI from our metrics:
Achieving SLA's relating to service and availability of products and systems.
  1. Low cycle time, that is, a high throughput of change.
  2. Low amount of unplanned work.
  3. Effective process improvement practices.
  4. Effective change management.
We can build metrics around these principles that give us usable metrics that bring value and motivation to a team.

Sunday, 28 September 2008

Selenium

Selenium is an automated web testing application that provides a real alternative to developing solid test scripts without investing in heavy weight test tools such as QuickTest Professional.

An open source project developed by ThoughtWorks, Selenium has been growing in popularity amongst some of the worlds biggest development centres. Google has deployed a selenium test framework to assist in many of its projects, and has also provided some of the principle contributors to the selenium project.

Selenium provides an IDE and a remote control serverto allow complete control over the development and implementation of test frameworks. How Selenium Works is a great article that provides an overview of the general functioning of a selenium framework.

Selenium is unique in that it provides support for a multitude of platforms, OS, and programming languages, making it a tool that can be integrated into many different test environments.

Selenium integrates well with existing test frameworks. The following article on thinkvitamin describes an innovative regression test framework using Selenium and and another open source project, Hudson, a build server designed for Java applications.

Selenium & Hudson

Selenium has a lot of support and some very interesting contributors. It is definitely a tool that should be considered when looking for a strong feature packed web testing application. The ease of use, integration, and its status as an open source project with innovative backers such as Google means that there is no doubt that this will become one of the standard web test tools in the near future.

Selenium:

http://selenium.openqa.org/