blazemeter - JMeter cloud testing

I've been using Apache's performance test tool JMeter for many years now, but struggle to find enough time and resource to create a decent distributed network of JMeter engines that can simulate large and realistic load. Today I stumbled upon http://blazemeter.com/ which allows just that. Its a cloud based performance test service that gives you access to multiple instances of JMeter that are geographically distributed on a variety of different server types.

To use blazemeter you either upload your own JMeter scripts or give blazemeter a url and let it write the script for you, you then create your scenarios before finally executing your test. Blazemeter has many great features, including some detailed analysis and test management capabilities that make this a compelling option when it comes to investing in performance testing.

Blazemeter operates a similar business model to browsermob.com (now Neustar Web Performance), where you pay for the number of test engines, different server types, on demand usage, etc.

There is a comprehensive free trial available that will allow you to experiment with features and tests for a couple of weeks.

For more information on JMeter visit http://jmeter.apache.org/.

When Should We Automate the User Interface in an Agile Environment?

Many misunderstand automation in the context of agile. Picking up a user interface test tool and automating from the word go is never going to work. Automation is often associated directly to UI tests but in an agile environment, automation refers to the whole test strategy. One of our primary goals in agile automation is to develop automated unit tests that will help us monitor the pulse of our project. From there we can look at developing integration tests, coupling together groups of unit tests, or we can look straight to UI automation. UI automation allows us to do both integration and system testing.

In an agile environment we have to look at the real return on investment when deciding what and how we automate in terms of the UI.

Examining the following factors may make this task easier:

1. Risk - A business or safety critical application is always a candidate. Use the other factors to assess the correct moment to automate. Low business impact applications should really be avoided unless work flows are simple.

2. Maturity/Stability - If the web application is still in primary development stages then there will be lots of changes to the UI. Sometimes it is better to wait for a Beta release before beginning UI automation. At this point there will usually be a lower frequency of change or less impacting changes. Waiting until this point saves a lot of time, and reduces the maintenance overhead.

3. Resource - Automation is labour intensive. If you can only devote a small amount of time to UI automation then you are probably not going to have success. Time block an automation project if necessary, it will bring benefit.

4. Change Index - Web applications with a high change index usually require the largest amount of maintenance. Keeping up with a site that has constantly changing layout or content can kill an automation project.

5. Complexity - Large complex systems that use a host of differing technologies, and contain work flows that cross these technologies, should be avoided. Unless, that is, you have the necessary resources and tools to combat this.

6. Technology - If you don't have well supported UI automation tools such as QuickTest Pro or Selenium, or the expertise to use the ones you have got, then your automation project could extend far beyond the original plan. The problem of not having expertise in your automation applications, or even in automation, is that you could find yourself with a very brittle framework that requires constant maintenance, increasing the cost and burden of automation.

Managing Defects

In an agile world we have to tackle defects in a different manner to cope with the pace and working practices. Here's one way of coping.

Develop a defect notebook, where defects are noted down and passed on immediately to the developer. Do this instead of logging every defect and you will save everybody time and greatly improve the quality of code that gets built.

Its a simple process.

When you detect a defect, inform the implicated developer immediately and discuss the defect. This is an important discussion and changes the way that we the code. In the developer/tester discussion make sure you cover the follow points.

• Is this a defect? Between you and the developer, you will be able to establish if it really is a defect or if it is expected behaviour. Get a product owner involved if necessary. Most importantly note it down in a note book.
• Can it be corrected now? The developer will be able to make a decision as to whether the defect can be fixed immediately or whether it will require more complex analysis. Ideally, the developer will fix it in the moment, not disrupting the work rate, and most importantly for the tester, reducing the need to raise a defect report.
• Does this defect require more analysis? If the case is yes, then before logging the defect, arrange a follow up the same day, or at the latest, the following day. This keeps the defect fresh and closer to the time that the original code was developed. This is important as the higher the turn around time for the developer writing the code and the defect being resolved, the slower the fix time will become. If a developer has to wait weeks before tackling a defect, it is likely that they will have forgotten the thought process that went into the development, thus increasing the time required to analyse and eventually fix.

If the defect requires more than a few hours to fix then log it. It maybe the case that the defect gets moved further down the backlog or requires input from others. It is in these cases where there is a benefit in tracking and monitoring a defect.The discussion with the developer can lead to an immediate solution or the necessity to invest more time in analysis. We only log the defect when we know that we have identified a serious issue.

Although not logging all defects, we still have our notes that we can use to prompt actions related to follow ups to analysis, or to ensure that the developer has done the promised fix. This note book can become vital, so look after it!

Your developers will more than likely adapt to this way of working as it helps them to have this quick quality reaction time to their code efforts.

Quick Tips: Learn From Others!

Its great to find other sources of great Agile QA info. Have a look a look at this great site full of more than just agile test tips.

http://www.quicktestingtips.com/tips/

Check out the authors for even more info:

• Jonathan Kohl
• Karen Johnson
• Mike Kelly
  

Checking Mail Automatically

In those situations where you need to automatically test the delivery of email , and you are unable to get access to a mail server through an api, what other options do you have to check this delivery automatically? Running a script to open up your mail client and check for mail can be cumbersome and unreliable. One method that I have used successfully has been identifying a mail service that gives access via an Atom or RSS feed.

Googlemail provides a limited but reliable Atom feed for its mail service. Sign in to googlemail account, open another browser tab or window and goto > https://mail.google.com/mail/feed/atom/. You will now see all your unread messages!

Using the .net WebClient class will allow you to input credentials against this url,you can then create an xml object and use the DOM to access particular tags and get the info you require.

Quick Tips: Wiki - Internal Team Information Sharing

When I'm working to get new processes published, or writing informal tests that need to be accessed by other team members, I sometimes feel restricted by corporate intranet and share point services. These tools are great for publishing formal information, or storing common access templates, but they limit your ability to express ideas and communicate effectively within smaller teams.

Using a hosted wiki service such as Google Sites within test or development environments can give you the freedom to develop processes, applications, and test documentation without formally committing information to clients or end users. You can freely communicate with each other using these very professional tools.

If you are creating processes for a test environment, it is far more productive to get practices that are being designed out to your team as soon as possible. This allows your process implementation to be a continuous movement, taking away the pain of implementing a multitude of practices in one go.

Wiki's provides the ability to publish work and forward relevant links or information to the team as and when you feel. Often, with an intranet, the content has to be managed and controlled and adhere to standards. With your internal team wiki, you can remove yourself from these restrictions and have a open shared information portal.

At some point some of the information that we draft out within the wiki needs to be formalised and perhaps published to a Sharepoint serves or intranet. When you finally do publish you at least know that the information is refined and in common use.

A Wiki is also a great tool for constructing adhoc test documentation that can be shared freely between team users. If I have to develop tests in an agile environment, I set up a page within a wiki and create tables that allow the input of values. I can share these tests with all team members very easily without having to formally publish them. This methodology can be used to construct tests within other test process environments.

Test Automation - Visual Basic Web Automation with IE - Part 3

Finally... Automation

The final part of this series on web application automation will focus on some examples using the concepts discussed in the previous two parts. We shall also discuss some of the limitations of using the DOM and the alternatives available to automate web application testing.

Automating a Web Page

The basic automation of web pages is simple and requires only the combination of the concepts discussed in the first two parts of this series. Before looking in more detail at the examples of automation we need to take into consideration the following DOM limitations mean that not all web pages can be easily automated.

• 
  Only HTML/DHTML web content can be automated. Web applications based on Java or other development frameworks such as .NET can not always be automated.
• 
  Web pages are not always developed with automation in mind. It is often necessary to create coding procedures within a development team to include the required properties within the elements to facilitate automation.

Locating Elements

There are various ways in which we can identify elements with the document.

• Viewing the source code directly.
  
• Using a DOM inspector
  

DOM inspectors are the preferred method as they usually allow the identification of elements by means of clicking on them or hover over them. Source code viewing is effective and can often uncover elements that have not been recognised by a DOM inspector. The only draw back with source code viewing is that it can be time consuming and not always accurate.

Automation

The following examples use Visual Basic Script to automate browser operations.

This function opens a browser and will then navigate to a web page.

Function OpenBrowser(url, sleepTime)
Dim IE
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate2 url
Do
WScript.Sleep(sleepTime)
Loop while IE.Busy = TRUE
Set OpenBrowser = IE
End Function

A function to click on link.

Function ClickLink( byval oIE, strLnk)
Dim Link
For Each Link In oIE.Document.links
If Instr(strLnk, Link.href) > 0 Then
ClickLink = Link.href
Link.Click
Exit Function
End If
Next
ClickLink = ""
End Function

A basic script that uses the functions OpenBrowser and ClickLink to open up a web page and click on a link.

Set objBrowser = OpenBrowser ("www.google.com", 1000)

Call ClickLink(objBrowser, http://images.google.com/imghp?tab=wi)

More Methods to access objects in a web page.

Searching on google using the getElementsByName method

Dim IE
Set IE = CreateObject(“InternetExplorer.Application”)
IE.Visible = True
IE.Navigate2 “http://www.google.com/”
Do
WScript.Sleep(1000)
Loop while IE.Busy = TRUE
IE.document.getElementsByName(“q”)(0).value=“Classical Guitar”
IE.document.getElementsByName(“btnG”)(0).click
Set IE=Nothing

Frames - Problems within the DOM

In many cases we find that with an application that contains frames, very often we can’t access the elements inside some of the frames. This is caused by the frame having a different domain to the principal web page, causing access to the DOM of that frame to be denied. This is a security feature that stems from cross frame scripting. It basically stops unwanted access and changes occurring from outside of the domain of the frame; you can read more about it here http://msdn2.microsoft.com/en-us/library/ms533028.aspx. In a development environment developers can be made aware of this and can include the document.domain property to reference the principal domain inside of the frame. This allows access to the frames DOM.

Even though there are limitations to the DOM, there are many other objects such as xml based objects, and methods that can be used to assist us when DOM limitations are reached. A series on test automation frameworks is planned in the near future which will expand on the concepts learned in this series. You will learn how to develop a test automation framework using just VBScript and the API of an application such as Excel.

Conclusion
In this series we have looked at the various concepts involved in the automation of Internet Explorer. We have also described some very basic methods used in web application automation. We now how have a solid base to develop a test harness that takes advantage of these concepts.

For more information on TEST AUTOMATION FRAMEWORKS using VB and Excel. Please contact me through insideSQA.