Test Doubles

Sometimes your system under test is made up of multiple systems. Some of those systems may not be in your control, and others may be very difficult to operate in a non production environment. When this happens you can use a test double to mimic those particular parts of the system.

A test double is a generic term used to describe the various ways of mimicking a system and can be classified as follows

• Dummy objects - objects that passed around but never actually used. Usually they are just used to fill parameter lists.
• Fake objects - objects that actually have working implementations, but usually take some shortcut which makes them not suitable for production (an InMemoryTestDatabase is a good example).
• Stubs - provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
• Spies - These are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
• Mocks - These are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don't expect and are checked during verification to ensure they got all the calls they were expecting.

Test doubles can be used across the entire test life cycle, facilitating the delivery of code by solving tricky integration or environment problems. However, at some point it may be necessary to remove those doubles and carry out more realistic system integration test. You should always assess the risk of using test doubles and have at the back of your mind that these are not real systems or objects, and are just helping development move along.

For more information read:

Martin Fowler - Test Double   

xUnit Patterns - Test Double

Browser Developer Tools

There is more to a browser than meets the eye! Not much more, but there are some great browser development tools that you should definitely pay attention to if you want to seriously test a UI manually through a browser.

I've added this to the list of things a tester should know or do as I still see many testers taking what is basically a point and click approach to manual browser testing. This is fine for simple user scenario based testing, but you could be missing valuable information just under the surface.

Take this simple scenario, on a login page, when a user inputs a correct user name, but an incorrect password, as it is bad practice to explain to that user the exact reason why they have not been able to log in to the system, the page displays a message stating "either the user name or password is incorrect". This is perfect for the user, but for a hacker, for example, trying to gain entry to the system, it’s not really giving any valuable detail about what their next attempt to enter the system should be based on.

At this point a hacker may look at the communication being sent between the user interface and any back end system. In this scenario, the user interface receives a message that contains an exception indicating that the log in failed, but it would not give the reason why it failed. However, not every developer follows good practice, and there maybe an instance where this message does contain enough detail to give a hacker more ammunition for their next attempt at breaking into the system.

I have seen something very similar to the following on a popular content management system, it’s a JSON object returned to the UI from a service after a failed log in attempt:

{
        "exception": "LOGIN_FAIL",
  "detail": "PASSWORD_ERROR",
}

Given that this scenario is a real possibility, and applicable to many other areas of a system, a tester needs to be able to easily assess these types of vulnerability.

Most browsers have a set of development tools built in that allow you to view the requests and responses that are processed by a browser. In any instance where you are informing a user of an action that has occurred through the user interface, and there is some degree of sensitivity or security related to that message or feature, then it always pays to have a look at what is going on in the background.

Don’t just stop at looking at the requests and responses, there are a whole host of over areas that you can look at such as the resources that are loading, the way css classes change, JScript errors, page performance, and much more.

Both chrome and firefox offer a decent tool set, either a feature or additional plugin

https://developers.google.com/chrome-developer-tools/

http://getfirebug.com/whatisfirebug

The Automation Pyramid

Think about using the test automation pyramid when planning your test automation strategy.

The test automation pyramid was used by Mike Cohn to describe the value of different types of automated tests in the context of an ntier application. The concept is very simple. Invest more time and effort in those tests that are lower down the pyramid than those at the peak, as those tests lower down the pyramid provide the most value in terms of quick feedback and reliability, whereas those at the peak are expensive to implement, brittle, and time consuming.

The traditional pyramid is split into three layers, Unit testing at the base, integration/API tests in the middle layer, and UI tests forming the peak of the pyramid. Many now opt to describe the UI layer as the ‘end to end’ layer as this phrase better represents those types of test.

Useful posts on the subject:

http://martinfowler.com/bliki/TestPyramid.html by Martin Fowler

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
by Mike Cohn

Automated smoke tests in production

If you can, don’t be afraid to run your automated tests in production. A production environment is a place where automated tests can give real value, especially after a release. Instant feedback on the success of a change in production could be worth a lot of money to your organisation.

As a minimum run automated smoke tests before and after a release in production, firstly, to baseline, and secondly, to assure nothing has broken after a release.

If you are limited by the data you can use or create during a test then just consider non transactional tests. Any way that you can speed up the feedback loop when a change has occurred is a bonus.

Obviously not all systems or organisations are conducive to this sort of strategy, so as a consideration when building a new system, it’s worth thinking about the ability to run automated tests in a live environment when designing a system.

This is how Microsoft does it.

Testing Webservices with SpecFlow

I have been looking for a way to test multiple soap web services as part of a complete integrated end to end workflow that at the same time can provide valuable business documentation. The requirements are quite simple:

• Workflows can be written using natural language
• Multiple web services can be easily executed in sequence
• Development time must be minimal

My immediate thought was to use a cucumber type test framework, and after a recommendation I started to investigate SpecFlow.

SpecFlow is a way of binding business requirements to code through specification by example in .NET. It supports both behaviour driven development (BDD ) and test driven development (TDD). SpecFlow, like any other natural language test framework, can also be used as a tool to combine documentation and testing of existing code, and that is exactly what I have used it for.

Using this method for generating an arbitrary web service, in a feature scenario using Gherkin I can specify the specifics of a web service, the contract location, the methods to be used, and what the response should be.  

In the binding statements, which SpecFlow uses to manage the logic required to execute the scenarios, I can execute the implementation of the call to the web service. There is a great example of this framework being used here, with multiple web services being called inside one feature.

This is probably not the most beautiful solution I have used to test services in a SOA environment but it provides the ability to get accessible system knowledge into the test and it’s extremely quick to set up. 

Developers in test.Yes, really!

I have mainly worked in high growth businesses, either in the form of start ups, or strategic projects in large corporations. My role typically involves promoting the use of sensible software engineering practices and software delivery patterns to help produce a product that works, and that can have frequent low risk change applied to it. In this type of environment, the team structure is very much organismic in nature. What this usually means is that there are very few people dedicated to, or specialising in, test activities. 

However, this does not mean that testing gets completely side stepped. We can still achieve the quality objectives of the organisation without dedicated specialists. Given the right means, I have found developers in this type of environment can become some of the best testers that you will come across.

How does that work?

There are a number of ways that we can use to bring effective testing to forefront of product engineering

Software Engineering Practices

I always ensure that developers are equipped with decision making power on how their work environments are structured, about the tools that they use, and about the delivery mechanism to push regular updates to a product. I ensure that teams use sensible practices such as CI, zero branching, infrastructure as code, contract testing, and the like. I push continuous delivery, and actively promote learnings made from many of the great people that I have worked with.

People

You need to hire engineers on the team that take a holistic and caring approach to software development. These are the people that have built successful products from the ground up, or have been pivotal players of very successful product teams. 

Test Activities

I find that coaching teams using principles from folk like James Bach and Michael Bolton to be incredible useful in up skilling developers quickly in the art of testing. These two guys have spent their careers honing a testing approach, and are so well drilled that you will always come away from any of their writings or teachings with more than a handful of powerful testing ideas. I personally think they are great guys that should be listened to a lot more. Their pragmatic, and often dogmatic approach, is contributing to the changing face of testing.

At some point organismic structures become mechanistic. This is when professional testers are hired. This is when test managers are hired, or may be a head of QA. At this point it is always really good to have facts and figures to assess just how successful the new order is compared to your pre-exising "testerless" state.  

Digging into Compiled Code

I recently had to test a number of changes to a .net web service which had no test automation, no regression tests, and no specification apart from the service contract and a subversion change log. In addition to this, there was also no indication as to when the last release of the service was so I had no idea from the change log which changes were live and which required testing.

Fortunately I had access to the live binaries which meant that I was able to decompile them using Red Gates Reflector, and drill into individual methods. This gave me the ability to cross reference whether the changes listed in the change log were actually live or not.

It took about an hour to analyse the decompiled code, but this reduced the potential test time from approximately four days down to less than one. It also gave a reassurance that no untested code would be released.

A decompiler is a great tool that gives you further insight into the code you are testing. Red Gate’s .Net Reflector, is one the most common for .net which I use a lot. For Java there are many plugins available for most common IDEs, I’m currently playing with the “Java Decompiler Project”.