The Automation Pyramid

Think about using the test automation pyramid when planning your test automation strategy.

The test automation pyramid was used by Mike Cohn to describe the value of different types of automated tests in the context of an ntier application. The concept is very simple. Invest more time and effort in those tests that are lower down the pyramid than those at the peak, as those tests lower down the pyramid provide the most value in terms of quick feedback and reliability, whereas those at the peak are expensive to implement, brittle, and time consuming.

The traditional pyramid is split into three layers, Unit testing at the base, integration/API tests in the middle layer, and UI tests forming the peak of the pyramid. Many now opt to describe the UI layer as the ‘end to end’ layer as this phrase better represents those types of test.

Useful posts on the subject:

http://martinfowler.com/bliki/TestPyramid.html by Martin Fowler

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
by Mike Cohn

Testing Webservices with SpecFlow

I have been looking for a way to test multiple soap web services as part of a complete integrated end to end workflow that at the same time can provide valuable business documentation. The requirements are quite simple:

• Workflows can be written using natural language
• Multiple web services can be easily executed in sequence
• Development time must be minimal

My immediate thought was to use a cucumber type test framework, and after a recommendation I started to investigate SpecFlow.

SpecFlow is a way of binding business requirements to code through specification by example in .NET. It supports both behaviour driven development (BDD ) and test driven development (TDD). SpecFlow, like any other natural language test framework, can also be used as a tool to combine documentation and testing of existing code, and that is exactly what I have used it for.

Using this method for generating an arbitrary web service, in a feature scenario using Gherkin I can specify the specifics of a web service, the contract location, the methods to be used, and what the response should be.  

In the binding statements, which SpecFlow uses to manage the logic required to execute the scenarios, I can execute the implementation of the call to the web service. There is a great example of this framework being used here, with multiple web services being called inside one feature.

This is probably not the most beautiful solution I have used to test services in a SOA environment but it provides the ability to get accessible system knowledge into the test and it’s extremely quick to set up. 

Scrum in under 10 minutes - video

A great introduction to Scrum in under 10 minutes!  

For more information on scrum, visit www.scrum.org.

Achieving an expected level of quality with limited resource and budget

Sometimes there is just no money for testing or QA. Testers leave your team and don’t get replaced. The team dwindles, but the developer base either maintains or grows. Your reduced team has more and more to do. The worst case scenario here is that your remaining testers become overworked, can’t do their job properly, get thoroughly de-motivated, and leave, and who could blame them. You now have even less resource.

Despite the scenario above, when it does happen, you will still hear the mantra of “quality is not negotiable”, and probably even more so, requests by product and company leaders to support everything and everyone.

So what is possible? How can you achieve the expected system and product quality with a limited budget?

Looking back at some of the successful projects on which I have been involved, and which have also been struck by similar limited test resource scenarios, it is possible to identify some common characteristics that contributed to their success from both a product and quality perspective.

- a product that the customer really needs
- customer input from the start
- a team that cares about what they are building
- a product manager that knows how to ship products and that trusts in the development team
- a strong work ethic
- innovation throughout the team
- the right tools
- a simple iterative development process

Without going into the psychological aspects of building the right team and processes, most of the above I would weigh as being far more important to foster or implement in a product development team than fretting too much about test and QA resource. Why? Having all the above, for me, goes a long way to ensuring the quality of both the idea and build of your product. Good people, processes, and tools will do far more for you than hammering your application to death, and don’t usually come out of your budget. If you don't have much of the above then life will be difficult.

As a final comment, if you are faced with the scenario described above, you should ask yourself, and maybe the business, the following questions:

- Can we compromise the quality of the system?
- Is quality negotiable for this product?
- Will the customers accept a less than perfect solution?

If the answer is yes to any of these questions then you have the answer as to why you have no  budget, and with this knowledge you can then focus your test and quality efforts in a different and more effective manner.

Kanban eBook

I blogged about Kanbanery a while back but failed to notice their really concise ebook on Kanban. Definitely worth a quick read if you are thinking of doing Kanban and need a quick introduction.

Kanban eBook

Kanbanery – Simple work management to keep you moving

Kanbanery is a great tool for managing tasks, your work load, and, of course, your Kanban process. Based on Kanban principles, it gives you a solid way to manage tasks for both solo and group work. With just three states you get a very clear view of what needs doing, what is being done, and what is done.

The tools simplicity focuses you on what is really important in your current work load. The price plan includes a free version that allows two people to collaborate on a project, whilst other very reasonably priced plans easily open this up to small and large teams.

Well worth a look. Check it out here – www.Kanbanery.com

Engineering Support Teams (DevOps)

In agile environments, product owners that aren't in touch with development or who are not familiar with engineering practices always seem to push out of the backlog those technical stories that are required for a successful project.

Sometimes we need technical stories that allocate time to set up a CI project, or create a mocking framework, or create an environment, or some automated deployment scripts in order to continue developing and releasing a project. When we don’t have these mechanisms in place, we need to hope that we have a team that can pick up the functionality as we fire it out and position it into the deployment pipeline so that it will drop out happily into production.

Without these technical stories and engineering practices in place, the cycle time (time between conception and release of a story) climbs and we either never release, or if we do release, it’s a pressurised situation of manic updates done manually on live servers by technicians that may have no idea about the code they are deploying.

If you have multiple teams or programmes, and if you have the budget, an engineering support team is essential. This is a team that engages primarily in DevOps, but will also be domain experts capable of taking a holistic view of the project and identify any unforeseen issues.

You will often see this team type in large organisations, but in those organisations which may need them the most, i.e. those start ups that need to be capable of continuous delivery to keep ahead of the competition, it is usually never seen due to the difficulty in justifying a couple of heads working on technical stories instead of functionality. It takes a really technically savvy manager to understand and sell their worth to the business.

If you don’t have this team, and you are not afforded the necessary time to do technical user stories in your sprints or iterations, then you need to think outside of the box and form a group that can focus on these software engineering practices.

A software engineering group is one of these and is usually a selection of the most ambitious or creative engineers from your teams that are willing to get together on a regular basis to throw around current engineering problems and work on providing solutions between them. By utilising slack days, or lunch times, the team can fire through some of these completely necessary technical elements of the delivery pipeline and keep the cycle time down to a minimum.

Agile Test Strategy (Updated)

A couple of years ago I posted a  very simple test strategy for agile based projects, here is an updated version.

Test and quality objectives


Objectives are a fundamental part of any strategy as they show us where we need to be going, and they allow us to invest in the right activities. My test strategies typically include the following objectives

• Provide the customer with a system that they really need with quality baked in
• Automate as much of the testing, configuration, and deployment as possible
• Engage the business and customers in testing
• Provide the business with the ability to confidently deliver features or complete systems without considering lengthy or multiple test iterations

Phase – Project conception & early estimation meetings


This is the most important part of the project. The wrong idea here could result in many man hours down the drain and a lot of money for your business. Who is really bothered about reducing the defect detection rate in a product or system that no one will ever use? Idea bugs are the most costly bugs and the ones that we should really focus on. This is sometimes out of the hands of most QA and test engineers, and is usually down to either having a very strong willed architect or a product manager that really understands what the customer needs and how to prove those ideas. I believe that this warrants a completely separate post, but you can find a very good explanation of idea bugs here. Whatever happens, there are tasks that we can take on.

• Encourage pretotyping to prove the product concept over writing stories for unproven ideas
• Understand any technology
• Begin to sketch out a test strategy for the project

Phase – Early stage sprinting


During the early sprints our focus should be on proving the architecture and system concepts, building the team, and putting in place development and test frameworks.
Planning

• Write technical stories or ensure the following tasks exist (These are all MUSTS)
• Set up a CI project on your CI server
• Set up unit test framework
• Set up framework for front end script automation (JSUnit)
• Set up build scripts
• Set up all the necessary environments for now and think about what you may need for the future (Identify any needs you have further down the line)
• Help scope out the stories and product
• Get acceptance criteria clear in each story
• Engage architects and technical experts in all planning discussions, never make assumptions on how things should work
• Liaise with test leads or test managers to ensure you have covered off any global test strategies or ideals
• Raise any performance considerations

Sprint

• If the architecture is not yet baked then get to understand proposed solutions, arguments for and against. If it is decided, get to know why it was chosen, capabilities, etc
• Ensure that plans are in place to make the code testable at more than just the unit level
• Work with the product owners to get more insight on what the product is supposed to be doing
• Update and test build and deploy mechanisms into the QA environments
• Look for opportunities for early automation
• Pair program / test / analyse

Phase – Normal sprinting


During normal sprinting we shape the product and deploy to UAT or beta environments on a regular basis to get feedback from customers, we may even be deploying direct to production. Regression testing is a key part of this process so must be efficient as possible. Throughout the strategy we are automating every process we can. Unit, integration, front end, and build scripts all combine to provide a high level of confidence in what we are delivering. 


Planning

• Clarify acceptance criteria
• Write testing tasks for all user stories
• Unit
• Integration
• Scripting
• Plan exploratory strategy
• Ensure environments are ready
• Plan crowd source testing
• Ensure that the correct people are involved in the different planning activities. The tester should become a conduit during the planning, ensuring that information is passing to the right people.
• Ensure UAT or beta environments are ready
• Set up performance environments

Sprint

• Write acceptance tests based on acceptance criteria
• Carry out automation tasks
• Maintain build scripts and environments
• Carry out exploratory testing
• Pair up with developers
• Performance test
• Test release mechanisms
• Write post release test plans

There are probably a lot more activities that we can include, and I would encourage folk to experiment. If you understand your objectives, you will definitely understand what activities you will need to implement.

UI Automation: Avoiding Failure

I have developed and worked on many UI automation frameworks using both commercial and open source tools, some of these have been more successful than others. Regardless of the tool you use there are some common indicators that you can help you identify if you are going down the wrong path.

No business backed strategy to begin UI automation

UI automation is expensive; it requires a lot of time and effort. Make sure you have enough secured resource to carry out your plans. If you don’t, that resource will be pulled somewhere else and you will end up with several unfinished automation projects that have cost the business money but bring no value.

No clear reason to automate

Why are we automating?  If you have no clear objectives, what’s the point?

Some of the reasons why we should automate

• to reduce the feedback loop between code implementation and possible defect detection
• to reduce the amount of time we spend in resource intensive activities such as regression testing
• to remove human error from test execution

       …and we should automate when:

• we have stable functionality
• we have the skill necessary to automate

 No design pattern is being applied to the test design

There is no need to reinvent the wheel with UI automation. Using a recognised design pattern like the page object model will reduce the time it takes to automate. It will also give you cleaner and easier to understand tests. Other engineers will be able to pick up the project and understand it.

Tests are brittle

If small changes to the AUT cause the tests to break then your tests are too brittle and not easily maintainable. 

Things to look out for:

• Too many checkpoints per test. Always ask yourself what the value of a checkpoint is before implementing it. The more checkpoints we have the greater the chance of test failing. In automation I only test for critical information. Many people will put checkpoints in for page layout, sizing, or styles applied. This is useful but is something that can become hazardous when you start cross browser testing, or testing with different resolutions.
• Hard coded data. Tests should be driven by a driver class or datasheet if data is required. If you don’t have complete control over your test data then it could change and your tests will break. Being able to feed data in gives you much greater control over the tests and makes them much more useful.
• Over engineered. Is your test code more complex than the code it is testing? Make sure that your tests are not failing due to over complicated code. I've seen this happen so many times!
• Tests that depend on the test results of another test. This is a very common mistake. If one test with dependencies fails all dependent tests will also fail.

 No control over the data that drives the tests. When the data changes the tests fail.

Data control is one of the most common reasons why UI automation project fail. Any automation project should have its own environment where data is guaranteed. If the environment is shared then you must have a way of protecting the data your tests use. It’s very frustrating when your data is tampered with and the result is failed tests.

Tests fail because the identification of page or application objects is using ambiguous names, or auto generated identifiers.

UI automation requires us to identify page objects through their properties or location. Many applications will generate unique changing names for objects. Using these names is not secure. Try to get developers to add decent identifiers to objects. It will help keep the tests from becoming brittle and save time during test creation.

Large xpath identifiers are also a common cause of test failure. Try to avoid!

More time is spent on maintaining tests than testing

This is a sure sign you have problems. 

Things to look out for

• A large number of tests fail on each test run
• Engineers are “Baby sitting” test runs. Helping the tests complete is not automation!
• You seem to be spending all your resource on one application. You never get to automate anything else
• New test creation drops

Nobody knows what the tests coverage is

One thing I’ve found in agile environments is that although UI automation is high on the agenda, there is little understanding of what the test coverage is. Developers and testers churn out tests, get excited, but there is no understanding of what really has been tested. This means that you still have to invest in manual regression runs to assure yourself. UI automation is a safety harness, so we need to know how much protection it is giving us.

Quick Tips: Defect Tracking with Google Wave

On an agile project I'd been using Skype to track defect conversations I was having with developers. We had no easy way to track bugs, and the need to get things fixed ASAP removed the benefit of formally tracking bugs. The biggest problem with using Skype was the distributed and fractured information.

Solution: Google Wave! Create a wave with all your developers and testers and call it after the name of your sprint. List defects one by one, give them an id and description. Use colour coding to indicate the state of the defect. Developers and testers can comment on each individual part of the wave/defect. Completely simple! All the information is contained within one location and is accessible to all. Next level communication!

Testing in a New or Transitional Agile Environment

Agile environments need many practices set up and functioning before a tester can really flourish – continuous integration, environment management, good software engineering practices, a solid development process etc. Without even these basic elements in place, the tester is left to manage an ad hoc flow of user stories, support issues, and goodwill.

Issues that seem to be common for testers in these environments:

• Iteration planning is a quick guessing meeting. This is the most important part of any iteration as it sets up the focus and objectives for upcoming work. It is also an opportunity for the team to extract decent acceptance criteria from the product owners.
• Test estimations reduced by product owners or developers. Just remember who the experts are here!  Don’t put yourself in a situation where you have to cram a full regression test into 3 minutes because a PO thinks that is enough time!
• Acceptance criteria either not identified or too vague to be of any real value (See above!). Not having good acceptance criteria means that a story has no real objective, and will be too vague to test. Without the defined goal posts that acceptance criteria gives us, testers will often find themselves beaten up over failed expectations if the story doesn’t do what the PO wanted it to do. Comments like “This hasn’t been QA’d properly” or “the testers didn’t catch this” are quite common in this situation and push accountability on to the test team.
• Stories getting to the tester too late. This usually happens when stories are ill defined and extend past the original estimation. Again, acceptance criteria will usually help focus estimations.
• In smaller environments where developers are multi-access resources, there is often a stream of “under the radar” work that eventually flows into the hands of the tester. This is work being done, perhaps for the good of the business that puts an extra burden on the team. In this type of environment, work never gets tested properly. Consider using Kanban if this is the case!
• No supporting development processes such as continuous integration, or automated testing. This means that the tester is usually engaged in large amounts of regression testing rather than exploring new functionality. Consider adding automation tasks to stories.
• No decent environment management system in place meaning that it’s very difficult to have consistency with test, development, and production environments. This is a must if you wish to be efficient and effective with your deployment pipeline. You will need to set aside developer, DevOps, and tester time to get this up and running. To secure this time you need to be able to sell the benefits to your management team. Reduced delivery time is always a good benefit to use in this circumstance.
• Testers being treated as a quality gate at the end of the iteration rather than an integral part of the team. This is a cultural change that is required in the team. A strong, test "savvy" development manager or a solid QA/test director should be pushing this change. Ground-up changes are usually quite difficult. Embedded cultural changes such as this usually require strong and determined leadership.
• No sense of quality ownership by the team. This is common in those teams with no test automation at any level, and where acceptance criteria are either weak or missing. This links in with many of the points above. The more we can infiltrate into the minds of the developers, the better! All the suggested practices above and below will help define this ownership.

What can the tester do to change this?

The tester in an agile environment needs to become a proponent for process improvement. To avoid some the issues above, an agile tester must engage in some of the following

• Highlight software engineering opportunities for the team, and be proactive in providing possible solutions. A great way to do this is to start a software engineering work group that gets together proactive and innovate developers and testers on a regular basis to implement the engineering practices that can improve the efficiency and effectiveness of the teams.
• Work to rule. This is tough, but if you have been asked to succeed in agile, then you must follow the basic work flows that have been designed. If things are not working, use the retrospectives to make changes that can be agreed on by the team.
• Be alert! Use the tools that you have to keep track of what is being developed, supported, and released to help you get an understanding of the work output. I have had a lot of success monitoring RSS feeds and change logs from source control systems as they give me the ability to hone test analysis to a specific part of the system. You also get information on changes to parts of the system that your developers may not have mentioned to you!
• Publish your ideal test architecture as part of the test strategy. This will allow others to see what your perspective is on what is needed to develop successfully, and may prompt them to help you out, especially if your ideas are compelling!
• Measure your Boomerang! This is work that comes back to the development process after it is released. In ISEB circles this is known as the defect detection rate. It is one of the most useful measurements we have as it is a real indicator of how effective your quality practices are.
• Measure whatever seems important as it can help you push the importance of what you are doing. This is one the most important things we can do. I once did this to get unit testing up and running in one environment. A simple weekly report on the number of tests per project provided the impetus to get developers writing tests on a regular basis.

Testing can be a thankless task, but being proactive during a transitional period will bring benefit to your team. The team will probably be going through a lot of change acceptance, and placing an integrated tester directly into the team is another difficult change to manage.

Here are a couple of great resources on agile development and lean development that could help you formulate ideas in these tricky environments.

http://blog.crisp.se/author/henrikkniberg

http://www.leanessays.com/

When Should We Automate the User Interface in an Agile Environment?

Many misunderstand automation in the context of agile. Picking up a user interface test tool and automating from the word go is never going to work. Automation is often associated directly to UI tests but in an agile environment, automation refers to the whole test strategy. One of our primary goals in agile automation is to develop automated unit tests that will help us monitor the pulse of our project. From there we can look at developing integration tests, coupling together groups of unit tests, or we can look straight to UI automation. UI automation allows us to do both integration and system testing.

In an agile environment we have to look at the real return on investment when deciding what and how we automate in terms of the UI.

Examining the following factors may make this task easier:

1. Risk - A business or safety critical application is always a candidate. Use the other factors to assess the correct moment to automate. Low business impact applications should really be avoided unless work flows are simple.

2. Maturity/Stability - If the web application is still in primary development stages then there will be lots of changes to the UI. Sometimes it is better to wait for a Beta release before beginning UI automation. At this point there will usually be a lower frequency of change or less impacting changes. Waiting until this point saves a lot of time, and reduces the maintenance overhead.

3. Resource - Automation is labour intensive. If you can only devote a small amount of time to UI automation then you are probably not going to have success. Time block an automation project if necessary, it will bring benefit.

4. Change Index - Web applications with a high change index usually require the largest amount of maintenance. Keeping up with a site that has constantly changing layout or content can kill an automation project.

5. Complexity - Large complex systems that use a host of differing technologies, and contain work flows that cross these technologies, should be avoided. Unless, that is, you have the necessary resources and tools to combat this.

6. Technology - If you don't have well supported UI automation tools such as QuickTest Pro or Selenium, or the expertise to use the ones you have got, then your automation project could extend far beyond the original plan. The problem of not having expertise in your automation applications, or even in automation, is that you could find yourself with a very brittle framework that requires constant maintenance, increasing the cost and burden of automation.

Tools: Web Service Testing

A colleague put me onto a great free XML development tool called Liquid XML Studio. This is an incredible feature rich XML development tool that apart from developing XML, it allows you to test and build SOAP requests in such an easy manner that I couldn't imagine a developer or tester being without such a tool! Its another great tool to add to your arsenal of agile test tools.

Some of the useful features:

• Web Service Call Composer
  
• XPath Expression Builder
  
• HTML Documentation Generation
  
• XML Diff - Compare XML Files
  
• Microsoft Visual Studio Integration (2005 & 2008)
  
• XML Schema Editor
  
• XML Data Binding

A great alternatives is soapui - This is an open source tool designed for web service testing. It allows you to inspect and invoke web services. This is a tool that has become a strong component of my current web test framework.

Managing Defects

In an agile world we have to tackle defects in a different manner to cope with the pace and working practices. Here's one way of coping.

Develop a defect notebook, where defects are noted down and passed on immediately to the developer. Do this instead of logging every defect and you will save everybody time and greatly improve the quality of code that gets built.

Its a simple process.

When you detect a defect, inform the implicated developer immediately and discuss the defect. This is an important discussion and changes the way that we the code. In the developer/tester discussion make sure you cover the follow points.

• Is this a defect? Between you and the developer, you will be able to establish if it really is a defect or if it is expected behaviour. Get a product owner involved if necessary. Most importantly note it down in a note book.
• Can it be corrected now? The developer will be able to make a decision as to whether the defect can be fixed immediately or whether it will require more complex analysis. Ideally, the developer will fix it in the moment, not disrupting the work rate, and most importantly for the tester, reducing the need to raise a defect report.
• Does this defect require more analysis? If the case is yes, then before logging the defect, arrange a follow up the same day, or at the latest, the following day. This keeps the defect fresh and closer to the time that the original code was developed. This is important as the higher the turn around time for the developer writing the code and the defect being resolved, the slower the fix time will become. If a developer has to wait weeks before tackling a defect, it is likely that they will have forgotten the thought process that went into the development, thus increasing the time required to analyse and eventually fix.

If the defect requires more than a few hours to fix then log it. It maybe the case that the defect gets moved further down the backlog or requires input from others. It is in these cases where there is a benefit in tracking and monitoring a defect.The discussion with the developer can lead to an immediate solution or the necessity to invest more time in analysis. We only log the defect when we know that we have identified a serious issue.

Although not logging all defects, we still have our notes that we can use to prompt actions related to follow ups to analysis, or to ensure that the developer has done the promised fix. This note book can become vital, so look after it!

Your developers will more than likely adapt to this way of working as it helps them to have this quick quality reaction time to their code efforts.

Quick Tips: Learn From Others!

Its great to find other sources of great Agile QA info. Have a look a look at this great site full of more than just agile test tips.

http://www.quicktestingtips.com/tips/

Check out the authors for even more info:

• Jonathan Kohl
• Karen Johnson
• Mike Kelly
  

Exploratory Testing

Even outside of the Agile arena I use exploratory testing. Its a great way to discover the workings of software and how to break it!

An agile tester is often told that they should only think about exploratory testing when testing outside of the basic user story acceptance tests. All other testing is covered by unit and integration testing. But how often is this the case? Agile development is never perfect, and to find a project that has a fully functioning, good quality CI implementation that executes all unit and integration tests is not as easy as some seem to think. So, as a tester we need some rules to allow us to implement exploratory testing only when it is really necessary. The following is a set of guide lines that I use to identify exploratory test opportunities.

• Always use in conjunction with planned tests on high impact stories. Cover as much as you can!
• Use when trying to reproduce system failure.
• Use when defect clusters have been identified. This will flush out even more defects.
• Always use when you have a good technical understanding of the system architecture. You will already be aware of what usually breaks certain systems.

And when executing exploratory testing:

• Demonstrate a plan of action. Even a quick outline of what you aim to achieve by carrying out certain actions will give confidence in your actions.
• Write down all tests that are performed. I use a test short hand that describes navigation/action/result in just one sentence. This enables you to create more tests further down the line.
• Let the system risk analysis guide you to critical areas of the application. This is where exploratory testing pays off.
• Sit near to, or with the development team to enable quick solutions to problems and questions.
• Never rely on just doing exploratory testing.

Remember - Completely unplanned random actions on an application is not exploratory testing, but rather, bad testing.

Agile Test Strategy

[THIS POST HAS BEEN UPDATED: Please visit Agile Test strategy Updated!]

Over the years I have had to design really heavy weight test strategies to help sell/communicate to the business the reasons why we are testing and what we will do during each project. However, I have always found that no matter how good the intention set out in the test strategy is, what actually occurs during the test phase of a project is almost unidentifiable. Fortunately, on the last project I worked on I had the opportunity to develop a lean test strategy that was useful, practical, reusable, and above all, CMMi friendly!

Although not strictly a company that practised agile or lean development, we were trying to reduce the bureaucracy of traditional technical processes. The following is probably as light weight as a test strategy can get, but it works.

The idea was to make a statement of intention that loosely binds some of the more important test practices that can help a team move forward. The phases are fairly typical of agile development, although they do not represent a definite task execution flow.

Phase: Project Set Up

• Understand the project
• Collect information about the project
• Create a test knowledge repository

Phase: Planning & Analysis or Release Planning

• Assist in the definition and scope of stories
• Develop test plans based on planning session

Phase: Development Iterations

• Risk analysis during the sprint/iteration planning
• Construct acceptance tests for each story
• Develop business functionality validation plan
• Document and write tests for defects
• Automate with both unit and UI tests
• Assist in functional review/demo
• Accept User stories

Phase: Hardening Iterations

• Regression test
• Business acceptance testing
• Develop release readiness plan
• Run performance tests

Phase: Release

• Assist in release readiness
• Plan test release data and tests
• Accept the Release

Using this basis for every project we can deliver software that has undergone rigorous testing but has not been delayed through the burden of traditional test documentation.

Agile Metrics

As a tester you will always find yourself absorbed in metrics creation, reporting on anything and everything that your management team wishes to report on. However, in agile development environments, sometimes its very difficult to assess the importance of metrics. I've been looking at some ways to provide management teams with interesting and useful information to measure our success.

One of the most fundamental approaches we can take towards metrics collection in an agile environment is the Return On Investment that a collection of metrics can give us. That is, how can our metrics help us provide more business value.

Using measurement goals such as only having 5 serious defects found in production after release give us more incentive to produce less buggy code.

Measuring the time it takes from the conception of a user story to seeing that user story providing business value also gives us an idea of how we are performing as a team. This is known as cycle time.

These metrics give us valuable information that provide us with the correct means to motivate a team.

Using the following principles, we can develop metrics that give us a decent ROI from our metrics:

Achieving SLA's relating to service and availability of products and systems.

1. Low cycle time, that is, a high throughput of change.
2. Low amount of unplanned work.
3. Effective process improvement practices.
   
4. Effective change management.

We can build metrics around these principles that give us usable metrics that bring value and motivation to a team.

Quick Tips: Wiki - Internal Team Information Sharing

When I'm working to get new processes published, or writing informal tests that need to be accessed by other team members, I sometimes feel restricted by corporate intranet and share point services. These tools are great for publishing formal information, or storing common access templates, but they limit your ability to express ideas and communicate effectively within smaller teams.

Using a hosted wiki service such as Google Sites within test or development environments can give you the freedom to develop processes, applications, and test documentation without formally committing information to clients or end users. You can freely communicate with each other using these very professional tools.

If you are creating processes for a test environment, it is far more productive to get practices that are being designed out to your team as soon as possible. This allows your process implementation to be a continuous movement, taking away the pain of implementing a multitude of practices in one go.

Wiki's provides the ability to publish work and forward relevant links or information to the team as and when you feel. Often, with an intranet, the content has to be managed and controlled and adhere to standards. With your internal team wiki, you can remove yourself from these restrictions and have a open shared information portal.

At some point some of the information that we draft out within the wiki needs to be formalised and perhaps published to a Sharepoint serves or intranet. When you finally do publish you at least know that the information is refined and in common use.

A Wiki is also a great tool for constructing adhoc test documentation that can be shared freely between team users. If I have to develop tests in an agile environment, I set up a page within a wiki and create tables that allow the input of values. I can share these tests with all team members very easily without having to formally publish them. This methodology can be used to construct tests within other test process environments.

Agile Testing

I've just started work in a development team that has recently adopted scrum processes. Part of my initial tasks were centred around the implementation of a test process that works within scrum.

Here are a few links that will help you on your way to discover the delights of agile testing:

• A good intro to agile testing practcies from google's Elizabeth Hendrickson
• Agile Software Development - Agile Testing blogs
• Agile testing principles - Elizabeth Hendrickson
• Extended agile test resources

Even if you're not moving towards agile development practices, some of the techniques, tools, and practices referenced in the above links can greatly improve the productivity and reputation of the tester as valuable member of the development team. Take a look.