Achieving an expected level of quality with limited resource and budget

Sometimes there is just no money for testing or QA. Testers leave your team and don’t get replaced. The team dwindles, but the developer base either maintains or grows. Your reduced team has more and more to do. The worst case scenario here is that your remaining testers become overworked, can’t do their job properly, get thoroughly de-motivated, and leave, and who could blame them. You now have even less resource.

Despite the scenario above, when it does happen, you will still hear the mantra of “quality is not negotiable”, and probably even more so, requests by product and company leaders to support everything and everyone.

So what is possible? How can you achieve the expected system and product quality with a limited budget?

Looking back at some of the successful projects on which I have been involved, and which have also been struck by similar limited test resource scenarios, it is possible to identify some common characteristics that contributed to their success from both a product and quality perspective.

- a product that the customer really needs
- customer input from the start
- a team that cares about what they are building
- a product manager that knows how to ship products and that trusts in the development team
- a strong work ethic
- innovation throughout the team
- the right tools
- a simple iterative development process

Without going into the psychological aspects of building the right team and processes, most of the above I would weigh as being far more important to foster or implement in a product development team than fretting too much about test and QA resource. Why? Having all the above, for me, goes a long way to ensuring the quality of both the idea and build of your product. Good people, processes, and tools will do far more for you than hammering your application to death, and don’t usually come out of your budget. If you don't have much of the above then life will be difficult.

As a final comment, if you are faced with the scenario described above, you should ask yourself, and maybe the business, the following questions:

- Can we compromise the quality of the system?
- Is quality negotiable for this product?
- Will the customers accept a less than perfect solution?

If the answer is yes to any of these questions then you have the answer as to why you have no  budget, and with this knowledge you can then focus your test and quality efforts in a different and more effective manner.

Managing Defects

In an agile world we have to tackle defects in a different manner to cope with the pace and working practices. Here's one way of coping.

Develop a defect notebook, where defects are noted down and passed on immediately to the developer. Do this instead of logging every defect and you will save everybody time and greatly improve the quality of code that gets built.

Its a simple process.

When you detect a defect, inform the implicated developer immediately and discuss the defect. This is an important discussion and changes the way that we the code. In the developer/tester discussion make sure you cover the follow points.

• Is this a defect? Between you and the developer, you will be able to establish if it really is a defect or if it is expected behaviour. Get a product owner involved if necessary. Most importantly note it down in a note book.
• Can it be corrected now? The developer will be able to make a decision as to whether the defect can be fixed immediately or whether it will require more complex analysis. Ideally, the developer will fix it in the moment, not disrupting the work rate, and most importantly for the tester, reducing the need to raise a defect report.
• Does this defect require more analysis? If the case is yes, then before logging the defect, arrange a follow up the same day, or at the latest, the following day. This keeps the defect fresh and closer to the time that the original code was developed. This is important as the higher the turn around time for the developer writing the code and the defect being resolved, the slower the fix time will become. If a developer has to wait weeks before tackling a defect, it is likely that they will have forgotten the thought process that went into the development, thus increasing the time required to analyse and eventually fix.

If the defect requires more than a few hours to fix then log it. It maybe the case that the defect gets moved further down the backlog or requires input from others. It is in these cases where there is a benefit in tracking and monitoring a defect.The discussion with the developer can lead to an immediate solution or the necessity to invest more time in analysis. We only log the defect when we know that we have identified a serious issue.

Although not logging all defects, we still have our notes that we can use to prompt actions related to follow ups to analysis, or to ensure that the developer has done the promised fix. This note book can become vital, so look after it!

Your developers will more than likely adapt to this way of working as it helps them to have this quick quality reaction time to their code efforts.

Exploratory Testing

Even outside of the Agile arena I use exploratory testing. Its a great way to discover the workings of software and how to break it!

An agile tester is often told that they should only think about exploratory testing when testing outside of the basic user story acceptance tests. All other testing is covered by unit and integration testing. But how often is this the case? Agile development is never perfect, and to find a project that has a fully functioning, good quality CI implementation that executes all unit and integration tests is not as easy as some seem to think. So, as a tester we need some rules to allow us to implement exploratory testing only when it is really necessary. The following is a set of guide lines that I use to identify exploratory test opportunities.

• Always use in conjunction with planned tests on high impact stories. Cover as much as you can!
• Use when trying to reproduce system failure.
• Use when defect clusters have been identified. This will flush out even more defects.
• Always use when you have a good technical understanding of the system architecture. You will already be aware of what usually breaks certain systems.

And when executing exploratory testing:

• Demonstrate a plan of action. Even a quick outline of what you aim to achieve by carrying out certain actions will give confidence in your actions.
• Write down all tests that are performed. I use a test short hand that describes navigation/action/result in just one sentence. This enables you to create more tests further down the line.
• Let the system risk analysis guide you to critical areas of the application. This is where exploratory testing pays off.
• Sit near to, or with the development team to enable quick solutions to problems and questions.
• Never rely on just doing exploratory testing.

Remember - Completely unplanned random actions on an application is not exploratory testing, but rather, bad testing.

Quick Tips: Wiki - Internal Team Information Sharing

When I'm working to get new processes published, or writing informal tests that need to be accessed by other team members, I sometimes feel restricted by corporate intranet and share point services. These tools are great for publishing formal information, or storing common access templates, but they limit your ability to express ideas and communicate effectively within smaller teams.

Using a hosted wiki service such as Google Sites within test or development environments can give you the freedom to develop processes, applications, and test documentation without formally committing information to clients or end users. You can freely communicate with each other using these very professional tools.

If you are creating processes for a test environment, it is far more productive to get practices that are being designed out to your team as soon as possible. This allows your process implementation to be a continuous movement, taking away the pain of implementing a multitude of practices in one go.

Wiki's provides the ability to publish work and forward relevant links or information to the team as and when you feel. Often, with an intranet, the content has to be managed and controlled and adhere to standards. With your internal team wiki, you can remove yourself from these restrictions and have a open shared information portal.

At some point some of the information that we draft out within the wiki needs to be formalised and perhaps published to a Sharepoint serves or intranet. When you finally do publish you at least know that the information is refined and in common use.

A Wiki is also a great tool for constructing adhoc test documentation that can be shared freely between team users. If I have to develop tests in an agile environment, I set up a page within a wiki and create tables that allow the input of values. I can share these tests with all team members very easily without having to formally publish them. This methodology can be used to construct tests within other test process environments.