Virtually every project I have worked on has at some time in its life had a close encounter with a well meaning, all things to all men, test automation framework, that has caused more harm than good, and cost a fortune to build in the process.
Maybe its due to the ubiquitous SDET (Software Development Engineer in Test) or a well meaning tester with no programming background, or a very demanding test manager, or just a plain old developer who should know better, but the continuing prevalence of massive test automation frameworks is slightly worrying. I both frequently hear about them, and very often see them. If they appear on my projects, they usually die a swift death.
So what's the problem?
The premise is that there is no way to execute or optimise automated testing apart from executing vast swathes of system tests from, usually, the highest interface into a system using a “special” test framework. If this is your only way to provide any kind of code coverage, or guarantee of safe delivery, then it points to a poor delivery process or a badly engineered system that is no doubt flaky and incredibly difficult to change and maintain.
These frameworks are typically maintained by a tester, test team, or a specialist bunch of mercenary developers.
You will normally find the tests executed by these frameworks to be long running, high maintenance, and vague in what they actually do. In many cases they will be driven through a UI, which typically means something like WebDriver, wrapped in an obviously completely necessary layer of abstraction, and maybe some fruity BDD framework to describe what the tests are doing.
Being system tests will typically mean that the tests are run in an integrated full stack environment. This in itself, regardless of test design, is a complexity that most would wish to avoid. Data management, permissions, versioning, infrastructure availability, etc. all come in to play here.
They very often sit in a separate test project that is completely decoupled from the code they are testing, meaning that version synchronisation issues at the feature level become a real problem. They are also hardly ever created by the people that need the feedback most from these tests - the developers.
The cost of both building and maintaining test frameworks for any enterprise sized solution can become astronomical in comparison to the actual value and risk reduction they deliver.
What I typically see is a very high test execution failure rate that can’t be attributed to actual code changes. I have seen some fairly large enterprise projects where the test run failure rate was between 69% and 91% with those failures not being tied to a code or configuration change. That is quite shocking. Equally, I have seen failure rates lower than 10%, but that does seem rare. If you couple the failure rate with the typical cost of building a test framework for any reasonably complex system, then the value becomes quite clear. Just multiply the day rate of all those involved in building a framework by the number of days it takes to build, and do the same for the ongoing maintenance cost, deduct all that from your expected profit and work out if that cost is justified.
What I typically see is a very high test execution failure rate that can’t be attributed to actual code changes. I have seen some fairly large enterprise projects where the test run failure rate was between 69% and 91% with those failures not being tied to a code or configuration change. That is quite shocking. Equally, I have seen failure rates lower than 10%, but that does seem rare. If you couple the failure rate with the typical cost of building a test framework for any reasonably complex system, then the value becomes quite clear. Just multiply the day rate of all those involved in building a framework by the number of days it takes to build, and do the same for the ongoing maintenance cost, deduct all that from your expected profit and work out if that cost is justified.
The other cost related issue lies with the constant rebuilding of test frameworks. As test frameworks are not typically used in production, the change management becomes less rigid. I'm not sure what the average lifespan of a test framework is, but based on experience I would hazard a guess at less than 2 years.
Essentially, these test frameworks are one of the symptoms of a badly planned test/design approach applied to your system.
and the Cause?
So do you really need to build this all singing, all dancing test automation framework? There are many causes and reasons why test frameworks appear.
Test Managers, testers, release managers, project managers etc, may have no understanding as to how developer driven tests maybe helping to reduce the risk of failure when adding and changing features. This encourages regression test phases, release test phases, pre prod test phases, all usually heavily laden with time draining manual and automated test framework antics. (Steve Smith goes into the detail of release testing, and how dubious that activity is, and that's without even talking about test frameworks!)
Engineers who have limited experience of commercial development, or engineers that should know better choose to knock out something that works over something that works and is maintainable. Very often you see the "responsibility of quality" foist upon testers or QA engineers within a team or organisation. These individuals very often don't have a programming background and will create, with the best intentions, a safety net in the form of a test framework that will typically be riddled with design anti-patterns, and difficult to maintain code. Again, this encourages the inclusion of multiple test phases due to the lack of confidence provided by the general lack of valuable and consistent testing feedback.
Project and test managers who believe that the tester or QA headcount can be reduced through the use of automation. This leads to attempts to automate the types of tests that a tester may execute. In reality, the things that get automated are just simple checks and not the complex interactive tests that a human being can execute. This in turn either leads to increased project costs, due to maintaining both manual and automation testers, or leads to poorer quality code being delivered due to the reduced amount of interactive testing that takes place. Either way, there is a hit on the money you have to spend.
Legacy Systems
We've probably all worked on a legacy system with no tests, and no documentation. If we need to make changes to this system, or refactor it into something more manageable, then we do need some kind of safety net. This can rear its head in the form of larges amounts of system tests being run by a test automation framework.
Whats the solution?
A decent test approach during the entire life cycle of your system, from concept to the end of its life.
The decent approach would typically include a combination of collaborative (isolated tests that use test doubles) and contract tests, coupled with those few tests that give you the warm fuzzy feeling that your system is hanging together nicely. See this great video from J.B. Rainsberger ("Integrated Tests Are A Scam") about how to design your tests to optimise not only how the system is tested, but also how it is designed as well.
If you join this all up with the testing executed by an expert interactive tester, then you will never have the need for a test framework ever again! Honest!
Of course, that's a very idealistic, and simplistic way to put it, but that is the bare bones of it. I think that some of the bigger problems really do come from those legacy systems with no tests, the ones that you need to refactor, but have no safety net. The instant reaction is to add a mountain of system tests driven by a framework to give you the sense that the risk of breakage is covered. In my mind, its probably more cost effective, and safer to have some good testers, and a small amount of warm and fuzzy smoke tests to help you through the hump of making a change, where each change sees some collaborative and possibly contract tests included in the work required for each change. Overtime, especially during a refactor, the test approach will modernise, and provide more built in safety.
Finally..
I have spent a number of years during my career either building or contributing to test automation frameworks. My experience is generally negative (hence this post). However, there is a test framework that I built in 2006 still running through a couple of hundred system tests every single day on an insurance policy management system. The cause of its existence here was, and still is, confidence, not in the software itself, but the infrastructure on which it is hosted. Its still there, still going strong. I've not done the sums yet to work out the value, but I'm sure either the company using it, or the consulting company maintaining it has a lot to thank me for. Value is in the eye of the beholder after all!
