Is Test Automation a Silver Bullet?

Although less common, there are still many instances where test automation is seen as the silver bullet to testing.  It’s easy to get caught in the trap, and everyone does, especially with the pace that technology is changing.  If we simply look at the term “Silver Bullet”, it is defined by Merriam-Webster as, “something that acts as a magical weapon; especially: one that instantly solves a long-standing problem.”  I think that we can unanimously say that test automation is not magical, nor is testing itself a long-standing problem.  At face value, the test automation silver bullet is a myth. That said, maybe there is a little saving it overall.  After all, if you are reading this, you likely believe that test automation has value.

The definition of silver bullet has two key components: instantly solves and long-standing problem. For the sake of argument and our definition, let’s remove instantly solves and change it to greatly reduces effort. For the second component, if we can admit that testing is not the problem, then the latter portion of the definition is really about the problems that testers face each day.

What automation does, and doesn’t do, well

While Artificial Intelligence (AI) and Machine Learning (ML) are out there and are improving each and every day, I’d like to focus on the test automation that is currently utilized more commonly and that most of the readers are familiar with or using. Test automation is really built to be a system of automated “checks,” as a part of testing. However, testing is about both validating successful operation and determining where the software doesn’t work – finding the bugs. You can see that those two items are quite different. Automated tests are executed by testers with the goal of checking to see if the software does what is expected under a set of known conditions. This can be done because computers are great at checks and processing for known inputs and known outcomes. The good news is that humans generally – and testers specifically – like the challenge of finding out how something works and, also, how they may be able to break it. Recognizing these different skillsets leads us down the path of using test automation to perform some level of regression testing while allowing the testers to work on building tests to ensure that new functionality is operating as it should. The next trick is to determine how much of the regression suite should a project automate. There are all kinds of methods to find and determine the right amount for a particular application/organization, but know that the method is almost always some level of test automation and almost never is the answer: “let’s not automate.” To help find the best method for you talk to the testers that are testing the application manually – they’ll know where to start. We explore this further on our podcast.

If a silver bullet even exists in test automation, it is important to properly select the problem area that we are trying to solve, as we know that testing overall is not the problem; it is, in fact, an important part of the solution. When selecting the problem area keep in mind that solving the problem in software development and testing is unlikely to make the problem disappear. Rather we are reducing risk of missed testing, expanding test coverage through more tests being executed, and improving timelines in testing, or a combination of all three. There are many areas that we can explore to use automation to assist the testing team including: test data creation, large data set validation, smoke tests, regression tests, etc. The key here is to identify where the testing team is spending their time performing tasks that a computer could do and likely do faster.

Where is test automation critical?

A misnomer that follows test automation is that it can save immense amounts of money. I think that is likely a true statement if you were performing all the tests today, but the more likely scenario is that much of the testing that should be done has been set aside due to time or lack of resources and is simply not being completed. It’s possible that your team performed some risk assessment, but more than likely, it was someone on the team making a judgement call as to what could be put in the backlog to test later.

Let’s take a quick look at a brief example that I am intentionally simplifying but holds true and will show how as a team we need to remember we are taking on risk with each release of software. For this example, we will assume that work grows as features are added. There are scenarios where this may not always be completely true, but for most scenarios this won’t be the case. For each release a developer will build one or more new features. The developer does not go back and update all features in the application or even necessarily check on them. The tester; however, is responsible for testing this new feature, in addition to performing regression testing on the entirety of the application. It is unlikely that the tester is provided with more time, or that additional testers are hired on each release and as a result the tester, while consulting with others, must decide what tests won’t be executed on any particular release. No testing team will ever have infinite time or money to release a piece of software, and this is where test automation is critical to the team’s ability to reduce the number of tests that are not being executed.

There is no other way than test automation to keep the testing costs consistent as the application(s) continues to grow and new features are added. However, test automation does need to be built and maintained, not an inconsequential task and investment, but it can take all those backlog tests and perform them to allow us to release with a more known state of quality. The path for consistency in testing costs is to implement test automation throughout the entire project so that it is there both during the build and into the maintenance phase. Remember though that implementing test automation throughout will have an additional cost associated with it and will require a commitment from the entire team and all stakeholders to maintain.

Technology helps reduce risk and improve effectiveness of your testing team

Technology has come so far in the past couple of decades, and I do believe that we will continue to see advancements in software testing solutions that will continue to make software testers lives easier and ensure that we are testing what needs to be tested. Artificial Intelligence and Machine Learning will bring solutions to testers that will allow them to create automated tests more quickly and this technology can be used to help human testers understand what changes have been made in development and what should be higher priority when executing tests. Again, this technology is helping you to reduce risk and make for a more effective testing team.

So, is test automation the Silver Bullet? It is clear there is no magic and that test automation can’t cut out the testing phase. I also think it’s quite clear that test automation is a precious component to successfully releasing software in a timely manner without continuously adding risk to the delivery of that software. The bottom line is that there is no quick fix to completing testing activities. Test automation can greatly improve consistency in testing, increase coverage, and improve speed. While it is not free, the question to ask yourself is how much are those tests that aren’t being executed really putting you and your business at risk.

PLATO can help develop test automation to improve your testing strategy. Learn more here.