Everyone in software development and Quality Assurance seems to be talking about automation these days. With all that chatter, there should be some discussion about what Regression Automation is and how it fits into a software development cycle.
There are a few different general types of test automation. Some automation is intended to be run by developers who are writing the actual software. Called Unit Testing, it is used to test if a particular piece of code functions, and they are generally very small tests that check the underpinnings of the code. That’s not the topic I’d like to cover here, and I'll leave that to one of our developers to cover in a future blog post.
Towards the end of a development cycle, when a software product is getting ready to be released (or when pieces of the code are functional in Agile development), QA resources are often called on to run regression testing on the product. Regression testing tests much, or when possible, all of the functionality that the software is designed to perform according to the requirements documentation. Regression testing is the final stamp of approval.
In an initial product release, regression testing makes sure everything works as expected. In subsequent product releases, it makes sure new functionality works as expected, and previous functionality still works as intended and wasn't inadvertently altered by new functionality.
Depending on how complex the software is, regression testing can be a daunting task. Repeating the same tests time and time again, without falling asleep, and noticing that the outcome was different 1 time, after having the same outcome 100's of times is a real challenge for any human being.
After all, what is the difference in a database query that returns "T" vs "F" to a human? A few pixels. What is that to a computer program? It is always definitively different. Considering how deeply embedded technology is in our world, in a hospital situation, those few pixels could mean the difference between life and death. On a stock trading application, it could mean millions of dollars. Those examples are at the extreme, but if your software is driving a system to make serious decisions, it's not too much of a stretch.
The Unblinking Eye
This is why automation is key. When your automated regression testing is set up to test that value, you can run and pass the test 10,000 times, and still catch a failure on execution # 10,001. A human's eyes (and brain) would be glazed over by then. A computer will even do it another 10,000 times without blinking (except maybe for its cursor).
Another aspect of automation is the availability of resources. There is no such thing as "Overtime pay" or "3rd shift" for a computer. Automated testing can be scheduled run at any time, not just when people are available and willing to execute tests.
So.. let’s automate EVERYTHING, right? It’s a nice thought. While possible in some situations, it’s not usually cost effective to automate all aspects of regression testing. There will always be some tests that are easier for a human to recognize as valid/invalid or passed / failed.
Importantly, automated regression testing won’t find problems that it is not specifically designed to look for. Computers are only as smart as we make them through programming, and often the time it would take to make them smart enough to handle all regression, or adapt to all scenarios is rarely worth it. The longer I work with automation, the more appreciation I have for the human brain and our ability to adapt to unexpected results. Unexpected results happen more often than I ever imagined before I started programming automation tools.
When a human QA resource runs a scripted test, we’re constantly looking for other areas that are not being tested directly. This ad-hoc testing often uncovers areas that may have been left out of regression, or new test scenarios / use cases that were not conceived of during development or requirements planning. Human QA resources will also be needed to maintain regression tests for new functionality. Unless your software product is in its final stage and won’t be getting any updates, the automation tests are going to always need adjustments along the way.
The Right Tool for The Right Job
“So I’ll save money by automating my regression testing, right?” Maybe. There’s a common misconception that automation will save money. It certainly can, but it's not always the case. Instead, regression automation should cover testing that you know can be tested reliably, repeatedly, and there’s a strong requirement for data accuracy.
In the end, a healthy balance of automating and human interaction can go a lot further than either one alone.