Different people from the software testing field can have completely opposing opinions about the process of test automation. Sometimes it happens that after a while, the initial expectations and hopes of this substantially costly investment are not justified at all. In simple words, test automation does not bring the expected profits. In this article, we will try to understand why such situations happen and how to avoid the most common mistakes.
Author: Nataliia Syvynska, TestMatick, https://testmatick.com/
To Automate or not to Automate
A web company definitely needs to implement test automation if:
- There are projects on software that needs to be maintained and enhanced in the long term. A lot of tests need to be run in the regression testing activities, their number is constantly increasing, and the routine should be sorted out first. QA specialists should test but not be passionate about passing the next test case;
- A company has distributed development teams. The programmer must be completely sure that his/her changes and edits will not break someone else code. If you don’t use test automation tools, he/she will know only after a week at best, or worse, from the client;
- Support of different versions of the released software is included in the company’s services, and there are project teams that are busy creating and releasing patches and service pack for various web products. Everything is very clear here: testing on various configurations is a simple routine that must be executed immediately;
- A company creates a service, whose main task is to process and edit various data. To perform manual data injection into the system and visually analyze its changes – these are out-of-date things now, people from the IT sector should not deal with this in the 21st century;
- The company follows an Agile approach and its aim is to achieve frequent releases. There is no time for complete manual regression testing at all. It is really important to understand that everything functions as it should, even in “very hard-to-reach places”.
If your projects are not like these, then you probably don’t even need to think about the process of test automation.
What is the Point of Automated Tests?
Automated processes are necessary to save a person from the routine. Test automation – is part of the same story. There is however a common opinion that automated tests should completely replace the work of a human being, and all tests can be shifted to the programmed scripts. This is a complete nonsense! There are still no testing scenarios in the world that could be ahead of human labor and human capabilities in terms of their functionality. There are no scripts that can test software from the very beginning. There are however groups of programmed scenarios that are aimed at performing certain cyclical operations in order to inform a person that something works wrong, in other words, to perform trivial tests.
Automated tests are used to obtain useful analytical data about any change in the quality of the tested software faster than a person could do. In addition, there are other requirements that make the automation process efficient: the extensiveness of the test coverage, the clarity, and truthfulness of the results, the costs of creating and maintaining the software, the easiness of use and editing of data, and so on. The basic performance indicators are speed, quality of coverage and price. So you should start from there.
Why Hopes Don’t Come True?
There are many reasons why test automation may not live up all the expectations placed on it. They are all based on making the wrong decisions in the engineering or managerial fields, and sometimes – in both of them at the same time.
The most harmful errors are:
- The desire to save and not hiring professionals in the field of software testing automation. If a project manager thinks that he/she can teach software testers who already work in the company how to automate by taking simple courses on Selenium, then he/she is absolutely wrong.
- Attempt to implement automation without a designed strategy and planning that were thought ahead, such as “you need to automate everything and quickly, and only then we will see.” Or even worse – automation for the fact of automation: company X automates, and why should we be worse?
- Very late “start”: writing automated tests when the testers who are involved in the project are already fully overwhelmed with the work.
Engineering solutions can be considered as such decisions that programmers make during software developing and at the initial stages of introducing automation: choosing the right automation tools, types of testing, current frameworks, etc.
Why is it Wrong to Automate Only UI Tests?
This is a very common mistake: starting to automate previously created tests exclusively through a graphical interface. Such decision does not seem bad at the time of its discussion and validation. But, sometimes it delays the solving of some issues for a very long time. On the other hand, it can be very effective if the software is in production and you are sure that the product will no longer be developed. But, often, this is a false strategy with long-term planning for actively developed projects.
UI tests are a set of tests that QA manipulate while observing software execution. Moreover, it is a kind of simulation of how customers will interact with it (web product) in the future. An automation process should be introduced under such a model. But there are some nuances, namely:
- UI tests are completely unstable;
- UI tests are very slow.
They are unstable because they completely depend on the graphical interface of the software. If the order of the buttons and forms on the screen are changed, the tests may simply fail. The automation parameter won’t be able to find the necessary element or will click on the wrong button, and the test logic will be completely different.
The more tests of this type you have on the project, the more time you need to spend on their editing and technical support. As a result, you don’t believe such tests results due to constant wrong results. At a certain point, a QA specialist will spend all the time on fixing tests that are down, and nothing new will be created.
They are slow because the GUI is very slow, it requires constant redrawing, loading of all resources, waiting for the demonstration of certain data. The verification script spends a lot of time waiting. And waiting is a financially costly thing. Moreover, the test may be down, as it tries to interact with an element that did not have time to show itself on a very slow UI.
When UI tests running takes from 1 to 2 days, even if an independent group of tests was launched simultaneously on multiple servers, it is very difficult to use such automation on an ongoing basis as a single indicator of absolute quality!
The Way Out
Stabilization
The first thing to think about is to talk to programmers so that they remember to write the original attributes for all the elements by which automation tools can easily identify them. It means you need to switch from using intricate xPath expressions or CSS selectors to the original name or id. Such a policy should be strictly spelled out in development guides or be a mandatory item in the definition of done for programmers.
Moreover, it will be useful to “train” the automation tool to wait for the moment when the element under test becomes available for system interaction with it.
Development Speed
If everything seems clear about the processes’ stabilization, then the problem of slow tests should be solved quickly, because it affects the whole process of software creation.
The first and most trivial thing to speed up the testing process is to deploy software and run tests on more “modern” hardware and avoid situations when network delay can affect the interaction between the system and the test. In other words, to solve the problem through better hardware and a stable test platform architecture.
The next thing you need to do is to lay the parameters of an independent parallel start in the test framework and the test cases design. The process of tests parallelization can significantly reduce the time of tests execution. But, there are some limitations. First, sometimes the declared software logic may not allow you checking several threads that started simultaneously. Such situations are specific and rare, but still, they can occur. Secondly, everything here “is stopped” by the hardware part: you can’t parallel timelessly.
And finally, try to create as few UI tests as possible. The fewer number of tests you have, the quicker you will get the result of their run.
Pyramid of Tests
The pyramid of tests is a very good metaphor that clearly shows the desired number of tests at each level of the system architecture. Probably there should be a lot of low-level tests and very few high-level UI tests.
A Complex Approach
You have to understand that unit tests check the program code, that means, they allow the programmer stating with confidence that the created part of code functions perfectly, as it was originally intended.
UI tests check the integrity of the system, exactly that part of the software which will interact with the client. You must have such tests available.
Our recommendation: you must always have all types of automated tests in the required quantity for each level of the pyramid. Then it will be possible to get the best efficient factor of such tests. At the level of API tests, all functional checks that QA specialists performed during the planned sprint should be established: positive, negative, combined ones. In such a way, a qualitative regression test package is created. Such installations relate to both the web development process and the development of mobile applications and systems.
At the level of UI tests, you need to leave only acceptance tests, as they are called – Happy Path or End-to-End scripts, which are shown during the demo version run.
If you just follow the recommendations of this pyramid, you can easily get effective tests and excellent coverage for a reasonable price of development and technical support.
Conclusion
Not all projects need full software testing automation: for some of them, it will be enough to write a banal set of several scripts that will make the software testers’ life much easier. But when it comes to a project that is developed, will be supported for a very long time, and many people are involved in the work on it, then you can’t do it without test automation. Good automation can be created if, at the beginning of work on the software, you make the right decisions regarding the creation of automated tests at each level of the system architecture. Monitored with the right test automation metrics, such a solution can be an effective key to success.
Thanks for sharing this…I agree with all the points and from now I also start focusing on that