Top 5 Mistakes in Automated Testing

Working in automated testing for over seven years, test automation is becoming a cornerstone of a successful and efficient product lifecycle, especially in the era of the rapid growth of the IT industry and the high pace of software development.

Author: Olexiy Vovk, Senior Test Automation Engineer at Sigma Software Group, https://sigma.software/

Having started at Sigma Software as an intern and gone through all the stages of growth here, I will mainly rely on the experience gained from 20+ projects completed with my participation.

What is test automation?

Test automation speeds up the verification and defect detection process and stands at the forefront of product quality assurance, guaranteeing its stable operation and compliance with the stated requirements. It solves several vital tasks: speed and efficiency of testing, repeatability of checks, early detection of errors, and continuous integration.

Advanced companies allocate significant resources to implement automation, realizing that with it, it is possible to achieve a high speed of new releases and, at the same time, maintain product quality.

However, without its many benefits, automation can become a source of unexpected problems and costs without a clear understanding and correct approach.

Below, I go over the most common mistakes that future or current automation professionals might face when implementing automated testing, which I encountered myself. I also provide recommendations on how to avoid such errors.

MISTAKE #1. Misunderstanding the automation goals

In my experience, one of the most common reasons for test automation failures is a distorted or incomplete understanding of its goals. Many teams mistakenly view automation as a way to reduce testing time or replace human engagement, lowering the cost of test engineers’ salaries. They start the test automation process without clearly understanding what they want to achieve. This can lead to misdirected efforts and, as a result, inefficiency of the entire process.

After all, without clearly defined goals, you can get “automation for the sake of automation.”

For example, I once witnessed how automation was added to a startup development project at a client’s request only because his friend also had automation. In another project, they developed a framework for a very complex and flexible UI to cover a few smoke scenarios.

This leads to the following possible negative consequences:

• Waste of budget – You can waste tens or hundreds of hours on a product that will not bring any benefit and often even take additional time to maintain.
• Loss of trust in automated tests – When they check the wrong thing, they often “break” or miss defects. The development team and the customer may be skeptical of the automation results.
• Incorrect test coverage – The wrong choice of test scopes can lead to automation of non-critical scenarios that are not particularly useful and can even be removed from the release scope. In such tests, automation simply does not pay off.

How to avoid such problems?

• First of all, define the main goals of automation before you start. This can include reducing regression testing time, automating time-consuming manual testing scenarios, critical scenarios, etc.
• Discuss with the team and ensure that automation will help reduce manual testing costs for such tasks.
• Make sure your product is technically feasible to automate.

MISTAKE #2. Failure to conduct a proof of concept before starting automation

Many teams, carried away by the idea of automation, neglect a critical stage – Proof of Concept (PoC). This stage is a small experiment that helps determine the viability of the chosen automation strategy and tools for a particular project.

Neglecting the PoC stage can lead to the team choosing inappropriate test automation tools or approaches, which will cause many problems and significantly increase the project’s cost. The PoC allows the team to assess risks early, identify potential issues, and determine whether the selected tools and approaches meet the requirements and whether it is possible to automate the product with the tools available on the market.

What negative consequences does this lead to:

• The cost of reworking the framework – In the absence of a PoC, the team may waste a lot of time and resources on tools or methodologies that must be changed or abandoned later.
• Loss of time – Time wasted trying to adapt to an inappropriate tool could be used much more productively.
• Team frustration – Unsuccessful implementation attempts can lead to frustration and demotivation of the team.

My recommendations:

• Before starting a PoC, clearly define what you want to get out of automation and what the main requirements are for the tools.
• Limit the time frame. The PoC should not be too long. It is meant to be a quick assessment, not a complete implementation.
• Analyze the results. After the PoC, carefully analyze the results, considering technical and business aspects.

Proof of Concept is not just another formality but an important step that can save time, money, and effort. Don’t neglect this stage if you want your test automation project to be successful and deliver the expected benefits.

MISTAKE #3. Automating too early

The following common mistake that teams make is implementing the automation too soon. This can happen for various reasons, such as business pressure, a desire to get ahead of the competition, or mere enthusiasm. However, starting automation before the product or functionality is sufficiently stabilized (unless you are 100% sure of yourself) is a strategy that can lead to many problems.

Early action often means that the team starts automating test scenarios for the functionality that can change significantly. This entails the need to constantly refine and adjust the automated tests or even the core of the framework, which can be a source of significant losses of time and money.

Possible negative consequences:

• Frequent revision of test scenarios – A functionality or product actively developing will require constant changes in automated tests.
• Increased costs – Constant changes and test improvements can lead to increased costs for their support.
• Unstable test results – Constant changes in functionality will lead to test failures. You will often receive red reports that will demotivate the team and make the automation look bad to the client.

Recommendations on how to prevent this:

• Assess the stability of the product. Before starting automation, make sure that the basic functionality of the product or application is stable enough and will stay the same.
• Start with critical scenarios. If you start automation early, focus on the most vital and stable scenarios. For example, you can start by testing the API and postpone UI testing until the user interface is stabilized.
• Plan for changes. If you know some product parts will change shortly, consider this when planning automation.

Starting automation at the right time is critical to its effectiveness and value. It’s essential to weigh the product’s readiness for automation and only begin when it’s genuinely appropriate and will bring the most value.

MISTAKE #4. Too complex test scenarios

One of the most common automation mistakes is creating overly complex and lengthy test scenarios, especially regarding end-to-end (E2E) user interface testing. When teams start to rely heavily on E2E tests, skipping other levels of testing, several problems can arise.

E2E tests test the system as a whole by simulating the actions of a real user. They are significant, but they are also the most expensive and time-consuming. The complexity of the scenarios and over-using of E2E tests can obscure the need for other types of testing, such as unit or integration tests.

Possible negative consequences:

• Long execution time – A large number of long E2E tests increases their execution time, which can slow down the development process.
• More false positives – E2E tests tend to be less stable due to many external dependencies, which leads to frequent false positives.
• High support costs – Due to their complexity and length, E2E tests require significant effort to create and maintain.

My recommendations:

• Testing pyramid. Follow the principle of the automated testing pyramid: a large base of quick unit tests, fewer integration tests, and even fewer E2E tests at the top.
• Shorten and optimize. Consider splitting long E2E scenarios into short ones or combining several tests.
• Use parallel execution. If you still have a lot of E2E tests, consider running them in parallel to speed up the testing process.

Although E2E tests are essential to the strategy, you shouldn’t rely on them as the main tool. The right combination of different testing levels will help create a compelling and sustainable automated system.

MISTAKE #5. Lack of metrics and reporting

In my experience, automated testing without proper monitoring and analysis is like traveling through unfamiliar territory without a compass.

Although your automation system may be successfully executing tests, the lack of proper metrics and reporting makes you blind to the actual performance and value of your testing.

Metrics and reports help you track test status and results and provide feedback on problem areas that need attention. They reveal patterns associated with frequent failures and can help the team make informed decisions about where to improve their code or tests.

Possible negative consequences:

• Inability to identify problem areas – Without metrics, it’s hard to determine which application areas are most vulnerable or crash frequently.
• Difficulty in determining ROI – Lack of reporting makes it difficult to prove the effectiveness and payback of automation implementation.
• Deterioration of product quality – Without understanding how and where errors occur most often, the team has fewer opportunities to eliminate them.

Recommendations:

• Identify key metrics. Decide which metrics are most important for your project. This may include the percentage of successful tests, lead time, number of defects found, etc.
• Use reporting tools. Integrate reporting systems into your automation pipeline to automatically generate reports after each run.
• Analyze data regularly. Take the time to review and analyze metrics to identify possible improvements constantly.

The right metrics and an effective reporting system can be your milestones to high-quality and efficient automation. They provide the team with the feedback they need and focus efforts on the most critical areas for improvement.

Summary

Test automation is a powerful tool that, if used correctly, can be the key to a successful and high-quality product. However, without a clear understanding of your goals, choosing the proper mechanisms, and constant support, this tool can turn against you. Following the above recommendations can avoid the main mistakes and make your automation process as efficient as possible.

About the author

Olexiy Vovk is Senior Test Automation Engineer at Sigma Software Group. He has been working in IT for over ten years, seven of which are in automated testing. He participated in 20+ automation projects in telecommunications, social networks, cloud services, banking, and other domains. He implemented automation from scratch and refactored existing solutions. He has extensive experience in creating turnkey automation.