As software development projects adopt frequent delivery approaches like continuous integration and continuous delivery, the speed of test execution become a key factor for their successful implementation. Distributed testing might be a solution that helps improving test execution speed. This article discusses the pros and cons of a distributed testing implementation process. It will also present the Selenium Grid open source testing tool.
Author: Alexey Rakitsky, Qulix Systems, http://www.qulix.com/
When new versions of software come up very frequently in Agile projects with short iterations or in organizations adopting continuous delivery, optimization of the testing process is the main goal for the QA team. Distributed testing is a very popular method as it significantly reduces the testing time. This is achieved running simultaneous tests on multiple computers interacting with each other.
The general objective of distributed testing is to run many tests in parallel in multiple environments. This makes it easier to run the tests against a vast combination of browsers and operating systems (OS). The motivation for such this approach ranges from reducing test execution time to achieving cross-platform coverage.
Imagine you work on a large-scale project with short iterations and you have more than 800 automated tests to run. Even high performance tests could need about eight hours to do their job depending on the test type. The development team cannot wait for hours until testing is finished. In this case, adopting distributed testing can reduce testing time to one hour if we add five more servers and run the tests in parallel.
In order to implement distributed testing, you need to take care of the following three conditions (it relates only to the distributed testing of one single test suite):
- First, all shared resources should operate correctly during the distributed testing process. The most frequent shared resource in test automation is a file system. It is especially important when files are uploaded or downloaded during the testing process.
- A separate data usage should be enabled during the parallel testing on one test stand. It is necessary that each test uses its own data to avoid any interference with other tests.
- Finally, test that includes environment configuration changes cannot be executed in parallel with others.
We are now going to present how the distributed testing process can be organized using the Grid feature of the Selenium open source tool.
Selenium Grid
One of the most popular open source tools that supports distributed testing is Selenium Grid. Selenium Grid allows running tests on remote machines in parallel. It represents a client-server solution that includes a master server and node servers. If you have a suite of 50 tests, you could set up Selenium-Grid to support four different machines (virtual or physical) to run those tests. Then your test suite should complete in (roughly) one-fourth the time as if your tests were performed sequentially on a single machine.
Image source: http://docs.seleniumhq.org/selenium-grid.jsp
Other open source and commercial tools can be used to perform distributed testing. In the open source category you can also explore the features of Apache JMeter, Appium, Tsung or Twister.
Advantages and Disadvantages of Selenium Grid
Advantages
First of all, Selenium Grid allows parallel test execution without extra efforts saving testing time. You can therefore drastically cut time for test execution.
Furthermore, it provides an option to select the Runtime Environment based on the required parameters. Tests can be easily setup and supported. If we have launched the tests but they failed for some reason, Selenium Grid provides us with tools to diagnose the problem and reconstruct the same environment for new test iteration.
It is possible to run test suites on a local machine that will be running on remote computer(s). With Selenium Grid we can assign particular tests to computers, which allows to flexibly managing the distribution of test to machines.
Disadvantages
The main peculiarity of Selenium Grid is that the code is executed on the same local machine where the tests are launched, whereas the remote machine only receives the browser/mobile device control commands. This creates additional issues: uploading/downloading files, launching of any additional software on the same machine together with a browser.
The performance of Selenium Grid is very low in case of a large amount of node servers.
Significant time and efforts might be required for the initial implementation of distributed testing, if a parallel test running was not set up before in the test automation framework.
Using Continuous Integration
Continuous Integration (CI) allows setting up execution of automated tests depending on the requirements. You will be able to schedule the tests for a particular time or event. Additionally, CI has an option to execute tests on a remote machine. The main difference is that a full copy of the testing code is executed on a remote machine rather than just transmitting browser/mobile device control commands.
Benefits
The CI approach provides flexibility for test planning and execution. You can for instance set up automatic test launch upon the event ‘Project commit’. In this case, the tests will starts to verify the new build when developers commit new functionality. It also saves test execution history to get the statistics report later on. For a large-scale projects it is particularly important to keep the history and have accountability on the product quality.
As full testing code is executed on the remote machine, no extra actions from automation tests are required to implement such an approach. It means that tests that require file download/upload are easier to configure via CI. Finally, it is possible to determine on which remote machine a particular test suite will be running.
Limitations
The system provides a possibility to set up a remote testing of the entire test suite. To execute a parallel testing of a single test suite, it is necessary to divide this suite into parts or use external tools to enable it.
If a test suite is split, some difficulties might arise. It will be more difficult to analyze general statistics of the whole test suite execution. There are no results on the whole suite just some separate results on each part of a suite.
Furthermore, it brings additional difficulties when it is required to add or delete a remote machine from the cloud environment.
Conclusion
According to my experience, the best solution is to use a combination of Continuous Integration and Selenium Grid. For the parallel execution, it is better to use Selenium Grid, however it might require some time for implementation. Usually, we use test development standards that allow launching tests on Selenium Grid by default, but this only concerns the web and desktop testing. Selenium Grid is by far the best tool to test on mobile devices.
About the Author
Alexey Rakitsky has been working as a Test Automation Manager at Qulix Systems for more than 9 years. He has a strong technical expertise in such areas as functional test automation as well as loading testing. Alexey has managed over 30 successful projects with duration from a couple of months up to several years.