Software testing is an important and tricky topic for mobile app development. With multiple operating systems versions and devices manufacturers, it is very time consuming to cover all possible combinations. This is why it is recommended automate testing with a parallel perspective rather than performing tests sequentially. In this article, Bakir Jusufbegović explains how to implement parallel mobile testing for iOS devices with Appium.
Author: Bakir Jusufbegović, Atlantbh, https://www.atlantbh.com/
This article assumes that you’ve already read ‘Parallel Mobile Testing with Appium – Part 1: Android’, which talks about motivations, approaches and challenges of establishing parallelism in mobile test execution with Appium and Android. The next step forward was to do the same thing with Appium and iOS. In general, the iOS platform in combination with XCode and the specifics of MacOSX usually lead to more headaches in mobile automation. We assumed this would be the case, but we didn’t expect the challenges we would face. This article will describe the problems we were trying to solve, our approaches as well as the simple solution we eventually found. Stay tuned for more details…
Initial problem
As explained in the previous article related to parallel mobile execution with Appium and Android, the general idea is to connect as many devices as possible to a machine (the number of connected devices is restricted by the number of USB ports), deploy the app on all connected devices and run Appium tests that will be executed against the target app on all devices in parallel.
We initially used the same approach for Appium and iOS, but we encountered one major limitation: only one instance of Apple Instruments can be run on MacOSX. Since Appium uses Apple Instruments underneath to run tests against an iOS app, that meant we are unable to spin up multiple Appium server instances on a single machine. We could spawn multiple Appium server instances (just like in the case of Appium/Android), but all of them would connect to the same Instruments instance (i.e. one iOS device per machine)
Our approach (first try)
When we encountered the limitation above, our immediate reaction was ‘What if we can spin up multiple MacOSX VM machines that are all hosted on one physical Mac machine?’ The way we envisioned this was:
* Launch a MacOSX virtual machine for each iOS device connected to the host.
* Connect the iOS devices to guest machines instead of the host (i.e. map physical USB ports to virtual USB ports on VMs).
* Provision test resources and Appium on each guest machine (i.e. start Appium, run tests, stop Appium).
In theory this made sense, but in practice, we found out that it was not a legal approach since Apple only allows users to run one MacOSX VM per one physical MacOSX machine, which did not help us in our scenario.
Our approach (second try)
When searching for various solutions, we found out that the limitation existed on XCode itself. XCode version < 6.3.2 didn’t support running multiple Instruments instances on one machine. By upgrading to a newer version of XCode and using Appium 1.5, which added support for spawning multiple Instruments instances, we could make pretty similar solution, like in the case of an Appium/Android combination, which can be found here on https://github.com/ATLANTBH/testing-research/tree/master/parallel_mobile_execution/ios.
This overall diagram describes how the flow works:
In short, the mobile devices are connected to a USB port. When the ios_runner_sh script is executed, it invokes the number of Appium server instances that is equal to the number of UDIDs and then executes the test script for each UDID (each real device). Each Appium server instance invokes one Instruments instance to communicate with a real device, which has its own tmp file for intermediate operations.
The full script can be found at https://github.com/ATLANTBH/testing-research/blob/master/parallel_mobile_execution/ios/ios_runner.sh, but here we will just talk about the most important parts of the script and how they differ from the Appium/Android script described in Part 1 of this article:
1. Discover all UDIDs using the system_profiler tool (in other words, discover the IDs of the connected iOS devices):
echo "[INFO] Getting all device UDIDs..."
udid_data=()
system_profiler_path=`which system_profiler`
udids=`$system_profiler_path SPUSBDataType | grep -o "[A-Za-z0-9]\{40\}"`
for udid in $udids; do
udid_data+=($udid)
done
2. Deploy the app using the ios-deploy tool on all the connected devices (in general, the ios-deploy tool is a helper tool for deploying iOS apps using the command line without using Xcode’s cumbersome cli tools):
function deploy_app() {
udids=$1
iosdeploy_full_path=`which ios-deploy`
ios_deploy_pids=()
for udid in $udids; do
$iosdeploy_full_path --id $udid --bundle $APP_FILE 2>&1 >/dev/null &
pid=$!
ios_deploy_pids+=($pid)
done
# Wait for apps to be deployed
wait ${ios_deploy_pids[*]}
}
3. Based on the number of UDID devices, spawn an equal number of Appium server instances with one subtle but important difference: specifying the instrument’s output tmp file (to spawn multiple instrument instances and have each Appium server work with a different instrument instance, you need to specify different tmp files for the instrument’s output. This is done using the –tmp flag):
function start_appium_server() {
appium_main_port=$1
appium_server_logs=$2
udid=$3
instruments_output=$4
echo "[INFO] Starting Appium server instance with main port: ${appium_main_port} for udid: ${udid}..."
appium_full_path=`which appium`
nohup $appium_full_path -p $appium_main_port --tmp $instruments_output > "$appium_server_logs.$udid" &
}
4. Start TestNG or RSpec tests for each combination of UDID and Appium server instance ports. This works in pretty much the same way as in the Appium/Android combination.
5. Clean up by shutting down the Appium server instances once the tests have been completed. This works in pretty much the same way as in the Appium/Android combination.
Conclusion
In general, mobile parallelization is becoming a real need these days, and more and more cloud mobile providers are offering solutions based on the fact that you can choose a farm of devices and execute any number of tests against them. Since this solution is currently very costly (as explained on the sample AWS Device Farm in part 1 of this blog), it is good to know that an in-house solution can be used for your own infrastructure. After all, why not try out parallelization (and see how tests work in general) at no charge and then decide to shift towards cloud mobile providers if you need more infrastructure. As explained in the first article, the diversity of mobile phones makes everyday testing of mobile apps a big problem so a good solution is very often somewhere in the middle (between using your own infrastructure and purchasing cloud mobile solutions). After careful planning and fragmenting devices that you want your app to be tested against, you can make a smart decision to use your own infrastructure for devices that are available to you and purchase cloud mobile solutions only for devices you are missing. Therefore, the solution we presented in this blog can be a good starting point to make a parallelization effort at no cost and utilize the infrastructure you already have.
About the Author
Bakir Jusufbegović is a Lead QA Engineer at Atlantbh with 7+ years experience in software testing, automation and DevOps. Particularly interested in process that improve complete QA pipeline and often include development, devops and documentation tasks also. He is always willing to explore/learn new technologies and enhance his skills as well as improve existing software solutions.
This article was originally published on https://www.atlantbh.com/blog/blog-parallel-mobile-testing-appium-part-2-ios/ and is reproduced with permission from Atlantbh .