Planning for Mobile Testing Challenges

When you develop applications, it is hard to ignore mobile devices. Testing mobile applications provides however some additional challenges than don’t exist for desktop or web applications. In this article, Kostyantyn Kharchenko explains some of the specific challenges that you should include in your mobile testing plans.

Author: Kostyantyn Kharchenko, Svitla, http://www.svitla.com/

It was already 4 years since Julia had started her career as a QA tester and she felt quite confident with any task. At one of the regular planning meetings, the company Product Owner announced that their application was going to have a mobile version as the market needed it. Julia was assigned to be in charge of all testing activities for this new version of the application and create a mobile testing strategy. The first thing that she was expected to do was to create a list of possible challenges and solutions for the next meeting to discuss and create a strategy for testing. This was a new experience for Julia but she understood that the time demands these changes and this was a great opportunity for her to acquire new skills and knowledge.

The research started and Julia created the following report:

Challenge #1. The Biggest Mobile Testing Challenge is Devices

There are approximately thousands of different mobile devices through which our customer may want to access our mobile app. So, testing of our mobile application should take into account all these devices. Ignoring some of the devices equals to loosing numerous potential customers.

Possible solutions:
1. We can test exclusively using real devices.
2. We can test exclusively with emulated devices.
3. We can use a combination of each.

Real devices

Of course, nothing can substitute real devices in full extent to be sure of all potential situations and bugs. However testing with real devices is expensive, as we will have to buy all of them. Maintaining the mobiles devices that your company needs for testing is a real challenge.

Emulated devices

The benefit of emulated devices is that we can switch among device models, simply loading a new device profile that displays our application the same way that the real device would.

The emulators were designed with testing in mind. They have full instrumentation to provide detailed diagnostics about the protocols that go back and forth between client and server at the various levels of the stack. Single platform with frequent updates of device profiles can be used to test any device on the market without buying it and this makes emulated devices very cost effective.

Real devices + Emulated devices

Unfortunately, emulated devices still lack some individual characteristics that only the real device have like providing of the pixel-perfect accurate rendering, performance and speed specific to different devices, influences of external conditions that can impact the behavior of the device.

A good way out is a mixture of both emulation and real device testing. Early in the development cycle, we can achieve most goals at a relatively low cost due to the high speed and device diversity that an emulator can provide and while we don’t need the pixel-perfect rendering of an actual device. The benefits are gained by increasing the number of test cases and device types covered in the test suite. Then, we will test on real devices later in the development cycle to validate that the applications are functioning as expected and certify that all development requirements and objectives have been met.

Challenge #2. Regional Network Challenge

Every country has its specific network operators. Totally there are about more than 800 mobile network operators worldwide. Each mobile operator may support multiple network technologies and protocols like LTE, CDMA, GSM. Most network operators have inserted mobile Web proxies (or gateway) to regulate how, when, and if their customers are able to connect to a particular site.

Besides the technologies, there are also location challenges that are no less important. To be able to test the full network stack on a particular operator network infrastructure, we must be connected to the target network. We should also take into account that the radio signals are not strong on cellular networks. Thus, if we want to test against Telecom Italia, we must be in Italy, and if we want to test against China Unicom, we must be in China. But traveling around the world for the testing needs is not mandatory at all with the following options:
1. We can bypass the lower layers of the network and simply test over the Internet or LAN
2. We can use the real network by using either phones or modems.

Network Bypass

To use this technique, we will need a device emulator, as most real devices are not capable of doing this. We can use TCP/IP to connect directly to the server and ignore the GPRS tunneling systems used by network operators. We may need to check first if our device emulator can perform network bypass by using the Internet. It can be nice to look for device emulators that are able to access the operator’s proxy to perform a more realistic test. With the network bypassing, we will not use and thus pay for airtime.

Real Networks

It is possible to test against real networks. One way is to use real devices at the target location, but again traveling is costly. There is another option – real device in the cloud. This type of solution still can be expensive because to the cost of the device we need to add the cost of the remote control hardware, remote control software, and local software. However, it is possible to rent testing time on a resource supported by a vendor that lets us access remote real devices at any time we need it. All we have to do is to open an account and buy testing time with a given model of device.

Challenge #3. Manual or automated scripting

We will need to choose the method that will used to actually execute the test scripts. Script execution can either be manual or automated. As there are many different mobile devices that have different structures, the automated scripting should be created so to be suitable for any device. A script with strict keystrokes on an iPhone will not be effectively performed on a Samsung mobile phone, as their user interfaces are absolutely different. The good news is that most automated testing software provide high-level scripting functions such as “go to URL” or “send SMS” that are not dependent on the particular menu structure of the target device. Most device emulators can be used for automating test execution with a higher-level, abstracted scripting language that is not device dependent.

The cost of creating automated scripting is typically higher than the cost of a single manual execution of a test script, however, if run on a periodic basis, it can really save time and money. Besides, many automated scripting tools have a special capability that can test an entire site with a single command. This means a set up of quick test that will look at every page of our mobile app for errors and inconsistencies.

Conclusion

In the next meeting, Julia presented this list of possible challenges and options of solutions. The discussion went productively and testers together with developers marked the skeleton of future testing plan. They made many compromises in selecting combinations of techniques. It was decided to search for device emulators for testing that have the instrumentation and the network options that can give a lot of flexibility, make sure that these emulated device solution contain a high-level scripting solution. In addition, it was agreed to search for a vendor to get access to remote real devices. Julia knew that new challenges would come, but none seemed impossible to overwhelm.

About the Author

As the Chief Technical Architect, Kostyantyn Kharchenko leads the technical department as our IT guru and sets up the strategy and processes standards. Kostyantyn has over 20 years of experience in complex software systems development and design. His strong engineering background has contributed to Svitla’s continued success. Kostyantyn’s previous work experience includes top technical positions in companies like Digital Equipment Corp., Compaq Corp, Intel and Panasonic. His deep understanding of the emerging industries and strong technical skills allow him to help Svitla’s clients create technologies of the future. Kostyantyn holds Masters and PhD degrees in Computer Science from the Kyiv Polytechnic Institute, one of the top academic institutions in Ukraine.

This article was originally published on Svitla’s blog http://svitla.com/blog/mobile-app-testing-pre-planning and is reproduced here with permission of Svitla.