Are you doing enough mobile testing? Most don’t, find out how you compare to other development and testing teams. XBOSoft surveyed global software development and software testing teams to understand mobile testing best practices. Use the results from their report to benchmark your own efforts. Below is an article based on the part of the report that covers what types of testing organizations do.
You can download the full report here: http://www.xbosoft.com/contact/research/mobile-testing-report-2013
Author: Alan Trefzger, XBOSoft, Senior Quality Consultant
Mobile Testing Practices
Testing practices range from, what type of testing is performed, to localization testing in different countries; from how many types of devices you test on to testing on different networks. Many of these examples reveal how much more complex mobile devices are to test than desktop platforms.
What type of mobile testing do you do?
Only 43% do any performance testing. This should be 100%. Letting your customers find your performance problems can be very expensive. For example: Google found that for every 1 second increase of page load time, a 20% loss in customers was recorded.
The challenges for performance of mobile testing can be divided into three parts:
1. Mobile App testing on the device [1]
2. Network issues
3. Server performance.[2]
For an Agile environment, many contend that regression testing can’t be done for each cycle unless you use automated testing. For an Agile team that has a production release each month, there is only so much time that can be spent on testing with such a short time frame, so only new features are tested. Regression testing of all those features and functions which shouldn’t have been affected is often neglected. Automation is the only way to solve this dilemma.
What type of platforms do you support?
Android and iOS continue to dominate with both having over 80% of the developer’s support, which tracks with most other surveys.
Windows has 41% support, which is very surprising since Windows market share is in the low single digits and 10% are developing for Windows to the exclusion of either iOS or Android. They must believe market research company IDC’s forecast report that Window’s market share will rise to 20% in the next few years and it is hard to believe that that the Nokia-Microsoft alliance won’t be successful. Window’s developers are half as likely to have a dedicated development or QA team.
How many different devices is your SW used on (approximately)? Please count one device that is used with 3 different OS versions as 3 devices.
How many actual devices do you test on? Please count one device that is used with 3 different OS versions as 3 devices.
Most mobile developers test with a small set of devices. Unfortunately, the number of devices on which your application will run is staggering. Even with testing on more than 24 devices (one company tested on 200 mobile devices), you are testing on only a small fraction of the devices that your mobile application will be used on. Since it is impossible to test on the full universe of devices that your application will be used on, you have to test smart and choose a minimum set of devices that gives you maximum coverage. This is not easy since that set of devices will change from quarter to quarter. This is where 3rd party QA experts can really help. They have sets of devices for different types of application and are continually updating them. They can match you up with one of their pre-determined sets of devices, and then modify and add to it based on your specific requirements.
Do you test how your application functions on different networks?
Almost 60% do not test their application on differing mobile networks. Networks are the primary cause of performance degradation on mobile devices. Those that do not test the performance of their application on different networks are going to lose customers and won’t know why. Mobile performance is drastically different on the different networks. It may work well on one type of network and not work at all on another type of network.
Is your application used in different countries?
Do you test how your application(s) function in different countries?
63% of the respondents plan to have their application used in more than one country. Performance of networks in different counties varies greatly. Even if the network in the other country is the same as the network in your own country that you tested on, the performance can vary greatly because the distance of the server from the user will be dramatically different. Tools can emulate the different network conditions around the world in a cost effective way. With 2/3rds in multiple countries, not only performance and security are issues, but also localization is important. 65% of those mobile applications in multiple countries are not tested in those countries.
Conclusion
A typical mobile development organization
* Likely to only perform Manual Functional Testing and Usability Testing,
* Support mainly Android and iOS,
* Test on less than 8 devices,
* Not test on different networks, and
* Be used in multiple countries, but not tested in multiple countries.
This is typical, but not necessarily best practices. ‘Which platforms to support?’ and ‘which countries to be in?’ are marketing decisions. But ‘how many devices?’, ‘which tests to perform?’, and ‘where to test?’ are definitely not getting enough attention. Best practices require the following:
* Performance, usability, as well as functional testing are required,
* Security and automation are needed or not depending on the application.
* Testing on a well thought out set of devices to get maximum coverage with a minimum of number of devices is usually in the range of 15-25 devices, depending on the application.
* Localization and network testing are needed to find out ‘under what conditions’ failure happens before your customers find them.
The interest in mobile is now ubiquitous. No organization can afford to ignore it and how it affects its profitability.
References
1. Testing Performance of Mobile Apps – Part 1: How Fast Can Angry Birds Run?
2. Testing Performance of Mobile Apps – Part 2: A Walk on the Wild Server Side
About
XBOSoft is a software quality assurance services company, providing both consulting services as well as testing services. XBOSoft wanted to understand the state of mobile testing so that it could better provide services to that market. The 2013 State of Mobile Testing Survey Report is divided into four parts:
Part 1 – The staffing issues with mobile testing.
Part 2 – The QA processes used by mobile developers.
Part 3 – The QA practices for mobile services, and
Part 4 – The companies who are developing mobile services.
In this article, XBOSoft used the results from Part 3: Mobile Testing Practices. The 2013 State of Mobile Testing Survey Report can be downloaded at http://www.xbosoft.com/contact/research/mobile-testing-report-2013