If you are a good software tester, but if you can’t communicate well what you find, you provide little value and are not very useful to your project team. This article gives you some hints specific to mobile applications testing on how to become a great bug reporter. Your teammates will love you for this and you will prove your value as a skilled team member.
Author: Jonathan Kohl, Kohl Concepts Inc.,www.kohl.ca
This article is based on the book “Tap Into Mobile Application Testing” by Jonathan Kohl currently available in the beta version on Leanpub: https://leanpub.com/testmobileapps. People who purchase the electronic beta book on Leanpub will get free updates to the book, including the finished version.
When you spot a problem, notice something weird, or have a negative reaction or suspicion, you need to investigate further so you can report this information in a useful way. The difference between getting an issue addressed and fixed to help improve the product experience and an issue or bug getting ignored often has to deal with the accuracy of the information. Here are some tips to follow when you notice something that should be dealt with, and how to report it well.
Determine if it is Reproducible
Sometimes we spot a problem, stop and write down the steps it took to reproduce it, and then move on with a task. A few minutes later when we decide to log the issue formally, we can’t repeat it. In other cases, we repeat the issue once ourselves, document the steps to reproduce, only to have the bug report sent back to us because the programmer can’t repeat it. We’ll talk about how to cope with intermittent bugs later in this chapter, but to reduce the waste in time and energy incurred by logging something others can’t repeat, I use Jerry Weinberg’s “Rule of 3.” When I am filling in a bug report, I make sure I can repeat the bug three times before I start recording it. If I can’t repeat it three times in a row, I need to spend more time researching and investigating before logging a report for the programmers. Some are quite easily repeatable, so I move on and start gathering information and logging the bug. Others may occur once out of three attempts. I will then try to repeat it more times, and I may only be able to repeat it five times out of ten, or thirteen times out of twenty attempts. If that’s the case, I add the word “Intermittent” to the bug title and include those metrics. That just means that the programmer will have to try to repeat it a few times rather than once to be able to repeat the conditions and start fixing the bug.
Clear Title and Description
I describe the problem in the simplest, clearest terms that I possibly can. This is like the headline of a news story, it needs to be compelling to draw in the reader. Examples include:
* “Poor performance during application launch on iPhone 3GS”
* “Application crashes when submitting purchase form with a weak wifi signal”
* “Incorrect login error message is obscured in French due to special characters”
I spend time on my titles to make sure they are clear and free of typos. It sounds easier than it is, I will often go through several ideas per bug until I get it right. I will frequently ask a colleague if my bug title looks ok or not if I have any doubts. I also have a brief description, usually based around a task or fulfilling a goal to put context around the bug. That helps the programmers figure out why they should fix the bug, and what sort of priority it should have. Here is an example: “While walking between wifi connections, I tried to submit a purchase, but the app crashed. This seems to happen when the wifi signal is weak.”
Exact Steps to Reproduce
This is incredibly important. Write down the steps, and practice it yourself. Also explore to see if there are any extra steps that you can remove and still see the problem. If you eliminate one or two steps and it no longer occurs, be sure to also note that in the report. This is useful information for diagnosing the source of the issue. Be sure to note anything in the environment when you are working with steps to reproduce bugs on mobile devices. Remember that movement activates sensors, different networks have better or worse performance, transitions between network types can cause problems, and low battery can cause the device to behave very differently. You may notice a problem when you are on the move, get back to your desk and be unable to reproduce it. I always take screenshots with the device (or take a picture with a different device) so that I can make sure I haven’t missed anything important. I also pay careful attention to what I am doing, and how different environments and activities can cause problems.
Once I work to repeat the bug, I will narrow things down. I had a low battery when the bug occurred. Does it also occur when the battery is charged? Maybe the wifi connection wasn’t at full connection. Can I reproduce it with full wifi? I noticed the bug while walking and moving the device, which gets sensors involved. Does it still occur when I am at my desk with the device sitting on it, without moving? In other cases, I might have been using a cellular network. Can I repeat it when I’m using a more powerful connection like wifi? Understanding these issues helps me create a much more accurate bug report.
What Should Have Happened
Testers have great ideas on how apps can provide value to customers and users. We often use these motivations as we generate test ideas. We notice that there is a problem, but it may not be obvious to everyone else. Furthermore, the programmer may only have a vague idea of a better solution, so our ideas will help them as they solve the problem, or as they confer with designers and other developers. You can also find information in specifications, requirements documents, or in developer guidelines that are supplied with each mobile development framework program. This can help provide credibility to your bug reports.
Device and Environment Details
At the minimum, always include the following:
* Device manufacturer and model number (e.g. HTC One X)
* Operating system version and sub-version (Android 4.0.3)
Notice how specific that is? I don’t say: “An HTC Android phone” or “HTC One X on Android 4”, because they are too vague and there are too many other variations to look into. You can find this information in the settings app on your device. Also be sure to include anything about the app, including:
* The version of the software you are testing (Project Xena 1.01 Build 24) Here are some device details to consider including when relevant:
* The date and time when you noticed the problem
* Wireless network type
* Low battery (can cause device to slow down)
* Movement and sensor interaction
* Using location services, such as GPS
* Other activity on the device: other apps running, interruptions by OS, communication, etc.
* Apps your app depends on, such as web browsers
* Device heat (can cause device to slow down)
Here are some environmental issues to include when relevant:
* Network connection strength, or if a transition between two networks occurred
* Light conditions and weather
* User movement and travel speed
Supplementary Information
The old saying that a picture says a thousand words holds true for bug reports. I may have a poor bug report, but an image or video capture demonstrating the problem may save the day. What I was unable to describe well I may be able to demonstrate perfectly. One of the most difficult pieces of supplementary information for beginners to gather are stack traces. Stack traces are complex error messages that are often suppressed from the user interface. You see a friendly message: “Oops, sorry, an error occurred” but in the background, more specific information is recorded and stored. To get this information requires development tools, so you will need to work with your friendly neighborhood programmer to figure out how to get this information. It’s worth the extra effort though, this information is incredibly helpful for programmers.