One of the most challenging topics we discuss in our Mobile Application Testing training course is how to determine what devices to purchase for testing and how to get the widest device coverage for tests.
When thinking through this there are several factors to consider:
- On what kinds of devices was my app designed to be used?
- Tablet or phone
- iOS or Android (or Windows or Blackberry) and which version!
- What devices do most of my customers use?
- Market research
- Analytics tools such as Google Analytics or Crashalytics
- What device features and sensors does my app rely upon?
- GPS, camera, bluetooth, fingerprint sensor
- High-resolution screen
- What kinds of devices/features are coming in the near future
- Wearables (Apple Watch, Gear)
- Larger tablets, stylus input
In many cases this results in a large number of potential devices. Especially if you are building an Android app. The challenge then becomes how to pare this down to a reasonable set of devices that your organization can manage. My strategy is to look at my answers to the above questions and start with the device/OS.
- Considerations for iOS apps:
- Apple users tend to upgrade regularly so the latest iPhone or iPad model running the latest version of iOS is a must.
- It is not necessary to obtain every variant of every available device model. A 32GB iPhone 6S is likely to function similarly to a 128GB iPhone 6S.
- Considerations for Android apps
- Android users are not as quick as iOS users to adopt the latest version of Android (if their carrier/device even allows it at all). Your focus should be on devices that run one of the previous two releases of Android.
- There are probably a few device manufacturers that top your list. Choose at least one device from your top two or three.
The next consideration should be the features that your application requires. You need to ensure that you cover all families of devices that implement that feature as well as some that don’t (in order to ensure that your app still functions properly, even if the lack of the feature inhibits some of your app’s capabilities). Equivalence partitioning is a good basic testing technique that can be applied in this context to narrow device choices down.
- If the feature is relatively new (i.e., iOS TouchID fingerprint sensors) you will need at least one device that has the feature and one that does not
- Similar features can be expected to perform similarly across different devices. It is not necessary to obtain every device that has a particular feature. One device can probably cover several different features.
- TouchID is likely to work the same on an iPhone 6 as it is an iPhone SE so there is not a need to obtain an iPhone SE specifically for the purpose of testing Touch ID.
- An 8 MP camera is likely to produce similar quality photos no matter what kind of device is being used
- Bluetooth 2.0 is likely to work the same no matter what device implements it
I hope that this helps get the thoughts flowing on how to choose devices to test. Look for future posts where I will talk about how to leverage testing tools to get broader device coverage without having to purchase and maintain many devices. Also, to get the full story on testing strategies, look for our Coveros training classes on Mobile Application Testing and other topics in your area.
