At the end of our previous installment, we touched on the fact that Apple instills tight control on its iOS. At the other end of the spectrum, you have the open Android platform; widely embraced in a society where freedom of information is a growing concern. But openness is a double-edged sword, and Android devices have already faced some issues with malware. It isn’t much of a stretch to assume that spyware, bloatware and especially junkware are trailing right behind or are out there already; after all, “Developers of malicious software are nothing if not creative” (https://quickstonesoftware.com/). And, this is all happening on devices many believe are posed to replace your credit cards by the year 2020 (https://mashable.com/2012/04/17/mobile-payments-2020/).
Jakob Nielsen, “The King of Usability,” identified three advantages of native apps, and I was pleasantly surprised to find that one of them is directly related to testing:
-
- Empirically, users perform better with apps than with mobile sites in user testing [emphasis added].
- Apps are much better at supporting disconnected use and poor connectivity, both of which will continue to be important use cases for years to come. When I’m in London and don’t feel like being robbed by “roaming” fees, any native mapping app will beat Google Maps at getting me to the British Museum.
- Apps can be optimized for the specific hardware on each device. This will become more important in the future, as we get a broader range of devices.
It could be argued that testing web mobile apps presents a narrower challenge, and while it certainly seems like less of a headache than testing multi-platform ports, there’s still quite a bit to do. In a perfect world, you would be able to create your mobile web app using standard tools and expect it to behave similarly across all platforms; leaving the portability work to the natively-coded interpreters in the browser (can we please call this the Java pipe dream?) A perfect world ours is not.
“A test configuration is a set of configuration variables that specify the correct setup required for testing an application. …The configuration variables include hardware, operating system, software, and any other characteristics that are important to use when you run the tests. Each test configuration can represent an entry in your test matrix.” (https://docs.microsoft.com/en-us/azure/devops/test/mtm/test-configurations-specifying-test-platforms?view=azure-devops)
Software testing as a discipline was relatively young when the first cellphones supporting third-party apps came around, but I can only imagine what the test configurations included in a test matrix might have looked like in those days. Every manufacturer had its own proprietary hardware design and OS, and the amount of phone offerings in the market was overwhelming. Developers turned to Java and, while this was logically the sensible option at that point, I firmly believe that in this case, a good old monopoly was welcomed by developers and testers around the world. These days, Android has just captured over half the market, followed by Apple with around a third and RIM (Blackberry) hanging in with just under half of that. Windows phones continue to trail in the market, struggling to reach 4% (https://www.comscore.com/esl/Prensa-y-Eventos/Infographics/Android-Captures-Majority-Share-of-US-Smartphone-Market?cs_edgescape_cc=CA).
The configurations list I’m looking at for my current assignment (mobile web app) has been narrowed down to a handful of devices and eight emulators mimicking a few OS releases. iPhone 4S and iPad 3 are in there, but the Android offering is reduced to the Samsung Galaxy (two models thereof). Risk management is a big part of software testing, so I’m kind of wondering if Sony should have been factored in that equation as well, being that they just came out with their shiny brand new HD, PlayStation-compatible, Xperia phones. The Blackberry makes an appearance as well with the 9900. Because the client is very interested in portability testing, Blackberry becomes an interesting case study here. While the Android design takes more than a few cues from the iPhone, and Blackberry is a distinctively different device. I must admit I am somewhat disappointed to see the 9900 featuring a touch-screen, as I was really looking forward to seeing how the app would perform in the complete absence of one (logically, this can still be attained through test case restrictions, but I guess I’m a purist at heart).
In any case, the future of test matrix complications derived from user interaction is still bright, with styluses making a comeback, voice commands being given yet another chance with Siri and split-screens being produced by some manufacturers.
I have waited until this point in the article to discuss the elephant in the room: we carry all of these networked devices around in our pockets. Have you ever considered how neat it is that your app display rotates to accommodate using the phone in portrait or landscape mode? BAM, your layout test cases just doubled. Also, they are phones, what happens if they ring while the app is running? Welcome to the magical world of interrupt testing (http://en.wikipedia.org/wiki/Mobile_application_testing). They have batteries and no back-up power – what does your app do if the battery disconnects or runs dry? What happens to the data? They are connected to a wireless data network that is sensitive to location, geography and weather conditions – what happens when the signal is gone?
Stay tuned for next month’s installment where I’ll talk about the challenges of – “A Continuously Moving Target”.