Why and When You Need to Test on Real Devices
February 9, 2017
Mobile app testing is not complete until executed on real devices. With the rise of mobile use, it’s clear why mobile testing matters now more than ever. Users have low tolerance for apps that are slow and buggy, competitors are fierce to usurp them in an abundant app market, and quite simply, business results count on them.
For quality assurance (QA) teams, this means testing is not only important but using the right tools to ensure app quality and performance is critical. Simply running tests on simulators for iOS and emulators for Android does not assure an application meets quality standards.
Why You Need to Test Your Mobile Apps on Real Devices
Emulators and simulators are virtual devices that do not have mobile hardware components and display on a computer screen. They offer primitive, important UI elements reflecting the internal behavior of a device and allow users to conduct limited functional testing. But to assure quality, testing teams need to consider hardware, operating system (OS), and carrier variations on real devices. This variation is referred to as fragmentation and is the inability to write code once and run it anywhere.
While application code is packaged in the same format (.apk for Android, .ipa for iOS), there is a variety of environments in which they can be used. Apple has more control over which software version applications can work with, however, Android fragmentation is more diverse. Manufacturers such as Samsung, HTC, and LG, for example, support all Android OS and platform versions.
But Android users still have less than a 1% adoption rate for the latest distribution of the OS version, and this creates more combinations of hardware and software that need to be tested as opposed to Apple whose adoption rate has almost reached 80%.
The challenge with fragmentation is test results can vary on different devices. Screen size and resolution can affect how UI elements render, network providers can regulate applications differently, and, when platform versions are updated, protocols can change, like how applications call services. Testing on real devices eliminates many shortcomings or unforeseen errors.
Real Devices vs. Emulators and Simulators
Real devices allow real-life variables to be included in test cases, like battery level or the effect of additional applications running in the background. Emulators, on the other hand, do not support these variables and thus cannot accurately represent potential real-user scenarios.
Emulators and simulators, for example, rely on the host computer’s network, whereas real devices can switch between wifi and cellular data. Emulators also depend on the host computer’s memory and battery life, and have no location services. How proximity alerts, messages, and push notifications are handled can vary across devices and platforms. For these reasons alone, it is imperative to test on as many real devices as possible to get the most complete insights on app quality and performance.
When You Need to Test Your Mobile Apps on Real Devices
It is never too early in the development process to run tests on real devices. The latest trend is testing after each feature is developed using Test-Driven Development (TDD) and Continuous Integration (CI).
Test-Driven Development begins with a tester writing a test case that targets a discrete feature of the app. A developer then uses that test case as a guide and codes only the features necessary to make that test case pass. Once the test case is in a passing state, both tester and developer move on to the next feature of the app. This process is repeated until all features of the app are created and tested.
By using TDD on real devices, functional tests can pinpoint bugs more accurately than using emulators/simulators because the application will be interacting with the hardware directly, not a hypothetical virtual instance. Through CI tools, this process can be streamlined with automated builds and tests, which can be run in parallel and reduce the time it takes to go through the test plan on multiple devices.
The most opportune time to test on real devices is at the end of release cycles for both development and pre-production builds. Beta and User Acceptance Testing should always be done on real devices because, in practice, their purpose is to test real-world situations.
It is also vital to run performance tests on real devices after each release cycle because every network provider handles traffic and volume differently. Performance testing across multiple carriers can discover latency and capacity issues. For example, a ticket-sale application known to have high volumes of traffic when concerts are announced should be tested on real physical devices.
Testing on emulators and simulators is not enough. Performance and functional tests behave differently on real devices because emulators are simply a virtual device. Fragmentation creates more variables that can affect functionality and performance, so the only complete way to test the lack of standardization is to run tests on as many real hardware and software combinations as possible.
It is never too soon to run tests on real devices. TDD and CI are the latest development trends that organizations are trying to adopt for mobile apps. Whenever there are major builds, whether they are development or production releases, testing on real devices helps isolate and discover more faults in applications. And with higher quality apps comes better business.