Five Keys to Mobile App Testing in an Agile Environment
The Speed of Change is Accelerating
How quickly is mobile app adoption growing? Consider this: consumers were expected to download 226 billion apps in 2015.1 Meena, Satish. “Consumers Will Download More Than 226 Billion Apps In 2015.” blogs.forrester.com. June 22, 2015. But convincing a user to download your app isn’t nearly as challenging as winning a loyal customer. As of 2014, 20 percent of downloaded apps were only used once—and just 39 percent were used 11 or more times.2Localytics. “App Retention Improves – Apps Used Only Once Declines to 20%.” info.localytics.com. June 11, 2014.
Why the poor metrics? It’s not just a matter of bad content—it’s often the result of poor app quality. And while 79 percent of users will give an underwhelming app a second chance, only 16 percent will give it a third chance.3Pramis, Joshua. “Are You a Rarity? Only 16 Percent of People Will Try Out an App More Than Twice.” Digital Trends CES Coverage. March 12, 2013.
If it were easy to produce high-quality mobile apps, everyone would be doing it. But mobile development is tough. Not only do consumers expect constant enhancements, but developers must also create a great experience on an ever-growing variety of devices.
It’s no wonder so many mobile app development teams are turning to agile development. Agile and mobile seem to be a perfect match. With agile development, you can iterate, test, and iterate again—quickly. Agile fosters cooperation across the entire development team, from testing to production. And agile’s speedy release cycles are ideally suited for mobile. Although most organizations are still in transition to agile development, Forrester reports that 24 percent of development teams have already gone 100 percent agile.4Source: Forrester’s Q2 2015 Global Agile Software Application Development Online Survey.
Making the Move to Mobile and Agile?
If you’re new to developing mobile apps, or if you haven’t already transitioned to agile development, you’re probably exploring both approaches. Before you make the move, stop and consider how agile development for mobile apps will affect your testing:
- Agile’s faster development cycles—which typically last two weeks—also require a compacted testing schedule
- Mobile requires testing across more platforms. You have to consider not only various OS versions within iOS and Android, but also take into account hardware diversity in the form of different screen sizes, controls, orientation, and so on.
The bottom line is that you’re being asked to provide a higher quality product while doing much more with less. This may sound daunting at first, but it’s not. You just need to follow the right best practices.
Five Ways to Meet the Testing Challenges of Agile Mobile Development
Here are five keys to making sure your mobile app testing meets the challenges of releasing high-quality apps while developing in an agile environment.
1. Automate testing whenever possible
Use automation to speed up your testing wherever you can. It’s the only reliable way to keep up with the need to test on today’s staggering diversity of hardware and software within faster release cycles.
Development teams often face the dilemma that they can’t save time with automation because they don’t have time to research and implement automation. If this applies to you, start small. Aim to automate even a small fraction of your testing, and you’ll soon see the benefits. Start with your build acceptance tests, which are typically the test cases that are run the most across all use cases. And use a real device automation tool that enables cross-device testing, which will allow you to write one test script that runs across devices.
So, is manual testing dead? No. You’ll probably never be able to automate all your testing—nor would you necessarily want to. After all, manual testing is especially useful in exploratory testing in which you need to put yourself in the place of the user. Ideally, your testing platform should allow both automated and manual testing.
2. Keep the user in mind
Apps are all about the user experience. So, at the risk of stating the obvious, don’t forget to run all your tests from the point of view of the user. To do that, you need to put yourself in the place of the user—which means figuring out which devices your users are using.
That’s not as difficult as it sounds. You can embed marketing analytics in your apps to gather this data. If you haven’t installed app analytics yet, device data from your website visitor analytics will often work in a pinch. Or you can just use the most popular devices for the general population.
It’s also possible to develop a device testing strategy that will give you a reliable sampling of the various OS versions and hardware configurations in your user base.
Here’s a simple example:
|iOS 9.2||iOS 9.0||iOS 8.4.1|
|iPhone 6S Plus||TEST||(NA)|
This table shows how you can get coverage across seven possible device/OS combinations by testing on just three devices.
3. Aim for continuous integration
Agile development teams create builds often. To ensure that builds are successful, and to find out if you have a problem as quickly as possible, it’s essential to automatically run each build through a series of acceptance tests.
Setting up a series of tests to kick off upon each new build will mean that in the midst of a two-week development sprint, your developers will be able to get feedback in minutes, rather than waiting a whole day.
4. Give testers and developers equal responsibility for testing
Agile is about doing as much work earlier in the process as possible, so that you can minimize the risk in your project. This includes testing. Your developers are responsible for quality control in the form of unit testing. Handing off code that doesn’t work as expected is a waste of the tester’s time, and the increased load on testing resources due to compressed timelines and numerous mobile platforms often makes testing a bottleneck in the development process.
Testers should be focused on the user experience, integration, and finding unintended consequences of code changes to other product features. Any unnecessary handoffs (or handbacks) in the process due to a lack of unit testing are time consuming and inefficient. Holding your Dev and Testing teams equally accountable for the process is a good way to trim as much lag time as possible out of the development cycle.
Where possible, give your developers an easy way to perform their own unit testing on real devices—ideally, on the same real devices that your testing team will be using. This will help ensure you’ll catch any issues earlier in the development process, when they’re much cheaper and easier to fix. The right testing technology can help here, too. Look for a solution that lets developers and testers alike automate tests, run them on real devices, share their work freely, and reproduce problems easily.
5. Maintain access to real devices
We’ve mentioned the need to let developers and testers run their tests on the actual platforms for which they’re developing apps. Yes, it’s helpful to run apps on emulators. But be aware that there can be not-so-subtle differences between emulators and real devices. For example, screen resolutions and screen sizes can differ from one device to the next. Controls may fit on one screen and fall off another. Emulators may not be as reliable as real devices for flagging memory or CPU issues. And can any emulator keep up with all the customizations available in Android devices?
Rather than relying on the proverbial box of phones, though, you can make life a lot easier for developers and testers by testing on real devices in the cloud. With just a few clicks, you can select the device of your choice and execute your test scripts—rather than waiting for Marc to come back from lunch so you can test your app on his rare flavor of Android.
Try Automated Testing for Yourself
Ensuring mobile app quality in an agile environment doesn’t have to mean more late nights, more stress, and fewer on-time deliveries. In fact, it should mean just the opposite. Give your developers and testers a mobile app testing platform that lets them test on real devices in the cloud, and watch what happens.
Keynote can help. We provide powerful cloud-based solutions for mobile app quality and performance, enabling companies like yours to reduce complexity, speed time to market, integrate with agile development, and deliver flawless mobile experiences that delight and amaze customers.
Try Keynote Mobile Testing for free for 7 days. Start your trial now >