Bringing Mobile Testing Up to Speed
By Josh Galde | October 24, 2014
Do you have your phone handy? (There’s a silly question.) So what little number is sitting on top of the “update” button in your app store app? An 8? 27? 42? Mine is 23. That’s how many of the apps on my phone are pushing out a new release since I last updated my phone, which was only about a week ago.
Compare that to how many times you’ve updated an app on your computer, even in the last year. Maybe a couple apps, a couple times? Desktop and mobile release schedules are moving further and further apart. Research from IDC shows that almost half of mobile apps are updated once a month or more frequently; another third are updated at least quarterly. More than half of the top 25 iOS apps are updated every 30 days or less.
The world is moving at the speed of mobile now. It’s a different process and pace than the desktop days. The question is: How do you ship updates faster without cutting testing corners and sacrificing quality? How can your mobile testing program keep up?
The answer has to be found in automation.
The math is pretty simple. Here’s a real-life example from a major financial institution. To get each new release ready, they do a build acceptance test that includes 300 test cases across one Android and one iOS device. That takes two QA engineers two weeks to complete — forever, in mobile years. Automating BAT for that test scenario cut testing time by 90%, to just 24 hours for the test run. And it freed up those test engineers to focus on more important things.
Hopefully, it will also enable them to expand their test pool beyond two phones.
A strategy that brings automation earlier in the process also brings big benefits. Freeing the development team from time-consuming manual testing chores lets them focus on coding or exploratory testing; automation also enables them to run more extensive tests in a shorter amount of time. Integrating the testing process into build systems can save hours every day.
What to do with the resources freed up by automation is a key point, highlighted by the financial institution example. One of the most valuable uses of those resources is to expand test coverage across a wider device pool. There are at least three iPhone form factors in widespread use today, and as many iPads. And hundreds of Android variations. Even if you restricted your testing to the 10 core OS-device combinations most popular with your users, you’re talking thousands of test cases. The only way to successfully test at that scale is with automation.
The real point is this: The mobile testing job just keeps getting bigger and bigger for everyone involved, from development on through into production, with QA bearing more than a fair share of the burden. There’s more devices, more functionality, more releases, more everything. And with release cycles shrinking to 30 days or less, everything has to be done faster. You can’t throw enough people at the challenge, even if you wanted to. The answer, instead, is to use automation to enable existing staff to handle the more-more-more.
Of course, Keynote offers extensive automated mobile testing capabilities. Our open platform has hooks to integrate many tools and frameworks, including IBM Rational, HP UFT, Worksoft, Selenium, and the increasingly popular Jenkins for continuous integration. The Keynote DeviceAnywhere real device testing cloud is without peer, putting hundreds of real devices and OS permutations at the virtual fingertips of automated test scripts. And Keynote’s centralized test case management lets teams across the country or around the world view everyone’s up-to-the-minute test reports.
No matter how big your QA department — or even your whole IT department — there aren’t enough fingers or devices to properly put your mobile app through its paces on the kind of schedule it takes to keep it up-to-date. Automate what you can. And then use the resources you’ve freed up to focus on the testing that can’t be automated, and on other ways to enhance your app.
Contact us today to learn more about how the automation capabilities in Keynote Mobile Testing can help you squeeze more quality from your shrinking development cycles.