The Quest for a Great Mobile Experience
How organizations are tooling & testing for mobile quality
Remember 2011? “The Year of Mobile”? And then there was 2012, “The Year of Mobile.” And 2013, most definitely The Year of Mobile. And 2014? Of course.
That such pronouncements continue on an annual basis is evidence that, while businesses clearly recognize the essential role that mobile now plays, many are still trying to sort out their strategies; put the right people, programs and resources into place; and organize their digital workflows to reach the top of their mobile game.
Who owns the apps and websites? Who develops them? Who buys the tools? Who makes sure they work? What happens if they don’t?
These are largely settled questions for traditional desktop sites. But in many cases, they’re still being answered for mobile. Keynote recently surveyed more than 1,600 mobile development and testing professionals to get a clearer picture of the current mobile testing and QA environment. The results, published in Keynote’s The State of Mobile Software Quality 2014, describe a discipline that, while universally regarded as business-critical, often still struggles to find its place within an organization and in its budget.
The mobile landscape in 2014
The need for effective apps and mobile websites is clearly recognized. In a recent Accenture survey of senior executives, more than three-quarters placed mobility in their top five priorities for 2014. Four in ten companies report that they have aggressively pursued and invested in mobile technologies. At the same time, however, overall most say they have not made substantial progress; less than half say their mobile efforts have been effective.
When Accenture probed further, they found that in about 7 of 10 companies, deficiencies included “the inability to keep pace with new mobile devices, systems, and services; no clearly defined, centralized ownership of mobility initiatives within the organization; failure to develop new, or redesigned business processes...and lack of internal and external skills. 1“Mobility: Fueling the Digital Surge,” Accenture Mobility Insights Report 2014, Accenture, 2/14”
Not surprisingly, some of these same themes surfaced in Keynote’s State of Mobile Software Quality 2014survey. Many, though not all, of the organizational and operational challenges of mobility overall are present for mobile testing in particular. These include device challenges, organizational structure, and another major stressor: speed.
Pressure to perform
“One of the questions we were trying to get answered in the survey was:Are these technology organizations inside these companies feeling crunched by what’s going on in mobile?” says Rachel Obstler, senior director of product marketing at Keynote. “And the answer is yes.
“The quality expectations for mobile apps are actually higher than what you see on the desktop. But the testing time allocated for mobile is roughly the same as desktop. The resources are actually lower. And the release schedules are typically much more compressed. So do more, faster, with less. That’s essentially what they’re being asked to do.”
Much of that pressure on the mobile testing team appears to be driven from the outside in, by users. Users very clearly are not adjusting their expectations to allow for slower mobile processors, slower networks, and inherently longer latency.
“You see a lot of metrics around the fact that, if people have a bad experience on a mobile device, they won’t go back again to that app or site,” Obstler says. “They expect sites and apps to load in three seconds. Some sources even say two seconds.”
That pressure to perform is being passed on to the mobile teams but, as noted above, they are not getting any extraordinary resources to deal with it. Many organizations still seem to be struggling to gain mastery over mobility.“Businesses are investing more in mobile because they know it’s important,” says Michael Sowers, CIO at Software Quality Engineering, a leading testing and development training firm that helped with the Keynote survey. “But it’s a challenge for testing teams to catch up. You’ve got to learn mobile. You’ve got to put mobile testing practices in place. You’ve got to get some tooling in place. And you’ve got to learn all these disparate platforms across manufacturers — not an easy thing.”
Where does mobile testing fit in?
Mobile testing responsibilities are handled differently in different organizations. Only a small majority, 55.5 percent, have dedicated mobile-specific testing groups, according to the Keynote survey. That 55.5 percent breaks out to19.7 percent witha centralized mobile testing group, and 35.8 percent with smaller mobile groups within business units or divisions.
“I think whatever is important to an organization must get the investment and focus,” Sowers says. “That said, in my experience, having a dedicated group for an emerging new technology such as mobile has the benefit of growing technical expertise rapidly — implementing a consistent, repeatable test process, and standards, and tool infrastructure. It’s too early to characterize trends; we see a mix of both [mobile-specific and centralized QA], as the survey results confirmed as well.”
While the testing groups themselves may be split between centralized and distributed, most companies are holding onto centralized purchasing authority. A little more than half report that mobile testing tool decisions are made by a centralized QA or tools group.
What’s being tested on mobile?
By far, the biggest focus is on functional testing — build acceptance testing (BAT), regression tests, new feature tests, etc. — with 47 percent of respondents “most concerned” about this testing area. Performance testing, both pre- and post-launch, was a distant second with 24.6 percent, followed by usability testing at 21.9 percent.
It would seem that much of that testing is being done on real devices, as survey respondents put “easy access to many device models” as their number one most important feature needed for functional testing. Some 58 percent of respondents reported that they do most of their mobile Web and app testing on real devices.
“There’s a pretty strong belief out in the market that, in order to ensure that your app or website is going to work on the specific device, it’s important to test on that specific device,” Obstler says.
Device and OS proliferation are an ongoing headache for mobile development, operations, and testing. Even Apple has seen its form factors and OS variants multiply; and of course, there are literally hundreds of variations of Android OSs and devices. Testers typically choose a manageable sample of devices that are representative of most of their user base, rather than trying to test on a very large number of devices. They’re also pushing validation with real devices earlier on in the development process, which has typically been emulator territory.
“One of the things we’re seeing with our customers is that they want to move the ability to quickly test or validate a new release on real devices further back, say into the development group,” Obstler says. “You’d typically be doing that kind of build acceptance testing on an emulator.”
“It may not be on 20 different devices,” she continues. “It may be on one Android, one iOS, five or ten devices, something like that. But this ability to take a new build and very quickly run through a series of automated tests across a sample of devices is very important to companies. This obviously enables them to deliver a much higher quality build to the QA team with high confidence that it has no fundamental problems.”
The question of automation
“Easy automation capabilities” was the second most important feature for functional testing, after access to many device models.But while nearly 60 percent of respondents have employed automation, only 14 percenthave automated more than half of their mobile testing. Resource constraints are one likely reason for this apparent underutilization of automation.
“Usually testers are running to catch up with new technology,” Sowers says. “We first do things manually, then later as time permits, begin to implement automation. The barriers are traditionally time and money in whatever we strive to automate.”
Automation in mobile is typically employed for regression testing of new releases — doing the grunt work of testing all of the existing functionality in an app or website to make sure none of it got broken in the new build. This speeds up the release, and frees up the QA people to do higher-value testing — like trying to break something.
The other main use for automation in mobile is the above-mentioned build acceptance testing. Automation supports a more agile process with more frequent releases that might be typical of some organizations, such as Facebook, for example, that may be doing multiple releases daily. Automated BAT supports a faster build-release cycle characterized by small changes and a more streamlined QA process.
Powerful New Scripting
Object-level scripting has enabled a new level of granularity and versatility for mobile testing.
“The big advance in mobile test automation over the last year or so has been the ability to do object-level scripting,” Obstler says, “giving us the ability to get at the object instead of just looking at what appears on the screen and matching it based on text recognition or image analysis. Object scripting is actually looking at the code.
“It’s a much better way to drive your scripts. It means the script that you write for Android device A is also going to work on Android device B. That means you have to write a lot less script. It’s much easier to automate. It also means that when you make small changes to your UI, it’s not going to break your script. It means a lot less maintenance.”
Top testing challenges
Perhaps surprisingly, there were no screaming red flags in terms of mobile testing challenges; on a scale of one to ten, none even broke a seven. Tools, time, and test devices were the top three challenges, followed by testing methodology/process and availability of mobile testing experts.
“Time, money, and expertise,” is how Sowers describes the challenges. “I think it starts with, how do I shift my focus from the skill set and tooling and process perspective? Which is about the expertise. Of course, growing that expertise requires an investment, which is the time and money part.”
For Obstler, it’s about putting the right resources into place, and finding the right partners to work with.
“The whole industry in general is playing catch-up to the fact that there’s a really steep adoption curve in mobile,” she says. “As a result, now budgets are being put into place and effort is being put towards it. It’s a challenge for sure. It’s a little bit lagging where the market is going.
“And you have to plan for staying ahead of the market. Right now people need to be looking at the vast increase in traffic, the increase in sales, the transactions that are being done on mobile devices, and ask, where is this going to be a year from now? What tools and processes do we need to put in place?”
One thing is certain: there isn’t going to be less traffic or transactions or sales on mobile devices. For many uses, mobile has already exceeded the desktop. It’s a top priority for retailers and financial institution and all manner of media companies.
It’s not too soon for test organizations to get their tooling, device access, and processes into place to be able to thoroughly vet the apps that are in production now, and the bigger and better ones that will be in production tomorrow. Because who knows? 2015 might just be The Year of Mobile.