From Killer Idea to Killer App
An overview of app testing and monitoring
Where in the world are your customers today? Are they at home, at work, or somewhere in between? At a soccer game? The gym?
And what about your employees? Are they at a customer site? In an impromptu meeting in a company hallway? Or working from home today?
It's nothing new that customers and employees are all over the place. But what is new is that no matter where they are, now they're likely connected via a smartphone or tablet. They've got their e-mail in their hand while they're waiting in line at the grocery store, and the company pricelist on their iPad while they're sitting across the desk from a prospect.
The stats paint a picture of a mobile industry that's roaring like a runaway freight train. The worldwide mobile industry will reach $1.3 trillion in revenue in 2011; 300 billion of that will be for data, and for the first time the majority of that will not be for messaging. In the United States this year, the majority of new mobile phone sales were smartphones, and next year, the majority of mobile phones in use in the U.S. are expected to be smartphones. 1www.chetansharma.com, "State of the Global Mobile Industry — Half Yearly Assessment 2011," Chetan Sharma
And when you're talking smartphones (and tablets), you're talking apps. Apps and the Web browser are the two major thoroughfares for accessing content on mobile devices. And the stats show increasing momentum for apps as the primary route for mobile content and entertainment.
According to one study, for the year ending in June 2011, app usage grew by 91percent; users now spent over 81 minutes a day on apps versus 74 minutes on the Internet through a Web browser. 2Flurry, Inc., "Mobile Apps Put the Web in Their Rear-view Mirror," by Charles Newark-French, 6/20/2011 According to Pew Internet, half of all U.S. adult cell phone owners have apps on their phones, and the percent who downloaded apps nearly doubled in the past two years. 3Pew Internet Project, "Half of adult cell phone owners have apps on their phones, by Kristen Purcell, 11/2/11
So it's no surprise that companies that don't have a viable mobile strategy are scrambling to formulate one, and companies that have already embarked on mobile are fast at work on ways to take their strategy to the next level.
"It's a mobile tsunami," says Faraz Syed, president of Keynote Device Anywhere. "Many of our customers have defined C-level and VP-level executives who are 100 percent mobile focused — who are mobile guides within their organizations. Because they know that not only do they have to provide access over mobile channels for their customers to products and services and for employees to enterprise information, but these are also channels they have to monetize."
Mobile apps, moving target
Companies may now recognize the compelling need to reach customers and employees through apps, but getting those apps built, tested, and to market is a new and complex challenge. Mobile is a many times more fragmented ecosystem than the desktop environment. Existing, "traditional" software development protocols need to be adapted; more importantly, mobile requires an entirely different set of quality assurance and performance protocols.
Rather than testing software in a single operating system environment, mobile requires development and testing for as many as five OS platforms — iOS, Android, BlackBerry, Windows, and Symbian. Complicate that further with a multitude of devices, each with its own form factor and display characteristics, add in the nuances of various cellular carriers both domestic and international, and you have a test scenario sure to send even the most sophisticated IT department reaching for the aspirin bottle.
It doesn't have to be such a headache, though. An effective, expeditious alternative is to go outside and utilize independent testing and monitoring partners that already have mobile infrastructures in place. The partner then becomes responsible for procuring the handsets and other devices, and establishing the carrier contracts in various geographies, typically on a scale far broader than would be realistic for an in-house department to manage. Such outsourcing is the direction the market is taking. International Data Corporation projects worldwide testing services to grow more than 15% a year through 2015, from a total spend of $9.4 billion in 2010 — and mobile applications are a fast-growing part of that total. 4Integration Developer News, "Cloud, Mobile Development to Propel Double-Digit Surge in Testing Services, IDC," by Vance McCarthy
Devices everywhere, how to test them all?
With customers and employees carrying so many different devices, the first decision is what platforms and devices to focus on for testing — there are simply too many to test them all. But by clearly defining the purpose of the application and examining market share and other data, it's possible to arrive at a manageable pool of devices that serves to validate an app with confidence.
"What we do is compile a lot of data," says Syed. "That data goes across multiple variables — market size, market share, penetration, as well as physical and OS characteristics of the different devices. We can give customers a sense that, if they tested, let's say 25 devices, they should be able to cover 75 to 80 percent of the market they're looking for."
The device decision is simpler for an enterprise app to be used by employees, because the enterprise will determine what devices are supported. In that case, it's definitely possible to test on all the devices for which the app is intended.
Testing: There's an app for that
Another decision that needs to be made early is what portion of the app testing, if any, will be done with emulators, and what portion with real devices. Emulators can be great timesavers, but they are far from perfect. Most experts recommend constraining their role.
"Emulator-based approaches are not recommended even by the device manufacturers," says Pradeep Kumar Govidasamy, head of the Mobile Testing Center of Excellence at Cognizant, the global IT and consulting firm. "But emulators are useful during the initial phases of development and testing, to validate sections of source code as it's being written. After that, though, you really need to use real devices for field testing, network testing, interruption testing — basically, for all of your functional testing."
Real tests use real devices
It's not that testing with real devices is the gold standard, though that's one way to look at it. The fact is, there is simply no way to emulate the behavior of a mobile app in the field unless the device is literally in the field. System overhead, memory usage, CPU speed — a host of variables impact app functionality in ways that just can't be reproduced in a lab. To get real test results, you have to deploy real devices.
But creating an internal test program using real devices is a challenge. The devices have to be bought, carrier contracts have to be established, test scripts created, users trained, results compiled and analyzed — for each geographic market. And then the devices and contracts have to be dealt with after the testing is done, or maintained for ongoing monitoring.
This complexity is why so many enterprises use an outside test partner that offers an established infrastructure with hundreds or thousands of devices deployed over a broad geography. The test provider works with the client company to develop the necessary scripts, and then leases time to them on its network to run the tests. This is testing in the "public cloud," which means that many clients utilize the same devices and infrastructure to conduct their tests. It's an ideal solution for most companies — there's no upfront capital expenditure and tests can be quickly executed on demand, on a budget-friendly pay-per-use basis.
But when a company has particular security concerns and is reluctant to expose its data on a shared infrastructure, or if it has ongoing testing needs, a "private cloud" solution is in order. In this case, the test provider procures devices and contracts to create a private test network exclusively for the use of a particular client company.
In either case, the client team has ready online access to the entire process, from scripting to running the tests to reading results.
Cloud-based app testing is an automated process. There's no human being working the phones out in the field; rather, the devices are remotely operated by machine, performing the interactions specified in the test scripts and providing feedback on the devices' responses.
"You can execute applications in their real native environment," explains Syed about Keynote Device Anywhere's automated testing, "with all the environmental factors that would affect that device if the app was running on a device in someone's hand."
The goal is to test every last bit of the app's functionality: to perform every task, load every screen, and process every input that a user might throw at it. Only when all the efforts to break it have failed is the app a candidate for release to its intended market.
Hoping for HTML5?
Yes, it's incredibly complicated to develop, test and monitor apps for the ever-changing multitude of mobile devices in the marketplace. That's why so many are pinning their hopes on HTML5, with its promise of "build once for all" simplicity.
"Enterprises are very, very keen on Web apps and on adopting HTML5 as the solve-all for their mobile development needs. And for obvious reasons," Syed says. "They've been trained on the Web front through the process of building lots of applications for the Internet. They have the trained personnel skilled to do Web development, and to the extent that they can leverage the same skills to produce mobile apps, that's what they want to do."
HTML5 does offer the potential to create a native-like app experience, with the ability to incorporate some native functionality such as gestures and GPS. And while some major players, such as Facebook and YouTube, have built apps with HTML5, we're a long way from matching the native experience. There are differences in how HTML5 is implemented across browsers and platforms, and performance is not always on par with native apps. Still, there are opportunities for HTML5.
"For example, mobile banking apps might be very well done in HTML5, or a restaurant recommendation review site may be very well done in an HTML5 set-up," says Syed. "It just depends on what the app needs to do on your platform, and therefore what coordinated features of the platform it needs to leverage. So performance is one big differentiator, and then access to the devices' core capability — these are the differentiators to use to determine whether or not you can do a Web app versus a native app."
Another consideration is distribution of the app. Native apps have app stores where they can be discovered and adopted by users; there is no such mechanism for Web apps. One solution that some app developers are adopting is to create an HTML5 app inside of a native app wrapper. That way, the bulk of the app — the HTML5 portion — can be created once, with just the wrapper to be customized. Then it can be marketed through the app stores like any other native app.
In any event, whether the app is built as purely native, hybrid, or an HTML5 Web app, the testing regimen and the imperative to test on real devices is the same.
The proof is in the performance
The app has been put through its paces, revised and retested over and over, and now every effort to break it simply fails. Break out the champagne, it's ready for release — but don't celebrate too long. Because no app is ever ready to be left all on its own.
With websites, apps, basically anything in the online world, things can and will go wrong. There could be overload issues (because your app is so popular!), server issues somewhere in your network, unexpected usage patterns. Any number of things can dull the sparkle of your shiny new app today, tomorrow or six months from now. Constant vigilance is the key to happy users and a successful app.
"There are people who look at it from a positive point of view who say, I want to have the fastest app and best user experience, "says Tim Murphy, marketing manager, mobile quality at Keynote Systems. "And there are people for whom maybe the glass is half-empty, who say, I just want to avoid headaches. But the solution is the same for either one."
That solution is monitoring. Perhaps not surprisingly, some of the same best practices for testing also apply to monitoring performance and availability, most notably, using real devices in real-life scenarios. How and what you monitor, though, is somewhat different from what you tested in the development phase.
"When you do testing, you want breadth and depth both, testing on dozens of devices, testing each and every workflow of the application," says Nisheeth Mohan, senior product manager, mobile, for Keynote. "But when it comes to monitoring, you typically focus on the most popular workflows on the most popular devices in the most popular markets."
Monitoring offers a similar public/private cloud choice as described above for testing. Keynote, for example, has a well-established infrastructure with a broad array of devices on different carriers in multiple locations around the world. An app owner can use this network for ongoing scheduled monitoring. Or, if security is a concern or if volume justifies it, Keynote will build a private cloud network with the devices and in the markets that matter most to the app owner. And with a private cloud, you have the luxury of doing more troubleshooting than you could do in a shared public environment, because you own all of the time on the devices. You can pause the application and monitoring script and interact with the app to see exactly what the end user sees and find out exactly what's causing a problem.
Public or private, the key to an effective monitoring program is to find out quickly about problems before they become crises.
"You don't want the news and media to be all over your issue, and find out about it from them," Mohan says. "You want to proactively fix issues before they escalate and have a big impact on your user base. You do that with global monitoring, and by having intelligence built into the system that proactively alerts you as soon as an issue starts occurring."
Mobile app monitoring covers two broad areas: availability and performance. A well-written monitoring script will confirm the availability of the mission-critical functions and entire transaction processes. At the same time, performance is measured in terms of how much time it takes a given transaction to complete. It's been well established in numerous research that users will not tolerate long delays — in fact, that they expect mobile performance to be on par with desktop performance — and not only will they abandon sites that are too slow, they won't come back.
Counting the stars
Performance is a vital key to satisfied users, and satisfied users are the key to broader adoption in the marketplace for one big reason: reviews. Poor performance means fewer stars, and that means fewer downloads.
"There are half a million apps in the iPhone app store," Mohan says. "How do you get visibility among all those apps? A lot of it comes down to customer reviews. Apps that have good customer reviews get higher ranking in the app store. So performance is critical. If you get bad reviews because of poor performance, your app is doomed."
Conclusion: Test early. Monitor always.
Whether you're aiming to create the next animal-throwing blockbuster or to connect your field employees with vital enterprise data, the success of your app depends on rigorous testing before release, and diligent monitoring after. Cloud solutions provide access to an established and far-reaching infrastructure of real devices — a critical criterion for the effectiveness — and bring speed, automation, and market reach to both the testing and monitoring processes.