Can Your Mobile App Pass the Test?
A Conversation with Keynote DeviceAnywhere President Faraz Syed
The explosion of the mobile app market is one of the top tech phenomena of the new century. Hundreds of thousands of apps are available for smartphones and tablets, and enterprises are quickly recognizing the business case for mobilizing their own workflows and data for their workforce. But it’s no simple matter to make sure that an app will perform in the field, on the myriad devices in users’ hands, as it was envisioned in the brainstorming session or on the back of a napkin. That’s where DeviceAnywhere comes in. Founded in 2003 and recently acquired by Keynote, Keynote DeviceAnywhere is a leader in mobile app field testing using real devices. Benchmark recently sat down with Keynote DeviceAnywhere President Faraz Syed to talk about the complexities, requirements and rewards of mobile app testing.
Benchmark: How did you get started in the mobile app testing business?
Faraz Syed: In 2001 I started at a company called Brience in San Francisco. My co-founder for DeviceAnywhere, David Marsyla, was also there at Brience. Brience was a company that was building an enterprise mobile platform.
Again, this was back in 2001, and the whole focus was, how can we mobilize existing enterprise applications? Can we provide the middleware to take these mobile? David was chief architect; he was building the platform. I was responsible for deploying the platform at customer sites.
What we realized very quickly was building mobile apps – certainly building the kind of apps that were being built back then –was relatively straightforward because the technology was quite young. However, making sure that it worked across all the different handsets that were already available was a huge challenge.
There were many issues getting those applications through specific devices because of browser differences, model differences, just lots and lots of differences between devices.
Benchmark: And this was before the advent of smartphones, right?
Faraz Syed: Absolutely, we were nowhere near smartphones. I mean, when we started our company, the first color phones were just coming out. But even then it was a pretty significant challenge trying to get apps to market with the surety that they had been tested across multiple devices.
So that’s how we basically came up with the idea that building applications is one thing, but testing is a whole different ball game. There was no technology out then that would help these customers in the mobile space accomplish their goals. There was a gap. So David and I started our company in 2003 and the rest is history, so to speak.
Benchmark: And certainly a whole lot has happened since that time.
Faraz Syed: Right. Our forecast was that, of course, mobile was bound to become a mainstream access mechanism for consumers. And of course, when that happens, businesses, whether they like it or not, are going to have to adopt it as mainstream access, a mainstream channel to their customers and to their employees.
It took a long time, but now especially given the rise of the smartphone platform, the adoption of mobility has become widespread. It’s everywhere. And our company has evolved from supporting feature phones that were prevalent back then to supporting smartphones, tablets, set-top devices, everything and anything that provides mobile access to consumers and to employees.
Over time we started getting engaged with more and more enterprise customers – banks, media companies, Web companies, e-commerce companies, healthcare, government. We’ve evolved with the mobile marketplace, where not only has the technology shifted from being feature phone-centric to very, very smartphone-centric, but we also see the center of our world shifting from mostly consumer-facing applications to also include employers looking at the real business value of mobile enterprise applications.
Benchmark: What sorts of businesses and what types of applications are they developing?
Faraz Syed: What we are seeing is that mobility is being adopted by everyone and anyone out there. It’s becoming mainstream, and mainstream enterprises – Fortune 2000, Fortune 5000 companies – are adopting mobility as strategic components of their execution strategy. There’s no way they can execute anymore without having a big and central mobile strategy around everything they do. They know that, and they’re executing on it.
We’re getting engaged with companies who are looking at their mobile strategies, both smartphones and tablets, to improve the productivity of their employees – providing more and more mission-critical information, more immediate information in real-time through these lightweight channels to employees who are mobile, even if that means just roaming around their hallways.
They’re enabling more immediate access to mission-critical corporate information, as well as enabling this additional very important channel to their consumers.
In fact, many of our customers have defined C-level and VP-level executives who are 100 percent mobile-focused – who are mobile guides within their organizations. Because they know that not only do they have to provide access for their customers to products and services, and for employees to enterprise information over mobile channels, but these are also channels they have to monetize.
Benchmark: Are you seeing momentum for your business in any particular area or is it just a mobile tsunami happening across the board?
Faraz Syed: It’s a mobile tsunami. Obviously, their are always fast and early adopters, especially people who have good connections to consumers. Most consumer brands have already embarked on and executed on sound mobile strategies that they’re continuing to build upon and build more presence around, whether it’s banks or consumer electronics or entertainment brands, for example.
At the same time everyone else – healthcare companies and government agencies and local government departments, and many, many others –are adopting mobility.
Benchmark: So is there a big difference in developing consumer-facing apps versus enterprise apps?
Faraz Syed: There isn’t a large difference at all actually. Our customer might be a bank or it might be a gaming company. Of course, the apps will differ in nature – a game versus a business app is two different types of requirements. But the fundamental need is that they want to make sure these apps work consistently across as wide an array of devices as possible.
On the flip side, if the same customer is building some sort of internal-facing app that their employees will use, then the requirements are somewhat more contained because to some extent, these companies can ordain to their employees that, ’hey, we will provide you support for these apps on the following platform or platforms.’ They don’t have to worry about every last device and all the permutations in the general marketplace.
So I would just say there is a slight difference, but within the platforms, the challenges and the efforts and the problems are about the same.
Benchmark: What are the biggest challenges mobile app developers are facing now?
Faraz Syed: The challenges haven’t changed much over the last several years, inasmuch as the problem is fragmentation. What we’re seeing right now is, obviously, most people are focused on the smartphone platforms – Apple and Android are the apparent winners at the moment. There are a couple of other players that could come out, like Windows, and maybe BlackBerry could resurface.
But within those two platforms still there is fragmentation. Apple is somewhat more contained except that they have a lot of older versions of the devices that are deployed. And then Android is an increasingly fragmented platform, because it has the same sort of problems that Java had as it evolved, in the shape of different vendors implementing it differently.
So the challenges haven’t gone away, haven’t been simplified. The sets of variables have changed.
Benchmark: One of the things you mentioned was that, if you’re just developing for the enterprise, your ecosystem is a bit more confined. When you’re doing a consumer-facing app, you have no control over that. How you go about helping a client to prioritize the platforms that they should be looking at?
Faraz Syed: We have a module called DevicePlanner. What we do is compile a lot of data, from third parties and from our own customers and developers, to figure out a recommendation to our customer for which devices and platforms they should target. That data goes across multiple variables – market size, market share, penetration, as well as physical and OS characteristics of the different devices. We can give customers a sense that if they tested, let’s say 25 devices, they should be able to cover 75 to 80 percent of the market they’re looking for.
We can say to a customer – who may be new to this whole business –for example, don’t worry about testing across these four different models. Because in essence these four are the same, so if you just target one of these four, you’ll basically be doing the same thing as targeting all four. Perhaps they’re just the same exact model branded differently across different carrier networks, or they are four different models but have the same OS version, same browser version, same screen size, etc.
So what we’re able to do is help our customers optimize the OS platforms and the devices they should target based on data we have collected.
Benchmark: Are demographics or qualitative data taken into consideration, or is it strictly about the devices?
Faraz Syed: No, our recommendation is more technical. It’s based on device characteristics and device penetration generally, without having any sort of demographic breakdown to it.
We try to take the widest walk of the marketplace out there, basically saying that, if you test it on these, say, 25 devices, then regardless of who the user is, you would cover about 75 percent of the mass consumer market out there. We also have an extended list that we can generate, which would then provide more like 95 percent of the market out there.
Benchmark: One of the things people keep talking about is the whole notion of Web apps, and specifically using HTML5 to create a native app experience without having to create a native app on various devices. What’s your point of view on Web apps?
Faraz Syed: Enterprises are very, very keen on Web apps and on adopting HTML5 as the solve-all for their mobile development needs. And for obvious reasons. They’ve been trained on the Web front through the process of building lots of applications for the Internet. They have trained personnel, skilled to do Web development, and to the extent that they can leverage the same skills to produce mobile apps, that’s what they want to do.
They don’t want their guys to have to learn and retrain on objective C and C++ and whatever else comes to the market for native app development. I can tell by working with and talking to my enterprise customers, they’re all waiting to adopt Web and HTML5 as the main way to produce mobile applications going forward.
There’s a certain gap in timing because HTML5 is not ready yet in terms of how it has been implemented. There are differences in implementations across different browsers, across different mobile phones, and so there’s still a gap there – it’s not mature yet. And the performance is not as good as native apps, so the trade-off is really in the kind of app that you’re developing. If you want it to be highly interactive, the native app is still your best bet.
And of course, native apps have a huge momentum behind them and there is a very clear path to market for native apps. Consumers know how to get native apps.
On the flip side, Web apps, HTML5 apps, may not perform as well today. Eventually they will. However, they are much easier to develop and much, much cheaper to maintain and distribute over the longer period of time.
Those are the pros and cons and the compromises people are looking at, but I can tell clearly that enterprises are keen to adopt HTML5 as their mechanism to get mobile strategies rolling.
Benchmark: Are there other tradeoffs that developers make when they go the HTML5 route versus a native app? With the native app we have gestures and accelerometers and GPS and all that additional functionality that’s such an integral part of the app experience. HTML5 takes advantage of some of that, but how close can it really come?
Faraz Syed: That’s exactly it – HTML5 only takes advantage of some of the core features that these smartphone devices are great for, which have caused all this adoption to occur. They can’t yet leverage all of them and so I think there will be a clear delineation between the kinds of apps that will be very suitable for HTML5 versus not.
For example, mobile banking apps might be very well done in HTML5, or a restaurant recommendation review site may be very well done in an HTML5 set-up. But a mapping application or a movie trailer application should probably be done as a native app.
It just depends on what the app needs to do on your platform, and therefore what coordinated features of the platform it needs to leverage. So performance is one big differentiator, and then access to the devices’ core capability – these are the differentiators to use to determine whether or not you can do a Web app versus a native app.
Benchmark: Do you envision that there will ever be parity there – that you’ll be able to access all of those features through HTML?
Faraz Syed: I’d imagine it probably will be, because even in the newer HTML5 they’ve given access to a bunch capabilities that were not previously available. So the trend is in that direction, where more and more of the platform’s capabilities are going to be exposed through HTML5 or to future versions of HTML, and therefore I think it will get there, but just with time.
Benchmark: What are the pros and cons of testing through emulation versus using real devices?
Faraz Syed: We believe firmly that the only way is to test using the real device platform that would be in the hands of a potential user.
And there’s a core reason for it. Emulators are good for maybe early stages of testing, when developers are doing their unit testing or just kind of doing rough testing, but what happens is an emulator runs within a larger-format, PC environment. So there is no way for it to be able to mimic and produce the exact environment that your app will run in, on a very restricted and constrained mobile platform.
For example, the CPU, the memory, the screen size, the network, the other apps running on it, and all those things are variables that you just cannot emulate to perfection.
That’s why we have built our business completely on providing testing capabilities on real devices, where you can access the device as if the device was in your hand. You can execute applications in their real native environment, with all the environmental factors that would affect that device if the app was running on a device in someone’s hand.
Benchmark: Where are your devices deployed? Can I test anywhere I want to?
Faraz Syed: We have a couple of different deployment models. We have a public cloud model, where we do have devices that are deployed in the U.S., Canada and Europe.
And then we have a private cloud model, where we can set up private device installations for any customer anywhere in the world. If you look at that, we have devices deployed in more than 40 countries around the world right now. But remember, it’s not a monitoring system, it’s a testing system. So providing access to the most pervasive national carriers in any particular country accomplishes the goal.
Benchmark: Are there any fundamental differences in the way you test a native app versus a Web app?
Faraz Syed: Not really; actually it’s about the same. All the considerations are the same, the problems, the challenges, the approaches are the same, and more and more these HTML5 apps behave very much like native apps – with local storage and local execution and lots of client-side interfacing. So the testing constrictions are the same.
Benchmark: What are some things that go into the ideal testing regimen from beginning to end? What does the process look like?
Faraz Syed: The one thing I want to say is that we have never claimed nor are we experts at the mobile test strategic planning process. There’s a difference in what we provide – it’s the technology platform that can be molded and interlaced into any company’s test strategy and test process. We don’t provide consulting or expert advice – we work with several partners who are experts in mobile testing and who have helped numerous customers on our side figure out the ideal test scenario for their needs and recommend that test process and implement those test scenarios.
But at the same time, what we do recommend to our customers is to start, obviously, testing early and test often and always test on real devices. Because what it does is reduce the number of defects as you progress through various stages of the development cycle. The more defects you find in the early stages, the cheaper they are to fix and the lower impact you have on your development efforts as well as your cost structure.
Benchmark: So your big news as a company is the fact that you have now combined forces with Keynote. What advantages does that offer to clients?
Faraz Syed: Well, obviously Keynote’s center of gravity has been monitoring and our center of gravity has been testing, and yet both of us have some presence in the other space.
What we’re now seeing is, first of all, the joint customer list is large and very strong and very comprehensive across verticals and industries. What these customers are looking for as they adopt mobile strategies is everything from developer-level testing to real solid QA testing to, once the applications have launched, live environment monitoring and then analytics beyond that. So our combined companies are able to provide a best-of-breed solutions set all the way from testing to monitoring.
The combined solutions get customers what they really need in order to put effective mobile strategies in play.
Benchmark: So if you can look into your crystal ball, what do you see the mobile app marketplace and its test environment looking like 18 months or two years from now?
Faraz Syed:First of all, I just think the mobile app marketplace is at a huge inflection point, growth-wise. It’s going to skyrocket because I can tell, sitting where I am, the kind of inbound leads and the kind of customer names that we are seeing – anyone and everyone, Fortune 2000, Fortune 5000 companies –getting into mobile. Some of them are new, first-year entrants into mobility, some are second- or third-year ’veterans’ of mobility.
But one thing is clear – mobile apps are here as a mainstream vehicle. And almost everywhere you go, just like you’re used to going on the Internet to a company’s website, you will expect to be able to access the company’s stuff over mobile channels in various ways.
Mobility is moving from being a side project, a ’nice-to-have,’ to a mainstream business priority that generates revenues and is a critical aspect of a company’s monetization strategy. We’re already seeing companies running the second or third year of their mobile strategy, allocating real significant resources to create platforms and infrastructure on top of which they can continue to grow their mobile strategies.
And I think that as the market continues to expand, with mobile as a mainstream strategy, enterprises will spend a lot more money in lifecycle management. These companies have been doing software for a long time; they’ve gotten very good at building software and they know that in order to build software, they need to create software management practices, and testing and monitoring are core components of that.
And I am positive, and we’re seeing it, that these same companies when they get into mobile software will want to implement the same kind of mobile software management practices as they move forward.
About Faraz Syed
Faraz Syed founded DeviceAnywhere (originally Mobile Complete, Inc.) in 2003 and has since led it to become a leader in the mobile testing industry, with more than 2,000 clients worldwide. Previously, Syed was director of solutions at Brience, a mobile WAP applications company, and technical architect at KPMG Consulting. He holds an MS in electrical engineering from Stanford University and a BS in electrical engineering from the University of Texas at Austin. Confirming the cerebral affinity between computer science and music, Syed is also lead singer in a San Francisco-area rock band, Kaif.