Going Back Outside the Box: The Video Timeline
By Product Management | July 29, 2013
Mobile app performance statistics—the Application Activity data I showed you in the last post—take us Inside the Box. Being inside the box is critical, but there's more to the story. It's good to see where performance clogs exist, but to get the complete diagnostic picture you have to see failures while you're watching the app in action. So, pack your bags, we've got to move once again—we're heading back outside the box.
Here, too, the toolkits for web applications point the way. Many of you are familiar with the filmstrip view you get from WebPagetest. While a page is loading, and load times are being monitored, images of the site as its being built are snapped. Web developers can then replay those images as a filmstrip to watch the page load along with the performance measurements.
For mobile app testers, that kind of video timeline is even more important.
First, we're watching something more than server downloads (that is, we're watching all the connections made while the app is running). With mobile apps, if there's a failure, we also need to understand the impact on the users—what's their experience? And then there's the whole mobility issue. In the real world, you're testing multiple devices in multiple locations. When an app fails in one of those locations, you're probably not there to be able to just pick up the device to watch it fail.
Let me show how useful a video timeline is for monitoring mobile apps. Here's an example of one I ran for my Facebook tests:Download Facebook Video Timeline
You can see I got a complete and accurate picture of what the app was doing, which would include any error messages the app itself generated.
In my case, I was 20 feet from the device. When I was alerted to an issue, it was easy to walk over and run the past timelines. I also ran ad hoc tests, watching the app while monitoring Application Activity results. But what if I my testing tool alerted me that some serious failures were occurring on a device that wasn't across the room, which was located in some other city, running a different mobile device, on a different network? That's the real-world testing environment. How do I see the app in action when it's on a device 3,000 miles away?
Sure, I could run the video timeline in my testing portal. But, here, past tests aren't enough: I need to run those tests right now, maybe multiple times right in a row, read the results and view the app in action. That's the only way to do remote troubleshooting. My video timeline tool lets me do that. I can take remote control of the device—no matter where I am physically. I run tests on the spot—and see the video timeline instantly. Now I can see the app go through its paces, and determine whether the failure was inside or outside the box.
I hope through these last four posts I've given you a good glimpse on the differences between testing web apps and mobile apps. Mobile app performance measurement should not be limited only to smartphones, but it should also be done for tablets and feature phones. There are rumors that Facebook will be announcing “Facebook for every phone”, a two-year secret project, which intends to make Facebook available on every mobile device around the world. When that happens, mobile app performance measurement tools, which provide Testing Inside the Box capability, will be even more needed to provide a clear, probably indispensable, view of mobile app performance across devices, networks and locations.