Visitor Monitoring Inside the Box: Big Picture Before the Deep Dive | Keynote
Blog

Inside the Box: Big Picture Before the Deep Dive

By Product Management | June 24, 2013

CATEGORIES: Mobile Quality

In my last post I introduced you to the concept of thinking inside the box when it comes to mobile app performance measurement. I'm going to use Facebooks' mobile app to show you what I meant. The mission? test three native functions of Facebook's mobile app: login, photo upload, and the bane of HTML5: news feed loading. These are tests, of course, that can't be performed with web testing tools, since those tools can't test, you've got it, "inside the box."

I ran the tests on an iPhone 4S, in Seattle, on an AT&T network.

The first step was to create the test scripts (in order to test inside the box I had to act like the box). Scripting for native apps is a lot like scripting for web apps, only more sophisticated—browsers don't manage log ins, or know about things like swipes and taps and pinches. It's more sophisticated but not more complicated: same models, steps, processes. So I wrote a single script that took the device through the three tasks I wanted to test. In these tests I set the script timer to pause during user input activity, so that I was monitoring only the actual time the device was in action.

At the end of the week I opened up the web portal where the results had been written, and looked at how Facebook had behaved.

Blog 2 image

Facebook App Performance over 1 week period

Here it is: my week of testing in a graph. You can see when performance was poor: in one case it took 44 seconds to complete a task that more often took only 15 seconds on average. I could see where the problems were. (Of course, in the real world I would have been alerted when each of these spikes hit. But here, I let that be.)

I honed in on one of the spikes and clicked on it to get a clearer snapshot of the situation. Here's what I saw:

Blog 2 image2

The snapshot gave me four critical pieces of information—particularly, how much time it took and how much data it downloaded: more than a half minute to transmit 200K of data doesn't look good. But it's still just the big picture. In order to reach a root cause diagnosis of why performance was slow, I've got to examine application activity at the connection level.

And that's what I'll talk about next time.

Back to Top