Inside the Box: Big Picture Before the Deep Dive
By Product Management | June 24, 2013
In my last post I introduced you to the concept of thinking inside the box when it comes to mobile app performance measurement. I'm going to use Facebooks' mobile app to show you what I meant. The mission? test three native functions of Facebook's mobile app: login, photo upload, and the bane of HTML5: news feed loading. These are tests, of course, that can't be performed with web testing tools, since those tools can't test, you've got it, "inside the box."
I ran the tests on an iPhone 4S, in Seattle, on an AT&T network.
The first step was to create the test scripts (in order to test inside the box I had to act like the box). Scripting for native apps is a lot like scripting for web apps, only more sophisticated—browsers don't manage log ins, or know about things like swipes and taps and pinches. It's more sophisticated but not more complicated: same models, steps, processes. So I wrote a single script that took the device through the three tasks I wanted to test. In these tests I set the script timer to pause during user input activity, so that I was monitoring only the actual time the device was in action.
At the end of the week I opened up the web portal where the results had been written, and looked at how Facebook had behaved.
Facebook App Performance over 1 week period
Here it is: my week of testing in a graph. You can see when performance was poor: in one case it took 44 seconds to complete a task that more often took only 15 seconds on average. I could see where the problems were. (Of course, in the real world I would have been alerted when each of these spikes hit. But here, I let that be.)
I honed in on one of the spikes and clicked on it to get a clearer snapshot of the situation. Here's what I saw:
The snapshot gave me four critical pieces of information—particularly, how much time it took and how much data it downloaded: more than a half minute to transmit 200K of data doesn't look good. But it's still just the big picture. In order to reach a root cause diagnosis of why performance was slow, I've got to examine application activity at the connection level.
And that's what I'll talk about next time.