Mobile Application and Services Monitoring – A Primer
Monitoring the performance of websites based on the end user’s experience isn’t a new concept.
Monitoring the performance of websites based on the end user’s experience isn’t a new concept. For years, website owners have recognized the need for continual monitoring on the Web from their customers’ perspective. Not only are site owners held accountable for high availability, they also appreciate the correlation between performance, the user experience, and business outcomes. In the world of digital interactions, the margin between success and disengagement or abandonment is measured in milliseconds.
Measuring performance is not a new concept to mobile either. However, mobile performance has historically been synonymous with voice quality, and only the concern of mobile telecommunications companies. Today, with the exploding adoption of advanced smartphones and tablets, companies are taking a mobile-first approach to engaging with customers and employees. And as these mobile initiatives are delivered at increasingly rapid rates, the quality and reliability of the mobile apps, mobile web and connected services that support them has become critically important.
For the technology teams delivering customer and employee services in the mobile channel, it is important to understand that performance monitoring solutions which work for the desktop cannot be simply applied to mobile. Managing the mobile end user experience requires an understanding of the challenges posed by the complexities of the mobile environment.
Accurate and meaningful monitoring, measurement, and understanding of the mobile end user experience demands approaches that manage this complexity. This paper will explain the challenges and opportunities of assuring mobile app and site performance, plus offer strategies for accurately monitoring mobile end user experience.
2. Understanding the Mobile Ecosystem
Shipments of smartphones and tablets have far outpaced purchases of laptops and desktops. Connected wearable devices are emerging as the “Internet of Things” becomes real. A complex mobile ecosystem has come into being as mobile technology continues to evolve. Coupled with trends in agile development and continuous delivery, mobile introduces new pressures on technology organizations.
Both developers and operations need to account for diversity across these variables which compose the mobile delivery chain:
- Operating Systems
- Application Factors
- Web Clients
- Wireless Network Connections
- 3rd Party Services
- Back-end Infrastructure
3. Mobile Performance
Delivering outstanding mobile performance requires understanding the dynamics that impact end user experience. It also depends on the route you chose to take to your customers and employees (i.e. mobile web, vs. mobile apps). The fragmentation and complexity of the ecosystem weighs heavily on a user’s mobile experience and is most commonly unappreciated and/or misunderstood by developers and site owners coming from the desktop environment. These differences can impact application performance on the magnitude of 2-100 times lower for mobile.
And yet the habits (leaving multiple applications open and running, navigating freely from site to site) and expectations (fast pages, beautiful design and rich interactivity) of users aren’t really any different from their desktop experiences. This compounds an already difficult challenge.
3.a. Device Constraints
The mobile world is fragmented—there are thousands of mobile device models, with different form factors, operating systems, operating system versions, and browsers. The range of variation can be wide—especially between smartphones and tablets. And compared to the desktop, mobile device processing power is less, memory is limited and power (represented in battery consumption) is a finite and precious resource. All of these dynamics inhibit both mobile web and app performance.
3.b. Wireless Network Architecture
Communication between a mobile device, the wireless network and the backend service(s) involves multiple connections resulting in much greater latency than found in the desktop environment. When a mobile device issues a request for data, the following steps occur:
- The mobile device connects to the network operator cell tower.
- The cell tower sends each request to the operator proxy server.
- A WAP Access Point Network (APN) may be used to access the operator’s proxy.
- The operator proxy communicates with the content provider’s Web server and delivers a content or data response to the mobile device.
Wireless data network performance can vary, even if the carrier and device support high speed LTE. LTE enabled devices can “fall back” to slower network speeds where high speed connections are unavailable, or if bandwidth is constricted. This can happen for instance, if nearby subscribers are using a lot of data bandwidth, or traffic spikes suddenly.
3.c. Mobile Web Content Delivery
One of the most important differences between mobile and desktop-based Web content is the need for mobile content to be tailored for the various characteristics and capabilities of a particular device. Mobile website content can be optimized to mitigate these differences and provide a mobile-appropriate experience. Some companies chose to optimize content based on recognition of the device/client. Others use Responsive Web Design (RWD) which leverages a common page that “adapts” to the user’s viewport, be it a desktop monitor, smartphone or tablet. Finally, as mobile content is routed to the user through a wireless operator network and proxy servers, different and/or additional operator-specific content may be added during the download.
When requesting content from a Web server, a mobile device intersperses headers that describe the type of device and its capabilities. The Web server recognizes the specific headers inserted by different devices and selects content that best supports the device making the request. For example, most Web servers have a selection of image types and choose which image to send depending on the header. In the mobile world, there are hundreds of different types of mobile devices. Each one can support only certain content types and therefore each one intersperses different headers.
Latency: The Enemy of Mobile Web Performance
Due to the nature of the mobile wireless network, mobile data exchanges experience significant latency. This most often creates the greatest impact to end user experience, especially on mobile websites. Because of this latency, DNS lookups can easily take 250 milliseconds each over a 3G network, and a single blocking call averaging 500 milliseconds in a serialized page can have the same impact as downloading 10 unique object requests.
3.d. Mobile App Performance
Companies publishing mobile apps today, either consumer-facing, or to enable their employees for productivity in the field and with the devices they bring to work (BYOD), are beginning to understand the challenges of assuring performance.
With the mobile web, front end clients (web browsers) are well documented, understood and stable. But mobile apps effectively put your company in the business developing the front end client, and the requisite burdens of quality, performance and reliability are assumed by you. Application quality directly impacts performance and must be accounted for earlier in the development process.
Application availability is dependent not only on the reliability of the application code running natively on a given device, in a given user workflow, but also on the data services and APIs that support the app. As with mobile web, third party services (social connectors, authentication services, ad services, analytics, etc.) can impact both app responsiveness and the availability of key user transactions or workflows.
4. Mobile Monitoring Strategies
Performance monitoring for mobile apps and websites should focus on the mobile end user experience. Site and app availability, as well as responsiveness and key transaction functionality are critical measures of performance and the first place to start in the development of a monitoring strategy.
If a company is deploying a mobile web channel that coincides with its desktop web channel, monitoring should be integrated as much as possible to provide a holistic view of performance.
Two approaches should be considered:
- Passive Monitoring
- Active Monitoring
4.a. Passive Monitoring
Commonly referred to as “real user monitoring” or RUM, passive monitoring collects data about the responsiveness of mobile web pages as your customers or employees download them. RUM solutions typically also collect additional data about the user’s mobile browser, the connection speed and location of the user. For mobile apps, RUM solutions track application event (e.g. “login”) responsiveness instead of tracking page response time, and may also collect data about the device.
RUM approaches are good for relating performance to business goals and understanding trends based on actual usage of your mobile website or apps. These may include finding locations where performance is poor, which pages or places in your app can use improvement and for correlating key user behaviors with responsiveness. By understanding the relationship between performance and metrics such as conversion, engagement and abandonment, you can prioritize changes and issues by business impact.
Keep in mind that the number of RUM measurements collected is dependent on the amount of usage of your app or pages. And RUM measurements can be highly variable due to the diversity of devices, device state, OS version, connection, location and other variables.
4.b. Active Monitoring
Active monitoring, also known as “synthetic monitoring” or “proactive testing”, simulates a customer or employee accessing a mobile device or client and measures the experience on a repeated basis. Active monitoring solutions can collect detailed data about all mobile web page components.
Active monitoring solutions use controlled environments hosted in specific locations often with mobile (wireless) operator connections to test performance, resulting in highly standardized measurements. This is useful for trending and regression analysis. Because active tests occur on a continuous basis, synthetic monitoring can determine application availability and report on service level compliance. Active monitoring is also well suited to provide performance benchmarks against your competitors so you can deliver a superior user experience.
Real Device Platforms
Accurate active monitoring of mobile applications requires the capability to interact directly with the operating systems (Android, iOS, Windows Phone, BlackBerry), touchscreen gestures, accelerometer, camera, etc. Using real devices is the only way to ensure that application interactions and complete user journeys can be tested.
Emulated Device Platforms
Emulated device monitoring is well suited for websites and mobile web services. Sites and services require only a browser to render the page and do not depend on the operating system of the device (for example, SMS). But to accurately monitor such services, the solution needs to be able to emulate a vast array of devices.
Emulation technologies must contain, along with a wide range of device profiles, an awareness of the multiple factors comprising the end user experience including a live wireless carrier (mobile operator) connection. It is important to make sure that the monitoring solution chosen is capable of accurately representing the target device—that is, the device for which the content is designed. In addition, they need to have the diagnostic capability to quickly pinpoint the root causes of failures when something goes wrong.
If an emulation solution is not specifically designed for mobile content, it may mistakenly use Web Access Point Network (Web APN) settings and connect directly to the Internet without going through the operator’s proxy.
Another critical part of emulation is the ability to provide device-specific information on the number of simultaneous browser connections the device can make.
5. Keynote Mobile Monitoring Solutions
With deep experience in mobile monitoring and measurement, Keynote has the most comprehensive products available for measuring the true end user experience of mobile content on mobile devices. Keynote’s broad selection of active monitoring emulation and real device capabilities plus real user monitoring means that you can get the solutions you need, tuned to your budget and your required outcome.
Keynote Mobile App Monitoring
Keynote Mobile App Monitoring reveals the performance of mobile apps from your customers’ and employees’ perspective. It is the only 100% cloud solution that requires no modification of the app being monitored to measure both app health and availability. Mobile App Monitoring allows you to:
- Fast and easily understand mobile app performance
- Improve mobile application quality and reliability
- Assure the return on your mobile initiative investment
Mobile App Monitoring integrates the DeviceAnywhere cloud of real smartphones and tablets with the industry’s largest global test and monitoring network and leading monitoring service. By fusing together the best on-demand mobile testing and monitoring platforms, Keynote offers a holistic solution that delivers unmatched value across Development, QA and Operations teams.
Keynote Web Monitoring
Keynote Web Monitoring can actively measure the mobile end user experience from around the world, emulating more than 2,000 mobile devices. All active mobile web tests comply with the WAP standards defined by the Open Mobile Alliance (OMA). Full device emulation is delivered by sending the same headers as a real device, with an authentic mobile network connection, resulting in the same mobile content downloaded as that of the emulated device. Every measurement is a true representation of the mobile experience.
With Keynote Web Monitoring for the mobile web, you can quickly identify the root cause of failures experienced by end users, leveraging the detailed diagnostics at the page component level, including a screen snapshot of the entire downloaded page, the number of resources, the page/object URLs, and more.
Keynote Web Monitoring also includes real user measurement and analysis which reveals the performance of your site for all your visitors, no matter what device they use, or where they live. Leveraging modern Web standards, big data intelligence and integration with active monitoring tests, the real user measurement of Keynote Web Monitoring helps you identify performance improvement opportunities by geography, site segment, page, mobile browser and platform.
6. Summary: Mobile-first Delivery Demands Fast Feedback Based on End user Experience
The value of any mobile initiative is ultimately measured in adoption, engagement, conversion and productivity. Mobile delivery challenges organizations to develop, test and deploy more rapidly and collaboratively than on the desktop web. Gathering feedback from your customers’ and employees’ perspective is the only way to feed the iterations needed for continuous improvement. Use today’s modern approaches for monitoring to analyze behavior and assure reliability in production and after app download. The quality of your measurements is critical to the success of the project. Keynote knows that taking shortcuts with incomplete device models or incomplete routing paths provides results that do not correspond with the real-world end user experience.
When you need a mobile monitoring solution that reflects the true user experience, make sure you ask:
- Does this accurately measure response time on the route taken on a mobile network to the target Web server?
- How do I download content that may have been added by the mobile operator in-stream during Web content download to accurately represent the end user experience?
- Can I ensure that I’m downloading the exact content served by the Web server depending on the requesting device?
- Does this solution capture the multithreaded behavior of a mobile browser for specific requesting devices?
When you deliver services across desktop, tablet and smartphone screens, make sure you ask:
- Can I easily compare end user experience across my apps, sites and the various screens of my users?
- Can I easily compare end user experience based on “clean room” tests/benchmarks against what my actual users are experiencing?
When you want to make sure that your native apps are providing a specific end user experience, make sure you ask:
- Does this measuring and monitoring process run its tests on real target devices?
When choosing a vendor to provide mobile measuring and monitoring technology, make sure that the vendor has the technology and the depth of resources to provide test results that reflect the true end user experience. Educate yourself on the issues and technologies available before making a decision. You are betting your company on it. Make the right choice. Choose Keynote.