Monitoring the User Experience of the Cloud
In this podcast, find out the best metrics to build into service level agreements and learn how to track the quality of service end users actually experience.
Listen to ZDNet’s Phil Waineright talk to Vik Chaudhary, VP of Product Management and Corporate Development at Keynote Systems.
In this podcast, find out the best metrics to build into service level agreements and learn how to track the quality of service users of SaaS and cloud applications actually experience.
Q&A With Vik Chaudhary, VP of Product Management and Corporate Development, Keynote Systems
Phil Waineright: I’m glad to have you with us because your — Keynote — is a SaaS provider itself, but you also actually work with SaaS and cloud companies who make use of your services, don’t you?
Vik Chaudhary: That’s right. In fact we do both. We started out as a cloud company and a SaaS company, well before those words were even invented, back 14 years ago. And today, we work with about 2800 different companies all over the world; SaaS companies are among them.
Phil Waineright: Right. So okay. So this part of the business is serving traditional enterprise businesses but part — a growing part I suppose of the business — is serving the cloud vendor community of one type or another.
Vik Chaudhary: As it turns out, the cloud vendor community is — especially in the SaaS world — is growing to include businesses that are using SaaS vendors very effectively. And because businesses typically care about their online performance and customer experience, they happen to look to us to help moderate the conversation between them and the SaaS providers so we can assure that performance and reliability of their applications are really top-notch.
Phil Waineright: Right. Because downtime, I know, is a big worry when businesses are thinking about using cloud providers. And I think there’s sometimes a lot of uncertainty about what are the bottlenecks that people should be thinking about, that providers should be monitoring and looking after. What are the performance bottlenecks you typically see?
Vik Chaudhary: Performance bottlenecks in the cloud first of all start out with the same performance bottlenecks that most globally distributed web applications have. And those typically fall into three categories.
One is data center latency. How far the users are from your data centers can lead to significant delays in serving up that particular application’s content, especially for those users who are far away, or further away, from the data centers. So, then you’ve got something like connection speeds. So, whether you’re coming on a DSL line, you’re coming on a T-1 line, whether you’re on dial-up in some countries — that makes a difference. And then finally, there’s the application’s own construction, how it’s developed and the good components that are being used. All these three things cause performance problems.
When you’re a cloud provider though, these get exacerbated because you’re serving up these applications at a high scale. You’ve got hundreds of thousands of users. You’ve got users who are sitting in banks at certain geographies. You have users who are sitting in China. So all of these get exacerbated heavily.
Phil Waineright: So what should people really be tracking? Are there metrics that you recommend people look out for so that they can keep on top of these performance issues?
Vik Chaudhary: You know, Phil, a couple of years ago, I was at a conference with a number of SaaS CEOs and I in fact asked that question of a panel of CEOs, SaaS CEOs. And it is very interesting, almost all of them said the number one hard metric that we track is response time. And that really in fact surprised me, because that was two years ago.
But it turns out that SaaS companies and their CEOs care very much about the response time of their applications, because if the response time is slow, they know first of all it has a deleterious effect on the experience of their users. But also, there’s this notion of SLAs and chargebacks if the application is down. So response time, availability, are two of the most common hard metrics that people should be tracking when they’re looking at a SaaS provider.
Phil Waineright: And when we are looking at response time, are we looking at response time up to the edge of the provider’s infrastructure or should that be response time at the user — at the client’s interface?
Vik Chaudhary: Response times should always be done from the client interface, and that’s because of a number of different reasons. One is the client, as I mentioned earlier, could be far away from your data centers. So that might affect his or her performance that they actually see.
Phil Waineright: I think that’s very important, of course, because the users are using the application to get a business result. And if they’re kept sitting waiting before they see those results then that is going to make them feel dissatisfied, isn’t it? So I think — I would agree with measuring at that point. But of course, people talk about SLAs. How do you build that into an SLA to make sure that there’s a standard that both sides are signed up to?
Vik Chaudhary: The primary SLAs that are typically signed in the IT industry happen to really fall in the area of availability and response time. But those are just the starting point. Most SaaS providers and their customers should be looking at a couple of different ways to measure their service levels. Number one is, how does my business transaction actually perform? If I connect to a cloud application and I login, and then I go and pick up my list of invoices, and then I click on an invoice and finally open it up, that’s a business transaction. How is that performing? So that’s an example of a business-specific SLA that we recommend that users of SaaS applications put into their SLAs with the SaaS providers.
Phil Waineright: I think makes a lot of sense. But how do you build the instrumentation into your infrastructure, as a provider, to actually measure that kind of thing?
Vik Chaudhary: If you’re a provider, here’s how typically they do it today. SaaS providers will monitor the health and availability of their entire infrastructure. They might use software that really tests if the database is up, or an application server is up, or the entire infrastructure is up, and the network is working. If you want to measure how well you’re doing from your customer’s perspective, you actually need to start by conducting what they call a synthetic or an active test that completely replicates what the end user does. Like the example I gave you before. And be able to repeat that transaction from multiple locations all over the world at every minute or every two minutes of the day, so that you know continuously how well you’re performing from the point of view of your users.
Phil Waineright: And that is the sort of infrastructure that I guess is better provided from an outside provider like Keynote, because you’ve got the nodes around the world that you’re using to test a whole load of different providers. So you can provide that level of testing that can say that global user base is either getting or not getting the response times that they need.
Vik Chaudhary: That is very true. If you have a SaaS application that might be used by users all over the world, we certainly have about 2600 locations all over the world. We have PCs and mobile devices that are monitoring the end user experience. So that you can do exactly what your end users do from these locations and be able to measure the response time. But I think it’s also important, Phil, to think about the enterprise users who are in specific locations. You might have a bank with particular users in a certain geography. You want to monitor from those locations as well.
Phil Waineright: Right and that also is a possibility with this kind of monitoring infrastructure, is it?
Vik Chaudhary: It is. You can take Keynote’s infrastructure and our measurement computers and place them in locations, like at a bank branch office, and be able to measure from there itself.
Phil Waineright: And is this something that is an added value service. I mean, is it an extra cost for the cloud provider or is it something that people can use to actually grow their business, grow their revenues, whatever?
Vik Chaudhary: That’s an excellent question. So let’s talk about the cost aspect first. Clearly, if you want to monitor from thousands of locations all over the world — and the more locations you have, it’s going to be more costly. However, if you’re a cloud provider or a SaaS provider, you might want to do some very high volume monitoring from specific locations. For example, you might want to measure the latency between your data center on the West Coast and your data center in London. And you can do that very cost effectively by just placing a single box on one end and doing that kind of monitoring. So you can cut costs as opposed to think about it as increasing your costs.
Now when it comes to thinking about how do you get a return on investment or how do you increase your revenues, we have to start thinking about it in a couple of different ways. First of all, your end user’s experience doesn’t just happen on a PC browser anymore, it also happens on mobile devices. So you’ve got a whole market of users who want to use mobile devices to do quick transactions like, look up their contact information on a CRM application. If you can improve the experience of that group of users, you’re going to get loyal customers that you haven’t been reaching to earlier. That’s one way to increase your revenues and expand your footprint into an organization.
Phil Waineright: Right and the ability to take advantage of that mobile opportunity is obviously one that a lot of cloud providers are very aware of these days. So thank you very much.