Visitor Monitoring Performance is a Competitive Advantage | Keynote
Webcast

Performance is a Competitive Advantage

About the Webcast

There is a digital transformation underway powered, in part, by the adoption of mobile. In this shifting landscape of increasingly fickle and distracted consumers, maintaining competitive advantage is more important than ever.

Competitive positioning is a relative art, defined differently for every business. But what's absolute is the science of digital customer experience and human behavior.

  • A 500 millisecond delay results in significant user frustration 
  • A 250 millisecond difference in user experience between competitors is all it takes to create advantage (or loss)

The reality is that faster pages drive higher engagement, but getting there takes comparative vigilance and constant feedback to development and digital teams.

So what signals should you watch to stay ahead? How can you create meaningful comparisons to other web and mobile experiences?

Watch the webcast for ideas you can use to out-maneuver your digital competitors with front-end performance intelligence.

Webcast Transcription

Yasmina Greco:         

Hello everyone.  Thank you for joining us today.  I’m Yasmina Greco.  I’m with O’Reilly Media, and I will be your host for today’s webcast.  We’d like to begin today’s event by saying a big thank you to Keynote for sponsoring our event, and we’d like to let you all know, having pioneered digital performance monitoring nearly 20 years ago, Keynote continues today as the leader in helping some of the world’s best known brands to make every digital interaction count. Keynote offers cloud-based testing, monitoring, and performance analytic solutions for web and mobile website and mobile apps.  They run the world’s largest, most accurate and reliable cloud testing, monitoring, and analytics network, collecting more than 700 million mobile and website measurements each day.  Thank you again, Keynote.  Folks, joining us from Keynote today is Aaron Rudger, and he’s going to talk to you about performance intelligence is a competitive advantage. 

Aaron is director of product marketing at Keynote, and he is a veteran in the web performance space, with over 15 years of industry experience.  As a senior product marketing manager, Aaron led product visions, strategy, and evangelization for Keynote’s industry leading web and mobile monitoring solutions.  Prior to Keynote systems, Aaron held key marketing positions at Right90, BMC Software, Remedy Corporation, and iPrint.com.  His cross-industry experience in CRM, IT management, and e-commerce, can be found in industry publications and technology blogs such as NBC News and Website Magazine.  Aaron is also a frequent speaker at industry events like O’Reilly’s Velocity Conference.  Folks, we’re really excited to have Aaron with us today to present this webcast for you all.  As we get the even started, I’d like to go over just a little housekeeping to help you get the most out of today’s webcast.  First, you’ll want to open your group chat widget, if you haven’t already done so.

This is where we can interact with each other during the event, and where you can submit your questions for Aaron.  We find that our audiences usually have a lot of good knowledge to share, so we encourage you all to chat freely during the event.  However, if you have questions for Aaron, please press SM with a capital letter Q.  That way we’ll know it’s for him, and we can make sure we see it for Q and A.  You can also open, move, and resize any of the other widgets.  If you would like to tweet from the Twitter widget today, you might need to give it permission to access your account.  The Twitter widget will automatically append the events hashtagged to your tweet so you don’t have to, and today, folks, our hashtag is velocityconf, all one word.  If you have any trouble with the event, please take a look at your help widget.  We do want to let you know that we are recording today’s webcast, and we will have the recording ready usually within 48 hours.  And folks at this time, I would like to turn the program over to Jason Yee, your velocity community manager, for a couple of announcements.  Hi Jason.

Jason Yee:                  

Hi Yasmina, thanks.  I just wanted to let everyone know that the Velocity Conference is nearly here, and it’s going to be next week from Wednesday through Friday at the Santa Clara Convention Center.  We’re going to have three amazing days of training and tutorials, keynotes, sessions, all based around web performance, operation distributed systems, dev ops, the things and the topics that we all know and love and are involved in much of our daily lives as we build performance websites.  Thanks to Keynote who’s sponsoring this webcast and also sponsoring the conference, you can receive 25 percent off all-access three-day and two-day passes by using the discount code keynote25.  We also have something fairly new this year, the Exhibit Hall Plus Pass, which gives you access to sponsored sessions, all the evening social and networking events, which we have many, and of course the exit hall itself, and you can get that pass for free, thanks to Keynote, using the code keynoteexpo. 

Next week, after the Velocity Conference, so two weeks actually, on June 10th, Jaz Humboldt, is going to be presenting on continuous delivery and how frequent automated deployments are critical to growing your business and your software’s reach.  You can learn more about our upcoming webcasts by visiting webcast.oreilly.com.  I’d love to see you all at the Velocity Conference, and our upcoming webcasts.  With all of that in mind, I’d love to turn it over to Aaron.

Aaron Rudger:           

Thank you Jason.  I appreciate that.  And thank you Yasmina for the wonderful introduction.  I’d just like to encourage everybody to absolutely consider if, at all possible, attending the Velocity Conference.  It’s a really great gathering of the entire community, dev ops community, web performance front end and back end, and the kind of content that you can experience there as well as the kind of technology that you can see, put your hands on, and really experience first-hand by virtue of the vendor presence as well, is truly unmatched in my opinion, having attended a number of the conferences.  So I definitely encourage you to continue to support the O’Reilly Conference organization and the community in general.  And that’s really cool that this year now the conference pass is being offered with that discount for free.  I didn’t even know that.  That’s super cool.  So good morning or good afternoon, perhaps even good evening, for those of you on the other side of the globe.

My name is Aaron, and I’m going to just kind of go through and share a bit of a story with you today.  It’s a story really anchored around four numbers, and as we go through the presentation, I’ll explain what these numbers mean and kind of how they relate to the overall story and the overall theme about really using performance as a competitive weapon and driving advantage for your company.  Because as performance enthusiasts, as technicians, engineers, developers, really at the end of the day, we’re all really supporting the goals and the objectives of the company for which we work.  And if we can’t do that in a way that’s meaningful and truly helps our company create a competitive advantage in the marketplace, then a lot of what we’re doing maybe is not as effective or as efficient as it really should be.  And we’re really, as a community, oriented around efficiencies and effectiveness and optimization, and hopefully some of the things that I’ll be talking about today will help you take a better appreciation for what the potential for achieving that can be relative to using performance in this context.

So let’s start with a few things first, in terms of the discussion about speed.  And speed, in the context of a lot of the content that the Velocity Organization or the Velocity Conference and community has been providing relative to what your apps and your websites need to do and how they need to perform, has really, I think, grown and matured, over especially the last two to three years.  And just to kind of put things back into a little bit of context, in the old days – air-quotes around “old days” – a lot of folks used to think that webpages needed to categorically, in absolute terms, load within three seconds, and mobile apps needed to be able to be launched within two seconds.  And I’ve been with Keynote for a long time, and I know that as a vendor, Keynote, we have also been very active in kind of perpetuating this notion of an absolute bar or benchmark for web-performance anchored around this number, three, or two.

And the reality is, is that speed and performance is really a function of a lot of things, and it can’t be measured in one single metric or in absolute terms.  And this is probably no shock to all of you on the phone today, right?  The reality is, is that your sites, your applications, are compositions of so many dynamic moving parts and pieces.  You’ve got functionality that includes search.  Should the search functionality perform within three seconds?  You’ve got functionality like user reviews or customer sat surveys, shopping carts, tracking pixels and security, there’s credit card validation, there’s analytics that help you understand the user behavior as your customers experience your sites and applications, and really the question is, within the context of the market that you serve – because again, we’re all about helping employees or consumers or partners, the end-user, experience and interact with the content and the digital services that we’re providing – that there are just simply different realities that exist, especially in different parts of the world.

And the question is, is that as you’re going about trying to determine how to get to some kind of a number, how to create a benchmark or a landmark for performance, the question is, really, at what cost do you attempt to achieve that goal?  What really is the cost, be it in hard dollars, or in maybe a little bit softer dollars: the amount of time and effort it takes to develop a different type of an approach to your front end experience.  Or integrate a different kind of a third-party service into your digital experience, or support your infrastructure with investments that help it become faster, either through content delivery networks or cloud services.  And at the end of the day, the question is, is that really helping you get ahead?  Are you really doing that in a way that is going to be, in relative terms, relative to your competition, actually moving the needle for you and helping you compete more effectively from a business context?

So back to the numbers.  That first number, 250, well the way that we look at it is, what we like to suggest looking at it is, is in terms of milliseconds, and 250 milliseconds being, perhaps, an alternative number to the three or the two second metric that we’ve all heard.  And the reasons why, are that 250 milliseconds now, really, is a measureable, demonstrable, supported number, in relative terms, that can help you achieve advantage.  So a couple of proof points here, supported in terms of the research from both Microsoft and the Google team, 250 milliseconds really can be, whether or not you’re slower or faster, a number that can really determine competitive advantage.  And so whether or not your page or your app is loading within maybe five seconds or eight seconds or maybe even only one and a half seconds, the delta between that and a relative comparison that is meaningful to your business, now is important in the context of this 250 millisecond benchmark. 

So just as a point of example, if we were to look at four different websites, four different home pages, let’s say, from the retail perspective, and we compared them, in this case here, none of them are achieving to that three seconds.  However, you can clearly see that there are different levels of advantage, perhaps, that some of these sites may have over others on the basis of how they relatively stack up.  And again, that three second rule, not necessarily the end of the story.  It may be one part of the story, and the 250 seconds is also another part of the story.  So let’s look at another one of these numbers; the number one.  Really the number one here is all about the notion of one time, looking at yourself, evaluating how you perform on a singular basis.  And really, that’s not particularly useful, right?  Because the reality is, as you’re looking to achieve advantage, and maybe achieve maybe that 250 millisecond advantage, is a one-time comparison enough to effectively do that and effectively accomplish that task?  I mean the reality is that your site is not frozen in time, right?  Your site is probably pretty dynamic.

And the other reality is, your competitor’s site is not frozen in time.  There are a lot of other services that are competing for the attention of your customers.  And at the end of the day, again, you’re all about helping to create an advantage which will translate into your customers feeling more engaged with your digital service and becoming more apt to convert, more apt to visit and read or consume your content, and more apt to maintain their loyalty to your brand.  So I thought I would ask a question of the audience here, just to kind of wake you guys up perhaps, as I’m trying to get myself going for the day out here in California.  How often does your site or app update? What is the frequency with which your team is performing updates?  And it may not necessarily be that you yourself are actually pushing those updates, right?  They’re probably in the audience today.  We have a number of folks that are practicing some pretty advanced discipline of continuous integration, continuous delivery, and whether or not you’re involved in the build automation process yourself, you’re part of the delivery chain, I’m sure.

And so looking at it from the big picture view, how does your company, how does your digital experience change, and what is the frequency of updates?  So if you could take just a moment to push one of these four buttons here, that shouldn’t be too difficult.  We’ll go ahead and submit that, and we’ll go ahead and take a look at the results here.  Okay, a lot of monthly updates, surprisingly, but daily updates as well.  And that’s good.  I mean, the reality is that the amount of update that occurs on the average or typical site is actually pretty high frequency, and it definitely exceeds just a pure monthly cadence.  What we’ve found is that in terms of constant improvement, having that notion of a comparison that is relative to what your competitors are doing, is very important.  And to do that in a way that you understand it more than once a year or more than once a quarter, because at a minimum folks are updating their sites on a monthly basis or even a daily basis.  In fact, research from L2, and I think we’ve got some stats here on the bottom from some other folks in terms of the amount of update frequency that typically occurs on a website, but research from L2 just this past January found that the average number of landing page updates – so now we’re not even talking about the core site itself, but just the landing page that you’re typically going to be using to help solicit and facilitate, in hopefully a frictionless way, interest and engagement from your broader audience involved through maybe search engine technologies, search engine marketing, search engine optimization, etc. 

But the average number of landing page updates per week was 2.7.  So just think about that.  Your competition is not only changing in terms of their homepage or the other core aspects of the site.  There’s a whole range of product offering, of price and promotion that’s occurring, but on that promotion side, there’s a lot of dynamicism involved as well, and it’s important to be able to understand and acknowledge the fact that things are continuously moving, and that the rate of change, and understanding that amount of change, is key to being able to understand how you can continuously improve.  So the next question to really ask is, well, what does make for a good comparison, right?  How do I get to apples to apples as opposed to apples to platypi?  I don’t know that that kind of a comparison is going to be particularly useful, let’s say, if you were in the apple business.  But the point is, is that simply doing a homepage to homepage comparison is probably not good enough.  You need to be a little bit more fine-grained, right?

In getting to a good comparison, and again, we’re not talking about managing to absolute numbers and absolute benchmarks but rather, relative comparisons and relative benchmarks.  You really need to start slicing and dicing how your digital assets are consumed by your customers, and how you can make a meaningful comparison to other experiences that are similar to yours that can help you understand how you’re going to gain advantage relative to those other options that your customers have.  And so these are just a few ways to start thinking about it.  And really, what we would suggest as a best practice is to create a matrix.  Look at your digital assets.  Look at how you’re going to market, and start understanding, how do those assets, from a categorization standpoint, align with other categories or other digital experiences that are available on the internet today, to your customers, as competitive alternatives?  One clear, easy way is by industry.  Are you in the media business?  Are you in the retail business?  Are you in the technology or travel industry, financial services, etc.?  That’s one point of comparison.  The next would be your market.  Geographically, where are you actually selling?  What are the aspects or expectations of customers in those geographies?  If your business is 100 percent local, then local considerations are critical.

But if you also have customers maybe in China or North America or Southern Europe, you need to understand how to compare yourself relative to options or alternatives, again, in those markets.  And that may involve understanding whether or not somebody, competitively, as an alternative, is taking advantage of certain technologies that may give them a certain advantage within those local marketplaces.  These last two, I think, are the most important ones that a lot of people forget.  And when you’re trying to start a problem of understanding how to create competitive advantage and looking at performance as a way of really leveraging that, so that you can create better business outcomes, understanding and creating kind of apples to apples comparisons on the basis of functionality and your page construction or your application design, is critical.  No one is going to be able to look underneath the cover, so to speak, at the core site architecture and do comparisons in that way, but certainly by looking out and exploring the world around you, you can start beginning to understand and put into nice boxes, for comparative purposes, certain key or core pieces of functionality, and align those with your business strategy and your business goals, so that once you can be able to start benchmarking that you can have conversations with the business.

And I’m going to talk a little bit about that a little bit later here.  But think about functionality and your design and your page construction in particular.  Let’s look at an informative example to help kind of drive this home a little bit.  So here, we’ve got two pages.  These are homepages, so that’s the first point of comparison, right?  We’re now looking at the homepage experience, which is going to be different than maybe a core page experience inside the website, and how you take a user through a journey to accomplish some task, or accomplish some business transaction.  And so just starting here at the top level of, like, a homepage, two very very close competitors, FedEx and UPS.  In the United States, these are delivery service providers.  They provide shipments, and everybody is probably familiar with UPS and FedEx.  They’re worldwide grants.  Their experiences are, in some ways, very similar.  Again, they’re trying to accomplish the same goals in terms of getting people to ship packages with them.

But the way in which they’re accomplishing those goals are actually slightly different when you start to peel back the layers on, underneath the covers here, of what experience they’re providing to: their end-users or their customers.  And if you notice, there’s a couple of things that kind of stick out as big differences in terms of the way that these sites, the homepages are composed.  One is that you’ll notice with FedEx, there are a lot of more open form fields that are available that accomplish a lot of functionality that the UPS homepage also accomplishes, but in a slightly different way.  And so that’s one point of comparison.  The other point of comparison is that the overall design and the style that’s used, in terms of imagery and font usage, etc., is also a bit different.  But then there are some other differences that you can’t necessarily see, and these align to strategic goals or business goals and objectives which, clearly, both businesses are looking at a little bit differently than the other.

So one, if we start kind of peeling back the onion here and looking underneath the surface, at the FedEx site, the overall user experience time here is over four seconds.  And this is one test, but on average it’s averaging around four seconds.  So again, it’s above the three second mark, which maybe is not a good thing.  But when we look at other aspects or other dimensions of that total user experience, when the page first starts to draw and when it becomes interactive, for example, there are other benchmarks that start to become available here that we can start to look at, right?  For example, the interactive page time is very very quick.  Now, relative to the UPS page, the user experience time is almost half.  It’s greater than half of the FedEx total user experience time for the page.  So does that mean that FedEx is twice as competitive?  Are they gaining two times the advantage of FedEx, right?  Well, really the answer is no.  There are other differences at play here, and one of the most important differences that you can’t see unless you start actually drilling even deeper into this page, is that the FedEx experience is supported by a number of additional services.

So there are additional marketing services, there are additional tracking services, these are all kind of third party or web services that are included in the WebEx page, and it’s actually delivered, if you looked at and you counted the number of IP connections that are open that represent the different services that are being integrated into that page experience on the FedEx site, there’s 10 of them.  On the UPS site, there’s only two.  There are five times more IP connections on the FedEx site as there are on the UPS site, and the reason that that exists is that the FedEx site is actually creating more personalization.  They’re leveraging more third party services to create a more personalized experience.  And if I had visited that page a second time and I had used some of the personalization available to me through the FedEx site, then the experience that I would receive would be different, and it would be richer.  And so the trade-off here is about experience.  And for FedEx, strategically that works for them, and they feel that they’re in a good place as a result of that.

The reality is, is that those decisions are made on an informed basis.  They understand what the trade-off is, and that’s really the key, is getting to an understanding of what those trade-offs involve.  And so when you start to look at the differences in terms of these comparisons that you might have, you can’t look at the difference at a very high level view without beginning to understand what is underneath the covers to make, really, a call, from a business standpoint, as to whether or not those differences are useful, they’re intentional, or if they’re unintentional and may be actually impairing your ability to perform more competitively, then you need that visibility in order to make those trade-offs.  Right?  And so adding visibility into the page construction on a comparative basis, across multiple points of comparison – In this case, the graphics that you’re seeing here are four points comparison going back to that original slide where we had those four retailers – looking at the actual components: the site structure, the core CSS, the amount of java-script, the third party components.

The third party components by various categories: advertising, tracking pixels, survey instruments, user review widgets, social widgets, for example.  Having that level of visibility now allows you to be able to understand whether or not the trade-offs that you’re making are actually contributing, again, intentionally, to achieve the business roles that you’re seeking to attain.  And if you’re doing any kind of AB testing to understand how performance is impacting the achievement of those business roles or outcomes, things like, again, user engagement, abandonment versus conversion, for example.  Understanding where the added value you may receive, in terms of functionality from your website, is being compensated for by the difference that you might see in terms of performance, and its impact on the user outcome or behavior.  You may see that actually your user behavior or your user outcomes, in terms of abandonment or conversion, continue to increase, even as your page weight increases, right?

And that’s because you’re making intentional decisions around certain functionality.  The key is to understand that continuously, again, in a constant view; daily, weekly.  Really daily.  Because the reality is that as your site changes, all of these decisions and all of these numbers are going to change, and more importantly your competitive comparisons are also going to be changing as well.  The last two numbers in the story here, 1/3 and 2/3.  These relate to the kind of components that you’re integrating from your third party services.  And again, third party components and services add value to your site.  They add richness and texture to the user experience.  And the reality is that on average, and we here at Keynote have looked across all of the different pages that we monitor and all of the different sites and experiences that we monitor, and find similar findings to the ones that other analysts and other vendors in the space have found as well, which is that, on average, more than 30 percent of site content today is actually driven, or coming from, these third-party services that are being integrated into the browser experience; the front end browser experience.

And we’re not talking about back-end services, right?  These are third party services that are integrated in the front end.  However, 2/3 of the impact from a performance standpoint, in terms of load time, is contributed by these third party services.  So it’s an inverse relationship.  And when we looked at, recently Keynote sponsored a survey with force to research, asking a number of both business decision makers as well as IT folks, operations folks, front-end folks as well.  We asked them about what has been their experience over the past 12 months with regard to performance issues, and what were the contributing factors?  25 percent of the respondents indicated that third-party services were actually a major contributor to those performance issues that they experienced, and I actually feel like that number is under-stated.  That in fact, probably more often than not, the end-user experience, your customer’s experience, is impacted negatively by misbehaving third-party contributors or third-party providers.

And yet the challenge is that it’s often difficult to really measure and understand that.  And the point here is that, again, as you’re looking at comparing your page construction, and you’re considering and evaluating third-party services as one of the classification areas or contributors to that comparison, and how that can impact your competitive positioning, right?  The question is, how is it contributing to you?  So you yourself have third party services and are probably integrating them today into your sites, and one of the questions that I would ask is how are you monitoring and managing those providers?  Do you have service level agreements in place?  And so I thought I’d ask again, another polling question here.  If you could, take a moment and let us know where you stand today, with regard to the management of your third-party services.  So again, these are things like social media widgets and other points of integration, security integrations, tracking pixels, of course your analytics providers that you might be integrating as java script in your applications.

What are the relationships?  Customer satisfaction, survey instruments, right?  What are the relationships that you have with those providers, and how are you actually managing them?  Is it active, or maybe a little bit passive?  So go ahead and submit.  Simple question.  Let’s take a look.  And not a whole lot of people, and this doesn’t surprise me.  What’s interesting about this is that I think it’s not for a lack of discipline or desire.  The real issue, the real opportunity, is that it’s difficult to know what that SLA should be.  When there’s an issue, how do you create, essentially, again, a benchmark to be able to hold that provider kind of accountable too, or at least to begin the discussion with your business partner around which third party to select, because there are multiple options that typically can exist?  It’s not a universe of one.  It’s typically a universe of many providers and many vendors.  But the question is that in making the selection purely on the basis of functionality and not considering what the performance implication is, and not having, again, an expectation and a sense of agreement around – before integrating them into your digital experience – what will be the contributing impact to that from a performance standpoint.

But also, as you move forward with that provider; how that provider is continuing to support your goals in terms of managing to an expectation or an agreement that you might have with them in terms of performance.  So in terms of alternatives, do you have alternatives and can you make meaningful decisions about them?  And so if you have, let’s say, a view of kind of a range or a universe of multiple third-party providers, and kind of an understanding of, on average, what their performance was, and could easily determine that, then you could have maybe more of these meaningful conversations around attributing and establishing a service level agreement of some type, right?  Or at a minimum, again, a frame of reference that is rooted in performance around how to choose, perhaps, a superior alternative to a provider that you might be using or integrating today into your service.  And so this kind of a view could be very useful, and I would encourage everybody to use the kind of tools that you use today in order to really understand averages over time, or medians, or whatever your metric of choice is going to be; a percentile of some type, whatever.  For user experience, what the average is for those third-party providers.

And then set that as a bar within your performance budget.  Ah, performance budget.  I just said performance budget.  Let’s talk about that, because I think as we wrap up here, that’s another concept that you may be familiar with, and you may be actively using today, but I just want to kind of hammer home a couple of points in regard to that.  The concept of a performance budget is fairly simple.  When you establish a sense of what your design and the user experience is going to be, what is going to be your customer experience in terms of your digital asset and how you’re going to create that digital experience for your customer, are you using the notion of a performance budget to help guide your decisions in terms of both the development, the design to development, and the delivery of that experience.  And the concept of the performance budget, as I said, is fairly simple.  You establish a number, a goal, and just like in the financial world, that goal becomes your budget.

That’s kind of your digital currency, if you will.  And the way that you spend that is by beginning the process of creating that experience and constructing that experience from a page component standpoint.  And you can think of every component that you’re adding into your page and the way in which you are creating your experience as, essentially, kind of like bills, right?  Bills that you have to pay.  Now some of those bills, you’ve got to pay no matter what.  Your CSS you have to pay, just like a utility.  It’s like the lights are on type of a thing, right?  Of course your base page.  That’s, absolutely, that’s like your mortgage.  You’ve got to pay that.  There are certain network costs that, as well, that, again, like the mortgage, you have to pay.  You don’t really have a whole lot of control over that.  But there are some disposable income items, right?  There are some disposable expenses, perhaps, that maybe you have a little bit more control over and a little more discretion.  Ah, that’s the right word, discretionary as opposed to disposable.  And so I think one of the keys is understanding what part of your performance budget is a little bit more discretionary than others, and how are you spending that digital currency?

So one, thing about creating your performance budget, again, informed by the comparisons – Those comparisons to your consumers, your customers, your employees, other alternatives, in terms of other digital experiences.  Use that to inform and establish your budget.  Make sure that you’re continuously monitoring, you’re continuously understanding how that budget is changing.  It’s kind of like income.  Sometimes income changes, and a lot of times it’s not due to any control that you might have over the amount of gross income that you are receiving, but rather, as perhaps like a form of a tax, that changes arbitrarily, your competitors are changing as well.  And because of that, they’re kind of changing your net income on a regular and recurring basis.  And so using that as a way to inform your budget, I think, is really important and a really good practice.  And then, again, managing how you spend that digital currency in terms of making choices, informed choices, about the kind of discretionary components that you may be able to adjust within the total user experience that you’re delivering, is a great way to make sure that you maintain a continuous advantage relative to your competitors.

And again, 250 seconds – I’m sorry, 250 milliseconds is the bar that really can be very useful in helping you understand where to start making investments in terms of performance, and how you get return on that.  So don’t make arbitrary assessments or arbitrary evaluations in terms of the establishment of your performance budget.  Get informed, based on your competitors, and the other kind of functional alternative that may exist, that your customers or your consumers may experience.  Make sure that you understand where you sit or where you stack relative to those alternatives, and then make sure that especially in the area of kind of that discretionary income, where you have, perhaps, a lot of the component of the performance budget being spent, again, sometimes as often or as much as 2/3 of your performance budget being spent on third-party providers.  Make sure that you’re understanding what their contribution is, and hold them accountable for maintaining that budget, so that you don’t have unpleasant surprises that essentially compromise your total customer experience.

So that’s kind of the story of the four numbers. Hopefully that was fairly useful for you.  I tried to make this kind of fun and a little bit high level, and I’m welcome to take questions, if anybody has them in the audience, and Yasmina, hopefully you can help me to parse some of those questions as we go through.

Yasmina Greco:         

Let’s do it, Aaron.  Great.  All right, folks.  You heard, we are at the Q and A portion now.  If you do have a question for Aaron and what he’s been talking to you about and showing you on the screen, open that group chat widget you have, type it in, and send it in, and we’ll take as many as we have time for.  We do have one question in from Mark, and Mark asks, “How would you deal with competitors blocking or slowing the traffic for your monitoring of their site?  Example: I work for XYZ Corp. monitoring and my competitor, ABC Corp., but they put a throttle or blocking on my source address because they know I’m monitoring them.”

Aaron Rudger:           

That’s a great question.  So I guess there are usually a lot of – There are numerous ways that you can kind of get intelligent about your competitive posture.  And so yes, there are practices that can occur where, depending on the tool that you choose; you may not necessarily be getting the full story, so to speak.  And so I think, like with anything, regardless of whether or not you were trying to understand how your side is performing, or how your users, your customers, your consumers are experiencing your site, you probably want to instrument in a number of different ways, and essentially triangulate.  I know we don’t have the luxury of being able to invest in multiple tools, let’s say, that are redundant, and I would not recommend that.  But there are definitely ways that you can triangulate from multiple points of input, in terms of data, to give to some kind of a trustworthy metric, or a trustworthy number.  And so certainly in terms of doing some kind of competitive analysis of a website or an application, there may be monitoring tools and monitoring technologies like Keynote which can be gained, so to speak.

So that’s why it’s probably good to also include, as a point of comparison, a data point from some other source, and there are a number of free sources that are available, webpage tests, right?  Would be of course one.  And I would encourage there to be a kind of a triangulative approach, if you were, to understanding what that outside-in view looks like.  And it’s also important on that note, regardless of the tool that you choose, to also understand what is that experience from the standpoint of a desktop user as well as a mobile user.  Mobile, of course, is becoming more important now than ever.  So, using tools, retesting tools that can help you to understand that experience, the mobile experience.  Keynote has tools that will basically help you to kind of benchmark that right from your desktop, so you’re not necessarily constrained to whatever gaming may be happening from the standpoint of a particular vendor.  But then I would also encourage talking to various vendors if you think that they are being gamed to understand whether or not they can address those issues or those concerns.

Most times we find, certainly at Keynote, that usually that kind of behavior is not necessarily beneficial for either side of kind of the story there.  Both the target, the customer, and so on.  Because certainly from the Keynote standpoint, we find that a lot of the companies that we’re monitoring today are actually customers, and so it’s not useful for them to game, essentially, themselves, and to create an experience that’s not going to be representative, because they’re really trying to understand the end-user experience as well.  So hopefully that answers that question there.

Yasmina Greco:         

Thank you so much, Aaron.  Folks, do you have any other questions for Aaron while he’s still with us please?  Open that group chat, type it in, and send it in, and Aaron, while we give folks just another moment there anything perhaps additional that you would like to leave the audience with, or a couple of comments?

Jason Yee:                  

Yeah, I think that in terms of, again, going back to the notion of the performance budget concept, really just trying to keep things simple and to think of things in terms of the way that maybe you might manage your own personal mint.com type of account, right?  And creating simple categories that can help you understand, again, where you’re spending that digital currency.  And I would definitely encourage folks to think about that, but to start first by really creating that matrix of what are the meaningful comparisons that you need to consider before even beginning to do any measurements?  And again, looking at things not only from the standpoint of averages, but other methods of doing the kind of calculation or analysis that allow you to understand relative to what you’re trying to achieve, in terms of your business goals, maybe a median or a percentile are more useful, especially based on kind of the variables that might occur, let’s say, within certain geographies.  Or based on the kind of services that you’re delivering.

I would just definitely encourage folks to think about all of those different dimensions. 

Yasmina Greco:         

Great.  Thank you so much, Aaron.  I’m not seeing any additional questions so with that, folks, we’re going to say a really big thank you to Aaron for spending his time with us today, and for presenting a really great webcast and for sharing all his knowledge and expertise.  Oh, and just as I say that, we do have one question that comes in.  Okay, this one is from Ronald, and Ronald is asking, “How do you measure customer frustration, which I often experience with a website?”

Aaron Rudger:           

Wow, that is a fabulous question.  So interestingly, I meant to kind of acknowledge this, but I think about two years ago Tammy Everts from Radware did some really great seminal research on the impact of performance on just that, user frustration, user intention, and overall kind of user emotion.  And she found that, or the study found, I should say, that as little as 500 milliseconds of slowness or latency can actually contribute to perceivable, like statistically perceivable and statistically valid difference in terms of user frustration.  And so of course this was a user experience kind of study so these are methods of measurement that involve actual panels with real human beings, right?  And so, not everybody can afford to assemble a panel of real human beings in a kind of a research setting to do that kind of analyses, especially for their own website.  And so the way in which you kind of get around that, the proxy, is to look at the real usage of your website and to be able to understand, based on the real usage of that website, how the customers that you’re providing your digital services for, how they interact with and experience the site in terms of the kind of journeys that they take throughout it.

And so every journey essentially has a beginning and an end.  Now sometimes that end can be a successful end from a business standpoint.  Hopefully that means it’s going to translate to revenue for you, or it’s going to help reduce your costs relative to maybe some other channel that you’re serving or providing service too.  Also, though, it can have a negative end-point, which is maybe an unsuccessful completion of the journey; an abandonment, right?  Sometimes it can be also expressed in terms of balance.  And so the way in which to really answer that question is to analyze your customer journeys in terms of the real usage of your site relative to the performance that they experience as they go on that journey.  And ideally, you do that again in an AB test way, where some population of your users experience a site that maybe burns with a little bit more latency, let’s say, versus another part of your population, and hopefully the majority of your population, that experiences the site in its most optimal way.  And then you compare the relative difference, in terms of the actual user behavior.

Did customers, as they go through those journeys, either convert at a higher rate or a lower rate or abandon at a higher rate or a lower rate?  What was the material impact on the behavior of users as they experienced the site?  Because frustration is measureable in that behavior.  And so when I’m frustrated, I tend to bail.  And so I think that’s probably the best answer.  You look at it from a controlled, comparative testing experiment, and you see, where does that frustration level happen?  What kind of cost, in terms of performance, exceeds a threshold that becomes really material to your business?  So hopefully that’s useful.

Yasmina Greco:         

Thank you so much, Aaron.  I don’t see any additional questions.  Folks, we’ll give you a couple of moments here.  If you do have anything additional, just type it in your group chat and send it in.  I do hope we see many of you at velocity next week, our conference, and we pushed out some details to you all in your group chat there.  Also the Keynote discount that can save you 25 percent, so do take advantage of that.  All right, and with that, folks, we’re going to say a really big thank you to you, Aaron, for spending your time with us today and for delivering a really great webcast.  Thank you. 

Jason Yee:                  

Thank you.

Yasmina Greco:         

Folks that attended our webcast today we thank you for attending and hope you’ve benefitted from it, and also a big thank you to those of you who got up early or stayed up late to attend our event, as we had people joining today from all over the world, and we really do appreciate your time.  Folks, as we close out, we do want to say thank you to Keynote for sponsoring our event today, and let you all know that Keynote has pioneered digital performance monitoring nearly 20 years ago, and Keynote continues today as the leader in helping some of the world’s best known brands to make every digital interaction count.  Keynote offers cloud-based testing, monitoring, and performance analytic solutions for web and mobile websites and mobile apps.  They run the world’s largest, most accurate, and reliable cloud testing, monitoring, and analytics network, collecting more than 700 million mobile and website performance measurements each day.  Thank you again, Keynote.  Folks, this will conclude our webcast today.  Goodbye, everyone.

Duration: 57 minutes

Back to Top