Visitor Monitoring Velocity 2014 - Ben Rushlo | Keynote
Video

Interview with Ben Rushlo at Velocity 2014 in Santa Clara

Video Transcription

Mike Hendrickson:            

Hi, this is Mike Hendrickson from Velocity 2014 in Santa Clara. I’m here with Ben Rushlo. Ben, how you doing?

Ben Rushlo:               

Good. It’s great to be at Velocity.

Mike Hendrickson:            

So you’re the VP of analytics at Keynote, and is analytics in your role internal-facing or external-facing, or is it a combination?

Ben Rushlo:               

It is a combination, but I think our vision is to sort of say we have all this great data. We provide customers lots of data. And traditionally, Keynote has been about alerting and alarming back in the history, but yet we don’t really leverage that data to provide context, to provide sort of metadata to make our data smarter for customers.

So part of our role in the analytics division at least, is to take our data and sort of put it on an analytics platform and make it smarter, better for our customers. So we’re not just giving them alarms, but we’re actually telling them what to fix, how to fix it, predict it. Things like that.

Mike Hendrickson:            

So you also do a health report, right?

Ben Rushlo:               

Yes, one of the baby steps in this area is that we said okay, we have all this data, we know what a healthy site looks like, why not sort of apply that rubric to all of the data points. So this health report is you can take a Keynote measurement or on your ongoing measurements and you get a score that says, you know, I got a C and then you can drill in and see, well, why is that?

Maybe I have too many Java scripts or I haven’t optimized my renderer, or whatever, but it allows even a non-technical person to kind of quickly understand how do I compare to best-in-class, and not only that, but how can I improve?

Mike Hendrickson:            

So that’s something like performance in context? Are you able to compare? So can you talk a little bit about performance in context?

Ben Rushlo:               

Yeah, so I talked this morning about it. I wish I had more time in the main session but what I think we find in this data-dense world is that people don’t know what to do with that data, whether it’s Keynote data or its traditional homage or Google analytics, web trends data, or systems data. We just had these data buckets, and it’s like, what do I do with it? Context for me means that we provide the ability – or we need to provide the ability for customers to say is this normal, what’s normal, and then how do I improve?

So it’s sort of taking this data, visualizing it so I see the right bits of data, and then from that, driving action. So that’s our vision and we’re doing that in sort of custom work, we’re doing that a little bit in the product like in the health score, but I think you’ll see Keynote do a lot of interesting things in the next 12 months around that.

Mike Hendrickson:            

So it seems like maybe there’s a couple disciplines coming together here where we have an event, called Strata, which is our big data and analytics is all part of that, and now we have grabops and performance here, and it seems like those two are kind of blending together more and more as you look at it, including companies like Aerospike that does kind of high-speed data and that sort of thing. So it seems like that’s kind of where you are sitting.

Ben Rushlo:               

Well, it’s where we want to go. So when people say big data analytics, I can’t tell you how many analysts I’ve talked to that are like, where do you fit into that, because it can go the gamut from system data to real-time stream data to BI tools, but for us, yeah, what it is means is that we have this big data platform right now. One of the largest synthetic measurement vendors in the world, and we have millions and millions of data pinpoints are being collected. We should leverage that and we could do things with analytics to make that data smarter.

So one other example that we’re thinking about is imagine that I am measuring my site in China, and I’m seeing all sorts of issues with, let’s say, Facebook performance, things like that. And what would be interesting is are other customers experiencing that. Are there other metadata that I can look at that say my problem is not unique? It’s actually happening with everybody in China. And we have that data now. So that’s not even a super-complex analytical problem where you’re mashing up data but that’s a very valuable business case.

Mike Hendrickson:

So does your data show the importance of mobile and addressing all the devices and all the different types of devices, and what a screen is and how it performs?

Ben Rushlo:               

Yeah. Mobile is extremely complicated. We’re even having to educate our customers because mobile has so many more variables than we ever had in the desktop world. And businesspeople especially, and we work a lot within my team with business owners, they don’t know how to wrap their heads around the fact that you have cell tower variability, you have cell signal variability, you have phone capability and variability, so it is a more challenging place but, you know, Keynote is really leading the way with real devices where we can actually do measurement on real devices and understand sort of the profile of that device.

Mike Hendrickson:            

So responsive design is actually a topic that’s exploded in the past couple years. Is it all that it should be, or is there a little smoke behind it?

Ben Rushlo:               

From a performance person’s perspective, it’s sort of a misnomer. I know the responsive in the name responsive design doesn’t mean performance. It means that the site is adapting to or responding to the device, but we find that it really is sort of embarrassingly true that a lot of responsive sites are just very slow across one of the screens. So to do a responsive site well that works across desktop, tablet, and mobile is very, very challenging.

As we work with a large customer that was doing some Super Bowl advertising and building a site for that, and they outsourced it to an agency. They said, we know responsive. So they built the site, and luckily, thankfully I could test it a few weeks before the Super Bowl, and it was unusable in the mobile context. They had images that were ridiculously large. Just very, very bad. In theory, it was responsive in the sense that it worked across three screens but it was done in a very poor way, which made it unusable. And we see that more and more with sites. People sort of dipping their feet into that area.

Mike Hendrickson:            

So did that customer not use testing to figure this out before?

Ben Rushlo:               

Well, they used us but it was such a tight timeline that we didn’t get to them until two weeks before, and believe it or not, they were able to make enough changes to pull off a very big Super Bowl campaign, but if they hadn’t used a company like Keynote, they wouldn’t have caught that and they would’ve launched and their site would’ve not worked on two of the three devices or three screens that they were using.

Mike Hendrickson:            

So what do you see in the future for Keynote and also the mixing of data and performance?

Ben Rushlo:               

In terms of what other terms of data, data mashup, and –

Mike Hendrickson:            

Yeah, where do you see data’s influence on the performance field and where do you see the performance influence on the data field?

Ben Rushlo:               

Yeah, I think for us, again, it’s about the performance and context piece because if you can start to collect things like user behavior data, you can start to collect system data, log data, and you can put it into a system that does smart correlation, causation (if that’s possible), at least correlation metrics and then you can start to really expose value to both the CIO and the CMO and that’s really our vision. We don’t necessarily just want to be talking to techie people.

We are techie, but we want to bring performance to business owner where the budget is and where the focus is and where the funding is, and help them understand why it matters. So the only way they’re gonna understand it is if you start to put things like if your site is slow, it’s not gonna convert, right?

People aren’t gonna convert. Or their user engagement is such that when the site is slow, people bounce. And you can only do that by putting data sources together, and starting to show this relates to that, right? And it’s a challenge because the natural question is does it really relate, is it something else, and so you have to have a lot of rigor around that as well, which I think will be challenging.

Mike Hendrickson:            

So data science can actually help the performance community by actually –

Ben Rushlo:               

I want to hire a data scientist, so that’s – we need to have that because you have to have the statistical models, the machine learning if eventually you’re gonna do predictive, right? To support the pretty graphs and charts. I mean, the businessperson wants a pretty graph and chart that say improve performance and improve conversion, but behind that, you have to have the rigor that actually is statistically supportable, I guess.

Mike Hendrickson:            

How about when we take data science and high performance into the machine world, the physical world, you know, the new M to M, is that gonna change anything or is it just another device that’s producing data?

Ben Rushlo:               

So in terms of the internet of things?

Mike Hendrickson:            

Exactly.

Ben Rushlo:               

I think what it’s gonna change is that we think about synthetic data, right, where we started which is very limited. You’re taking measurements from maybe hundreds of locations and that data volume is relatively small. And then we have real user data, right, which is what a lot of the vendors are doing, Keynote as well. And then you think about internet of things, which is like exponentially larger than any of that data set.

So I think it’s gonna just make things much more complex. And it will require people who really know what they’re doing in terms of big data architecture and data science and digitalization. And then I think the other big challenge is people want not only batch data, but real-time data. And so that’s layered on top of all of this, which is okay, it’s great if you can batch that up, but I really want to make decisions in real time about my data. And to have all the data sources coming together, not in a data warehouse, but in a real-time stream, super challenging.

Mike Hendrickson:            

And react.

Ben Rushlo:               

And react, right. Super challenging. So I think it’s an interesting time to be doing what we’re doing, but it’s challenging.

Mike Hendrickson:            

Absolutely. Ben, thank you. And thank you for speaking here today.

Ben Rushlo:               

Yeah, thank you. It was great.

Duration: 10 minutes

Back to Top