Streaming on an Olympic Scale
iStream Planet's Bhavesh Upadhyaya talks about streaming the world's most-watched games
Every two years, people around the world gather around their screens to watch the best athletes on the planet compete for the gold. And with each biennial edition of the Games, more and more of those screens are something other than a television. Two years can be an eon in “Internet time,” and 2012 promises to be the most streamed, most social, most mobile Olympics ever. iStream Planet’s Vice President of Operations Bhavesh Upadhyaya has been on the front lines of streaming for the past two Olympics, and won an Emmy for his role in the 2008 Beijing games. He’s one of the few individuals who has a first-hand sense of the massive scale and complexity of live-streaming the scores of events and thousands of hours that make up the Olympics.Benchmark recently spoke with Bhavesh to learn about his experiences with the Olympics and his perspective on the upcoming London games.
Benchmark: You worked on streaming the Beijing Olympics. What was your role?
Bhavesh Upadhyaya: I was a consultant for NBC. I was brought in 14 months before the Olympics to help architect the video workflow. And that two-month project turned into a 14-month project and ended up with me being in Beijing for six weeks. It was absolutely a once in a lifetime experience to put something like that together.
Benchmark: That must have been really exciting. So NBC did the whole thing in-house? Is that how it worked?
Bhavesh Upadhyaya: Yes. I was there from the beginning when it was just a team of three people, all the way to when we were there in Beijing inside the broadcasting center delivering, at that time, what I think was the largest live video event ever in the history of the Internet.
Benchmark: Can you give us a sense of the scope of that?
Bhavesh Upadhyaya: At that time, we basically delivered – with the exception of the major sports – every single moment of the Olympics live online from Beijing to the entire audience of the United States. The major sports were recorded so that they could be watched on TV first and then, after a short time, they were available on-demand online after the fact.
But up until that point, there were thousands of hours of live content being delivered, plus more on-demand content of clips that were custom-created just for the Internet audience, or amazing clips that came from what was done on TV.
Benchmark: Since then you’ve joined iStream Planet. What role has iStream Planet played in the Olympic Games?
Bhavesh Upadhyaya: For the Vancouver Games, we delivered all of the live video. With Vancouver, we reached another threshold – it was the first HD video delivery. I think we had at the peak 27 simultaneous live streams going, delivered in 720p HD over the Internet. And we did the delivery not just for the United States this time for NBC, but also CTV and NRK in Norway. So all the Canadians, and the United States and Norway audiences saw the bits that were generated by us.
That was another wonderful experience. Part of the progression that started back then to what it is today was the increase in audience, the increase in video delivery bit rates, the fact that we're moving towards adaptive bit rate technologies and adding on different avenues to watch it – not just simply online, but also mobile devices and over the top as well.
Benchmark: Are you involved in the upcoming Olympics in London?
Bhavesh Upadhyaya: We are, but we’re not allowed to talk about what we’re doing with them right now. We’re under tight NDAs [non-disclosure agreements] right now.
Benchmark: But it’s safe to say that you will be on the scene and doing some things with the London Olympics?
Bhavesh Upadhyaya: The way things are moving now with video delivery and how it’s becoming more ubiquitous, we don’t have to be on the scene per se. But again, that is under tight NDA right now, so we really can’t talk too much about it. Hopefully you’ll see press releases in the future. I’m just a techie guy.
Benchmark: Perhaps then we could speak in general terms about the concept of dealing with an event like the Olympics, based on your experience. What’s different now, in the couple of years since the last Olympics in 2010? You mentioned HD video and mobile. What else has changed?
Bhavesh Upadhyaya: From the United States perspective – and I think this is something that's going to be potentially hitting the other countries as well, too – it's the authentication process to validate who is actually allowed to watch these streams. Do you have the right cable accounts, et cetera, to authorize you to view these live streams?
The other big thing is just scalability. From what we've seen, the utilization of online video is doubling – every year you see more and more devices that are capable of watching video. I think with the advent of the iPad and Android tablets, iPhones and Android phones, et cetera, now people are more willing to watch videos wherever they are. That's really driving a lot of the growth. It's no longer somebody stuck behind a computer or behind a television. People want to watch anywhere they go, and now the challenge is how to test and scale that side of the model, too.
You're no longer just simply validating that your software can support one delivery mechanism. Now you've got three or four, if not more, delivery mechanisms and you're adding onto that online advertising as well. It's just becoming so much more complex because people are trying to get to the point where what they watch online is as simple as turning on a television. And we'll get there.
Benchmark: How do you prepare to handle an event of this magnitude? How do you make sure that the streaming is going to work without a hitch?
Bhavesh Upadhyaya: It's about two aspects – two partnerships that you have to generally set up. One is creating a resilient infrastructure on the content creation end. And then the other part is getting the delivery part, the relationship with the CDNs, available so they can actually spread it out to the last mile. We want to make sure that we segregate the difference between content creation and content delivery.
The content creation part, we're doing our best job making sure that we're generating the best quality video for the lowest possible bit rate that we can do. And then once we've developed the video, how do we ensure that it's available to the audience the right way? And that's working with companies to help test and hammer not only the origin infrastructure that we've created, but also the CDN infrastructure to validate that it can support it as well.
With the Olympics especially, that is super critical, because you are going to be, up to a point, saturating lots of end nodes. You're going to get to the point where, potentially for some major cities, you might be saturating an ISP with all the video that's being delivered.
And then how do you ensure that the problems you're seeing when you're monitoring those live streams are not a general problem? Is this – the fact that you're seeing buffering, or the fact that you're seeing dropped frames or loss of connectivity – is that a systematic issue? Or is that a regional issue specific to an ISP or a specific town or what have you?
So monitoring has to happen both before – just to make sure that you're capable of handling the load – and then during, to make sure that you're able to triage any issues that are happening in real time.
Benchmark: And what happens before is load testing, essentially? Are you stressing the system out at certain levels to see how far you can go?
Bhavesh Upadhyaya: Absolutely. You can't go into a large event like this without having a comfort level to know, at least to a certain degree, what is your limit? At what point should you start to get nervous?
If you have not done that type of testing, your stress levels will peak right from the get-go, because you won't know where you're comfortable in getting to. In the past, we've done load testing to a degree where it's gone well beyond what we thought the potential audience was going to be.
So now when we're watching live streams and we're doing our monitoring, we are comfortable with a pretty large audience coming through, and knowing that down to the last mile – unless there's a specific town or a specific ISP where they've got issues – there will not be a systematic problem. It can handle X amount of load. That's a great comfort level to walk into an event with.
Benchmark: Is the load testing you do an ongoing process, or are you continuously ratcheting it up? Or do you look at an event like the Olympics or the Super Bowl and say well, we estimate it’s going to be X number of streams, X number of viewers, and then test to that specifically? Or both?
Bhavesh Upadhyaya: You picked the two events where there's no way to accurately figure out what the audience is going to be. Good lord, when we did the Super Bowl we thought the audience was going to be X, and we actually – this was Las Vegas, remember –put some friendly bets around it. No money traded hands, but we said to each other, 'Yeah, where do you think the audience is going to peak out?' And it was well beyond expectations.
I think NBC put out the press release that there were 2.1 million unique viewers during the Super Bowl. And that was quite exhilarating to be onsite when we were delivering the live stream. That came through with five different camera angles, and it was just incredible to work with that.
That one worked out OK. But there's no way to accurately figure out what that audience might be until you actually get to the event. So you have to take a look at past events where you think you might have had a similar audience, and then you go from there and say 'OK, let's load test to this amount.'
The work that we did together [with Keynote] was for the Xbox events for Microsoft. They're the guys who hired Keynote in order to manage the load test. Because this is something for them that's critical. This is the first time out, so you have to make sure that, especially in a pay per view model like what they're doing, you've got to guarantee that your video quality and your network infrastructure is going to support this audience coming through. Because the last thing that you want is people calling in saying, 'I don't like this stream, I want a refund.' That will kill the ability to continue to do online the right way.
Benchmark: To skip back to your Super Bowl example, how’d you load test it to that audience that went beyond your expectations?
Bhavesh Upadhyaya: No, we didn't go to the 'well beyond expectations.' But we had load tested up to a certain point, not to the level where we finally saw the audience. But with those events we worked not only with Microsoft and Keynote, we also had the CDN partner onsite as well to validate and ensure that they were not seeing any issues on their end.
So the testing that we did from there, when you have events that are this large, it's all hands on deck from every partner in your ecosystem. They made sure that they dotted every 'i,' crossed every 't,' did all the testing on their side, and ensured us that the proper capacity was there on the CDN side as well, to manage X amount of load.
And they did their testing internally. You go to our safe point, that we know we've tested to, and then if it gets beyond that, then we start sweating a little bit. But again, everything worked out OK.
Benchmark: Have you had any situations where it hasn’t worked out?
Bhavesh Upadhyaya: Not that I can talk about. There have been instances, not on live production events, but on test events when we’re putting systems together. And I think that’s when certain customers or certain upper management says, ‘Well, we don’t care about load testing, let’s just get the thing moving.’ And then you do it, and it doesn’t meet the expectations, and you go back and say, ‘Well, let’s go back to our recommendation and actually load test again. And let’s see what we can do and let’s take a look at all the parameters, and get things done the right way.’
Benchmark: So the lesson learned from the ones that you can’t talk about is, you really should be load testing?
Bhavesh Upadhyaya: You need to make sure that your infrastructure is comfortable to a certain level of delivery.
Benchmark: What are the important considerations for testing while the event is happening? The monitoring as it’s happening?
Bhavesh Upadhyaya: It's not just the aggregate of what's your audience, what's your peak and current, and what is the bandwidth being utilized across the board. If you really want a true idea of what's going on, you need to have good analytics from the end user, or from end-user type clients.
So either plugins into the player itself that are sending that information back that's getting aggregated. Or agents that are out there at these end locations sending back information, samples of information about what they're seeing. How is their reaction and behavior to the video stream? Are they getting all the bits at the right time in the right order, synchronized with video and audio? Are they getting all the statistical data, or whatever other metadata that's being supplied with the stream? Is that showing up the right way?
Are they having any issues at all picking up information and then delivering that back to a central monitoring dashboard that says, 'Looks like we're green across the board everywhere. Looks like we're yellow in, let's say Wichita, Kansas, because we've saturated one of the cable providers or DSL provider or what not, and they're starting to lose the ability to view streams properly.'
OK, well if it's a localized issue, we really can't do much with that. Or we see something is happening with a certain region, in which case then you know that OK, let's talk to the content distribution network provider, because if it's a full region and it's across multiple ISPs, there might be a bad edge node for the CDN. You've got to constantly monitor and get samples from a wide variety of things so you can actually triage and figure out, at what level is this issue occurring? Is it at the origin delivery level? Is it one of the peers, and how widespread is that peer? How much is that affecting you in terms of your audience size? Or is it all the way down to an ISP in a last-mile scenario?
Knowledge like that helps you ensure that your event is going OK. Or if you start seeing buffering increasing or what not, how can you start proactively adjusting things on your end in order to ensure that they're seeing the stream properly.
Benchmark: You mentioned the word ‘constantly.’ When you’re dealing with one of these events that’s three or four hours long, or 10 days worth of events that you’re monitoring, monitoring is pretty much continuous and nonstop throughout that?
Bhavesh Upadhyaya: Yes. Live events are hard business.
Benchmark: Just nerve wracking, it seems. To jump back to the London Olympics – they are being billed as the first ‘social Olympics.’ Does that play into streaming at all?
Bhavesh Upadhyaya: It absolutely will. The fact that you have all this video out there to watch is one thing. But the other side of it is, people have to find it. Especially when you have so much of it available, how do you actually know what's enticing to watch right now?
Most people aren't going to be looking at every single one of the channels, constantly going back and forth figuring out what's going to happen. They want to know if something exciting is happening at this event, and social is the way right now. It's the best way for people to communicate and say, 'There's something really cool going on in track and field right now.' Or, 'Oh my god! You've got to see this amazing shot put event.' Or 'Something awesome is happening inside rowing.'
People are going to be using social to talk and mention the video streams and you've got to make sure that you've got the links in place to allow people to share it properly and have them jump right into that video. The last thing you want to do is have people talk about the fact that there's something amazing happening, and they go to the site and there's no way to find it. You lose that entire live audience.
The live event means 'now.' The more people that can find out about it now and the more people that can get to that point, the better your audience is going to be –and that means you're going to be able to monetize it a lot better as well. You want to get that audience to watch what you want them to and not get frustrated and give up because they can't find it.
Benchmark: What you’re talking about is an organic referral and recommendation process, where event news is spread by individuals on the social networks.
Bhavesh Upadhyaya: Absolutely. And there’s a second part of it, which is, in order to do that the right way, the people who are designing the video players and the people who are designing the websites that have the links into players need to do a really good job of integrating the social plugins, to ensure that people from within the video stream can simply say, ‘I want to share this on Twitter, or Facebook or pin it on Pinterest.’ And make it easy for them to get to the point where they can tell all their friends, ‘Something really cool is happening here right now. Check it out, watch with me.’
Benchmark: Will any of the outlets be streaming directly into Facebook? If NBC has a page, would they actually put the streams on there so people can watch it within Facebook?
Bhavesh Upadhyaya: I doubt it. I don't know of any that are, but from my knowledge of the IOC rules of where you're allowed to put live streams, there are very strict rules as to where a stream can show up. So I don't think so. Unless someone has gotten special permission on their end, they're not allowed to put up a stream on something that is not approved by the IOC.
Those rules are very, very strict in terms of what you're allowed to show, how much advertising is on a page on a live Olympic stream, et cetera. The people who go beyond those rules are severely punished.
Benchmark: What is the impact of smartphones and tablets? Do you have a sense of how much of the audience has gone over to that side and how has that complicated things for streaming?
Bhavesh Upadhyaya: I've only got an anecdotal sense. We're not tracking the level of audience, the ratio between what's online versus mobile. We know certain numbers for certain events here and there, but for the most part, for us what it entails is a full different streaming infrastructure. The video delivery itself is not much of a problem. We can transmux on the edge or on our origins, so we can create the streams in any which way possible.
The trick for every single one of these different outlets is integration with the ad providers, because every single one has a different mechanism to do it. And that's where the complexity happens in terms of making sure that as people are watching, the appropriate ad tones are delivered, the appropriate ads are inserted in the right location and they're captured so that the owners of the video can actually get their money for the ad call that came through.
Benchmark: And is that a whole separate load testing and monitoring regimen?
Bhavesh Upadhyaya: It will be. The load testing and monitoring for sure is there for the live delivery. The ad insertion part of it is what adds the complexity on that end, but that doesn't necessarily have to be load tested. That's more of a unit test to make sure that the ad is available and, as the whole, on-demand video is delivered in the right way. That's actually the easiest part about it, making sure that the on-demand video is available, because it simply gets cached on the edges.
The monitoring and the load testing of the streams for mobile are critical as well. Apple has actually made it relatively simple from the iOS perspective. As long as it works on the iOS devices, it will work across the board.
The Android devices create more of a testing burden, because every single different version has a different video player inside it. They have different frame sizes. There's a little bit of everything. I think for people to try to test that individually on their own, they can go ahead and say either, 'I'm going to try and capture the widest possible audience and make sure that I test everything,' or they're going to say, 'You know what? I really can't afford to spend the time to take care of everything. I'm just going to test for a particular device class and a particular operating system level and that'll be it. Unfortunately I won't be able to reach the rest of my audience.'
So I think load testing, and verifying and testing like that is critical.
Benchmark: You said that verifying the streams as they’re happening is less of a problem?
Bhavesh Upadhyaya: The ad insertion part of it is less of a problem. You still have to validate the streams. Normally when we’re working on our side, we’ve got a single location which is testing it, but that’s not the same as having multiple locations around your entire population zone testing it and validating that it’s working there as well.
Benchmark: To wrap things up, since you’re really entrenched in the field, what do you see a couple of years down the road in terms of live streaming and online video in general? What trends do you think are impacting things and how do you think things will change for you?
Bhavesh Upadhyaya: We are going to see more and more live linear television out there. We’re going to be seeing more and more IP-capable devices that are out there and more avenues to deliver video. And the challenge is going to be to scale properly and validate that you can create one inbound stream and have it delivered to over-the-top devices, desktops, laptops, mobile devices, tablets, a little bit of everything. And how do you make sure that everything is happening the right way and continue to scale that as more and more people start developing and acquiring these devices?
Benchmark: It seems we’re just moving to more and more of an on-demand world in general.
Bhavesh Upadhyaya: I think we're actually moving to a more live world as well. We're seeing a lot more people saying, 'I want to do live linear television and make it available over the top.'
Have you seen the Verizon FIOS app for Xbox? Now for the first time, there's live television on the Xbox and people can watch it without having to get a cable box. I think that's the trend, that's where people are going to be starting to move to. It's getting more and more of that live video out there.
On-demand video is, in complexity, nothing compared to live. Live means you've created the bits and you've pushed them out and around the country or around the world within a couple of seconds, wherever the latency is at that point, and you've got to make it available for everybody. For on-demand, you're simply saying, 'Here's the file. Let me replicate it out wherever it's going to be.' And it doesn't matter if it's five or ten minutes late. You can't be five or ten minutes late for a live event.
People are starting to scale that more. You're going to see more announcements about people putting together live linear offerings for all of their content.
Benchmark: What are the challenges in doing that? It’s a huge undertaking ultimately to have it to the degree you’re talking about, so what are the barriers?
Bhavesh Upadhyaya: I'm not sure. Do we have another two hours? Those are the problems that we're trying to solve. It's really scaling and creating a dynamic infrastructure that will help people pull up live events and pull them down when they're done.
Honestly they're going to run out of infrastructure in their own locations if they're trying to do it. How do you start pushing the complexity out of your own organization and do it in the cloud where you can scale a little bit faster?
There's a whole untapped area right there to look forward to. There's a whole lot of problems right now with the demand for video that's showing up versus the ability to deliver the video. There's a wide gap growing and I think that's the problem that everyone is trying to solve right now. How do you get that video created, get the content created, make it live, make it available and send it out to everybody with your current infrastructure – and then expand your current infrastructure and beyond.
Benchmark: Beyond the infrastructure and the technology of the people who are creating it and pushing it out, it’s the Internet itself. Will it be able to handle it all?
Bhavesh Upadhyaya: It’s a race.
Benchmark: And an interesting one to watch. Thank you, Bhavesh, you’ve been real generous with your time. We appreciate it.
About Bhavesh Upadhyaya
Bhavesh M. Upadhyaya is iStream Planet’s vice president of operations, and leads teams responsible for acquisition, management, security and publishing of customers’ live and on-demand video assets. Under Bhavesh's leadership, his teams encoded and streamed every moment of the 2010 Vancouver Winter Olympics. His teams claim several Internet firsts, including the first IIS Smooth Streaming 3D event and first IIS Smooth Streaming 1080p web broadcast.
Bhavesh worked for 14 months with NBC on the 2008 Beijing Olympics as a video workflow architect and as the video player lead for NBCOlympics.com. The end result was a video player and website that received over 1.3 billion page views, 75 million video views and more than 10 million hours of video watched in 17 days. The NBC team was nominated for the George Wensel Technical Achievement Award, and Bhavesh won the 2008 Sports Emmy for “Outstanding New Approaches - Sports Event Coverage” for the Games of the XXIX Olympiad for NBCOlympics.com. Bhavesh has since spoken at the 2011 OTTCon, Digital Media Innovation Forum, and the Henry Stewart Conferences.