Continuous Mobile Testing Using Jenkins: A How To Guide
Joe Lewis, Solutions Consultant:
Welcome. Today’s presentation will help you understand what the mobile boom means for today’s developers and testers, how to test at the speed of mobile, who uses Jenkins, how Jenkins supports mobile testing, and how to test your mobile app or website on real mobile devices.
So what’s going on in the mobile market today? What’s this mobile app imperative that we’re talking about? Clearly, users are demanding access via mobile. For example, there were one billion app downloads in 2013 according to a survey from Gartner. And it was predicted there will be $77 billion in anticipated revenue generated through mobile apps by 2017, also according to a survey from Gartner.
In the retail sector, mobile has become the number one technology in 2014. And in financial services, two out of three banks surveyed predict that 100 percent of their customers will be using their mobile services by 2017. Whether we’re talking about consumer apps or internal business applications, users want access on demand via their smart phones and tablets. And this is easily seen in the increase of traffic in sales in recent years.
Users have become less accepting of poor quality in their mobile experience. They may have made allowances in the past when the technology was new, but now they expect their mobile experiences to be designed for mobile and perform well for mobile, and not just accommodating to mobile or good enough for mobile. This lack of quality can impact sales and perception. With so much at stake, in many cases one brand, testing these apps has become more than a nice-to-have. It’s become imperative in providing quality mobile apps to your users.
When developing a mobile app, you run into many headaches. You need it to run on the major OSs, on as many devices as possible, and you don’t have time to learn new technologies and you need to iterate quickly. Testing for mobile has become a moving target. Take for example the recent Google announcements. Not only do devices change but operating systems change, carriers change, phone specs change, and the capabilities, the browsers, everything is constantly changing, being iterated and improved upon. And guess what, with actual development, your mobile app will demand rapid iteration and reiteration, which means more change.
The picture you see here is an actual mobile QA desk from one of our customers. I’ve seen desks like that myself and I’ve had desks like that. I’ve had the cabinet full of phones….is my charger there, my sim card there, do I need to do an ENS swap? There are a lot of different things to worry about when you’re talking about working with all of these different mobile devices, and I’m sure a lot of you are starting to face those challenges too.
So you need to be able to deliver and manage across all of that dynamic complexity. Unfortunately, there’s no time for your team to learn those new technologies. Maybe a new phone OS comes out, maybe they need to learn the new IOS 7 or IOS 8 or the new Android. There’s also not a lot of time to do all the testing that you need across all the different mobile devices. There’s a lot of anxiety about whether the application will perform at a scale when deployed in production.
So how does Keynote help you solve those problems? First, we provide a global, cloud-based testing environment with the broadest range of devices and testing capabilities on the planet. This picture shows our testing environment. You can see some of our devices that we’ve built out. So what you’re seeing are real devices. When you’re testing, you want to be testing on real devices. So for that, you need to test on real devices.
This picture is what we do for you. We have these devices built out. They’re hosted in our data center. They’re not cluttering up the desk; they’re not in the cabinet somewhere. They are there when you need them. They’re charged, they’re ready to go, and they have service. Our global network is the most comprehensive in the industry and we work closely with all major carriers so that we can test every possible configuration of connectivity and platform.
Of course, we know that you don’t use Keynote in isolation, so it’s critical that we integrate with and leverage your existing implementations and development and testing tools from major partners like SAP, IBM, HP, and Jenkins, which will be showcased on today’s webcast. This is also where test automation can be of most value, and the ability to run hundreds of test repeatedly on a large number of devices.
Our object scripting will give high reusability across the same type of content, which is web, or across the same platform for mobile. While object scripting is certainly the most important part of automation, it isn’t the entire picture. Object scripting can drive a script based on elements and objects, but it can’t truly tell you if the object rendered properly, you know, whether the picture actually showed up or if it’s pixilated properly aligned. So we also have image- and text-based recognition as well. And you can use all of these in concert to ensure that you’re testing your apps across all of the different use cases that you need.
We support native and web element scripting, native and web and hybrid applications, UI-based scripting like I mentioned, OCR text recognition, and image-based recognition. You’re able to capture your playback. You’re able to script out your things either using your UI-based scripting or Java API to create your own tests through whatever IDE you choose. You’re able to schedule those tests, to run them when they’re convenient for you.
You’re gonna be able to create thousands of test cases and unlimited concurrent tests and users. We provide a lot of different ways to access these real mobile devices. We have our own Java-based client, which you can download and install from your desktop and access those devices for either manual or automated testing. We have a Java API or we have integrations with leading testing tools including UFT, or some of you may be a little old-school and know it as QTP, IBM Rational, and Selenium.
We also discussed today how we integrate with our continuous integration tools such as Jenkins. So let’s talk a little bit about Jenkins since that’s the topic of today’s webcast. The history of Jenkins starts eight years ago. It was built at Sun Micro and it takes agile development to the extreme. There’s been a release of Jenkins every week for the past eight years, totaling more than 500 releases. And that’s the number-one continuous integration and continuous deployment server.
The key to Jenkins is that it’s very easy to get started. It’s very easy to set up and deploy. All you need to do is download the war file and then execute in a console, Java minus Jar Jenkins.war. And there you have it. Jenkins will be up and running on your machine. Jenkins has close to 78,000 installations all over the world, and a 60 percent increase in the last year alone.
So what’s contributed to that massive adoption? Well, the ease of use that we spoke about. But also the ability to write a new plug-in to support a new tool or resource, leveraging its inherent architecture. Several years ago, they had about 60 or so plug-ins contributed and now there’s more than 900 plug-ins by more than 500 contributors with one plug-in per day being added. And those are all available through the Jenkins tool.
Part of the success of an open source community is to see how many people contribute back, and if you look at this graph, that’s about 12 percent of those people using Jenkins have contributed back into the open source with almost 25 percent who are interested in writing a plug-in. In fact, it’s this easy plug-in capability that has allowed Jenkins to build its plug-in to support mobile testing from Jenkins.
Also, part of the success is the staying power of Jenkins. It’s one thing to launch something, but it’s a whole other issue to maintain satisfaction and usability. Today, Jenkins enjoys an 87 percent satisfaction rate. Most of us really new to continuous integration with mobile and I’ve talked to a lot of folks out on the floor today that are just getting started in mobile in general. So let’s try to understand why this is so important and how Jenkins and mobile can work well together.
If we look at the chart here, we can see that internet traffic for mobile is skyrocketing. Over 85 percent of mobile traffic is in direct result to mobile apps. Not mobile web or browsers, but just applications installed on devices, thus supporting the need to understand the importance of investing in mobile app development, which I don’t think anyone doubts. It’s a good reminder. Setting up a continuous integration for mobile means the developers can focus on developing the application. This makes it much easier for developers. There are many plug-ins like we mentioned earlier that you can use for continuous integration with mobile.
You can also scale out your workflow to do distributed builds and create different clients, so you can do matrix-based projects across device testing. Today, when we talk about mobile, we’re really talking about Android and IOS, which are the two major players. Windows is growing, but that’s more in Europe, and the rest of the U.S. has not really reached the level of saturation that we see with IOS or Android to warrant significant support.
Our mobile testing integration with Jenkins enables you to execute test cases directly from Jenkins and to upload applications available in the Jenkins workspace onto Keynote devices. The ability to automatically deliver your build to devices and run a set of test cases to validate that build increases your productivity and streamlines your testing process. As you can see by the screenshot, you’re able to run any existing Jenkins built script on a real mobile device, all from your desktop and without creating a new script. With Keynote’s integration with Jenkins, you can perform automated sanity testing with each mobile application built to support earlier defect identification.
You can find out right away if your build is bad by getting the test report right as the build comes out. It goes on the devices and says, does it work or not? You can run your regression tests automatically. You can build and schedule your mobile application automated regression suite directly from the build machine to increase confidence enhanced test coverage in a shorter test cycle. It also gives immediate feedback to your developers on the quality, functionality, or system-wide impact of the code that they’re writing.
All right. So this is a video that I recorded from my machine beforehand. We don’t have network here but this is gonna show us what Jenkins does with our plug-in so here’s the plug-in interface. This is how you would install a plug-in. It’s just an HPI file that you would upload from your machine and install just like any other Jenkins plug-in. And then this is gonna add some new features to your configuration. You’re gonna be able to go in there and define where your test cases are and also where your username and login for the Keynote tool are and that’s gonna go ahead and grab all of those test cases that you’ve built out, all of your regression tests, and they’re gonna be available for you now through this tool.
For the interest of speed, I’m gonna go ahead here and skip over to some of our previous results because some of those test cases take – it’s running against an actual device going through it just like an end-user would. And those results are gonna look like this. They’re gonna pop up here in just a second in the portal. You’ll be able to see screenshots of the device that was being tested on so it’s very easy to find out exactly what happened and where any failures are and what the screen looked like when they failed.
And using a tool to run your regression tests means that you don’t have to waste a lot of your QAs tester’s time. They’re gonna be able to do a lot more manual testing on the things that matter. Regression testing and sanity checks on builds, if you can do that automatically, you might as well do it and save some time and money. So here are some more steps through there. You can see we’re running through the IMDV app, plus you’re able to see what we’re looking for, what we actually saw in the device in the comparison between those.
So now we’ll go back here. You’re gonna be able to get started using these tools by going to the URLs that you see on the screen. You can also learn more about them by going to the second URL. And this presentation will also be available and posted online after the show.
So now I think we have some time for some Q&A. I know that Josh will be running the online Q&A for the folks attending virtually. But if anyone here has any questions, I’ll be here to answer those for you as well. I see there’s one down there.
Okay. So the question is how we got the tests into Jenkins. So we showed you earlier in the presentation. We talked about the Keynote mobile testing platform. We showed you the devices that are in the bank so those test cases are built ahead of time in that Keynote mobile testing tool. There’s an automation tool that you can build all your automation test cases, your regression test cases. They’re in the Keynote infrastructure. The Jenkins plug-in that we’ve built will then – once you log in through that interface that you saw.
It’ll actually go and look at your count and it’ll get all of your automation test cases and it will list all the devices you have access to and then you can select all of those from that drop-down. So for example, the banking test case that you saw it run was one that we had built on that tool to run through for that banking application. So once you select those, then those test cases will be run as part of that build process. So there are test cases that have been built in the Keynote mobile testing tool. And then the Jenkins plug-in is uploading the new version of that application to the device and then running your regression test cases that you’ve built against it.
These are actually all real devices. We don’t have any emulators or anything. I think it’s important to test on real devices, and what we’re doing is actually in that picture that we showed you. So here you can see we’ve got those devices in they are these two U high rack mountable boxes. What we do is we actually break them apart. We wire into the video chip set on the device and all the inputs and outputs and then we control them remotely so what you see on your screen when you’re testing with these devices is actual video from a real device on the end. It’s a live device on a cellular network or on wifi, whichever you prefer, and you’re just controlling it remotely through our infrastructure. And we actually have some of those set up out on the floor if you’ve like to come by and take a look at those.
So the follow-up question was if it’s more of a SaaS model. And we do offer kind of a SaaS model where we have a very large cloud of devices that we host. That kind of keeps you from having to go out and buy all those devices. But we also have other deployment models where we could, for example, host a system in your own lab or you could be utilizing our public cloud of devices where we’ve got a very large bank and then have a smaller subset of private devices that we host or that can be hosted wherever it is that you want them hosted.
So one of the nice things about this platform, since we’re using real devices, we’re not limited to certain types of functionality of those devices. So since we’re just controlling the device remotely but as an end-user would, we’re able to do testing across anything that we would do on that device. So I can do native applications, I can do hybrid apps, I can even do web testing. Some folks use this just for SMS testing. So they say does this device get SMS from our short code? So basically anything that you can do on that device in your hand, you’re gonna be able to do with this tool because it’s just like having your hands go through the internet and touch the device through the wires.
Thank you so much for coming by and taking a look at what we’ve got. Be sure to contact us to ask some more in-depth questions if you there is anything else not covered in this presentation.
Duration: 20 minutes