Visitor Monitoring Mobile Test Automation: A Collaborative Strategy | Keynote
Webcast

Mobile Test Automation: A Collaborative Strategy

About the Webcast

The fast pace of mobile app delivery demands a high level of test automation with testers, developers, operations, and product owners all working together.

Various advantages can be gained when the tribal testing knowledge and code hidden in silos is leveraged across organizational lines. By crossing these lines, the lessons learned include:

  • How to bring mobile development, test, and QA closer together
  • Whether standardization is the answer for you
  • If continuous integration can be supported for faster time to market.

Watch the webcast to explore solutions to these mobile test automation challenges with a focus on using open source technologies to bring teams closer together.

You’ll learn about:

  • QA/Test and development automated test suites
  • Open source tools for cross-team collaboration
  • How real-device testing figures into continuous integration

Silos photo © Peter D. Freeman

Webcast Transcription

Josiah:

Hello and welcome to today's web seminar, A Mobile Test Animation: A Collaborative Strategy, featuring Aaron Rudger, Director of Product Marketing at Keynote, and Bean Chua, Senior Product Manager at Keynote.  I'm the host and moderator, Josiah.  Thank you for joining us today.  Before I hand it off to the speakers, let me explain the control from this console.  If you experience any technical issues, please send a note through the questions located beneath the panel or the on-hand help as quickly as possible. A response is in the web in the Q&A panel to the left of the slides.  It's also the same way to submit questions.

We welcome all comments and inquiries during today's event.  Feel free to ask as many as you'd like.  We'll answer as many as possible during the Q&A session after the presentation.  This presentation includes a live demo.  To optimize the bandwidth, the best viewing experience possible, please quit any unnecessary applications that may be running in the background.  At the bottom of the console is widget toolbar.  These widgets open panels.  The console opens three on the screen, the speaker, Q&A, and the slide panels.  All of these panels can be move around and resized to your liking.  If you accidentally close one, click the respective widget to reopen the panel in its original place. 

Other cursor over these icons will be labeled identifying each widget.  Through these icons, you can download this for the slides and show this event withcolleagues.  This event is being recorded and were made available for on-demand hearing.  Once the recording is ready, you'll receive email of instructions on how to access the presentation and slides on demand.

With that said, let's introduce and pass control to our first speaker, Aaron.

Aaron:

Thank you, Josiah.  Welcome, everybody.  Good morning, good afternoon, good evening.  Wherever you are today, we're really glad that you're here to join us.  I'm joined myself by Bean Chua who's part of our product organization.  My name again is Aaron Rudger.  I'm the Director of Product Marketing here at Keynote.  We're very excited that you picked in a bit of time from your day to join us as we talk about collaboration specifically as a strategy for mobile test automation.  It's gonna be a great talk.  We're gonna talk a little bit about some kind of, some best practices, give you some thoughts as to kind of Keynote's philosophy around mobile test automation. 

But then we also wanna help you to understand a few different perspectives as it relates to technology and approaches and hopefully it'll be practical and tangible for you and we're hoping to get some good use out of the content in the next 40 to 50 minutes or so that you granted us to share with you today. So I'll start off by just kind of sharing with you some thoughts about, a high level, from the highest level possible, what we're all faced with today as our companies, the companies for whom we work are going to market to provide interesting, engaging and really dynamic customer experiences for what is really emerging as a truly mobile native customer base.

You hear a lot about customer experience these days and really customer experience in this new digital age is really the battleground that businesses are fighting for the minds and hearts of the customers that we're trying to reach out and to engage with today.  And so improving that customer experience, providing new stimulating interactive experience specifically through the mobile channel is really what as a business we're finding as we talk to our customers, people are prioritizing the resources, their code, their energy, their budgets all to work. 

In terms of the strategies for delivering digitally, we recently partners with a company called Forrester Research to do a study to ask folks how should, and it's not showing up very well here, I'm surprised unfortunately, that to ask folks what their top visual strategies are for success over the course of the next 18 months or so.  Unfortunately, you can't feel the various responses here.  So I'm gonna read out just a few of them. 

Really they focused around improving that customer experience, expanding their digital assets specifically in terms of mobile and testing in more people, more skills versus maybe reducing costs and basically taking approaches that are more kind of cost slant in nature and probably know surprisingly at least this part of the slideshow up here pretty well for you, not surprisingly what ended up being reported that five companies and we surveyed over 200 or so different folks representing both high key leaders as well as business leaders, folks from the line of business that are actually in charge of and have responsibility for the digital assets that our companies are providing to your customers.

We ask them the question around the priorities, not surprisingly they came up with their top three of their responses being all focused around growth and specifically customer experience, mobile assets and the extension of digital services and what's interesting about this is that you know, in another I would say non-scientific poll, we ask developers and quality teams how these innovation priorities are actually being communication down to them from the business, right?  What we heard was basically the no. 1 response is basically something like this. 

Now it's basically we just need to stuff shift faster and that innovation needs to come at a higher pace, and a higher velocity and actually this is technically inaccurate, if not ludicrous, this should really be plaid I believe but anyways, we're digressing just a little bit here.  To get back on point, the reality is how this articulates in terms of what the industry is in, that's the compression of release cycles and release by a velocity is coming fast and furious and if you are in the Android mobile app development world, your release frequency is about every 28 days on average and then appear in the iOS world, it's about every 59 days. 

It's basically monthly, right?  We're looking at monthly release cycles and yet the customer expectations, further applications, again slide not showing up correctly but the statistics are that if you're not publishing a high quality application, the 20 functionally stable and that's gonna be high performing, customers are just simply not going to adopt it.  They're not going to engage with it, right?  So the bar continues to be raised ever higher for functionality and performance.  It's that overarching customer experience we're trying to create with fundamentally if the features are great, the design is intuitive but it's brittle from the standpoint of quality then you're going to be under delivering to those customer expectations and not being able to provide satisfactory engagement environment for them.

So given their – with high quality, right, takes code and it takes collaboration and so that's really the crux of what we really want to talk about here is this nexus between the points in which the creation of the code and the testing of that code so that it can be properly built and delivered and published ultimately up to the app store could be ultimately expressed in a very efficient manner in a very fast responsive manner.  So for quality teams that want to accelerate app delivery, it's important to acknowledge a few key development realities, right?

These are just three of these realities.  The first one of course being specialization that coders are experts in the languages that they develop against you know, whether it's objective C or Java or C# or Ruby, whatever, right?  They want to work within that environment because that's what they're inherently geared to be able to do.  That's their passion.  That's what they do best.  But first also prefer open source tools and frameworks.  It gives them control and it gives them flexibility and because they're enabling, able to extend and customize the kind of goals and objectives they want to achieve, being comfortable with the model of community support enhancement there's a natural gravitation to open source.

Lastly, testing, and so it would be inaccurate for us to suggest that testing only happens in the realm of quality assurance.  That's simply not true.  Testing happens in development as well and it's actually growing in terms of the amount and volume of testing that occurs in the development cycle and so developers themselves, they’re increasingly taking responsibility especially in mobile application development for their in-testing themselves and there's a number of great reasons, right?  

Again, they know how to automate stuff.  They write automation all day, so a test is an easy extension.  It was always very natural but also the reduction in the cost of defects that could be identified early in the process is amazing and as we have had stats, you probably are very familiar with them those stats that basically set the cost of a bug, you know, the further I could get towards production and ultimately went into production can be orders of magnitude greater then if they're identified early in the application life cycle, right?

Also it's because oftentimes there's a very focused expertise that a [inaudible] [00:10:43] might have specifically in mobile that maybe other folks in the quality assurance team might not in terms of their level of specialization or expertise.  So these are important realities to just kind of acknowledge and then to also understand that we end the testing, in development occurs, it has a chance to focus around certain characteristics or their certain qualities associated with that testing. 

Oftentimes the very focus on units, right, small units, they're gonna be oriented around basically handling some of the non-core functionality that they're working on from a feature extension on enhancement standpoint, right?  So once they are able to enhance or extend their application, they then run very simple units to simply validate that those features are working in the context of the application and they be [inaudible] [00:11:41] very simple units that are core to helping them get to that point in their feature development that they need to be able to get to.

So that also is gonna happen within the context of their development environment, their IDE, right?  So the language they're gonna write those units in, the way that they're gonna automate those, it's going to be probably relayed back to that IDE.  They're gonna be using a mix of emulators and maybe some personal devices, maybe it's some phone that that they have on their desk or in drawers, you know, phones they, or smart or tablets that they've accumulated over time and has every easy access to that will be basically, you know fairly, I should say intimate to that individual developer and then they share those by sneakerheading them around the office but that's kind of the reality of the way that they're going to develop and sometimes when you do that, you end up with creating units that look like this.

Here we've done, we've fit everything into our template but it's not exactly what we were looking for in the end.  So let's talk about QA because really when things come together, that's at the quality assurance stage and QA teams have access to resources and a different set of their, going around a different set of objectives than the developer and so their testing needs and their requirements, and their priorities are going to be focused differently and you guys being QA professionals in the industry, I don't need to go through this list because you literally [inaudible] every day but you know, I think one of the key tech ways here is that you're focused on having a bias for the totality, the end to end aspect of the quality of an application is a really important distinction that differs from maybe the developer's mindset as they go about doing testing, right?

Again, we're talking today about collaboration and these differences in this approach sometimes results in silos, testing ends up being silo-ed and if we want to eliminate, we were, we want to reduce friction, we want to improve efficiency, we need to think about ways of reducing these silos and helping our testing become more collaborative, right?  It's not, it's just simply not necessary today to partition off tests into these individual repositories that can't be leveraged by other teams especially if you're in larger organizations, that becomes a very difficult problem to overcome but it's not something that can't be overcome and so let's kind of talk about some of the approaches maybe. 

One particular approach that we wanted to explore with you in terms of bridging that we found that's being something that our customers have been kind of talking to us about is Appium. Now here is a test of mobile test automation framework that's open source, really helps to kind of close some of these gaps or bridge some of the divides in helping development and QA teams get on the same page a little bit more with regard to testing because it has the natural benefits associated with it in regards to allowing folks to create automation in the scripting line or in the programming language that's native to them.

It allows folks to have access to step automation frameworks that can be ported to emulators as well as real devices and it just kind of helps people to kind of get on the same page, right?  You may have your own approach and Appium is not necessarily the right approach for everybody.  It's one approach that we wanted to kind of talk about today and focus on but you might be a shop that is dedicated entirely to Microsoft or HP and you might be using application life cycle management tools or frameworks in those environments that are closed by nature because they're vendor proprietary but they help to kind of accomplish some of the same things.  I just wanna leave you with is that regardless [inaudible] [00:16:07] talk about Appium here that regardless of whether or not we choose to go that route that the practices and the processes to explore and apply to those types of environments as well. 

We thought this might be a good opportunity to just pause for a second and ask you to just give us your sense of whether or not Appium has ever come up on your radar and if you could, to just take a moment to answer this polling question, does your company use Appium today and the list of responses are yes, we use it or no, we're thinking about or we actually are planning to use it in the next six months or so or absolutely not, we're just a different kind of environment, open source doesn't really work for us, we haven't really thought about Appium at all and so we don't have any plans to use it right now.

You can take a moment to respond to that question, really appreciate it, I think we'd like to share these responses with you as an audience as well.  Bean, if I could just kind of bring you into the conversation at this point, you know, as we talk to customers and that Keynote is in the business of providing quality products specifically mobile testing solutions so that quality assurance teams worldwide can improve the quality of their applications.  So we talk to customers all the time and in your conversations, what have you been hearing from the folks that you've been talking to about Appium?

Bean:

Yes, we certainly have a customer who actually using Appium now and since we use the Appium support, they've been talking to us in term of extending their test cases into the mobile devices in our Keynote labs. 

Aaron:

Okay, great.  Super, well, I think that we're just gonna go ahead and wrap up the poll here.  It looks like we've got a really good healthy base of answers so I'm gonna go ahead and close the poll.  Any last minute rejections you wanna put in there?  Go ahead and put them in.  Let's go ahead and see what the results are.  Okay.  So the results are that 15 percent of you guys are using Appium today to automate your mobile testing.  That's great and then 25 percent of the audience is considering using Appium within the next six months or so.  That's pretty interesting and then about 60 percent of the audience is currently not using, doesn't really have any plans for using Appium and that's totally fine.

As I mentioned, it may be because from environmental standpoint, it just simply is not a good fit or match for your company, for your application development process.  This is great feedback.  So you know, in terms of creating an environment where we can foster collaboration, specifically in terms of the benefits that QA an receive when you're able to better collaborate with development in terms of leveraging and extending the kind of testing that's happening in development, there's some interesting opportunities or some advantages that present themselves if you can exploit that.

So, one of them is that you can actually leverage assets without sacrificing capability.  You can take those tests and you can extend them into an environment especially when, where you can validate your quality and you're tested with new device feedback which is a very important aspect.  Secondly, you could take very narrow units and enhance them to provide additional coverage and so again from a QA perspective, you're really focused on making sure that there's a healthy, very appropriate amount of coverage in terms of the kind of testing that you're doing.

So for example, take and extend that coverage by instead of using one fed data to simply passing unit to stand the range or the array of data that's being put into that particular unit to make it broader in terms of the kind of coverage that it's gonna provide to you.  Then of course taking multiple units and combining them into full cases and if you're doing end to end testing, you really want to ensure that the full quality, that the full customer experience is reliable and stable and functional for our customers then being able to stitch together or compose or orchestrate multiple units into a full end to end test case, how often is that? 

If you're in the development side of the spectrum, when QA is able to leverage your unit test, you get to receive certain benefits as well and so this notion of the source of truth, the single source of truth, my test passed, your test failed, you must have written your test incorrectly.  Well, that goes away if you're actually sharing the same test, right?  Additionally, you can improve the veracity of the fixes that are applied once you have identified some kind of defects.  So if in the development side of the house you're able to provide that to them a test case or a specific unit that is actually consistent and written in the same construct that the developer's written had been writing a day, then that just improves the ability of the developer to ensure that the fix that they put in is actually the right fix.

So they can type the test that you created that you extended in quality assurance and then apply that if they're able to run it in their environment and to ensure that the fix that they put in is actually correct and it's going to look for the desired outcome.  Of course there's improving the validity and the accuracy of all your testing by the inclusion of real devices into the mix, right?  Actually to be biased towards during a lot of testing if not all of it against real devices.  These are some interesting benefits that folks can realize.

When you're building a bridge, you don't want to build that bridge to nowhere and so one of the things that's really important we think in creating collaboration strategy is to ensure that it is truly collaborative, that you're not just talking about test access and be able to use those but to be able to do that in a way where the environments within which they are shared can truly foster collaboration across teams and maybe within your team but there also has to be a cross team.  So there are certain aspects of consistency that need to be applied on especially when dealing and testing with real devices that you wanna make sure actually is provided. 

So a little bit about Keynote, so Keynote does provide a mobile testing solution.  We deliver a real device mobile testing solution that stays in the cloud.  We have technology that allows folks to improve the quality, the overall quality of the application in their mobile websites by having remote access programmatic of remote access to real devices all over the world that actually connected with live connections to real mobile wireless carriers and have full functionality over those devices or full control over the functionality of those devices.

So that means cameras, and key, they have accelerometers, etc. and so that's kind of our bread and butter.  That's what we do at Keynote and we offer a number of different ways in which you can actually deploy this technology. IT could be through our shared public cloud and devices.  They're available 24/7 for folks to interact with remotely or 50 through other configuration for their private devices that are hosted either remotely in the cloud or locally and so a number of flexible options that we provide to you, to basically have direct access to real devices in a very clean consistent way. 

That's really the key differentiator for us between having a device to pull out of your desk, it's very personal and intimate to you versus a consistent bag of devices that's being managed in an enterprise fashion so that the results that you would see are consistent and can be repeated and reproducible every single time.  So Keynote, we talked a little bit about Appium as being one of the great ways of maybe fostering this collaboration in a more leverage-able and enhanced way between specifically developers and QA teams.

Keynote has just released recently an integration with or for Appium.  It does a lot and you can use Appium basically to write tests in Appium and to drive those tests and target them to real devices that you can get feedback on whether or not functionality and quality exists from a real device perspective, again, using Keynote's very consistent enterprise managed cloud-based approach.  So that gives you a lot of opportunities to then integrate these real devices into your overall automated testing approach and structure.

What we thought we'd do is instead of talking about it, we're gonna actually show you it.  So I think at this point what we'd like to do is just do a live test and see if we can actually open up our collaboration environment here to screen cap for you what this looks like and show you how to run a test that you might have written in some nice programming language that you prefer in your own comfortable IDE and target that to a remote device in Keynote cloud and watch that actually run on that real device and see how that can help you to improve your process for quality assurance.

So Bean is gonna drive this demonstration.  So Bean, what are we looking at here?

Bean:

Hi, thank you, Aaron.  We are looking at a web-based interface we have for the device interventions here.  So in today's demo, I'm gonna cover, I'm gonna run through this run test case in three different methods, first is to manual testing and then through Appium, and then through the Keynote built-in recorder for the automation scripting.  So it's just to show you that the support for the different types of users you have, manual testing of advanced user or the beginner user and what you're looking at here is a large device here so this is real device in our, this particular one is in our demo environment so it's connected in our office and as you can see here, I'm looking through the camera of the front and on the front desk.

Aaron:

I should run outside like stand over there by the desk so people could see me.

Bean:

Yeah, so sometimes you can actually see people walking buying the [inaudible] [00:28:29] thing.

Aaron:

It's a slow time in the office here. 

Bean:

Okay, so I'm gonna quickly go through some of the functionality of this UI so if you can see from here is this is a list of all the devices in the package that I have access to.  I can certainly make them my favorite, that way I can quickly get to them, in this case, these two devices here that I have that I'm gonna run my demo on.

Aaron:

And so these are the devices that Keynote have available in that shared cloud environment.  I think what, there's like 200, 300 some are  

Bean:

320 devices plus or minus and then whatever the product devices that you have.

Aaron:

Anything I could want.

Bean:

Basically, yes, and you have a different filtering that can look, can express, narrow down the list that you want to look at, okay, and over here on the phone, I can do anything that you, if you have your phone in front of you, touching the screen, getting into apps, getting up, and there's a whole bunch of data there you can collect given your manual testing gear then to your products from the device or exporting the frames, frame by frame from your testing.

Aaron:

Now that to me would seem very much like wow as opposed to having a device in my hand, being able to do my testing through this [inaudible] [00:30:04] interface A, I may be able to access a device I couldn't, that wasn't in my drawer.  So with the B, even if I was, even if I did have a Samsung Galaxy like the one you have here then it's watching all my testing activity and have ways to share that so that you can collaborate with other folks in your team as you go through this process. 

Bean:

Yes, okay.  So I'm gonna run through this one simple test case from recording an expense that I incurred.  I'm going to run this Expense Manager here.  Now this one, this app, I just installed it from the free store.  So you can upload the app that is pre-production or you can just install the one from the product emulations and in this case this is Expense Manager that I just installed.  I'm gonna click this through the steps that you will see some steps being run from Appium and also from the automated script.  okay?

All we do is come in here, click on the expense, put in some numbers and then within the payee, in this case I'm just going to use B O A, changing the category.

Aaron:

So you're clicking on the keys, the keyboard basically of the device by just what, using your mouse?  Is that how you're doing that?

Bean:

Yes.  So in this case, the mouse will be finger. 

Aaron:

Got it.

Bean:

Come in here, change this, save, okay, just going to go back to this expense and then delete this.  Okay?  So as you can see, I've gone through and use different functionality of the phone, touching, type entering and handling the product screen.  All this can be done automatically also.  So all I did here is just manually go into that, okay?  That's it.

Aaron:

Cool. 

Bean:

Now, I'm gonna go to the [inaudible] [00:32:26] part, okay?  I have an Appium script that is created.  In here as you can see, these are the standard library stuff that you include and here

Aaron:

Just, you went in Eclipse window here, is that what this is?

Bean:

Yeah, thank you.  I am using Eclipse but you could use any IDE that you can write the scripts in Appium.  We tested Appium on multiple languages but as you probably know if you're familiar with Appium, there is more language that it supports then we can cover all in one shot.

Aaron:

Yeah, that's kind of one of the beauties of Appium, right, to support all these various different languages so that's great.  Obviously a lot of different flexibility for people to discuss.

Bean:

Yeah, so on the screen here; this is what the Keynote needed in terms of connecting to the device.  Just using the regular Android driver to connected device and these are the things that you need to add to the Appium script, after that, this part is Appium script itself.  There's no modification inside the scripts, okay?  So you can take your Appium scripts at the device acquisition to the beginning and release of the device in the end and that's it.  Everything else inside should have the highlighted portion on your own product.

If you want to save screenshot, any additional information, you would have to do it in the script, okay?  Then later on I'm going to show you a little bit in terms of what Keynote provided if you're using the Keynote view in recording and scripting capabilities, okay?  So in this case, I'm running through the same step here as I manually come through, going to the Expense Manager and then click into the buttons.  So I'm going to run this on two devices at the same time.  So after I click on this one

Aaron:

Wait a second, you're gonna run this on two devices at the same time?

Bean:

Yep.

Aaron:

Wow.

Bean:

So as you can see, the phone on the right started and then phone on the left started, both going to Expense.

Aaron:

So these are different.  Now, they're both Android phones but these are different phones.  One I could see is LG and then you have a Galaxy there, right?

Bean:

Yeah, the left is LG G2 and then the right is the Samsung S6 Edge.

Aaron:

Got it.

Bean:

So you have one of the latest device from this year and one old device from a couple years ago.

Aaron:

All right, interesting, good.

Bean:

Then if you can notice on this side on my desktop screen, there's a screenshot there that was just saved here.

Aaron:

Yeah, the two little files down there in the left, I see them.  So now that was part of your script.

Bean:

Yes,

Aaron:

To capture a screenshot.

Bean:

That's part of okay, so the script just finished so the phone will return to its original state and then I'm coming back to the scripts, okay, and as you can see from this screenshot, these are the ones that I saved through the Appium script, okay.

Aaron:

So these are files that were just snapshotted to your local laptop that you're write in your IDE.

Bean:

Yes.

Aaron:

Got it.

Bean:

I saved it in the desktop just for demo purpose.  You can save it to wherever the place you want to save it, okay?  This is for the Appium's script spot is very simple, you come in, acquire the device using acquire the device and then run your script and that's it, okay?  Now, this is for user who, or the customer who are very familiar in, comfortable with scripting everything themselves, okay?  We also have the UI so let me close out this.

Aaron:

So you're closing that out, so then again, I might have a development colleague who's run some simple unit test, could've been an Appium script like that, they can target the device. They may run it against one device but you could then take, if you're running in the QA team, you could take that strip and run it again.  Obviously just those two devices but however many devices you wanted to run?

Bean:

Yep.

Aaron:

That's great.  So I can really improve my coverage.  Now of course the script would have to be read in a way presumably from an object standpoint, we're ending up with an apples to apples type of a scenario.  But that's pretty cool.

Bean:

Okay, so now I'm in, this is the Keynote built-in interface.  This GUI base and you can script here basically through graphic interface by dragging the command one by one.  In this one I just clicked use the large apps command and there's nothing secret to it.  It's just two of our great commands, check out here, okay, and it's very easy to launch an app. All you need to do is pick from the list.  These are the list of all the apps available on this phone and then once you pick it and you launch it then you are in the apps yourself.

Okay, now in this, for the good case, let's say I don't want to script, right?  I just want to record the commands.  So I click on the record button, click on Expense, as you can see, it recognized that all the objects from the screen, you save it and then you highlight there but then with all the objects that I touch on.  Now I can continue to click on the buttons to continue recording or I can add a verification step here and let's say I want to make sure that payee show up before I want to continue to the next step.

I can do all these things, okay?  during the recording still active, I can actually go back to one other step and I decided I don't wanna click on that button.  I wanna click on the other button or I want to look for specific buttons.  This is the object filter that allow me to quickly find the objects that I'm looking for.  The reason for this is that this screen is very simple but there are screens that you have, we have up to a few thousand objects that the customer put in and of course you can also look at the object create itself and as you pick the different items then you can see that it's highlighting the item for you.  You can do switching also here.

Well, there's tons of different ways and people are going to another demo and that would take about two hours.

Aaron:

Which we do not have unfortunately.

Bean:

So I'm gonna stop the recording here and then go to the script I already recorded previously and as you can see here, just gonna check this out.

Aaron:

When you were doing the recoding, essentially you could have just simply reproduced that that same steps or interaction that you have showed us originally when you first kind of manually interacting with the device.  It's a very similar mode, is it not?

Bean:

Yes.

Aaron:

And you're in that kind of a coy mode I’m just kind of pushing it, buttons, I type, or I use my mouth to type in values, etc. and the UI just can simply capture all of that interaction.

Bean:

Yes.

Aaron:

Got it.

Bean:

So for example, this was the recorded step that we have.  If the verification added to make sure that your transfer [inaudible] [00:40:55] show up before I go to the next which is talking on Expense and then entering the data.  So in this case it's entering the amount 923 into the box, okay?  This stuff is the same one that I've gone through when I do the manual testing and also the same stuff that I've gone through in Appium.

Aaron:

Got it.

Bean:

So now I'll explain it quickly. One is as you can see, is launching an app, close it first and then open it and then included those with the app expense portion, these are the steps.  I'm not touching the mouse now but it's going to jump through.  You can see the highlight here.  It's going to walk you through the next step as it's executed on the phone.

Aaron:

I see, the highlight, it's moving through those little icons in your grid and putting a little gray circle around that as it steps through your particular process.

Bean:

This step that just passed through as you can see is very interesting.  You are able to get a swiping in the same command so that if you have a long list of items then you can swipe through that.  So this part is done and then the next one, the reason I put in two, actually just to break up just like you normally do when you're doing programming you put in sub functions, otherwise all of them can be in one big giant function, too.  Just group in logical grouping and then use later.  So that's it.

So this part just done, now the few item is

Aaron:

That looks like a very good test by the way because it passed every single time we ran it.

Bean:

In case I didn't pass then you do have a detailed result that I saved.  So from here once you upload the data, what it would see is on the, we have a result that saves all this data and once you click on this, one of these rings, you can see the details that come through and for each step is all the relevant information for you.  So this is the high level here showing you what device you used and kind and the essential information and then I click on one of the commands that in this every step, you get the clip image of before the command started and after the command.

Aaron:

Now wait a second, so with every single step you get this?

Bean:

Yes, so these are done automatically for you.  You do not have to include a function call so the screenshot is taken.

Aaron:

So you had to manually do that when you're writing the Appium version of this but in this case the Keynote system actually does it for you.

Bean:

Yes.

Aaron:

Understood.

Bean:

And you can see that the bottom here is saved all the relevant information and the object that is found on this particular step.  It gives you the coordinate also on where it touched but it actually recognize the object it touched by the object name and the actual coordinate is for your reference just to help you note down where you actually touch it because sometimes on the screen it could be multiple objects with the same information.

Aaron:

Right, you ran the step on the device that has a clunky [inaudible] [00:44:36] tablet right?  Much bigger screen or a new port that's obviously very different from the one you're using here and the bug ended up like in the upper right hand corner, it would either receive the coordinates.

Bean:

Yes, so we provide a lot of information to debug the issues and then if this is for one of the, one that failed, there would be a red mark and then it'll show you the information it collected. 

Aaron:

Our test number failed but I don't know.

Bean:

Well, we want it to fail so that you can catch the problem before, right?

Aaron:

That's true. 

Bean:

Okay and that's it for the demo.  Just to reiterate, we've gone through many of the testing.  We've gone through Appium testing on multiple devices and then I'm showing this recorder based scripting in the Keynote system.  There's a lot more things that we haven't talked much on is for example the executions, you can schedule the job and then when you, you can do the execution from like driving from external or from internal and a whole bunch of other things that we can play with.

Aaron:

That might be the topic of a webcast for yet another day.

Bean:

Yes, sir.

Aaron:

Because we do also obviously but not obvious but for our audience's benefit, what's really interesting is when you can really start combining a lot of this automation in other context like for example build automation so Jenkins is another environment that commonly our customers talk to us about, integrating with to continuous integration and so to be able to kick off, you just kind of build acceptance from oriented tests, in a way every time a new build is created then that obviously, that's something a lot of our customers are very interested in supporting as well and so that's I think definitely a topic for discussion on another day.

Bean:

Yeah.

Aaron:

Excellent.  Well, thank you, Bean.  I'm hoping the audience got some good news out of this.  Just a final closing thought as we wrap up here and get to your questions, really as we're thinking about collaboration, hopefully you kind of can see or have a good sense of the feel for the kind of collaboration that you can drive if you are using some framework like an Appium framework that really has a lot of inherent advantages for folks let's say in your development team that can be then leveraged and extended and repurposed in the context of truth in the end QA or quality assurance.  You get a lot of great benefit but really there's too much like benefits where you can bring those two together. 

We can build that bridge but really the key, one thing that we really wanna emphasize that if you build that bridge and still yet you maybe sharing the same kind of tests or the actual code of the tests themselves, then you're not sharing the same device, how good is the consistency going to be or the reproducibility of the results that you're getting in the context of that testing?  All that collaboration again that bridge, may be a bridge where if you can't have certainty around the reproducibility of those results.

So we really think it's important to think that you have a strategy for ensuring that the devices and the environment in which you're actually executing your testing delivers that kind of consistency so that you can have the assurance from knowing that your results are true and accurate.  That's our last parting thought for today, hopefully [inaudible] [00:48:27] to you.  I guess, Josiah, what we'd like to do maybe is to turn it over to you to open up the conversation with our audience and answer some of their questions.

Josiah:                        

All right, absolutely.  Before we start Q&A, you can ask Aaron and Bean questions by typing in the field beneath the panel and clicking the submit button.  We'll try to get to as many questions as possible but for those we're unable during Q&A, some will be answered offline.  All right guys, first question, is the Keynote driven test approach supported by Appium be default without any extra effort?

Bean:

Yes, so as you can see the demo for the Appium portion, whatever is supported by Appium, we allow them to execute on the devices.  So you can use different methods that have been, allow you to identify the objects and also to pass through the parameters. 

Aaron:

And just for everybody's reference, obviously, at Keynote, we try to be as open and extensible as we possibly can so Appium is one framework.  We don't profess to be experts in Appium but we do support all of the native capabilities of Appium in and of itself so anything that you can envision, that you can do in Appium really is supportable by the Keynote system and so I would encourage folk to explore the Appium community, it's an open source community, if that's of interest to them but yeah, keyword support absolutely.

Josiah:                        

All right, thank you very much.  Next question, do developers need to change their code to let Appium grab object identifiers?

Bean:

Sorry, can you repeat the question?

Josiah:                        

Yeah, no problem.  Do developers need to change their code to let Appium grab object identifiers?  It's question no. 23 in the chat.

Bean:

Thank you.  Well, for the Appium portion, you do need to allow Appium to recognize the objects and I believe it's also using accessibility API but those are by default should be supported in the Android devices. 

Josiah:                        

All right, thank you very much.  Next question, does Keynote provide services in addition to the mobile test infrastructure and mobile test automation tool?

Bean:

Yes, Keynote does have a professional services group that can help you with providing guidance.  For example if you're doing your own automations, they can actually provide the complete solution of doing the automation for you and on top of that, you can also do monitoring.  So Keynote does have a full suite of different products. 

Josiah:                        

A little bit more specific question, do you have to recompile or made up apps specific library or code block to automate using Appium? 

Bean:

For Keynote portion, they don't need to recompile a code.  So as you can see from the example, the main item for Keynote is to acquire the device in managing the device but once you acquire the device, whether the code that you have, that you've written in Appium, you can run those and remember the app is just a normal build.  They don't require you to rebuild or Keynote portion does not require to add libraries into your code.  The one additional requirement for the iOS app is that we do need the reassign the apps but there's no libraries to add to your code either. 

Josiah:                        

All right, and thank you very much in doing this one a little bit longer.  In Keynote's Appium portion a proprietary wrapper around the open source Appium or can it execute a Keynote Appium script on another solution that supports Appium like Soft Labs?

Bean:

Frankly, there's no Keynote Appium script.  The Keynote support Appium as it, we don't change your Appium script as you can see from the example.  We don't modify your Appium script.  We don't require you to add additional code into your apps to support Appium.  If Appium has required you, that's Appium, but Keynote itself does not require you to do anything with the Appium code.

Josiah:                        

All right, next question, how can you determine the amount of coverage are tested on specific devices, how should I choose the phone, the tablets I should be tested on?

Bean:

Yes, we do have a choice that can help you based on the criteria that you can enter.  On top of that, there's also, have to look at the market that you're focusing on.  So there's the [inaudible] [00:54:05] in terms of the device models and over the, that are popular in different market segments but overall, at the very high level worldwide Android has the most market share and iOS is next and depends on the market segment that you're going into, Windows for example could go up to like 10 percent and those are across the different OS smart phone. 

In terms of the different phones and different resolutions per OS and those also depend on how we do covering because mainly it goes by the screen size so we're targeting large, medium, and small screen size and that covers most of these cases.

Aaron:

Correct me if I'm wrong but I believe that we do suggest or recommend the best practice but when you are developing mobile applications and of course you're doing your testing and all that then we do consider and apply as part of your strategy a mobile application analytics portion as well.  Analytics are so critical to being able to help you understand and drive greater, not only greater usage and adoption but higher quality as well.  This goes beyond just crash analytics but full functional usage, understanding of your application.  I think that's pretty much the best practice and really you wanna use that empirical data to drive the kind of decisions you make with regard to device coverage.

Bean:

Yep, absolutely. 

Josiah:                        

All right, next question, do I need to install and run Appium locally in order to test on Keynote devices remotely?

Bean:

No, so the Appium server portion, those are on the server that devices are connected to.  So in your [inaudible] [00:56:15] you do need to have the IDE of course that is compatible and then the Appium libraries to be included in your scripts and then runs your KPR, you'll be running on the Keynote device and Keynote server so you don't have to have the [inaudible] on your machines.

Aaron:

We take care of it all.

Bean:

Yeah.  You just need to tell us which device to run on and we'll run it.

Aaron:

That's right.

Josiah:                        

All right, well, thank you very much, guys.  That's the end of our event for today.  I'd like to thank the speakers, Aaron and Bean, for their time and I'd like to thank Keynote for sponsoring everybody.  Also a special thank you for the community audience who spent the last hour with us.  Have a great day and hope to see you at a future event. 

Aaron:

Than you.

Bean:

Thank you.

Duration: 57 minutes

Back to Top