*** The PDF version of this white paper can be downloaded from: Blended Test Automation Concepts (BTAC) – White Paper ***
– Leverage Automation Architecture
-Increase testing productivity
-Obtain critical information
-Improve your product
-Take better decision
White Paper, Version 1.0
October 1st 2015
Author: Yagnesh Shah
- Step 1: Select the best choice of Tool for your AUT: App vs Tool Assessment for Web, Mobile, Flash Automation
- Step 3: Implementation of blended Test Automation Architecture + Custom libraries: Selenium Webdriver Learn Explore Contribute – A complete architecture solution
- Step 3: Google reporting: TestNG report Parser + Google doc update on real-time basis
- Step 4: Our Reporting Solution: Test Automation Dashboard – An Open Source Tool
In the present case study, we will discuss in detail, our test-automation solution as provided to one of the major project we worked on. We’ll work through all the aspects of the problem, deal with why we chose the solution in contrast to the available alternatives, alongside the challenges and improvements we’ve made while at it.
Our client is a major provider of “Mobile Payments & Banking” solution based products.
When coupled with the Aadhaar identity platform, payment services and financial inclusion can be achieved at scale. Our Client is working on mobile solutions that make financial transactions possible anytime, anywhere. Our clients’ efforts include a multitude of portals, NoC (Network Operations Centre) dashboards and reports, that will display insights from processing big data in the form of visualizations for quick decision making.
The product is a combination of multiple sub-projects & the end-product is a combination of multiple platforms including Mobile App, Web Portals, integrating hardware’s like Biometric Scanner & Bluetooth printer.
We started automation effort for the project with mobile automation. The tool we selected was Appium & started designing tests for the app. As time passed we were challenged with multiple problems in terms of Technology & Process. This document is aimed to discuss those problems & the solution for the same.
- Tool Selection
- Web/Mobile Automation Architecture & Data-Driven
- Flow of Suites
- Extra Project Level Configurations
- Dev Process & code pre-requisites
In terms of technology here are some of the challenges we faced and how our team overcame this as the time passed.
Product is a combination of Mobile: multiple APK’s, Web, External Hardware: Biometric scanner, Bluetooth printer
We started Automation with Appium Tool for Mobile Automation. Soon we were facing tool capability issues with respect to the features/modules supported by our Application Under Test.
- Unable to interact with ‘Toast Error Handling Messages’ for error validation tests
- Which Tool supports Multiple APK’s for Automation?
- Which Tool interacts with all elements on code level/UX level?
- Unable to interact with field elements like username, first name, etc
- Code Coverage support from Automation perspective
We needed to finalize a tool which could support the capabilities of our product to the maximum extent.
Tool suggested from Client side was Monkey Talk.
We quickly explored Monkey Talk for Mobile automation & our product perspective. Combining the findings from Monkey talk & other expertise we had for Appium, Robotium, Calabash we came up with a thorough Tool comparison doc.
Client was happy to know about our expertise & based on all discussion from the Tool comparison doc, they let us decide which Tool is best suited for their product Automation.
We proceeded with Appium for Mobile Automation for the following major benefits:
- Supports multiple APK’s (Major advantage over tools like Robotium/Calabash)
- Supports mobile device we need to automate
- Rich online tech support & libraries available
For the assistance of Testing community, we have also posted complete Analysis of Tool comparison as a Blog for your reference: Blended Automation – Part 1:: App vs. Tool Assessment
Earlier, we made scripts data-driven & follow the automation architecture guidelines for better script maintenance & reusability. But, soon few questions were brought to our light with regards to Web Automation Architecture:
- End to end architecture of Automation which can take care of custom libraries, scripting, reporting, and so on. Aim is to make framework as reusable as possible & assist new people to script easily
- What happens to headings/labels, if they are changed or their order of position is changed in GUI? Do we re-work on scripts for necessary updates?
- What if the fields of a ‘Create Users’ form changes in future?
- What if total tabs displayed as Admin or any other role are changed in future?
We need to make sure that script re-work is not needed even if the order of elements/tabs/fields changes in the web GUI. Rework should be at a minimum until the feature/requirement itself is updated.
- Implementation Scope of Data-driven architecture & it’s design was upgraded to make sure we can change data in excel itself if the fields/tabs/elements need to be changed in the future. By doing so, script rework was minimized to a great extent.
- Here are the areas where the scope of Data-driven was made more powerful apart from normal data-driven activities:
- Error handling message text are maintained via excel sheet
- Headings for each column for pages like users, organization, transactions, devices and so on are maintained via excel since the position of those headings “<td> attribute in HTML DOM” could change
- Master data validation for different fields across the site
- Form submission fields & labels
- Example design 1 :: Error handling messages being maintained via excel sheet
For any create user form page, there are lots of fields & lots of error handling messages. What if the error handling message format keeps changing? Do you keep changing the message in script each time in future?
Data-driven concept works great for this situation. Here is a sample for one of the message:
Column 0: contains error name which is used as variable in script
Column 1: contains error message
Now, all we need to do is refer ‘orgSaveResponseAlertForCommentsField’ as variable in our script. Whatever error message is changed in future, all we need to do is update the same in excel sheet respectively. No change require on script level.
- Example design 2 :: Tabs displayed as different role based login
As admin role we see 7 tabs & as bank user role we see 5 tabs on web GUI. What if as Bank user, one or two old tabs are removed & another third tab is added (which before was accessible only as Admin) is now also displayed as Bank user?
We made role based excel sheet containing following:
By doing so, our script will read appropriate data from excel sheet for appropriate Role based login. Based on the data it will make sure to check the presence of specific tabs
Thus, if in future if somebody decides to change the tabs as a Bank user, then we don’t need to change the script. All we need to do is update data in excel sheet & the script will assert appropriately.
- For details refer: Architecture implementation sample code and selenium webdriver project template + General Utilities
- Custom libraries, reporting, package presentation, scripting examples which can be reused were developed in process
- Scope of Data-driven concept was made stronger & implemented on a larger scale than before to make sure script re-work is reduced
- Error handling messages text are maintained via excel sheet
- Headings for each column for pages like users, organization, transactions, devices and so on are maintained via excel since the position of those headings could change
- Master data validation for different fields across the site
- Form submission fields & labels
For any Automation project having a single point of control for project level configurations is must.
Being a new project we were not sure of what configurations to provide for a project at different levels of control.
Ex: Browser under execution, DB params, and so on.
We analyzed more than hundreds of scripts for the level of central configurations we need to have & made a collection of configuration properties which is very helpful for us to work on any new Automation project.
Reference link for complete list: Extra Automation Project Level Configurations
Here is a small glimpse of such configurations:
Being a banking product & a combination of Web & Mobile, we needed to make extra patches & configurations which were inserted in-between of suites or scripts to make sure reliable execution of all scripts.
- Updating limits for transaction from default allowed value in DB, then perform test execution, then reset limits to default values in DB.
- Make users & organizations. But in order to reuse same script again, make sure we delete the same from DB directly as there were no delete options available in GUI.
- Flow of test execution: Perform script execution of negative & positive scenarios with appropriate design of data & suites. It’s critical to make sure to execute suite2 after suite1 if it’s dependent on its output.
We needed to make sure the flow/order of each test execution. Being a newly released bank product we had very limited real user data, organizations supported, other test data. So, the test data creation & utilization for each test required in depth analysis.
We analyzed more than 100 scripts for the test data & the flow/order of test execution. Here is a small glimpse for flow of test scripts/suite execution:
Being a large product comprising of many small features & sub products. Our next challenge was automation reporting.
- We knew that TestNG, Ant, ReportNG reports would not serve our needs, as they are limited to the scripts or suites we execute currently.
- What happens if our business heads need information on the entire product containing 100’s of scripts & 10-20(N) suites? For them, the current suite result information is of no use to make decisions. They need info on entire automation results rather than individual suite reports.
- They need a quick way of making decisions like looking at pie-chart for the results of 100’s of scripts. Our client had a dashboard which makes the pie-chart from the given CSV as input, but making input for that CSV involved manual effort on updating results in Goggle sheet for every suite we execute.
We needed a reporting solution which could report everything automatically on the Google sheet on real-time basis & dashboard once the suites are executed and TestNG reports are generated.
- We made our own TestNG report Parser Tool. It extracts info from all TestNG suite reports & updates the same automatically via Google API in Google sheets.
- This tool updates script status on real-time basis for current script execution status.
- We also have made our own dashboard which assists people to view entire history of suites execution status which could have been executed in past.
- We have 4 TV screens on the wall, one of which showcases the real-time execution status of 100’s of scripts
For detailed reference:
- For Google reporting: TestNG report Parser + Google doc update on real-time basis
- For Tool download links refer: Test Automation Dashboard – An open source tool
In terms of technology here are some of the challenges we faced.
Soon, it was becoming difficult to maintain the scripts due to changing requirements. Also, the position of elements from App like various fields were changing.
- xPath are not so reliable for automation(Note: Variable xPath, Relative xPath are good but not absolute xPath starting from HTML body in the DOM)
- We couldn’t use id’s as Appium doesn’t interact with app on code level. This lead to breaking of scripts & script failure
How do we make scripts reliable & and easier to maintain? We needed to make sure it does not break due to changing xPath, id, CSS locators and so on from dev side.
- We suggested the addition of “Content description” attribute for all the elements for Mobile apps & “Id’s” for web build & made sure to follow this as a dev process. Keeping this locators as constant made sure that scripts will never break even on changing element position in HTML DOM.
- Scripts became more reliable & maintenance efforts were reduced.
Our approach & architecture for Automation has been updated constantly with time & experience. Here is a quick glimpse of different factors to take care of while designing automation architecture & framework.
- Easier to maintain & more re-usability (until the feature automated itself is updated)
- More Reliable
- Extra project level configurations
- Custom libraries for Web, Mobile, DB, Reporting, and so on
- Intelligently designed test data & flow of test execution
- Any change in elements of web or Mobile are maintained via dedicated Page Object files on feature/module basis. So, we can make the changes at a single place & all scripts refer the change of Page Object
- Fixed Dev process on pre-requisites for Code & Page Objects
- Our own awesome Reporting Dashboard & Tool for 100’s of scripts. Script status can be viewed on real-time basis for 100’s of script from anywhere anytime.
Stay Tuned & Happy Testing!!