Thorough testing and validation are necessary steps in delivering a quality mobile application that meets requirements. But how can we alleviate the many pain-points associated with testing mobile applications while still ensuring that testing is given ample attention? This post will discuss emerging automated testing tools and provide testing recommendations, specifically for mobile applications.
Why Automated Testing?
Manual testing of applications can be a slow, repetitive and inefficient process. For example, testing the flow of creating a username, logging in, changing the password and submitting a new post to a message-board type application is easy enough…the first few times. But, this same core functionality should be retested with every new build of the application to ensure that it wasn’t broken by new code. With many software packages pushing one or many builds a day, this process becomes time consuming and increasingly tedious. However, this type of testing effort (regression testing) can now be automated to run in a fraction of the time as compared to a human resource providing the input.
Automated testing can also be applied to almost any phase of the testing cycle including integration testing and end-to-end testing, but the biggest savings in time and costs come when applied to regression and functional testing. When automated, functional testing can be run with many different inputs to ensure that actual meets expected results. Depending on the tool, external databases can be accessed as a library of input variables to further increase efficiency.
Regression testing can be a tester’s nightmare when done manually. Having to tap and swipe the same screens over and over is time-consuming and repetitive. Regression efforts will sometimes be neglected or delayed due to sheer loathe of the exercise. However, automated testing can remove the human interaction almost completely. Some testing tools even pair with continuous integration servers like Jenkins to automatically run regression tests after new builds are deployed. Without even having to “click go,” a tester can check a JUnit or RSpec report for pass/fail outcomes.
The Case for Manual Testing
Manual testing may now seem primitive; however, it has many benefits that cannot be achieved through automated testing. Interacting with an application through taps and swipes, the same way as the end user experiences it, will always be necessary. Testing edge cases and new functionality requires new and complex development for test scripts. While having an automated test script would be nice for repeatability, the automated test script development effort can exceed the effort in manually testing non-core scenarios.
Using Both Manual and Automated Testing
In order to obtain an efficient, effective, flexible and thorough testing process a mix of automated and manual testing should be chosen. By switching to a mix of manual and automated testing, costs savings can come in the form of freeing up resources to focus on other areas of validating the application. In addition to reducing human testing hours, automated tests can be rerun in frequencies and on schedules unable to be executed by a dedicated team of testers. This allows the testers to spend valuable hours ensuring that an application meets all requirements and service level agreements for performance.
Comparison of Automated Testing Tools
There are many automated testing tools vying to become the market leader. It is important to first decide how automated testing will be incorporated to your testing framework before a decision can be made. Below is a chart that compares some of the offerings available today and details the pros and cons of the products which support testing on both iOS and Android.
- Modeled after Selenium’s well established web-testing framework
- Test scripts can be written in nearly any language
- Tests can be run on real devices and simulators
- Not all mobile gestures currently supported
- No built-in reporting
- Only one test can be executed at a time, per hosting machine
- Easy to create tests with support for complex multitouch gestures
- Can run on could against many devices simultaneously
- Robust reporting and scheduling included in suite
- Cannot test on real devices
- TouchTest wants portion of profits made from app
- Lite, free and easy to write Cucumber tests out of the box
- With test cloud, can be run on many platforms and devices simultaneously
- No GUI or test suite to assist with touches or gestures
- Difficult to identify element/object IDs for writing scripts
- Easy to create test scripts using GUI
- Multiple devices can be referenced in one script (i.e. one device posts, one comments)
- Tests can be written in many languages
- Product still in Beta
- Requires subscription
- Tests can be run on hundreds of actual devices
- Monitors device performance such as memory use and responsiveness
- Tests can be kicked off by Continuous Integration servers
- Pricing based on test hours
- Cannot test on devices in-hand
With pressure to take an application to market quickly, some chose to skimp on testing and fix issues after deployment. This can cause users to lose confidence in the app, as well as discourage future use. Instead of rushing to distribute a buggy app, developers should incorporate automated testing into the existing testing processes, enabling them to deliver quality products on time.