No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Enhanced Approach for Maximizing Coverage in Automated Mobile Application Testing
... Therefore, in the context of our experiment, we selected applications from a F-Droid, 3 which is a well-known repository that indexes more than 3000 Android apps of different categories, sizes and complexities. This repository has been used in several studies on testing mobile applications (Samir et al., 2019;Li et al., 2019;Jha et al., 2019;Mao et al., 2016;Li et al., 2014;Lamothe and Shang, 2018;Eler et al., 2018;Silva et al., 2018b). ...
Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners - who design approaches supporting mobile testing - to understand developer's needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.
With the prevalence of Android-based mobile devices , automated testing for Android apps has received increasing attention. However, owing to the large variety of events that Android supports, test input generation is a challenging task. In this paper, we present a novel approach and an open source tool called EHBDroid for testing Android apps. In contrast to conventional GUI testing approaches, a key novelty of EHBDroid is that it does not generate events from the GUI, but directly invokes callbacks of event handlers. By doing so, EHBDroid can efficiently simulate a large number of events that are difficult to generate by traditional UI-based approaches. We have evaluated EHBDroid on a collection of 35 real-world large-scale Android apps and compared its performance with two state-of-the-art UI-based approaches, Monkey and Dynodroid. Our experimental results show that EHBDroid is significantly more effective and efficient than Monkey and Dynodroid: in a much shorter time, EHBDroid achieves as much as 22.3% higher statement coverage (11.1% on average) than the other two approaches, and found 12 bugs in these benchmarks, including 5 new bugs that the other two failed to find.
Unique challenges arise when testing mobile applications due to their prevailing event-driven nature and complex contextual features (e.g. sensors, notifications). Current automated input generation approaches for Android apps are typically not practical for developers to use due to required instrumentation or platform dependence and generally do not effectively exercise contextual features. To better support developers in mobile testing tasks, in this demo we present a novel, automated tool called CrashScope. This tool explores a given Android app using systematic input generation, according to several strategies informed by static and dynamic analyses, with the intrinsic goal of triggering crashes. When a crash is detected, CrashScope generates an augmented crash report containing screenshots, detailed crash reproduction steps, the captured exception stack trace, and a fully replayable script that automatically reproduces the crash on a target device(s). Results of preliminary studies show that CrashScope is able to uncover about as many crashes as other state of the art tools, while providing detailed useful crash reports and test scripts to developers. Website: www.crashscope-android.com/crashscope-home Video url: https://youtu.be/ii6S1JF6xDw
Software systems today are complex and have many possible configurations. Products released with inadequate testing can cause bodily harm, result in large economic losses or security breaches, and affect the quality of day-to-day life. Software testers have limited time and budgets, frequently making it impossible to exhaustively test software. Testers often intuitively test for defects that they anticipate while less foreseen defects are overlooked. Combinatorial testing can complement their tests by systematically covering t-way interactions. Research in combinatorial testing includes two major areas (1) algorithms that generate combinatorial test suites and (2) applications of combinatorial testing. The authors review these two topics in this chapter.
Smartphones are becoming more important in our everyday lives and it is increasingly common to perform critical tasks on these devices, such as making payments. For this reason, ensuring the quality of these applications is an important task. One way to do this is through software testing. However, the testing of these applications presents major challenges due to the wide variety of devices available in the market. In this context, automated testing gains more relevance. There are dynamic test approaches for testing mobile applications, but there are some challenges that need to be overcome for good results, such as, being able to explore the complete behaviour of the application (e.g., overcoming blocking points); choosing appropriate input data; testing dynamic behaviour; testing specific characteristics of mobile applications, such as specific forms of interaction, e.g., long press, and so on. This paper presents a dynamic exploration approach of Android mobile applications that aims to overcome some of the problems identified. During the exploration process, the algorithm builds a Finite State Machine where states are traversed screens and transitions between states describe events that allow moving from one screen to another. This approach is implemented as an extension of the iMPAcT tool. The approach is validated over real Google Play apps and the test coverage results achieved are presented, compared and discussed.
Context: Mobile applications are Event Driven Systems (EDS) that take Graphical User Interface (GUI) event sequences as input and respond by changing their state. EDS are often tested with event sequences that exercise system functionality. Much of prior work focuses on testing random event sequences. Combinatorial-based techniques are often used to systematically generate event combinations and may be extended to test behavior that occurs only when events are executed in a particular order. We expand upon the state-of-the-art by using combinatorial-based techniques to systematically test Android applications with automatically generated GUI event sequences.
Objective: This paper describes a combinatorial-based technique for automatic construction of Android application test suites. The goal is to minimize redundant execution of events, maximize coverage of event combinations, and increase the likelihood of testing behavior that occurs when GUI events are executed in a particular order.
Method: A greedy online algorithm selects and executes GUI events that maximize coverage of n-way event combinations, where n is a specified event combination strength. We compare our combinatorial-based technique to random and frequency-based techniques. We use a two-hour time budget to generate test suites for ten Android applications and empirically evaluate the test suites in terms of code and event coverage.
Results: Our 2-way and 3-way combinatorial-based test suites achieve better code and event coverage compared to random and frequency-based test suites in the majority of our subject applications. The results show that there is no significant difference in code or event coverage between 2-way and 3-way combinatorial-based test suites.
Conclusion: Given the time budget, the combinatorial-based technique is more effective than random and frequency-based techniques, but its effectiveness may vary depending on specific characteristics of the application under test.
Automated GUI testing based on behavioral model is one of the most efficient testing approaches. By mining user usage, test scenarios can be generated based on statistical models such as Markov chain. However, these works require static analysis before starting the exploration which requires too much prerequisites and time.
To address these challenges, we propose a behavioral-based GUI testing approach for mobile applications that achieves faster and higher coverage. The proposed approach does not conduct static analysis. It creates a behavioral model from usage logs by applying a statistical model. The events within the behavioral model are mapped to GUI components in a GUI tree. Finally, it updates the model dynamically to increase the probability of an event that rarely or never occurs when users use the application. The proposed approach was evaluated on four open-source Android applications, and compared with the state-of-the-art tools and manual testing. The main evaluation criteria are code coverage and ability to find errors. The proposed approach performed better than the current state-of-the-art automated testing tools in most aspects.
We introduce Sapienz, an approach to Android testing that uses multi-objective search-based testing to automatically explore and optimise test sequences, minimising length, while simultaneously maximising coverage and fault revelation. Sapienz combines random fuzzing, systematic and search-based exploration, exploiting seeding and multi-level instrumentation. Sapienz significantly outperforms (with large effect size) both the state-of-the-art technique Dynodroid and the widely-used tool, Android Monkey, in 7/10 experiments for coverage, 7/10 for fault detection and 10/10 for fault-revealing sequence length. When applied to the top 1, 000 Google Play apps, Sapienz found 558 unique, previously unknown crashes. So far we have managed to make contact with the developers of 27 crashing apps. Of these, 14 have confirmed that the crashes are caused by real faults. Of those 14, six already have developer-confirmed fixes.
Mobile devices are poised to challenge PCs as the application platform of choice, with 500 million mobile internet devices expected to ship in 2012 compared to 150 million PCs. The convergence of all digital devices into mobile platform model augments the software companies, software developer, and venture capitalist firms to turn their focus into mobile application platform (for example mobile social networking application like face book and mobile VOIP like Skype) a futuristic platform for increased revenue, new challenges and growth potential. But the commercial success of these applications depends on their working smoothly and securely on a wide variety of handheld devices and wireless networks. More and more virtual mobile application stores are built on the web. The web itself is in the transforming form to adapt to the mobile devices to thrive on. The sudden growth in the mobile application and the complexity in the divergence of the devices that uses these applications present increased challenges and opportunities for the software testing companies and software testers to conquer this small device. Performing such testing quickly and cost-effectively greatly expands the market for such applications. This paper deals the nuances of Automated Test Case Design Strategies for Mobile Software Testing.
We present a system Dynodroid for generating relevant inputs to unmodified Android apps. Dynodroid views an app as an event-driven program that interacts with its environment by means of a sequence of events through the Android framework. By instrumenting the framework once and for all, Dynodroid monitors the reaction of an app upon each event in a lightweight manner, using it to guide the generation of the next event to the app. Dynodroid also allows interleaving events from machines, which are better at generating a large number of simple inputs, with events from humans, who are better at providing intelligent inputs.
We evaluated Dynodroid on 50 open-source Android apps, and compared it with two prevalent approaches: users manually exercising apps, and Monkey, a popular fuzzing tool. Dynodroid, humans, and Monkey covered 55%, 60%, and 53%, respectively, of each app's Java source code on average. Monkey took 20X more events on average than Dynodroid. Dynodroid also found 9 bugs in 7 of the 50 apps, and 6 bugs in 5 of the top 1,000 free apps on Google Play.
Iterative improvement partitioning algorithms such as the FM
algorithm of Fiduccia and Mattheyses (1982), the algorithm of
Krishnamurthy (1984), and Sanchis's extensions of these algorithms to
multiway partitioning (1989) all rely on efficient data structures to
select the modules to be moved from one partition to the other. The
implementation choices for one of these data structures, the gain
bucket, is investigated. Surprisingly, selection from gain buckets
maintained as last-in-first-out (LIFO) stacks leads to significantly
better results than gain buckets maintained randomly (as in previous
studies of the FM algorithm or as first-in-first-out (FIFO) queues. In
particular, LIFO buckets result in a 36% improvement over random buckets
and a 43% improvement over FIFO buckets for minimum-cut bisection.
Eliminating randomization from the bucket selection not only improves
the solution quality, but has a greater impact on FM performance than
adding the Krishnamurthy gain vector. The LIFO selection scheme also
results in improvement over random schemes for multiway partitioning and
for more sophisticated partitioning strategies such as the two-phase FM
methodology. Finally, by combining insights from the LIFO gain buckets
with the Krishnamurthy higher-level gain formulation, a new higher-level
gain formulation is proposed. This alternative formulation results in a
further 22% reduction in the average cut cost when compared directly to
the Krishnamurthy formulation for higher-level gains, assuming LIFO
organization for the gain buckets
Review of Mobile Applications Testing with Automated Techniques
- A Kaur
JaCoCo Java Code Coverage Library
- Eclemma
Appium: Mobile App Automation Made Awesome.
- Appium