Conference Paper

Compatibility Testing Service for Mobile Applications

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Abstract—As more and more mobile applications are developed, mobile app testing and quality assurance have become very important. Due to the diversity of mobile devices and platforms, compatibility testing for mobile apps has been identified as one urgent and challenging issue. There are two major reasons contributing to this issue. They are: a) the large number of mobile devices with diverse features and platforms which are upgraded frequently; b) a higher cost and complexity in mobile app compatibility testing. This paper proposes one optimized compatibility testing strategy using a statistical approach to reduce test costs, and improve engineer’s operation efficiency. The paper provides a solution to generate an optimized compatibility test sequence for mobile apps using the K-Means statistical algorithm. A compatibility testing service has been proposed for mobile apps. Moreover, two case study results are reported to demonstrate its potential application and effectiveness.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Moreover, the search string was validated by considering a test set of seven control studies [21,34,46,50,72,75,91] previously identiied by one of the authors. The test set was deined before we started to deine the search string. ...
... [50], [42], [91], [56], [15] Portability testing is a very time-consuming task [ 6.2 RQ2 2 Mobile Platforms Figure 8 shows the distribution of the mobile platforms considered in the primary studies. As reported in the igure, Android has been the most exploited platform as it was considered in 91.1% (51/56) of the selected works, followed by iOS, 4.4% (2/56). ...
... Tests are made by sequences of user events used to request the UI of the apps. Test cases are executed on multiple devices to test the portability [42,50,56,91] and the compatibility [50] of the apps. Sequences of events resembling speciic usage scenarios are executed while the performances and energy consumption of the apps are monitored to test their performance eiciency [6,21,53,83]. ...
Article
Full-text available
Context: The mobile app market is continually growing offering solutions to almost all aspects of people’s lives, e.g., healthcare, business, entertainment, as well as the stakeholders’ demand for apps that are more secure, portable, easy to use, among other non-functional requirements (NFRs). Therefore, manufacturers should guarantee that their mobile apps achieve high-quality levels. A good strategy is to include software testing and quality assurance activities during the whole life cycle of such solutions. Problem: Systematically warranting NFRs is not an easy task for any software product. Software engineers must take important decisions before adopting testing techniques and automation tools to support such endeavors. Proposal: To provide to the software engineers with a broad overview of existing dynamic techniques and automation tools for testing mobile apps regarding NFRs. Methods: We planned and conducted a Systematic Mapping Study (SMS) following well-established guidelines for executing secondary studies in software engineering. Results: We found 56 primary studies and characterized their contributions based on testing strategies, testing approaches, explored mobile platforms, and the proposed tools. Conclusions: The characterization allowed us to identify and discuss important trends and opportunities that can benefit both academics and practitioners.
... As Android increasingly gains its momentum, compatibility issues in Android also have been on the rise [30,39,43,47,48]. The open-source nature of Android has led to the diversification of Android devices [34] and customized Android platforms (e.g., the Android operating system kernel) [11]. ...
... While Android fragmentation is a lasting cause of these issues, it is not the only cause. Zhang et al. [47] proposed an approach to testing the compatibility of apps for reducing the testing cost. Recent studies [28,32] focus on compatibility issues induced by SDK evolution and API changes, where these changes, along with developers' strategies dealing with the issues, are characterized. ...
Conference Paper
Full-text available
The rapid expansion of the Android ecosystem is accompanied by continuing diversification of platforms and devices, resulting in increasing incompatibility issues which damage user experiences and impede app development productivity. In this paper, we conducted a large-scale, longitudinal study of compatibility issues in 62,894 benign apps developed in the past eight years, to understand the symptoms and causes of these issues. We further investigated the incompatibilities that are actually exercised at runtime through the system logs and execution traces of 15,045 apps. Our study revealed that, among others, (1) compatibility issues were prevalent and persistent at both installation and run time, with greater prevalence of run-time incompatibilities, (2) there were no certain Android versions that consistently saw more or less app incompatibilities than others, (3) installation-time incompatibilities were strongly correlated with the minSdkVersion specified in apps, while run-time incompatibilities were most significantly correlated with the underlying platform’s API level, and (4) installation-time incompatibilities were mostly due to apps’ use of architecture-incompatible native libraries, while run-time incompatibilities were mostly due to API changes during SDK evolution. We offered further insights into app incompatibilities, as well as recommendations on dealing with the issues for bother developers and end users of Android apps.
... In this SMS, we found three (3) studies that have addressed the issue of compatibility testing [CMT9,CMT13,CMT19]. Firstly, a strategy for compatibility testing was proposed in CMT9 as a solution for overcoming the challenges faced in mobile application due the complexity of mobile app and devices. ...
... We believe it is important to provide more convincing results related to feasibility, efficiency and validity aspects. Only two studies [CMT8,CMT9] representing 8.7% reported case studies in evaluating their proposals (methods/ algorithm). ...
Conference Paper
Full-text available
Mobile applications and devices play significant roles in boosting a global business, which encompasses various domain such as health, education, bank, transport, to mention a few. They become indispensable tools for everyday activities, thus these applications are developed in a very large quantity with a diverse features and platforms. This brings about new set of difficulties and security challenges. To address this problem and ensure quality and security of these applications, a rigorous and systematic testing especially, using a cloud-based environment is required. Using a systematic mapping study (SMS) method is significant in reporting emprirical studies that address issues cloud-based mobile application testing. This paper presents a total of 23 primary studies that investigate a cloud based mobile application testing and the effect of a Testing as a Service (TaaS). Majority of these studies (56.5%) contribute to literature with a number of framework proposals. We discover that most of the studies (60.9%) focus on Android applications and usually supporting one type of mobile app testing. In addition, despite a pletho-ra of services offered by TaaS, majority of the studies (52.2%) fail to focus on TaaS. The results of this SMS identify a clear gap: 1) lack of general and scalable approaches to support diverse types of mobile apps testing for applications using various platforms and 2) lack of evaluation methods such as case study to validate the proposed approaches .
... The above mentioned GUI testing techniques focus on functional testing, while our work is more about nonfunctional testing i.e., GUI visual issues which will not cause app crash, but negatively influence the app usability. The UI display bugs detected by our approach are mainly caused by the app compatibility [72], [73] due to the different devices and Android versions. It is highly expensive and extremely difficult for the developers covering all the popular contexts when conducting testing. ...
Preprint
Full-text available
Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the upgrading of mobile devices and the development of aesthetics, the visual effects of the GUI are more and more attracting, and users pay more attention to the accessibility and usability of applications. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, component occlusion, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a fully automated approach, Nighthawk, based on deep learning for modelling visual information of the GUI screenshot. Nighthawk can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. At the same time, training the model needs a large amount of labeled buggy screenshots, which requires considerable manual effort to prepare them. We therefore propose a heuristic-based training data auto-generation method to automatically generate the labeled training data. The evaluation demonstrates that our Nighthawk can achieve average 0.84 precision and 0.84 recall in detecting UI display issues, average 0.59 AP and 0.60 AR in localizing these issues. We also evaluate Nighthawk with popular Android apps on Google Play and F-Droid, and successfully uncover 151 previously-undetected UI display issues with 75 of them being confirmed or fixed so far.
... There is empirical evidence to suggest that usability testing is very compulsory to eliminate usability problems [4][5][6]. ...
Conference Paper
Full-text available
In a system development process, usability evaluation is a crucial part of the process. The purpose of this paper is to test the usability of new developed application called “DARSGAH” for getting optimum outcomes from m-learning (Mobile Learning) technology. This application provides learning opportunities to University students. Initially, pilot test was conducted with 10 participants. After slight word refinement and minor changes in the application full scale usability test was conducted with other 100 participants who were selected randomly from the University by generating random numbers (student IDs) using SPSS. The data was collected using System Usability Scale (SUS) questionnaire immediately after the test, to obtain students’ overall perception about the application. The findings revealed that the application is user friendly, effective, and efficient and the users were satisfied as the overall average score of SUS is 84. The total task completion rate was 100 % which means that the application was very effective.
... The model reflects four features of the test environment: mobile devices, platforms (various operating systems), connectivity, and application program interfaces (APIs). Another graph model, which represents mobile device feature compatibility and user environments, is considered in Zhang et al. (2015), in which a Mobile Application Compatibility Testing Feature Tree Model and a special statistical algorithm are used to cluster mobile devices with similar compatibility features. A case study for two mobile apps includes the coverage of such features as the Android OS version, camera, screen size, and screen resolution. ...
Article
Full-text available
This paper evaluates the effectiveness of coverage approaches for selecting mobile devices (i.e., smartphones and tablets) to test mobile software applications. Due to the large number of such devices on the market and the variations in their characteristics, it is hard to guarantee that an application will work as intended on all devices. For this reason, multi-device testing is necessary. The goal of this research was to determine how many devices must be tested and which methods for device selection are best for revealing device-specific faults. We experimentally investigated a simple coverage of all values of each device’s features separately and the each-choice coverage (i.e., the coverage of all device characteristics at the same time). To collect the experimental data, 15 Android applications were tested on 30 mobile devices and 24 device-specific faults were detected. Our research shows that a random selection of 13 devices achieved 100% effectiveness. However, coverage of device characteristics in the selection process yielded an acceptable 90% level of effectiveness with a set of only five devices. The most successful approaches were the coverage of different types of Android operating systems and the each-choice coverage. Our results include recommendations for increasing the effectiveness while decreasing the costs of mobile testing.
... To test an app, users need to interact the app first and record the UI events using the tool and the recorded test seems not able to be executed on multiple devices or emulators in parallel. Zhang et al. [34] presented a method to choose a small set of Android devices to execute compatibility testing for mobile apps in order to save test cost. Basically, the method aims to generate a test sequence that covers most compatibility features, such as camera, screen resolution, and network connections, for a set of Android devices. ...
Article
Full-text available
Along with the widespread use of smartphones, Android has become one of the major platforms for multimedia applications (apps). However, due to the fast evolution of Android operating system and the fragmentation of Android devices, it becomes important for an Android multimedia app to be tested on different devices to ensure that the app is compatible with and run well on any of the devices so as to provide consistent user experiences. This paper presents a cloud testing platform (CTP) that allows Android multimedia apps to be tested automatically against a scalable number of physical devices in parallel. Particularly, CTP provides five types of testing to ensure the compatibility of apps from different perspectives, including GUI testing, acceptance testing, stress testing, crash testing, and installation/uninstallation testing. Further, to facilitate identifying the bugs of apps, in addition to test results, CTP also provides the video, screenshots, and performance data corresponding to the tests. Moreover, CTP can also generate a GUI state diagram that can be used to analyze app’s behavior and is helpful for crash diagnosis and debugging. The case study shows that CTP can be effective in ensuring the compatibility of Android multimedia apps while saving test time and effort. © 2018 Springer Science+Business Media, LLC, part of Springer Nature
... A few works in the literature have investigated API compat- ibility issues [9][10][11][12]. Most recently, Wei et al. have focused on fragmentation-induced compatibility issues [9]. ...
Conference Paper
The Android Application Programming Interface provides the necessary building blocks for app developers to harness the functionalities of the Android devices, including for interacting with services and accessing hardware. This API thus evolves rapidly to meet new requirements for security, performance and advanced features, creating a race for developers to update apps. Unfortunately, given the extent of the API and the lack of automated alerts on important changes, Android apps are suffered from API-related compatibility issues. These issues can manifest themselves as runtime crashes creating a poor user experience. We propose in this paper an automated approach named CiD for systematically modelling the lifecycle of the Android APIs and analysing app bytecode to flag usages that can lead to potential compatibility issues. We demonstrate the usefulness of CiD by helping developers repair their apps, and we validate that our tool outperforms the state-of-the-art on benchmark apps that take into account several challenges for automatic detection.
... Graphical user interface [21,22,[31][32][33][34][23][24][25][26][27][28][29][30] Mobile devices have a small screen. Keeping the constraint in mind the app should be developed and tested well to check if it operates differently on varied screen size and orientation. ...
Article
Context: In the last few years, the exigency of mobile devices has proliferated to prodigious heights. The process of developing the mobile software/application proceeds amidst testing phase to verify the correctness of the mobile app. The estimation of testing plays a vital role in the effective completion of testing. Objective: To identify how estimation of test effort for mobile applications is distinct from other software via published literature and from mobile software organizations. Second is to recognize different issues in adapting traditional test estimation methods to the mobile domain and if suggestions from survey results could be helpful in providing an improved test estimation model for mobile applications. Method: A systematic literature review is conducted followed by a survey through an online questionnaire filled from experienced mobile application developers and testers. Results: The results from SLR cover identification of mobile app specific characteristics and reports test effort estimation techniques in the mobile domain. Findings from survey corroborate that a) Function Point/Test Point Analysis is highly adapted traditional test estimation technique to mobile domain; b) Challenges like uncertain requirements, no tool support for test estimation, complexity in testing, client miscommunication etc. are reported; c)Suggestions to improve test estimation process include proper test planning, adoption of agile methodology, healthier communication among client, developer, and tester etc.; d) On the basis of responses, Analytical Hierarchical Process (AHP) identifies “Diverse Devices and OS” along with “Type of App” as highly influential mobile app characteristic on the test estimation process. Conclusion: Results conclude that the importance of identified mobile app characteristics from SLR cannot be ignored in the estimation process of mobile software testing. There might be a possibility to improve existing test estimation techniques for mobile apps by giving weight to mobile app specific characteristics and by considering suggestions from experienced developers and testers.
... Because of this verity of smartphone devices, the compatibility testing for Mobile Application has been known as challenging problem. [11] The two main reasons causing this issue. One is, the huge number of different mobile devices with various platforms and which upgrade very often and second is, the complexity in Mobile Application compatibility testing and its cost. ...
Conference Paper
Full-text available
Mobile Applications are turn out to be necessary in recent years. Mobile Applications are very much different from desktop apps and web apps, and these are progressively moving to be used in mission critical fields. Mobile Application needs a different methodology for assuring their quality, similarly these also need an efficient testing approach to develop high quality Mobile Applications. Assuring the quality of Mobile Applications is a highly related issue since the application failure may leads to some serious consequences, i.e. damage of organization repute or maybe fiscal loss. In this study, we analyzed how effective the different Mobile Applications testing tools are (being widely used by software testing professionals) and their comparison. As a result, we analyzed different software testing challenges in Mobile Applications testing tools in term of unit, integration, regression and system testing.
... Одна или несколько конкретных функций разрушают приложение [28] Запрос на добавление функции (I6) В приложении необходимо добавить функции [35] Проблема совместимости (I7) У приложения имеются проблемы на конкретном устройстве или версии ОС [36] Проблемы при установке (I8) Сбой происходит в процессе установки [28] Проблемы подключения к сети (I9) У приложения имеются проблемы подключения к сети, например, возникает отставание сети (network lag) [37][38] Большой расход ресурсов (I10) Приложение потребляет слишком электропитания или памяти [39][40][41][42] Время отклика (I11) ...
Article
Full-text available
In the current scenario, the popularity of smartphones has led to the emergence of an ample collection of mobile applications (apps). Mobile apps are dynamic in nature; therefore, classical software development approaches are not suitable. Individual needs of the customer, new technology, battery consumption, and many more issues force app developers regularly introduce new apps to the market. But due to the unavailability of any formal and customized practices of app development, various issues occur in mobile apps. These issues may adversely affect the application and user acceptance of the end product. In this paper, fifteen issues in mobile apps have been identified. Then we applied Fuzzy-DEMATEL (Decision Making Trial and Evaluation Laboratory) method to analyze the critical mobile issues (CMIs) and divide these issues into cause and effect groups. Firstly, multiple experts evaluate the direct relations of influential issues in mobile apps. The evaluation results are presented in triangular fuzzy numbers (TFN). Secondly, convert the linguistic terms into TFN. Thirdly, based on DEMATEL, the cause-effect classifications of issues are obtained. Finally, the issues in the cause category are identified as CMIs in mobile apps. The outcome of the research is compared with the other variants of DEMATEL like G-DEMATEL and E-DEMATEL and the comparative results suggest that fuzzy-DEMATEL is the most fitting method to analyze the interrelationship of different issues in mobile apps development. The outcome of this work definitely assists the mobile apps development industry to successful identification of the serious issues where professionals and project managers could really focus on.
... This is done as determined by the requirements stage. Zhang et al. [38], have proposed cost-effective compatibility testing strategy using the K-Means algorithm for selecting mobile devices and their diverse platforms and configurations. ...
Article
Full-text available
Software testing is the investigation process required to give sufficient information about the quality of the program and its conformity with the specifications that have been identified by the customer and to find problems and errors before they are discovered when the system or program is run by the user. It has an important role in software engineering, which is essential to Software Quality Assurance (SQA). Due to increasing numbers of mobile applications users, testing of mobile application is an essential phase in developing any application to ensure that software has zero defects. For that, many software testing techniques and tools are developed, each having its own features to test software. This paper describes and gives general overview of different strategies of software testing techniques that commonly used for detecting errors in mobile applications, which are: White box testing, Black box testing, Gray box testing, Mutation testing, Regression testing and Fuzzy testing. The primary goal of this paper is to emerge the effective technique of the mobile application testing. As a result of the research, most of testing techniques are currently used in mobile testing include Mutation, Regression and Fuzzy testing. The White box testing and Black box testing techniques are rarely used. Also, the latest research into mobile application testing focuses on GUI testing of mobile applications.
... The study by Zhang et al. (2015 ) [S65] presents a compatibility testing method based on a statistical approach for testing mobile applications on mobile devices. The testing method takes into consideration both the large diversity of mobile devices and maintaining a low testing cost. ...
Article
The importance of mobile application specific testing techniques and methods has been attracting much attention of software engineers over the past few years. This is due to the fact that mobile applications are different than traditional web and desktop applications, and more and more they are moving to being used in critical domains. Mobile applications require a different approach to application quality and dependability and require an effective testing approach to build high quality and more reliable software. We performed a systematic mapping study to categorize and to structure the research evidence that has been published in the area of mobile application testing techniques and challenges that they have reported. Seventy nine (79) empirical studies are mapped to a classification schema. Several research gaps are identified and specific key testing issues for practitioners are identified: there is a need for eliciting testing requirements early during development process; the need to conduct research in real-world development environments; specific testing techniques targeting application life-cycle conformance and mobile services testing; and comparative studies for security and usability testing.
Article
Full-text available
Mobile applications and devices have played a significant role in boosting global businesses that encompass various domains such as health, education, banking, and transportation. These tools have become indispensable for everyday activities, and its applications have been developing rapidly with diverse features and platforms. However, this has created new problems and security challenges. To ensure the quality and security of these applications, a rigorous and systematic testing using cloud-based environment is required. By employing systematic mapping study (SMS) method, this paper will examine the empirical studies that address the issues on cloud-based mobile application testing. This paper presents a total of 23 primary studies that investigate cloud based mobile application testing and the effect of Testing as a Service (TaaS). The majority of these studies (56.5%) contribute to literature with a number of framework proposals. A large proportion of the studies (60.9%) analyzed Android applications, and usually supported a single type of mobile app testing. Other than that, the majority of the studies (52.2%) have failed to investigate the outcomes of TaaS, despite a plethora of services that offers TaaS. The SMS method conducted in this paper has identified gaps in literature, which are: 1) there is a lack of general and scalable approaches to support the diverse types of mobile app testing for applications using various platforms, and 2) the lack of evaluation methods such as case study to validate the proposed approaches.
Article
In the current scenario, the popularity of smartphones has led to the emergence of an ample collection of mobile applications (apps). Mobile apps are dynamic in nature; therefore, classical software development approaches are not suitable. Individual needs of the customer, new technology, battery consumption, and many more issues force app developers regularly introduce new apps to the market. But due to the unavailability of any formal and customized practices of app development, various issues occur in mobile apps. These issues may adversely affect the application and user acceptance of the end product. In this paper, fifteen issues in mobile apps have been identified. Then we applied Fuzzy-DEMATEL (Decision Making Trial and Evaluation Laboratory) method to analyze the critical mobile issues (CMIs) and divide these issues into cause and effect groups. Firstly, multiple experts evaluate the direct relations of influential issues in mobile apps. The evaluation results are presented in triangular fuzzy numbers (TFN). Secondly, convert the linguistic terms into TFN. Thirdly, based on DEMATEL, the cause-effect classifications of issues are obtained. Finally, the issues in the cause category are identified as CMIs in mobile apps. The outcome of the research is compared with the other variants of DEMATEL like G-DEMATEL and E-DEMATEL and the comparative results suggest that fuzzy-DEMATEL is the most fitting method to analyze the interrelationship of different issues in mobile apps development. The outcome of this work definitely assists the mobile apps development industry to successful identification of the serious issues where professionals and project managers could really focus on.
Article
Full-text available
Context Testing is a critical and costly activity in the life cycle of a mobile application, due to the growing request of new applications and to the rapid evolution of mobile devices and frameworks. Testing automation may represent an effective solution to improve the quality of mobile applications and to reduce testing costs. Objective We have performed a systematic mapping study to find, analyze, and classify papers in the scientific literature that are related to the automation of functional testing of mobile applications with the aim to provide a classification scheme useful for researchers and practitioners to have a clear view of the state of the art and to easily find existing solutions to their issues. Method We have conducted the study on the basis of a set of 18 research questions. Search queries have been formulated and applied to 7 search engines and the resulting papers have been filtered by considering sets of inclusion and exclusion criteria. The selected papers have been systematically classified and, in addition, a bibliometric analysis has been performed. Results A systematic map including 131 papers has been obtained and is publicly available. The papers have been classified on the basis of the supported testing activities, the characteristics of the techniques and tools they present, and the evaluation methodologies adopted to validate them. The bibliometric analysis has allowed the identification of the most active researchers, the most attractive venues, and the most influential papers. Conclusions The analysis of the systematic mapping has allowed the identification of some research trends and gaps in this field of study. For example, we have observed a strong prevalence of Android-based approaches, a lack of contributions from industry, and the absence of specific venues and journals focused on mobile testing automation.
Chapter
There are always communication fragmented regions in opportunistic networks, and Ferry nodes which can periodicity commute between different fragmented regions always be placed in opportunistic networks. At present, the research on Ferry nodes in opportunistic networks mainly focus on the cache management, energy balance and routing algorithm optimization, meanwhile, researches on identifying Ferry nodes in a strange network are less. On the basis of the importance of structure holes and k-cores, this paper puts forward the index to evaluate the dynamic importance of nodes in opportunistic network, and proposes an importance evaluation algorithm of nodes in opportunistic networks based on it, which is used to identify the Ferry nodes clusters in strange networks. Conclusions can draw through experiments that the proposed model has good applicability and can identify Ferry nodes in networks accurately.
Article
Full-text available
Recently, testing mobile applications is gaining much attention due to the widespread of smartphones and the tremendous number of mobile applications development. It is essential to test mobile applications before being released for the public use. Graphical user interface (GUI) testing is a type of mobile applications testing conducted to ensure the proper functionality of the GUI components. Typically, GUI testing requires a lot of effort and time whether manual or automatic. Cloud computing is an emerging technology that can be used in the software engineering field to overcome the defects of the traditional testing approaches by using cloud computing resources. As a result, testing‐as‐a‐service is introduced as a service model that conducts all testing activities in a fully automated manner. In this paper, a system for mobile applications GUI testing based on testing‐as‐a‐service architecture is proposed. The proposed system performs all testing activities including automatic test case generation and simultaneous test execution on multiple virtual nodes for testing Android‐based applications. The proposed system reduces testing time and meets fast time‐to market constraint of mobile applications. Moreover, the proposed system architecture addresses many issues such as maximizing resource utilization, continuous monitoring to ensure system reliability, and applying fault‐tolerance approach to handle occurrence of any failure.
Chapter
Along with the widespread use of smartphones, Android has become one of the major platform for multimedia applications (apps). However, due to the fast evolution of Android operating system and the fragmentation of Android devices, it becomes important for an Android multimedia app to be tested on different devices to ensure that the app is compatible with and run well on any of the devices so as to provide consistent user experiences. This paper presents a cloud testing platform (CTP) that allows Android multimedia apps to be tested automatically against a scalable number of physical devices in parallel. Particularly, CTP provides four types of testing to ensure the compatibility of apps from different perspectives. Further, to facilitate identifying the bugs of apps, in addition to test results, CTP also provides the video, screenshots, and performance data corresponding to the tests. The case study shows that CTP can be effective in ensuring the compatibility of Android multimedia apps while saving test time and effort.
Conference Paper
The Compatibility Test Suite (CTS) is provided by Google to help manufactures to ensure if their Android devices are in compliance with the Android compatibility standards. However, the CTS contains a huge number of test cases and it usually would take several hours to complete the CTS tests. This could seriously affect the development schedule of Android devices, especially when the CTS test is included in the daily system integration. To reduce the time to perform CTS tests and shorten the time-to-market of Android devices, this paper presents an approach for improving the CTS test efficiency. Particularly, the CTS test is decomposed into multiple tasks to be executed on different devices concurrently. In addition, the task scheduling and partitioning methods are considered in the approach. A cloud-based testing platform is developed to support the proposed approach. The experimental results show that the efficiency of CTS test can be much improved as the number of devices increases. Moreover, the results also indicate that the Longest Job First (LJF) scheduling and mixed partitioning methods can result in better test efficiency.
Conference Paper
Mobile applications have become highly pervasive in recent years. The quality of mobile applications for business use, in particular, is relevant for all stakeholders since application failures can lead to serious consequences, such as damage of corporate reputation or financial loss. For other applications, a reasonable level of quality is also required to convince users to use them. The goal of this work is to identify approaches that address the issue of quality assurance for mobile applications. We present an overview of the identified approaches based on certain viewpoints, such as the focused test level and the addressed quality, and show current research challenges. In order to drive the systematic mapping study, we derived seven research questions based on the stated goal. Then two researchers identified 3,192 records from four digital libraries based on a search string related to terms regarding quality assurance for mobile applications and predefined selection criteria. Ultimately, 230 articles were selected. We created clustered views to answer our seven research questions. In addition, we used surveys found to complement our overview of current challenges. The results show an overall upward trend of publications since 2003. Important topics include automation of GUI tests and assurance of non-functional qualities. Aspects of future research could be the establishment of test environments and the focus on defects addressing stronger the specific characteristics of mobile applications.
Article
Full-text available
Usability testing a mobile application in the laboratory seems to be sufficient when studying user interface and navigation issues. Aki Kek?l?inen
Conference Paper
Full-text available
As mobile cloud computing applications are becoming widely used to mitigate the limitations of mobile devices and exploit cloud services, testing these applications becomes critical. Testing mobile cloud application is complicated further due to dynamic assign of cloud resource to applications that run on mobile devices, due to the increased multiple factors in test case generations and execution. This paper will critically review the three related areas of mobile, cloud and mobile cloud applications testing, in terms of features and models. Furthermore, it will show the necessity of a mobile cloud computing applications testing model due to its uniqueness from both native mobile application testing models and cloud applications testing models. Based on this review, although testing mobile cloud computing applications is critical, there is currently no model to test mobile cloud computing applications.
Conference Paper
Full-text available
App markets are stirring a paradigm shift in the way software is provisioned to the end users. The benefits of this model are plenty, including the ability to rapidly and effectively acquire, introduce, maintain, and enhance software used by the consumers. This paradigm shift, however, has given rise to a new set of security challenges. In parallel with the emergence of app markets, we have witnessed increased security threats that are exploiting this model of provisioning software. The key obstacle is the ability to rapidly assess the security and robustness of applications submitted to the market. The problem is that security testing is generally a manual, expensive, and cumbersome process. This is precisely the challenge that we have begun to address in a project targeted at the development of a framework that aids the analysts in testing the security of Android apps. The framework is comprised of a tool-suite that given an application automatically generates and executes numerous test cases, and provides a report of uncovered security vulnerabilities to the human analyst.
Conference Paper
Full-text available
This paper describes an approach for increasing the effectiveness of mobile software system testing. A Cloud Testing of Mobile Systems (CTOMS) framework is presented in the form of a cloud service that provides the ability to run tests on a variety of remote mobile devices. This framework is based on a heterogeneous networked system that connects operational computers, mobile devices, and databases with software applications. Our research focuses on building a concept and a prototype of CTOMS that supports testing Android mobile applications in the cloud. CTOMS allows multidirectional testing, providing the opportunities to test an application on different devices and/or operating system (OS) versions and new device models for their compatibility with the newest OS versions and the most popular applications. Another new aspect is to embed the test model, specifically the appropriate testing techniques for mobile development, within the framework. For users, this model will provide suggestions from CTOMS about the test methods, criteria, coverage, and possible test cases. These suggestions are based on available configurations, statistics, and resource constraints.
Article
Full-text available
By changing the way software is delivered to end-users, markets for mobile apps create a false sense of security: apps are downloaded from a market that can potentially be regulated. In practice, this is far from truth and instead, there has been evidence that security is not one of the primary design tenets for the mobile app stores. Recent studies have indicated mobile markets are harboring apps that are either malicious or vulnerable leading to compromises of millions of devices. The key technical obstacle for the organizations overseeing these markets is the lack of practical and automated mechanisms to assess the security of mobile apps, given that thousands of apps are added and updated on a daily basis. In this paper, we provide an overview of a multi-faceted project targeted at automatically testing the security and robustness of Android apps in a scalable manner. We describe an Android-specific program analysis technique capable of generating a large number of test cases for fuzzing an app, as well as a test bed that given the generated test cases, executes them in parallel on numerous emulated Androids running on the cloud.
Article
Full-text available
Major software companies, such as Apple and Google, are disturbing the relatively safe and established actors of the mobile application business. These newcomers have caused significant structural changes in the market by imposing and enforcing their own rules for the future of mobile application developments. The implications of these changes do not only concern the mobile network operators and mobile phone manufacturers but also bring additional opportunities and constraints for current mobile application developers. Therefore, developers need to assess what their options are and how they can take advantage of these current trends. In this paper, we take a developer’s perspective in order to explore how the identified trends will impact the mobile application development markets. Our preliminary analysis leads us to suggest eight propositions which summarize our findings and can be the starting points for future empirical research.
Article
Full-text available
The development of wireless application has recently received more attention due to the increment in the number and in the power of mobile devices such as PDA's and cellular phones. Different meth-ods and techniques have been developed to ease the design and de-velopment of applications for these kind of devices. Also, different languages have been proposed to provide support for such platform, such as J2ME and Brew. On the other hand, few attention has been given to testing activity in this scenario. Some works try to test the functional aspects of a given application, others try to perform load, usability and stress testing. In this article we present a strategy to support coverage testing for mobile device software in such a way that the applications can be tested not only on emulators, but also on their real target mobile devices with the aid of structural cov-erage assessment. We also present an environment which supports the proposed strategy. Such environment is implemented in a tool, named JaBUTi/ME. A simple case illustrating how JaBUTi/ME can be used is also presented.
Conference Paper
Full-text available
In this paper we discuss a pilot usability study using wireless Internet-enabled personal digital assistants (PDAs). We compared usability data gathered in traditional lab studies with a proxy-based clickstream logging and analysis tool. We found that this remote testing technique can more easily gather many of the content-related usability issues, but device-related issues are more difficult to capture.
Conference Paper
The testing of a mobile application is a difficult task keeping in mind the diversity in mobile devices and the runtime environment. Also the need of resources to test a mobile application varies from mobile phones to tablet. At present there are tools which can simulate the mobile environment to test mobile applications. These stimulators simulate only the functionality of a particular operating system but not the processor cores, speed, memory, cache size of the mobile device. In order to overcome the above short comings we move to cloud testing. In cloud computing we can easily allocate the required resources using virtualization and also it is easy to scale up our resources anytime without affecting the entire system. The major advantage in cloud is that it is cost-effective. In this paper we define a cloud based mobile testing model which can test application for different mobile environments and platforms.
Conference Paper
Android fragmentation remains a compatibility issue for third-party Android application (app) developers. Most of development teams may not have enough mobile devices for compatibility testing because of limited budget. They may not have enough time to finish the testing on tens kind of devices one by one even devices are sufficient, since the applications have to be launched on time under market pressure. In this paper, we present Mobile App Automated Compatibility Testing Service (AppACTS), which is aimed at helping developers to improve mobile application compatibility testing efficiency, save cost and ensure mobile application quality and reliability. Developers could upload their apps through a web interface, designate interesting mobile device models, and review testing report after finishing testing. The whole compatibility testing process of AppACTS includes the installation, startup, random key and screen actions, and removal of mobile apps. The architecture of AppACTS is scalable and the mobile devices of AppACTS are real devices and geographically distributed.
Article
With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile subscriptions. This drives a strong demand on mobile application testing on mobile devices. Since mobile APPs are native to mobile devices, an underlying mobile platform becomes the basic foundation of their test environments. To achieve effective test automation, test solutions must be compatible, deployable, and executable on different mobile platforms, devices, network, and appliance APIs. This paper is written to provide an approach to modeling mobile test environments based on a Mobile Test Environment Semantic Tree (MTE_ST). Based on this model, the paper discusses test complexity evaluation methods for test environment. Furthermore, some case study results are reported to demonstrate and analyze the proposed testing models.
Conference Paper
With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile subscriptions. This drives a strong demand for development and validation of mobile APPs and SaaS applications on mobile web. This paper is written to offer informative and insightful discussion about mobile testing-as-a-service (MTaaS), including its basic concepts, motivations, distinct features and requirements, test environments, and different approaches. Moreover, it presents a test process in MTaaS and three different approaches. Furthermore, the paper proposes one mobile test cloud infrastructure for mobile TaaS, and discusses the required mobile test frameworks and environments. Finally, the paper addresses existing issues, challenges, and emergent needs.
Article
Mobile application testing is a specialised and complex field. Due to mobile applications' event driven design and mobile runtime environment, there currently exist only a small number of tools to verify these applications. This paper describes the development of JPF-ANDROID, an Android application verification tool. JPF-ANDROID is built on Java Pathfinder, a Java model checking engine. JPF-ANDROID provides a simplified model of the Android framework on which an Android application can run. It then allows the user to script input events to drive the application flow. JPF-ANDROID provides a way to detect common property violations such as deadlocks and runtime exceptions in Android applications.
Conference Paper
Modern software is developed to work with multiple software and hardware architectures, to cooperate with various peer components and can be installed in many different configurations. In order to test it, all possible working environments needs to be created. This requires software and hardware resources like servers, networks and software licenses and most important: man-hours of qualified engineers that will have to configure and maintain them. Because resources are usually limited we have to choose a set of configurations with highest impact on quality of software under test. In this paper we present a method of measuring effectiveness of given software environment for discovering defects in software by introducing environment sensitivity measure. We also show how it can be used in simple algorithm used to select best configurations by using only a selected subset of them and progressively modifying it thougout software development process.
Article
To cope with frequent upgrades of mobile devices and technologies, engineers need a reusable and cost-effective environment for testing mobile applications and an elastic infrastructure to support large-scale test automation.
Article
The usability of mobile applications is critical for their adoption because of the relatively small screen and awkward (sometimes virtual) keyboard, despite the recent advances of smartphones. Traditional laboratory-based usability testing is often tedious, expensive, and does not reflect real use cases. In this paper, we propose a toolkit that embeds into mobile applications the ability to automatically collect user interface (UI) events as the user interacts with the applications. The events are fine-grained and useful for quantified usability analysis. We have implemented the toolkit on Android devices and we evaluated the toolkit with a real deployed Android application by comparing event analysis (state-machine based) with traditional laboratory testing (expert based). The results show that our toolkit is effective at capturing detailed UI events for accurate usability analysis.
Article
Software components are increasingly assembled from other components. Each component may further depend on others, and each may have multiple active versions. The total number of configurations—combinations of components and their versions—in use can be very large. Moreover, components are constantly being enhanced and new versions are being released. Component developers, therefore, spend considerable time and effort doing compatibility testing—determining whether their components can be built correctly for all deployed configurations–both for existing active component versions and new releases. In previous work we developed Rachet, a distributed, cache-aware mechanism to support large-scale compatibility testing of component-based software with a fixed set of component versions.
Article
Users increasingly rely on mobile applications for computa-tional needs. Google Android is a popular mobile platform, hence the reliability of Android applications is becoming increasingly important. Many Android correctness issues, however, fall outside the scope of traditional verification techniques, as they are due to the novelty of the platform and its GUI-oriented application construction paradigm. In this paper we present an approach for automating the testing process for Android applications, with a focus on GUI bugs. We first conduct a bug mining study to understand the na-ture and frequency of bugs affecting Android applications; our study finds that GUI bugs are quite numerous. Next, we present techniques for detecting GUI bugs by automatic gen-eration of test cases, feeding the application random events, instrumenting the VM, producing log/trace files and analyz-ing them post-run. We show how these techniques helped to re-discover existing bugs and find new bugs, and how they could be used to prevent certain bug categories. We believe our study and techniques have the potential to help devel-opers increase the quality of Android applications.
Conference Paper
Mobile applications are becoming more and more powerful yet also more complex. While mobile application users expect the application to be reliable and secure, the complexity of the mobile application makes it prone to have faults. Mobile application engineers and testers use testing technique to ensure the quality of mobile application. However, the testing of mobile application is time-consuming and hard to automate. In this paper, we model the mobile application from a black box view and propose a distance metric for the test cases of mobile software. We further proposed an ART test case generation technique for mobile application. Our experiment shows our ART tool can both reduce the number of test cases and the time needed to expose first fault when compared with random technique.
Conference Paper
As the advance of software component technology, engineers encountered different issues and challenges in testing and automation of configurable components and component-based programs. One of them is how to validate configurable components and programs to achieve adequate test criteria and support test automation. This paper uses a test model, known as a semantic tree, to assist engineers to model and analyze diverse composite components and configurable software in terms of configurable environments, organization structures and functions. Based on this model, well-defined test criteria are presented to address the adequate testing issues. In addition, the paper discusses two test complexity evaluation methods for configurable components and software. Furthermore, some case study results are reported to demonstrate the testing complexity of diverse configurations.
Conference Paper
Java ME (Java micro edition) technology is being widely used in the mobile computing field. Testing of Java ME software is an important and challenging task. This paper reviews the most popular Java ME testing frameworks, i.e. JMUnit, J2MEUnit and Mobile JUnit, and addresses their defects. And then the JT Harness/ME Framework is discussed and studied, which is somewhat currently ignored but is more effective. Based on the analyses, a design scheme of a testing system that integrates the JT Harness/ME framework into the NetBeans platform is proposed. Testing results show this method leverages the advantages of both the JT Harness/ME Framework and NetBeans, and offers a convenient and flexible testing approach with the capability of powerful test management.
Conference Paper
With the development of mobile computing and pervasive computing, smart mobile devices such as PDAs or smart-phones are gradually becoming an indispensable part of our daily life. However, as the software running on these devices becomes more and more powerful and complex, the testing of these mobile applications poses great challenges for mobile application vendors and phone manufacturers. In this paper, we introduce MobileTest, a tool supporting automatic black box test for software on smart mobile devices. The objectives and the design of the testing tool are thoroughly discussed. The paper also adopts a sensitive-event based approach to simplify the design of test cases and enhance the test cases' efficiency and reusability. Finally, we conducted an experiment in a real testing project. Measurement data of the testing process shows that MobileTest can effectively reduce the complexity of automatic test on smart mobile devices.
Using GUI tripping for automated testing of android applications
  • D Amaltano
  • A R Fasolino
  • P Tramontana
  • S D Carmine
  • A M Memon
D. Amaltano, A. R. Fasolino, P. Tramontana, S. D. Carmine, and A. M. Memon, "Using GUI tripping for automated testing of android applications", In: Proceedings of the 27th IEEE International Conference on Automated Software Engineering, 2012, pp.258-261.