Content uploaded by Hossein Shirazi
Author content
All content in this area was uploaded by Hossein Shirazi on Apr 16, 2018
Content may be subject to copyright.
A Classification Framework for Automatic Test Case Generation
Techniques for web applications
1 Mohammad Reza Keyvanpour, 2Hajar Homayouni, 3Hosein Shirazee
1, First Author Computer Engineering Department, Alzahra University, Vanak St., Tehran, Iran,
Postal Code. 1993893973, Keyvanpour@alzahra.ac.ir
*2,Corresponding Author Computer Engineering Department, Alzahra University, Vanak St., Tehran,
Iran, Postal Code. 1993893973, Homayouni.hajar@gmail.com
*3,Corresponding Author Computer Engineering Department, Azad University, Qazvin Branch,
Qazvin, Iran, Shirazee@gmail.com
Abstract
With the growing complexity of web applications, testing is essential to ensure that they provide
reliable qualities. An essential task in software testing is the generating of test cases, which is in
general, a costly and labor-intensive process. As web applications have been evolved through recent
decades, various methods of generating test cases have been proposed according to their features and
complexity. In this paper an all-around classification framework for existing automatic test case
generation approaches for web applications are introduced. Different techniques for both traditional
and modern web applications are compared by defining general evaluation criteria and the results are
analyzed.
Keywords: Test Automation, Test Case Generation, Web Application, Classification Framework
1. Introduction
Today, web applications as complicated, ever evolving, and rapidly updated systems, play an
increasingly essential role in people activities. Faulty web applications can have far-ranging
consequences on businesses, economies, scientific progress, and health. Therefore, it is crucial to test
web applications completely to ensure that the applications are reliable and meet their design
specifications [1]. With the development of internet technology, the web applications become more and
more complex with strict requirements of quality reliability.
Software testing is an integral part of quality assurance activity in the domain of software
engineering [2], [3]. Testing is the process of detecting errors and failures in software systems.
However, manual testing requires extensive human effort, which comes at significant cost [4].
Testing of web applications is a kind of software testing, which has been studied for a long time by
researchers and has long been recognized as a hard task [5], [6]. Traditional testing methods and tools
are not sufficient to address the distinctive features of web applications [7]. As web application
becomes more and more complicated, testing them is a hard and time consuming activity, thus testing
is often poorly performed by practitioners. Test automation can avoid this situation [3].
Test case generation is one of the most challenging and extensively researched activities of software
testing [8]. A good set of test cases are one that has a high chance of identifying previously unknown
errors. To reduce tedious manual efforts in generating promising test cases for web applications,
various automated techniques have been proposed.
Web applications have been evolved through last decade to satisfy requirements of different users.
Its evolution process started from a simple static page-sequence client/server system [9] into a dynamic
medium of user-created content and rich interaction. The complete evolution steps are discussed in [9].
In this paper we classify web applications into two general groups i.e. traditional and Modern, and
review the existing automatic test case generation methods for each class. We compare different
methods with appropriate evaluation criteria, and indicate which method is suitable for which kind of
requirements; we also show that methods for generating test cases for traditional web applications are
not sufficient for testing new generation of web applications. The remainder of this paper is organized
as follows. Section 2 is related works. Section 3 introduces the proposed classification framework.
Section 4 is some discussions and finally section 5 is conclusion.
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
International Journal of Information Processing and Management(IJIPM)
Volume4, Number3, May 2013
doi:10.4156/ijipm.vol4.issue3.3
26
2. Related Works
There has been a significant amount of works in automatic test case generation for web applications
that attempts to increase the amount of observed behavior. Despite of these wide researches, there have
been few efforts on representing an all-around classification, which cover all existing automatic test
case generation approaches for web applications. A classification for Model-based test case generation
techniques for web applications is given in [10]. In [8] automatic test case generation approaches are
classified into three general groups i.e. Specification-based techniques, Sketch diagram-based
techniques, and Source code-based techniques. Sebastian Elbaum et al. [11] presented a classification
for traditional multi-page web applications. Another classification was suggested by A. Arora et al.
[12] in which testing methods for web applications are separated into two general groups i.e. white box
and black box testing. These classification frameworks don’t cover all the features of different web
applications.
3. Proposed Classification Framework
In general, for the purpose of classifying different techniques of automatic test case generation, we
first separate web applications into two general categories. The reason for this classification is that the
features of each category make the test case generation method different from other one.
3.1. Traditional Web Applications
Traditional web applications are based on a multi-page user interface model, in which interactions
are based on a synchronous page-sequence pattern [9]. This class of web applications typically
involves complex, multi-page, multi-tiered architectures containing Web sites, applications, database
servers, and clients [13], and heterogeneous execution environments [7]. These applications have
unpredictable control flow, as users jump to arbitrary URLs [14]. They are based on the stateless HTTP
protocol. Generally speaking, we can mention the following attributes for traditional web applications
[3]:
• Heterogeneous execution environments: Different hardware, network connections, operating
systems, web servers and browsers.
• Heterogeneous nature: Different technologies, programming languages, and components.
• Dynamic nature: Web pages can be generated at run time according to user inputs and server
status.
The architecture of traditional web applications is presented in figure 1.
Figure 1. Traditional web applications architecture
3.2. Modern Web Applications
Modern web applications (web 2.0 applications) as dynamic medium of user-created content and
rich interaction rely on asynchronous client/server communication. In general they have the following
features [9]:
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
27
• Strong participation of web users as a source of content.
• User collaboration and information sharing.
• Rich but simple to use web user interfaces.
• Software as a service through the web.
Rich Internet Applications are usually developed using Web 2.0 technologies, such as Ajax,
Silverlight, or Flex [15]. Among the technologies that are being developed to implement its features,
Ajax is one of the most promising and mature. Ajax supports the development of rich-client Web
applications, by allowing the execution of asynchronous requests and dynamic update of the page
structure and content. Ajax Web applications usually are single page in which elements are updated in
response to callbacks activated asynchronously by the user or by a server message. Actually, with Ajax,
developers can implement asynchronous communications between client and server. Due to the event-
driven elaborations, the interface of Ajax-based applications may be considered like an event-driven
software system (EDS) or similar to the GUI of a desktop application. In these applications HTML and
CSS are utilized for information presentation, the DOM is used to access and modify the display
information, XMLHttpRequest object is exploited to retrieve data from web server, and XML is
employed to wrap data. Javascript code is executed upon callback activation [6]. Figure 2 shows the
architecture of Modern (Ajax-based) web applications.
Figure 2. Modern web applications architecture
In next section we will describe the existing test case generation approaches applied to each type of
web applications.
3.3. Automatic Test Case Generation for Traditional Web Application
For this type of web applications, test cases are defined as sequences of web pages that are
potentially fault prone [13]. Many approaches have been proposed to address traditional Web
application test case generation for different testing goals. An all-around classification for these
methods is presented in figure 3, and the main idea, applications, pros and cons of each technique are
explained as follows. Table 1 summarizes the different approaches of generating test cases for
traditional web applications.
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
28
Figure 3. Automatic Test Case generation methods for traditional web applications
3.3.1. Random Based Method
In this method, test cases i.e. page sequences, are generated randomly with no specific strategy.
Among other existing approaches this randomly generating test cases is the most simple, fully
automated and easy to implement choice for a test case generation tool [16]. However, this approach is
not a promising way to produce test cases which are able to achieve a high level of coverage. This is
due to the low probability of selecting sequences of interest from the input domain of the program. On
the other hand, there may be some infeasible sequences among the test cases resulted from applying
random technique. Due to the fact that this method isn’t sufficient for testing complicated web
applications, it is sometimes used beside other techniques or for evaluating the performance and
accuracy of other methods.
3.3.2. White Box Method
Similar to desktop applications, this method is based on knowledge about the internal structure of
the SUT. The test case adequacy is assessed by evaluating the level of structure coverage they reach.
These methods are implemented in a simple way. However, generating test cases is usually done very
late in the software development cycle, thus errors discovered are very difficult and costly to correct,
since changes affect large portions of the design, implementation, and testing procedure [17]. This
method can be applied to web applications in different ways i.e. code coverage and Model based.
1) Code coverage based: These techniques generally use a control flow graph of program to identify
paths to be covered and generate appropriate test cases for those paths [8]. The control flow graph is
derived from source code in which nodes represent the statements that are executed by the web server
or by the client, while edges represent control transfer [18]. Although this method is easily
implemented, it needs to support and trace Web code that, often, is a mix of languages such as HTML,
Javascript, and JSP, and currently, tools with these characteristics are not available. A significant
scalability challenge for this method is how to handle the exponential number of paths in the code. This
technique can further be classified as static and dynamic [8]. In Static method test cases are generated
based on symbolic execution of the program, in which the main idea is to use symbolic values instead
of actual data as input values, and to represent values of program variables as symbolic expressions.
Therefore, the outputs computed by a program are expressed as a function of the symbolic inputs [19].
Dynamic techniques obtain the test cases by executing the program under test.
In [6] a two-phase approach to generate test cases automatically by analyzing structure of the Web
application is proposed. The dependence relationships, data dependence and control dependence, in the
Web application are defined from source code and improve the way of test case generation. [4], [20]
proposed a method based on symbolic execution of program for generating test data in PHP coded web
applications. In [21] traditional flow graph testing techniques is extended to Web applications.
2) Navigation Based
A navigating model of web application under test is a graph where each node represents a Web page
and each edge represents a link. Test cases, i.e., sequence of pages, are extracted by traversing the
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
29
navigation model of the application under test according to a coverage target (e.g., nodes, edges, or
paths). For the purpose of creating navigation model, application source code is analyzed to extract
user navigation map which is composed of all the possible URLs from Web application source code,
then through the navigation graph, a set of paths is selected and test cases are generated for each path
[18]. Due to the fact that there are a huge number of pages in traditional web applications, in this
method there is a need to apply an appropriate search strategy in navigation model. Indeed this method
suffers from state explosion problem.
In [14] Application source code is analyzed to extract relevant information about the URLs and their
parameters. This information is used to construct and simplify a graph in which nodes represent URLs
and edges represent links between URLs. A set of paths through the graph is selected for each path to
generate the test cases.
3.3.3. Black Box Method
In this method the Web application is considered as a black-box, with no information about internal
structure. Functionality of the web application can be tested using this technique. The existing
approaches using this strategy for generating test cases are based on creating a model of the web
application under test. This model is generated using software specification. The model can be used for
generating test cases according to some scenarios of interest which is a graph which specifies the web
application. An algorithm is then developed and used to traverse the model in such a way that ensures
it satisfies the desired coverage criteria, and ensures that all aspects of the web application are tested.
This approach has the advantage of supporting direct response of user needs [22]; furthermore, testing
can begin much earlier in the software design process, therefore inconsistencies and ambiguities in
specifications are found earlier by software testers and errors discovered are simply and not much
costly to correct. On the other hand, specifications are complex and, therefore, difficult to understand
and also implementation should exactly correspond with specification. In General it is hard to represent
an exact identification of system specification in real web applications. There are different tools for
representing software specification. Some examples are Data Flow Diagram, Use Case Diagram, Petri
Nets, Z language, and Object Constraint Language.
Jia [5] uses formal specifications in XML syntax for generating test cases to test functionality,
security and performance of web applications. In [22] test sequences are generated by using an initial
set of sequences obtained from the specification (use case model) and extending these initial sequences
to cover the dependence edges in the model for testing Multi-tier Web Applications. [23] Uses use case
diagram as to build the behavioral model of web application and generate test cases to satisfy path
coverage. A path in the behavioral model is defined as a scenario in the use case. [24] A graphical web
model is driven from specification document, and the model is traversed using “All Link Coverage” to
generate test cases.
3.3.4. User Session Based
Each user session is a collection of user requests in the form of URL and name-value pairs. In other
words, a user session is defined as beginning when a request from a new IP address reaches the server
and ending when the user leaves the web site or the session times out [25]. Each logged request of the
user session is changed into an HTTP request that can be sent to a web server. A test case consists of a
set of HTTP requests that are associated with each user session. Generally speaking, Test cases are
generated by analyzing user logs on the Web server to build models of a web application. Logged user
data captures dynamic behavior of an application. However, there is some redundant or unimportant
information in data captured from user sessions. Therefore in this approach there is a need for data
reduction instead of exhaustive search in all user sessions. Due to the fact that these data capture the
real interactions of users with the web application, it seems to be an appropriate method for testing the
real world applications. The general architecture of this approach is shown in figure 4. There are lots of
examples using this method. In [2] it divides the user sessions into different groups, each of which is
called a test suite, and then prioritizes the test suites and the test cases of each test suite. So, the initial
test suites and test cases, and their initial executing sequences are achieved. Genetic algorithm is then
employed to optimize the results of grouping and prioritization. [9] Automatically builds statistical
models of user sessions and automatically derives test cases from these models. Bošnjak et. el [26] give
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
30
an insight into the possibility of merging data mining techniques with Web access logs analysis for
achieving a synergetic effect of Web usage mining and its utilization in Web Applications testing. [27]
Present a test case generation approach named US-RDG in terms of gray-box testing, which combines
user sessions with request dependence graph (RDG) of web application, to generate test cases. [28] Is a
user session based testing which considers state dependences from multi-user interactions and claims
that generated test cases are more realistic than traditional user sessions and their replay covers
different code and exposes different faults. In [29] user sessions are collected by parsing Apache server
log and used to generate test cases of high code coverage.
Figure 4. Session based test case generation architecture [9]
Table 1. Automatic test case generation approaches for traditional web applications
Approach Main Idea Pros Cons Domain
Random
Based
Generate test cases
randomly
Simple,
Fully automated,
Easily implemented.
Low efficiency,
Low probability of
selecting feasible
sequences,
Not sufficient for
testing complicated
software.
For evaluating other
approaches,
Beside other
approaches.
White BoxGenerate test cases
using the internal
structure of
Software
Easily implemented Postponing the test
case generation
process to software
implementation,
Hard to correct
detected errors.
Structural testing
Black BoxGenerate test cases
using the
specification of
software
Considering user
requirements,
Testing software in
first steps of
software
development cycle,
Easy to correct
detected errors.
Specifications are
complicated,
Implementation
should be a direct
mapping of
specification,
Hard to present an
accurate definition
for specification.
Functional testing
User Session
Based
Generate test cases
by analyzing user
logs on the Web
server
Are suitable for real
world problems
since user sessions
capture the real
interactions of users
with the web
application.
Redundant data
among user sessions.
Regression testing
3.4. Automatic Test Case Generation for Modern Web Applications
For these applications, test cases are sequences of events that are potentially fault prone [30]. The
testing techniques applied to traditional multipage web applications are not sufficient for testing
modern single page web applications. In more detail, the properties of Ajax make them extremely
difficult to test. Static analysis techniques are not able to reveal many of the dynamic dependencies
present in modern web applications. The highly dynamic nature of Ajax user interfaces and their
client/server delta communication adds an extra level of complexity to the traditional web analysis and
testing challenges [9]. Furthermore, traditional web testing techniques are based on the traditional page
request/response model, and not taking into account client side functionality [31]. In [18] the adequacy
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
31
and effectiveness of the most famous web testing methods are applied to these modern web
applications and results shows that they are inadequate for testing AJAX based web applications. The
classification framework for automatic test case generation methods for modern web application is
shown in figure 5. Since Ajax Web applications are heavily based on asynchronous messages and
DOM manipulation, the faults associated with these two features are relatively more common than in
other kinds of applications. Thus, most Ajax testing techniques are directed toward revealing faults
related to incorrect manipulation of the DOM [6]. Table 2 shows the summarization of different
approaches for Automatic test case generation for modern web application.
Figure 5. Automatic test case generation methods for modern web applications
3.4.1. Random Based
The test cases i.e. sequences of events are generated randomly. In other words, there is a random
walk through the web application. This is not a strategic way of testing software, However can be
useful beside other techniques.
3.4.2. Model Based
To bridge the gap between current web testing and the main new features provided by AJAX, [18]
proposed a Model-based testing technique represented by a FSM to describe the behavior of an AJAX
Web page, according to its DOM structure and content in which nodes represent DOM states and edges
are events which modify the DOM. An example is shown in figure 6 for cart web application. States in
the model represent the set of DOM configurations that can be reached by the client-side pages of the
application during its execution. Test cases are obtained by extracting some of the state machine paths
according to a specific coverage criterion. This method contains four main steps: FSM construction,
Path extraction, Test cases generation which converses FSM paths into test cases, and Test cases
execution. To extract the FSM model of a web application, different methods have been proposed
using execution traces, crawlers or user sessions.
Figure 6. FSM of cart web application
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
32
1) Using execution traces
A set of executions is needed to construct the initial FSM. Although the implementation is simple,
this approach suffers from state explosion problem. On the other hand, the FSM produced by execution
traces is incomplete and may miss some states and transitions which are never exercised in the
considered scenarios. Hence, some manual step is sometimes needed in which the user augments the
resulting FSM with missing states and transitions [6].
This method is first proposed by [18] which generate test cases from FSM model of Ajax based web
applications. Test cases are obtained by extracting some of the state machine paths according to
transitions coverage. In [6] Test cases are derived from the state model in a same way as previous
method. However, the coverage criterion for this method is the number of semantically interacting
events, and resulted to more efficient test cases. Events e1 and e2 interact semantically if there is a state
S0 such that their execution in S0 does not commute which means that the following conditions hold:
On the other hand, search space of the semantically interacting event sequences is huge, as it can
grow exponentially with the event sequence length. Hence in [32], search-based algorithms, namely
hill climbing and simulated annealing, are applied to the problem of generating maximally diverse
event sequences of various lengths in order to select those that are most promising, based on a measure
of test case diversity. In other terms, they re-formulate the problem of test case generation as an
optimization problem which can be solved by applying heuristic search algorithms guided by an
objective function which select only the most diverse test cases, without any constraint on their length.
Four types of objective functions, based on the notion of test diversity were proposed:
ED: the execution frequency of each event which labels a transition in the FSM model exercised by
the test cases of a test suite; EDm (Eq. 2) is the diversity of test suit which is measured by computing
the minimum distance between the event frequencies of each test case and the event frequencies of
other test cases in the same test suite. EDa (Eq. 3) is the diversity of test suit which is measured by
computing the distance from the average frequencies.
PD: the execution frequency of each pair of semantically interacting events labeling FSM
contiguous transitions exercised by each test case of a test suite;
(1)
(2)
(3)
(4)
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
33
TC: the FSM-event coverage reached by the test cases in a test suite;
TL: the semantically interacting events sequences length in the test cases of a test suite;
2) Using Crawlers
Another way of extracting a FSM Model from a web application is the use of Crawlers, clicking
buttons and other UI-elements, and generating a test suite covering the paths obtained during crawling.
Although this is an efficient method of extracting FSM, but there is no direct way of obtaining all
clickable elements in a DOM-tree, therefore the resulted FSM may be incomplete.
Mesbah et al. [33] have proposed an approach that infers a graph of client-side interface states
using an automatic crawler (CRAWLJAX ), automatically clicking buttons and other UI-elements, and
generating a test suite covering the paths obtained during crawling for identifying specific types of
fault that can occur in those states, thus exercising the client-side UI functionality. Figure 7 shows the
Processing view of the CRAWLJAX architecture. The algorithm makes use of a set of candidate
elements, which are all exposed to an event type such as click, MouseOver, etc. After finding the
candidate elements, the algorithm proceeds to determine whether these elements are clickable. For each
candidate element, the crawler executes an event type on the element in the browser.
(5)
(6)
(2)
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
34
Figure 7. The processing view of the CRAWLJAX architecture [33]
Amalfitano also used a crawling algorithm called CrawlRIA to obtain the FSM of rich internet
applications and generate test cases using the extracted model [15]. CrawlRIA is a prototype crawler
that automatically generates execution traces of an RIA by triggering the events found in rich internet
application interfaces by either a depth first or a breadth first visiting strategy; moreover it is able to
cluster interfaces and events to generate an FSM model.
3) Using User Sessions
This approach has been already applied with success both for traditional and modern Web
application testing and aims at automatically generating test cases composed of event sequences which
are deduced by analyzing user interactions with a version of the application. The obtained test cases
can be replayed for generating a FSM model of application which can be used for deriving test cases
automatically. However, there is some redundant or unimportant information in data captured from
user sessions. Therefore in this approach there is a need for data reduction instead of exhaustive search
in all user sessions. Due to the fact that these data capture the real interactions of users with the web
application, it seems to be an appropriate method for testing the real world applications.
Amalfitano [15] defined a testing technique that exploits concrete execution traces of an application,
produced by real users (or tester users) using a tool called CReRIA, to transform traces into executable
test cases. CReRIA is an interactive tool for dynamic analysis that supports the semi-automatic reverse
engineering of a Finite State Machine (FSM) modeling the behavior of an Ajax application user
interface. The tool offers an integrated Web browser which allows a user to navigate the RIA and to
trace and record his user sessions.
3.4.3. GUI Based
Although Ajax application has specific features which make it different from traditional GUI
applications [6] there are some GUI testing techniques which claim to be appropriate for testing Ajax
based web applications.
1) Agent Based
It is a test case generation technique that builds the GUI model and produces the test cases
incrementally, while interacting with the application under test. In this method the problem of
generating test cases for an unknown application is turned into the problem of an agent that must learn
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
35
how to effectively act in an unknown environment. This method does not rely on an initial set of
executions; therefore it does not suffer from the limitations of pre mentioned approaches. This
technique is used for functional testing of modern web applications. AutoBlackTest tool [34] uses
reinforcement learning, in particular Q-Learning, to learn how to interact with the application under
test and stimulate its functionalities. The obtained results show that AutoBlackTest can automatically
generate test cases that cover a relevant portion of the statements in the target applications.
2) Optimization Based
In this technique, the task of automatically generating test cases is transformed into an optimization
problem. The main idea is to define a quality criterion i.e. a fitness function and search for test cases
which maximize this value. Due to the fact that the search space of all possible test cases is often huge
and has a complex structure, one could try to exploit metaheuristic techniques. [35] Presents an
approach for finding input sequences (test cases), using a metaheuristic algorithm named Ant Colony
Optimization for the purpose of GUI testing. The information gathered from scanning GUI is used to
derive the set of alternative actions. Considering the information learned from the fitness values of
earlier sequences, promising action is then selected.
Table 2. Automatic test case generation approaches for Modern web applications
Approach Main Idea Pros Cons Domain
Random BasedGenerate test cases
randomly
Simple,
Fully automated,
Easily implemented.
Low efficiency,
Low probability of
selecting feasible
sequences,
Not sufficient for
testing complicated
software.
For evaluating
other approaches,
Beside other
approaches
Model BasedGenerate test cases
using Finite State
Machine extracted
from web
application
Easily implemented,
Representing an
understandable
model of web
application.
State explosion
problem,
Incomplete state
machines,
Manual efforts
needed to complete
state machines.
Structural testing,
Functional testing
GUI Based Generate test cases
using GUI testing
of desktop
applications
approaches
Covers most of
applications states
and transitions
Not sufficient to
test all aspects of
modern web
applications that
have specific
characteristics
which differs from
desktop
applications
Functional testing
4. Discussion
In this part we have a brief look at both traditional and modern web application test case generation
approaches and compare those using different criteria. Table 3 shows comparison results for automatic
test case generation of traditional web applications and table 4 shows the same result for modern web
applications.
Table 3. Automatic test case generation approaches for traditional web applications
Complexity
Coverage
Supporting safety-
critical software
Support real-
time software
approach
Low
LowNoYes
Random
Middle
HighYesNo
White box
High
High Yes No
Black box
High
HighYesNo
Session
based
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
36
Table 4. Automatic test case generation approaches for modern web applications
Complexity
Coverage
Supporting
safety-critical
software
Support real-time
software
approach
Low
LowNoYes
random Middle
High No No
Model
based High
MiddleYesNo
GUI based
As table 3 shows Random based approach is appropriate for testing real-time software, because it
can generate test cases very quick, however the obtained test cases by this approach are not reliable
enough to be used for testing safety-critical software because it leads to low coverage. Other three
approaches are not quick enough to be utilized for testing of real-time software because of their high
complexity; however, they can be improved and customized for safety-critical software testing purpose.
The information in table 4 for modern applications also shows that only random based approach is
appropriate for real-time software testing, since other methods haven’t got efficient enough to be as
quick as random one. In addition to random approach the Model based method is not reliable to test
safety-critical software because the approaches in this group don’t cover all the states of the application
and there may be an unpredictable state. The GUI based technique can be improved for safety-critical
software testing purpose.
5. Conclusion
This paper reviews different methods proposed for automatic test case generation for both
traditional and modern web applications. Modern web applications are analyzed apart from traditional
ones, since the existing methods for test case generation of traditional web application are not
applicable for modern web applications. The advantages and disadvantages of each technique are
illustrated with different examples and some defined criteria are presented for comparing various
approaches. Clearly, automatic test case generation is a wide open area of research, especially for new
generation of web applications with their specific features which bring new challenges to the task of
test case generation. As the researches in this paper shows there haven’t been an efficient and complete
method which covers all of the defined criteria. Our feature work is to propose an optimize method of
testing modern web applications to overcome the existing challenges of current methods.
6. Acknowledgment
This work is supported by Research institute for ICT under Grant T/500/19236. The authors are
grateful to anonymous referees of this paper for their constructive comments.
7. References
[1] C. Kung, David; Liu, Chien-Hung ; Hsia, Pei ;, "An Object-Oriented Web Test Model for Testing
Web Applications," in Computer Software and Applications, 2000.
[2] Z. Qian , "User Session-Based Test Case Generation and Optimization Using Genetic Algorithm,"
J. Software Engineering & Applications, vol. 3, no. 6, pp. 541-547 , 2010.
[3] B. Garc´a and J. C.Due'nas, "Automated Functional Testing based on the Navigation of Web
Applications," in WWV 2011, 2011.
[4] G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura and Z. Su, "Dynamic Test Input
Generation for Web Applications," in international symposium on Software testing and analysis,
New York, 2008.
[5] X. Jia and H. Liu, "Rigorous and Automatic Testing of Web Applications," in IASTED
International Conference on Software Engineering and Applications, SEA, 2002.
[6] A. Marchetto, P. Tonella and F. Ricca, "State-Based Testing of Ajax Web Applications," in
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
37
International Conference on Software Testing, Verification, and Validation, Lillehammer, 2008.
[7] Y. Huang and L. Lu, "A Methodology for Test Suit Reduction in User-session-
b
ased Testing,"
Journal of Convergence Information Technology, vol. 5, no. 9, pp. 864- 868, 2010.
[8] N. Kosindrdecha and J. Daengdej, "A Test Generation Method Based On State Diagram," Journal
of Theoretical and Applied Information Technology, vol. 18, no. 2, pp. 28-44, 2005.
[9] A. Mesbah, Applications, Analysis and Testing of Ajax-based Single-page Web, Karaj: TuDelft,
2009.
[10] P. N. Boghdady, N. L. Badr, M. Hashem and M. F. Tolba , "Test Case Generation and Test Data
Extraction Techniques," International Journal of Electrical & Computer Sciences IJECS-IJENS,
vol. 11, no. 3, pp. 87-94, 2011.
[11] S. Elbaum, G. Rothermel, S. Karre and M. Fisher II, "Leveraging User-Session Data to Support
Web Application Testing," IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, vol. 31,
no. 3, pp. 187-202, 2005.
[12] A. Arora and . M. Sinha, "Web Application Testing: A Review on Techniques, Tools and State o
f
Art," International Journal of Scientific & Engineering Research, vol. 3, no. 2, 2012.
[13] Y.-H. Tung, S.-S. Tseng, T.-J. Lee and J.-F. Weng, "A Novel Approach to Automatic Test Case
Generation for Web Applications," in International Conference on Quality Software, Zhangjiajie,
2010.
[14] P. Frankl and J. Wang, "Testing web database applications," Workshop on Testing, Analysis and
Verification of Web Services, vol. 29, no. 19, 2004.
[15] D. Amalfitano, "Reverse Engineering and Testing of Rich Internet Applications," Ph.D Thesis,
2011.
[16] M. Papadakis and N. Malevris , "Automatic Mutation Test Case Generation Via Dynamic
Symbolic Execution," in IEEE 1st international Symposium on Software Reliability Engineering,
California, 2010.
[17] W. T. Tsai, D. Volovik, T. Keefe and M. E. Fayad, "Automatic Test Case Generation from
Relational Algebra queries," IEEE Transactions on Software Engineering, vol. 16, no. 3, 1990.
[18] A. Marchetto, . F. Ricca and P. Tonella, "A case study-
b
ased comparison of web testing
techniques applied to AJAX web applications," Int J Softw Tools Technol Transfer, vol. 10, no. 6,
p. 477
–
492, 2008.
[19] . C. Cadar, P. Godefroid, . S. Khurshid, . C. S. Pasareanu, . K. Sen, N. Tillmann and W. Visser,
"Symbolic Execution for Software Testing in Practice–Preliminary Assessment," in International
Conference on Software Engineering, Honolulu, 2011.
[20] S. Artzi, J. Dolby, F. Tip and M. Pistoia, "Directed Test Generation for Effective Fault
Localization," in international symposium on Software testing and analysis, Trento, 2010.
[21] C. Hung Liu , D. C. Kung, P. Hsia and C.-T. Hsu , "Object-Based Data Flow Testing of Web
Applications," in The First Asia-Pacific Conference on Quality Software, Hong Kong, 2000.
[22] Z. Dai and M.-H. Chen , "Automatic Test Case Generation for Multi-tier Web Applications," in
9th IEEE International Workshop on Web Site Evolution, Paris, 2007.
[23] J. J. Gutiérrez , M. J. Escalona , M. Mejías and J. Torres , "An Approach to Generate Test Cases
from Use Cases," in the 6th international conference on Web engineering, 2006.
[24] G. deep and D. J. Sengupta , "Automatic Generation of Regression Test Cases for Web
Components using Domain Analysis and Modeling," International Journal of Computer
Applications, vol. 11, no. 12, pp. 14-17, 2010.
[25] S. Sampath, V. Mihaylov, A. Souter and L. Pollock, "A Scalable Approach to User-session based
Testing of Web Applications through Concept Analysis," in the Automated Software Engineering
Conference, 2004.
[26] S. Bošnjak, M. Marić and Z. Bošnjak, "The Role of Web Usage Mining in Web Applications
Evaluation," Management Information Systems,, vol. 5, no. 1, pp. 31-36, 2010.
[27] X. Peng and L. Lu, "User-session-based automatic test case generation using GA," International
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
38
Journal of the Physical Sciences, vol. 6, no. 13, pp. 3233-3245, 2011.
[28] J.-T. Yang, J.-L. Huang, F.-J. Wang and W. C. Chu, "Constructing Control-Flow-Based Testing
Tools for Web Application," in Software Engineering and Knowledge Engineering Conference
(SEKE), 1999.
[29] X. Fu, "Relational Constraint Driven Test Case Synthesis for Web Applications," in TAV-WEB,
2010.
[30] D. Turner, M. Park, J. Kim and J. Chae, "An Automated Test Code Generation Method for Web
Applications using Activity Oriented Approach," in 23rd IEEE/ACM International Conference on
Automated Software Engineering, 2008.
[31] A. Mesbah, A. v. Deursen and D. Roest, "Invariant-Based Automatic Testing of Modern Web
Applications," Software Engineering Research Group, Department of Software Technology,
Faculty of Electrical Engineering,Mathematics and Computer Science, Delft University o
f
Technology, 2011.
[32] A. Marchetto and P. Tonella, "Using search-
b
ased algorithms for Ajax event sequence generation
during testing," EmpirSoftwareEng, vol. 16, no. 1, p. 103–140, 2011.
[33] A. Mesbah, A. Van Deursen and D. Roest, "Invariant-Based Automatic Testing of Modern Web
Applications," IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 35-53, 2012.
[34] L. Mariani, M. Pezz`, O. Riganelli and S. Mauro, "AutoBlackTest:Automatic Black-Box
Testingof Interactive Applications," in IEEE Fifth International Conference on Software Testing,
Verification and Validation, Montreal, 2012.
[35] S. Bauersfeld, "A Metaheuristic Approach to Automatic Test Case Generation forGUI-Based
Applications," Humboldt university, 2011.
A Classification Framework for Automatic Test Case Generation Techniques for web applications
Mohammad Reza Keyvanpour, Hajar Homayouni, Hosein Shirazee
39
