ArticlePDF Available

Abstract and Figures

New technologies for the development of Web applications, such as AJAX, support advanced, asynchronous interactions with the server, going beyond the submit/wait-for-response paradigm. AJAX im-proves the responsiveness and usability of a Web application but poses new challenges to the scientific community: one of them is testing. In this work, we try to apply existing Web testing techniques (e.g., model based testing, code coverage testing, session based testing, etc.) to a small AJAX-based Web application with the purpose of understanding their real effectiveness. In particular, we try to answer the following ques-tions: "Is it possible to apply existing testing techniques to AJAX-based Web applications?"; "Are they adequate to test AJAX applications?"; and, "What are the problems and limitations they have with AJAX test-ing?". Our preliminary analysis suggests that these techniques, especially those based on white-box approaches, need to be changed or improved to be effectively used with AJAX-based Web applications.
Content may be subject to copyright.
Testing Techniques applied to
AJAX Web Applications
Alessandro Marchetto
1
, Paolo Tonella
1
, and Filippo Ricca
2
1
Fondazione Bruno Kessler - IRST, 38050 Povo, Trento, Italy
marchetto|tonella@itc.it
2
Unit`a CINI at DISI
?
, 16146 Genova, Italy
filippo.ricca@disi.unige.it
Abstract. New technologies for the development of Web applications,
such as AJAX, support advanced, asynchronous interactions with the
server, going beyond the submit/wait-for-response paradigm. AJAX im-
proves the responsiveness and usability of a Web application but poses
new challenges to the scientific community: one of them is testing. In
this work, we try to apply existing Web testing techniques (e.g., model
based testing, code coverage testing, session based testing, etc.) to a
small AJAX-based Web application with the purpose of understanding
their real effectiveness. In particular, we try to answer the following ques-
tions: “Is it possible to apply existing testing techniques to AJAX-based
Web applications?”; “Are they adequate to test AJAX applications?”;
and, “What are the problems and limitations they have with AJAX test-
ing?”. Our preliminary analysis suggests that these techniques, especially
those based on white-box approaches, need to be changed or improved
to be effectively used with AJAX-based Web applications.
Keywords: AJAX-based Web applications, Web te sting techniques.
1 Introduction
Traditional Web applications are based on the client-server model: a browser
(client) sends a request asking for a Web page over a network (Internet, via
the protocol HTTP) to a Web server (server), which returns the requested page
as response. During this elaboration time the client must wait for the server
response before visualizing the requested page on the browser, i.e., the model is
based on synchronous communications between client and server.
During the last few years, several Web applications (such as Google Suggest,
Yahoo Instant Search, Google and Yahoo Mail, etc.) have been developed using
a new technology named AJAX [11, 4]. AJAX breaks the traditional Web page
paradigm, in which a Web application can be thought of and modeled as a
graph of Web pages, and invalidates the model of traditional Web applications.
It introduces additional asynchronous server communication to support a more
?
Laboratorio Iniziativa Software FINMECCANICA/ELSAG spa - CINI
responsive user interface: the user interacts directly with items within the page
and the feedback can be immediate and independent of the server’s response.
The validation of Web applications that are based on asynchronous commu-
nication with the server and employ new technologies, such as AJAX, Flash,
ActiveX plug-in components, is an area that deserves further investigation. For
example, until now, little rese arch efforts have focused on how to test Web appli-
cations employing AJAX. Since AJAX does not c omply with the classical Web
application model, several techniques presented in literature will not work any
longer. This s ee ms to be the case of model based Web testing: using a Web
Crawler
3
to extract the model seems not possible anymore. Before devising new
techniques specific of AJAX, we think it is imp ortant to address the following
questions: “Which testing techniques are able to work with AJAX-based Web
applications?” and, “To what degree?”
In this paper, we apply some existing testing techniques to a simple Web
application based on the AJAX technology. Our purp ose is to detect effectiveness
and advantages as well as limitations and problems, of each examined testing
technique.
The rest of the paper is organized as follows. We start, in Section 2, explain-
ing the main characteristics of AJAX. Section 3 presents the existing testing
techniques and tools examined. In Section 4 we analyze each technique in order
to understand and evaluate its applicability to AJAX-applications. In the same
Section we apply the analyzed techniques to a sm all AJAX-application. Section
5 summarizes our preliminary res ults and, finally, in Section 6 we conclude the
paper.
2 AJAX
AJAX (Asynchronous Javascript And XML) is a bundle of existing technolo-
gies used to simplify the implementation of rich and dynamic Web applications.
HTML and CSS are used to present the information, the Document Object
Model is used to dynamically display and interact with the information and the
page structure, the XMLHttpRequest object is exploited to retrieve data from the
Web server, XML is used to wrap data and Javascript is exploited to bind “every-
thing together” and to manage the whole process. With AJAX developers can
implement asynchronous communications between client and server. To achieve
this, client-side scripts and a special AJAX component named XMLHttpRequest
are used. Thanks to AJAX, Web developers can update parts of the client-side
page independently: in AJAX the units of work are the page elements (e.g., text
area, HTML form, DOM structure) rather than the whole page, as happening
with traditional page-based Web applications. For this reason, AJAX breaks the
traditional Web page paradigm. Every element of an HTML page may be tied to
some AJAX action; every action may generate a server request, associated with
3
A Web crawler (also known as a Web spider or robot) is a program that automatically
traverses the Web’s hyperlink structure and retrieves some information for the user.
a URL, so that many HTTP requests can be performed by a single client-side
page.
The main technological novelty of AJAX is the XMLHttpRequest object used
to exchange request and data, wrapped into XML packages, between Web com-
ponents. In particular, through this object a client-side component (e.g., HTML
page) can send HTTP requests to the Web se rver and capture its response in
order to update pieces of the same HTML page. XMLHttpRequest allows to send
asynchronous GET/POST HTTP requests to a given Web server without show-
ing any visible effect to the user and, more importantly, without stoping the
component execution, since a special AJAX engine controls HTTP requests and
responses in background, using an event listener. In other terms, it allows Web
developers to specify event handlers that change the client-side component state
whenever a server response is received asynchronously.
3 Web Testing Techniques
Existing approaches for Web Application testing can be divided into three classes:
white-box, black-box and session-based testing.
3.1 White-Box testing
Similarly to traditional software, white-box testing of Web applications is based
on the knowledge about the internal structure of the system under test. This
approach can be applied to Web applications either by representing the structure
at the high-level, by means of the navigation model (Model-based testing), or
at the low-level, by means of the control flow model (Code-coverage testing).
In the white-box category we consider also Mutation-based testing, which
requires internal knowledge of the Web application under test.
1. Model-based testing [1, 3, 6]. In this approach, reverse engineering tech-
niques and a Web crawler are used to build the Web model of the target
application. The built model is a graph where each node represents a Web
page and each edge represents a link (e.g., HTML links, submits, automatic
redirections). Test cases are extracted by traversing the navigational system
(i.e., the Web model) of the application under test. A test case is composed
of a sequence of pages plus input values.
2. Code-coverage testing [10]. This approach is based on knowledge of the
source code of the Web application under test [10]. In code coverage testing,
the level of coverage reached by a given test suite can be determined by
instrumenting (through trace instructions) the branches in the control flow
model. Since the execution on the server involves one (or more) server side
languages (e.g., Java, PHP, SQL) and the execution on the client involves
additional languages (such as HTML and JavaScript), the control flow model
has different node kinds. Examples of tools used to pe rform code-coverage
testing for Java software are Emma
4
and C lover
5
.
3. Mutation-based testing [2, 8]. Code mutation has bee n used in the lit-
erature for s everal purposes. In the context of Web applications, it can be
used to recover the Web model [2]. In this approach, code mutation is ap-
plied to the server-side code in order to automate the difficult task of model
construction. More traditional uses of mutation consist of applying mutation
operators to the source code in order to generate code mutants. Mutants are
exercised through suites of test cases in order to evaluate their effectiveness
in finding faults. For instance, Elbaum et al.[8] use a fault-based approach
to evaluate the effectiveness of test s uites constructed by means of a session-
based approach. Finally, another use of this defect-injection method is the
selection a subset of test cases, based on a fault-coverage criterion.
3.2 Black-Box testing
The test of the functional requirements can be conducted by considering the Web
application as a black-box. Web applications may have documents describing
the requirements at the user level, such as: use-cases, user stories, functional
requirements in natural language, etc. From these documents, it is possible to
create a list of test cases. Each test case, i.e., ordered list of Web pages plus
user inputs, describes a scenario that can be accomplished by a Web visitor
through a browser. Output pages obtained by navigating the application and
providing the requested inputs are compared with the results expected from
that interaction according to the requirements. In this work, we consider two
black-box approaches: capture/reply and xUnit.
1. Capture and Reply. The most common class of black-box testing tools
provide an infrastructure to support the capture and replay of particular user
scenarios. During black-box testing of Web applications the interaction with
the user c an be simulated by generating the graphical events that trigger
the computation associated with the application interface. One of the main
methods used to obtain this result consists of recording the interactions that
a user has with the Web application and repeating them during the (regres-
sion) testing phase. A lot of functional and regression testing tools, based
on capture/replay facilities, are available as free and commercial software.
Examples are Mercury WinRunner
6
, IBM Rational Robot
7
and MaxQ
8
.
2. xUnit testing. Another approach to black-box testing is based on HttpUnit
9
.
When combined with a framework such as Junit
10
, HttpUnit permits pro-
4
http://emma.sourceforge.net
5
http://www.cenqua.com/clover
6
http://www.merc-int.com
7
http://www-306.ibm.com/software/awdtools/tester/robot
8
http://www.bitmechanic.com
9
http://httpunit.sourceforge.net
10
Http://www.junit.org
grammers to write Java test cases that check the functioning of a Web appli-
cation. HttpUnit is a Java framework well suited for black-box and regression
testing of Web applications, which allows the implementation of automated
test scripts based on assertions. In the xUnit family are also other tools, such
as JsUnit
11
and PHPUnit
12
. Several xUnit tools (e.g., Selenium
13
) use the
capture and reply mechanism to record scripts of test. These scripts can be
completed and enriched with assertions by the user.
3.3 Session-based testing
Another approach to testing Web applications is user-session based testing. It
relies on capturing and replaying real user sessions. This approach avoids the
challenge of building an accurate model of a Web application’s structure. Each
user session is a collection of user requests in the form of URL and name-value
pairs (i.e., input field names and values). A user session begins when a user makes
a new request to a Web application and ends when the user leaves it or the session
times out. To transform a user session into a test case, each logged request is
changed into an HTTP message that can be sent to the Web server. A test case
consists of a set of HTTP requests that are associated with each user session.
Different strategies can be applied to construct test cases from the collecte d
user sessions stored in a log file [5]. The simplest strategy is transforming each
individual user session into a test case. Other, more advanced, strategies are also
possible [7]. An example of access log is given in Figure 5.
4 Testing of AJAX-based Web Applications
In this section we analyze white-box, black-box and session-based testing in
terms of their applicability to AJAX-applications, evaluating limitations and
problems, considering existing tools and applying them to a small “case study”.
Fig. 1. customerID UML model
11
http://www.jsunit.net
12
http://www.phpunit.de
13
http://www.openqa.org/selenium
Fig. 2. customerID screenshot
For our study we selected a small application named customerID
14
. Although
small, it represents a typical AJAX application, which can be analyzed, tested
and described in detail. Figure 1 shows the UML class diagram of this applica-
tion. The custom erID application is composed of:
a c lient-side HTML page (client.html );
an HTML form (customer ) used by client.html;
a Javascript code fragment (JS ) used by client.html;
a s erver-side page Customer.jsp.
JS retrieves the customer identification number written by the user (see Figure 2)
in the HTML form and uses an XMLHTTPRequest object (activated by the user
through the “submit” button of the form) to send this number to the server-side
page Customer.jsp. Customer.jsp is a JSP component that receives a customer
id from the client and returns first and last name of the customer associated
with the received id, using a XML-based object for data transmission. Then,
the client-side component JS captures this XML package and uses the contained
data to update the client page. In that operation, a fragment of the Javascript
code JS is used to get the data and to update the state of the HTML page
client.html.
Fig. 3. Model-based testing applied to customerID
4.1 Model-based testing
Model-based testing are only partially usable to test AJAX applications, since
the associated model is not adequate for this kind of applications. Existing
14
http://www.ics.uci.edu/cs122b/projects/project5/AJAX-JSPExample.html
model-based techniques describe a Web application through a navigation model
composed of Web pages and hyperlinks. The problem is that AJAX applications
are essentially composed of a single page that changes its state according to the
user interactions. Other problems with the model-based approach are: First, ex-
isting Web crawlers, used to build the Web model, are not able to extract the set
of dynamic links created by AJAX applications. Second, client and server com -
ponents tend to exchange small XML data fragments instead of entire HTML
pages. This pec uliarity of AJAX makes it impossible to capture the server re-
sponse in order to extract the hyperlinks of the navigation model, as done in
conventional model-based testing. Third, existing Web models don’t capture the
evolution of a Web page in terms of successive states (i.e. DOM configurations)
and they are not able to represent the XML data exchanged among components.
Summarizing, to perform model-based testing of AJAX-based applications one
needs to:
1. improve the “capability” of the actual Web crawlers;
2. extend the Web model.
These limitations are evident when we apply model-based testing to cus-
tomerID. The model pro duced by the Web crawler is partial (Figure 3 shows
the model extracted according to Ricca and Tonella [6]). It is composed only
of the page client.html, its HTML form and the page customer.jsp. The result
of executing customer.jsp on the server is missing in the model because XML
packages cannot be represented in this model.
4.2 Mutation-based testing
Two studies[2, 9] apply mutation-based testing to Web applications. Bellettini
et al. [2] use mutation to recover the Web mo del, while Sprenkle et al. [9] use
mutation to evaluate the effectiveness of a test suite by inserting artificial defects
(mutations) into the original application. No work to the best of the authors’
knowledge tried to apply mutation to the client-side code of Web applications.
The definition of mutation operators for AJAX could be quite complicated, due
to the possibility of run-time changes of the DOM structure and page content.
We think that mutation-based testing may be useful to support the testing
approaches for AJAX applications. This involves studying and defining AJAX-
specific mutation operators.
We tried to apply the technique proposed by Bellettini et al. [2] to cus-
tomerID, but we found a problem: This testing technique cannot be directly
used with customerID because it applies mutation operators only to server-side
pages and because the response of the server is an XML package and not an
entire page, as expected by the technique.
4.3 Code Coverage testing
In theory, it is possible to apply code coverage testing to AJAX-based applica-
tions. In practice there are some problems. The first is a technological problem.
Fig. 4. Output of our co de-coverage tool applied to client.html
Coverage tools for Web applications need to trace Web code that, often, is a mix
of languages such as HTML, Javascript, JSP and currently, tools with these char-
acteristics are not available. Moreover, currently, it is impossible to use existing
coverage tools to trace dynamic changes of the DOM structure or dynamically
generated code. We think that this list of problems limits in practice the effec-
tiveness of the code coverage testing approach.
Some tools such as Cobertura
15
, JCover
16
, Clover
17
and Emma
18
cannot be
used to test complex and real Web applications bec ause they can trace and
analyze only Java (i.e., server-side) code. An example of code coverage tool for
Javascript is the commercial tool Coverage Validator
19
. It displays statistics for
each Javascript file that is being monitored for code coverage.
A coverage tool able to work with a mix of Web languages is under devel-
opment at our laboratory. Figure 4 shows the output of our tool applied to the
client.html page of the customerID. The code (HTML and Javascript) exercised
during test case execution is traced and shown in light gray. The tool reports
(see top of the figure) som e code-coverage metrics such as condition (60%) and
statement coverage (80%).
When applying our tool to customerID, some problems become evident. It is
difficult to trace dynamic code and dynamic changes of Web pages (e.g., DOM
changes). For instance, we cannot trace the run-time changes of the HTML form
embedded in the client.html page. The reason is that the AJAX component, used
by the same page, dynamically updates the form (name and lastname fields in
Figure 4) through asynchronous communications with the server page. Hence
dynamic changes of the form field values (lines 38-39 on Figure 4) cannot be
traced by the coverage tool.
127.0.0.1 - - [04/Dec/2006:11:31:25 +0100] “GET /ajax1/client.html HTTP/1.1” 200 1916
127.0.0.1 - - [04/Dec/2006:11:31:27 +0100] “GET /ajax/customer.jsp?customerID=134
HTTP/1.1” 200 29
127.0.0.1 - - [04/Dec/2006:11:32:03 +0100] “GET /ajax1/ HTTP/1.1” 200 1563
127.0.0.1 - - [04/Dec/2006:11:32:04 +0100] “GET /ajax1/client.html HTTP/1.1” 200 1917
127.0.0.1 - - [04/Dec/2006:11:32:07 +0100] “GET /ajax1/customer.jsp?customerID=245
HTTP/1.1” 200 29
Fig. 5. Tomcat log-file of customerID
4.4 Session-based testing
This testing approach is apparently applicable to AJAX applications, because in
a log-file we can capture both kinds of HTTP requests: traditional (triggered by
15
http://cobertura.sourceforge.net
16
http://www.mmsindia.com/JCover.html
17
http://www.cenqua.com/clover
18
http://emma.sourceforge.net
19
http://www.softwareverify.com/javascript/coverage/feature.html
user clicks), as well as AJAX-sp e cific (triggered by AJAX-components). How-
ever, session-based testing techniques are adequate to verify only non-interacting,
synchronous request-response pairs, because of two kinds of limitations:
1. Using (only) log-files information it is not possible to reconstruct the state
of Web pages that are modified during the execution of a given application.
The reason is that the data exchanged between client and server in a given
AJAX application are only “pieces of data” wrapped into XML messages,
that are eventually turned into page changes;
2. Some techniques used to mix log file information (i.e., sequence of hyperlinks
and input values) cannot be used to generate new navigation sess ions (i.e.,
new sequences of links and input values) useful to e xercise the application
under test. In fact, it may be hard to reproduce the context where the log
information used in the original application can be re-inserted in a different
scenario.
Figure 5 shows a fragment of the Tomcat log-file for the customerID ap-
plication. Given this log file, we can reply the two customerID usage sessions
performed by the user but we cannot take advantage of the information in the
XMLHTTPRequest object used by customerID. For this reason, we can not re-
peat its behavior. Moreover, we can use the log information to verify each single
response to the HTTP requests stored in Tomcat log files. But, we cannot use
it to tes t the entire application customerID, since it is impossible to reconstruct
the state of the application when each HTTP request was issued and when the
response was received. Furthermore, by mixing the log-extracted HTTP requests
we might end up with inconsistent requests, since knowledge of the AJAX-objects
used by customerID cannot be derived from log-files only. Some examples of test
cases derived from the log-file in Figure 5 are the following:
1. GET client.html verify the HTML code s ent by the server on response;
2. GET client.html send ID=134 to customer.jsp using a GET request
verify the returned XML package;
3. GET client.html verify the HTML code s ent by the server on response;
4. GET client.html send ID=134 to customer.jsp using a GET request
send ID=234 to customer.jsp using a GET request verify the returned
XML package.
4.5 Capture and Reply
Capture and replay is in principle applicable to AJAX-based Web applications,
since it exercises the application from a user point-of-view using the GUI. How-
ever, the real applicability to AJAX-based Web applications depends on the
actual testing tool in use. Several implementations of this technique would need
to be improved to be eff ectively used to test AJAX applications. To be success -
fully applied to AJAX applications, a capture and reply tool should:
1. support Javascript;
function testupdateFirstLastName() {
<?xml version=‘‘1.0’’ encoding=‘‘UTF-8’’?>
<LogiTest:test
xmlns:LogiTest=‘‘http://www.logitest.org’’>
<LogiTest:name>Untitled</LogiTest:name>
<LogiTest:description />
<LogiTest:resetCookies>false</LogiTest:resetCookies>
<LogiTest:resource
url=‘‘http://localhost:8080/ajax1/client.html’’
method=‘‘GET’’ delay=‘‘0’’ />
</LogiTest:test>
Fig. 6. LogiTest applied to customerID
2. be able to capture dynamic events associated with user input;
3. be able to capture dynamic changes of the DOM structures;
4. be able to perform asynchronous HTTP requests to the Web server.
Fig. 7. Selenium applied to customerID
Examples of capture and reply tools are LogiTest
20
, Maxq
21
and Badboy
22
.
These tools don’t work well with AJAX applications. Logitest doesn’t sup-
port Javascript while Maxq cannot record dynamic events. Badboy supports
Javascript but it is not able to capture some dynamic events (e.g., “onblur”
on form input fields) and run-time changes of the DOM structures. Thus, it is
not adequate to test AJAX applications. The commercial tool eValid
23
and the
tool Origsoft
24
promise to test the AJAX applications. Indeed, they are able to
capture dynamic e vents and store dynamic DOM changes.
20
http://logitest.sourceforge.net
21
http://maxq.tigris.org
22
http://www.badboy.com.au
23
http://www.soft.com
24
http://www.origsoft.com
We have tried to apply a tool of this category to customerID. Figure 6 shows
the navigation-script stored by LogiTest for an usage session of our customerID.
It is clear that this script is not adequate to test customerID since it doesn’t
contain information related to the asynchronous HTTP requests performed by
the application.
4.6 xUnit testing
The xUnit approach is also in principle applicable to test AJAX applications.
Actually, it focuses on the functional behavior rather than the implementation.
However, the real applicability to AJAX Web applications depends on the actual
implementation of the tool in use. To be used with AJAX-applications, xUnit
tools must support, at least, Javascript, asynchronous HTTP requests and DOM
inspection.
Examples of xUnit testing tools are: Latka, HTTPUnit, InforMatrix, HTM-
LUnit, JsUnit, Canoo WebTest, squishWeb and Selenium. Latka
25
and HTTPUnit
26
cannot be used to test AJAX-applications because they are not able to manage
asynchronous HTTP requests and DOM inspection. Some tools such as Infor-
Matrix
27
, HTMLUnit
28
and Canoo WebTest
29
have be en recently improved to
support Javascript and AJAX components. Unfortunately, their Javascript sup-
port is still limited and in real AJAX applications they can hardly be applied.
Instead, software such as squishWeb
30
and Selenium
31
can be used to test AJAX
applications, since they fully support Javascript, asynchronous HTTP requests
and DOM inspection.
We have tried to apply some tools of this category to customerID. Infor-
Matrix and HtmlUnit can not be used to test customerID: they don’t support
some DOM-events actions, while they partially support Javascript. Differently,
Selenium, thanks to the assertion “wait conditions”, can be success fully used to
test customerID. Figure 7 shows two screenshots of Selenium. The considered
test case is the following:
1. load client.html type the number “123” in the form ID click the submit
button (i.e., send data to the customer.jsp) verify the form name has been
updated with “John”.
In this example, we use two different assertions to verify the output of the
above test case. In the first case , we use a conventional “assert” command instead
of the correct “waitForValue” (see Figure 7, right). Without this “waitForValue”
25
http://jakarta.apache.org/commons/latka
26
httpunit.sourceforge.net
27
http://www.informatrix.ch
28
http://htmlunit.sourceforge.net
29
http://webtest.canoo.com
30
http://www.froglogic.com
31
http://www.openqa.org/selenium-ide/
the test case verification fails (see Figure 7, left) since the AJAX application cus-
tomerID uses asynchronous communications and thereby a conventional “assert”
cannot be used to capture the server response.
5 Discussion
Testing adequate problems tools customerID
Model-based no Web models extracted research not ok
are partial; existing Web
crawlers are not able
to download site pages
Mutation-based no mutant operators are not-existing not ok
never being applied
to client Web code;
the application of
mutant operators
is difficult
Code Coverage partially it is difficult to cover Javascript: Coverage partially ok
dynamic events and validator
DOM changes; coverage Java: Cobertura,
tools managing a mix Emma, Clover, etc.
of languages are not Languages mix :
available not available
Session-based no it is impossible to research not ok
reconstruct the state
of the Web pages
using only log-files
Capture&Reply yes Javscript, asynchronous not ok : Maxq, it dep end s
and xUnit HTTP requests and HTTPUnit, on the
DOM analysis are InforMatrix etc. tool
not always partially ok : Badboy, implementation
supported HTMLUnit etc.
ok: squishWeb,
Selenium etc.
Table 1. Web testing techniques applied to AJAX-based applications
Table 1 summarizes our preliminary analysis of e xisting testing techniques
applied to AJAX Web applications:
1. model-based: test cases are derived from the Web model. The model doe sn’t
consider all the states that a single HTML page may reach during the exe-
cution of the Web application. So, this technique appears applicable but not
adequate to test AJAX-applications.
2. mutation-based: to our knowledge, no work defines mutation operators
that apply to client-side code of Web applications. Moreover the application
of mutation operators is complicated by the specific nature of the AJAX Web
applications (HTML pages can dynamically change their DOM structure and
content). Additionally, specific tools for Web applications are not available.
So, this technique is promising, but far from being available in practice.
3. coverage-based: this technique is applicable and adequate to test AJAX
applications. However, its real effectiveness has to be verified in practice since
it is difficult to instrument and trace AJAX dynamic code (in particular,
dynamic e vents and DOM changes). Another problem is that coverage tools
for Web applications must trace a mix of languages and currently, tools
with these characteristics are not available. For this reason, this technique
is considered only partially adequate to test this kind of software.
4. session-based: in this technique test cases are derived using the data recorded
in the log-files. Since in a log-file we can capture traditional HTTP requests
as well as AJAX-specific ones, this technique is apparently applicable to test
AJAX applications. However, session-based testing is not fully adequate be-
cause it is hard to reconstruct the state of the Web pages that are exercised
during the exec ution of a given application using only log-files information.
5. capture/reply and xUnit: this category of tools is adequate to test AJAX
applications because it does not consider the internal structure of the tar-
get application. The capture/reply and xUnit tools verify the functionalities
of the Web applications based only on requirements and expected output
(black-box testing). However, often, existing tools don’t support AJAX or
the support is still limited. A tool of this category, to be used with AJAX-
based Web applications should support at least: Javascript, asynchronous
HTTP requests and DOM inspection.
6 Conclusions
In this paper we have applied some existing testing techniques to a simple AJAX
We b application with the purpose of detecting effectiveness and advantages, as
well as limitations and problems, of each examined technique.
The results of our preliminary analysis can be summarized as follows: model
and session-based testing techniques need to be modified and improved to let
them test AJAX Web applications. Mutation-based testing needs to be adapted
to be used with the client-side components of AJAX applications. Code-coverage
testing can be only “partially” used with AJAX-applications: (1) the dynamism
of this kind of technology limits its effectiveness; (2) tools are not available to
manage mix of languages as required. Currently, capture/reply and xUnit testing
are the only tools able to work with AJAX. However, some of them have to be
extended/improved to support Javascript, dynamic changes/events of the DOM
and AJAX-c omponents.
Our preliminary analysis suggests that to test AJAX-applications new ap-
proaches/tools are needed, since the existing ones have severe limitations and
problems, especially white-box and session-based testing techniques.
References
1. A. Andrews, J. Offutt, and R. Alexander. Testing Web Applications by Modeling
with FSMs. Software and System Modeling, Vol 4, n. 3, July 2005.
2. C. Bellettini, A. Marchetto, and A. Trentini. Dynamic Extraction of Web Appli-
cations Mo de ls v ia Mutation Analysis. Journal of Information, 2005.
3. C. Bellettini, A. Marchetto, and A. Trentini. TestUml: User-Metrics Driver Web
Applications Testing. 20th ACM Symposium on Applied Computing (SAC 2005),
Santa Fe, New Mexico, USA. March 2005.
4. J. Eichorn. Understanding AJAX: Using JavaScript to Create Rich Internet Ap-
plications. Prentice Hall, 2006.
5. S. Elbaum, G. Rothermel, S. Karre, and M. Fisher. Leveraging user session data to
support Web Applications Testing. IEEE Transactions on Software Engineering,
Vol. 31, n. 3, March 2005.
6. F. Ricca and P. Tonella. Building a Tool for the Analysis and Testing of Web
Applications: Problems and Solutions. Tools and Algorithms for the Construction
and Analysis of Systems (TACAS’2001), Genova, Italy. April 2001.
7. E. Sampath, S. Gibson, S. Sprenkle, and L. Pollock. Coverage Criteria for Testing
Web Applications. Technical Report 2005-17, Computer and Information Sciences,
University of Delaware., 2005.
8. S. Sprenkle, E. Gibson, S. Sampath, and L. Pollock. Automated Replay and Failure
Detection for Web Applications. 20th IEEE/ACM International Conference on
Automated Software Engineering (ASE 2005), Usa. 2005.
9. S. Sprenkle, E. Gibson, S. Sampath, and L. Pollock. A Case Study of Automatically
Creating Test Suite from Web Applications Field Data. Workshop on Testing,
Analysis and Verification of Web Services and Applications (TAV-WE B 2006),
Usa. 2006.
10. P. Tonella and F. Ricca. A 2-Layer Model for the White-Box Testing of Web
Applications. International Workshop on Web Site Evolution (WSE’04), Illinois,
USA. September 2004.
11. E. Woychowsky. AJAX: Creating Web Pages with Asynchronous JavaScript and
XML. Bruce Perens’ Open Source Series, 2006.
... Traditional web applications used standards such as HTML, XHTML, CSS [15]. Virtual lab experiments are generally content rich [7]. ...
Conference Paper
Full-text available
Biotechnology and biology education has been known to show declining student interest due to classroom environments and instructor teaching styles, hence we introduced virtual labs as an interactive self-learning material in a blended environment. With ICT-based education becoming ubiquitous, virtual and remote triggered labs have become a novel platform that helps users to engage in a proactive learning process. A promisingly new trend in virtual labs-based education is the development of remote laboratories that are available over the internet and can be accessed by students and teachers. We implemented and deployed a low-cost light microscope using a simple front-end to enable users to have anytime-anywhere access. This paper reports the implementation, deployment and user-case studies on the learning and usage based on the remote-triggered virtual lab. This study also focuses on the analysis of using remote-triggered experiments as supplementary laboratory resources for overcoming the problems faced in a traditional lab environment. The study used online feedback surveys for evaluating the learning outcome and the flexibility of user-interactions with the remote labs and reports the status of usage of remote triggered techniques in biology courses. The statistical analysis suggests that remote labs are an easy learning and interactive platform for users from different places.
... In contrast, there is a general field of practice in usability studies; however, most work conducted in computer usability studies is in general areas like design and computer science and is not applied to instructional environments (e.g., Chen, 2001;lehman, 2008;Marchetto, Ricca, & Tonella, 2008;nielsen, 2000;Subraya, 2006). ...
Article
Full-text available
In this paper, data obtained from a university website accessibility and usability validation process are analyzed and used to demonstrate how the design process can affect the online experience for users with disabilities. Interviews, observations , and use data (e.g. where users clicked on a page or what path taken through a site) were collected. Findings indicate that using automated validation tools does not necessarily ensure complete accessibility. Students with low vision found many of the pages hard to use even though automated validation did not indicate issues for visual disabilities. While the pages were accessible for blind users, low vision students who did not use specialized software had access problems. Findings from this study are used to present principles for web designers interested in creating and testing usable and accessible websites.
... Traditional web applications used standards such as HTML, XHTML, CSS [15]. Virtual lab experiments are generally content rich [7]. ...
... Researchers have indicated the usefulness of mutation analysis for JSWAs [4]. Recently, some researchers have started applying mutation analysis techniques on JSWAs [5]- [7]. ...
Conference Paper
Full-text available
When developers test modern web applications that use JavaScript, challenging issues lie in their event-driven, asynchronous, and dynamic features. Many researchers have assessed the adequacy of test cases with code coverage criteria; however, in this paper, we show that the code coverage-based approach possibly misses some faults in the applications. We propose a mutation analysis approach for estimating the faultfinding capability of test cases. We assume that developers can find overlooked fault instances and improve the test cases with the estimated capability. To create a set of faulty programs, i.e., mutants, we classify the JavaScript features in web applications and then define a comprehensive set of mutation operators. We conducted a case study on a real-world application and found that our approach supported the improvement of test cases to expose hand-seeded faults by an extra ten percent.
Article
BACKGROUND. The term Rich Internet Applications (RIAs) is generally associated with Web applications that provide the features and functionality of traditional desktop applications. Ten years after the introduction of the term, an ample amount of research has been carried out to study various aspects of RIAs. It has thus become essential to summarize this research and provide an adequate overview. OBJECTIVE. The objective of our study is to assemble, classify, and analyze all RIA research performed in the scientific community, thus providing a consolidated overview thereof, and to identify well-established topics, trends, and open research issues. Additionally, we provide a qualitative discussion of the most interesting findings. This work therefore serves as a reference work for beginning and established RIA researchers alike, as well as for industrial actors that need an introduction in the field, or seek pointers to (a specific subset of) the state-of-the-art. METHOD. A systematic mapping study is performed in order to identify all RIA-related publications, define a classification scheme, and categorize, analyze, and discuss the identified research according to it. RESULTS. Our source identification phase resulted in 133 relevant, peer-reviewed publications, published between 2002 and 2011 in a wide variety of venues. They were subsequently classified according to four facets: development activity, research topic, contribution type, and research type. Pie, stacked bar, and bubble charts were used to depict and analyze the results. A deeper analysis is provided for the most interesting and/or remarkable results. CONCLUSION. Analysis of the results shows that, although the RIA term was coined in 2002, the first RIA-related research appeared in 2004. From 2007 there was a significant increase in research activity, peaking in 2009 and decreasing to pre-2009 levels afterwards. All development phases are covered in the identified research, with emphasis on “design” (33&percnt;) and “implementation” (29&percnt;). The majority of research proposes a “method” (44&percnt;), followed by “model” (22&percnt;), “methodology” (18&percnt;), and “tools” (16&percnt;); no publications in the category “metrics” were found. The preponderant research topic is “models, methods and methodologies” (23&percnt;) and, to a lesser extent, “usability and accessibility” and “user interface” (11&percnt; each). On the other hand, the topic “localization, internationalization and multilinguality” received no attention at all, and topics such as “deep Web” (under 1&percnt;), “business processing”, “usage analysis”, “data management”, “quality and metrics” (all under 2&percnt;), “semantics”, and “performance” (slightly above 2&percnt;) received very little attention. Finally, there is a large majority of “solution proposals” (66&percnt;), few “evaluation research” (14&percnt;), and even fewer “validation” (6&percnt;), although the latter have been increasing in recent years.
Article
Asynchronous Javascript And XML (AJAX) is a recent technology used to develop rich and dynamic Web applications. Different from traditional Web applications, AJAX applications consist of a single page whose elements are updated dynamically in response to callbacks activated asynchronously by the user or by a server message. On the one hand, AJAX improves the responsiveness and usability of a Web application, but on the other hand, it makes the testing phase more difficult. In this paper, our state-based testing technique, developed to test AJAX-based applications, is compared to existing Web testing techniques, such as white-box and black-box ones. To this aim, an experiment based on two case studies has been conducted to evaluate effectiveness and test effort involved in the compared Web testing techniques. In particular, the capability of each technique to reveal injected faults of different fault categories is analyzed in detail. The associated effort was also measured. The results show that state-based testing is complementary to the existing Web testing techniques and can reveal faults otherwise unnoticed or hard to reveal with the other techniques.
Article
Full-text available
As web applications evolve and their usage increases, their complexity also increases, thus creating a great demand for techniques and tools to ensure well-tested, reliable applica-tions. While program-based coverage and fault detection capability can be used to measure the quality of test suites, the dynamic characteristics of web applications motivate ad-ditional criteria to complement these traditional test ade-quacy criteria. This paper presents novel dynamic coverage criteria customized for web applications—criteria intended for testing at a page level. Based on a changing universe of test requirements, as indicated by evolving usage of the application, our criteria avoid the difficulties of building an accurate static model of a web application's structure. We define a class of dynamic coverage criteria, present the sub-sumption relation among them, and describe two case stud-ies to demonstrate their usefulness. Among other possible uses, the proposed criteria can be used to compare the qual-ity of test suites, to select test cases, and to examine how usage of a web application changes over time. The proposed criteria complement traditional program coverage and fault detection capability criteria.
Conference Paper
Full-text available
Web applications have become very complex and crucial, especially when combined with areas such as CRM (Customer Relationship Management) and BPR (Business Process Reengineering), the scientific community has focused attention to Web application design, development, analysis, and testing, by studying and proposing methodologies and tools. This paper describes techniques for semi-automatic test case definition and for user1-driven testing (based on statistical testing or coverage analysis) from Web applications reverse engineered UML models. These techniques are implemented as tools in the WAAT project. WebUml is a reverse engineering tool that generates class and state diagrams through static and dynamic Web application analysis. TestUml is a testing suite that uses generated models to define test cases, coverage testing criteria and also reliability analysis.
Conference Paper
Full-text available
User-session-based testing of web applications gathers user sessions to create and continually update test suites based on real user input in the field. To support this approach during maintenance and beta testing phases, we have built an au- tomated framework for testing web-based software that fo- cuses on scalability and evolving the test suite automatically as the application's operational profile changes. This paper reports on the automation of the replay and oracle compo- nents for web applications, which pose issues beyond those in the equivalent testing steps for traditional, stand-alone applications. Concurrency, nondeterminism, dependence on persistent state and previous user sessions, a complex ap- plication infrastructure, and a large number of output for- mats necessitate developing different replay and oracle com- parator operators, which have tradeoffs in fault detection effectiveness, precision of analysis, and efficiency. We have designed, implemented, and evaluated a set of automated replay techniques and oracle comparators for user-session- based testing of web applications. This paper describes the issues, algorithms, heuristics, and an experimental case study with user sessions for two web applications. From our results, we conclude that testers performing user-session- based testing should consider their expectations for program coverage and fault detection when choosing a replay and or- acle technique. Categories and Subject Descriptors: D.2.5: (Software Engineering): Testing and Debugging-testing tools General Terms: Reliability, Experimentation
Conference Paper
Full-text available
Creating effective test cases is a difficult problem, especially for web applications. To comprehensively test a web applica- tion's functionality, test cases must test complex application state dependencies and concurrent user interactions. Rather than creating test cases manually or from a static model, field data provides an inexpensive alternative to creating such sophisticated test cases. An existing approach to using field data in testing web applications is user-session-based testing. Previous user-session-based testing approaches ig- nore state dependences from multi-user interactions. In this paper, we propose strategies for leveraging web application field data to automatically create test cases that test vari- ous levels of multi-user interaction and state dependencies. Results from our preliminary case study of a publicly de- ployed web application show that these test case creation mechanisms are a promising testing strategy for web appli- cations.
Article
Full-text available
Web applications are vital components of the global information infrastructure, and it is important to ensure their dependability. Many techniques and tools for validating Web applications have been created, but few of these have addressed the need to test Web application functionality and none have attempted to leverage data gathered in the operation of Web applications to assist with testing. In this paper, we present several techniques for using user session data gathered as users operate Web applications to help test those applications from a functional standpoint. We report results of an experiment comparing these new techniques to existing white-box techniques for creating test cases for Web applications, assessing both the adequacy of the generated test cases and their ability to detect faults on a point-of-sale Web application. Our results show that user session data can be used to produce test suites more effective overall than those produced by the white-box techniques considered; however, the faults detected by the two classes of techniques differ, suggesting that the techniques are complementary.
Article
AJAX: fast mastery for experienced Web developers!Already an experienced Web developer? Apply your skills in today's fastest-growing area of Web development: AJAX!Building on what you already know, this fast-paced guide will show you exactly how to create rich, usable Internet applications. Joshua Eichorn teaches through sophisticated code examples, including extensive server-side PHP code.You won't just learn how to code AJAX applications: Eichorn covers the entire development lifecycle, from use cases and design through debugging. He also presents detailed application case studies, including a start-to-finish update of a non-AJAX application that addresses everything from feature improvements to changing usage patterns. Coverage includes:ï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ How AJAX changes the conventional Web development cycleï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Problems created by the AJAX paradigm -- and how to avoid themï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Adding AJAX to existing Web applications: key considerationsï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Using core AJAX technologies, including the XMLHttpRequest objectï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Consuming data returned to an AJAX application using both XSLT and JSONï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Building more usable AJAX applications: guidelines and downloadable resourcesï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Use cases: solving real-world problems in the AJAX environmentï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Libraries and toolkits for simplifying AJAX development, including Sarissa, scriptaculous, and HTML_AJAXï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ A complete guide to AJAX debuggingï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ Supporting browsers without XMLHttpRequest by using IFrames or cookiesï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ JSON data encoding debugging guide, which covers tools for Firefox and Internet Explorerï¾·ï¾ ï¾ ï¾ ï¾ ï¾ ï¾ A list of libraries, which includes PHP, .NET, Java, and other libraries that can be used with any server language
Conference Paper
Web applications are becoming increasingly complex and important for companies. Their design, development, analysis and testing need therefore to be approached by means of support tools and methodologies. In this paper we consider the problems related to building tools for the analysis and testing of Web applications and we try to provide some indications on possible solutions, based upon our experience in the development of the tools ReWeb and TestWeb. The definition of a proper reference model will be discussed, as well as the impact of dynamic pages during Web site downloading and subsequent model construction. Visualization techniques addressing the large amount of extracted data will be presented, while infeasibility problems will be considered with reference to the testing phase.
Article
Researchers and practitioners are still trying to nd eective ways to model and test Web applications. This paper proposes a system-level testing technique that combines test generation based on nite state machines with constraints. We use a hierarchical approach to model potentially large Web applications. The approach builds hierarchies of Finite State Machines (FSMs) that model subsystems of the Web applications, and then generates test requirements as subsequences of states in the FSMs. These subsequences are then combined and rened to form complete executable tests. The constraints are used to select a reduced set of inputs with the goal of reducing the state space explosion otherwise inherent in using FSMs. The paper illustrates the technique with a running example of a Web-based course student information system and introduces a prototype implementation to support the technique.
Conference Paper
White-box testing exercises a software system by ensuring that a model of the internal structure is covered by the test cases. Extending this approach to Web applications is far from obvious, because at least two abstraction levels can be considered to represent the internal structure of a Web application: the navigation model and the control flow model. To further complicate the matter, dynamic code generation must be taken into account in both models. In this paper, the two alternative models are presented and white-box testing criteria are defined on them. Their usage for the white-box testing of a real-world Web application is described, highlighting the associated costs and benefits.
AJAX: Creating Web Pages with Asynchronous JavaScript and XML
  • E Woychowsky
E. Woychowsky. AJAX: Creating Web Pages with Asynchronous JavaScript and XML. Bruce Perens’ Open Source Series, 2006.