No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Automatic data-gathering agents for remote navigability testing
Abstract
The Tirsus Project, developed at the Laboratory of Object-Oriented Technologies at the University of Oviedo, Spain, comprises the design of several hypermedia tools for teaching history in the exhibition rooms of the Archaeological Museum of Asturias, Spain. These tools must be designed for a wide variety of users. We use the Object Oriented Hypermedia Development Model to structure the information that historians, archaeologists, and the museum curator provide. This tool lets us create several navigation models to comply with the expectations of different kinds of users, including special versions of the knowledge base in three languages: English for international support, Spanish for national support, and Asturian, the local language, for local support. The research team at Oviedo tested the prototypes' usability and navigability concurrently, employing usability testing techniques. The test participants were history students and volunteers of the Friends of the Archaeological Museum of Asturias Society, two groups who closely matched our intended average users.
... Logs Logs in general Rodriguez, 2002;Rodden, Hutchinson. and Fu, 2010;Andrica & Candea, 2011;Inversini, Cantoni, and Bolchini, 2011;Gordillo et al., 2014;Harrati et al., 2015;Maalej et al., 2015;Garrido et al., 2017;Grigera et Error rate Au et al., 2008;Rodden, Hutchinson, and Fu, 2010 Visitor metrics Organic clicks Rodden, Hutchinson, and Fu, 2010;Bakaev, Bakaev, and Mamysheva, 2016;Lee et al., 2016 Bounce rate Bakaev, Bakaev, and Mamysheva, 2016;Lee et al., 2016 Traffic; the number of unique visits Rodden, Hutchinson, and Fu, 2010;Bakaev, Bakaev, and Mamysheva, 2016;Lee et al., 2016 User activity Rodden, Hutchinson, and Fu, 2010;Bakaev, Bakaev, and Mamysheva, 2016;Lee et al., 2016 Visual metrics Session recording, video recording Oertel & Hein, 2003;Maalej et al., 2015 Eye movement tracking Oertel & Hein, 2003 Heat maps Gordillo et al., 2014;Courtemanche et al., 2017;Grigera et al., 2017 changes are made to a product/service to measure the effect (Rodden, Hutchinson, and Fu, 2010;Lee et al., 2016;King, Churchill, and Tan, 2017). ...
... The examples given here outline a need for a more cost effective and time efficient solution. The usability community has recognized and responded to this need with various remote testing methodologies, many of which are automated and have replaced key processes within the user centered cycle (e.g. Rodríguez & Gutierrez, 2000). But despite the benefits of automation, the need for a human moderator during the evaluation cycle has persisted and forced practitioners to consider a remote extension of local user testing. ...
In the past, remote user testing has offered promising results when compared to its local counterpart. Nonetheless, the comparison studies conducted insofar have relied on small independent samples, measuring key metrics and have not focused on the qualitative objectives of user testing. The current study addresses these issues in order to reinforce the efficiency of the remote method in the key metrics but to also examine its ability to reveal qualitative data. Mainly, our data suggests that remote user testing requires more time to complete, remote users adhere more strictly to the think aloud protocol application and online post-test questionnaires yield higher quantity and quality responses. In addition to attaining the previous two objectives, our analysis also accounts for the effect of website difficulty during user testing. Our results indicate that remote and local user testing yield similar results during the assessment of an easy or difficult website.
... By simply keeping track of traditional measurements like visited pages, time spent on each one and requests made through forms or AJAX interfaces and by relating that to the context data gathered through the devices sensors we can obtain enough information to perform the analysis. Existing systems like [5] or [14] can be used to gather web-interaction data, leaving the task of context data gathering to the context observer module of our system. If a particular case requires ad-hoc event tracking we can, depending on the situation, simply add it to the context tracking system or take the previously described approach to study the internal events of the web browser.Figure 2 : High-level architecture of the proposed system . ...
In this paper we describe a system for silently studying users' behaviour through automated data gath-ering in mobile devices. Today's handheld devices have rich sensing capabilities that can be used to perceive information about users' physical environment. The automated gathering of usage and context data allows not only the study of how users behave in terms of interaction with the device but also the analysis of relations be-tween those interactions and the physical context in which they happen. This might help to improve the quality of the tested applications by offering a deeper understanding of users' perceptions and needs.
... Pavlopoulou and Young's Perpetual Testing extends testing, and in particular the statement coverage adequacy criterion to normal software operation [15]. ANTS' Remote Testing relies on actual usage of selected users to test the navigability of web sites [17]. Orso et al.'s Gamma gathers field-data from deployed software, to support both impact analysis and regression testing [13]. ...
Component-based software systems raise new problems for the testing community: the reuse of components suggests the possibility of reducing testing costs by reusing information about the quality of the software components. This paper addresses the problem of testing evolving software systems, i.e., systems obtained by modifying and/or substituting some of their components. The paper proposes a technique to automatically identify behavioral differences between different versions of the system, to deduce possible problems from inconsistent behaviors. The approach is based on the automatic distilling of invariants from in-field executions. The computed invariants are used to monitor the behavior of new components, and to reveal unexpected interactions. The event generated while monitoring system executions are presented to software engineers who can infer possible problems of the new versions.
... ANTS was able to record the exact position of the user on a navigational map as well as record the duration of how long a user spends on each page. Although the tool could be useful for future information retrieval concerning navigability, it is still at a starting point, as the results showed little distinctive relationship between user behaviour and navigability [19]. ...
In recent years, navigability has become the pivot of website designs. Existing works fall into two categories. The first is to evaluate and assess a website's navigability against a set of criteria or check list. The second is to analyse usage data of the website, such as the server log files. This paper investigates a metric approach to website navigability measurement. In comparison with existing assessment and analysis methods, navigability metrics have the advantages of objectiveness and the possibility of using automated tools to evaluate large-scale websites. This paper proposes a number of metrics for website navigability measurement based on measuring website structural complexity. We will validate these metrics against Weyuker's software complexity axioms, and report the results of empirical studies of the metrics.
... In many cases an automated assessment is the best way to discover incorrect site structure. Based on the archive of navigational patterns some automatic simplifications of path existing in the system can be recommended [8]. Typical association rules and their indirect version were used for creation of recommendation ranking list and as the minor importance research for the assessment of hyperlinks. ...
One of the basic methods of web usage mining are association rules that indicate relationships among common use of web pages. Positive and confined negative association rules are the components of the new quality measures: Positive and Negative Quality function, respectively. These functions are used to evaluate the quality of hyperlinks existing on web pages. A number of statistics and the expert validation revealed the usefulness of association rules for the assessment of hyperlink usability.
... In many cases an automated assessment is the best way to discover incorrect site structure. Based on the archive of navigational patterns some automatic simplifications of path existing in the system can be recommended [20, 23]. Typical association rules and their indirect version were used for creation of recommendation ranking list and as the minor importance research for the assessment of hyperlinks . ...
The new approach to structure verification of the Web hyperlinks is presented in the paper. It utilizes several types of patterns extracted from the Web usage data: positive and negative association rules, positive sequential patterns as well as the new kind of patterns - sequential patterns with negative conclusions. All they enable to appraise the usefulness of hyperlinks in both the positive and negative way. Based on this knowledge, the content managers can adequately promote the trustworthy hyperlinks and remove the negatively verified ones.
... Orso et al. gather field data both to predict impact of updates and to drive the regression testing [8] Rodriguez monitors user activities to infer the way the system is used [10]. Rodriguez's approach has been used for navigability testing of web sites, but the idea can be extended to CBSs too. ...
Component-based system (CBS) technology supports rapid development of complex heterogeneous evolving systems by enhancing reuse and adaptability. CBSs can be extended and adapted by modifying one or more components. The same component can be used in several systems, and the same system can be deployed in many configurations that differ for some components. Traditional test and analysis techniques make little use of quality information about components and subsystems when testing the whole system. Thus, reusability for quality assessment and reduction of quality related costs are not fully exploited. Moreover, verification of CBSs is hardened by the frequent lack of information about components that are provided by third parties without source code and with incomplete documentation. This framework reduces the applicability of many traditional testing and analysis techniques for CBSs. Main goal of my PhD research is the definition and experimentation of testing and analysis techniques that allow to efficiently test CBSs in presence of limited information about design and code by reusing behavioral information that can be gathered from previous usage of the components.
Local user testing accrues high costs when solely relied on during the iterative cycle of user- centered design. A number of automatic evaluation methods have emerged with the aim of reducing costs incurred by repeated local tests, international user testing and post-deployment assessments. When considering costs, it is important not to overlook the benefits provided in human-moderated user testing. This paper focuses on the benefits of human moderation, provides test administration protocols for human-moderated remote testing and examines those protocols in real life settings. Local user testing, at a portable or established laboratory, has been the customary method used in determining the effectiveness of a product. Local user tests are conducted throughout the product development lifecycle, each time with different objectives. Depending on those objectives, several types of tests are implemented. Exploratory tests validate the product, determine users' mental models and suggest an appropriate design strategy. Assessment tests verify the soundness of the exploratory test. Task completion and ease of use are examined. Finally, validation tests serve as verification of a product. Performance measures are evaluated to ensure that the product objectives have been met (Rubin, 1994).
Browsing the web is inecient for blind web users because of persistent accessibility prob- lems. While previous eorts have tried to improve web accessibility through developer awareness, technological improvements, and legislation, these attempts have failed to adequately address accessibility concerns. Blind web users have proven quite adept at overcoming the shortcom- ings of the web and existing tools, but have been only marginally involved in improving the accessibility of their own web experience. In this paper, we first present a survey of existing accessibility issues and then discuss technology that has been developed to overcome them. We next discuss user studies that have been performed in order to understand the user experience of blind web users. These studies along with related work will inform new remote studies that will observe blind web users as they browse the web using their own equipment. Finally, we propose a new approach to improving web accessibility that leverages the skill and intelligence of blind web users. This approach seeks to enable blind web users to independently improve the accessibility of their own environment in a way that can be shared with both web users and developers.
Laboratory navigability testing is a powerful technique to obtain a picture of the user’s mental representation of the navigation
model of a web site. However, bringing volunteers to the laboratory to test our prototypes is quite expensive and even impossible
for certain users with specific interaction needs. Since the tests are performed using equipments different from those employed
by the real user, the impact that the computer performance has on navigation is missed. In this paper we propose a remote
testing approach, performing navigability testing in the user’s home, employing special silent data-gathering software agents,
which are able to measure the user accuracy when performing navigation tasks.
The first stage in the design of a user interface is the quest for its ‘typical user’, an abstract generalization of each
user of the application. However, in web systems and other scenarios where the application can be used by dozens of different
kinds of users, the identification of this ‘typical user’ is quite difficult, if not impossible. Our proposal is to avoid
the construction of interactive dialogs during the design stage, building them dynamically once the specific cognitive, perceptual
and motor requirements of the current user are known: that is, during the execution stage. This is the approach used by GADEA,
an intelligent user interface management system (UIMS) able to separate the functionality of an application from its interface
in real time. The system adapts the components of the interface depending on the information stored in a user model which
is continuously updated by a small army of data-gathering agents.
In years, navigability has become the pivot of Web site designs. Existing works fall into two categories. The first is to evaluate and assess a Web site's navigability against a set of criteria or check list. The second is to analyse usage data of the Web site, such as the server log files. This work investigates a metric approach to Web site navigability measurement. In comparison with existing assessment and analysis methods, navigability metrics have the advantages of objectiveness and the possibility of using automated tools to evaluate large-scale Web sites. This work proposes a number of metrics for Web site navigability measurement based on measuring Web site structural complexity. We validate these metrics against Weyuker's software complexity axioms, and report the results of empirical studies of the metrics.
In this paper we analyze the process of hypermedia applications design and implementation, focusing in particular on two critical aspects of these applications: the navigational and interface structure. We discuss the way in which we build the navigation and abstract interface models using the Object-Oriented Hypermedia Design Method (OOHDM); we show which concerns must be taken into account for each task by giving examples from a real project we are developing, the Portinari Project. We show which implementation concerns must be considered when defining interface behavior, discussing both a Toolbook and a HTML implementation of the example application. Keywords: Hypermedia Design, Methodology, Modeling, Object Orientation, Navigation, Interfaces 1. Introduction In the past three years there has been growing interest in hypermedia design approaches [Izakowitz 95, Garzotto 93, Lange 94]. There are many different problems the hypermedia designer has to deal with, since the combinatio...
Multiple navigational graphs can be obtained as the result of the design stage of a hypermedia-based artifact. The only way to know which one adapts better to the user navigational metaphor is by mean of usability testing. This technique is expensive in terms of the number of human resources needed to perform it, and it isn’t able to record spontaneous user behavior. The use of automatic testing tools is an interesting and cheap alternative that avoids the problems commented. We tested our own automatic navigability testing system (ANTS) conducting an experiment to determine where users expect to find the navigational bar of a web site.
Temporal properties of interaction have a profound impact upon the usability of human computer interfaces. Delays in response time can lead to frustration and error. The simultaneous presentation of many different pieces of information imposes heavy demands upon the cognitive and perceptual resources of system operators. These problems have been investigated by a number of recent research initiatives. Unfortunately, it has been difficult to replicate the results that have been obtained from experimental investigations. This creates significant problems for designers if these results are to guide the future development of interactive systems. This paper briefly describes how an multi-disciplinary team has addressed this problem during the Temporal Aspects of Usability (TAU) project.
There has been little research into the study of long term adaptation by users or the extent to which usage is enhanced over a substantial period [1]. However, there is general agreement that some interfaces, such as Unix shells and certain editors, take years to master. In this paper we present evidence that users do indeed change their actions in the long term. Some implications of our findings are discussed.
As machine response delays vary, the most important effect on users may be not their annoyance but that they change the way they use an interface. Even the very simple task of copytyping three digit numbers gives rise to at least three different user strategies (i.e. procedures). However the effect seems not be a simple function of delay length, contrary to earlier reported work. Instead users are probably shifting between strategies more fluidly.
Usability testing is normally used to determine the navigation map that better adapts to the average-user mental navigation
model. However, usability testing is an expensive process, misses the influence that the computer environment has on navigation,
and it is not able to record spontaneous user behaviour. The remote testing technique is an interesting and cheaper alternative
that avoids the problems commented. We have developed our own set of tools based on data gathering agents for supporting remote
testing. In this paper, we comment our experiences designing and developing such kind of systems and the results obtained
when conducting human-computer interaction experiments based on access to web sites.
Temporal Aspects of Usability: How Machine Delays Change User Strate-gies
- P O Donnell
- S W Draper
P. O'Donnell and S.W. Draper, " Temporal Aspects of Usability: How Machine Delays Change User Strate-gies, " SIGCHI Bull., vol. 28, no. 2, Apr. 1996, www.acm.org/sigchi/bulletin/1996.2/Steve-Draper.html.
Microsoft Secrets: How the World's Most Powerful Software Company Creates Technology, Shapes Markets, and Manages People,
- M A Cusumano
- R W Selby
M.A. Cusumano and R.W. Selby, Microsoft Secrets: How the World's Most Powerful Software Company Creates Technology, Shapes Markets, and Manages Peo-ple, Free Press, New York, 1995.
"Temporal Aspects of Usability: Long-term Variation in User Actions,"
- R C Thomas
R.C. Thomas, " Temporal Aspects of Usability: Long-term Variation in User Actions, " SIGCHI Bull., vol. 28, no. 2, Apr. 1996, www.acm.org/sigchi/bulletin/1996.2/ Rich-Thomas.html.
WebCAT: Web-CATegory Analysis Tool
- Usability Visualization
- Group
Visualization and Usability Group, " WebCAT: Web-CATegory Analysis Tool, " http://zing.ncsl.nist.gov/ WebTools/WebCAT/overview.html.
Is Timeout to Be the Big Is-sue? " Proc. Time and the Web Workshop
- A Byrne
- R Picking
A. Byrne and R. Picking, " Is Timeout to Be the Big Is-sue? " Proc. Time and the Web Workshop, British Com-puter Soc., Wiltshire, UK, 1997, www.soc.staffs.ac.uk/ seminars/web97/papers/picking.html.
"What's the Web Worth? The Impact of Retrieval Delays on the Value of Distributed Information,"
- C Johnson
C. Johnson, " What's the Web Worth? The Impact of Retrieval Delays on the Value of Distributed Informa-tion, " Proc. Time and the Web Workshop, British Com-puter Soc., Wiltshire, UK, 1997, www.soc.staffs.ac.uk/ seminars/web97/papers/johnson.html.
Abstraction, Composition and Lay-Out Definition Mechanism in OOHDM
- D Schwabe
- G Rossi
D. Schwabe and G. Rossi, " Abstraction, Composition and Lay-Out Definition Mechanism in OOHDM, " Proc. ACM Workshop Effective Abstractions in Multi-media, ACM Press, New York, 1997, www.cs.tufts. edu/~isabel/schwabe/MainPage.html
The JavaBeans 1.0 API specification
- Sun Microsystems
Automatic Low-Level Interactive Discourses for Multimodal User Interfaces," <i>
- M G Rodríguez
Is Timeout to Be the Big Issue?" <i>Proc. Time and the Web Workshop,</i>
- A Byrne
- R Picking
Ergolight Operation Recording Suite
- Ergolight
WebVIP: Web Variable Instrumented Program
- Usability Visualization
- Group
Ergolight Interactive Site Usability Re-ports, " www.ergolight-sw
- Ergolight
Ergolight, " Ergolight Interactive Site Usability Re-ports, " www.ergolight-sw.com/pub/Sites/ ErgoLight/First/Q1CY2001/Reports/Home.html.
Low-Level Interactive Discourse Adaptation Based on Cognitive Models," <i>
- M G Rodríguez
"Is Timeout to Be the Big Issue?"
- A Byrne
- R Picking
"Low-Level Interactive Discourse Adaptation Based on Cognitive Models,"
- M G Rodrguez
"Automatic Low-Level Interactive Discourses for Multimodal User Interfaces,"
- M G Rodrguez
"what's the web worth? the impact of retrieval delays on the value of distributed information,"
- johnson
"abstraction, composition and lay-out definition mechanism in oohdm,"
- schwabe
"is timeout to be the big issue?"
- byrne