ArticlePDF Available


User engagement is a key concept in designing user-centred web applications. It refers to the quality of the user experi-ence that emphasises the positive aspects of the interaction, and in particular the phenomena associated with being cap-tivated by technology. This definition is motivated by the observation that successful technologies are not just used, but they are engaged with. Numerous methods have been proposed in the literature to measure engagement, however, little has been done to validate and relate these measures and so provide a firm basis for assessing the quality of the user experience. Engagement is heavily influenced, for ex-ample, by the user interface and its associated process flow, the user's context, value system and incentives. In this paper we propose an approach to relating and de-veloping unified measures of user engagement. Our ulti-mate aim is to define a framework in which user engagement can be studied, measured, and explained, leading to recom-mendations and guidelines for user interface and interaction design for front-end web technology. Towards this aim, in this paper, we consider how existing user engagement met-rics, web analytics, information retrieval metrics, and mea-sures from immersion in gaming can bring new perspective to defining, measuring and explaining user engagement.
Towards a science of user engagement (Position Paper)
Simon Attfield
University of Middlesex
Gabriella Kazai
Microsoft Research
Mounia Lalmas
Yahoo! Research
Benjamin Piwowarski
University of Glasgow
User engagement is a key concept in designing user-centred
web applications. It refers to the quality of the user experi-
ence that emphasises the positive aspects of the interaction,
and in particular the phenomena associated with being cap-
tivated by technology. This definition is motivated by the
observation that successful technologies are not just used,
but they are engaged with. Numerous methods have been
proposed in the literature to measure engagement, however,
little has been done to validate and relate these measures
and so provide a firm basis for assessing the quality of the
user experience. Engagement is heavily influenced, for ex-
ample, by the user interface and its associated process flow,
the user’s context, value system and incentives.
In this paper we propose an approach to relating and de-
veloping unified measures of user engagement. Our ulti-
mate aim is to define a framework in which user engagement
can be studied, measured, and explained, leading to recom-
mendations and guidelines for user interface and interaction
design for front-end web technology. Towards this aim, in
this paper, we consider how existing user engagement met-
rics, web analytics, information retrieval metrics, and mea-
sures from immersion in gaming can bring new perspective
to defining, measuring and explaining user engagement.
Categories and Subject Descriptors
Systems—Human factors
General Terms
User experience, User engagement, Evaluation
Over the last two decades the nature of human computer
interaction has been transformed. In the 1980s the dominant
model was of a single person sat in front of a workstation,
probably in an oce. At the end of the day computing was
left in the oce and interaction came to an end. Since then
Copyright is held by the author/owner(s).
WSDM’11, February 9–12, 2011, Hong Kong, China.
ACM 978-1-4503-0493-1/11/02.
computers and connectivity have come to pervade all as-
pects of our lives. We have computers in our homes and in
our pockets. The continually evolving web provides unprece-
dented choice of things we can do; we can educate ourselves
or others, buy and sell, watch movies, keep up with friends,
write blogs, share content, play games, or simply indulge
apassingcuriosity. Withthistheconceptualisationofthe
user has also changed. Kuutti [23] describes this evolution as
being from a cog in an organisational machine and a “source
of error”, to a partner in social interaction and ultimately
consumer, and more recently, with Web 2.0, to content cre-
In line with this transformation, researchers and prac-
titioners in user-centred design have increasingly seen the
need to extend traditional performance-based notions of us-
ability, such as eectiveness, eciency and satisfaction (ISO
9241-11), to encompass non-utilitarian aspects of interaction
(e.g., [12, 24]). Under the broad rubric of user experience,
there is emphasis on understanding and designing for the
subjective aspects of technology encounters (e.g., [24, 47]).
In a world full of choices where the fleeting attention of
the user becomes a prime resource, it is essential that tech-
nology providers do not just design systems but that they
design engaging experiences [31]. Indeed, in an environment
of choice, failing to engage the user “equates with no sale on
an electronic commerce site and no transmission of informa-
tion from a website; people go elsewhere” [28]. The question
then becomes: how do we do this, and closely related to
that, how do we assess the experience as being the kind we
would like to design?
The answer lies in the study of user engagement, a quality
of user experience that emphasises the positive aspects of in-
teraction, and in particular the phenomena associated with
being captivated by technology (and so being motivated to
use it) [28]. Successful technologies are not just used, they
are engaged with; users invest time, attention, and emotion
into the equation.
To promote user engagement we need to be able to design
for it. To know when we have done this we need some way
of assessing and measuring it. Only by knowing when and
why engagement occurs can we understand what is eective.
As discussed in this paper, many works on user engagement
have been written, but they are spread over a range of dif-
ferent areas of research, thus hindering the possibility of
building upon previous research. We thus need a framework
in which user engagement can be studied, measured and
explained, leading to, for example, recommendations and
guidelines for user interface and interaction design, that is,
we need a science of user engagement.
Towards this goal, this paper contains our proposal to es-
tablish the “science of user engagement”. It has two main
contributions. First, we provide a review and classification
of prior research on user engagement and associated mea-
sures (Section 2), with a special focus on the work carried
out outside large-scale web log analysis. Based on this re-
view, and our insights, we then present our proposed re-
search agenda providing a detailed roadmap on how to de-
fine and set up a framework in which user engagement can
be studied, measured, and explained (Section 3).
In this section, we provide a review of prior research on
user engagement. We start with our own definition based
on our review of that literature.
User engagement is the emotional, cognitive and
behavioural connection that exists, at any point
in time and possibly over time, between a user
and a resource.
This definition is intentionally broad. By identifying emo-
tional, cognitive, and behavioural factors, it emphasises the
holistic character of user engagement and is also sugges-
tive of aspects that are open to measurement. It also refers
equally to user engagement in terms of a single session or
a more long-term relationship across multiple session. User
engagement with a technological resource is not just about
how a single interaction unfolds, but about how and why
people develop a relationship with technology and integrate
it into their lives.
2.1 Characteristics
In the following we discuss some characteristics associ-
ated with user engagement, either as presented by previous
studies or as suggested by us. These elaborate the notion
of engagement over the three broad dimensions: emotional,
cognitive and behavioural. While some of the characteris-
tics have stronger ties with one of the dimensions, most are
a combination of the three. They also provide some orienta-
tion for thinking about the designable causes and observable
consequences of user engagement.
Fo c us e d a t t e nt i on .
Being engaged in an experience involves focusing attention
to the exclusion of other things, including other people (un-
less social interaction enhances the engagement) [28]. This
phenomenon relates to distortions in the subjective percep-
tion of time during interaction [29], which has been shown to
be an eective indicator of cognitive involvement [2] and has
been used as a component in measures of games immersion
[18]. The more engaged someone is, the more likely they are
to underestimate the passage of time. Together with concen-
tration, absorption and loss of self-consciousness, distortions
in the subjective perception of time have led to parallels be-
ing drawn between engagement and the idea of flow as an
optimal experience [9], where flow refers to a mental state
in which a person is fully immersed in what they are doing.
Positive Aect.
“Engaged users are aectively involved” [28]. Aect relates
to the emotions experienced during interaction. For exam-
ple, O’Brian & Toms [28] found that a lack of fun can act
as a barrier to shopping online and that fun during a web-
cast can draw the user in, although a lack of awareness of
other viewers can mitigate against the quality of the fun ex-
perience. Jennings [19] argues that aective experiences are
intrinsically motivating and that, in relation to engagement
on the web, an initial aective hook can induce a desire for
exploration or active discovery. This, they argue then en-
courages greater emotional involvement and contributes to
customer loyalty (see endurability below). They regard af-
fect as “an emotional investment that helps create a personal
link to an experience or activity” [19].
Aesthetics concerns the sensory, visual appeal of an interface
and is seen as an important factor for engagement [29]. In
the context of online shopping, web searching, educational
webcasting and video games, O’Brian & Toms [28] relate
aesthetics to factors such as screen layout, graphics and the
use of design principles such as symmetry and balance. In
the context of multimedia design, Jennings [19] relates aes-
thetics (in this case media quality) to positive aect and
suggests that aesthetic experiences promote focused atten-
tion and stimulate curiosity. The significance of aesthetics
was well demonstrated by Tractinsky & Ikar [45] who, in an
experiment in which users interacted with a computerised,
surrogate Automated Teller Machine, found a positive cor-
relation between perceived usability and aesthetic appeal,
even though no such correlation existed between perceived
usability and actual usability.
People remember enjoyable, useful, engaging experiences and
want to repeat them. This aspect of engagement refers to
the likelihood of remembering an experience and the willing-
ness to repeat it [35]. It also relates to a users’ willingness
to recommend an experience to others, and to their per-
ceptions of whether an experience met their expectations of
being “successful”, “rewarding”, or “worthwhile” [29]. In the
context of the web, endurability can be related to the notion
of ‘sticky’ content and the goal of holding the user’s atten-
tion and encouraging them to return. O’Brian & Toms [28]
found that having fun, being rewarded with convenience and
incentives, and discovering something new promoted later
Interactive experiences can be engaging because they present
users with novel, surprising, unfamiliar or unexpected ex-
periences. Novelty appeals to our sense of curiosity, en-
courages inquisitive behaviour and promotes repeated en-
gagement [29]. It can arise through freshness of content or
innovation in information technology [13]. In e-commerce
applications, shoppers can enjoy the experience of becoming
sidetracked, browsing just to see what is there [28]. It has
been shown that learners experience higher levels of engage-
ment during multimedia presentations that exhibit higher
levels of variety [48]. However, there is also suggestion of
the importance of a subtle balance between novelty on the
one hand and familiarity on the other. For example, in gam-
Characteristic Definition Measures Ref.
Focused At t e n -
Focusing att e n t ion to the exclusi o n o f o t h e r
Distorted perception of time, follow-
on task performance, eye tracking
[28, 29, 2, 18, 9, 15]
Positive Aect Emotions experienced during interaction Physiological sensors (e.g. face detec-
[28, 19, 15]
Aesthetics Sensory and visual appeal of an interface Online activity (curiosity-driven be-
haviour), Physiological sensors (e.g.,
eye tracking), perceived utility
[29, 28, 19, 45, 15]
Endurability Likelihood of remembering an experience
and the willingness to repeat or recommend
Online activity (e.g. bookmarking,
sending emails)
[35, 29, 28, 32, 34,
Novelty Novel, surprising, unfamiliar or unexpected
Physiological sensors (e.g., blood
[29, 13, 28, 48, 39,
Richness and
Levels of richness and control Online activity (e.g., interaction with
the site, time spent), Physiological
sensors (e.g. mouse pressure)
[37, 32, 46]
Reputation, trust
and expectation
Global trust users have on a given entity Online activity (returning user, rec-
[22, 27, 44, 10, 20,
User Context User’s motivation, incentives, and benefits Online activity (location, time, past
[26, 25, 30, 11]
Table 1: In this ta b l e , w e s u m m a r ise the i d e nt i e d ch a r a c t e risti c s o f u s e r e n gage m e nt di s c u s s e d in Sect i o n 2 ,
give the possible ways to objectively measure them (beyond questionnaires, if any), and give the references
where such characteristics are discussed
ing the level of novelty can determine whether engagement
is sustained [28], although some familiarity with a game en-
vironment can lead to faster engagement [39] and reduced
disorientation [48].
Richness and control.
The “Richness, Control and Engagement” (RC & E) frame-
work [37] explains levels of engagement in terms of the levels
of richness and control that are shaped by the features of
a product and the user’s expertise. Richness captures the
growth potential of an activity by assessing the variety and
complexity of thoughts, actions and perceptions as evoked
during the activity (e.g., variety, possibilities, enjoyment, ex-
citement, challenge). Control captures the extent to which
a person is able to achieve this growth potential by assessing
the eort in the selection and attainment of goals (clarity,
ease, self confidence, freedom). Through experiments on a
digital voicemail system, where they varied the number of
features of the user interface, the amount of voicemail con-
tent and the type of task, Rozendaal et al. [37] showed that
level of engagement could be predicted according to the level
of richness and control experienced.
Reputation, trust and expectation.
Trust is a necessary condition of user engagement. Repu-
tation can be seen as the trust users invest globally in a
given resource or provider. Trust is not only a matter of the
insurances oered by technology (e.g. encrypted commu-
nications), but also depends on implicit contracts between
people, computers, and organisations [22]. Organisations in
“real life” do rely on consumers’ faith in the information or
service provided. For example, telephone companies are re-
quired to correctly connect calls and protect the integrity
of phone numbers. Similar requirements can be issued when
transposing trust to the web. Where users interact with each
other, the reputation of the web site has a strong influence
on the extent to which users trust each other in their trans-
actions, e.g., on eBay [27]. Similarly, the perceived fairness
of the scoring system in Q&A sites influences users’ trust
in the service and their willingness to engage [44]. In social
networks, trust naturally has an additional dimension re-
lated to the perception of trustworthiness of user-generated
content [10]. Trust can also be related to engagement with
web search services, where authoritativeness and popular-
ity features are used to rank search results [20]. Finally,
a consequence of trust or reputation is expectation, which
harnesses engagement before a user has even reached a web
site [41].
User context.
User’s motivation, incentives, and benefits aect the experi-
ence more than in traditional usability [26]. User experience
is also very context dependent [25], so the experience with
the same design in dierent circumstances is often very dif-
ferent. This means that user engagement evaluation cannot
be conducted just by observing user’s task completion in a
laboratory test [30]. Similarly, the range of available choices,
as well as the accepted social norms, values and trends im-
pact on how users engage [11]. The user’s personal prefer-
ences and priorities over aspects that influence engagement,
such as trendiness, coolness, or fun are likely to change in
dierent usage scenarios and domains. In addition, dier-
ent forms of engagement are likely to suit dierent types of
personality (e.g., couch potatoes, critics, or creators).
These characteristics go some way to elaborate the emo-
tional, cognitive and behavioural components of user en-
gagement. They also say something about how engagement
might be recognised, and even promoted. However, research
is still very patchy. We know little about how dierent char-
acteristics might be more or less significant for dierent sce-
narios of web interaction and dierent user demographics.
There may be more significant characteristics yet to be iden-
tified. Researchers are only beginning to explore this space.
We return to this in Section 3. For now these characteristics
provide some initial leverage on approaches to measuring
user engagement. In the next section we explore some of
these opportunities.
2.2 Measurement
Having defined user engagement and elaborated some of
its key characteristics we now look into potential approaches
to its assessment. Since user engagement is multi-faceted (as
witnessed by the list above) there are (and should be) many
approaches to its measurement.
User experience evaluation metrics can be divided into two
broad types: subjective and ob jective. Subjective measures
record a user’s perception, generally self reported, of the
media at hand. User’s subjective experiences are central
to user engagement and we consider methods for assessing
these. Subjective experiences, however, can have objectively
observable consequences, and so we also consider objective
measurements that may be indicative of user engagement.
These include independent measures such as the passage of
time or number of mouse clicks to complete a task.
2.2.1 Subjective measures
Where the subjective characteristics of an interactive ex-
perience have been defined, it is possible to construct a post-
experience questionnaire to measure them in relation to a
given interactive experience. Such an instrument was de-
veloped by O’Brian, Toms and colleagues. They first [28]
report a literature review and exploratory interview study
with users in four domains (online shopping, web searching,
educational webcasting, and video games). Through their
analysis they identify a set of attributes of user engagement
(some of which are listed above). They then used this initial
conceptualisation to develop a questionnaire to assess user-
engagement among online shoppers [29]. They administered
their survey to 440 online shoppers and used factor analysis
to reduce the characteristics to a set of key constructs. Their
survey provides a standardised instrument for eliciting users
engagement assessments.
O’Brian et al’s aim was to produce a general purpose user
engagement questionnaire. However, user engagement al-
most certainly has dierent characteristics in dierent appli-
cation domains and for dierent demographic groups. The
web oers a diversity of experiences relevant to a diversity
of users. The properties of engagement during instant mes-
saging, for example, may dier from the properties of en-
gagement with a news portal. Aesthetic appeal may be an
important factor for engagement with a films website whilst
trust has been found to be a key factor in engagement with
health websites [50]. Related to this, dierent user popula-
tions may have dierent priorities. Whereas fun may be an
important characteristic for engaging children [35], ease of
navigation may be a higher priority for adults. Some char-
acteristics of engagement may generalise well, others may
This raises the question of generating new user engage-
ment instruments relevant to dierent application domains
and user groups. As O’Brian and Toms demonstrate, an
approach is to begin with exploratory, qualitative studies
designed to uncover engagement characteristics for specific
kinds of interaction and user. Analysis can then draw out
key variables indicative of engagement in this context, fol-
lowed by the creation and validation of questionnaires that
probe on these variables.
One issue that arises with post-experience questionnaires
and tests, however, is that they are not sensitive to ways
in which an interaction changes over time. Interactions are
dynamic and engagement fluctuates [28]. In the interests
of dierentiating aspects of an interactive experience it may
be valuable to measure such temporal changes. In this case
post-experience questionnaires may not be the best tool, and
objective measures seem more well suited.
2.2.2 Objective measures
There are a number of drawbacks to questionnaires and
other subjective measures. These include their reliance on
the user’s subjectivity [42], post-hoc interpretation and their
susceptibility to the halo eect1[38]. A common strategy
to overcome these is to develop ob jective measures that can
reliably indicate subjective states [16]. This is done, for
example, in studies of presence [40] and games immersion
This raises the question of what objective phenomena are
indicative of engagement. To date little has been reported
on this, but since engagement is multifaceted and complex,
a number of dierent variables may be useful to measure.
In the following we discuss some measures used in previous
research. What becomes evident is that each objective mea-
sure tend to target a very specific aspect of engagement, un-
like questionnaires, which can address a range of variables.
In light of this and our proposal that dierent engagement
characteristics are relevant for dierent kinds of situation,
we anticipate that it will be both useful and necessary to
establish a range of objective measures and use this range
to predict engagement.
The measures we consider here are: the subjective per-
ception of time, follow-on task performance, physiological
sensors, online behaviour, and information retrieval metrics.
In Table 1, we suggest how these measures could be related
to dierent aspect of user engagement.
The subjective perception of time (SPT).
Assessing the subjective perception of time involves asking a
user to make some estimation of the passage of time during
an activity. This can involve retrospectively probing for how
long an activity lasted or asking a user to indicate the dura-
tion of fixed time-intervals during the activity [2]. Despite
its name, we regard SPT as an objective measure given that
it is assessed against actual time. To ask somebody to re-
port how long an activity has lasted or estimate an interval
is to ask them a question they can get wrong (by dierent
degrees). In a sense it is a measure of performance. As dis-
cussed above, SPT has been used in attention research and
is indicative of cognitive aspects of engagement.
Follow-on t a s k p erf o r m a n ce.
Another potential measure of cognitive engagement is how
well somebody performs on a dierent task immediately fol-
lowing a period of engaged interaction. Games researchers
have found that the more immersed a person is when playing
1This a cognitive bias whereby the perception of one trait
(i.e. a characteristic of a person or object) is influenced by
the positive evaluation of another trait (or several traits) of
that person or object.
a game, the longer it takes them to complete an unrelated
puzzle task immediately afterwards [18]. A related measure
is to assess how well somebody performs on a secondary task
that periodically interrupts an engaged interaction.
Physiological measures.
Physiological data can be captured by a broad range of sen-
sors [15] related to dierent cognitive states. Examples of
sensors are eye trackers (diculty, attention, fatigue, mental
activity, strong emotion), mouse pressure (stress, certainty
of response), biosensors (e.g. temperature for negative af-
fect and relaxation, electrodermal for arousal, blood flow for
stress and emotion intensity), oximiters (e.g., pulse), cam-
era (e.g., face tracking for general emotion detection). Such
sensors have several advantages over questionnaires or on-
line behaviour, since they are more directly connected to the
emotional state of the user, are more objective (involuntary
body responses) and they are continuously measured. They
are however more invasive and cannot be used in large scale
Ikehara and Crosby [15] have used such sensors to assess
the cognitive load of a ”game” (tracking fractions). Jennett
et al. [18] also report the use eye-tracking data to assess
immersion. They found that eye-movement increases over
time during a non-immersed experience and reduces over
time during an immersed experience. Users seem to focus
on fewer targets during engaged attention.
In general, such measures could be highly indicative of im-
mersive states through their links with attention, aect, the
perception of aesthetics and novelty. Research is required
to refine our understanding of precisely what physiological
states are indicative of engagement and how to dierentiate
against negative states of high arousal such as stress.
Online behaviour.
The subjective perception of time, follow-on task perfor-
mance and physiological indicators are objective measures
that are potentially suitable for measuring a small number
of interaction episodes at close quarters. In contrast, the
web-analytics community has shown some interest in mea-
suring user engagement through various approaches that as-
sess users’ depth of engagement with a site. In this context
Peterson [32] defines engagement as,“An estimate of the de-
gree and depth of visitor interaction against a clearly defined
set of goals.” For example, Peterson [32] considers measures
that indicate visitors who consume content slowly and me-
thodically, return directly to a site, and whether they sub-
scribe to feeds, and have defined a user-
engagement metric calculated on the basis of the number of
comments per posting on their site [46].
So-called interaction patterns, like for example web search
logs, can be instrumental in studying user engagement. A
step in this direction can be found in [34] who proposed
to summarise a series of actions performed by the user by
a single value. A related approach can be found in the
patent Methods and systems for detecting user satisfaction
(US Patent 7587324) where the inventors propose to mea-
sure user satisfaction by assigning a utility value to each se-
quence of actions (between a user and a computer interface)
and comparing this value to a baseline. User interaction pat-
terns have also be used to predict search engine switching
behaviour [49], which is a facet of user engagement.
From a commercial perspective, assessments of online be-
haviour fall within the scope of customer engagement (CE).
CE is an instrument of marketing referring to the engage-
ment of customers with one another, with a company or a
brand. Its more recent form considers online CE. Online CE
enables organisations to respond to the changes in customer
behaviour that the internet has brought about, as well as the
increasing ineectiveness of the traditional“interrupt and re-
peat” broadcast model of advertising. Supporting “deep” en-
gagement is seen as an important source of competitive ad-
vantage, whether through advertising, user generated prod-
uct reviews, FAQs, forums where consumers can socialise or
contribute to product development. In this space, a number
of initiatives have emerged to address the question of CE
metrics from organisations such as the World Federation of
Advertisers (WFA), the Association of National Advertisers
(ANA) and Nielsen Media Research. As in the academic
literature, engagement is seen as multifaceted and open to
a range of potential metrics [43].
Information retrieval (IR) metrics.
Among the dierent lines of research in IR metrics, three
are directly related to measuring user engagement. The first
is to develop metrics for interactive IR, e.g., [5, 17] where
users and their contexts are taken into account. This line
has brought up the idea of simulated search scenarios, where
a subject is asked to follow a search scenario that specifies
what, why, and in which context the user is searching.
The second line of research is to develop metrics that in-
corporate enriched user interaction models, e.g., [33, 21, 51].
This is needed when old assumptions typical of traditional
IR (linear browsing, binary relevance) do not hold. The work
of e.g. Piwowarski & Dupret [33] and Kazai et al. [21] pro-
vides formal user models that could be adapted to defining
the behaviour of a user when interacting with a web service,
and hence a way to provide measures correlated with user
Finally, the third line of research relates standard IR ef-
fectiveness metrics to user satisfaction, e.g., [1, 14]. The
research of Al-Maskari et al. [1] shows that we can expect
some correlation between carefully chosen metrics and user-
oriented measures, even with metrics that were not specifi-
cally designed to capture such aspects of the retrieval pro-
cess. However, more research is needed in order to validate
and extend these results.
2.3 Building on Foundations
Interest is growing in user engagement and its measure-
ment, as well as that of related concepts. However work
is somewhat fragmented, even if more holistic frameworks
begin to appear [36]. This framework has been developed
for defining large-scale user-centred metrics, both attitudinal
and behavioural, coined HEART (Happiness, Engagement,
Adoption, Retention, and Task success), and relating it to
product goals. Our approach is broader and more system-
atic, in the sense that we want to capture and model users,
and not only design user metrics, and relate these models
and measures to design principles for web applications.
It is unclear exactly what to measure and how to mea-
sure it and what the important measures are for specific
scenarios. It is also unclear how dierent aspects of user
engagement relate to each other in these scenarios. A main
observation is that there seems to be a lack of understand-
ing of how to integrate subjective and objective measures.
Note, for the former, experiments were specifically designed
to study user engagement or something related to it. And
whilst the latter hold the promise of being able to assess dy-
namic fluctuations in engagement, precise correlations with
engagement are yet to be established.
An important next step is to look carefully at the work
described above, and explore the possible mappings between
subjective and objective variables in specific contexts. Tak-
ing into account a range of interaction contexts adds to
the complexity of this task, but is imperative for enabling
progress. Experiments will have to be carefully designed,
and validation will need to be large-scale. Here we have ad-
dressed a number of approaches with potential for making
progress in this area. Whilst new ground needs to be made,
however, our review outlines a solid foundation of existing
work in HCI, IR and web evaluation which can provide the
basis for this work. Importantly, we aim to integrate map-
pings as empirically grounded models. In this way we want
to move towards a “science” of user engagement that can
have fundamental impact in how to design the user experi-
Our goal is to define a framework in which user engage-
ment can be studied, measured, and explained, and, as an
ultimate aim, lead to recommendations and guidelines for
user interface and interaction design for front-end web tech-
Our approach is holistic but convergent, following a gen-
eral progression from: (i) Empirical and analytic exploration
of the characteristics of user engagement (ii) Correlating and
resolving these to integrated models and reliable measures
both objective and subjective (iii) Design and validation of
Within the approach, we propose the following three main
directions of work: (1) identification of interaction patterns
and development of engagement measures for a range of con-
texts and services; (2) adaptation of immersion concepts
from gaming; and (3) designing for user engagement. The
first two lines of research try to define user engagement in
various contexts, along with metrics and methodologies for
measuring it. The third gives direct and practical means to
deploying definitions, metrics and methodologies for design-
ing engaging technologies.
These lines of research are not independent from each
other, but represent related aspects that will undoubtedly
influence and inform the overall approach for defining and
measuring user engagement. We elaborate on each in the
next sections.
3.1 Developing measures and models of user
The goal of the first line of investigation is to obtain an
understanding of how a range of user engagement character-
istics relate to specific user contexts and scenarios and how
they can be calculated using state-of-the-art web analytic
metrics, IR metrics, and existing or novel user engagement
This would expand on the work of O’Brian and colleagues
at the University of Dalhousie [28, 29]. In our opinion, their
work presents the most significant progress towards under-
standing and assessing user engagement to date. However,
whereas the Dalhousie group assumes generality across ac-
tivity domains and user groups, an important goal will be
to derive reliable and reusable metrics for user engagement
that are specific to particular contexts and online services.
In order to do so, it is necessary to collect and catalogue
existing and new engagement characteristics and indicators.
Such characteristics can be tested for correlation with a
range of existing measures, such as standard IR metrics for
assessing web search engines and web analytics measures.
This would involve setting up a series of “lab-based” user ex-
periments (combinations of virtual, online lab experiments,
actual lab environments, and in-situ studies) around specific
user tasks and usage scenarios in which users’ perceived val-
ues are studied for various characteristics through question-
naires and other subjective measures. Data would also be
collected through observation and logging.
It is important that experiments are carried out on a suf-
ficiently large and diverse sample of the user population and
using a (feasibly) wide range of design variants, user tasks
and motivations to ensure generalisability. To reduce bias,
appropriate sampling methods with respect to the user pop-
ulation have to be used. Psychological techniques such as
experience sampling2(e.g., [8]) can be used to gain insights
on user interactions.
From these experiments, dependency relationships can be
established among the dierent engagement characteristics,
leading to classes of user models for the dierent contexts
and task types. A specific point of interest is the relationship
between incentive structures, rewards, benefits and engage-
ment. The derived user engagement models can then be
validated through further experiments.
In order to assess the full range of variables of engagement,
it is necessary to test a multitude of scenarios, interaction
and graphics designs, for example, dierent page layouts,
result presentations, modified user interaction models (such
as result peek, scratch pad), etc. We anticipate the need for
continuous refinement to this methodology, allowing fluid
adaptation to the latest findings and the research questions
that such results raise.
Summarising, this line of research will deal with: (1) Clas-
sifying characteristics of user engagement as relevant to par-
ticular contexts, user tasks, and incentives, and in particu-
lar, validating characteristics relevant to web interactions
(extending our current work); (2) Identifying whether char-
acteristics can be automatically derived using standard or
more advanced web analytic measures or IR metrics and
developing prediction models when applicable; (3) Identi-
fying patterns of user engagement (leading to the develop-
ment of models of user engagement), both from qualitative
(surveys, questionnaires, focus groups, participatory design,
etc.) and quantitative points of view (through log analy-
sis); (4) Developing from this a methodology for measuring
and understanding user engagement, including methods for
comparative measures across brands and methods for failure
3.2 New perspectives from immersion in gam-
ing and related areas
The goal of the second line of research is to develop novel
concepts of user engagement through an examination of the
notion of immersion as it is used in gaming. Being immersed
2Essentially, participants fill out several brief questionnaires
every day by responding to alerts, where the questionnaires
ask about their current activities and feelings.
in a computer game involves being “drawn in”, with atten-
tion focussed entirely on the game [18]. As such, this phe-
nomenon has much in common with engagement and many
ideas emerging from this and related research may be ap-
plicable to more general notions of user engagement and
notions of user engagement on the web in particular. Some
early work on the application of the concept of immersion is
Bhatt’s [3] preliminary study of interaction with a range of
e-commerce web sites. Bhatt finds the relative importance
of immersion (compared to interactivity and connectivity) to
be genre-dependent; for example, it is far more significant
to the fashion industry than it is to the financial industry.
This finding concurs with our initial view of the applicabil-
ity of immersion to user engagement and also the context
dependant nature of user engagement.
In relation to immersion, of particular interest is the work
of Cairns and colleagues. Their early work [6] reported a
user study of immersion in gaming. They observed three
progressively deeper levels of immersion, which they referred
to as engagement, engrossment, and total immersion. In a
more recent paper [7] they review the general notion of user
experience in the light of their Core Elements of the Gaming
Experience (CEGE) model. CEGE is a hierarchical model
that elaborates a set of necessary conditions of an enjoy-
able gaming experience. For example, a positive experience
depends on the user taking “ownership” of a game, where
ownership involves developing high-level strategies and goals
and receiving rewards. But this itself is built upon lower-
level factors such as “control” elements (e.g., interface fac-
tors, small actions, point-of-view) and “facilitators” (such as
time to play and the user’s aesthetic values). Calvillo-Gamez
et al. [7] argue that many of these factors can be meaning-
fully extrapolated to other kinds of interaction, such as the
use of productivity applications.
We propose further exploration of the concept of immer-
sion and concepts from related domains (such as flow, and
presence in Virtual Reality experiences) in order to expand
and enrich existing notions of engagement. Underpinning
this work will be the aim of expanding our understanding
of the necessary constituents of an engaging experience. We
believe that this can bring valuable new perspectives to the
measurement of user engagement on the web.
The expected outcomes are: (1) Understanding the links
(and contrasts) between user engagement and immersion
(and to some extent flow and presence); (2) Leveraging this
understanding to enrich our model of user engagement; (3) De-
veloping additional metrics relevant to search and dierent
genres of web interaction; (4) Conceptual input into larger-
scale experimentation in tandem with the first line of work
(Section 3.1).
3.3 Designing for user engagement
This last line of research explicitly focuses on designing
for user engagement, by providing a useful source of vali-
dated design ideas. This is based on insights from the other
two lines of research to inform the development, validation
and refinement of design concepts (initiated in the first line
of research – Section 3.1) for services and tools aimed at
promoting user engagement.
To conduct such a research, an iterative, user-centred de-
sign process incorporating evaluation throughout, should be
adopted. Initial qualitative field studies can be used to iden-
tify and characterise particular kinds of interaction scenario
and their requirements. Scenarios should include individual
interaction as well as collaboration, sharing and social inter-
actions. Each scenario would provide a basis for developing
conceptual designs, which implement generalisable strate-
gies for promoting engagement appropriate to the context.
Prototypes ranging from low to high-fidelity can be devel-
oped and evaluated iteratively, with early formative evalu-
ations leading to the development of design improvements.
Summative evaluations can then be used to assess design al-
ternatives in terms of user engagement metrics developed in
other areas of the work. Overall the process will be aimed
at generating and testing generalisable design strategies for
promoting engagement in dierent contexts, ultimately re-
sulting in reusable recommendations, guidelines and exem-
plars which can be employed by other designers.
An approach to the cataloguing and communication of
design strategies, which we favour, is to record these in the
form of interaction design patterns [4]. Patterns are an ap-
proach to capturing reusable solutions to recurring design
problems within semi-formal representations. As such they
oer a convenient basis for generating a ‘lingua franca’ of
design solutions that encapsulate a particular set of design
values (in this case the promotion of engagement).
The expected outcomes of this line of research are to:
(1) Evaluate prototype solutions for promoting user engage-
ment in a range of contexts; (2) Produce generalisable design
recommendations in the form of reusable interaction design
In this paper, we advocate the development of an ap-
proach for studying, measuring, and explaining user engage-
ment. We first define user engagement, and then report on
and classify various characteristics and potential methods
for measurement. User engagement is a multifaceted, and
complex phenomenon; this applies to both its definition and
empirically grounded “signature” characteristics. Both of
these give rise to a number of potential approaches for mea-
surement whether objectively or subjectively.
We proposed three lines of research that have the potential
to shape what we like to refer as the science of user engage-
ment. The first two define methodologies for measurement,
whilst the third outlines how these methodologies can lead
to user engagement aware front-end technology design. Our
proposed approach is motivated by the aim of achieving an
innovative yet principled approach to the problems associ-
ated with measuring user engagement, building on existing
solid and fundamental work and methodology, and taking
inspiration from other related areas.
Acknowledgements Benjamin Piwowarski is supported by
EPSRC Grant Number EP/F015984/2. This paper was
written when Mounia Lalmas was a Microsoft Research/RAEng
Research Professor at the University of Glasgow.
[1] A. Al-Maskari, M. Sanderson & P. Clough. The
relationship between IR eectiveness measures and user
satisfaction. ACM SIGIR, 2007.
[2] D. Baldauf, E. Burgarda & M. Wittmann. Time perception
as a workload measure in simulated car driving. Applied
Ergonomics, 40(5):929-935, 2009.
[3] G. Bhatt. Bringing virtual reality for commercial web sites.
Human-Computer Studies, 60(1):1-15, 2004
[4] J. O. Borchers. A pattern approach to interaction design.
AI & Society, 15(4): 359-376 (2001)
[5] P. Bo r l u n d & I. Ruth ven . I n troduct i o n t o t he sp ecia l i s s u e
on evaluating interactive information retrieval systems.
IP&M, 44(1): 1-3 (2008)
[6] E. Brown & P. Cairns. A grounded investigation of
immersion in games. ACM CHI, pp 1297-1300, 2004.
[7] E. Calvillo-Gamez, P. Cairns & A. Cox. From the Gaming
Experience to the Wider User Experience. BCS HCI 2009,
pp 520-523, 2009.
[8] S. Consolvo & M. Walker. Using the experience sampling
method to evaluate ubicomp applications.
[9] M. Csikszentmihalyi. Flow: The psychology of optimal
experience. Harper and Row, 1990.
[10] J. Golbeck. Computer Science: Weaving a Web of Trust.
Science, 321(5896):1640-1641, 2008.
[11] R. Harper, T. Rodden, Y. Rogers & A. Sellen. Being
Human: Human-Computer Interaction in the Year 2020.
Microsoft, Cambridge, UK. 2008.
[12] M. Hassenzahl & N. Tractinsky. User experience - a
research agenda. Behaviour and Information Technology,
25(2):91-97, 2006.
[13] M. Huang. Designing website attributes to induce
experiential encounters. Computers in Human Behavior,
19(4):425-442, 2003.
[14] S. B. Human & M. Hochster. How well does result
relevance predict session satisfaction?. ACM SIGIR, 2007.
[15] C. S. Ikehara and M. E. Crosby. Assessing cognitive load
with physiological sensors. In Proceedings of the 38th
Annual Hawaii International Conference on System
Sciences, Washington, DC, USA, 2005. IEEE Computer
[16] W.A. Ijsselstein, H. de Ridder, J. Freeman & S.E. Avons.
Presence: Concept, determinants and measurement. SPIE,
[17] P. In g werse n & K . J a r ve l i n . T he Turn: I n t egra t i o n o f
Information Seeking and Retrieval in Context.
Springer-Verlag, 2005.
[18] C. Jennett, A. Cox, P. Cairns, A. Dhoparee, A. Epps, T.
Tijs & A. Walton. Measuring and Defining the Experience
of Immersion in Games. International Journal of
Human-Computer Studies 66(9):641-661, 2008.
[19] M. Jennings. Theory and models for creating engaging and
immersive e-commerce websites. ACM SIGCPR, 2000.
[20] G. Kazai & N. Milic-Frayling: Eects of Social Approval
Vot e s o n S e arch Perfo r m a nce. IT N G , 2 009.
[21] G. Kazai, N. Milic-Frayling & J. Costello. Towards
methods for the collective gathering and quality control of
relevance assessments. ACM SIGIR, 2009.
[22] R. Khare & A. Rifkin. Weaving a Web of trust. World
Wide Web J., 2(3):77-112, 1997.
[23] K. Kuuttii. Hunting for the lost user: From sources of
errors to active actors and beyond. Cultural usability
seminar, University of Art and Design Helsinki, 2001.
[24] E. L-C. Law, N. Bevan, G., Christos, M. Springett & M.
Larusdottir, (Eds.). Proceedings of the COST294-MAUSE
Wor k s h o p on Mea n i n g ful Me a s u r es: Val i d U s e ful Us e r
Experience Measurements (VUUM), 2008.
[25] E. Law, V. Roto, M. Hassenzhal, A Vermeeren & A. Kort.
Understanding, scoping and defining user eXperience: A
survey approach. ACM CHI, 2009.
[26] A. M¨
a & J. Fulton Suri. Supporting Users’ Creativity:
Design to Induce Pleasurable Experiences. International
Conference on Aective Human Factors Design, 2001.
[27] M. M. L. Mui. A Computational Model of Trust and
Reputation for E-businesses. HICSS - Volume 7, 2002.
[28] H. O’Brian & E. Toms. What is user engagement? A
conceptual framework for defining user engagement with
technology. JASIST, 59(6):938-955, 2008.
[29] H. O’Brian, E. Toms, K. Kelloway & E. Kelley. The
development and evaluation of a survey to measure user
engagement. JASIST 61(1):50-69 (2010)
[30] M. Obrist, V. Roto, & K. V¨
anen-Vainio-Mattila. User
experience evaluation: do you know which method to use?
CHI Extended Abstracts, 2009.
[31] K. Overbeeke, T. Djajadiningrat, C. Hummels, S.
Wen s veen & J. Fr e n s . Lets ma k e thing s e n g aging .
Fun o l o gy. Klu wer, 2 0 0 3 .
[32] E.T. Peterson. How do you calculate engagement? Part I.
Web A n a l y tics D e m i s tifie d ( b l o g)
do-you-calculate-engagement-part-i.html (accessed Oct
2009), 2006.
[33] B. Piwowarski & G. Dupret. Evaluation in (XML)
information retrieval: expected precision-recall with user
modelling (EPRUM). ACM SIGIR, 260-267, 2006.
[34] B. Piwowarski, G. Dupret & R. Jones. Mining user web
search activity with layered Bayesian networks or how to
capture a click in its context. ACM WSDM, 2009.
[35] J. Read, S. MacFarlane & C. Casey. Endurability,
engagement and expectations: Measuring children’s fun.
Interaction Design and Children (2):1-23, 2002.
[36] K. Rodden, H. Hutchinson, and X. Fu & Measuring the
User Experience on a Large Scale: User-Centered Metrics
for Web Applications. Proceedings of CHI, 2010.
[37] M. Rozendaal, D.V. Keyson & H. Ridder. Product features
and task eects on experienced richness, control and
engagement in voicemail browsing. Personal Ubiquitous
Computing, 13(5):343-354, 2009.
[38] F. Saal, R. Downey & M. Lahey. Rating the ratings:
assessing the psychometric quality of rating data.
Psychological Bulletin, 88 (2):413-428, 1980.
[39] N. Said. An engaging multimedia design model, Interaction
Design and Children. Interaction design and children:
building a community, pp 169-172, 2004.
[40] M.V. Sanchez-Vives & M. Slater. From presence to
consciousness through virtual reality. Nature Reviews
Neuroscience 6(4):332-339, 2005.
[41] J. Saipe. User Experience (UX) and Improving
Engagement. E-insight Blog: Usability & UX.
[42] M. Slater. Measuring Presence: A Response to the Witmer
and Singer Presence Questionnaire. Presence 8(5). 1999.
[43] P. Summerfield. Getting to engagement: Media agencies
start to realize it’s part art, part science. Strategy
Magazine, June, 2006.
[44] C. Tong Lee, E. Mendes Rodrigues, G. Kazai, N.
Milic-Frayling & A. Ignjatovic. Model for Voter Scoring
and Best Answer Selection in Community Q&A Services.
WI, pp. 116-123. 2009.
[45] N. Tractinsky, A.S. Katz & D. Ikar. What is beautiful is
usable. Interacting with Computers (13):127-145, 2000.
[46] E. Ulken. Measuring user engagement: Lessons from
BusinessWeek. Online Journalism Review
(accessed Oct 2009), 2009.
[47] K. V¨
anen-Vainio-Mattila, V. Roto & M. Hassenzahl.
Now Let’s Do It in Practice: User Experience Evaluation
Methods in Product Development. CHI, 2008.
[48] J. Webster & H. Ho. Audience engagement in multimedia
presentations ACM SIGMIS, 28(2):63-77, 1997.
[49] R.W. White & S. T. Dumais. Characterizing and predicting
search engine switching behavior. ACM CIKM, 2009.
[50] J. Wortham. The Blog That Ignited a Privacy Debate on
Fac e b o o k , N e w York Tim e s . Fe brua r y 1 8 , 2009.
[51] E. Yilmaz, M. Shokouhi, N. Craswell & S. Robertson,
Incorporating user behavior information in IR evaluation.
SIGIR UUIR workshop, 2009.
... • Attentional Engagement (AE): More user time spent indicates higher engagement (Attfield et al., 2011). We model AE of a post by examining whether it has editted replies, and the information specificity in the replies. ...
... We model AE of a post by examining whether it has editted replies, and the information specificity in the replies. • Behavioral Engagement (BE): Human behavioral features closely correlate with their engagement state (Attfield et al., 2011), and we model BE by examining Reddit post scores, adjusted by popularity. • Reply Engagement (RE): Following definition from (Ghazarian et al., 2020), if a certain post is very likely to be continued by following threads, it is considered engaging; reply_counts are also popularity adjusted. ...
... Measuring user engagement of a website is not a straightforward task. In existing works, measuring user engagement commonly involves three approaches, which are self-reported engagement assessment, cognitive engagement assessment, and online behavioral metrics [8], [9]. Self-reported engagement assessment uses questionnaires or interviews to get user engagement attributes. ...
Full-text available
User engagement is a part of user experience that measures how attracted the users are in using a certain product or services. The level of user attention while navigating a website is a key factor in determining its effectiveness. Web usage mining can be performed on the weblog of a website to extract user navigational behavior and gain valuable insights into the useful activities of the users. The user navigational behavior can then be further evaluated to measure the user engagement of the website. This paper presents a new framework that uses user navigational behavior extracted from web usage mining to measure the engagement level of the users on the website. In this proposed framework, web session clustering is first performed on the pre-processed weblog to group similar user access patterns from web sessions. After that, four proposed engagement metrics, which are the hourly activeness, the hourly traffic, the daily activeness, and the daily traffic are calculated for each cluster to determine the engagement level of the users on the website. According to the engagement level determined by the user engagement metrics, an attention score is formulated to show how attracted the users in the session cluster are to the website. In this paper, two weblogs from different websites are used to measure their user engagement with the Cluster-N-Engage framework. The framework shows that attention score can assess the website’s effectiveness in achieving its objectives.
... The word 'user engagement' refers to the relationship that exists between users and the technology that they utilise. There are several descriptions and definitions of user engagement available, including one by Attfield et al., 15 who describes it as 'the emotional, cognitive, and behavioral connection that exists, at any point in time and possibly over time, between a user and a resource' (p. 10). ...
Full-text available
Objective Despite the fact that user engagement is critical to the efficacy of mobile health (mHealth) interventions in the Global South, many of these interventions lack user engagement features. This is because sociotechnical aspects of such initiatives are frequently ignored during the design, development, and implementation stages. This research highlighted the importance of considering sociotechnical factors when developing mHealth apps. The intended users for the mHealth technologies in this study are care professionals. Materials and Methods Five semi-structured interviews and a pilot interview were conducted to identify user engagement facilitators and barriers. The interview data were analysed using NVivo. The Capability, Opportunity, Motivation – Behaviour (COM-B) model is then used to map the facilitators and barriers to mHealth app engagement, allowing researchers to better understand how users engage/disengage with mHealth apps. Results and Discussion Capability facilitators included features that assist users in learning more about the app (e.g. a user manual and statistical data) as well as features that assist users in developing a routine. The lack of app skills and cognitive overload limit capability. While social connectedness and offline functionality were identified as facilitators of user engagement, non-user-friendly design and cultural dimensions were identified as barriers. Early user engagement and rewards were identified as motivational facilitators that influence user engagement. Furthermore, perceived non-utility and a lack of encouragement were identified as motivational barriers to engagement. Conclusion Several factors were discovered across all COM-B model components that could be used to develop more engaging mHealth apps. Adopting a techno-centric approach that ignores sociotechnical factors can reduce user engagement. The design process engagement enhancement system (DECENT) framework was proposed based on the findings.
... Online shopping and e-commerce have created an innovative and profitable business model. Both consumers and sellers connect to social media and ask questions about products and monitor products and their features (Attfield et al., 2011). The COVID-19 pandemic has empowered social media users to create new partnerships that will likely lead to stronger communication over time (Almansoori and Habtoor, 2018). ...
The use of social media to promote clothing and textile businesses has revealed several changes in consumer behavior during the post-COVID-19 era. In this study, the social media tools which provide useful information in regards to consumer preferences and their decision-making processes in the fashion industry are presented. Moreover, the marketing strategy used and the strong presence of social media influencers in the fashion industry both during and after the COVID-19 pandemic period are also examined. A mixed research methodology is applied through the conduction of interviews with 20 owners of small fashion businesses and a questionnaire survey of 200 consumers who use social media for their purchases from the region of Thessaly in Central Greece. Furthermore, the results are analyzed and the research findings are discussed in detail. Finally, suggestions for further research are also included.
Full-text available
In the context of digitization, traditional journalism is facing transformation and innovation. Among them, the interactivity of visual graphics in data journalism is crucial for attracting and retaining online users, but few studies have examined public perceptions and attitudes toward it. In this study, we proposed a model to validate the relationship between users’ perceived interactivity and their attitudes toward data journalism, and we included user affective and cognitive factors (enjoyment and engagement) related to this as possible mediating variables in the model for validation. We conducted experiments (n = 75) using data journalism containing map visualizations with three levels of interactivity (low, medium, and high) in China. Furthermore, an exploratory evaluation of the experimental group provided further insights into the differences in interactions between groups, and the emerging five key concepts of data journalism design. Overall, all our hypotheses are supported, with enjoyment and engagement mediating the relationship between perceived interactivity and users’ attitudes toward the news. In addition, the experimental group with higher interaction potential also reported more positive attitudes toward journalism. Therefore, if data journalism and visualization designers want to attract and retain users in the future, enhancing user interaction on news pages will be a proven method.
Full-text available
Website is one of the OGD initiatives that provide input and spread information to the public. This research’s objective is to develop a framework for citizen engagement in using OGD’s website, i.e. Ministry’s websites. This research proposes a few combinations of determinants from the Theory of Human Behavior, Technology Acceptance Model (TAM), Theory of Planned Behaviour (TPB), Theory of Reasoned Action (TRA), and Information System Success Model (ISSM). This research will provide input and information on how the implementation of OGD in Malaysia can become a sustainable innovation in developing a user-friendly website to attract engagement from the public.
Novelty, anomaly, or out-of-distribution (OOD) detection has been an active research area for decades, since the 1960s [1, 2], due to its widespread applications in various domains [3, 4], such as financial surveillance, health and medical risk, AI safety, network intrusion detection, etc. Recently, this research has also received increased attention in the Natural Language Processing (NLP) domain. For example, many researchers have studied the problem of text classification [5,6,7,8,9], but, as opposed to classical text classification that works under the closed-world assumption, novelty or OOD detection works under the open-world setting, which allows the class of a test instance to be unknown. However, the existing text classification methods that perform novelty or OOD detection are mainly coarse-grained and topic-based. Given a text document, their goal is to detect whether the text belongs to a known topic or an unknown topic. Some examples include detecting product reviews of unknown product categories [7], news articles of unknown topics [6], and natural language commands of unknown intents/domains [9]. This chapter is based on two papers published in 2021 and 2022 [10, 11]. They introduce a new novelty detection problem in the NLP field—fine-grained semantic novelty detection in texts, which we also call contextual novelty detection. In this chapter, we want to present a learning-based agent for semantic novelty detection in text. The agent does not actively interact with the world. The perceptual operators are Deep Neural Networks (DNN) built on text features created from large language models and external knowledge bases.
The classification of living organisms is essential to understand biodiversity and, therefore, to support its conservation. The taxonomic system classifies living organisms into hierarchically organized groups. Classification is not always a simple task, as even today taxonomists face the challenge that data are frequently revised, and species may be reclassified into other groups. Visualization of hierarchies, particularly biological taxonomies, has been a relevant area of research to support this challenge. Previous user studies determined that expert taxonomists preferred the edge-drawing method to visualize changes between two versions of a taxonomy, using the tool called Diaforá, which presents a single view. In this paper we propose a visualization that consists of using the coordinated multiple view (CMV) technique and focuses on displaying only the differences between two alternative versions of a biological taxonomy. The hypothesis is that the proposed visualization would provide a better cognitive approach to users, facilitating the identification of taxonomic changes. The work consisted first in designing other views and, through a user study, selecting one of them to be integrated into Diaforá to form a CMV system. Then, a second user study was carried out to contrast the original version of Diaforá with its extended version. Participants performed a series of tasks and we obtained measures of effectiveness, efficiency, trust rate and overall engagement rate. Results indicate that multiple coordinated views (CMV) improved users’ performance in identifying differences.KeywordsInformation visualizationHuman-computer interactionHierarchy comparisonCoordinated multiple views
Full-text available
Many evaluation measures in Information Retrieval (IR) can be viewed as simple user models. Meanwhile, search logs provide us with information about how real users search. This paper describes our attempts to reconcile click log in- formation with user-centric IR measures, bringing the mea- sures into agreement with the logs. Studying the discount curve of NDCG and RBP leads us to extend them, incorpo- rating the probability of click in their discount curves. We measure accuracy of user models by calculating 'session like- lihood'. This leads us to propose a new IR evaluation mea- sure, Expected Browsing Utility (EBU), based on a more sophisticated user model. EBU has better session likelihood than existing measures, therefore we argue it is a better user-centric IR measure.
Full-text available
In this paper we discuss the different elements of the gaming experience and their relation to other concepts within HCI. The objective is to showcase how the different elements that form the gaming experience can be used to understand further issues regarding user experience. The objectives of games are, after all, to provide players with a positive experience. Understanding the elements that eventually lead players to have a positive experience should provide feedback about the wider user experience concept. Although video-games and non-game applications seem to be two different domains of study, in terms of experience, they both aim to improve the individual's experience.
Full-text available
Over the last decade, 'user experience' (UX) became a buzzword in the field of human – computer interaction (HCI) and interaction design. As technology matured, interactive products became not only more useful and usable, but also fashionable, fascinating things to desire. Driven by the impression that a narrow focus on interactive products as tools does not capture the variety and emerging aspects of technology use, practitioners and researchers alike, seem to readily embrace the notion of UX as a viable alternative to traditional HCI. And, indeed, the term promises change and a fresh look, without being too specific about its definite meaning. The present introduction to the special issue on 'Empirical studies of the user experience' attempts to give a provisional answer to the question of what is meant by 'the user experience'. It provides a cursory sketch of UX and how we think UX research will look like in the future. It is not so much meant as a forecast of the future, but as a proposal – a stimulus for further UX research.
Full-text available
This article presents the preliminary findings from an explorative case study concerning barriers, benefits and use of SMEs adoption of business-to-business Internet commerce. The main findings were that SMEs embrace the Internet mainly just by chance ...
Full-text available
Reviews research that is concerned with evaluating the psychometric qualities of data in the form of ratings (rating errors) and that has been plagued with conceptual and operational confusion and inconsistency. Following a brief historical survey, inconsistencies in definitions, quantifications, and methodologies are documented in a review of more than 20 relevant articles published in Journal of Applied Psychology, Organizational Behavior and Human Performance, and Personnel Psychology (1975–1977). Empirical implications of these inconsistencies are discussed, and a revised typology of rating criteria, combined with a multivariate analytic approach, is suggested. (65 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
This paper describes a toolkit for measuring fun with children aged between 5 and 10. The relationship of fun to usability is discussed and three dimensions of fun; Endurability, Engagement, and Expectations are described. A set of tools that can be used in empirical studies to measure these dimensions of fun is described in detail. These tools include a Smileyometer, a Fun-Sorter and an Again-Again table. Three user trials, in which the tools have been used, are described and suggestions are made for the use of each of the tools with children. A discussion of the efficacy of the tools presents the findings that there is a sufficient subset that can be used to give a reliable measure of fun. Dependent on the nature of the event being evaluated and on the number of activities being compared; the subset will be differently constructed. The paper concludes with observations about the difficulties of measuring fun with children, and some areas for further work are presented.