Content uploaded by Morten Hertzum
Author content
All content in this area was uploaded by Morten Hertzum on Jan 01, 2018
Content may be subject to copyright.
P. Kotzé et al. (Eds.): INTERACT 2013, Part II, LNCS 8118, pp. 461–478, 2013.
© IFIP International Federation for Information Processing 2013
Do Usability Professionals Think about User Experience
in the Same Way as Users and Developers Do?
Torkil Clemmensen1, Morten Hertzum2, Jiaoyan Yang3, and Yanan Chen3
1 Department of IT Management, Copenhagen Business School, Denmark
tc.itm@cbs.dk
2 Computer Science and Informatics, Roskilde University, Denmark
mhz@ruc.dk
3 Institute of Psychology, Chinese Academy of Science, Beijing, China
{chenyn,yangjy}@psych.ac.cn
Abstract. In this paper, we study how usability professionals’ thinking about
system use relates to that of system developers and end users. We conducted 72
repertory-grid interviews to capture how usability professionals, developers,
and users describe their system use. The participants in each stakeholder group
were from China, Denmark, and India. Our results indicate that usability
professionals focus on emotion-related aspects of system use, while users focus
more on context in terms of utility and degree of usage. There are no
interactions between stakeholder group and nationality, although both
stakeholder group and nationality independently influence how participants
think about usability and user experience. We recommend that to understand
users’ concerns, researchers should study context more.
Keywords: Usability professionals, UX professionals, user experience,
stakeholder differences, cultural differences.
1 Introduction
In this paper, we focus on how usability professionals’ thinking about system use is
different from other stakeholder groups, in particular system developers and end users.
By usability professionals, we mean people with titles such as usability professional,
user experience manager, user experience specialist, user researcher, customer-
experience architect, human factors specialist, interaction designer, usability engineer,
and more. Differences between usability professionals and other stakeholders in their
thinking about system use can be expected for several reasons. First, usability
professionals’ and system developers’ thinking about system use may differ from that
of end users, because professionals have systems as the objects of their professional
work, whereas users apply systems to do their activities. That is, the “computer
application, from the user’s perspective, is not something that the user operates on but
something that the user operates through on other objects or subjects.” [1, p. 1].
Second, usability professionals’ thinking about system use may differ from that of
developers and users, because usability professionals will tend to be in an evaluation
462 T. Clemmensen et al.
mode (trained to reflect explicitly about use), in contrast to system developers and
users. Third, there are cultural differences in how people experience the use of
interactive systems and products, and also in the way people respond to usability
evaluation techniques, and in how they share their experiences of using a system [22].
The fit between the different stakeholder groups’ view of system use is a basic
raison d'être for the usability profession. A better fit between users and usability
professionals in their thinking about system use, compared to users and system
developers, has been the main argument for a usability profession since the beginning
of the profession [20]. According to Putnam and Kolko [17, p. 2021] a HCI
professional [their term] should be able to “walk in the users’ shoes”. The usability
profession emerged because design-oriented developers did not have such empathy
with the users.
Recently, two developments have made it imperative to study in depth how
usability professionals think differently than users and developers about the use of
systems. First, the usability professionals’ identity has expanded into a much broader
User Experience (UX) professional identity [3]. According to Dumas [3], some
usability professionals with backgrounds in psychology and human factors perceived
this development as a degradation of the skills of the profession, while other usability
professionals accepted it as an expansion of the profession. To reconcile such
incompatible beliefs, more knowledge about what distinguishes the thinking of a
usability professional from that of developers and users is much needed.
Second, the globalization of the interactive systems industry has led to a worldwide
quest for locally available usability and UX skills. The tasks of a global UX team can
include designing a new interactive system for global users, localizing an existing
system for national users, designing completely new system concepts for the local
users, providing UX management for local developers, and more [19]. In such
projects, the geographical location of stakeholders and the availability of specialized
UX skills in a particular geographical location are important. UX skills are not
distributed evenly across geographical locations but depend, among other things, on
the mix of multidisciplinary university and professional training available locally
[19]. According to Rosenberg and Kumar [19], local UX professionals may be needed
to study domain specific requirements, while general issues like UI patterns and
guidelines might be shared across countries. It appears to be a best practice in the
computing industry to hire local professionals because they are assumed to be in some
way different from professionals in other countries.
In this paper, we investigate differences between usability professionals, users, and
developers in how they think about system use. We do this based on 72 repertory-grid
interviews. We also examine whether nationality (China, Denmark or India) affects
the differences between the three stakeholder groups. We use four UX classifications
to make a content analysis of the interviews. This paper contributes with insights into
the unique operational constructs held and used by usability professionals, compared
to the operational constructs used by other stakeholder groups in systems
development. These insights are important to usability professionals’ practices when
they design and evaluate UX. The insights should also improve communication about
UX between stakeholder groups in systems development.
Usability Professionals Think about User Experience in the Same Way 463
2 Background
Redish and Barnum [18] stated that from the late 1990s the usability profession opened
up to technical communicators, information architects, interaction designers, content
strategists, and others who were not part of the human factors or software engineering
traditions from the beginning of the usability profession. As a part of this evolution, the
usability profession gradually became a broader UX profession, culminating with
Usability Professionals’ Association (UPA) changing its name to User Experience
Professionals Association (UXPA). Dray [2, p. 5] described how it became more
difficult to identify a usability professional as one who practiced the usability test
methodology: “In the past, usability could lay claim to a professional identity based on
a particular set of practices or methodology—usability testing.”. So what is it that
distinguishes the usability professional from other IT professionals? Putnam and Kolko
[17] proposed that we should study the differences between professionals who bear
user-oriented job titles such as usability engineer and those who have design-oriented
job titles such as developer. In a pilot study, they identified three professional groups:
(1) design-oriented such as developers and designers, (2) UX oriented such as UX
architects and usability professionals, and (3) management oriented. They found
significant differences between the groups in their use of a range of methods to study
and communicate about end-users, for example, the UX group reported a higher
average use of contextual inquiry methods. They also found that the design-oriented
group showed more empathy in how they considered users, compared to the UX
oriented group, as measured by scales of emphatic concerns and personal distress
[17]. However, the differences in empathy were not significant. Putnam and Kolko
[17] concluded that we need to study differences, including regional differences,
among the professionals who create products and those who conduct user research.
According to Lindgaard [15], usability professionals traditionally had their basic
training in the behavioural and social sciences. Shackel [20] described the usability
profession as being particularly focused on the evaluation of users’ task completion
with interactive tools in specified environments. Thus, while being a part of the
design process, usability professionals’ work was from the beginning of the
profession defined as having to do with the analysis of user tasks and needs, and
the forwarding of the resulting knowledge to developers. According to Lindgaard
[15], the focus was on performance measures all the way from usability metrics of job
task effectiveness and efficiency to broad business-level concerns. To the extent that
the users’ emotions have been in focus, the efforts have been directed toward helping
the user avoid frustration, see for example [14]. Among others, Dumas [3] and
Lindgaard [15] point out that although the concepts of attitude and satisfaction toward
an interactive system were part of the classic understanding of usability, the broader
field of user experience was still to come.
Early in the history of the profession, ‘user’ was the term used to denote the
stakeholder group that interacted with the system to obtain some goal, and who did
not have much interest in, or knowledge about, the technology itself. As Shackel [20,
p. 339] put it: “For many users the informatics system is essentially the terminal or
workstation which they are using, and that is the central computer as they see it”.
Thus, although the user was not attending to the system as such, s/he was acutely
aware that she or he was using a system. In contrast, Shackel [20] claimed that
464 T. Clemmensen et al.
developers tended to forget about the user and focus on the technological artefact.
Shackel [20, p. 342] describe this: “Too easily this [setting goals and specifying
attributes of the design] may be seen solely as the work of the designers… [but] to
achieve usability in the ultimate design it is essential that users and managers are fully
involved”. The stakeholder groups of user, developer, and usability professional were
thus defined as being different in their relation to each other and to the technology.
Irrespective of the recent efforts of the HCI community to define usability and
institute UX standards [10], people relate to systems through their personal
experience and concepts. Hertzum, Clemmensen, Hornbæk, Kumar, Shi and
Yammiyavar [9] analysed 48 repertory-grid interviews with developers and users to
see how their personal usability constructs were affected by nationality (Chinese,
Danish, or Indian) and stakeholder group (developer or user). For stakeholder groups,
a significant overall difference between developers and users was found. The
differences included that users associated ease of use with leisure and, conversely,
difficulty in use with work-relatedness, while developers expressed the reverse
associations. There was no significant overall difference across nationalities in
participants’ thinking about usability.
Hertzum and Clemmensen [8] analysed how 24 usability professionals’ thinking
about usability fitted with analytic usability definitions, and found that usability
professionals made use of more utilitarian than experiential, i.e. UX related, constructs.
This indicates that goal-related performance was central to usability professionals’
thinking about usability, whereas they had less elaborate sets of experiential constructs.
The usability professionals tended to construe usability at an individual level, rather
than at organizational and environmental levels. However, the usability professionals’
constructs went considerably beyond what was captured by the ISO 9241 definition of
usability, thereby indicating a discrepancy between this definition of usability and the
thinking of the professionals concerned with delivering usability.
We are unaware of studies in which usability professionals are compared
systematically with other stakeholder groups. Thus, it remains unknown whether
usability professionals think more like users than developers do. We need to know more
about the usability constructs of usability professionals and other stakeholder groups,
and in particular, how they may differ in the way that they think about user experience.
3 Method
In this paper, we merge the data from our two previous studies [8, 9] and, for the first
time, provide an analysis of how usability professionals’ thinking about user
experience compare to that of developers and users. As described above our previous
studies have analysed usability professionals in isolation [8] and compared developers
and users [9]. While the data in the current paper are re-used from the two previous
studies, the analysis of the data is completely new.
We analyse repertory-grid interviews with 72 people to capture how they described
their system use. The repertory-grid technique was developed by Kelly [11] as part of
his personal construct theory. It has been successfully used in many studies of
Usability Professionals Think about User Experience in the Same Way 465
systems development, design, and use [7, 21]. We used local interviewers in the three
involved countries to help make the data collection cross culturally valid.
3.1 Participants
The 72 participants were 24 usability professionals, 24 system developers, and 24 end
users. Within each of these three stakeholder groups there were eight participants from
China, eight from Denmark, and eight from India (see Table 1). To ensure that the
participants were representative of the general cultural context of computer use in their
country, participants from each country had to be a resident of the specific country and
have been raised in the country. We recruited through our personal networks, and the
participants were thus a convenience sample. All participants had good English skills.
All participants used text-processing, web applications, and email every day, or nearly
every day. They all stated that their computer hardware was sufficient, that they were
comfortable with computers, and that they felt neither anxious nor relaxed/indifferent
when they ran into a problem with a computer or application.
Table 1. Demographic information about the 72 participants
Group Gender Age (years)
IT experience
(years)
Male Female M SD M SD
Chinese usability prof. 5 3 26.6 ±3.2 9.8 ±2.4
Chinese developers 5 3 31.5 ±1.9 10.6 ±1.7
Chinese users 5 3 27.3 ±1.9 8.4 ±1.9
Danish usability prof. 3 5 32.5 ±4.6 18.6 ±6.1
Danish developers 5 3 36.6 ±5.8 19.3 ±5.8
Danish users 5 3 36.8 ±6.2 16.9 ±3.6
Indian usability prof. 7 1 29.9 ±1.6 9.5 ±2.7
Indian developers 8 0 29.6 ±1.7 9.9 ±2.5
Indian users 5 3 29.0 ±4.0 7.0 ±2.1
3.2 Procedure
The procedure was similar to the procedure proposed by Kelly [11]. Participants were
interviewed individually at their workplace by a local interviewer; that is, a person
with the same nationality as the participant. Participants filled out a questionnaire
about their background and tried to elicit constructs with the repertory-grid technique
on a couple of training tasks. Then the actual repertory-grid interview was conducted.
It consisted of two steps: selection of systems and elicitation of constructs.
In selecting systems, the participant was asked to consider “the array of computer
applications you use for creating, obtaining, revising, managing, and communicating
information and documents in the course of your day-to-day activities.” We
maintained a focus on the participants’ work by interviewing them at their workplace
and by encouraging them to look for candidate systems at their workplace computer
(in the start menu). Participants had to select a system within each of six categories:
466 T. Clemmensen et al.
my text processing system, my email, a useful system, an easy-to-use system, a fun
system, and a frustrating system.
To elicit construct-contrast words, the participant was given groups of three of the
selected systems and asked: “Can you think of some important way in which your
personal experience using these three systems makes two of the systems alike and
different from the third system?” Having indicated the two similar systems, the
participant wrote down a short phrase that explained how these two systems were
alike – the construct – and another short phrase that explained how the third system
differed – the contrast. Then, a seven-point rating scale was defined with this
construct/contrast pair as its end points, and the participant rated all six systems
according to this rating scale. The construct-elicitation step was repeated for all
twenty combinations of three systems, in random order, or until the participant was
unable to come up with a new construct for two successive combinations. In this way,
each participant elicited his or her personal grid of constructs.
Following local customs, Danish and Indian participants received no compensation
for their participation in the study while Chinese participants were paid a small
amount in RMB for their participation. Each interview lasted about 1.5 hours.
The repertory-grid interviews were conducted by three local interviewers: a
Chinese, a Dane, and an Indian. The interviewers were all HCI researchers. We
followed closely the prescriptions from Kelly [11] for doing repertory grid interviews.
After a pilot interview, we met for sharing experiences and creating consensus about
how to do the interviews, while maintaining the benefits of having local interviewers.
3.3 Data Analysis
Microsoft Word was the dominant choice of text-processing system among the
participants, and Microsoft Outlook was the most frequently selected email system. For
the four other types of system, there was more diversity among the selected systems.
Most of the selected systems were software that is used all around the world. The
selection of these systems did not appear to be biased by the participants’ nationality.
We analysed the elicited constructs by categorizing them according to four UX
classifications, Table 2. The first classification distinguished between system, user,
and context of use. The ISO standard for UX [10] lists these three factors as those that
influence the subjective UX. The second classification contrasted subjective and
objective experiences, by following the suggestion in Han [5, 6] that objective
performance and subjective image/impression are both important dimensions. This
classification was chosen because it gave detailed definitions of the two categories
and could be expected to capture broadly any UX related constructs [5, 6]. In
addition, the classification seemed to reflect the utilitarian/experiential dimension
used by other researchers. The third and the fourth classifications were chosen to
maximize the differences among the four classifications. We included Kujala, Roto,
Väänänen-Vainio-Mattila, Karapanos, and Sinnelä [12]’s classification of UX into
general relation to the system, attractiveness of the system, ease of use, utility, and
degree of usage because all of these categories were defined with a focus on the IT
Usability Professionals Think about User Experience in the Same Way 467
system. In contrast to this system-oriented UX definition, we chose McCarthy and
Wright [16]’s four threads of experience: compositional, sensual, emotional, and
spatio-temporal, because they focus on human experience in general.
Table 2. The classifications used in categorizing the constructs
Classification Category definitions
ISO 9241-210 user experience
System-related All components of an interactive system that provide information
and controls for the user to accomplish specific tasks with the
interactive system
User-related Emotions, beliefs, preferences, perceptions, physical and
psychological responses, behaviours and accomplishments that
occur before, during and after use.
Context of use-related Other people, tasks, equipment, and the physical, social, temporal,
organizational and cultural environments in which a product is used
Objective vs. subjective UX
Performance How well users perceive and interpret the interface of a system,
how fast the users get used to the product and how well they
remember it, and the users control activity and its results;
Image/Impression Basic sense (the primitive and direct image/impression stemming
from the design characteristics), description of image of a system,
and evaluative feeling
System-oriented UX
General relationship and
user experience with the
system
Any experience with systems that users find meaningful and
important
Attractiveness of the
system
General attractiveness (appeal) of the product in the users’ own
eyes and those of their friends; more than users’ rational or
practical experiences
Ease of use of the system Product is easy and effortless to use
Utility of the system Product serves an important function for the user
Degree of usage of the
system
Degree of usage which affects user experience over time, related to
quality of experience
Human experience of technology
Compositional The way that different elements of experience form a coherent
whole; the narrative structure, action possibility, plausibility,
consequences and explanations of actions
Sensual The concrete, palpable (to take in your hand), and visceral (in flesh
and blood) character of experience that is grasped pre-reflectively
in the instant situation
Emotional Value judgments (e.g., frustration and satisfaction) that ascribe
importance to other people and things with respect to the user’s
needs and desires
Spatio-temporal What draws attention to the quality and sense of space-time that
pervades experience
468 T. Clemmensen et al.
We categorized all constructs according to the first classification before we
proceeded to the second classification, then categorized all constructs according the
second classification, and so forth. For each classification, the categorization of
the constructs involved five steps. First, the first author thoroughly discussed the
classification with to the two coders, authors three and four, in order to ensure a
common understanding of the classification. Second, a randomly selected training set,
consisting of 20% of the constructs, was categorized by each coder independently.
Each construct was assigned either to one of the categories of the classification or to
an ‘other’ category. Third, all disagreements in the coders’ categorizations of the
training set were discussed to reach consensus about the categorization of the
constructs. The rationale for the second and third step was to create a shared
understanding of how the classification was used. Fourth, the remaining 80% of the
constructs were categorized by the two coders independently to be able to assess the
quality of the categorization. Fifth, all disagreements in the coders’ categorizations of
these 783 constructs were discussed and a consensus was reached, in order to arrive at
the final classification. used in the analysis.
The Kappa values of the agreement between the coders in their coding of the 783
non-training constructs were 0.55, 0.62, 0.61, and 0.68 for the ISO 9241-210 UX,
objective vs. subjective UX, system-oriented UX, and human experience of
technology classifications, respectively. Whereas all four Kappa values indicate
statistically significant agreement, the value of 0.55 for the ISO 9241-210 UX
classification is below the minimum threshold of 0.60 recommended by Lazar et al.
[13]. Since this classification was important, and the agreement was approaching the
threshold, we decided to keep the classification in the analysis.
4 Results
The 72 participants elicited a total of 977 construct/contrast pairs, corresponding to an
average of 13.57 pairs per participant. The minimum number of construct/contrast pairs
elicited by a single participant was three, the maximum 20. Below, we first analyse the
constructs for effects of stakeholder group, then look into effects of the participants’
nationality, and finally investigate interrelations across the classifications.
4.1 Differences between Stakeholder Groups
To analyse whether usability professionals had different user experiences than
developers and users Table 3 gives, for each stakeholder group, the average
percentage of constructs in a category across the participants in that stakeholder
group. We used multivariate ANOVAs to test for stakeholder differences in the
distribution of constructs across all categories in a classification and univariate
ANOVAs for the individual categories. The statistical analysis was performed on the
percentage distributions of the constructs (columns 2 to 4 of Table 3); this was done
to assign equal weight to participants, irrespective of the number of constructs elicited
by a participant. Before conducting the statistical analyses, the percentage values for
each participant were arcsine transformed because percentages cannot be assumed
Usability Professionals Think about User Experience in the Same Way 469
normally distributed [4]. All pair-wise comparisons reported below were Bonferroni
adjusted to compensate for multiple comparisons.
For the classification of the constructs into system-, user- and context-related ISO
9241-210 UX there was a significant effect of stakeholder group, Wilks’ λ = 0.77, F(8,
132) = 2.35, p < 0.05. While there were no differences across stakeholder groups for the
percentage of system-related constructs (e.g., “Address register/No address register”),
F(2, 70) = 0.99, p = 0.38, and other constructs (e.g., “Copyright protected (license
needed)/Download from the internet for free”), F(2, 70) = 0.19, p = 0.83, there were
significant differences across stakeholder group for user-related constructs (e.g., “Lot of
fun to use/Hard job”), F(2, 70) = 5.55, p < 0.01, and for context- related constructs (e.g.,
“Communication tool/Work tool“), F(2, 70) = 6.36, p < 0.05. Pair-wise comparisons
showed that usability professionals had a higher percentage of user-related constructs
than both developers (p < 0.05) and users (p < 0.01); developers and users did not differ
significantly from each other. In contrast, usability professionals had a lower percentage
of context-related constructs than users (p < 0.001).
For the classification of the constructs according to objective vs. subjective UX
there was a significant effect of stakeholder group, Wilks’ λ = 0.81, F(6, 134) = 2.47,
p < 0.05. While there were no differences across stakeholder groups for the
percentage of performance constructs (e.g., “"Efficiency/Productivity”), F(2, 70) =
1.79, p = 0.18, and other constructs (e.g., “Environment driven...peer group team
members involved in its usage/Personal, self-driven“), F(2, 70) = 2.06, p = 0.14, there
was a significant difference across stakeholder group for image/impression constructs
(e.g., “Overwhelming interface [means that the interface has too many
features]/Enjoyable, rich interface”), F(2, 70) = 5.60, p < 0.01. Pair-wise comparisons
showed that usability professionals had a higher percentage of image/impression
constructs than developers (p < 0.05) and users (p < 0.01); developers and users were
not significantly different from each other.
The classification of the constructs according to system-oriented UX showed a
significant effect of stakeholder group, Wilks’ λ = 0.63, F(12, 128) = 2.82, p < 0.01.
We found significant differences for the categories of attractiveness (e.g., “Nice-
looking UI/Ugly UI”), F(2, 70) = 7.83, p < 0.01, and utility (e.g., “For playing
movies/Do scientific analysis“), F(2, 70) = 6.87, p < 0.01. Pair-wise comparisons
showed that usability professionals had a higher percentage of attractiveness constructs
than developers (p < 0.05) and users (p < 0.01); developers and users were not
significantly different from each other. In contrast, usability professionals had a lower
percentage of utility constructs than users (p < 0.01). There were no differences across
stakeholder groups for general relationship and user experience (e.g., “Bright and fluid
[means good to look at…organic]/Static [means very rigid in structure…not
organic]“), ease of use (e.g., “Demands more clicks/Demands fewer clicks“), degree of
usage (e.g., “Frequent use/Seldom use“), and other (e.g., “Install package is small/Size
is too large“), Fs(2, 70) = 2.35, 1.50, 0.94, and 0.94 respectively (all ps > 0.1).
For the classification of the constructs according to human experience of technology
there was a significant effect of stakeholder group, Wilks’ λ = 0.76, F(10, 130) = 1.95,
p < 0.05. While there were no differences across stakeholder groups for the categories
compositional (e.g., “Can be used independent of other applications/Have to be used
with other applications“), sensual (e.g., “Complex/Simple“), spatio-temporal (e.g.
470 T. Clemmensen et al.
“Available everywhere/May not be available everywhere“), and other (e.g., ”On
cd/Downloaded“), Fs(2, 70) = 1.15, 1.33, 0.65, and 1.01, respectively (all ps > 0.2),
there was a significant difference in the percentage of emotional constructs (e.g.,
“Live, active/Dead“), F(2, 70) = 5.80, p < 0.01. Pair-wise comparisons showed that
usability professionals had a higher percentage of emotional constructs than developers
and users (both ps < 0.05).
Table 3. Frequency (N = 977 constructs) and percentage (N = 72 participants) of constructs
within each classification, averaged across participants from the same stakeholder group
Category Frequ-
ency
Usability
profs Developers Users Total
M
SD
M
SD
M
SD
M
SD
I
SO 9241-210 UX *
System-related 306 33 ±18 34 ±14 27 ±20 31 ±17
User-related** 204 30 ±19 19 ±15 16 ±12 22 ±17
Context-related** 427 33 ±18 44 ±20 52 ±19 43 ±20
Other 40 4 ±5 3 ±6 4 ±6 4 ±6
Objective vs. subjective UX *
Performance 414 38 ±14 47 ±21 44 ±13 43 ±1 7
Image/impression** 353 44 ±13 34 ±16 31 ±14 36 ±15
Other 210 18 ±10 20 ±15 25 ±14 21 ±13
System-oriented UX **
General relationship and user
experience with the system 77 11 ±8 7 ±9 6 ±8 8 ±9
Attractiveness of the system** 75 14 ±13 6 ±6 4 ±6 8 ±10
Ease of use of the system 178 22 ±11 17 ±16 16 ±13 18 ±14
Utility of the system** 405 30 ±16 41 ±23 54 ±20 42 ±22
Degree of usage of the system 46 3 ±4 7 ±7 4 ±4 5 ±5
Other 196 20 ±14 22 ±17 16 ±15 20 ±16
H
uman experience of technology*
Compositional 547 53 ±14 59 ±17 57 ±14 57 ±1 5
Sensual 95 12 ±8 10 ±9 8 ±7 10 ±8
Emotional** 94 15 ±11 7 ±6 7 ±9 10 ±9
Spatio-temporal 178 15 ±11 18 ±14 20 ±11 18 ±12
Other 63 5 ±6 5 ±8 7 ±9 6 ±8
*p < 0.05, ** p < 0.01, indicate significant differences between stakeholder groups.
4.2 Nationality Differences
We tested for interactions between nationality and stakeholder group using multivariate
ANOVAs, but found no overall interactions between nationality and stakeholder group
for any of the four classifications. Stakeholder group and nationality appeared to be
two independent factors, each influencing how participants think about UX. Regarding
nationality, Table 4 gives, for each nationality, the average percentage of constructs in a
category for a single participant.
Usability Professionals Think about User Experience in the Same Way 471
The international standard for UX lists three factors as those that influence
subjective UX (system-, user-, and context-related). For the classification of the
constructs into these three ISO 9241-210 UX categories there was a significant effect
of nationality, Wilks’ λ = 0.54, F(8, 132) = 5.88, p < 0.001. There were significant
differences across nationality in the percentage of system-related constructs, F(2, 70)
= 12.44, p < 0.001; user-related constructs, F(2, 70) = 3.63, p < 0.05; context-related
constructs, F(2, 70) = 3.30, p < 0.05; and other constructs, F(2, 70) = 5.68, p < 0.01.
Pair-wise comparisons showed that Chinese participants had a higher percentage of
system-related constructs than Indian and Danish participants (both ps < 0.001) and a
lower percentage of user-related (p < 0.05) and, tentatively, context-related (p = 0.05)
constructs than Indian participants. Chinese participants had a higher percentage of
other constructs than Indian participants (p < 0.01).
Table 4. Percentage of constructs in the categories of each classification, averaged across
participants with the same nationality, N = 72 participants
Category Chinese Danish Indian
M
SD
M
SD
M
SD
I
SO 9241-210 UX ***
System-related*** 44 ±16 26 ±12 25 ±17
User-related* 15 ±11 24 ±13 27 ±22
Context-related* 35 ±17 46 ±14 48 ±26
Other** 6±6 4 ±6 1 ±3
Objective vs. subjective UX
Performance 44 ±17 38 ±14 47 ±18
Image/impression 40 ±16 39 ±11 30 ±17
Other 16 ±8 24 ±12 23 ±18
System-oriented UX ***
General relationship and user experience 6 ±7 11 ±9 8 ±9
Attractiveness 7 ±6 8 ±9 10 ±13
Ease of use 22 ±12 13 ±10 20 ±16
Utility* 32 ±16 44 ±19 50 ±27
Degree of usage*** 5±6 7 ±5 2 ±4
Other*** 29 ±15 17 ±14 12 ±12
H
uman experience of technology*
Compositional*** 59 ±14 49 ±13 62 ±16
Sensual 12 ±10 11 ±6 7 ±7
Emotional 8±6 11 ±9 11 ±12
Spatio-temporal** 13 ±9 24 ±13 16 ±12
Other 9±9 5 ±7 4 ±7
*p < 0.05, ** p < 0.01, *** p < 0.001.
472 T. Clemmensen et al.
The classification of the constructs according to objective vs. subjective UX
showed no effect of nationality, Wilks’ λ = 0.85, F(6, 134) = 1.84, p > 0.05. Thus,
Chinese, Danish, and Indian participants did not display a different distribution of
their constructs. The percentages of performance and image/impression constructs
tended to be roughly similar.
The classification of the constructs according to system-oriented UX showed a
significant effect of nationality, Wilks’ λ = 0.55, F(12, 128) = 3.73, p < 0.001.
Significant effects were found for three categories: utility, F(2, 70) = 4.36, p < 0.05,
degree of usage, F(2, 70) = 6.88, p < 0.01, and other F(2, 70) = 9.71, p < 0.001. Pair-
wise comparisons showed that Indian participants had a higher percentage of utility
constructs than Chinese participants (p < 0.05), and that Danish participants had a
higher percentage of degree-of-usage constructs than Indian participants (p < 0.01).
Chinese participants had a higher percentage of other constructs than Danish (p <
0.05) and Indian (p < 0.001) participants. As much as 29% of the Chinese
participants’ constructs were not covered by the system-oriented UX classification.
For the classification of the constructs according to human experience of technology
there was a significant effect of nationality, Wilks’ λ = 0.72, F(10, 130) = 2.38, p < 0.05.
There were significant differences in the percentages of compositional constructs, F(2,
70) = 5.00, p < 0.01, and spatio-temporal constructs, F(2, 70) = 5.32, p < 0.01. Pair-wise
comparisons showed that Indian participants had a higher percentage of compositional
constructs than Danish participants (p < 0.01), and that Danish participants had a higher
percentage of spatio-temporal constructs than Chinese participants (p < 0.01).
As stated above, we did not find any overall interaction effects between
stakeholder group and nationality. Since this was an interesting and surprising result,
we decided to analyse whether the influence on UX of stakeholder group and
nationality were also independent for each category of the classifications. We found
only two interaction effects when looking at the individual categories of the UX
classifications. First, for the ISO 9241-210 classification, there was a significant
interaction effect between nationality and stakeholder group for system-related
constructs F(4, 68) = 2.95, p < 0.05. Chinese users and Chinese usability
professionals used more system-related constructs than the same stakeholder groups
from India and Denmark, while for developers there was no difference. Second, for
the objective vs. subjective UX classification, there was a significant interaction effect
between nationality and stakeholder group for other constructs F(4, 68) = 4.16, p <
0.01. Users from India were quite different from developers and usability
professionals in the degree to which they construed UX in terms categorized as
“other”. The objective vs. subjective UX classification was more exhaustive for all
Chinese and Danish stakeholder groups.
4.3 Interrelations of Constructs across Classifications
To analyse the interrelations between categories in different classifications, Table 5
shows the ISO 9241-210 classification crossed with the three other classifications.
Comparing the ISO 9241-210 classification with the Objective vs. subjective UX
classification, the differences across stakeholder groups were unevenly distributed
across the Objective vs. subjective categories. The usability professionals had overall
Usability Professionals Think about User Experience in the Same Way 473
most subjective image/impression constructs (139/105/109), and had in particular more
user related (55/26/32) subjective image/impression constructs, compared to the other
stakeholder groups. They had fewer context-related constructs in the performance
(35/73/58) and other (46/78/61) categories, compared to users and developers.
Comparing the ISO 9241-210 classification with the system-oriented UX
classification, the usability professionals had more user-related constructs, in particular
about attractiveness (36/13/17) and ease of use (32/20/26). The usability professionals
also had fewer context-related constructs, in particular about utility and degree of usage.
Comparing the ISO 9241-210 classification with the human experience of
technology classification, the usability professionals had more user-related emotional
constructs. Again, the usability professionals had fewer context-related constructs, in
particular compositional constructs, compared to the other stakeholder groups.
Table 5. Cross-tabulation of stakeholder groups’ ISO 9241-210 and the other UX classifications
Subjective vs.
Objective UX System-oriented UX
Human experience
of technology Total
Objective performance
Subjective image/impression
Other
General relationship
Attractiveness of system
Ease of use
Utility
Degree of usage
Other
Compositional
Sensual
Emotional
Spatio-temporal
Other
I
SO 9241-210 UX
Usability professional
System 48 55 3 16 2 24 29 0 35 76 24 0 3 3 106
User 31 55 4 14 36 32 5 0 3 35 9 42 2 2 90
Context 35 26 46 4 2 8 63 11 19 55 3 2 45 2 107
Other 5 3 5 0 0 4 1 0 8 2 1 0 0 10 13
Total 119 139 58 34 40 68 98 11 65 168 37 44 50 17 316
User
System 44 49 1 5 0 24 33 0 32 63 18 0 2 11 94
User 27 26 2 11 13 20 8 1 2 22 3 22 2 6 55
Context 73 27 78 3 1 5 138 11 20 106 1 3 61 7 178
Other 10 3 3 1 0 6 4 1 4 7 2 0 2 5 16
Total 154 105 84 20 14 55 183 13 58 198 24 25 67 29 343
Developer
System 57 49 0 10 4 21 24 1 46 74 22 1 6 3 106
User 23 32 4 8 17 26 5 1 2 25 6 21 4 3 59
Context 58 23 61 4 0 5 94 19 20 81 1 3 50 7 142
Other 3 5 3 1 0 3 1 1 5 1 5 0 1 4 11
Total 141 109 68 23 21 55 124 22 73 181 34 25 61 17 318
Total 414 353 210 77 75 178 405 46 196 547 95 94 178 63 977
474 T. Clemmensen et al.
5 Discussion
5.1 Usability Professionals Differ from Other Stakeholder Groups
We find three main differences between usability professionals and the other
stakeholder groups in their thinking about usability and UX: Usability professionals
focus more on user-related constructs and subjective UX than developers and users,
and they focus less on context-related constructs than users, as illustrated in Fig. 1
which shows selected information from Table 3. Below we discuss these differences.
Fig. 1. Usability professionals focus more on users and subjective UX, less on context
Usability professionals focus more on user-relatedness and subjective UX than
developers. This supports the main rationale behind the profession, which is that
usability professionals compared to developers have more empathy with users and
their situation [20]. The implication of this is that while developers might need to be
convinced to become more user centred, usability professionals are ready for more
advanced theory and techniques that go beyond simply being user centred.
Usability professionals focus more on user-relatedness and subjective UX than
users. This finding is somewhat against the idea of usability professionals having the
same view on system use as users. Usability professionals may focus more on
subjective UX, emotions, attractiveness and so forth than users, because they as
professionals have a greater range of constructs to describe and distinguish between
UX-related emotions. The distinction between objective performance and subjective
image/impression by Han [5] appears on the surface to be similar to a distinction
between utilitarian and experiential constructs. However, in this study usability
professionals attended a lot to subjective aspects of UX, whereas Hertzum and
Clemmensen [8] found that usability professionals made use of more utilitarian than
experiential constructs. A utilitarian/experiential distinction divides usability into a
utilitarian factor concerned with goal-related performance and an experiential factor
concerned with satisfaction [8]. Satisfaction is, partly, about freedom from
discomfort, that is, when there is no frustration there may be satisfaction [14]. In
contrast, in the objective/subjective classification, the subjective dimension was
inspired by Kansei-engineering and focused on the emotions actually experienced by
the user: “…collecting the subjective feelings of a product in words, extracting the
key feelings…” [5, pp. 478-479]. Furthermore, the subjective dimension covers both
Usability Professionals Think about User Experience in the Same Way 475
the immediate image/impression that the user gets from interacting with the product
in that moment, the user’s description of impressions of an interaction, and their
judgment and attitude toward this interaction [6]. Thus, a different set and range of
emotional processes involved in interacting with computers may be covered by the
subjective image/impression category, compared to an experiential dimension.
Usability professionals focus less on context-related constructs than users. We
speculate that the difference in focus on context-relatedness may be related to the
difference in focus on subjective UX. Users may be concerned with getting the work
done, a context-related concern, to a larger extent than usability professionals, who in
contrast focus more on the emotions experienced during use, a less context-related
concern. According to Hertzum et al. [9], users associate ease of use with leisure and
difficulty in use with work-relatedness, while developers express the reverse
associations. The usability professionals in the present study rarely distinguished
between work use and other use (e.g., “Working Tools/For Fun“). Only three of the
107 context-related construct/contrast pairs elicited by usability professionals
concerned a work-leisure distinction. A possible explanation for the near absence of a
distinction between work and leisure in the usability professionals’ constructs may be
that they consider UX-related emotions relevant to leisure as well as work. If a system
is sufficiently mature in that most performance and utilitarian issues have been taken
care of, the way to improve the system further may be to focus on the emotional
aspects of the user experience, irrespective of whether the system is for work or
leisure. This explanation may apply to our study because most of the systems selected
by the participants were mature products, such as MS Word and Outlook. In addition,
usability professionals may be more business-oriented than users in their thinking
about usability and UX. Though we did not find any direct evidence of this, usability
professionals may tend to think about how the user experience can give a product a
competitive edge on the market, and therefore value emotional over context-related
constructs.
5.2 Global Software Development and Inclusive UX Definitions
Stakeholder group and nationality independently influence how people think about
UX, as illustrated in Fig. 2. This means that usability professionals are different
from the other stakeholder groups, irrespective of nationality. The lack of
significant interaction effects between stakeholder group and nationality suggests
that the ways in which usability professionals differ from local user groups will be
somewhat similar across national borders. This result is promising for the global IT
industry because it indicates that usability professionals’ competences may be
globally applicable, thereby meeting a need pointed out by the global IT industry,
see [19]. However, since we found clear main effects of nationality – people from
different countries think differently about their system use – it seems to vary from
one cultural context and country to another what exactly usability professionals do
when they, paraphrasing Putnam and Kolko [17, p. 2021], “…walk in the [local]
users’ shoes”.
476 T. Clemmensen et al.
Fig. 2. Effects of stakeholder group and nationality on the ISO UX classification (%)
The present study confirms that UX is an inclusive concept. In the ISO definition,
UX is a broad concept that aims to capture most, if not all, of what people experience
when interacting with computer systems. ISO [10, p.3 ] defines the user experience as a
“person's perceptions and responses resulting from the use and/or anticipated use of a
product”. The four UX classifications were all good or very good at capturing the 977
constructs used by the 72 participants to describe their everyday system use. All four
UX classifications had better coverage (average 87%) compared to the coverage found
in an earlier study [8] for the ISO usability definition (average 53%). This study
provides empirical support that selected UX classifications, including the ISO definition
of UX, match the variation in how core stakeholder groups think about their system use.
5.3 Limitations
Three limitations should be remembered in interpreting the results of this study. First,
the stakeholder groups of developer, user, and usability professional evolve over time.
In particular, the usability profession may be evolving into a UX profession with a
broader or different focus. Hence, we characterize usability professionals’ thinking
about system use at a time when their profession is in a process of discussing its
identity. Second, the Danish participants were older than the Chinese and Indian
participants were, and the Indian participants were gender skewed toward males. The
Percent
Usability Professionals Think about User Experience in the Same Way 477
age difference may reflect that in Denmark people are relatively older when they
finish their education. However, we acknowledge that these issues related to the
sample and recruitment of participants could have affected our results. Third, with
respect to the developers and usability professionals there may be a gap between their
personal constructs and their professional knowledge. While we asked participants for
their personal constructs, their descriptions of their user experiences might be
influenced by explicit definitions of usability and UX, learned during their education
rather than experienced during the use of the selected systems.
6 Conclusion
We find differences in how usability professionals think about their user experiences,
compared to developers and users. Therefore, if the usability/UX profession in the
future wants to continue to claim that they are the users’ advocate, it may be
worthwhile to review the profession’s key constructs. To meet users’ concerns, it may
be time for HCI researchers to revisit the context of use and focus more on the
physical, social, temporal, organizational, and cultural environments in which a
product is used. In addition, people’s nationality influences all stakeholder groups’
thinking about system use, according to our study of a sample of Danish, Chinese, and
Indian people. Finally, the four UX classifications used in this study have emerged as
inclusive concepts that captured nearly all the ways in which our participants thought
about system use. This inclusiveness is encouraging from the point of view of
devising analytic UX conceptualizations that encompass most of the variability of
different stakeholders’ actual user experiences.
Future research may investigate how varying levels of professional experience
influence usability professionals’ thinking about UX. The usability professionals
participating in this study were intermediate-level to experienced usability
professionals. Their thinking about UX may gradually have moved away from that of
users and developers. Novice usability professionals may think more like users, or
they may more directly apply textbook definitions of usability and UX.
Acknowledgements. This study was co-funded by the Danish Council for Independent
Research. Thanks to Institute of Psychology, Chinese Academy of Science, Beijing,
for hosting the first author during data analysis in 2012. Thanks to Jyoti Kumar,
Qingxin Shi, Xianghong Sun, Pradeep Yammiyavar, and the interviewees for taking
part in this study.
References
1. Bødker, S.: Through the Interface: A Human Activity Approach to User Interface Design.
Lawrence Erlbaum Associates Inc., Hills dale (1990)
2. Dray, S.M.: Engaged Scholars, Thoughtful Practitioners: The Interdependence of
Academics and Practitioners in User-Centered Design and Usability. Journal of Usability
Studies 5(1), 1–7 (2009)
478 T. Clemmensen et al.
3. Dumas, J.: The Great Leap Forward: The Birth of the Usability Profession (1988-1993).
Journal of Usability Studies 2(2), 54–60 (2007)
4. Fleiss, J.: Statistical Methods for Rates and Proportions. Wiley, New York (1981)
5. Han, S.H., Hwan Yun, M., Kim, K.J., Kwahk, J.: Evaluation of Product Usability:
Development and Validation of Usability Dimensions and Design Elements Based on
Empirical Models. International Journal of Industrial Ergonomics 26(4), 477–488 (2000)
6. Han, S.H., Yun, M.H., Kwahk, J., Hong, S.W.: Usability of Consumer Electronic Products.
International Journal of Industrial Ergonomics 28(3), 143–151 (2001)
7. Hassenzahl, M., Wessler, R.: Capturing Design Space from a User Perspective: The
Repertory Grid Technique Revisited. In: International Journal of Human-Computer
Interaction, pp. 441–459 (2000)
8. Hertzum, M., Clemmensen, T.: How Do Usability Professionals Construe Usability?
International Journal of Human-Computer Studies 70(1), 26–42 (2012)
9. Hertzum, M., Clemmensen, T., Hornbæk, K., Kumar, J., Shi, Q., Yammiyavar, P.:
Personal Usability Constructs: How People Construe Usability across Nationalities and
Stakeholder Groups. International Journal of Human-Computer Interaction 27(8), 729–761
(2011)
10. ISO 9241-210: Ergonomics of Human System Interaction-Part 210: Human-Centred
Design for Interactive Systems (Formerly Known as 13407). International Organization for
Standardization (ISO). Geneva, Switzerland (2009)
11. Kelly, G.A.: The Psychology of Personal Constructs. Theory and Personality, vol. 1.
Norton, New York (1955)
12. Kujala, S., Roto, V., Väänänen-Vainio-Mattila, K., Karapanos, E., Sinnelä, A.: UX Curve:
A Method for Evaluating Long-Term User Experience. Interacting with Computers 23(5),
473–483 (2011)
13. Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human-Computer Interaction.
Wiley, Chichester (2010)
14. Lazar, J., Jones, A., Shneiderman, B.: Workplace User Frustration with Computers: An
Exploratory Investigation of the Causes and Severity. Behaviour & Information
Technology 25(3), 239–251 (2006)
15. Lindgaard, G.: Early Traces of Usability as a Science and as a Profession. Interacting with
Computers 21(5-6), 350–352 (2009)
16. McCarthy, J., Wright, P.: Technology as Experience. Interactions 11(5), 42–43 (2004)
17. Putnam, C., Kolko, B.: HCI Professions: Differences & Definitions. In: CHI 2012
Extended Abstracts, pp. 2021–2026. ACM, New York (2012)
18. Redish, G., Barnum, C.: Overlap, Influence, Intertwining: The Interplay of UX and
Technical Communication. Journal of Usability Studies 6(3), 90–101 (2011)
19. Rosenberg, D., Kumar, J.: Leading Global UX Teams. Interactions 18(6), 36–39 (2011)
20. Shackel, B.: Usability-Context, Framework, Definition, Design and Evaluation. Interacting
with Computers 21(5-6), 339–346 (2009)
21. Tan, F.B., Hunter, M.G.: The Repertory Grid Technique: A Method for the Study of
Cognition in Information Systems. MIS Quarterly 26(1), 39–57 (2002)
22. Walsh, T., Nurkka, P., Walsh, R.: Cultural Differences in Smartphone User Experience
Evaluation. In: MUM 2010, Limassol, Cyprus, (Article No. 24). ACM, New York (2010)
