ArticlePDF Available

User involvement: A review of the benefits and challenges

Authors:

Abstract and Figures

User involvement is a widely accepted principle in development of usable systems. However, it is a vague concept covering many approaches. This study first clarifies the nature of user involvement and its expected benefits, and secondly reviews three streams of research, to evaluate the benefits and problems of varied user involvement approaches in practice. The particular focus of this study is on the early activities in the development process. An analysis of the literature suggests that user involvement has generally positive effects, especially on user satisfaction, and some evidence exists to suggest that taking users as a primary information source is an effective means of requirements capture. However, the role of users must be carefully considered and more cost-efficient practices are needed for gathering users' implicit needs and requirements in real product development contexts.
Content may be subject to copyright.
User involvement: a review of the bene®ts and
challenges
SARI KUJALA
Helsinki University of Technology, Software Business and Engineering Institute, Metsa
Ènneidonkuja 10,
P.O. Box 9600, FIN-02015 HUT, Finland; e-mail: sari.kujala@hut.®
Abstract. User involvement is a widely accepted principle in
development of usable systems. However, it is a vague concept
covering many approaches. This study ®rst clari®es the nature
of user involvement and its expected bene®ts, and secondly
reviews three streams of research, to evaluate the bene®ts and
problems of varied user involvement approaches in practice.
The particular focus of this study is on the early activities in the
development process. An analysis of the literature suggests that
user involvement has generally positive eects, especially on
user satisfaction, and some evidence exists to suggest that
taking users as a primary information source is an eective
means of requirements capture. However, the role of users must
be carefully considered and more cost-ecient practices are
needed for gathering users' implicit needs and requirements in
real product development contexts.
1. Introduction
The goal of user-centred design is the development of
usable systems (Gould and Lewis 1985, Karat 1997).
One of the principles of user-centred design is the early
and continual focus on users, and it is generally agreed
that usability is achieved through the involvement of
potential users in system design (Karat 1997, Wilson et
al. 1997, Bekker and Long 2000).
As user needs and use contexts became increasingly
important in system development, ISO 13407 (1999)
recommends the active involvement of users for under-
standing user and task requirements. Karat (1997)
describes it in this way: `We don't consider usability as
limited to the display and keyboard interface between
human and machine, but rather we recognise that it
encompasses how any artefact ®ts into a complex work
or home environment'. Thus, it is apparent that
documents are insucient as sources of information
and direct contact with users is crucial in order to
understand the various contexts of use. Moreover, in
theory, user involvement is most ecient and in¯uential
in the early stages of system development as the cost
involved in making changes increases during system
development (cf. Ehrlich and Rohn 1994, Noyes et al.
1996).
On the other hand, a clear de®nition of user
involvement is lacking. It has been used synonymously
with `focus on users' (Wilson et al. 1997), `consulting
end ± users' (Noyes et al. 1996), `contacting with system
users' (Grudin 1991a), and `participation of users'
(Heinbokel et al. 1996). User involvement can be seen
to be a general term describing direct contact with users
and covering many approaches. For example, in
participatory design, users take active roles in many
design activities, but in other approaches users are
involved as providers of information, commentators or
objects for observations. The level of user involvement
can be broadly characterized as being somewhere on the
continuum from informative, through consultative to
participative (Damodaran 1996).
One of the diculties in involving users and under-
standing user requirements is that part of the users'
knowledge has become tacit through automation (Wood
1997). In well-learned tasks, much of the relevant
knowledge is no longer consciously available for the
person and everyday self-evidences are dicult to
articulate. Thus, the type and level of user involvement
need to be carefully considered. A promising approach
is to perform ®eld studies, whereby qualitative methods
are used to study users and their activities in their own
environment (cf. Bly 1997, Wixon 1995). Users do not
need explicitly to articulate their needs, but the under-
lying problems and possibilities are understood by
studying the future context of use.
Even if user involvement is generally approved,
research into its eciency tends to be varied and
BEHAVIOUR & INFORMATION TECHNOLOGY, 2003, VOL. 22, NO. 1, 1±16
Behaviour & Information Technology
ISSN 0144-929X print/ISSN 1362-3001 online #2003 Taylor & Francis Ltd
http://www.tandf.co.uk/journals
DOI: 10.1080/0144929021000055530
fragmented. For example, Clement and Van den
Besselaar (1993) found in their retrospective look at
participatory design projects that there were no
systematic surveys of these experiences and that the
projects did not adequately re¯ect the longitudinal
aspect of their course. Clement and Van den Besselaar
(1993) contacted the authors of these reports and asked
them to complete a short open-ended questionnaire in
order to bring the ®ndings up-to-date. They reviewed 10
projects. The results were somewhat contradictory and
showed that user participation does not necessarily lead
to users' more positive attitudes toward the technology
or market success.
On the other hand, the bene®ts of usability engineer-
ing have been clearly demonstrated (Bias and Mayhew
1994, Karat 1997). In general, for a given project the
cost-bene®t analysis identi®es the costs associated with
the usability work for the project and attempts to
quantify the potential sources of bene®t. The dierence
between the costs and the bene®ts is used to demonstrate
the value that usability engineering brought to a project
(Mayhew and Mantei 1994).
However, Lund (1997) argues that the importance of
matching design with user need should be emphasised.
Indeed, cost-bene®t analyses are generally based on
usability evaluation, but the cost-eectiveness of under-
standing user needs is dicult to evaluate. For example,
the case of Digital Equipment Corporation (Wixon and
Jones 1996) is an often-referred example of cost-bene®t
analysis that supports early user involvement. A baseline
new product designed with little involvement by human
factors professionals experienced disappointingly low
initial sales. The second version of the product was
developed with extensive involvement by usability
professionals. Wixon and Jones (1996) claimed that
their methods had a great eect on the product's
commercial success, and indeed sales exceeded predic-
tions by 30 to 60%. As a variety of usability techniques
were used in the Digital Equipment Corporation case
study, no speci®c grounds or mechanisms for the
improvement could be shown.
New approaches such as ®eld studies appear attrac-
tive, but the real bene®ts and costs of such approaches
should be considered. The paper of Curtis et al. (1999) is
one of the very few which thoroughly cover reporting
the costs of ®eld studies. They found that their project
made a signi®cant contribution to a large organization's
customer understanding, but the amount of data to be
analysed and represented was large. They spent as many
as 50 engineer months and $65 000 in gathering and
analysing data.
The purpose of this study is to improve the under-
standing of early user involvement and its worth in
practice. User involvement is a vague concept covering
many approaches. All these approaches may have varied
bene®ts and challenges. In this study, the main
approaches are identi®ed and their bene®ts and chal-
lenges are reviewed. Information about users and their
needs is needed in the early stages of system develop-
ment, but what kind of user involvement could uncover
this kind of informal and non-verbal information? Most
literature on user involvement concerns user involve-
ment throughout all the phases of the lifecycle, however
this article focuses on what is known about early
involvement and then what is known about user
involvement in general.
The ®rst section identi®es the dierent approaches to
user involvement and clari®es expected bene®ts. The
next section reviews and organizes the literature to
evaluate the bene®ts and problems of user involvement.
The literature is organized according to three research
streams: ®eld studies, qualitative, and quantitative
research. The article concludes with a summary and
evaluation of the expected bene®ts of user involvement
according to the research streams and dierent ap-
proaches.
1.1. Approaches to user involvement
It was stated earlier that user involvement has been
loosely translated as `direct contact with users' thus
covering many approaches. For example, Muller et al.
(1997) list 61 `participatory' practices including dierent
system development approaches like Joint Application
Design `JAD' (Carmel et al. 1993), Soft System
Methodology (SSM) (Checkland and Scholes 1990),
and ETHICS (Mumford 1993).
Bekker and Long (2000) reviewed the similarities and
dierences between ®ve of these `practices'. They
evaluated the dierences in user role, developer role,
user control, user involvement rationale, timing, and
process in order to enable designers to compare and
choose between various design approaches. However,
they did not compare the usefulness or eectiveness of
these approaches, possibly because this information is
not available. This makes it dicult to choose rationally
between the varied system development approaches. In
addition, the approaches do not always tell how to
implement user involvement in practice. Bekker and
Long (2000), for example, found that SSM and JAD are
less explicit about how to select and involve an
appropriate set of users.
So, what is the role of system development ap-
proaches in user involvement? Can we ®rst of all
understand the phenomenon of user involvement and
its eects? How should user involvement be implemen-
ted in practice? If we try to classify the main approaches
S. Kujala
2
to user involvement instead of particular development
approaches, we may suggest that the main approaches
are user-centred design, participatory design, ethnogra-
phy, and contextual design. For example, these
approaches are represented in Readings in Human-
Computer Interaction-book (Baecker et al. 1995), and
the latter three are considered as frameworks of ®eld
research by Wixon and Ramey (1996).
The roots and methods of the approaches are closely
linked and most of the approaches can be found in the
`Participatory Design' book edited by Schuler and
Namioka (1993). In addition, task analysis can involve
users (see Diaper 1989, Johnson 1989, Hackos and
Redish 1998). All these approaches include a rationale
explaining why to involve users and a methodology on
how to involve users. Table 1 demonstrates the
dierences for the user involvement approaches.
The goal of user-centred design is the development of
useful and usable products. There appears to be no
agreed de®nition or process for it (Karat 1997).
However, the principles that Gould and Lewis (1985)
present are generally accepted. The principles are:
1. Early focus on users and tasks.
2. Empirical measurement.
3. Iterative design.
The principles include the idea of user involvement:
Gould and Lewis (1985) recommend bringing the design
team into direct contact with potential users, as opposed
to hearing or reading about them through human
intermediaries. The second principle implies that, early
in the development process, intended users should use
simulations and prototypes to carry out real work, and
their performance and reactions should be observed,
recorded, and analysed.
Usability engineering tends to overlap with user-
centred design and the two are often used interchange-
ably (e.g. Mayhew and Mantei 1994). Wixon and
Wilson (1997) de®ne usability engineering as a process
for de®ning, measuring, and thereby improving the
usability of products. Methodological approaches to
usability engineering have been introduced by a number
of authors such as Mantei and Teorey (1988), Nielsen
(1993), and Mayhew (1999).
Participatory or co-operative design is an approach of
Scandinavian origin (Floyd et al. 1989, Ehn 1993).
Designers and workers have collaborated on under-
standing users and their tasks when planning and
designing new business practices and interfaces. Users
participate by analysing the organizational requirements
and by planning appropriate social and technical
structures to support both individual and organizational
needs. Democratic participation and skill enhancement
are important features of participatory design (Ehn
1993).
The early work in Scandinavia has been comple-
mented in other countries (Muller et al. 1991, Clement
and Van den Besselaar 1993, Schuler and Namioka
1993) and the approach has been applied in several
research projects of the in-house or contract develop-
ment type, and also in product development (Kyng
1994). Kuhn and Muller (1993) say that outside of
Scandinavia, the ®eld is more varied, with some
theorists and practitioners pursuing a locally adapted
form of democratic decision-making, and others
emphasizing eective knowledge acquisition and pro-
duct quality.
Ethnomethodological ethnography is a sociological
approach that is also used to inform design of systems.
It is most in¯uential within the research communities of
computer-supported co-operative work (CSCW), but
also increasingly in HCI research (Dourish and Button
1998). It has become a shorthand or simpli®cation to
speak of ethnography instead of ethnomethodology in
CSCW, while within sociology and anthropology
themselves ethnography denotes rather little (Shapiro
1994).
Ethnography describes human activities and culture
with a focus on the social aspects of human co-
operation. Blomberg et al. (1993) characterize it with
four principles:
User involvement: a review of the bene®ts and challenges 3
Table 1. User involvement approaches.
User-centred Participatory Contextual
design design Ethnography design
Emphasis Usability Democratic Social aspects of Context of work
participation work
Typical methods Task analysis,
Prototyping, Workshops, Observation, Contextual
Usability Prototyping Video-analysis inquiry
1
evaluations Prototyping
1
Contextual Inquiry is a ®eld interviewing method which combines observing and interviewing (Beyer and Holtzblatt 1998).
1. It takes place in natural settings.
2. It is based on the principle of holism, that is,
particular behaviours must be understood in the
respective context.
3. It develops descriptive understanding in contrast
to prescriptive.
4. It is grounded in a member's point-of-view. The
main methods are observation and video-analysis.
The earliest attempts at linking ethnographic studies
of work and design were with CSCW systems (e.g.
Hughes et al. 1992). In a design context the aim of
ethnography is to develop a thorough understanding of
current work practices as a basis for the design of
computer support (Blomberg et al. 1996, Simonsen and
Kensing 1997). Lewis et al. (1996) describe the process
of using ethnographic data for product development.
Kensing et al.'s (1998) MUST-method combines the use
of ethnographic techniques and intervention within the
participatory design tradition in the context of in-house/
custom development. Viller and Sommerville (1999a,b)
have also developed an ethnographically informed
method for requirements engineering and design pro-
cess.
Contextual design is focused on studying people in
their work (Holtzblatt and Beyer 1993, Beyer and
Holtzblatt 1996, 1998, 1999). Users, usually one-at-a-
time, are watched and talked with about their work while
working in their own environment. The idea is to study
the work processes and to describe and redesign them by
changing role structures, supporting tasks, automating
and eliminating unnecessary steps. The approach in-
cludes a general philosophy of visiting users. Beyer and
Holtzblatt (1999) themselves describe contextual design
as an approach to designing products.
Task analysis covers a wide range of methods in
order to analyse a system function in terms of user
goals and the sub-goals inherent in performing the task
(Johnson 1989, Kirwan and Ainsworth 1993, Hackos
and Redish 1998, Richardson et al. 1998, Annett and
Stanton 2000). Much of the task analysis literature is
devoted to the analysis of data, but task analysis also
involves the users as informants (Diaper 1989, Jeries
1997). In addition, task analysis may be used as a part
of larger design methodologies. The identi®ed goals,
task sequences and hierarchies can be used in design in
recognizing the familiar paths for users and the
problems they have.
1.2. Expected bene®ts of user involvement
To truly understand user involvement we should have
an understanding of the bene®ts such involvement
brings about. The expected bene®ts of user involvement
therefore serve as hypotheses to be tested.
According to Damodaran (1996) a variety of studies
show that eective involvement in system design yields
the following bene®ts:
1. Improved quality of the system arising from more
accurate user requirements.
2. Avoidance of costly system features that the user
did not want or cannot use.
3. Improved levels of acceptance of the system.
4. Greater understanding of the system by the user
resulting in more eective use.
5. Increased participation in decision-making within
the organization.
The list is somewhat participatory design focused, but
it aptly illustrates the underlying assumptions regarding
the bene®ts of user-centred design and usability en-
gineering. For example, Gould et al. (1987) report their
®ndings on the bene®ts of user centred design as: `Extra
eort in the early stages leads to much less eort later on
and a good system at the end'. They also recommended
that one must focus on users early on, in order to learn
the type of system required. Also Nielsen (1993) states:
`Users often raise questions that the development team
has not even dreamed of asking. This is especially true
with respect of potential mismatches between the users'
actual task and the developers' model of the task'.
The bene®ts 4 and 5 are mainly related to the features
of participatory design. In participatory design, democ-
racy is one of the main themes, the aim being that the
workforce should be active participants in all decisions
aecting their working lives.
2. Literature review
Research into user involvement is widely dispersed,
ranging from descriptive case studies to cross-sectional
surveys and covering many approaches, many types of
products, development contexts and ®rms. The starting
point for selection of the papers to review was the user
involvement approaches suggested earlier. All relevant
papers published in Interacting with Computers 1997 ±
2000, Human ± Computer Interaction 1995 ± 2000, Inter-
actions 1996 ± 2000, and Communications of the ACM
1997 ± 2000, among others, were considered for inclu-
sion. In addition, books such as Field Methods Case-
book for Software Design edited by Wixon and Ramey
(1996), Participatory Design edited by Schuler and
Namioka (1993), and Task Analysis for Human ±
Computer Interaction edited by Diaper (1989) were
included. The CSCW area was excluded from this
S. Kujala
4
review. Plowman et al. (1995) provide a review of this
area.
Three dierent research streams are reviewed. First,
we review literature to ®nd out what has been learnt
about user involvement in ®eld studies, which represent
descriptive case studies including direct user involve-
ment. The main goal of these case studies is not to
evaluate user involvement but to give actual examples of
®eld research (Wixon and Ramey 1996). Then we review
qualitative research work, which focuses more directly
on the helping and hindering factors of user involve-
ment. Finally, the quantitative research on the eects of
user involvement on system success is evaluated.
2.1. Field studies
The book `Field Methods Casebook for Software
Design' edited by Wixon and Ramey (1996) is full of
positive experiences of user involvement. The frame-
works of the cases were ethnography, participatory
design, and contextual design. Many of the case
descriptions present the bene®ts in the same vein as
Ramey et al. (1996) do `we feel con®dent that the
method has proved useful'. Wixon et al. (1996) describe
their ®ndings and present a number of positive customer
responses. Dray and Mrazek (1996) state that valuable
insights were gained. However, no objective measure-
ment of the bene®ts is presented and only a few authors
describe the costs incurred on the cases.
Muller and Carr (1996) state that they spent less than
100 h in ®eld studies and found major bene®ts to be in
understanding, redirection of eort, downstream tech-
nology understanding, and improved mutual under-
standing and work relationships among all the
stakeholders. Rowley (1996) estimates that two-and-a-
half weeks of travel and data gathering was spent on the
initial stage of their project. He mentions two main
bene®ts found: (i) customers usually viewed the visits as
a form of respect and appreciation; and (ii) the decisions
of software developers were more likely to match the
needs of the users. He also mentions that relatively little
time was spent in change control meetings discussing
design changes which originated from the study because
the credibility of the source reduced the controversy that
often contributes to the amount of time spent determin-
ing priorities. He also reported a number of ®eld-study
challenges: the amount of raw data collected during the
study can be overwhelming; impacting the design can be
dicult if ®eld-oriented methods are not an accepted
part of the development process; and gaining direct
access to customers can be dicult.
Brown (1996) does not mention how much time they
spent on ®eld studies. She states that they received data
that was invaluable in supporting decisions, but that the
time needed for the actual studies, the communication,
and management of large amounts of data remained
their biggest problems. She also found that users were
generally happy, although some users began to request
that changes be made to their system. Bauersfeld and
Halgren (1996) also found that they gathered useful data
and experienced considerable success in turning data
into design, but they suggested that a more ecient way
to compare subjective data across users should be
developed.
Juhl (1996) describes the experiences of using con-
textual inquiry in Microsoft product development. The
research project was completed in 60 days and the
project team felt that this cost was reasonable, given the
depth and breadth of information obtained about
customers' tasks, and the bene®ts being realized from
data reuse. The design teams explained that contextual
inquiry and follow-on users centred design activities
provided them with a long-term vision for product
development eorts and helped them in the under-
standing of the customers' needs. On the other hand, the
projects revealed that contextual inquiry studies are too
time-, labour-, and attention-intensive for them and it
was dicult to generate clear deliverables for upper
management.
Coble et al. (1996) spent approximately 1600 h in
contextual inquiry sessions and data analysis. They feel
the bene®t gained was the ability to gather accurate and
comprehensive information about their users' needs.
One of their users even commented that the users were
impressed with the comprehensiveness of the resulting
requirements.
Blomberg et al. (1996) combined ethnography and
participatory design in their law ®rm project. They
found that ongoing relations between developers and
strategically selected worksites can deepen developers'
understanding of the problems that workers face, but
that making work-oriented design an integral part of
system development will require resources to be
committed to alternative forms of design practice.
In summary, the ®eld studies into user involvement
were positive. The authors felt they gathered invaluable
data from users, the data helped them in understanding
of the customers' and users' needs and customer and
user responses were positive. Conversely, costs and
other hindering factors were mentioned such as:
.the overwhelming amount of raw data collected;
.the diculty in impacting design;
.the diculty gaining direct access to customers;
.the time spent on studies, communication and
management of large amounts of data;
.users requesting changes to their system.
User involvement: a review of the bene®ts and challenges 5
Karlsson (1996) also reported the positive aspects of
®eld studies. She completed four empirical studies of
four requirements gathering processes using dierent
methods like interviewing and observing. The results of
the empirical studies showed that dierent data collec-
tion methods contributed to the overall results, as well
as to the requirements formulated. In general, a more
complete picture was gained by using the ®eld studies.
Rockwell (1999) reports extremely positive experi-
ences of contextual design. He found that contextual
techniques resulted in a better-targeted product, higher
customer satisfaction, reduced development time, and
better team synergy and focus for delivery. The costs of
analysing the data were not reported.
Not everyone ®nds ®eld studies as successful. Butler
(1996) reported that `If you have talked to colleagues
who have tried these methods, you'll ®nd that although
this is important and informative work, it is also very
hard to do. It's dicult to ®nd users willing to let you
watch them work, it can be a tough sell to get developers
out of the oce and into the ®eld, and it's hard to make
sense of the data that you collect'.
Butler (1996) reported the problems he encountered
as follows:
.users were reluctant to let the researchers watch
them work;
.®nding users willing to let the researchers watch
them work took much longer than ®nding users for
typical usability sessions;
.users rarely did real work when the researchers
came to visits. It was dicult to arrange their day
so that they would have real work to do when
observed.
Their solution was to conduct `roundtables', where
anybody from the team who wanted to, could `sit
around' in pleasant conference rooms and chat with a
user who had brought in examples of the work they do
everyday.
To conclude, the experiences gathered seem to
con®rm some of the expected bene®ts of user involve-
ment. It was felt that more accurate user requirements
were gathered and user needs were better understood.
Positive customer and user responses were reported. But
many challenges to improve ®eld study techniques exist,
e.g. how to spend less time in using them, how to analyse
a large amount of data and how to compare subjective
data across users.
Plowman et al.'s (1995) review of workplace studies
and ethnographically oriented studies brought out
similar results within the area of CSCW. The main
outcome of these studies was in the dierent forms of
insight, which were usually reconceptualized at a more
abstract level. However, at this point, most of the
research ceased and few publications reported further
developments. Moreover, ®eldworkers were found to
have problems with communicating results to system
developers and with eecting design work. The authors
accordingly suggest that the majority of designers do not
have the time, inclination or expertise to consider ®eld
study ®ndings and that information may not always be
of practical use to system developers.
2.2. Qualitative research: helping and hindering user
involvement
Wilson et al. (1996, 1997) tried to directly assess the
relationship between the costs and bene®ts of user
involvement. The results are summarized in table 2. The
®rst study of Wilson et al. (1996) was a cross-sectional
survey of 25 practitioners. The intention of the survey
was to receive an initial indication of the costs and
bene®ts of user involvement. Indeed, 15 out of the 25
practitioners who responded reported some degree of
user involvement in the design project in which they had
been recently engaged. The respondents were asked to
list the strengths and weaknesses of each design activity
and the bene®ts and problems associated with the
involvement of users.
Wilson et al. (1996, 1997) continued this work with a
longitudinal study of one design project where the
designers were interviewed. Wilson et al. (1996) identi-
®ed the costs and bene®ts of user involvement, whereas
Wilson et al. (1997) concentrated on the obstacles and
facilitators of user involvement from both the user and
designer points of view.
All in all, the data of Wilson et al. (1996, 1997) tends
to be qualitative in nature and therefore the relative
signi®cance of the costs and bene®ts is impossible to
estimate. The ®ndings were similar to the other studies
described earlier. Users provided information, they were
generally satis®ed and accepted the design, but also
drawbacks and obstacles to the user involvement were
apparent. Hirschheim (1989) had obtained similar
results earlier.
Findings now being reported by Wilson et al. (1996,
1997) involved diculties in communicating between
users and developers. The authors concluded that
ideally, all stakeholders should be motivated and users
should be educated about the entire design process. The
rationale for these dierent problems with user involve-
ment seems to be that the user involvement in these
design projects was more participative in nature and no
special technique of user involvement was used.
Grudin (1991b) tries to understand the obstacles of
user involvement in the product development context.
S. Kujala
6
His conclusions rely on an earlier survey and interviews
with over 200 interface designers from several product
development companies; experiences in product devel-
opment; and conversations with fellow developers. The
bene®ts are not discussed, simply because Grudin
(1991b) believed that user involvement is necessary in
order to understand user requirements. He describes the
obstacles in development environments preventing user
User involvement: a review of the bene®ts and challenges 7
Table 2. The obstacles and bene®ts of user involvement.
Design
context Obstacles Bene®ts
Grudin (1991b):
a survey and
interviews of over
200 interface
designer,
conversations and
experiences
Product
development
in large
organizations
.Motivating the developers were
challenging.
.Identifying appropriate users was
dicult.
.Obtaining access to users and
motivating the users.
.Developers did not know how to
bene®t from user contact and how
to obtain feedback from existing users.
.Not enough time.
The bene®ts were out of the focus of the
paper. User involvement is believed to
be necessary in order to understand
user requirements.
Wilson et al. (1996):
a questionnaire of
25 practitioners
Not reported In the preparation phase:
.Users lacked information as to what
the designers needed to know.
.Users lacked information about what
design process meant.
In the design phase:
.Little consensus across users, the
problem was in ®nding compromises
between groups.
.Users introduced new concepts, and
there were generally too great a volume
of feedback.
In the evaluation phase:
Overall:
.Users became more exacting.
.There were often too many user
groups.
In the preparation phase:
.Users provided information and
feedback.
In the design phase:
.Users identi®ed interaction issues,
which had to be addressed by users
within the speci®c application
domains, provided ideas and oered
a practical view.
In the evaluation phase:
.Involvement from users, comments,
feedback, suggestions, commitment,
criticism, acceptance, improved
usability, learning by designers and
project leaders.
.The feedback brought the user
interface closer to task
Overall:
.Users were satis®ed.
.Users accepted the design.
Wilson et al. (1996):
a longitudinal study
of a one design
project, interviews
of designers
In-house .Users had to be educated about design.
.Users were unaware of
implementation constraints.
.Designers spent lots of time contacting
users and arranging meetings.
.Users provided useful information
and ideas.
.Users helped de®ne the scope of
the project.
.The system from the customers point
of view, was improved.
.Users were happy with the results.
.Users learnt about their job and
organization.
Wilson et al. (1997):
a longitudinal study
of a one design
project, interviews
of designers and
users
In-house .Limited time for the ®rst phase of the
design.
.Users were very busy.
.Some users lacked con®dence or
motivation and were reluctant to talk
to the designers.
.Some users did not understand the
task model used.
.Users were unaware of
implementation constraints.
.Users were eager to participate, because
they wanted to in¯uence on the out-
come. (The bene®ts were out of the
focus of the paper.)
involvement (table 2.). The problems are similar to those
mentioned by Wilson et al. (1996, 1997) concerning the
motivation of designers and users. In product develop-
ment, identifying appropriate users was also found to be
problematic, as the actual users of a product are not
identi®able until the product is actually bought.
Grudin (1991b) found that certain aspects of the
interface and user involvement are undervalued in
decision making in these organizations and that inter-
face quality is readily compromised. He suggests that
more positive conditions of direct user involvement in
product development can be achieved by altering the
structure of organizations and product development
processes. He recommends organizations to follow
Gould's (1988) principle of putting all the usability
aspects under a single management and the use of
methods such as user involvement in iterative design
with prototyping.
2.3. Quantitative research: the eect of user involvement
on system success
Part of the research work has focused on the direct
eects of user involvement on dierent aspects of system
success. Mantei and Teorey (1988) introduced the topic
of cost-bene®t analysis of usability engineering by
discussing the cost of incorporating a wide range of
usability engineering activities into the development
cycle. Bias and Mayhew (1994), Karat (1997) and Lund
(1997) provide a framework for cost-bene®t analysis and
several excellent examples of cost-bene®t analysis,
demonstrating that usability activities bring value to
corporations. The following are brief examples of the
estimated bene®ts.
2.3.1. Increased sales: Based on `buy decision' data
from usability tests and surveys, it is estimated that the
new usability-engineered system will have sales that are
25% higher in the ®rst year compared with `product
development as usual' (Karat 1994).
2.3.2. Increased user productivity: In one case, the
reduction in user time to complete the ®rst three tasks
from the initial to the ®nal version was 4.67 min after
three iterations of usability design and testing (Karat,
1997). The application had 22 876 end users, so the
working time saved was 1781 h. The evaluated cost-
bene®t ratio of task analysis, development of a low-
technology prototype, three iterations of usability
testing, and redesign was evaluated to be 1:2. In
another case, the reduction in time on task from ®rst
to ®nal user interface was 9.6 min on average after a
benchmark test, development of a high-technology
prototype, three iterations of usability prototype
testing, and redesign (Karat, 1997). The cost-bene®t
ratio of the usability work was evaluated to be 1:100.
2.3.3. Decreased training costs: Dray and Karat (1994)
estimate that a well-designed system could decrease
training costs by 35%. The project team conducted
iterative usability evaluations for prototypes and moved
their oces so that they were in constant contact with
users and the context in which they performed their
work.
2.3.4. Decreased user support: Microsoft announced
that the number of support calls dropped dramatically
as a result of usability testing and problem identi®ca-
tion, leading to a revised design (Reed 1992). The
average time per call fell to less than 10 min instead of
the earlier 45 min. Similarly, the Ford Motor Company
changed 90% of their accounting software for their
small car dealerships as a result of usability testing and
they were able to drop the help-line calls to zero (Kitsuse
1991). Earlier, it took the car dealers three help-line calls
merely to get started.
As Bias and Mayhew (1994) conclude, after the worth
of usability engineering is realized, it becomes a question
of how much resource to expend and how to apply that
resource. The cost-eectiveness of dierent types of
usability evaluation methods has been evaluated in
several research studies. Jeries et al. (1991) compared
four user interface evaluation techniques: heuristic
evaluation, software guidelines, cognitive walkthroughs,
and usability testing. They found that heuristic evalua-
tion by several UI specialists revealed the most serious
problems with the least amount of eort. However, it
also identi®ed a large number of low-priority problems
and required several highly skilled UI professionals.
Usability testing was the next most eective method; it
was particularly reliable in discovering relatively serious
and recurring problems.
On the other hand, Karat et al. (1992) compared
individual and team usability walkthrough methods,
including heuristic evaluation and usability testing. They
found that across two systems, empirical testing
identi®ed the greatest number of problems; it identi®ed
a signi®cant number of relative severe problems that
were missed by the walkthrough methods. The total
number of usability problem tokens revealed by
empirical testing was approximately four times the total
number of problems identi®ed by team walkthroughs,
and about ®ve times the total number found by
individual walkthroughs. Empirical testing required
the same or less time to identify each problem compared
to walkthroughs.
S. Kujala
8
Furthermore, Desurvire (1994) describes a set of
studies in which heuristic evaluations were compared to
empirical testing. In one such study, human factors
experts using heuristic evaluation, revealed only 29% of
the most serious problems identi®ed by empirical
testing. One explanation for the dierences of the results
compared with Jeries et al. (1991) is the dierent way
of evaluating problem severity and validity. A method is
required for discovering if predictions of usability
problems were realized. Jeries et al. (1991) used UI
specialists to evaluate the problem severity, but Desur-
vire (1994) assumes that usability testing simulates to
some extent the problems in reality.
In summary, usability inspection yields somewhat
dierent results than empirical testing, the most
dicult problems are usually revealed by testing with
real users, and these methods can be seen as
complementary (Karat et al. 1992, Desurvire 1994,
Karat 1994, Nielsen 1994). However, as eective as
usability inspection methods are, as Desurvire (1994)
points out, primary to a system's success is whether it
can facilitate a users' job, task, or life in some useful
way, and the usability evaluation methods do not
address these issues (see also Wixon et al. 1994). Kujala
and Ma
Èntyla
È(2000) oer an example of how even
large-scale usability testing does not necessarily lead to
the ideal, if users and context of use are misunderstood.
Furthermore, Lund (1997) argues that cost-bene®t
analyses should rather emphasize an important con-
tribution of usability engineering ± matching design
with need, identifying new product or business oppor-
tunities and thus supporting the revenue strategy of the
company. The next step could be to evaluate the cost-
eectiveness of the methods that take into considera-
tion user needs and context of use.
User involvement is also a recurrent theme in
management literature where the focus is on user
participation. Ives and Olson (1984) already identi®ed
over 30 empirical studies where user involvement was a
key variable. In general, the studies relate user involve-
ment and system quality, system usage, user attitudes,
and user information satisfaction.
However, the studies did not provide consistent
evidence of the bene®ts. It was argued that serious
theoretical, methodological, and measurement problems
were associated with the past research (Ives and Olson
1984). For example, the operational de®nitions of user
participation and user involvement may have been
imprecise. The participation measures rarely re¯ect
actual user in¯uence on the physical development of
the system and objective indicators of system success are
rarely employed. Some later studies maintain that the
results of user participation depend upon various
contextual factors (Saleem 1996).
Saleem (1996) suggests that the functional expertise of
users modi®es the relationship between user participa-
tion and system acceptance. User in¯uence on system
development becomes vital for system acceptance when
users are perceived to possess greater system-related
functional expertise than other members of the design
team do. This in¯uence becomes less critical when users
appear to possess less expertise than those in¯uencing
system design do. Similarly, Hunton and Beeler (1997)
found that low self-ecacy perceptions might inhibit the
user's desire to participate in development activities.
Hawk and Dos Santos (1991) found in their ®eld study
that user participation was more closely related to user
information satisfaction when the system was used for
decision support and not for transaction-processing and
when users were at higher levels in the organization.
The results of some recent studies have been
summarized in table 3. The results show that the user
participation has positive eects overall. Baroudi et al.
(1986) report that the correlation between user involve-
ment and system usage is 0.28 and between user
satisfaction and user involvement 0.18, in their survey
of 200 production managers. These correlations are
statistically signi®cant, but weak in eect. User involve-
ment is explaining 8% of the variance in the system
usage and 3% in user satisfaction. Barki and Hartwick
(1991) also found a positive, although insigni®cant
correlation between user participation and system usage
(r= 0.17). In their study, a signi®cant correlation
between user participation and personal relevance of a
system to its users was found (r= 0.36).
McKeen and Guimares (1997) found a positive and
signi®cant correlation (0.42) between user participation
and user satisfaction in the systems development of 151
projects. Thus user participation was explaining 18% of
the variance in user satisfaction. Data was collected
from the project leader in charge of the development
and from the primary end user(s) of each system.
Dysfunctional user participation was never found in
these 151 projects. User participation, even in low-need
situations, was positively related to user satisfaction.
Nevertheless, it was observed that users need to be far
more involved in cases of high task and/or system
complexity. If task and/or system complexity were low,
only the core set of user participation behaviours
improved user satisfaction and other behaviours like
project de®nition were not linked with satisfaction.
Foster and Franz (1999) found there to be a strong
signi®cant correlation between users' self-perceptions of
participation and system acceptance indicators of
functional features (r= 0.42 and 0.32). Users' self
perceptions of participation had a weak but statistical
signi®cant negative correlation with an acceptance
indicator of generic system attributes (r=70.05).
User involvement: a review of the bene®ts and challenges 9
Thus, users valued the functional features of a system
that are directly useful to them in performing their tasks,
rather than generic system attributes. Analysts' percep-
tions of user participation correlated strongly and
signi®cantly with all indicators of acceptance (r= 0.81,
0.75, 0.55 and 0.50).
Keil and Carmel (1995) showed that direct links to
users and customers related to projects that were
evaluated as successful by managers. However, the term
`link' was used broadly. It included many kinds of
activities from observational study to support line.
Direct links were those in which the customer and
developer deal directly with one another and not
through intermediaries or customer surrogates.
Similarly, Chatzoglou and Macaulay (1996) found
that in projects where users and documentation are used
as primary sources of information, the number of
iterations needed for the completion of the requirements
capture process is one or two. In contrast, in projects
where users are a secondary rather than primary
information source, the number of iterations increases
and three or more iterations are then needed. In
S. Kujala
10
Table 3. A summary of the eects of user involvement on system success.
Design context Negative eects Positive eects
Barki and Hartwick
(1991): a survey of
105 users
In-house .User participation had a positive,
although nonsigni®cant correlation
with system usage (r=0.17).
Participation correlated statistically
signi®cantly with personal relevance
of a system to its users (r=0.36).
Baroudi et al. (1986):
a survey of 200
production managers
Varied
information
systems
.User involvement in the development
of information systems enhanced
statistically signi®cantly system
usage (r=0.28) and the user's
satisfaction (r=0.18).
Foster and Franz (1999):
a questionnaire of
87 users and 107
analysts
Varied
information
systems
.Users' self perceptions of
participation had a moderate
signi®cant correlation to system
acceptance (r=0.42 and 0.32 for
acceptance indicators of functional
features).
.Analysts' perceptions of user
participation correlated strongly
and signi®cantly with all indicators
of acceptance (r=0.81, 0.75, 0.55
and 0.50).
Heinbokel et al. (1996):
a longitudinal ®eld
study of 29 projects
External and
in-house
User participation in software
development was associated
with project diculties:
.lower overall success (r=70.47);
.fewer innovations (r=70.40);
.lower degree of ¯exibility (r=70.44);
.lower team eectiveness (r=70.45).
Keil and Carmel (1995):
an interview of
development managers
of 17 companies
Product
development
and in-house
.More successful projects employed
more direct links to users and
customers.
McKeen and Guimares
(1997): interviews and
questionnaires to users
and developers from
151 projects
In-house
business
applications
.Positive and signi®cant relationship
between user participation and
user satisfaction was found (r=0.42).
User participation was never
dysfunctional in these 151 projects.
addition, Blackburn et al. (2000) found that more time
and eort invested in the early stages of a software
project yields faster cycle times and improved produc-
tivity.
The only study demonstrating purely negative eects
of user involvement on system success was the long-
itudinal ®eld study by Heinbokel et al. (1996), who
assessed quality factors of the development process and
the product in two measurement periods during the
development process, using interviews and question-
naires. The participants were 200 team leaders and
users' representatives from 29 application software
development projects in Germany and German speaking
Switzerland. The participants were asked to evaluate
among other things the amount of user participation,
the overall success of the project, the amount of
innovations made during development, ¯exibility of
the project (i.e. reaction of the project to unpredicted
events), and ful®lment of time and budget requirements.
Users' responses to the ®nal products were not studied.
In this study, in the projects where the level of user
participation was high, the system analysts and pro-
grammers, team leaders, and user representatives
evaluated the overall success of the project to be lower.
Similarly, in their opinion, these projects showed fewer
innovations and a lower degree of ¯exibility. The results
also suggest that high user participation and even user
orientation correlates negatively with the evaluated team
eectiveness and quality of team interaction. Heinbokel
et al. (1996) explain that user participation disturbs the
process of software development. The participation
projects had to deal with several problems related to
developer-user relations that were not present in projects
without user participation. For example, users proposed
new ideas and demanded changes in a later stage of
development.
The results of Heinbokel et al. (1996) appear at ®rst to
be contradictory. However, the work was the ®rst survey
to target team leaders, developers and the relationship
of project work to user participation. User participation
was de®ned on the basis of the number of user
representatives in a project. So the user involvement
was participative in nature, and the participation was
presumably informal. The overall success of the project
was measured by a single subjective scale. User
participation was negatively (r=70.31) but not statis-
tically signi®cantly correlated with the measure of on
time/in budget in any signi®cant manner. However,
those projects that were thought to be inecient were, in
fact, less often completed within the second measure-
ment time than were projects that were thought to be
ecient (r=70.40, n= 26, p50.05). One interpreta-
tion of the results is that the projects encountered
communication problems with user representatives and
this in turn had a negative eect on the perceptions of
project success and the ful®lment of time and budget
requirements.
3. Summary and Conclusions
In this study, we have discussed the nature of user
involvement ± its bene®ts and challenges. In particular,
the purpose has been to understand early user involve-
ment and its value in requirements gathering even before
a prototype of the system exists.
The three streams of research reviewed here seem to
have similar results. User involvement is clearly useful
and it has positive eects on both system success and
user satisfaction. The streams of research reveal some
evidence relating to the Damodaran's (1996) expected
bene®ts of user involvement. Table 4 shows the expected
bene®ts, which were supported, by the dierent research
streams.
The results of ®eld studies and qualitative research
suggest that developers experience that they get more
accurate user requirements by involving users. The
bene®ts of prototyping and iterative usability evaluation
are clearly demonstrated, but it is more dicult to prove
empirically the cost-eectiveness of user involvement in
User involvement: a review of the bene®ts and challenges 11
Table 4. Evidence oered by the three streams of research supporting the expected bene®ts of user involvement.
Research streams
Expected bene®ts Field studies
Qualitative
research
Quantitative
research
More accurate user requirements X X X
Avoiding costly system features that the user did
not want or cannot use
XXX
Improved levels of acceptance of the system X X X
Greater understanding of the system by the user
Increased participation in decision-making in the
organization
X
gathering user needs before a prototype exists. Only the
work of Chatzoglou and Macaulay (1996) demonstrates
it, albeit indirectly, by showing that users as the main
source of information decreased the number of itera-
tions needed.
The bene®t of avoiding costly system features is not
directly shown, but it can be evaluated that usability
engineering can reduce the time and cost of development
eorts through early identi®cation and resolution of
usability problems (Karat 1997). The improved levels of
acceptance of the system were found in qualitative and
quantitative research; the relationship between user
involvement and user satisfaction is particularly strongly
supported by all three research streams. The positive
eects of user involvement in system usage also have
some support (Baroudi et al. 1986, Barki and Hartwick
1991).
The bene®ts of a greater understanding of the system
or increased participation in decision-making were not
the focus of these studies. Only Wilson et al. (1997)
mentioned that users were eager to participate, because
they wanted to in¯uence on the outcome, and Cherry
and Macredie (1999) argued that the improvement of
work organization and industrial democracy were the
key bene®ts of participatory design in their case study.
Clement and Van den Besselaar (1993) found in their
review of 10 participatory design reports that the
relationship between user participation in systems
design and the pursuit of workplace democracy is a
complex one. The authors of the reports generally note
that the local participants increased their competence on
new technology and became more willing to take the
initiatives with it. Nevertheless, Clement and Van den
Besselaar (1993) found that participatory design is
characterized by isolated projects with little indication
that it leads to self-sustaining processes within work
settings.
The eects of user involvement seem to be positive
overall, but complicated. Figure 1 summarizes the
eects of early user involvement. The early user
involvement may be a positive value for users and
customers as such, as described in ®gure 1. However, the
main eects come through intermediate factors such as
better user requirements.
Early user involvement additionally aects the per-
formance of the product development team but in
somewhat contradictory ways. The results of Chatzo-
glou and Macaulay (1996), Keil and Carmel (1995), and
Poltrock and Grudin (1994) suggest that user involve-
ment can have positive eects on project work and the
number of iterations required. Poltrock and Grudin
(1994) found that designers viewed marketing as
ineective in obtaining the information needed in order
to de®ne their product requirements and they were
frequently frustrated by the diculty of deciding what
to do without the relevant information from users.
Good's (1992) case study also reveals some initial
evidence to the eect that an understanding of the
user's world can lead to more innovations.
However, implementation of user involvement in
projects can be demanding. In particular, Heinbokel et
al. (1996) and Wilson et al. (1996) show that the
participative approach to user involvement may have
negative eects on project work. Normally, it is reported
that usability engineering reduces the development time
(Karat 1994, Mayhew and Mantei 1994). For example,
the value of correcting usability problems early is
estimated by making the assumption that changes made
early cost only one-quarter of what the same changes
made late would cost (Mantei and Teorey 1988,
Mayhew and Mantei 1994). However, Heinbokel et al.
(1996) and Wilson et al. (1996) report that when users
are participating in the design project, problems arise
when users demand changes in a late stage of develop-
ment or designers must resolve con¯icts between user
groups. As Hawk and Dos Santos (1991) point out, user
involvement in the form of participation is a costly
process that requires time and eort on the part of users
as well as developers. Developers and users tend to have
diculties in communication and users have to be
educated in what design process actually means (Wilson
et al. 1996). The costs can be said to be reasonable given
that user involvement has positive eects on project
success and user acceptance, but there are challenges to
improve user involvement in practice.
My overall interpretation is that involving users is not
an easy task for designers. Early involvement of users
appears to be promising, on the condition that user
involvement methods are developed further and the
roles of users and designers are carefully considered.
Designers should take an active role in user involve-
ment. Users are experts in their own ®eld, but they do
not need to be experts on design. Field studies are a
particularly promising approach for understanding
users' implicit and non-verbal needs. Users are not just
asked about their needs, but the analysts try to
understand their behaviour and the future context of
use. Users may not be able to communicate their precise
S. Kujala
12
Figure 1. The eects of early user involvement.
requirements, but they are able to explain their goals
and how they approach their tasks. Using this kind of
information a designer can work out on behalf of the
users the solution they need. Contextual inquiry and
ethnographic methods seem promising, but challenges
exist in the use and analysis of the huge amount of raw
data collected.
However, we could ask if it is designers task to gather
user needs or should projects use some kind of expert to
gather information from users. At least, Keil and
Carmel (1995) argue that indirect links between devel-
opers and customers (including users) are less desirable
to use because of information ®ltering and distortion. It
is clear that developers may have diculties in under-
standing users and empathizing with them if they have
never seen them.
3.1. Agenda for future research
As remarked earlier, our understanding of how user
involvement aects product development is incomplete.
These shortcomings present opportunities for future
research. For example, even if usability is the principal
goal of user involvement in HCI, too little eort has
been dedicated to evaluating how and how much the
early involvement of users (e.g. ®eld studies) contributes
to the usability of the ®nal product. Thus, it is hard to
convince companies of the importance of ®eld studies
(see Kaasgaard 2000). This has led up to a situation
whereby many companies focus their usability eorts on
usability testing. Methods and handbooks do exist for
usability testing, but in this manner usability is treated
primarily as error detection and elimination (Lund
1997).
In addition, an opportunity exists in examining the
eects of early user involvement in system development.
Problems may arise, for example, when users get new
ideas and demanded changes during a late stage of
development. However, developers may also spend a
considerable time in arguing with each other. As Nielsen
(1993) describes it: `It is amazing how much time is
wasted on certain development projects by arguing over
what users might be like or what they may want to do.
Instead of discussing such issues in a vacuum, it is much
better (and actually less time-consuming) to get hard
facts from the users themselves'. Thus, user involvement
can also have positive eects on development, if it
provides facts on which decisions may be based and the
user involvement is implemented in a designer-con-
trolled way. This also helps minimize design errors and
the need to make expensive changes.
Perhaps simple user participation is not enough;
developers need techniques on how to understand users
and their needs. These techniques exist, but all of the
varied approaches attract both supporters and critics
and few objective comparisons of methods or ap-
proaches are available. Maiden and Rugg (1996) have
tried to evaluate the requirements ± acquisition methods,
including ethnography. They evaluated that require-
ments engineers need considerable training in the use of
ethnographic methods and that the methods may take a
considerable time to master. Even the supporters admit
these problems and they identify the principle problem
to be the presentation of the results of ethnography in a
form that is readily usable by designers (Hughes et al.
1995).
We also know that the participatory role of users may
lead to problems in development (Heinbokel et al. 1996)
and that contextual inquiry may lead to a vast amount
of raw data. The problem these approaches have in
common seems to be that there needs to be a closer
connection to system development work. As Millen
(2000) points out, the ever-increasing pace of new
product development requires more time ecient
methods.
Fortunately, the approaches develop constantly.
Hughes et al. (1995), for example, have developed more
focused `quick and dirty' ethnography, in which
®eldworkers undertook short focused studies to gain a
rapid understanding of the work setting or to check the
sanity of an already formulated system proposal. Millen
(2000) also introduced rapid ethnography, and Viller
and Sommerville (1999a,b) have developed an ethno-
graphically informed method for requirements engineer-
ing and design process. Wood (1997) has developed
more ecient ethnographic interviewing techniques. In
their book Hackos and Redish (1998) introduce
practical guide for user and task analysis combining
dierent approaches and techniques to task analysis.
Lafrenie
Áre (1996) has developed a collaborative low-cost
approach to task analysis. Currently, a research
challenge exists to evaluate these new approaches and
their eectiveness in real developmental contexts.
The approaches are beginning to resemble one
another. In the end, the question may not be what
approach and methods to select, but what we can learn
from these methods and approaches, and what methods
we should use may depend on the situation. Participa-
tory design is the bottom rung for the user involvement
philosophy and users' rights: it introduced the idea of
bringing end users into direct contact with designers.
Field methods, such as contextual design and ethno-
graphy, provide methods for communicating with users
and understanding users' implicit needs. Contextual
design proposes, for example, the good principles of
visits to users and user ± developer relations. Ethnogra-
phy oers information on how to study social aspects of
User involvement: a review of the bene®ts and challenges 13
work. Task analysis demonstrates the importance of
goals, tasks, and task sequences.
References
ANNETT, J. and STANTON, N. A. (eds) 2000, Task Analysis
(London: Taylor & Francis).
BAECKER, R. M., GRUDIN, J., BUXTON, W.S. and GREENBERG,S.
(eds), 1995, Readings in Human-Computer Interaction:
Toward the Year 2000 (CA: Morgan-Kauman Publishers).
BARKI, H. and HARTWICK, J. 1991, User participation and user
involvement in information system development. Proceed-
ings of the Twenty-Fourth Annual Hawaii International
Conference on System Sciences, 1991. Volume iv, pp. 487 ±
492.
BAROUDI, J. J., OLSON, M. H. and IVES, B. 1986, An empirical
study of the impact of user involvement on system usage and
information satisfaction. Communications of the ACM,
29(3), 232 ± 238.
BAUERSFELD, K. and HALGREN, S. 1996, ``You've got three
days!'' Case studies in ®eld techniques for the time-
challenged. In D. Wixon and J. Ramey (eds) Field Methods
Casebook for Software Design (New York: Wiley), pp. 177 ±
196.
BEKKER, M. and LONG, J. 2000, User involvement in the design
of human-computer interactions: Some similarities and
dierences between design approaches. In S. McDonald,
Y. Waern and G. Cockton (eds) People and Computers XIV
(Proceedings of HCI'2000) (London: Springer), pp. 135 ±
147.
BEYER, H. and HOLTZBLATT, K. 1996, Contextual techniques.
Interactions,3(6), 44 ± 50.
BEYER, H. and HOLTZBLATT, K. 1998, Contextual Design:
De®ning Customer-Centered Systems (San Francisco: Mor-
gan Kaufmann Publishers).
BEYER, H. and HOLTZBLATT, K. 1999, Contextual Design.
Interactions,6(1), 32 ± 42.
BIAS, R. G. and MAYHEW, D. J. (eds) 1994, Cost-Justifying
Usability (San Diego, CA: Academic Press).
BLACKBURN, J., SCUDDER,G.andVAN WASSENHOVE, L. N. 2000,
Concurrent Software Development. Communications of the
ACM,43(11), 200 ± 214.
BLOMBERG, J., GIACOMI, J., MOSHER, A. and SWENTON-HALL,P.
1993, Ethnographic ®eld methods and their relation to
design. In D. Schuler and A. Namioka (eds) Participatory
Design: Principles and Practices (Hillsdale, NJ: Lawrence
Erlbaum), pp. 123 ± 155.
BLOMBERG, J., SUCHMAN, L. and TRIGG, R. H. 1996, Re¯ections
on work-oriented design project. Human-Computer Interac-
tion,11, 237 ± 265.
BLY, S. 1997, Field work: Is it product work? Interaction,4(1),
25 ± 30.
BROWN, D. S. 1996, The challences of user-based design in a
medical equipment market. In D. Wixon and J. Ramey (eds)
Field Methods Casebook for Software Design (New York:
Wiley), pp. 157 ± 176.
BUTLER, M. B. 1996, Getting to know your users: Usability
roundtables at Lotus development. Interactions,3(1), 23 ±
30.
CARMEL, E., WHITAKER, R. D. and GEORGE, J. F. 1993, PD and
Joint Application Design: A transatlantic comparison.
Communications of the ACM,36(6), 40 ± 48.
CHATZOGLOU, P. D. and MACAULAY, L. A. 1996, Requirements
capture and analysis: A survey of current practice. Require-
ments Engineering,1(2), 75 ± 87.
CHECKLAND, P. and SCHOLES, J. 1990, Soft Systems Methodol-
ogy in Action (New York: Wiley).
CHERRY, C. and MACREDIE, R. D. 1999, The importance of
context in information system design: An assessment of
participatory design. Requirements Engineering,4, 103 ± 114.
CLEMENT, A. and VAN DEN BESSELAAR, P. 1993, A retrospective
look at PD projects. Communications of the ACM,36(6),
29 ± 37.
COBLE, J. M., MAFFIT, J. S., ORLAND, M. J. and KAHN,M.G.
1996, Using Contextual Inquiry to discover physicians' true
needs. In D. Wixon and J. Ramey (eds) Field Methods
Casebook for Software Design (New York: Wiley), pp. 229 ±
248.
CURTIS, P., HEISERMAN, T., JOBUSCH, D., NOTESS, M. and WEBB,
J. 1999, Customer-focused design data in a large, multi-site
organization. Conference on human factors in computing
systems (CHI), pp. 608 ± 615.
DAMODARAN, L. 1996, User involvement in the systems design
process ± a practical guide for users. Behaviour &Informa-
tion Technology,15(6), 363 ± 377.
DESURVIRE, H. W. 1994, Faster, cheaper!! Are usability
inspection methods as eective as empirical testing? In J.
Nielsen and R. Mack (eds) Usability Inspection Methods
(New York: John Wiley & Sons).
DIAPER, D. 1989, Task observation for Human-Computer
Interaction. In D. Diaper (ed) Task Analysis for Human-
Computer Interaction (New York: Wiley), pp. 210 ± 251.
DOURISH, P. and BUTTON, G. 1998, On ``technomethodology'':
Foundational relationships between ethnomethodology and
system design. Human-Computer Interaction,13, 395 ± 432.
DRAY, S. M. and KARAT, C. 1994, Human factors cost
justi®cation for an internal development project. In J.
Nielsen and R. Mack (eds) Usability Inspection Methods
(New York: John Wiley & Sons).
DRAY, S. M. and MRAZEK, D. 1996, A day in the life of a
family: An international ethnographic study. In D. Wixon
and J. Ramey (eds) Field Methods Casebook for Software
Design (New York: Wiley), pp. 145 ± 156.
EHN, P. 1993, Scandinavian design: On participation and skill.
In D. Schuler and A. Namioka (eds) Participatory Design:
Principles and Practices (Hillsdale, NJ: Lawrence Erlbaum),
pp. 41 ± 77.
EHRLICH, K. and ROHN, J. A. 1994, Cost justi®cation of
usability engineering: A vendor's perspective. In R. G. Bias
and D. J. Mayhew (eds) Cost-Justifying Usability (San
Diego, CA: Academic Press), pp. 73 ± 110.
FLOYD, C., MEHL, W., REISIN, F., SCHMIDT, G. and WOLF,G.
1989, Out of Scandinavia: Alternative approaches to soft-
ware development and system development. Human-Com-
puter Interaction,4(4), 253 ± 350.
FOSTER, S. T. and FRANZ, C. R. 1999, User involvement during
information systems development: a comparison of analyst
and user perceptions of system acceptance. Journal of
Engineering and Technology Management,16, 329 ± 348.
GOOD, M. 1992, Participatory design of a portable torque-
feedback device. Conference on human factors in computing
systems (CHI), pp. 439 ± 446.
GOULD, J. D. 1988, How to design usable systems. In M.
Helander, T. K. Landauer and P. Prabhu (eds) Handbook of
Human-Computer Interaction, 2nd edn. (Amsterdam: Else-
vier), pp. 757 ± 789.
S. Kujala
14
GOULD, J. D., BOIES, S. J., LEVY, S., RICHARDS, J. T. and
SCHOONARD, J. 1987, The 1984 olympic message system: A
test of behavioral principles of system design. Communica-
tions of the ACM,30(9), 758 ± 769.
GOULD, J. D. and LEWIS, C. 1985, Designing for usability: Key
principles and what designers think. Communications of the
ACM,28(3), 300 ± 311.
GRUDIN, J. 1991a, Interactive systems: Bridging the gaps
between developers and users. IEEE Computer,24(4), 59 ±
69.
GRUDIN, J. 1991b, Systematic sources of suboptimal interface
design in large product development organization. Human-
Computer Interaction,6(2), 147 ± 196.
HACKOS, J. T. and REDISH, J. C. 1998, User and Task Analysis
for Interface Design (New York: Wiley.)
HAWK, S. R. and DOS SANTOS, B. L. 1991, Successful system
development: The eect of situational factors on alternate
user roles. IEEE Transactions on Engineering Management,
38(4), 316 ± 327.
HEINBOKEL, T., SONNENTAG, S., FRESE, M., STOLTE, W. and
BRODBECK, F. C. 1996, Don't underestimate the problems of
user centredness in software development projects ± there
are many! Behaviour &Information Technology,15(4), 226 ±
236.
HIRSCHHEIM, R. 1989, User participation in practice: Experi-
ences with participative systems design. In K. Knight (ed)
Participation in Systems Development (Columbia, Maryland:
GP Publishing), pp. 194 ± 212.
HOLTZBLATT, K. and BEYER, H. 1993, Making customer-
centered design work for teams. Communications of the
ACM,36(10), 93 ± 103.
HUGHES, J., KING, V., RODDEN, T. and ANDERSEN, H. 1995, The
role of ethnography in interactive systems design. Interac-
tions,2(2), 56 ± 65.
HUGHES, J. A., RANDALL, D. and SHAPIRO, D. 1992, Faltering
from ethnography to design. Conference on Computer
Supported Cooperative Work, pp. 115 ± 122.
HUNTON, J. E. and BEELER, J. D. 1997, Eects of user
participation in systems development: A longitudinal ®eld
experiment. MIS Quarterly, December 1997, 359 ± 388.
ISO 13407, 1999, Human-centred design processes for inter-
active systems. ISO/TC159/SC4. International Standard.
IVES, B. and OLSON, M. 1984, User involvement and MIS
success: A review of research. Management Science,30(5),
586 ± 603.
JEFFRIES, R. 1997, The role of task analysis in the design of
software. In M. Helander, T. K. Landauer and P. Prabhu
(eds) Handbook of Human-Computer Interaction, 2nd edn.
(Amsterdam: Elsevier), pp. 347 ± 359.
JEFFRIES, R., MILLER, J., WHARTON, C. and UYEDA, K. 1991,
User interface evaluation in the real world: A comparison of
four techniques. Conference on human factors in computing
systems (CHI) (New York: ACM), pp. 119 ± 124.
JOHNSON, P. 1989, Supporting system design by analyzing
current task knowledge. In D. Diaper (ed) Task Analysis for
Human-Computer Interaction (New York: Wiley), pp. 161 ±
185.
JUHL, D. 1996, Using ®eld-oriented design techniques to
develop consumer software products. In D. Wixon and J.
Ramey (eds) Field Methods Casebook for Software Design
(New York: Wiley), pp. 215 ± 228.
KAASGAARD, K. 2000, Software Design &Usability. Making
usability research usable, A talk with Stephanie Rosenbaum
(Copenhagen: Copenhagen Business School Press),
pp. 153 ± 176.
KARAT, C. 1994, A comparison of user interface evaluation
methods. In J. Nielsen and R. Mack (eds) Usability
Inspection Methods (New York: John Wiley & Sons).
KARAT, C. 1997, Cost-justifying usability engineering in the
software life cycle. In M. Helander, T. K. Landauer and P.
Prabhu (eds) Handbook of Human-Computer Interaction,
2nd edn. (Amsterdam: Elsevier), pp. 653 ± 688.
KARAT, C., CAMPBELL, R. and FIEGEL, T. 1992, Comparison of
empirical testing and walkthrough methods in user interface
evaluation. Conference on human factors in computing
systems (CHI) (New York: ACM), pp. 397 ± 404.
KARAT, J. 1997, Evolving the scope of user-centered design.
Communications of the ACM,40(7), 33 ± 38.
KARLSSON, M. 1996, User Requirements Elicitation, A Frame-
work for the Study of the Relation between User and
Artefact. Thesis for the degree of Doctor of Philosophy
(Go
Èteborg: Chalmers University of Technology).
KEIL, M. and CARMEL, E. 1995, Customer-developer links in
software development. Communications of the ACM,38(5),
33 ± 44.
KENSING, F., SIMONSEN, J. and BéDKER, K. 1998, MUST: A
method for participatory design. Human-Computer Interac-
tion,13(2), 167 ± 198.
KIRWAN, B. and AINSWORTH, L. K. (eds) 1993, A guide to task
analysis (London: Taylor & Francis).
KITUSE, A. 1991, Why aren't computers. . . Across the Board,
28(October), 44 ± 48.
KUHN, S. and MULLER, M. J. 1993, Participatory design.
Communications of the ACM,36(6), 24 ± 18.
KUJALA, S. and MA
ÈNTYLA
È, M. 2000, How eective are user
studies? In S. McDonald, Y. Waern and G. Cockton (eds)
People and Computers XIV (Proceedings of HCI'2000)
(London: Springer), pp. 61 ± 71.
KYNG, M. 1994, Scandinavian design: Users in product
development. Conference on human factors in computing
systems (CHI), April 24 ± 28, 1994 (Boston, United States:
ACM), pp. 3 ± 9.
LAFRENIE
ÂRE, D. 1996, Cuta: A simple, practical, low-cost
approach to task analysis. Interactions,3(5), 35 ± 39.
LEWIS, S., MATEAS, M., PALMITER, S. and LYNCH, G. 1996,
Ethnographic data for product development: A collabora-
tive process. Interactions,3(6), 52 ± 69.
LUND, A. M. 1997, Another approach to justifying the cost of
usability. Interactions,4(3), 48 ± 56.
MAIDEN, N. A. M. and RUGG, G. 1996, ACRE: selecting
methods for requirements acquisition. Software Engineering
Journal,11(3), 183 ± 192.
MANTEI, M. M. and TEOREY, T. T. J. 1988, Cost/bene®t for
incorporating human factors in the software lifecycle.
Communications of the ACM,31(4), 428 ± 439.
MAYHEW, D. J. 1999, The Usability Engineering Lifecycle (San
Francisco, Morgan Kaugmann Publishers).
MAYHEW, D. J. and MANTEI, M. 1994, A basic framework for
cost-justifying usability engineering. In R. G. Bias and D. J.
Mayhew (eds) Cost-Justifying Usability (San Diego, CA:
Academic Press), pp. 73 ± 110.
User involvement: a review of the bene®ts and challenges 15
MCKEEN, J. D. and GUIMARAES, T. 1997, Successful strategies
for user participation in systems development. Journal of
Management Information Systems,14(2), 133 ± 150.
MILLEN, D. R. 2000, Rapid ethnography: Time deepening
strategies for HCI ®eld research. In Proceedings on Design-
ing interactive systems: processes practices, methods, and
techniques, pp. 280 ± 286.
MULLER, M. J., BLOMBERG, J. L., CARTER, K. A., DYKSTRA,E.
A., GREENBAUM, J. and HALSKOV MADSEN, K. 1991, Panel:
Participatory design in Britain and North America: Re-
sponses to the ``Scandinavian challenge''. Conference on
human factors in computing systems (CHI), pp. 389 ± 392.
MULLER, M. J. and CARR, R. 1996, Using the CARD and
PICTIVE participatory design methods for collaborative
analysis. In D. Wixon and J. Ramey (eds) Field Methods
Casebook for Software Design (New York: Wiley), pp. 17 ±
34.
MULLER, M. J., HALLEWELL HASLWANTER, J. and DAYTON,T.
1997, Participatory practices in the software lifecycle. In M.
Helander, T. K. Landauer and P. Prabhu (eds) Handbook of
Human-Computer Interaction, 2nd edn. (Amsterdam: Else-
vier), pp. 255 ± 297.
MUMFORD, E. 1993, The participation of users in systems
design: An account of the origin, evolution, and use of the
ETHICS method. In D. Schuler and A. Namioka (eds)
Participatory design: Principles and practices (Hillsdale, NJ,
Erlbaum), pp. 257 ± 270.
NIELSEN, J. 1993, Usability Engineering (London: Academic
Press).
NIELSEN, J. 1994, Heuristic evaluation. In J. Nielsen and R.
Mack (eds) Usability Inspection Methods (New York: John
Wiley & Sons).
NOYES, J. M., STARR, A. F. and FRANKISH, C. R. 1996, User
involvement in the early stages of the development of an
aircraft warning system. Behaviour &Information Technol-
ogy,15(2), 67 ± 75.
PLOWMAN, R., ROGERS, Y. and RAMAGE, M. 1995, What are
workplace studies for? The Fourth European Conference on
Computer-Supported Cooperative Work, 10 ± 14 September
1995, (Stockholm, Sweden: Kluwer), pp. 309 ± 324.
POLTROCK, S. E. and GRUDIN, J. 1994, Organizational obstacles
to interface design and development: Two participant ±
observer studies. ACM Transactions on Computer-Human
Interaction,1(1), 52 ± 80.
RAMEY, J., ROWBERG, A. H. and ROBINSON, C. 1996, Adaptation
of an ethnographic method for investigation of the task
domain in diagnostic radiology. In D. Wixon and J. Ramey
(eds) Field Methods Casebook for Software Design (New
York: Wiley), pp. 1 ± 16.
REED, S. 1992, Who de®nes usability? You do! PC/Computing,
(Dec), 220 ± 232.
RICHARDSON, J., ORMEROD, T. C. and SHEPHERD, A. 1998, The
role of task analysis in capturing requirements for interface
design. Interacting with Computers,9, 367 ± 384.
ROCKWELL, C. 1999, Customer connection creates a winning
product, building success with contextual techniques. Inter-
actions,6(1), 50 ± 57.
ROWLEY, D. E. 1996, Organizational considerations in ®eld-
oriented product development: Experiences of a cross-
functional team. In D. Wixon and Ramey, J. (eds) Field
Methods Casebook for Software Design (New York: Wiley).
SALEEM, N. 1996, An empirical test of the contingency
approach to user participation in information systems
development. Journal of Management Information Systems,
13(1), 145 ± 166.
SCHULER, D. and NAMIOKA, A. (eds) 1993, Participatory Design:
Principles and Practices (Hillsdale, NJ: Lawrence Erlbaum).
SHAPIRO, D. 1994, The limits of ethnography: combining social
sciences for CSCW. The Conference on Computer Supported
Cooperative Work (New York: ACM), pp. 417 ± 428.
SIMONSEN, J. and KENSING, F. 1997, Using ethnography in
contextual design. Communications of the ACM,40(7), 82 ±
88.
VILLER, S. and SOMMERVILLE, I. 1999a, Social analysis in the
requirements engineering process: from ethnography to
method. International Symposium on Requirements Engineer-
ing (Limerick: IEEE Computer Soc. Press), pp. 6 ± 13.
VILLER, S. and SOMMERVILLE, I. 1999b, Coherence: An
approach to representing ethnographic analyses in systems
design. Human-Computer Interaction,14(1&2), 9 ± 41.
WILSON, A., BEKKER, M., JOHNSON, H. and JOHNSON, P. 1996,
Costs and bene®ts of user involvement in design: Practi-
tioners' views. People and Computers XI, Proceedings of
HCI'96 (London: Springer Verlag), pp. 221 ± 240.
WILSON, A., BEKKER, M., JOHNSON, P. and JOHNSON, H. 1997,
Helping and hindering user involvement ± A tale of
everyday design. Conference on human factors in computing
systems (CHI) (Atlanta: ACM), pp. 178 ± 185.
WIXON, D. 1995, Qualitative research methods in design and
development. Interactions, 2(4), 19 ± 26.
WIXON, D. and JONES, S. 1996, Usability for fun and pro®t: A
case study of the design of DEC rally version 2. In M.
Rudisill, C. Lewis, P. B. Polson, T. D. McKay (eds) Human-
Computer Interface Design: Success Stories, Emerging
Methods, and Real-World Context (San Francisco: Morgan
Kaufman Publishers), pp. 3 ± 35.
WIXON, D., JONES, S., TSE, L. and CASADY, G. 1994, Inspection
and Design Reviews: Framework, history, and re¯ection. In
J. Nielsen and R. Mack (eds) Usability Inspection Methods
(New York: John Wiley & Sons).
WIXON,D.PIETRAS, C. M., HUNTWORK, P. K. and MUZZEY,D.
W. 1996, Changing the rules: A pragmatic approach to
product development. In D. Wixon and J. Ramey (eds) Field
Methods Casebook for Software Design (New York: Wiley),
pp. 57 ± 89.
WIXON, D. and RAMEY, J. (eds) 1996, Field Methods Casebook
for Software Design (New York: Wiley).
WIXON, D. and WILSON, C. 1997, The usability engineering
framework for product design and evaluation. In M.
Helander, T. K. Landauer and P. Prabhu (eds) Handbook
of Human-Computer Interaction, 2nd edn. (Amsterdam:
Elsevier), pp. 653 ± 688.
WOOD, L. E. 1997, Semi-structured interviewing for user-
centered design. Interactions,4(2), 48 ± 61.
S. Kujala
16
... The forms of involvement vary from collecting extensive data to asking individuals to assess early products [65]. The level of involvement can also be adjusted between informing (as in interviews), advising (as in reviewing concepts), and doing (as in having stakeholders co-perform research or design work) [71]. However, more active or laborious stakeholder involvement risks asking individuals to do more than what is realistic, reasonable, or affordable [21,72]. ...
... This is often the case when individuals are asked to be co-designers without adequate training in design, compensation for their contribution, or understanding of the problem space. Although some involvement is essential to UCD, more is not always better [71]. ...
... Having designers interact directly with stakeholders, and especially end users, has been shown to yield better results [78] than hearing about the stakeholders and end users from another source [71]. Continual interaction with stakeholders during the UCD process helps designers gain a firsthand experience of the domain [78,79]. ...
Article
Full-text available
Background User-centered design (UCD) is a powerful framework for creating useful, easy-to-use, and satisfying mobile health (mHealth) apps. However, the literature seldom reports the practical challenges of implementing UCD, particularly in the field of mHealth. Objective This study aims to characterize the practical challenges encountered and propose strategies when implementing UCD for mHealth. Methods Our multidisciplinary team implemented a UCD process to design and evaluate a mobile app for older adults with heart failure. During and after this process, we documented the challenges the team encountered and the strategies they used or considered using to address those challenges. Results We identified 12 challenges, 3 about UCD as a whole and 9 across the UCD stages of formative research, design, and evaluation. Challenges included the timing of stakeholder involvement, overcoming designers’ assumptions, adapting methods to end users, and managing heterogeneity among stakeholders. To address these challenges, practical recommendations are provided to UCD researchers and practitioners. Conclusions UCD is a gold standard approach that is increasingly adopted for mHealth projects. Although UCD methods are well-described and easily accessible, practical challenges and strategies for implementing them are underreported. To improve the implementation of UCD for mHealth, we must tell and learn from these traditionally untold stories.
... On Human Resources, Kujala (2002) It was also observed that any quality process encompasses organisation employees through permanent audits of the process. However, it does not mean that, in their recruitment, a specific knowledge of this matter has been met. ...
... In their study none of the certified companies in quality uses this form of recruitment, unless it is selecting someone for the specific area of quality clarifying that. Kujala (2002) and Hofstede (2002) also argue that cultural aspects are really crucial in organizations" management. From those findings, culture also becomes a resource that can enhance knowledge or capacity, and may result in higher competence able to foster a better performance. ...
... found out that due to ISO quality implementation; employees have more training and more knowledge. Culture in an organisation (Schein, 1999;Kujala, 2002) becomes a resource either as knowledge based or dynamic capability perspective (Teece, 1994; Dirickx & Cool, 1989) resulting in higher competence and better performance. As to a ...
Article
Full-text available
The President of Kenya in the year 2010 directed that all public institutions be ISO Certified by 2012. Since then, ISO 9001:2008 has become part of the performance management framework by the Government of Kenya to enhance realization of quality service provision within the public sector. However, the effect of this initiative presents a grey area considering that the performance of the Kenya public service according to studies and successive customer satisfaction surveys remains significantly low. This is despite most of the Government Ministries and Departments being ISO 9001:2008 certified and having adopted the quality processes. The government is a country"s single biggest buyer of goods and services and this is done through the Supply Chain Departments. The Government Ministry Headquarters, being a central component in public service delivery through their Supply Chain Departments presented an opportunity for the researcher to interrogate the level of significance of this ISO 9001:2008 Management System in enhancing performance in these organizations. The paper focuses on one of the key Supply Chain Management activities within the ISO 9001 Quality Management System process of Resource Management Processes and how it influences performance of Supply Chain Departments at the Ministries" Headquarters in Nairobi. The study was conducted through a randomized quasi experimental design to analyze the pre and post certification conditions in which the influence of the independent variable was measured against the possible performance of the SCM departments in GOK Ministries. Using purposive and systematic random sampling respectively, respondents were sampled from the staff and management of these Ministry Headquarters. Data was collected through questionnaires and analysis of Ministry SCM Department records. The SPSS software was used to analyze the data. Inferential data analysis was done using correlation analysis. Regression models were constructed using multiple regression analysis and hypothesis testing done using ANOVA. The findings were presented in tables, graphs and literal descriptions. The findings indicate that resource management processes had a positively significant effect on performance of SCM departments. The study therefore recommends to the Principal Secretaries of the various Ministries to emphasis on resources management processes that enhance productivity and other best practices in human resource management and development.
... -Difficulty to translate certain types of data into design. (Kujala, 2003;Nielsen, 2008) Value Sensitive Design -Create concepts around user perceptions. ...
Thesis
Full-text available
Learning Analytics (LA) is a new promising field that is attracting the attention of education providers and a range of stakeholders including teachers, learning designers, academic directors and data scientists. Researchers and practitioners are interested in learning analytics as it can provide insights from student data about learning processes, learners who may need more help, and learners’ behaviours and strategies. However, problems such as low educator satisfaction, steep learning curves, misalignment between the analytics and pedagogical approaches, lack of engagement with learning technologies and other barriers to learning analytics development have already been reported. From a human-centred design perspective, these problems can be explained due to the lack of stakeholders’ involvement in the design of the LA tools. In particular, learners and teachers are commonly not considered as active agents of the LA design process. Including teachers, learners, developers and other stakeholders as collaborators in the 𝘤𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 of LA innovations can bring promising benefits in democratising the LA design process, aligning analytics and pedagogy, and meeting stakeholders’ expectations. Yet, working in collaboration with stakeholders to design LA innovations opens a series of questions that are addressed in this thesis in order to contribute to closing the gap for effective co-design of LA innovations. The questions addressed in this thesis are the following: 1. How can co-design techniques assist in the integration of diverse stakeholders in the LA design process? 2. What are the roles of the co-design practitioner/researcher in the LA design process? 3. What are the challenges in engaging stakeholders in the LA design process? Based on co-design principles, and following a Design-Based Research process, this thesis explores the critical challenge of engaging educators and students, the non-technical stakeholders who are often neglected, but who should ultimately be the main beneficiaries of LA innovations. In this research work, three case studies have been used to test, analyse and verify various co-design techniques in diverse learning contexts across a university to generate a co-design toolkit and recommendations for other co-design practitioners: i) learners and educators engaged in simulation-based healthcare scenarios, ii) learners, educators and other stakeholders in a Data Science Masters program , and iii) educators interested in providing personalised feedback at scale. This thesis presents three contributions to knowledge for effectively collaborating with educational stakeholders in the LA co-design process: 1. Inspired by archetypal challenges reported in classic and contemporary co-design literature, and in current LA research, the thesis identifies, exemplifies and reflects on five key challenges for LA co-design: power relationships, surveillance, learning design dependencies, asymmetric teaching/learning expertise, and data literacy. 2. By adopting and adapting well established co-design techniques, across the three case studies, the thesis provides empirical evidence of how these techniques can be used in LA co-design, reflecting on their affordances, and providing guidance on their usage. These detailed findings are distilled into a 𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘈𝘯𝘢𝘭𝘺𝘵𝘪𝘤𝘴 𝘊𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 𝘗𝘭𝘢𝘺𝘣𝘰𝘰𝘬, published under an open license to assist adoption and improvements. 3. Recognising the importance of the co-design practitioner in ensuring that the design process is participatory, the thesis documents and discusses the key functions and skills that this position requires. The role is further complicated when the practitioner is not only a 𝘧𝘢𝘤𝘪𝘭𝘪𝘵𝘢𝘵𝘰𝘳 serving a project, but also a 𝘳𝘦𝘴𝘦𝘢𝘳𝘤𝘩𝘦𝘳 of co-design. This motivates guidelines on the role of the co-design practitioner/researcher when working with stakeholders, and simultaneously studying the LA co-design process, tools and methods.
... Several different types of benefits have been found when involving users and experts in a co-design process. For example, a better fit between the system and users' needs, unique benefits and better value for users, improvement of mutual learning and understanding, combination and integration of different people's ideas, enhancement of communication and collaboration between people (Alam, 2002;Muller, 2002;Kujala, 2003;Steen, Manschot and De Koning, 2011). It is noteworthy that the seniors are experts of their experiences, such as social circumstances, habits and attitudes to values and preferences (Cottam and Leadbeater, 2004). ...
Conference Paper
Full-text available
This work-in-progress study reviews co-designing processes through the lens of possibility-driven design (PDD). A knowledge management model (KMM) is co-designed by facilitating the development work of senior and regional innovation actors who share ideas, experience and information in the development of smart products and services for an age-friendly smart living environment. The empirical part is divided into three stages: an orientation workshop, two panel meetings and three co-design and validation workshops where an appropriate knowledge management model is co-designed through iteration rounds. The first stage maps the regional innovation actors, relevant organisations in the region and data flows between all the parties. Ideas of suitable ways to manage knowledge are gathered from the panel meetings of the second stage and are methodologically supported by the strategic options development and analysis (SODA) approach. At the time of writing this paper, the third stage consisting of three workshops with appropriate iteration rounds is on-going. The findings of the study provide insights regarding the use of PDD activities with an inclusion of the SODA approach when facilitating the co-design of a KMM with a multi-professional group of experts. The study contributes to the theory of PDD by integrating systematic methodological aspects to it when working on complex problems.
... The effectiveness of the involvement of user research in the design process has been called into question in many studies (see e.g. Kujala, 2003;Mao, Vredenburg, Smith, & Carey, 2005 for detailed discussions; Vredenburg, Mao, Smith, & Carey, 2002), although it is commonly accepted that the use of user research in the design process has particular benefits for organizations. It is certain that the methods and tools used in obtaining this information have major impacts on the effectiveness of user research, and there is a broad range of literature on how this may be conducted to assure success. ...
Thesis
Full-text available
It is a commonly held belief that the integration of user research data into the design process can bring great benefits; and there have been many studies that not only examine these benefits, but have also suggested how these researches may be carried out. However, effective integration relies as much on the way information gathered from user researches is delivered to the designer as the quality of the information gathered. Examples of how user research findings are communicated can be found in literature; but what is lacking is a structured approach to developing deliverables with a framing of discussions about effectiveness, considering the practitioner’s needs and expectations. This study aims to investigate how user research findings should be communicated to the designers in order to maintain effectiveness in integration of the findings to the design process. A model and strategies and guidelines to achieve effective communication are proposed as the result. In order to propose them the methodology involves three main stages, including a literature search, an in-depth interview with the practicing designers and a verification questionnaire to confirm the findings of the previous two stages. The results of the study reveal expected outcomes of the user research activity by designers as the dimensions of effective communication of user research findings. Moreover qualities of the delivery mediums and informational content of the deliverables are identified from practitioners’ perspectives. The outcome of the study is a set of strategies and guidelines that the researches should consider, while designing new deliverables and planning communication activities for delivering user research findings to the design process.
Chapter
For small and medium-sized companies the fit between their business processes and their Enterprise Resource Planning (ERP) system is a critical success factor. The functions and features for essential tasks must be geared to the demands and skills of the individual users. This paper reports on the usefulness of several methods for eliciting user input which served as a basis for requirements for a personalized ERP system. It describes the yield of heuristic evaluations, both by experts and by developers, and a focus group with six users representing the main user types. The focus group consisted of an identification of the most important functions, task demonstrations, and a mini design workshop. As a demonstration of the results of the various user-focused methods, some noteworthy findings on the personalization of ERP systems are presented.
Chapter
Applications for interactive TV (iTV) addressing social aspects will only be successful, if the intended user and user community is taken into account during the development process. Existing methods for evaluating usability and user experience aspects of social interactive TV applications are not really enough to address and focus on the broad range of factors related to social user experience. This chapter presents various forms of user-centered methods enhancing usability and user experience aspects of interactive TV applications in general. Giving an overview on currently used user-centered methods to enhance and evaluate usability and user experience, some selected methods are presented for early and late development stages, reflecting the peculiarities stemming from the non-traditional environment the applications are used in—the home. The chapter will show how the methods can be adopted to focus more precisely on social aspects, especially social user experience. It enables the reader to get an overview on currently used user-centered methods in interactive TV and to learn about benefits and shortcomings of these methods as well as how to choose appropriate methods for their own purposes.
Article
Background The number of public eHealth services that support patient self-management is rapidly increasing. However, the implementation of these eHealth services for self-management has encountered challenges. Objective The purpose of this paper was to analyze the challenges and opportunities of implementing eHealth services for self-management by focusing on the fit between the technical solution and clinical use. Methods We performed in-depth interviews with 10 clinical project coordinators and managers who were responsible for developing and implementing various eHealth services for self-management interventions in five university hospitals in Finland. The results were analyzed using content analysis and open coding. The Fit between Individuals, Task, and Technology (FITT) framework was used to interpret the findings. Results The implementation of self-management services involved many challenges related to technical problems, health professional acceptance, patient motivation, and health organization and management. The implementers identified practices to manage the identified challenges, including improving the design of the technology, supporting health professionals in the adoption of the eHealth services, changing the work processes and tasks, involving patients, and collectively planning the implementation inside an organization. The findings could be mostly attributed to the dimensions of the FITT framework. Conclusions The FITT framework helped to analyze the challenges related to the implementation, and most of them were related to poor fit. The importance of patients as stakeholders in eHealth services for patient self-management needs to be highlighted. Thus, we propose that patients should be added as a different type of individual dimension to the FITT framework. In addition, the framework could be extended to include organization and management in a new context dimension.
Article
Aims and objectives The aim of this study is to learn more about factors that promote or inhibit user involvement among health professionals when implementing welfare technology in home care services. Background It is a health policy goal to increase the use of welfare technology in order to address some of the challenges that health care services are facing. Health professionals’ involvement is important for the successful implementation of welfare technology in home care services. Design The study has an explorative and descriptive longitudinal design based on a qualitative approach. Five focus group interviews were conducted with 16 nurses and assistant nurses from three different municipalities over a period of two years. The data were analysed using reflexive thematic analysis. The COREQ checklist was used. Results The analysis led to five main themes: competence a critical component, information and information lines, new ways of working, choice of welfare technology, and change in patient services. From health professionals’ perspective, there appeared to be a lack of preparedness for the change the implementation of welfare technology entailed for home care services. The respondents experienced limited facilitation and opportunities for user involvement. Conclusion Health professionals want to be more involved but emphasised that competence, information and collaborative arenas are necessary factors if involvement in the process is to be increased. Competence affected some of the respondents’ attitudes and willingness to use the technology. The respondents also experienced that the managements’ focus on facilitation and interest in user involvement in addition to infrastructures that functioned in various ways, had an impact on user involvement. Relevance to clinical practice The knowledge gained from this study about factors that promote or inhibit user involvement among health professionals could contribute to better preparedness for further implementation of welfare technology in the field of home care.
Article
Background Co-design with multiple tools is useful when end users’ knowledge is important, especially when designers work with people unfamiliar with design. Many studies have highlighted the importance of nurses’ participation in design, and such participation requires the development of techniques and tools to facilitate collaboration. This article analyzes how nurses participated in designing a general intensive care unit in a walk-in virtual environment (VE) and examines how their work-related knowledge can be transferred to the design process of spaces. Method In this action research study, the design process was conducted by using virtual mock-ups, which were evaluated by multi-occupational groups in a walk-in VE. Nurses were the largest occupational group. Their work processes were under modification, since existing multi-patient rooms were being redesigned as single-patient rooms. The design of single-patient rooms was performed in three iterative cycles in the walk-in VE. Results The nurses could specify their requirements in the walk-in VE, and their suggestions were incorporated into the architectural design process. The nurses were satisfied with their role in the design process. Conclusion Co-design with virtual mock-ups in walk-in VE is appropriate when designing new healthcare facilities and when the opinions of workers are important. Virtual mock-ups in walk-in VE can be used collaboratively, facilitating simultaneous feedback from multiple users. Virtual reality (VR) technology has evolved, and changes can be made rapidly and at a lower cost. Another advantage of VR is that it allows one to design larger spaces, thus providing larger layouts of facilities for evaluation.
Article
Increasingly users ® nd themselvesinvolved' in IT design projects. This occurs because the organizational culture of the parent organization purports to promote participation, or because structured design methods are being used which require users to play a part. In either case users who ® nd themselves required to participate in IT projects are frequently unclear about what this requires. In most organizations surprisingly little brie® ng on the users' role in design projects is provided. Users are therefore confused about their brief and concerned about their lack of expertise in computing. Although research reports on participatory design (PD) projects abound, little coherent guidance for the key stakeholders representing users' interests is available. The contents of this paper go some way towards ® lling the gap. Clear differentiation is made in the paper between the roles of the different players involved. Detailed guidance is provided for meeting the varied requirements of the different roles. For example, the roles oftop' management andmiddle' management in supporting user involvement are explored, their special responsibilities speci® ed and required actions listed. The need for an infrastructure to support user involvement and how to create one is discussed. Guidance is provided on, for example, the representation process and the factors to consider in selecting user representatives. The role of user representatives is particularly problematical and therefore receives particularly close attention. Finally guidance is given regarding the common pitfalls in Quality Assurance procedures and especially how to avoid the procedures becoming a meaninglessrubber-stamping' exercise. The guidance presented is grounded in the extensive experience of the author in participative design processes in a wide variety of contexts including the footwear industry, a major UK government department and a telecommunications and broadcasting company.
Article
This chapter begins with the description of the usability engineering and then describes its origins. The chapter then structures a discussion around each of the steps involved in doing usability engineering. Usability engineering is a process for defining, measuring, and thereby improving the usability of products. Usability engineering evolved because of a need to move usability from the realm of personal opinion to an attribute that is quantifiable like other engineering attributes. Practitioners can use the definitions and examples as a basis for their work. The chapter discusses the issues and pitfalls that provide background for establishing, marketing, and refining a usability engineering process in a development organization. Notes for the theorist on the conceptual underpinnings of usability engineering and the interrelationships among usability engineering and other methods like contextual inquiry, inspections, and scenarios are provided throughout the chapter. Finally, the chapter lists the ways in which other design models and methods complement the usability engineering framework.
Article
Many poor interface features are the result of carelessness, ignorance or neglect in the development process. For these features, methods such as user involvement in iterative design with prototyping, the us e of checklists and guidelines, and even formal evaluation can be of great help. However, there are strong forces present in development environments that block the use of such methods and distort interface designs in a systematic way. Because these forces serve legitimate goals, such as making a design simpler, more easily communicated, or more marketable, they are more difficult to counter; beca use developers are skilled at working toward those goals, the tangential effects on the interface usually pass unnoticed. This descriptive, empirical article describes these forces in the context of large organizations developing commercial off-the-shelf software products. Most points are supported by examples and by a logical argument. Not all of the phenomena may appear in a given development organization, but the overall picture of a complex environment in which interface development requires unwavering attention is quite general.
Article
This paper reviews user involvement in the design of human—computer interactions, as advocated by a selection of different approaches to design. The selection comprises: User-Centred Design; Participatory Design; Socio-Technical Design; Soft Systems Methodology; and Joint Application Design. The review reveals a preliminary identification of non-configurable and configurable ‘attributes’ of user involvement in design, and their associated ‘values’, which characterise the similarities and differences between the design approaches. The attributes and values are intended in the longer term to support designers to compare and contrast various design approaches and to make more informed choices about the configuration of user involvement in design practice. Requirements for future research into the better understanding and configuring of user involvement are proposed.
Article
This chapter summarizes a process of system design that, if followed, helps a person create usable computer systems for people—systems that are easy to learn, easy to use, contain the right functions, and are liked. There are four key points in this usability design process: early focus on users; empirical measurement; iterative design; and integrated design, where all aspects of usability evolve together from the start. This chapter reviews 20-30 informal methods to carry out these four points. Many of the methods can be used without extensive training in human factors. This chapter also summarizes how system design process is used to develop a new comprehensive software development system, called ITS. ITS was created to make it easier for developers to carry out this design process particularly incremental, iterative design. As a part of iteratively developing ITS, ITS was used to build serious applications, particularly multimedia ones, used by thousands and millions of users. This chapter describes these case studies, with emphasis upon the human factors methods used and the iterative, incremental approach.