Conference PaperPDF Available

'Include', a Toolbox of User Research for Inclusive Design


Abstract and Figures

In order to empower more people to become more self-reliant in society, interactive products and services should better match the skills and values of diverse user groups. In inclusive design, relevant end-user groups are involved early on and throughout the design and development process, leading to a better user experience. However, for IT businesses not operating in the academic domain, getting access to appropriate user research methods is difficult. This paper describes the design and prototype development of the Include Toolbox, in close cooperation with practitioners of small to medium sized enterprises (SMEs) in IT. It consists of an interactive app paired with a book. The app helps to find suitable research methods for diverse user groups such as older people, people with low literacy, and children. The book offers background information on the advantages of inclusive design, information on different user groups, and best practices shared by other companies.
Content may be subject to copyright.
‘Include’, a Toolbox of User Research for Inclusive Design
Hanna Zoon
Eindhoven Technical University
Department of Industrial Design
P.O. Box 513, 5600 MB
Eindhoven, The Netherlands
Anita Cremers
P.O. Box 23, 3769 ZG
Soesterberg, The Netherlands
Berry Eggen
Eindhoven Technical University
Department of Industrial Design
P.O. Box 513, 5600 MB
Eindhoven, The Netherlands
+31 40 247 5227
In order to empower more people to become more self-
reliant in society, interactive products and services should
better match the skills and values of diverse user groups. In
inclusive design, relevant end-user groups are involved
early on and throughout the design and development
process, leading to a better user experience. However, for
IT businesses not operating in the academic domain, getting
access to appropriate user research methods is difficult.
This paper describes the design and prototype development
of the Include Toolbox, in close cooperation with
practitioners of small to medium sized enterprises (SMEs)
in IT. It consists of an interactive app paired with a book.
The app helps to find suitable research methods for diverse
user groups such as older people, people with low literacy,
and children. The book offers background information on
the advantages of inclusive design, information on different
user groups, and best practices shared by other companies.
Author Keywords
Inclusive design; user research methods; low literacy;
elderly; children; SMEs; IT; societal participation
ACM Classification Keywords
H.5.m. Information interfaces and presentation (e.g., HCI):
The world changes at an amazing pace, and technological
innovations happen faster and faster [1]. Behaviour, needs,
wants and likes of people also change, and many user
groups are less homogenous than before; fewer and fewer
people can be characterised by a traditional stereotype [31].
Older people: a 70-year old internet pioneer taking
programming classes, and a 90-year old going shopping by
bicycle. Disabled people: a 16 year old teen who battles
BMX riders in the skatepark and calls himself an extreme
sportsman, but also happens to be in a wheelchair [33].
Businesses cannot rely on existing knowledge alone; they
need increasingly up-to-date insights into the perception
and behaviour of their diverse customers, and need
inspiration on how to shape the future of their enterprise.
As public services and products become more interactive
and are increasingly offered solely online, it is important to
not exclude anyone from using these services. In many
countries, there is already legislation to ensure access to
essential online services and products for everyone, for
example ensuring a blind person can buy a train ticket, or a
low literate person can apply for benefits online. But also
for less essential interactive services and products, it can be
an advantage to design them in an inclusive way, taking
into account the diversity of the user group.
Inclusive design can be defined as the design of mainstream
products or services that are accessible to, and usable by as
many people as reasonably possible [20]. In inclusive
design, relevant end-user groups are involved early on [29]
and throughout the design and development process [3],
leading to more varied inspiration and more user-friendly
solutions. Making a product more inclusive, usually means
better usability for everyone [7]. It is not only a way to
solve problems, but also a strategy to identify problems to
solve. With the growing importance of brand image and
sustainable business practice, inclusive design can be a
selling point and may broaden the potential customer base.
It can seem easier and cheaper to self-reference than to
involve users. But to cater an application to a 28-year-old
male programmer, is actually designing for a tiny minority.
It can be very costly to launch an application and only then
discover it does not suit the people who were intended to
use it. From our previous research, it can be concluded that
awareness of inclusive design is still lacking. Unfamiliarity
with its advantages and methods seems to be a barrier to
practicing it.
User involvement does not need to be difficult, time-
consuming or expensive. For a business seeking inspiration
from customers, or wanting to get a feel of the perception of
their product by users, smaller informal sessions suit an
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
Copyright is held by the author(s).
Published in: van Leeuwen, JP, Stappers, PJ, Lamers, MH, Thissen,
MJMR (Eds.) Creating the Difference: Proceedings of the Chi Sparks
2014 Conference, April 3, 2014, The Hague, The Netherlands.
iterative design process much better and are more practical.
Sometimes, even one participant can be enough to get
inspiration for new business ideas [26], and a session with
five participants can uncover most of the general attitudes
about a subject [6]. User research methods are generally
very robust and even shorter, simplified execution will give
usable results.
It can be challenging to find a user involvement method
suitable for the goal of a project, time frame, resources and
the type of target users. Academic as well as popular
literature describes hundreds of research, ethnography and
usability methods [Sanders, E.B.N., personal communica-
tion], [4]. Especially for businesses not operating in the
academic domain, getting access to and finding the right
research methods can be a challenging task, that is often not
even started. As a result, users are often not included in the
development process.
There are many websites and toolkits available that offer
techniques to help getting new inspiration by thinking
differently, and tools for visual design. There are only a few
that offer user involvement methods, or are designed for use
by practitioners in the information technology domain (IT).
In this paper, we will describe the design and prototype
development of a toolbox for inclusive design and end-user
involvement, during which we worked closely together with
practitioners of small to medium sized enterprises (SMEs)
in the field of IT, that had little or no prior experience with
user research.
The Include Toolbox prototype that resulted from this
cooperation consists of an interactive app paired with a
book. The Include App (Figure 1) helps to find suitable
research methods for diverse user groups such as older
people, people with low literacy, and children. The Include
Book offers background information on the advantages of
Inclusive Design, information on different user groups, and
describes best practices shared by other companies.
The Include Toolbox can help bridge the knowledge gap
between academia and practice, and make Inclusive Design
possible for more businesses. Ultimately, the toolbox could
lead to interactive products and services that match the
skills and values of diverse target groups better, in order to
empower these people to become more self-reliant in
Figure 1. Home page of the Include Toolbox App (below)
Amongst the many toolkits available for designers, most
offer inspiration techniques and tools for visual or product
design. There are only a few that offer user involvement
methods, of which a selection is described here.
IDEO cards [17]
The classic IDEO method cards are widely used and the
methods on them became accepted practice in a lot of
businesses. The deck of 51 sturdy cards, developed by the
IDEO design firm, describes mostly inspirational methods
and also some user involvement methods, and are
specifically geared towards product designers. Already over
10 years old, it is one of the most well-known collections of
methods and is often used as basis for other card sets and
toolboxes, such as HCD Connect and Designing with
People (see below). On the cards is a colour-coded system
that helps designers to pick a suitable method. Of the
toolkits described here, it is the only one that is not free.
HCD Connect - Bill & Melinda Gates Foundation / IDEO
The Human-Centered Design Toolkit was designed
specifically for people, nonprofits, and social enterprises
that work with low-income communities throughout the
world. The HCD Toolkit walks users through a human-
centred design process and offers design- as well as user
involvement methods, bases on the IDEO cards. It is
divided in three parts: Hear, Create, Deliver; with the
appropriate methods listed under pictograms.
Designing with People - Helen Hamlyn Centre for
Design, Royal College of Art [11]
Designing with people is a website with very complete
information on research methods, design methods, user
groups background and stories, personas, and tips on doing
user research. It is set up in a clear way, so that it is easy to
decide which part of the information to view. The research
methods that are available in this toolbox, are based on the
IDEO cards and described in a compact way but further
references are also given. The Designing with People
website is one of the outputs from i~design 3, the final
phase of a collaborative research programme on inclusive
design funded by the Engineering and Physical Sciences
Research Council. It has been designed to work in
conjunction with the Inclusive Design Toolkit developed by
project partner, the Engineering Design Centre at
Cambridge University.
Inclusive Design Toolkit - Engineering Design Centre,
Cambridge University [15]
The Inclusive Design Toolkit, next to explaining inclusive
design and offering business rationale and design patterns,
offers physical tools to assist inclusive design, including a
Vision and Hearing Impairment Simulator, Cambridge
Simulation Gloves and Glasses, and an Exclusion
calculator. The design patterns mention user involvement
methods but for descriptions refers to the closely linked
Designing with People website. The Engineering Design
Centre also offers courses on inclusive design.
UCD Toolbox - Tristan Weevers UCD / Delft University of
Technology [12]
The UCD Toolbox is a very complete resource of methods
for user centred design. It contains an overview of 35
design methods, which can be filtered by criteria: type of
product, design goal, resources, participants and method
characteristics. Also, a pre-selection of methods can be
made for various target groups: elderly, children, physically
challenged, visual/hearing impaired or cognitively
challenged. However, no background information on
specific target groups is offered and it does not become
clear why the methods are suitable for the target groups.
Method descriptions contain a lot of information and it is
not always clear how much time, skill and effort a method
will take. Because of the extensiveness of this toolbox, it
can be hard for IT practitioners to find the specific
information they need.
UX Toolbox: Better Web for Citizens - British Columbia
Government [16]
A resource for government interaction and web designers,
the UX Toolbox is a complete manual containing
information on user experience, design research, web
strategy, information architecture, content design, and web
standards. The design research section describes research
methods in detail, some of which involve end-users. It
includes, research plans, reporting and managing tips. The
methods are sorted into categories based on the type of
research method, of which the terminology can be hard to
understand for non-experienced researchers. The UX
Toolbox also has a team that practitioners can contact for
information and help.
55plusToolbox Saxion [13]
The 55plusToolbox (in Dutch) focuses on topics that
change the innovation process as a result of choosing the
user group of people over the age of 55. Target users of the
toolbox are entrepreneurs, focusing on both product
development and marketing and sales. It contains
information on the user group as well as case studies.
Suitable tools for the particular phase and user group are
suggested and illustrated in factsheets containing step by
step guidance, visualisations, relevant links and references.
Universal Methods of Design - Bruce Hanington, Bella
Martin [10]
The fact that Universal Methods of Design is a physical
book, allows for another kind of interaction. When flipping
through the pages, one gets a clear overview but not too
much information at once, even though the book describes a
total of 100 methods. The book contains design-, research-
and user involvement methods, ordered alphabetically and
coded for each design phase. The target users of the book
are designers, but it is written in a very accessible way.
There is a short description given for each method,
completed with references and examples.
All toolkits and related work mentioned above are in theory
also usable for IT practitioners. However, because of
factors such as terminology and emphasis on creative
techniques, some are clearly meant to be used by designers,
not programmers. The British Columbia UX Toolbox is
made for the IT-practitioner target group, but requires
previous knowledge of and experience with user research.
In most existing toolboxes, it can be difficult to find a
suitable method, sometimes because there are so many,
sometimes because it is difficult to get an overview [28].
Often exact descriptions of a method are not given,
sometimes there are references to this information. But
even then it is often still hard to determine the level of
difficulty of the method, and the amount of time it will take.
Few toolboxes offer sufficient background information on
the needs and abilities of different target groups. Many of
the resources mentioned above, cover some aspect of what
we think an inclusive design toolbox needs to offer, but
there is not yet a resource available that caters specifically
to the needs and wishes of practitioners of SMEs in IT.
In earlier research [3], a workshop setting was used to
gather preliminary requirements for an inclusive design
toolbox, which formed the basis for this project.
To further define and refine the requirements, we worked
together with a total of 11 practitioners from seven SMEs in
IT. The project was set up in an Agile way [18] with many
short iterations. In the earlier stages of the project, focused
interviews were conducted about existing business
processes and views on inclusive design. In later stages,
two practitioners worked with the toolbox in their projects
[19] (Figure 2), providing feedback on the practical applica-
Figure 2. Practitioner performing user research using the
Business Origami method from the Include Toolbox
bility, while others provided feedback on design and
business aspects. Throughout the project, new companies
were introduced to get fresh insights from people who had
never seen the toolbox before. Five companies were smaller
(5-20 employees) privately owned businesses, one
practitioner was a PhD. student in the field of computer
science and one company was medium-sized (~200
employees), all based in the Netherlands. The business
activities included designing, developing and building
websites, intranets, e-learning applications and apps, and
In the beginning of the project, a very persistent attitude we
saw in the companies was that user research would be
difficult, expensive and not useful. Statements such as
‘users do not know what they want’ and ‘they never say
anything useful’ were heard. With further questioning, it
became clear that the people we interviewed were more
frustrated with the lack of success, than convinced of the
uselessness of user research. For example, they hated it
when their app for helping elderly using public transit came
on the market but was rejected by the target group, who
claimed they did not need help. The company was now
wondering if working with older people in a more
structured way from the beginning, could have prevented
the need for an expensive redesign. Another business owner
saw the business opportunities of making a clients website
more inclusive for elderly, attracting more customers that
By working closely together with practitioners and owners
from these and other companies, and using the knowledge
gathered in our earlier research, requirements were refined
and tested through prototyping throughout the project. This
process allowed for a thorough understanding of the needs
and business practice of our target user group, SMEs in IT.
It also made it possible to adapt and change the design and
setup of the toolbox continuously, to suit the requirements
more optimally.
The requirements we found for an Inclusive Design
Toolbox for SMEs in IT:
1. Easy to find the right method
Especially because it is so difficult for most people, and
therefore also end-users, to verbalise needs and imagine
products or services that do not yet exist, it helps to employ
a structured process for this task [25]. For academics,
searching a scientific literature library is easy. Finding a
specific, suitable method without knowing its name is
already more difficult. For non-academics there are books
and websites on user research, but because this is such a big
field, the amount of information is typically very large, and
the information is often geared towards experienced
researchers, not IT practitioners. The practitioners from our
interviews were often not aware of the existence of
scientific libraries or that there are many different user
research methods.
2. Decision aid versus freedom of choice
When looking for suitable user research methods, practi-
tioners wanted the autonomy to explore the methods
themselves first, to see what there was on offer. Only after
that, they would request toolbox recommendations. It was
mentioned many times that they would like to keep their
options open and choose themselves, and have the freedom
to deviate from recommended methods.
3. Methods: fast, robust and cheap
For a scientific research project, it can make perfect sense
to do a three-week long experiment with 120 participants.
But in the IT world, with Agile development cycles, the
time or budgets for this are often simply not there. And as
Nielsen [23] and Dix [6] argued, with about 3 to 5 people,
you will see most of the common behaviours from an entire
group, and each extra participant gives only limited extra
insights. Limiting the amount of participants limits the
expense of time and money, making user research more
likely to take place.
The methods should be robust so that even inexperienced
researchers making mistakes, can get useful information
from them. Therefore methods should be less dependent on
facilitator skills, and require only limited previous
knowledge of the target groups. Since sometimes the people
in IT businesses who want to do user research, have to
convince others, or need some convincing themselves, the
methods that are offered must be well-known or validated
methods, to inspire credibility.
4. Background information
Once the practitioners became more convinced of the
importance of taking the needs of end-users into account in
the development process, they often recommended that the
toolbox would have ready-made personas and background
information on user groups.
5. Examples (best practices)
Another requirement that came out of the interviews, was to
supply examples or best practices, so that the practitioners
could see how other businesses had approached user
research. This way, it would also be possible to give
examples of the type of knowledge one could expect to gain
from user research, without presenting it as if this particular
knowledge is generally applicable.
6. Business processes
Amongst the companies we worked with, there was a range
of business processes practiced. Practitioners who were
used to Agile or Lean [27] ways of working, were often
looking for ways to verify early concepts with users within
a short timeframe, because these methodologies require
new versions or changes to a product to be validated of
before the next iteration can begin. In more traditional
linear (waterfall) processes, there appeared to be a bigger
concern with the cost and validity of user research in
general. If a project went over budget, evaluations were one
of the first things to be skipped to save costs. User research
was often only done when they needed specific proof to
back up decisions. But even here, there was a general
acceptance of the idea that it is cheaper to identify issues
sooner rather than later.
7. Brand image / Commercial value
Next to the user research activities, it seemed important to
companies to be able to show these activities off, and be
able to use inclusive design as a sales tool. While an app or
online toolbox would be practical in use, a physical
component that can be shown in the office or given to
clients is also important.
Being able to talk about inclusive design as a complete
strategy, rather than separate user research methods, can
show a brands involvement in sustainable business
practices. It is important that the toolbox allows IT
practitioners to learn about inclusive design as a holistic
concept, to be able to sell it convincingly.
8. Terminology and tone
Some of the practitioners we worked with, joked about the
words ‘design’ and ‘research’ as concepts they were not
interested in. They saw themselves first as developers, as
clients often provided the visual design and not much
research was done. To avoid estranging practitioners with
design jargon or academic terms, it is important to use more
generic language or use terms from the IT world. On the
other hand, the people that we worked with were highly
educated entrepreneurs and independent thinkers, who did
not appreciate to be talked down to.
For a small to medium-sized company that works in the
field of IT, ‘Include’ is a toolbox of user research methods
providing an easy and efficient starting point for inclusive
design. The toolbox is an interactive application
accompanied by a hardcover book. The interactive part
gives an overview of easy user research methods, complete
with descriptions and workbooks, and helps filter them
according to the company’s needs and their user groups.
The book serves as a visual reminder, a reference for
background information and a way to show off the use of
inclusive design practice in the office.
The toolbox is currently available as a free application in
Beta, that has the purpose of creating awareness about
inclusive design in the IT sector. For future versions of the
toolbox, different business models could be considered. The
main functionalities and design elements of the toolbox are
described below and structured according to the
requirements listed above.
1. Directly from the home screen of the app, it is easy to
find the right method
Incorporated in the interactive toolbox are currently 10 user
research methods. This number of methods was arrived at
by weighing different factors: there must be enough
methods to suit four development phases and three specific
target groups, but not so many that it would become
overwhelming, especially for first-time users. Some
methods are suitable for more than one phase or user group.
The methods are all existing, validated methods, with
references. The methods can be browsed from the main
screen of the app, where after clicking a short description is
offered next to a recognisable picture for each method
(Figure 3). There is a filter bar, that can be hidden when
not in use, that contains two drop-down menus: “My end-
users include: [Older People, Children, People with Low
Literacy]” and “My goal is to: [Get Inspiration, Evaluate an
Idea, Review a Scenario, Test a Prototype]”. Using the filter
creates a recommendation of three selected methods, that
can be browsed through before making a choice.
Figure 3. Example of a method description card
2. The toolbox can act as a decision aid but also allows
for making alternative choices
Offered are two ways of looking for a method (browsing
and filtering). This has the benefit of being able to give
people who know what they want direct access to the
methods, while users looking for guidance can get it while
keeping the option to choose open.
3. The methods are fast, robust and cheap
From the requirements description, we distilled the
following criteria for user involvement methods that could
be useful to SMEs in IT:
it must be possible to do the preparation, session and
analysis in a total of 8 hours, one half day at the
beginning of the week and another at the end
no special skills or prior experience are necessary
it must be a well-known or validated, existing method
it looks like a fun activity for both the practitioner and the
all the necessary materials can be provided in a PDF
workbook in the form of tips or worksheets
get useful results with 3 to 5 participants
From a literature study, the following methods were
selected based on the aforementioned criteria:
A day in the life [9]
Cultural Probe [8]
Business Origami [10]
Mission from Mars [5]
Card Sorting [30]
Peer Tutoring [21]
Co-Constructing Stories [25]
Co-research [32]
Technology Tea Party [2]
Think Aloud [22]
The methods are described in the following way: first a
short introduction, on the basis of which the method is
chosen. Then there are seven cards used within the app:
1.overview of the method
2.description of the preparation, step-by-step, but short
(Figure 4) for the session itself, idem for a quick and effective analysis, idem
5.filling in the name of the project and a short description
6.choosing how to use the PDF workbook: print or digital,
and after the research, archiving the work
7.rating the method on suitability for the user group and
ease of use of the method itself, providing tips and
comments, choosing to share or not
4. and 5. Background information and best practices are
available in-app and in a hardcover book
User group characteristics change, and due to market
fragmentation, user groups are not as homogenous as before
[31]. Providing pre-made personas (in contrast to carefully
constructed personas based on relevant user research data)
would mean to overly simplify and stereotype the people of
a user group [24], ignoring the needs and wants of sub-
groups or failing to spot underlying patterns in a user group
specific to a certain project. Therefore, even though
practitioners asked for ready-made personas, we decided
against this.
Figure 4. Example of a method workbook card
The toolbox does provide general background information
about the larger target groups, to get a first idea of the
characteristics of the group, and prepare the user
involvement sessions more efficiently. In the app, a menu
can be shown for access to more in-depth background
information, such as about inclusive design, the methods,
user group information and best practices.
From the method descriptions, there are also links to this
information. Because it is unlikely that users are going to
read many pages from within an app, the complete
information is also available in the form of a hardcover and
e-book. The contents of the book and the app overlap, to
accommodate both easy reading and flipping through paper
pages, as well as the ease-of-use, portability and the help of
links within in the app.
6. Implementation in iterative business processes
Because of the limited preparation and analysis time needed
for the methods covered by the toolbox, they are suitable to
be implemented in business processes with short iterations
such as Agile or Lean. User centred design based on design
cycles is also a good fit. For other project structures, early
user research can still be beneficial to starting the project on
a relevant course, and also wherever there is a decision
moment. In some cases, using the toolbox may even help
practitioners to see their development process clearer,
because the preparations for user research often include
describing and focusing on the end-goals of the project.
7. Brand image / Commercial value
Research results can be used as proof to clients, validating
development decisions. Being able to show that their
company is practicing inclusive design was also perceived
to be a selling point towards new clients. The book
accompanying the interactive app allows for that, and is
also a visual reminder in the office to think about special
user groups more often.
Another aspect is that being able to show customers what
user research is, can convince them a project needs it. It
also makes it easier to talk about inclusive design, if clients
can learn about it from a third party. More efficient
development processes combined with a better brand
image, can increase the overall competitiveness of a
8. Terminology and tone
The descriptions of the methods are void of design jargon
and academic terminology as much as possible. The
background information in both the book and the app, is
written from a perspective of an SME entrepreneur, or IT
practitioner, linking back to what they can get out of it in
their work, or what it will do for their businesses.
Furthermore, the steps of each method are balanced so that
they give just the necessary amount of information without
overwhelming or condescending.
In addition to working with different SMEs in IT
throughout the project, the final prototype of the interactive
toolbox app was evaluated qualitatively on visual and
interaction design and user perception with five
participants, in three sessions. The participants of these
sessions were practitioners in IT, with limited previous
experience with user research. The evaluation method used
was based on the Co-Constructing Stories [25] interview
All participants said they thought they could easily execute
the methods offered by the Include Toolbox, and
appreciated the detailed workbooks. In general they were
surprised each method could be executed in 8 hours and
realising this was often a pivot point in the interview; from
talking about difficulty and hurdles to talking about the
benefits of involving users. They agreed that the toolbox
language should be available in Dutch for ease of use with
end-users from the Netherlands, especially for the target
groups older people, people with low literacy, and children,
who may not speak English. Another overall feedback was
that the participants could appreciate the visual language of
the toolbox, which was seen as colourful and inviting, and
the information architecture, which was perceived as easy
to use.
Participants commented on how they would use the toolbox
app, by saying they would browse the overview page for a
while first, before choosing a method to work with.
Methods with ‚research’ in the name came across as boring
and formal, and some inspirational methods were seen as
too creative. Getting a recommendation of three selected
methods when using the filters was a good number for
participants. In practice, participants would like to spend a
minimum amount of time in the app, just quickly choose a
method, print everything out and get to work.
Some participants would have liked to see more best
practices, some said they would need much more different
user groups and more specific user group information
because for each project the target user is different. At the
time this evaluation took place, the content of the toolbox
prototype was not yet complete, and some participants were
eager to get the remaining method descriptions, perhaps
showing their motivation to get started with the methods.
Participants also noted that it seemed to them that
answering the questions in the filter bar of the app, would
already bring some awareness to practitioners about
inclusive design. Using the toolbox to educate their clients
about user research, could give the clients more confidence
to discuss the details of user research, making them more
satisfied with the company in the long run. There were
some concerns if the option to share the research result as a
best practice, would safeguard the anonymity of the end-
users enough.
The hardcover book that accompanies the Include App
(Figure 5), was not yet finished by the time the final
evaluations took place. During the sessions, an interesting
contradiction arose when discussing background
information about topics such as inclusive design, user
involvement methods, user group descriptions and best
practices. For many practitioners, it was important to know
it was offered and how they could reach it easily. However,
when it was visually represented in the toolbox, links to the
information were appreciated, but not opened. In an
informal evaluation in an exhibition setting that did show
the book, people were mainly attracted to the big size, and
colourful layout.
Figure 5. The Include Book
Even though there were some criticisms from the
participants in the final evaluations, the overall feedback
was positive, with participants envisioning their use of the
Include Toolbox and expressing motivation to use it in their
work. The toolbox app could perhaps reach more
practitioners if it allows for more company profiles, next to
the currently very specific: practitioners from SMEs in IT
with little or no experience in user research.
Some participants would have liked to see more best
practices in the toolbox, and this is indeed planned by
allowing users of the toolbox to share their own
experiences. This way, over time the toolbox content
becomes an increasingly complete reference guide. Further
progress towards a more complete and continuously
updated toolbox could be made through making it possible
for practitioners to add or edit information, for example add
new methods, update target group information, provide tips
and rate methods.
Other practitioners wanted more specific user group
information, but by their own admission the specifics are
unique to each project. It is our intention to add support for
more end-user groups such as teenagers, non-native people,
or people with physical or mental disabilities. In our
opinion, next to using the general information in the user
group descriptions, it would also be necessary to work
closely with users in each project, thereby making sure of
having the most specific, unique and up-to-date insights as
In our evaluations where practitioners used a prototype of
the toolbox in real projects, we and the practitioners found
that inclusive design was within practical reach for them.
The final evaluations showed acceptance of and enthusiasm
for the visual design and the research methods offered.
The next phase of this project is the departure from the
prototype stage, starting the development of the application,
incorporating everything learned from the prototype phase.
Compared with existing toolboxes and resources, we
believe the Include Toolbox is specifically suitable for
SMEs in IT, due to its visual design, setup, and the
difficulty level of the methods offered, more so than other
toolboxes. However, this is something that needs to be
further validated with a comparative study of toolkits
amongst IT practitioners.
Although the first informal evaluations have been positive,
more formal empirical study is needed to verify our claim
that with the Include Toolbox, IT practitioners are able to
perform user research with a quality adequate for their
commercial purposes. Also we need to further substantiate
the business rationale of inclusive design and user research.
To find out more about the need and use for background
information and a physical book, further evaluations need
to be done to find out if a book is the best form to present
the information, and how much background information is
in fact required by the SMEs.
The current work was carried out as part of COMMIT, a
Dutch national public-private research program on
information and communication technology, project P02:
Interaction for universal access (
We thank Bas Holleman for his help with the interaction
1. Brand, R., & Rocchi, S. Rethinking value in a changing
landscape. A model for strategic reflection and business
transformation. Philips design (2011).
2. Coventry, L., Jones, E. The role of tea parties to elicit
technology requirements to support the mobility of older
adults. Proc. PETRA (2012).
3. Cremers, A. H. M., Neerincx, M. A., & de Jong, J. G.
M. Inclusive design: bridging theory and practice.
Engineering Psychology and Cognitive Ergonomics,
Applications and Services. Springer Berlin Heidelberg
(2013). 323-332.
4. Curedale, R. Various titles, Design Community College
Incorporated (2012-2013).
5. Dindler, C., Eriksson, E., Iversen, O. S., Lykke-Olesen,
A., & Ludvigsen, M. Mission from Mars. Proc. IDC
6. Dix, A. Humancomputer interaction: A stable
discipline, a nascent science, and the growth of the long
tail. Interacting with Computers, 22(1), (2010), 13-27.
7. Eikhaug, O., Gheerawo, R., Plumbe, C., Berg, M. S., &
Kunur, M. Innovating with People- The Business of
Inclusive Design. Norsk Designråd (2010).
8. Gaver, B., Dunne, T., & Pacenti, E. Design: cultural
probes. Interactions, 6(1), (1999), 21-29.
9. Gillen, J., Cameron, C. A., Tapanya, S., Pinto, G.,
Hancock, R., Young, S., & Gamannossi, B. A. A day in
the life, Early Child Development and Care Vol. 177,
Iss. 2, (2007).
10.Hanington, B. & Martin, B. Universal Methods of
Design: 100 Ways to Research Complex Problems,
Develop Innovative Ideas, and Design Effective
Solutions. Rockport Publishers (2012).
17.IDEO, Method Cards: 51 Ways to Inspire Design. Palo
Alto (2003).
18.Jongerius, P., Offermans, A., Vanhoucke, A.,
Sanwikarja, P., & van Geel, J. Get Agile!: Scrum for
UX, Design & Development. BIS Publishers (2013).
19.Kayal, A., Brinkman, W.P., Zoon, H., Neerincx, M.,
Van Riemsdijk, M. B. Using norms for social
applications to provide value-sensitive support to
families. Submitted for publication (2014).
20.Langdon, P., & Thimbleby, H. Inclusion and interaction:
Designing interaction for inclusive populations.
Interacting with Computers, 22(6), (2010). 439-448.
21.Markopoulos, P., Read, J. C., MacFarlane, S., &
Hoysniemi, J. Evaluating children's interactive products.
Morgan Kaufmann (2008).
22.Nielsen, J. Thinking Aloud: The #1 Usability Tool.
NNgroup (2012). Via
thinking-aloud-the-1-usability-tool/, retrieved 2013.
23.Nielsen, J., Landauer, T.K. A mathematical model of the
finding of usability problems. Proc. CHI, INTERACT
(1993), 206-213
24.Nielsen, L. Constructing the user. Human-computer
interaction: theory and practice. Part II 2 (2003), 430
25.Ozcelik-Buskermolen, D., & Terken, J. Co-constructing
stories. Proc. PDC (2012).
26.Razak, F. H. A. Single person study: Methodological
issues. Doctoral dissertation Lancaster University
27.Ries, E. The Lean Startup: How today's entrepreneurs
use continuous innovation to create radically successful
businesses. Random House Digital, Inc (2011).
28.Seffah, A. Bridging the educational gap between SE and
HCI: what software engineers should know? Proc. IET
(2004), 88-95.
29.Sluis-Theischeffer, W., Bekker, T., & Eggen, B. Adding
user creativity to the UX toolbox: Exploring the use of
Creative UX methods. Proc. CHI Netherlands (2009).
30.Spencer, D. Card Sorting, O'Reilly Media, Inc. (2009).
31.Tidd, J., & Bessant, J. Managing innovation: integrating
technological, market and organizational change. Wiley
(2011), Chapter 1.3.
32.Van Doorn, F., Stappers, P. J., & Gielen, M. Design
research by proxy. Proc. SIGCHI (2013).
33.Wheelchair teen in record-breaking backflip, Metro UK
... The data acquisition work done in this study has also been used by Cremers et al. [71] to provide a more general overview of lessons learned about using anthropological methods with special-needs demographics. The methods presented here have also been included in the Inclusive Design Toolbox [176], a collection of inclusive design methods aimed at supporting SMEs, as examples of how to approach a low-literate target audience. ...
... These are described in turn. 176 7 ...
Full-text available
This thesis presents the research, design, and evaluation of the learning support system VESSEL: Virtual Environment to Support the Societal participation Education of Low-literates. The project was started from the premise that people of low literacy in the Netherlands participate in society less often and less effectively than literate people do: Their lower ability to read, write, speak, and understand the Dutch language hampers their ability to independently be part of society. Our goal was to create learning support prototypes with a re-usable design rationale, aimed at helping these people of low literacy learn to improve their societal participation. To achieve this, low-literate learners participated throughout the entire design process, ensuring that we addressed their wants and needs with regard to learning and the perceived shortcomings of existing learning materials and kept in mind their skills and capabilities in order to ensure effective learning. Particularly, we investigated the possible ways that digital learning, Virtual Learning Environments (VLE), and Embodied Conversational Agents (ECA) could help fulfill the societal participation needs of this target group. We used the Socio-Cognitive Engineering (SCE) methodology to organize and structure this research, distinguishing the foundation, specification and evaluation of the VESSEL design. Two studies provided a grounded foundation for VESSEL, which was refined and worked out into three subsequent studies that provided the consequential design specifications and prototype evaluations (all prototypes have been tested with a human ’Wizard of Oz’ simulating VESSEL functionality). In the first study, we collected necessary information for the foundation of VESSEL in three categories. The first category consisted of the operational demands, which form an overview of the context of use: Demographic information about low-literate learners in the Netherlands, a description of the crucial practical situations of participating in Dutch society, and important attributes of learning societal participation in the Netherlands. The second category encompassed human factors knowledge, consisting of literature about adult learning and ICT-supported learning. The third category contained technology insights, which we gathered by looking at both existing learning support software in the areas of low literacy and societal participation in the Netherlands, and the envisioned capabilities of VLEs. In the second study, we extended and refined our knowledge of the operational demands (as the foundation for VESSEL). We spoke to low-literate language learners in the Netherlands, in order to gain qualitative insights into their daily life experiences related to participating in Dutch society. We used participant workshops and Cultural Probes to obtain large amounts of rich data pertaining to these experiences, and we used the Grounded Theory method to transform these data into the Societal Participation Experiences of Low-Literates (SPELL) model. This model describes the four attribute categories of societal participation experiences: Personal attributes, formal societal attributes, information societal attributes, and information-communication attributes. In the third study, we used our foundation of information to create a first prototype, a ’proof-of-concept’ VESSEL. This prototype consisted of four interactive scenario-based learning exercises: Two exercises (’Easy’ and ’Hard’) about conducting online banking, and two exercises about talking to a city hall service desk employee. The prototype also contained our ECA ’coach’, Anna, who could provide three types of learning support: Cognitive learning support based on scaffolding, affective learning support based on motivational interviewing, and social learning support based on small talk. This prototype was evaluated with low-literate language learners throughout the Netherlands in an empirical mixed-method experiment, in which users did all four exercises both with and without coach support. Results showed that all learners managed to complete all exercises with coach support, while almost no learners completed all exercises without coach support. Participants interacted with the coach in a natural manner: They asked for her help and even talked to her without external prompting. A majority of participants appreciated her presence and help. In the fourth study, we formalized the coach’s cognitive learning support capabilities for the design and evaluation of the second prototype. We used existing scaffolding literature and our own experiences from the third study to define five levels of cognitive learning support: Prompt, Explanation, Hint, Instruction, and Modeling. We created a large corpus of standardized speech utterances for the coach in the context of the Hard Online Banking exercise, and wrote detailed rules describing which type of utterance the coach should use in any given situation, how long the coach should wait between utterances, and what kinds of user-uttered keywords she should react to and how. The model describes that the coach should always offer the lowest level of support (Prompt) for any new topic, that support should always go up in level and never repeat itself unless asked, and that the coach should wait a certain amount of time after any utterance. Two support models were made to describe this timing: The Generalized Model, in which the coach always waits 20 seconds, and the Individualized Model, in which the coach adapts the support wait time to the individual participant’s previous performance. The second prototype was created, focusing on an expanded version of the Hard Online Banking exercise, and an empirical mixed-method experiment was conducted with low-literate learners to test the differences between the two support models: Learners completed three exercises in either the Generalized condition (with a consistent 20 second support wait time) or the Individualized condition (in which their support time in exercises two and three depended on their results in exercises one and two). We hypothesized that both the Generalized and Individualized Models would increase learning effectiveness, and that the Individualized Model would increase learning effectiveness significantly more than the Generalized Model. Results support the first hypothesis: Support from either model resulted in high learning effectiveness and higher self-efficacy for low-literate learners, and low-literate learners managed to use the new keyword-based speech recognition without the need for explanation. The second hypothesis was not supported: No differences in learning effectiveness were found between the two support models. In the fifth study, we formalized the coach’s affective and social support capabilities for the design and evaluation of the third prototype. We used existing motivational interviewing literature to define four levels of affective learning support (Reflective Listening, Normalizing, Affirmation, and Self-Efficacy Supporting) for three emotional states (Anger, Fear, and Sadness) at three levels of specificity (General, Specific, and Very Specific), and created a corpus of affective support utterances: For each combination of emotional state and specificity (General, Specific, or Very Specific Anger, Fear, or Sadness), one or two support utterances were created for each level of affective support. We used the Shimmer photoplethysmographic sensor and the FaceReader facial expression recognition software to infer learners’ affective states from their heart rate and facial expressions (respectively), and connected this to new affective support rules: Whenever the coach inferred an emotion at a certain level of specificity, it should use one Reflective Listening utterance relevant to that particular combination, one Normalizing utterance, one Affirmation utterance, and one Self-Efficacy utterance, in that order. We also used existing small talk literature to write a simple branching small talk script for the coach, focused on bonding with the user and introducing the new Volunteer Work exercise, in which learners had to fill out a volunteer work background information form and then talk to an ECA about their answers. A third and final Wizard-of-Oz prototype was created and evaluated with low-literate learners in an empirical mixed-method experiment, in which learners completed the full exercise once with only cognitive learning support and once with cognitive, affective, and social learning support. Results did not show strong significant differences between the two conditions. We identified three potential explanations: Our exercises did not manage to evoke emotional reactions in learners strong enough for our sensors to detect, our affective support model was not effective in the way we intended, and/or our experimental setup limited the amount of emotional reactions learners could experience. However, the prototype in general did work as intended: Learners completed every exercise, requested and used the coach’s support, and reported higher self-efficacy at the end of the experiment. This experiment also reported differences between NT1 and NT2 learners and between men and women, suggesting more careful study into demographic differences will be required. Overall, results from our studies show that VESSEL seems to be increasing learning effectiveness. Learners across studies reported that working with VESSEL made for a positive learning experience, and after doing challenging societal participation exercises for the first time, learners’ self-efficacy regarding the exercise topic (online banking / volunteer work) increased and stayed on the new level throughout. However, it proved difficult to clearly identify distinctive effects for specific VESSEL functions: For instance, positive learning outcomes could not clearly be attributed to the adaptive timing of the support or the constructive scaffolding used for cognitive support, and the positive experience of interacting with the coach could not be attributed to the presence of scripted small talk and affective support. Crucially, our results show that learners were able to use VESSEL as intended: they interacted with the exercises as intended and with the coach as envisioned, without the need for prior explanations or tutorials (save a brief introduction given by the coach). This suggests that we have managed to incorporate the actual capabilities, shortcomings, and wants and needs of people of low literacy into the design of VESSEL. However, it is not clear whether these positive outcomes would apply to all low-literate learners: While we attempted to recruit low-literate participants from different backgrounds and skill levels, on reflection, the majority of our participants were relatively high-skilled and intrinsically motivated. This is further complicated by the relatively low number of participants in our experiments, which calls the power of the results into question. Just as importantly, we regularly saw that learners socially engaged with ’Anna’: They responded to her questions, asked questions of their own, thanked her for her help, and even occasionally talked to her as if she was a real person – telling stories and making jokes. Learners were grateful for the support, and generally indicated that they would like to receive more support like this in the future.
... The user research process includes (Portigal, 2013): studying people deeply, ideally in their context; exploring not only their behaviour, but also the meaning behind the behaviour; making sense of data by using inference, interpretation; analysis and synthesis; and using those insights to point toward a design, service, product or other solution. When relevant end-user groups are involved early on and throughout the design and development process, a better user experience is achieved (Zoon, Cremers and Eggen, 2014). ...
... As public services and products become more interactive and are increasingly offered solely online, it is important to not exclude anyone from using these services (Zoon, et al., 2014). The lack of ICT skills in the public sector is a major challenge for e-Government initiatives (Nkohkwo and Islam, 2013). ...
Conference Paper
Full-text available
The Internet is reshaping government through improved public services. A deep understanding of the users' needs is crucial for building a successful digital service. User research can be used to understand how people live their lives, in order to more effectively respond to user needs with informed and inspired design solutions. What remains, largely missing, from government monitoring is the citizens' experience of government services and the systematic use of this evidence to improve performance. Users interacting with government websites often experience that not enough has been done to anticipate their needs or make information easily available and locatable. The lack of citizen inclusion is found to be key in inhibiting e-Government initiatives. User Experience (UX) activities, such as user research, are typically not included in the Systems Development Lifecycles of South African Governments. User research plays a crucial role to ensure that the promise of technology, such as e-Government, is realised to its full potential. The focus of this paper is on user research conducted to inform the information architecture of government websites. Focus groups and one-on-one interviews were conducted in rural areas in the Western Cape in South Africa. Grouping services by categories according to user interest is considered as a building block for e-Government success. Grants, jobs and business government service categories are identified. The results achieved wider user research buy-in in government and a mandate to categorise government services online according to citizen needs. The results will be specifically of value for national, provincial and local government in Southern Africa. The results of this study have implications for government officials, information technologists, usability evaluators and specifically e-Government website designers.
... In this paper, we describe our design method, consisting of adaptations of several methods commonly used in user-centred design processes for interactive product development, but until now seldom applied to the context of automating journalists' work practice. User-centred design methods involve end-users, in this case journalists, from the start and during the entire design process, causing better acceptance and a more positive user experience of the end product (Zoon, Cremers & Eggen, 2014). ...
Conference Paper
Full-text available
We propose our user-centred design method for automated journalistic tools with the goal of supporting journalists' efficiency, allowing for more time to be allocated to investigative and reflective tasks that lead towards in-depth journalism. Our method allows for a journalist-focused, positive perspective on algorithmic news: instead of viewing journalism as a practice in need of replacing, we provide a holistic way to look at journalistic ways of working where there are many promising opportunities for innovation using natural language generation and other artificially intelligent technologies.
... The data acquisition work done in this study has also been used by Cremers et al. (2014) to provide a more general overview of lessons learned about using anthropological methods with special-needs demographics. The methods presented here have also been included in the Inclusive Design Toolbox (Zoon, Cremers, & Eggen, 2014), a collection of inclusive design methods aimed at supporting SMEs, as examples of how to approach a low-literate target audience. Using Grounded Theory, we turned our rich empirical user experience data into the SPELL model (Fig. 4). ...
Specialized learning support software can address the low societal participation of low-literate Dutch citizens. We use the situated Cognitive Engineering method to iteratively create a design specification for the envisioned system VESSEL: a Virtual Environment to Support the Societal participation Education of Low-literates. An initial high-level specification for this system is refined by incorporating the societal participation experiences of low-literate citizens into the design. In two series of user studies, the participant workshop and cultural probe methods were used with 23 low-literate participants. The Grounded Theory method was used to process the rich user data from these studies into the Societal Participation Experience of Low-Literates (SPELL) model. Using this experience model, the existing VESSEL specification was refined: requirements were empirically situated in the daily practice of low-literate societal participation, and new claims were written to explicate the learning effectiveness of the proposed VESSEL system. In conclusion, this study provides a comprehensive, theoretically and empirically grounded set of requirements and claims for the proposed VESSEL system, as well as the underlying SPELL model, which captures the societal participation experiences of low-literates citizens. The research methods used in this study are shown to be effective for requirements engineering with low-literate users.
... As public services and products become more interactive and increasingly offered solely online, it is important not to exclude anyone from using these services (Zoon, Cremers and Eggen, 2014). The WCG has Cape Access e-Centres in rural areas throughout the Western Cape. ...
Conference Paper
Full-text available
Youth-at-risk in the Western Cape are faced with various challenges such as poverty, unemployment, drug and alcohol abuse and an uncertain future. Online youth information can help young people to find the right information and take their own informed decisions. The Internet is a powerful source of information and communication, as well as an integrated part of the social environment of young people. A deep understanding of the users’ needs, the youth, is crucial for building a successful online service. An online service must meet the user needs, and one way to understand those needs is through the creation of personas. The personas method allows for describing the users’ characteristics, goals, motivations behind those goals, skills and demographic profile. User research (interviews and focus groups) were conducted in rural areas of the Western Cape in South Africa to build personas of youth in the Western Cape. The user research, together with a literature study of services offered on current South African youth websites, were conducted to produce a list of online service categories required by youth of the Western Cape. The resulting deliverables of this study are youth personas and proto-personas for the Western Cape, as well as recommended categories of services for an online presence, such as government websites.
Full-text available
The paper discusses four paradigms of economic value creation that completely transform how companies create and deliver value.
Interactive products designed for childrenwhether toys, games, educational products, or websitesare increasingly embedded in childrens lives and school experiences. Making these products safe, effective, and entertaining requires new methodologies for carrying out sound and unbiased evaluations for these users with unique requirements, environments, and ethical considerations. This book directly addresses this need by thoroughly covering the evaluation of all types of interactive technology for children. Based on the authors' workshops, conference courses, and own design experience and research, this highly practical book reads like a handbook, while being thoroughly grounded in the latest research. Throughout, the authors illustrate techniques and principles with numerous mini case studies and highlight practical information in tips and exercises and conclude with three in-depth case studies. Essential reading for usability experts, product developers, and researchers in the field. * Presents an essential background in child development and child psychology, particularly as they relate to technology. * Captures best practices for observing and surveying children, training evaluators, and capturing the child user experience using audio and visual technology. * Examines ethical and legal issues involved in working with children and offers guidelines for effective risk management.
Conference Paper
While most software developers may have been exposed with basic concepts such as GUI design guidelines, few developers are able to understand the human/user-centered design (UCD) toolbox at a level that allows them to incorporate it into the software development lifecycle. This paper describes an approach for skilling developers and students enrolled in an engineering program in critical user-centered design techniques and tools. The proposed approach starts from the analysis of the usability and software engineer's work context, identifies critical UCD skills and then associates relevant learning resources with each of the identified skills. Our approach suggests a list of learning patterns and resources extracted from the immediate software developers' work.
Conference Paper
We investigate how to create a mobile social application to support families with elementary school children, assisting them in exploring their so-cial and geographical environment (for example by helping them staying safe, making new friends or getting to know their neighbourhood). Existing social ap-plications such as Foursquare, Twitter and Life360 already provide functionality that supports these tasks to some extent. However, while promoting important user values such as safety and family security, these applications can negatively impact other values such as privacy and independence. This stems from the fact that people's values play out differently with varying family situations and social contexts, which current social applications cannot accomodate. In this paper we investigate how social applications can become more adaptive in order to provide value-sensitive support to families. We propose as a solution that users should be able to express rules of behavior (norms, viewed as concretized values) to the so-cial application, mirroring real-life social norms. In multi-agent systems reseach various generic models have already been proposed for expressing social norms, but it has not been investigated how these can be used for influencing the behavior of social applications for families. We develop a tailored normative model in three steps: 1) identification of important family values in this application domain, 2) analysis of current capabilities of social applications in terms of norms, show-ing that they lack expressivity on several dimensions (such as context-sensitivity) which negatively impacts family values, and 3) a normative model that provides the required expressivity. We evaluate the model through a qualitative user study with parents and children, showing that their requirements for the behavior of the social application can be expressed using our normative model.
Conference Paper
Mobility is fundamental to aging well, maintaining independence and quality of life. Physical mobility is a fundamental component of overall mobility. This paper presents our "technology tea party" methodology. We have used this method to explore older adults' reaction to utilizing different technologies to maintain their mobility, in particular, games consoles to promote exercise at home. At our tea party, participants experienced problems with initially getting started with the console but with minimal support were able to interact with the system and start to explore the benefits they could gain from using such a system. They saw the benefits not only as exercise but also as promoting social interaction. We are now using this method in the design process for an "intelligent" walking aid to improve the ability of older adults to walk around shopping centres. In this paper we advocate the involvement of older adults throughout the design process and we present tea parties as a potential method to use in early stages to elicit requirements.
Conference Paper
Large groups in society lack the necessary skills to be sufficiently self-reliant and are in need of personal assistance. These groups could be supported by information and information technology (ICT), but only if this technology is designed to fit their (cognitive) abilities. Inclusive design theory and methods have already been developed in research contexts, but there is still a gap between theory and practice. There is a need for a practical aid, that helps to create awareness of inclusive design among ICT developers, and offers easy-to-use information and tools to actually apply the methods for diverse target groups. This paper describes the first steps taken towards an inclusive design toolbox for developing ICT applications that offer cognitive support for self-reliance. Dutch ICT companies were interviewed and participated in a co-design workshop, leading to a number of initial needs, user requirements, and an on-line community, that form input for further development of the toolbox.
In this paper we introduce a participatory design technique for early, formative concept evaluations to elicit in-depth user feedback and suggestions, revealing attitudes and motivations of users. The technique is motivated by the link between memories, experiences and dreams, and is based on the assumption that users can make better judgments about novel design concepts if they link them to their past experiences. The technique involves user sessions consisting of two main phases, one focusing on recollecting past experiences in related contexts, and one focusing on envisioning future experiences that can be enabled by the use of the concept. In both phases, designer-user dialogue is established through storytelling. Storytelling is used by the designer to set the stage and to present the concept and by the user to communicate his past and anticipated future experiences. The technique results in joint stories about novel concepts. In this paper we explain the technique in detail. We discuss its theoretical background and relation to other user research methods. We share the insights that we gathered through first pilots. The paper concludes with a discussion about the use of this technique in the design process and future research.