Content uploaded by Stefano Bussolon
Author content
All content in this area was uploaded by Stefano Bussolon on Jul 31, 2016
Content may be subject to copyright.
Online card sorting: as good as the paper version
Stefano Bussolon
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
stefano@bussolon.it
Barbara Russi
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
Fabio Del Missier
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
delmissier@form.unitn.it
ABSTRACT
Netsorting is a web based cardsorting tool. Our research
group used Netsorting to run a number of experiments
on cognitive science and on information architecture. In
the study we are presenting here we compared two
couples of card sorts with the same data: two performed
with Netsorting, the other two with the traditional paper
based card sorting. We measured the performance of the
two groups with two indices: the number of correct
classification and the correlation among sampled
subsets of participants. The participants who used
Netsorting performed as good as the ones who used the
paper sorting.
Keywords
Card Sorting, Information Architecture, Web Based,
Usability
INTRODUCTION
Card Sorting is a user-centred design method, aimed to
improve the findability of a web site. The sorting task
requires that the users group a list of labelled cards in
coherent sets. After the task participants could be asked
to suggest a label for each set they formed. The card
sorting is an effective tool to elicit and represent the
implicit mental models of the users (Rugg and
McGeorge, 1997). This helps the information
architecture experts to better organize the informations
that will be easier to find and to use, improving the
quality of the site.
A number of computer based and web based card
sorting tools are available (for a review see
http://www.iawiki.net/CardSorting). Nonetheless, we
developed our own card sorting tool because we are
using it for a number of experiments, and therefore we
can adapt it for our experimental and statistical
requirements.
To test the validity of our web application we were
interested to know if there is any difference among the
paper based and the web based card sorting.
Methods
Design
Two factors:
1) interface: paper card sorting versus Netsorting;
2) domain of categorization: a list of animals and
a list of gifts.
Participants
60 participants took part to the paper based version of
the card sorting. Each participant performed both the
sorts (animals and gifts). 363 participants, recruited on
line, completed the card sorting of animals using
Netsorting. 736 participant, recruited on line as well,
completed the card sorting of a list of gifts using
Netsorting. We randomly selected 60 participant for
each online group for the calculus of the correlation
indexes.
Materials
A list of 60 names of animals, belonging to 4 categories:
15 mammals, 15 reptiles, 15 fish, 15 birds in the sorting
of animals. A list of 60 possible gifts in the sorting of
the gifts.
Procedure
We asked the participants to group the items in 4
different categories on what concerns the animals, in 6
categories for the gifts
Results
To estimate the goodness of the categorization of each
group we used an algorithm proposed by Tullis and
Wood (2004): we randomly divided each experimental
group in two subsets of 30 participant. One subset was
used as the control subset. The participants of the other
subset were randomly sampled to form groups of
different sizes (2, 4, 8, 12, 16, 24, 30 participants).
Correlation coefficients were calculated between the
similarity matrix of the control subset and the similarity
matrices of each sample. We therefore obtained, for the
four groups (animals online and offline, gifts online and
offline) a correlation index for each sample size (see the
first figure).
The correlation indexes of the animals sorting was
higher than the ones of the gifts sorting. The indexes
were pretty similar for the two interface condition
(paper versus Netsorting) in both the sortings: there is
no difference among the correlation index of the offline
and the online groups.
If all the users of the online experiments are used, the
indexes of the online condition are higher than the ones
of the offline condition, in both the sortings.
On what concerns the card sorting of the animals, we
also calculated the number of correct classifications
made by each participant. The mean correct
classification for the online (Netsorting) group was
calculated among those participants who classified at
least 54 items. The selected participants were 174.
The mean of correct classifications where different for
the two groups: online: 54.04, offline: 49.87. The
difference reached significance: F (1,232) = 6.114, p =
0.014.
Conclusions
The results of our web based card sorting are as good as
those obtained with the traditional paper card sorting
version. Increasing the number of participants improves
the quality of the sorting. Recruiting participants using
an online software is by far easier than using the paper
card sorting; furthermore, no data entry is needed using
a tool like Netsorting. The use of such a tool allows
therefore to test a greater number of participants, with
lower costs; as a result, better classifications can be
obtained.
REFERENCES
IAWiki: Cardsorting. Available as
http://www.iawiki.net/CardSorting
Rugg, G. and McGeorge, P. (1997). The sorting
techniques: a tutorial paper on card sortis, picture
sorts and item sorts. Expert Systems, 14(2):80 – 93.
Tullis, T. and Wood, L. (2004). How many users are
enough for a card-sorting study? In Proceedings
UPA’2004, Minneapolis, MN.