ArticlePDF Available

Online card sorting: as good as the paper version

Authors:

Abstract

Netsorting is a web based cardsorting tool. Our research group used Netsorting to run a number of experiments on cognitive science and on information architecture. In the study we are presenting here we compared two couples of card sorts with the same data: two performed with Netsorting, the other two with the traditional paper based card sorting. We measured the performance of the two groups with two indices: the number of correct classification and the correlation among sampled subsets of participants. The participants who used Netsorting performed as good as the ones who used the paper sorting.
Online card sorting: as good as the paper version
Stefano Bussolon
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
stefano@bussolon.it
Barbara Russi
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
Fabio Del Missier
Dipartimento di Scienze della
Cognizione e della Formazione
Università di Trento
Via Matteo del Ben, 5
38060 Rovereto (TN)
delmissier@form.unitn.it
ABSTRACT
Netsorting is a web based cardsorting tool. Our research
group used Netsorting to run a number of experiments
on cognitive science and on information architecture. In
the study we are presenting here we compared two
couples of card sorts with the same data: two performed
with Netsorting, the other two with the traditional paper
based card sorting. We measured the performance of the
two groups with two indices: the number of correct
classification and the correlation among sampled
subsets of participants. The participants who used
Netsorting performed as good as the ones who used the
paper sorting.
Keywords
Card Sorting, Information Architecture, Web Based,
Usability
INTRODUCTION
Card Sorting is a user-centred design method, aimed to
improve the findability of a web site. The sorting task
requires that the users group a list of labelled cards in
coherent sets. After the task participants could be asked
to suggest a label for each set they formed. The card
sorting is an effective tool to elicit and represent the
implicit mental models of the users (Rugg and
McGeorge, 1997). This helps the information
architecture experts to better organize the informations
that will be easier to find and to use, improving the
quality of the site.
A number of computer based and web based card
sorting tools are available (for a review see
http://www.iawiki.net/CardSorting). Nonetheless, we
developed our own card sorting tool because we are
using it for a number of experiments, and therefore we
can adapt it for our experimental and statistical
requirements.
To test the validity of our web application we were
interested to know if there is any difference among the
paper based and the web based card sorting.
Methods
Design
Two factors:
1) interface: paper card sorting versus Netsorting;
2) domain of categorization: a list of animals and
a list of gifts.
Participants
60 participants took part to the paper based version of
the card sorting. Each participant performed both the
sorts (animals and gifts). 363 participants, recruited on
line, completed the card sorting of animals using
Netsorting. 736 participant, recruited on line as well,
completed the card sorting of a list of gifts using
Netsorting. We randomly selected 60 participant for
each online group for the calculus of the correlation
indexes.
Materials
A list of 60 names of animals, belonging to 4 categories:
15 mammals, 15 reptiles, 15 fish, 15 birds in the sorting
of animals. A list of 60 possible gifts in the sorting of
the gifts.
Procedure
We asked the participants to group the items in 4
different categories on what concerns the animals, in 6
categories for the gifts
Results
To estimate the goodness of the categorization of each
group we used an algorithm proposed by Tullis and
Wood (2004): we randomly divided each experimental
group in two subsets of 30 participant. One subset was
used as the control subset. The participants of the other
subset were randomly sampled to form groups of
different sizes (2, 4, 8, 12, 16, 24, 30 participants).
Correlation coefficients were calculated between the
similarity matrix of the control subset and the similarity
matrices of each sample. We therefore obtained, for the
four groups (animals online and offline, gifts online and
offline) a correlation index for each sample size (see the
first figure).
The correlation indexes of the animals sorting was
higher than the ones of the gifts sorting. The indexes
were pretty similar for the two interface condition
(paper versus Netsorting) in both the sortings: there is
no difference among the correlation index of the offline
and the online groups.
If all the users of the online experiments are used, the
indexes of the online condition are higher than the ones
of the offline condition, in both the sortings.
On what concerns the card sorting of the animals, we
also calculated the number of correct classifications
made by each participant. The mean correct
classification for the online (Netsorting) group was
calculated among those participants who classified at
least 54 items. The selected participants were 174.
The mean of correct classifications where different for
the two groups: online: 54.04, offline: 49.87. The
difference reached significance: F (1,232) = 6.114, p =
0.014.
Conclusions
The results of our web based card sorting are as good as
those obtained with the traditional paper card sorting
version. Increasing the number of participants improves
the quality of the sorting. Recruiting participants using
an online software is by far easier than using the paper
card sorting; furthermore, no data entry is needed using
a tool like Netsorting. The use of such a tool allows
therefore to test a greater number of participants, with
lower costs; as a result, better classifications can be
obtained.
REFERENCES
IAWiki: Cardsorting. Available as
http://www.iawiki.net/CardSorting
Rugg, G. and McGeorge, P. (1997). The sorting
techniques: a tutorial paper on card sortis, picture
sorts and item sorts. Expert Systems, 14(2):80 – 93.
Tullis, T. and Wood, L. (2004). How many users are
enough for a card-sorting study? In Proceedings
UPA’2004, Minneapolis, MN.
... In addition, the card-sort in Study 1 was conducted online. Card-sorting was initially developed for physical cards (Conrad & Tucker, 2019) and although Bussolon et al. (2006) argue that online card-sorting is comparable to physical card-sorting, the differences in procedure may have influenced the outcomes. Therefore, we conducted a second study using physical cards and a larger sample to investigate robustness of the outcomes of Study 1. ...
... Comparing the outcomes of the online card-sorting (Study 1) with the offline card-sorting (Study 2), we support the notion that online and offline methods are comparable for large card-sorting tasks (Bussolon et al., 2006). Personal observations of the interviewer indicate that the task of sorting 80 cards was substantial, but the burden was manageable. ...
Article
Full-text available
Replacing animal-derived proteins with plant-based proteins has environmental and health benefits. Despite increasing consumer acceptance of plant-based proteins, most individuals do not frequently consume them. Understanding how different consumer groups perceive and categorise plant-based proteins in comparison to animal-derived proteins can support the protein transition, as it might provide insights into both consumer acceptance and how to position plant-based proteins. Based on categorisation theory, we investigated how different consumers – omnivores, flexitarians, vegetarians and vegans – categorise various sources of proteins and to what extent they use taxonomic, goal-derived or cross-categorisation approaches. 121 Dutch participants free-sorted 80 product cards (30 plant-based proteins, 20 animal-derived proteins, 5 hybrids (animal-plant) and 25 non-protein products). Forty participants elaborated on their categorisations in follow-up interviews. Our findings show that similar strategies can result in different categories depending on consumer group. Taxonomic categorisation strategies are dominantly applied by all consumer groups, but specific categories differ. With decreasing animal protein consumption, omnivores, flexitarians, vegetarians and vegans become increasingly strict in their categorisations. Omnivores do not separate proteins as strictly as flexitarians, vegetarians and vegans. All groups separate animal-derived meat from plant-based meat alternatives, but hybrid meat is ambiguous for omnivores and flexitarians. Variations in categorisations of plant-based proteins between groups give directions to marketers on how to tailor positioning of these products in a way that consumers identify and adopt plant-based proteins, to guide and accelerate the protein transition.
... We designed a card sorting task [4,18] to explore attitudes towards 16 hypothetical approaches of increasing a VR user's awareness of reality whilst in VR. We opted for a card sorting task to capture both quantitative and qualitative data on participants' preferences. ...
... Due to COVID-19 restrictions in our local area we conducted the study remotely (which prior work has established is valid for card sorting tasks [4]). Zoom [39], a videotelephony service, was used to meet with the participants and Trello [1], an online collaboration tool which allows users to sort labelled cards into categories, was used to perform the sorting task ( Figure 1). ...
... Writing notes in a shared workspace helps both the individual and the group to brainstorm ideas, while also providing an overview of all notes, with existing notes providing inspiration for new ideas (Aiken et al., 1996;Lockton et al., 2016;Michinov & Jeanson, 2021;Wilson, 2006). Grouping notes with similar themes together can highlight similarities and differences between individual notes, similar to a card-sorting task (Bussolon et al., 2006). These notes would then be visible on the whiteboard, allowing the facilitator and experts to further discuss the evolving themes. ...
Thesis
Full-text available
Comfort is an important factor that affects user acceptance and the subsequent uptake of automated vehicles (AVs). In highly and fully automated driving, the transition of control from drivers to the automation system transforms the role of onboard users from active drivers to passive riders. This transition removes the need to control the vehicle and monitor the environment, which allows users to engage in non-driving-related activities. This, in turn, makes it difficult for users to predict the vehicle’s manoeuvres, which potentially challenges user comfort. Evidence suggests that designing AVs’ driving styles in certain ways, such as mimicking users’ manual driving styles, may affect user comfort. However, our knowledge about the influences of AVs’ driving styles on user comfort is limited. There also remains a significant gap in understanding the complexities of the concept of user comfort in automated driving. Addressing these research gaps is crucial for a comprehensive understanding of user comfort in automated driving and improving cross-study comparability. This thesis aims to investigate user comfort in highly automated driving, and how different driving styles of AVs affect comfort. The research examined a) users’ subjective evaluations of different driving styles, b) the relationship between objective vehicle metrics and subjective evaluations, and c) a conceptual model explaining how driving styles affect user comfort, involving related concepts and factors. This thesis adopted a mixed-method approach. Based on a driving simulator experiment, quantitative methods were used to understand users’ subjective preferences for human-like versus non-human-like driving styles and the effect of vehicle metrics on such subjective evaluations. Based on a focus group workshop with experts, qualitative methods were used to establish a conceptual model of user comfort. The quantitative exploration showed that two representative human-like driving styles (defensive and aggressive) were perceived as more comfortable and natural than the non-human-like, robotic, driving style. Particularly, the defensive one was rated as the most comfortable, by both low and high sensation seekers, especially for more challenging roads. Results further showed that several lateral and rotational kinematics of the vehicle were significantly associated with both comfort and naturalness evaluations, while only one longitudinal factor was associated with comfort. Results also suggested that enhancing the human-likeness of automated driving by aligning it with users’ manual driving, in terms of several vehicle metrics like speed, could improve user comfort and naturalness. However, it also noted that such human-like patterns in lateral jerk might adversely affect evaluations. The qualitative study found a range of aspects related to comfort in automated driving, such as physical comfort, design expectations, and pleasantness. Several aspects of discomfort were also identified, which differ from those associated with comfort. The study further led to the development of a conceptual framework. The framework explains how AVs’ driving styles, as well as other non-driving-related factors, affect user comfort in automated driving. It incorporates a range of concepts, such as trust, naturalness, expectations, and privacy concerns. This thesis contributes to a better understanding of user comfort in automated driving, empirically and theoretically. It clarifies the effect of driving styles on user comfort from both subjective and objective perspectives. Moreover, it reveals the multifaceted nature of the concept of user comfort in automated driving. The implications drawn from this work provide design guidelines to assist in the development of more comfortable, pleasant, and acceptable automated vehicles for users.
... Writing notes in a shared workspace helps both the individual and the group to brainstorm ideas, while also providing an overview of all notes, with existing notes providing inspiration for new ideas (Aiken et al., 1996;Lockton et al., 2016;Michinov & Jeanson, 2021;Wilson, 2006). Grouping notes with similar themes together can highlight similarities and differences between individual notes, similar to a card-sorting task (Bussolon et al., 2006). These notes would then be visible on the whiteboard, allowing the facilitator and experts to further discuss the evolving themes. ...
Article
Full-text available
The driving style of an automated vehicle (AV) needs to be comfortable to encourage the broad acceptance and use of this newly emerging transport mode. However, current research provides limited knowledge about what influences comfort, how this concept is described, and how it is measured. This knowledge is especially lacking when comfort is linked to the AV’s driving styles. This paper presents results from an online workshop with nine experts, all with hands-on experience of AVs and a long track record of research in this context. Using online tools, experts were invited to introduce concepts they considered relevant to comfort/discomfort in currently available modes of transport which offer a ride (taxi/bus/train) to users and compare these to the concepts used to define comfort and discomfort in AVs. Results showed that a wide range of terms were used to describe user comfort and discomfort for both modes. Although all terms used for existing vehicles were found to apply to AVs, additional terms were proposed for determining comfort/discomfort of AVs. For example, to enhance comfort in AVs, designers should consider good communication channels, as well as ensuring that the AV’s capabilities match users’ expectations. Results also revealed that more terms were used, overall, to define discomfort, and that a comfortable ride in AVs is not just about mitigating discomfort. New concepts specific to AVs were also revealed when considering what increases their discomfort, such as whether riders’ safety and privacy are affected, or if they feel in control. Experts’ input from the workshop was used to enhance and expand a simple conceptual framework, explaining how AV driving styles, as well as other, non-driving-related factors, affect user comfort. It is hoped that this framework provides a more comprehensive list of the concepts affecting user comfort, also allowing more accurate measurement of the concept. As well as allowing for a more accurate comparison between empirical studies measuring comfort in AVs, this study will facilitate the design of more comfortable and acceptable automated driving for future vehicles.
... La técnica se puede realizar manualmente, mediante tarjetas de papel o cartón, pero también mediante programas de software de fácil uso [17,18,19]. La ventaja del sistema manual con tarjetas de papel en una mesa es su naturalidad y es más intuitivo para los usuarios que utilizar un programa. ...
Article
Full-text available
Software engineers can use various techniques to elicit relevant information from stakeholders to specify the requirements that a software-intensive product must satisfy. However, few are aware of the differences and advantages they have from each other. This article presents an exploratory study of the vision that requirements engineers have on twelve requirements elicitation techniques. An empirical study has been carried out using two classification techniques: card sorting and repertory grid. The perceptions obtained with each technique were compared, showing that there are differences and coincidences in the visions captured by each technique. This study can help guide training objectives on software requirements and conduct future empirical studies that show the adequacy of each elicitation technique.
... Typically, participants are given a series of cards to sort based on the underlying principle or similarities of solution (Chi, Feltovich, and Glaser 1981, 124). The sorting itself may be done in person through one-on-one interviews (Chi, Feltovich, and Glaser 1981), as an in-class activity (Smith et al. 2013, 630), or through an online platform (Lapierre and Flynn 2020), with no clear bias observed between physical or virtual sorting (Bussolon, Russi, and Missier 2006). It is often specified that participants are not to solve the problems, just to sort them, with emphasis that this is not a test and there is no incorrect way to sort them. ...
Article
Full-text available
Universities are recognizing the need to prepare graduates to think conceptually and have the ability to take on complex, real-world problems. Strategies to assess conceptual knowledge are limited and often require more time and effort to complete than is accessible for most undergraduate courses. Card sorting is a very broad technique for understanding how people group concepts, but in higher education has typically been used to show a student’s development towards expert-like thinking in a discipline as a whole. However, it typically does not give much insight into how we should change our teaching. In this paper, using the novel setting of two terms of a first-year, earth and ocean science lab that uses problem-based learning (PBL), we show how one can generate a card sort that is built using course learning goals and then use the analysis to make actionable improvements to course instruction. Using a card sort designed so that the expert sort corresponds to learning goals supported by the lab activities, we found that in both offerings of the course students generally moved towards expert-like sorting with a reduction in novice-like sorting. A striking feature stood out in both terms of the course, with one question scoring significantly lower than any other expert pairings, despite a change in the wording of that question between terms. This suggests that our course materials do not promote this specific conceptual connection that we had expected and gives us a clear place to look for issues in our course material. In a broader context, our results suggest that tailoring card sort questions to material at a course level, rather than at the discipline level, can provide a manageable, routine assessment of conceptual knowledge in students, while also providing feedback on the quality of course materials.
Article
Full-text available
Handheld devices such as smartphones, tablets, and PDAs are commonly used for online shopping. In order to satisfy web page requests on these devices, web developers are using a responsive web design approach, where websites dynamically adapt to different screen sizes and orientations. One of the challenges faced by web developers is to provide a usable experience across different devices. Motivated by this, a mobile-first usability guideline was established by extracting good usability practices from five popular responsive e-commerce websites. Validation of the usability guideline was conducted, and results indicate that it will have a positive impact on the overall user experience.
Chapter
Card sorting is a popular way for creating website information architectures based on users’ mental models. This paper explores the effect of participants’ self-efficacy on card sorting results. A two-phase study was carried out. The first phase involved 40 participants rating their self-efficacy on a standardized scale, followed by an open card sort experiment. The median self-efficacy score was used to split the open card sort data into two groups: one for low and one for high participants’ self-efficacy. These two datasets were analyzed following state-of-the-art techniques for open card sort data analysis, which resulted in two information architectures for the eshop. In the second phase, two functional prototypes were first created for the eshop, one for each information architecture of the first phase. Subsequently, 30 participants interacted with both prototypes in a user testing study. This paper found that users interacting with the information architecture produced by open card sort participants with low self-efficacy made statistically significantly more correct first clicks, significantly less time to find content items, rated the tasks as significantly easier, and provided higher perceived usability ratings compared to when they interacted with the information architecture produced by users with high self-efficacy.
Chapter
How users interact with an intelligent system is determined by their subjective mental model of the system’s inner working. In this paper, we present a novel method based on card sorting to identify such mental models of recommender systems quantitatively. Using this method, we conducted an online study (N ⁣= ⁣170N \!=\!170). Applying hierarchical clustering to the results revealed distinct user groups and their respective mental models. Independent of the recommender system used, some participants held a strict procedural-based, others a concept-based mental model. Additionally, mental models can be characterized as either technical or humanized. While procedural-based mental models were positively related to transparency perception, humanized models might influence the perception of system trust. Based on these findings, we derive three implications for the consideration of user-specific mental models in the design of transparent intelligent systems.
Conference Paper
Full-text available
A study was conducted to assess the minimum number of participants needed for a card-sorting study. Similarity matrices and tree structures from various sample sizes were compared to those based on a set of 168 participants. Results indicate that reasonable structures are obtained from 20-30 participants.
Article
Although sorting techniques (e.g. card sorts) are widely used in knowledge acquisition and requirements acquisition, they have received little formal attention compared to related techniques such as repertory grids and laddering. This paper briefly describes the main sorting techniques, and then provides a detailed tutorial on one variety (repeated single-criterion sorts), using a worked example. Guidelines for choice and sequencing of techniques are given, both in relation to varieties of sorting technique and in relation to other techniques. It is concluded that the sorting techniques are a valuable part of the elicitor's methodological toolkit.