Content uploaded by Thomas Tullis
Author content
All content in this area was uploaded by Thomas Tullis
Content may be subject to copyright.
UPA 2004 Presentation—Page 1
A Comparison of Questionnaires for Assessing Website Usability
Thomas S. Tullis and Jacqueline N. Stetson
Human Interface Design Department, Fidelity Center for Applied Technology
Fidelity Investments
82 Devonshire St., V4A
Boston, MA 02109
Contact: tom.tullis@fidelity.com
ABSTRACT:
Five questionnaires for assessing the usability of a website were compared in a study with
123 participants. The questionnaires studied were SUS, QUIS, CSUQ, a variant of
Microsoft’s Product Reaction Cards, and one that we have used in our Usability Lab for
several years. Each participant performed two tasks on each of two websites:
finance.yahoo.com and kiplinger.com. All five questionnaires revealed that one site was
significantly preferred over the other. The data were analyzed to determine what the
results would have been at different sample sizes from 6 to 14. At a sample size of 6, only
30-40% of the samples would have identified that one of the sites was significantly
preferred. Most of the data reach an apparent asymptote at a sample size of 12, where two
of the questionnaires (SUS and CSUQ) yielded the same conclusion as the full dataset at
least 90% of the time.
Introduction
A variety of questionnaires have been used and reported in the literature for assessing the
perceived usability of interactive systems, including QUIS [3], SUS [2], CSUQ [4], and
Microsoft’s Product Reaction Cards [1]. (See [5] for an overview.) In our Usability Lab, we
have been using our own questionnaire for the past several years for assessing subjective
reactions that participants in a usability test had to a web site. However, we had concerns
about the reliability of our questionnaire (and others) given the relatively small number of
participants in most typical usability tests. Consequently, we decided to conduct a study to
determine the effectiveness of some of the standard questionnaires, plus our own, at
various sample sizes. Our focus was specifically on websites.
Method
We decided to limit ourselves to our own questionnaire plus those in the published literature
that we believed could be adapted to evaluating websites. The questionnaires we used were
as follows (illustrated in Appendix A):
1. SUS (System Usability Scale)—This questionnaire, developed at Digital Equipment
Corp., consists of ten questions. It was adapted by replacing the word “system” in
every question with “website”. Each question is a statement and a rating on a five-
point scale of “Strongly Disagree” to “Strongly Agree”.
2. QUIS (Questionnaire for User Interface Satisfaction)—The original questionnaire,
developed at the University of Maryland, was composed of 27 questions. We
dropped three that did not seem to be appropriate to websites (e.g., “Remembering
names and use of commands”). The term “system” was replaced by “website”, and
the term “screen” was generally replaced by “web page”. Each question is a rating
on a ten-point scale with appropriate anchors at each end (e.g., “Overall Reaction to
the Website: Terrible … Wonderful”).
3. CSUQ (Computer System Usability Questionnaire)—This questionnaire, developed at
IBM, is composed of 19 questions. The term “system” or “computer system” was
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 2
replaced by “website”. Each question is a statement and a rating on a seven-point
scale of “Strongly Disagree” to “Strongly Agree”.
4. Words (adapted from Microsoft’s Product Reaction Cards)—This questionnaire is
based on the 118 words used by Microsoft on their Product Reaction Cards [1]. (We
are grateful to Joey Benedek and Trish Miner of Microsoft for providing the complete
list.) Each word was presented with a check-box and the user was asked to choose
the words that best describe their interaction with the website. They were free to
choose as many or as few words as they wished.
5. Our Questionnaire—This is one that we have been using for several years in usability
tests of websites. It is composed of nine statements (e.g., “This website is visually
appealing”) to which the user responds on a seven-point scale from “Strongly
Disagree” to “Strongly Agree”. The points of the scale are numbered -3, -2, -1, 0, 1,
2, 3. Thus, there is an obvious neutral point at 0.
Note that other tools designed as commercial services for evaluating website usability (e.g.,
WAMMI [6], RelevantView [7], NetRaker [8], Vividence [9]) were not included in this study.
Some of these tools use their own proprietary questionnaires and some allow for the
construction of your own.
The entire study was conducted online via our company’s Intranet. A total of 123 of our
employees participated in the study. Each participant was randomly assigned to one of the
five questionnaire conditions. Each was asked to perform two tasks on each of two well-
known personal financial information sites: finance.Yahoo.com and Kiplinger.com. (In the
rest of this paper they will simply be referred to as Site 1 and Site 2. No relationship
between the site numbers and site names should be assumed.) The two tasks were as
follows:
1. Find the highest price in the past year for a share of <company name>. (Note that a
different company was used in each task.)
2. Find the mutual fund with the highest 3-year return.
The order of presentation of the two sites was randomized so that approximately half of the
participants received Site 1 first and half received Site 2 first. After completing (or at least
attempting) the two tasks on a site, the user was presented with the questionnaire for their
randomly selected condition. Thus, each user completed the same questionnaire for the
two sites. (Technically, “questionnaires” was a between-subjects variable and “sites” was a
within-subjects variable.)
Data Analysis
For each participant, an overall score was calculated for each website by simply averaging
all of the ratings on the questionnaire that was used. (All scales had been coded internally
so that the “better” end corresponded to higher numbers.) Since the various questionnaires
use different scales, these were converted to percentages by dividing each score by the
maximum score possible on that scale. So, for example, a rating of 3 on SUS was
converted to a percentage by dividing that by 5 (the maximum score for SUS), giving a
percentage of 60%.
Special treatment was required for the “Words” condition since it did not involve rating
scales. Before the study, we classified each of the words as being “Positive” (e.g.,
“Convenient”) or “Negative” (e.g., “Unattractive”). (Note that they were not grouped or
identified as such to the participants.) For each participant, an overall score was calculated
by counting the total number of words that person selected and then dividing that number
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 3
into the number of those words that were “Positive”. Thus, if someone selected 8 positive
words and 10 words total, that yielded a score of 80%.
Results
The random assignment of participants to the questionnaire conditions yielded between 19
and 28 participants for each questionnaire. The frequency distributions of their ratings on
each questionnaire for each site, converted to percentages as described above, are shown in
Figures 1 through 5. Figure 6 shows the average scores for each site using each
questionnaire.
SUS
0
10
20
30
40
50
60
20% 40% 60% 80% 100%
Percentage of Maximum Rating
Frequency
Site 1
Site 2
Figure 1. Results using SUS.
QUIS
0
20
40
60
80
100
120
140
160
180
200
20% 40% 60% 80% 100%
Percentage of Maximum Rating
Frequency
Site 1
Site 2
Figure 2. Results using QUIS.
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 4
CSUQ
0
20
40
60
80
100
120
140
160
20% 40% 60% 80% 100%
Percentage of Maximum Rating
Frequency
Site 1
Site 2
Figure 3. Results using CSUQ.
Survey 4: Words
0
2
4
6
8
10
12
20% 40% 60% 80% 100%
Percentage of Maximum Score
Frequency
Site 1
Site 2
Figure 4. Results using Microsoft’s Words
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 5
Our Questionnaire
0
20
40
60
80
100
120
20% 40% 60% 80% 100%
Percentage of Maximum Score
Frequency
Site 1
Site 2
Figure 5. Results using our questionnaire.
Comparison of Means
73% 74% 74% 72%
50% 48% 48%
38%
52%
66%
0%
10%
20%
30%
40%
50%
60%
70%
80%
SUS QUIS CSUQ Words Ours
Survey
Mean Score
Site 1
Site 2
Figure 6. Comparison of mean scores for each site using each questionnaire.
All five questionnaires showed that Site 1 was significantly preferred over Site 2 (p<.01 via
t-test for each). The largest mean difference (74% vs. 38%) was found using the Words
questionnaire, but this was also the questionnaire that yielded the greatest variability in the
responses. Both of these points are apparent from examination of Figure 4, where you can
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 6
see that the modal values for the two sites are at the opposite ends of the scale, but there
are some responses for both sites across the entire range.
The most interesting thing to look at now is what the results would have been using each
questionnaire if the study had been done with a smaller number of participants. We chose
to analyze randomly selected sub-samples of the data at size 6, 8, 10, 12, and 14. We felt
these samples represented sizes commonly used in usability tests. This was an empirical
sub-sampling in which 20 random sub-samples were taken from the full dataset at each of
these different sample sizes, and a t-test was conducted to determine whether the results
showed that Site 1 was significantly better than Site 2 (the conclusion from the full
dataset). Figure 7 shows the results of this random sub-sampling.
% of "Correct" Conclusions
20%
30%
40%
50%
60%
70%
80%
90%
100%
6 8 10 12 14
Sample Size
SUS
QUIS
CSUQ
Words
Ours
Figure 7. Data based on t-tests of random sub-samples of various sizes. Twenty sub-
samples were taken at each sample size for each site and each questionnaire. What is
plotted is the percentage of those 20 tests that yielded the same conclusion as the
analysis of the full dataset (that Site 1 was significantly preferred over Site 2).
As one would expect, the accuracy of the analysis increases as the sample size gets larger.
With a sample size of only 6, all of the questionnaires yield accuracy of only 30-40%,
meaning that 60-70% of the time, at that sample size, you would fail to find a significant
difference between the two sites. Interestingly, the accuracy of some of the questionnaires
increases quicker than others. For example, SUS jumps up to about 75% accuracy at a
sample size of 8, while the others stay down in the 40-55% range. It’s also interesting to
note that most of the questionnaires appear to reach an asymptote at a sample size of 12.
The improvement by going to a sample size of 14 is small in most cases. Also, due to the
different variances of the responses, some of the questionnaires reach a higher asymptote
than others. For example, SUS and CSUQ reach asymptotes of 90-100% while the others
are in the 70-75% range. Of course, the other questionnaires would have continued to
yield improvement if larger samples had been tested.
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 7
Conclusions
First, some caveats need to be pointed out about the interpretation of these data. The
primary one is that they really only directly apply to the analysis of the two sites that we
studied. We selected two popular sites that provide financial information,
finance.Yahoo.com and Kiplinger.com. We chose these sites because they provide similar
kinds of information but in different ways. Had the two sites studied been even more
similar to each other, it would have been more difficult for any of the questionnaires to yield
a significant difference. Likewise, if they had been more different, it would have been easier
for any of the questionnaires to yield a significant difference.
Another caveat is that the users’ assessments of these sites were undoubtedly affected by
the two tasks that we asked them to do on those sites. Again, we did not choose tasks that
we thought would be particularly easier or more difficult on one site vs. the other. We
chose tasks that we thought were typical of the tasks people might want to do on these
kinds of sites.
It’s also possible that the results could have been somewhat different if we had been able to
collect data from more participants using each questionnaire. The minimum number of
participants that we got for any one questionnaire was 19. Some researchers have argued
that still larger numbers of participants are needed to get reliable data from some of these
questionnaires. While that may be true, one of our goals was to study whether any of these
questionnaires yield reliable results at the smaller sample sizes typically seen in usability
tests.
Finally, this paper has only addressed the question of whether a given questionnaire was
able to reliably distinguish between the ratings of one site vs. the other. In many usability
tests, you have only one design that you are evaluating, not two or more that you are
comparing. When evaluating only one design, possibly the most important information is
related to the diagnostic value of the data you get from the questionnaire. In other words,
how well does it help guide improvements to the design? That has not been analyzed in this
study. Interestingly, on the surface at least, it appears that the Microsoft Words might
provide the most diagnostic information, due to the potentially large number of descriptors
involved.
Keeping all of those caveats in mind, it is interesting to note that one of the simplest
questionnaires studied, SUS (with only 10 rating scales), yielded among the most reliable
results across sample sizes. It is also interesting that SUS is the only questionnaire of those
studied whose questions all address different aspects of the user’s reaction to the website
as a whole (e.g., “I found the website unnecessarily complex”, “I felt very confident using
the website”) as opposed to asking the user to assess specific features of the website (e.g.,
visual appearance, organization of information, etc). These results also indicate that, for
the conditions of this study, sample sizes of at least 12-14 participants are needed to get
reasonably reliable results.
REFERENCES
1. Benedek, J., & Miner, T. (2002). Measuring desirability: New methods for evaluating
desirability in a usability lab setting. Proceedings of UPA 2002 Conference, Orlando, FL,
July 8-12, 2002.
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 8
2. Brooke, J. (1996). SUS: A Quick and Dirty Usability Scale. In: P.W. Jordan, B. Thomas,
B.A. Weerdmeester & I.L. McClelland (Eds.), Usability Evaluation in Industry. London:
Taylor & Francis. (Also see http://www.cee.hw.ac.uk/~ph/sus.html)
3. Chin, J. P., Diehl, V. A, & Norman, K. (1988). Development of an instrument measuring
user satisfaction of the human-computer interface, Proceedings of ACM CHI '88
(Washington, DC), pp. 213-218. (Also see
http://www.acm.org/~perlman/question.cgi?form=QUIS and
http://www.lap.umd.edu/QUIS/index.html)
4. Lewis, J. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric
Evaluation and Instructions for Use. International Journal of Human-Computer
Interaction, 7 (1,) 1995, 57-78. (Also see
http://www.acm.org/~perlman/question.cgi?form=CSUQ)
5. Perlman, G. (Undated). Web-Based User Interface Evaluation with Questionnaires.
Retrieved from http://www.acm.org/~perlman/question.html on Nov. 7, 2003.
6. WAMMI: http://www.wammi.com
7. RelevantView: http://www.relevantview.com/
8. NetRaker: http://www.netraker.com/
9. Vividence: http://www.vividence.com/
Appendix A: Screenshots of the Five Questionnaires Used
SUS
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 9
QUIS
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 10
CSUQ
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 11
Words (based on Microsoft’s Product Reaction Cards)
A Comparison of Questionnaires for Assessing Website Usability
UPA 2004 Presentation—Page 12
Our Questionnaire
A Comparison of Questionnaires for Assessing Website Usability
