Content uploaded by James R. Lewis
Author content
All content in this area was uploaded by James R. Lewis on Oct 08, 2023
Content may be subject to copyright.
This article was downloaded by: [James Lewis]
On: 10 April 2013, At: 11:14
Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK
International Journal of Human-Computer Interaction
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/hihc20
Psychometric Evaluation of the T-CSUQ: The
Turkish Version of the Computer System Usability
Questionnaire
Oğuzhan Erdinç a & James R. Lewis b
a Turkish Air Force Academy, Istanbul, Turkey
b IBM Corporation Software Group, Boca Raton, Florida, USA
Accepted author version posted online: 24 Jul 2012.
To cite this article: Oğuzhan Erdinç & James R. Lewis (2013): Psychometric Evaluation of the T-CSUQ: The Turkish Version of
the Computer System Usability Questionnaire, International Journal of Human-Computer Interaction, 29:5, 319-326
To link to this article: http://dx.doi.org/10.1080/10447318.2012.711702
PLEASE SCROLL DOWN FOR ARTICLE
Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to
anyone is expressly forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents
will be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses should
be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims,
proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in
connection with or arising out of the use of this material.
Intl. Journal of Human–Computer Interaction, 29: 319–326, 2013
Copyright © Taylor & Francis Group, LLC
ISSN: 1044-7318 print / 1532-7590 online
DOI: 10.1080/10447318.2012.711702
Psychometric Evaluation of the T-CSUQ: The Turkish Version
of the Computer System Usability Questionnaire
O˘
guzhan Erdinç1and James R. Lewis2
1Turkish Air Force Academy, Istanbul, Turkey
2IBM Corporation Software Group, Boca Raton, Florida, USA
This article describes the development of a standardized
computer system usability questionnaire for use with speak-
ers of the Turkish language, the Turkish Computer System
Usability Questionnaire, Short Version (T-CSUQ-SV). This new
questionnaire, based on the English-language CSUQ, underwent
careful translation and transformation through comprehensive
psychometric evaluation. The results of the psychometric eval-
uation revealed an acceptable level of reliability, appropriate
construct validity, and sensitivity to manipulation, indicating
that Turkish usability practitioners should be able to use the
T-CSUQ-SV with confidence when conducting user research.
1. INTRODUCTION
In accordance with the mapping of usability metrics to the
construct of usability (Sauro & Lewis, 2009), usability assess-
ment of computer systems typically addresses both objective
and subjective aspects of usability. Objective aspects mea-
sure one or more elements of the performance of the user
(e.g., efficiency and effectiveness). Subjective aspects involve
perception and satisfaction of users toward the system under
study. Thus, subjective user data can aid in the identification
of usability problems and improvements to increase user sat-
isfaction. Although there are a number of English-language
instruments for the assessment of user satisfaction with com-
puter systems, there has been a need for a valid and reliable
questionnaire for the usability assessment of computer systems
for the Turkish-speaking population.
The primary purpose of this research reported in this arti-
cle was (a) to translate Lewis’s (2002) Computer System
Many thanks to Yasemin Gönenç, ¸Sule Ertu˘
grul Yüksel, Zarife
Öztürk, Vanessa Larson, and Göksel Günay for their involvement in
the translation process. Thanks also to Canay Esin, Y.Alperen Gürel,
Duygu Pekgönenç and Murak Duruk for their valuable support in data
collection. We also express our appreciation to the two anonymous
reviewers who provided helpful suggestions on the first draft of this
paper.
Address correspondence to James R. Lewis, IBM Corporation,
8051 Congress Avenue, Suite 2088, Boca Raton, FL 33487. E-mail:
jimlewis@us.ibm.com
Usability Questionnaire (CSUQ) into Turkish and (b) to
conduct a psychometric evaluation of the translated ques-
tionnaire (T-CSUQ) to develop a standardized instrument.
Standardized satisfaction measurements offer many advantages
to the usability practitioner. Specifically, they enable objectiv-
ity, replicability, quantification, economy, communication, and
scientific generalization (Nunnally, 1978).
1.1. The CSUQ
The CSUQ was originally a 19-item instrument designed for
the purpose of assessing users’ perceived satisfaction with their
computer systems in the field (Lewis, 1995), modeled directly
on the Post-Study System Usability Questionnaire (PSSUQ) for
conducting this type of assessment at the end of a lab-based
usability study. In its current form, the CSUQ (as well as the
PSSUQ) is a 16-item instrument (Lewis, 2002, 2012; Sauro &
Lewis, 2012). It had its origin in an internal IBM project called
SUMS (System Usability MetricS), headed by Suzanne Henry
in the late 1980s. The mission of SUMS was to document and
validate procedures for measuring system usability, including
performance, usability problems, and user satisfaction.
The most widely used English-language usability ques-
tionnaires are the System Usability Scale (SUS), CSUQ,
Questionnaire for User Interaction Satisfaction (QUIS), and
Software Usability Measurement Inventory (SUMI; Sauro &
Lewis, 2012). The major qualities that led to the selection
of the CSUQ for transformation into a Turkish instrument
include its
1. Desirable psychometric qualities (Lewis, 1995, 2002; for
details, see section 1.2)
2. Successful record of practical and academic applications
in its original (e.g., Bagheri & Ghorbani, 2009; Chow &
Chan, 2010; Frias-Martinez, Chen, & Liu, 2009; Kartakis &
Stephanidis, 2010; Saleem et al., 2011) and modified forms
(e.g., Periera et al., 2012)
3. Continuing relevance to current researchers—although
developed in the late 1980s and first published in the early
1990s, the CSUQ has been cited in more than 60 published
319
Downloaded by [James Lewis] at 11:14 10 April 2013
320 O. ERDINÇ AND J. R. LEWIS
research papers since 2010 by researchers in North America,
Europe, Asia, Australia, and Africa
4. Applicability in usability evaluation of computer sys-
tems in various areas including medicine (Saleem et al.,
2011), inclusive e-learning systems (Savidis, Grammenos, &
Stephanidis, 2006), clinical learning systems (Chow & Chan,
2010), and 3D land visualization systems (van Lammeren,
Houtkamp, Colijn, Hilferink, & Bouwman, 2010), as well as
in research into the measurement of the construct of usability
(e.g., Christophersen & Konradt, 2011; Finstad, 2010;
MacDorman, Whalen, Ho, & Patel, 2011; McNamara &
Kirakowski, 2011)
5. Intentional multidimensional structure (Bagheri &
Ghorbani, 2009; Chow & Chan, 2010; Kartakis &
Stephanidis, 2010; Sauro & Lewis, 2012; van Lammeren
et al., 2010; in contrast to SUS)
6. Noncommercial use, which enables researchers and practi-
tioners to perform cost efficient analyses (in contrast to QUIS
and SUMI)
Furthermore, in a study of the sensitivity of five methods
used to assess satisfaction with usability (SUS, QUIS, CSUQ,
Product Reaction Cards, and a Fidelity questionnaire), Tullis
and Stetson (2004) examined randomly selected subsets of data
from evaluations of two websites and found the SUS was the
fastest to converge on the final (correct) conclusion regarding
which website was more usable, reaching 75% agreement at a
sample size of 8 and 100% agreement when n=12. The CSUQ
was the second fastest, reaching 75% agreement at a sample
size of 10 and 90% agreement when n=12. In contrast, even
when n=14, the other methods were in the low- to mid-70% of
agreement with the correct decision. Thus, for unidimensional
measurement, the SUS appears to be an excellent instrument.
For studies that would benefit from multidimensional assess-
ment of usability, however, the data indicate that practitioners
should consider the CSUQ.
1.2. Brief Review of Psychometric Theory
The goal of psychometrics is to establish the quality of psy-
chological measures (Nunnally, 1978). Is a measure reliable
(consistent)? Given a reliable measure, is it valid (measures the
intended attribute)? Finally, is the measure appropriately sen-
sitive to experimental manipulations? Here is a brief review of
some basic elements of standard psychometric practice.
Reliability goals. In psychometrics, reliability is quanti-
fied consistency, typically estimated using coefficient alpha
(Nunnally, 1978). Coefficient alpha can range from 0 (no relia-
bility)to1(perfect reliability). Measures of individual aptitude
(such as IQ tests or college entrance exams) should have a
minimum reliability of .90 (preferably a reliability of .95). For
other research or evaluation, measurement reliability should be
at least .70 (Landauer, 1997).
Validity goals. Validity is the measurement of the extent
to which a questionnaire measures what it claims to measure.
Researchers commonly use the Pearson correlation coefficient
to assess criterion-related validity (the relationship between
the measure of interest and a different concurrent or predic-
tive measure). Moderate correlations (with absolute values as
small as .30 to .40) are often large enough to justify the use of
psychometric instruments (Nunnally, 1978).
Another approach is to assess a questionnaire’s construct
validity, usually done with factor analysis. Factor analysis is a
statistical procedure that examines the correlations among vari-
ables to discover clusters of related variables (Nunnally, 1978).
Because summated (Likert) scales are more reliable than single-
item scales (Nunnally, 1978) and it is easier to present and
interpret a smaller number of scores, it is common to conduct a
factor analysis to determine if there is a statistical basis for the
formation of summative scales.
Sensitivity goals
A questionnaire that is reliable and valid should also
be sensitive—capable of detecting appropriate differences.
Statistically significant differences in the magnitudes of ques-
tionnaire scores for different systems or other usability-related
manipulations provide evidence for sensitivity.
1.3. Previous Psychometric Evaluations of the CSUQ
Previous psychometric evaluations (Lewis, 1991, 1992a,
1992b, 1995) indicated that the items used for the CSUQ
and PSSUQ produced a very reliable overall composite score
and had three reliable factors (all coefficient alphas greater
than .80). The labels assigned to these factors were System
Usefulness (SysUse), Information Quality (InfoQual), and
Interface Quality (IntQual).
Investigations into scale validity found that the overall score
correlated highly with other measures of user satisfaction taken
after each scenario, and the overall score, SysUse, and IntQual
all correlated significantly with the percentage of successful
scenario completion. Sensitivity analyses indicated that the
scales respond appropriately to manipulations of system and
user groups.
An analysis of 5 years of lab-based use of the CSUQ
and PSSUQ replicated the previous findings for consistently
high reliability and consistent construct validity (Lewis, 2002).
Analyses of variance to assess sensitivity indicated that vari-
ables such as the study, developer, stage of development, type
of product, and type of evaluation significantly affected scores.
Other variables, such as gender and completeness of responses
to the questionnaire, did not. Item analysis indicated that three
items (3, 5, and 13 in the original numbering) contributed rel-
atively little to its reliability, leading to the recommendation
to remove those items and use the resulting shorter 16-item
version.
Downloaded by [James Lewis] at 11:14 10 April 2013
PSYCHOMETRIC EVALUATION OF THE T-CSUQ 321
2. METHOD
2.1. Translation of the CSUQ Into Turkish
A multidisciplinary research group of reviewers and lan-
guage professionals adapted the full CSUQ (19 items) into
Turkish. The group included two native Turkish reviewers and
two bilingual reviewers, a native Turkish language expert, and
the first author, who was the group’s usability expert and
who coordinated the translation process. To initiate the pro-
cess, the first author wrote a draft Turkish version of the
CSUQ from a usability perspective. Next, the research group
reviewed the original and the draft Turkish versions through
a three-stage review process, modifying the Turkish version
to keep the meanings of the items as close as possible to the
original.
First stage. The native Turkish and bilingual reviewers
independently examined the original and the draft Turkish ver-
sions of the CSUQ. They assessed the semantic, idiomatic, and
conceptual equivalence of items and identified translation prob-
lems such as discrepancies and ambiguities among items. The
bilingual reviewers further examined if items in the Turkish ver-
sion reflected and covered the content of items in the original
version.
Among the foremost issues addressed was the similarity
between Turkish synonyms of the terms: “effectively,” “effi-
ciently,” and “productive” used in Items 3, 5 and 8, respectively.
The terms used to express effectiveness, efficiency, and pro-
ductivity in Turkish (“etken,” “etkin,” and “verimli”), used
interchangeably in spoken Turkish, were not sufficiently dis-
tinctive. The greatest similarity was between Turkish synonyms
of “efficiently” and “productive.” The word “verimli” fits “effi-
ciently” in that in a usability context it means accomplishing
tasks with minimum resources. Another word, “üretken,” which
is similar to “verimli,” was used in Item 8 for “productive”
because it implies producing more outcomes during work. The
other issues mostly involved selection of the best expression and
the most suitable form of the verbs. In total, this review stage led
to the modification of 14 items.
Second stage. The version of the T-CSUQ from the first
stage was adapted for usability assessment of web-based course
management software used in a state university. Twelve par-
ticipants (10 students and two instructors) independently com-
pleted the T-CSUQ and provided comments to indicate any
problematic terms and expressions. Also, a statistician unfamil-
iar with the software assessed the understandability of the items.
None of the participants reported any major problems with the
clarity of the items.
The Turkish language expert reviewed this tested version
of the T-CSUQ. He assessed the compatibility of the items
with the spoken Turkish language and proposed modifications
toward the selection of the best expression among alternatives.
Finally, the two native Turkish reviewers reassessed the word-
ing of the items independently, with both of them suggesting
minor modifications.
Third stage. The first author used the modifications sug-
gested by the second-stage participants, the Turkish language
expert, and the native Turkish reviewers to revise the T-
CSUQ. The primary modification, suggested by the Turkish
language expert, was replacement of the Turkish synonym of
“to complete” (“tamamlamak”) in Items 3,4,5 and 14 with the
Turkish synonym of “to do” (“yapmak”). Although the for-
mer is understandable, the latter is a better conceptual choice
in Turkish. Another significant modification was to replace
“güzel” (initially used for “pleasant” in Item 16) with “be˘
gen-
mek” to connote likeability rather than beauty—a more suitable
adjective.
The author communicated all modifications and the revised
version of the T-CSUQ to the research group. Each member of
the research group independently reviewed the original, tested,
and revised versions. The research group found the modifica-
tions suitable and reached a consensus about equivalence and
applicability of the items for the final version of the T-CSUQ.
Table 1 shows the original English and final Turkish versions of
the items.
2.2. Psychometric Evaluation of the T-CSUQ
Even with careful translation of items, there is no guarantee
that the T-CSUQ would have the same psychometric proper-
ties as the English version (van de Vijver & Leung, 2001).
Thus, the next step was to conduct a psychometric evaluation of
the T-CSUQ. This evaluation included assessment of reliability,
validity, sensitivity, and factor structure for long (19 items) and
short (16 items) versions of the questionnaire.
The recommended minimum sample size for psychometric
analysis (specifically for factor analysis) is to have at least five
participants per item, which, for the 19-item CSUQ, would be
95 participants (Nunnally, 1978). The evaluation process started
by inviting the voluntary participation of 120 Turkish employ-
ees and managers who used document management software
to prepare, sign, and circulate official documents (memos) in a
state institution. Participants evaluated this software using the
T-CSUQ and had the option to indicate problematic terms and
expressions. The return rate was 88.3% (106 questionnaires).
Of these, nine questionnaires had missing or invalid data, so the
final sample size was 97 (87 male, 10 female), achieving the
desired minimum sample size.
The age of the participants ranged between 22 and 42 years
with a mean of 30.54 (SD =4.99). Using 5-point scales, par-
ticipants also indicated their typical frequency of use of the
system (1 =less than one time per day, n=56; 2 =one to
five times per day, n=8; 3 =six to 10 times per day, n=8;
>10 times per day, n=5; continuous use, n=17) and dura-
tion of daily use (1 =0–2 hr, n=58; 2 =2–4 hr, n=13;
4–6 hr, n=6; 6–8 hr, n=12; >8 hr per day, n=4). Most
participants (n=63) used the system to prepare documents; the
remainder (n=34) used the system to read, initialize, or sign
the documents.
Downloaded by [James Lewis] at 11:14 10 April 2013
322 O. ERDINÇ AND J. R. LEWIS
TABLE 1
Items of the CSUQ and Their Translation Into Turkish
Original Version of CSUQ Turkish Version of CSUQ
1. Overall, I am satisfied with how easy it is to use this
system.
1. Genel olarak, sistemin kullanım kolaylı˘
gından memnunum.
2. It is simple to use this system. 2. Sistemi kullanmak basittir.
3. I can effectively complete my work using this system. 3. Sistemi kullanarak i¸slerimi etkin bir ¸sekilde yapabiliyorum.
4. I am able to complete my work quickly using this system. 4. Sistemi kullanarak i¸slerimi hızlı bir ¸sekilde yapabiliyorum.
5. I am able to efficiently complete my work using this
system.
5. Sistemi kullanarak i¸slerimi verimli bir ¸sekilde
yapabiliyorum.
6. I feel comfortable using this system. 6. Sistemi rahatlıkla kullanabiliyorum.
7. It was easy to learn to use this system. 7. Sistemi kullanmayı ö˘
grenmem kolay oldu.
8. I believe I became productive quickly using this system. 8. Sistemi kullanarak kısa zamanda üretken hale geldi˘
gime
inanıyorum.
9. The system gives error messages that clearly tell me how to
fix problems.
9. Sistemin verdi˘
gi hata mesajları, problemleri nasıl
giderece˘
gimi açıkça anlatmaktadır.
10. Whenever I make a mistake using the system, I recover
easily and quickly.
10. Sistemi kullanırken yaptı˘
gım hataları, kolay ve hızlı bir
¸sekilde düzeltebiliyorum.
11. The information (such as on-line help, on-screen messages
and other documentation) provided with this system is
clear.
11. Sistemin verdi˘
gi bilgiler (çevrim-içi yardım, ekran
mesajları, di˘
ger bilgiler, vb.) açık ve nettir.
12. It is easy to find the information I need. 12. Sistemde ihtiyaç duydu˘
gum bilgilere ula¸smak kolaydır.
13. The information provided with the system is easy to
understand.
13. Sistemin verdi˘
gi bilgiler kolayca anla¸sılmaktadır.
14. The information is effective in helping me complete my
work.
14. Sistemin verdi˘
gi bilgiler i¸slerimi yapmama yardımcı
olmaktadır.
15. The organization of information on the system screens is
clear.
15. Sistemin ekranlarındaki bilgiler açık ve anla¸sılır biçimde
düzenlenmi¸stir.
16. The interface of this system is pleasant. 16. Sistemin arayüzünü be˘
gendim.
17. I like using the interface of this system. 17. Sistemin arayüzünü kullanmak ho¸suma gidiyor.
18. This system has all the functions and capabilities I expect
it to have.
18. Sistem, bekledi˘
gim bütün i¸slevlere sahiptir ve yeterlidir.
19. Overall, I am satisfied with this system. 19. Genel olarak sistem tatmin edicidir.
The anchors: The anchors:
Strongly Strongly Kesinlikle Kesinlikle
agree 1234567disagree katılıyorum 1 2 34567katılmıyorum
Note. CSUQ =Computer System Usability Questionnaire.
3. RESULTS
3.1. Factor Analyses
The purpose of the first factor analysis was to investigate
the correspondence with previous evaluations of the English
CSUQ, which, in addition to the Overall score (Items 1–19),
has three factors:
• System Usefulness: Items 1–8
• Information Quality: Items 9–15
• Interface Quality: Items 16–18
Table 2 shows the varimax-rotated three-factor solution for
all 19 items, and Table 3 shows the same type of solution for
the short version (16 items) that corresponds to the short version
of the English CSUQ. Although the item-factor alignment for
these two versions of the T-CSUQ was similar to that for the
English CSUQ, the structures were not identical. For System
Usefulness, seven of eight items aligned as expected (the excep-
tion was Item 4). The three Interface Quality items loaded most
strongly on the same factor. The alignment discrepancies were
especially notable for Information Quality.
To create a short version of the CSUQ-T with item-factor
alignment closer to that of the English CSUQ and using the
results shown in Table 3 as a guide, we conducted a factor
analysis on Items 1, 2, 3, 6, 7, and 8 (expected to align with
System Usefulness), 9, 11, and 14 (expected to align with
Information Quality), and 16, 17, and 18 (expected to align
with Interface Quality), plus the overall Item 19. Unfortunately,
Downloaded by [James Lewis] at 11:14 10 April 2013
PSYCHOMETRIC EVALUATION OF THE T-CSUQ 323
TABLE 2
Varimax-Rotated Three-Factor Solution for
T-CSUQ (19 Items)
Factor
Item 1 2 3
1.604 .206 .246
2.735 .257 .092
3.697 .250 .272
4 .343 .480 .160
5.588 .475 .005
6.556 .355 .241
7.814 .048 .098
8.680 .089 .277
9 .384 .233 .376
10 .510 .336 .314
11 .291 .212 .490
12 .238 .487 .353
13 .184 .242 .847
14 .191 .527 .251
15 .266 .506 .394
16 .190 .679 .271
17 .187 .625 .223
18 .082 .640 .092
19 .163 .747 .045
Note. T-C S U Q =Turkish Computer System
Usability Questionnaire. Numbers in bold indicate
the strongest factor loading for an item.
Item 14 did not align with Information Quality. Substituting
Item 13 for 14, as shown in Table 4, did result in a stable
Information Quality factor.
3.2. Final Short Version of the T-CSUQ (T-CSUQ-SV)
Based on the psychometric evaluation, the recommended
short version of the T-CSUQ contains 13 items, as shown in
Figure 1. Note the renumbering of the items from 1 to 13. For
the original item numbers, refer to Table 4.
3.3. Reliability
Coefficient alphas for the recommended version of the T-
CSUQ-SV were as follows:
• Overall (Items 1–13): .85
• System Usefulness (Items 1–6): .88
• Information Quality (Items 7–9): .71
• Interface Quality (Items 10–12): .73
All coefficient alphas exceeded the minimum criterion of
.70 (Landauer, 1997), indicating sufficient reliability for the
intended use of the T-CSUQ-SV. The reliability results for the
TABLE 3
Varimax-Rotated Three-Factor Solution for
T-CSUQ (16 Items)
Factor
Item 1 2 3
1 .241 .589 .192
2 .193 .609 .338
4.558 .429 .041
6 .430 .629 .025
7 .029 .849 .160
8 .103 .694 .260
9 .156 .294 .632
10 .385 .549 .208
11 .216 .284 .470
12 .547 .297 .174
14 .466 .061 .518
15 .508 .276 .398
16 .677 .169 .259
17 .633 .170 .203
18 .634 .068 .138
19 .690 .141 .178
Note. T-C S U Q =Turkish Computer System
Usability Questionnaire. Numbers in bold indicate
the strongest factor loading for an item.
TABLE 4
Varimax-Rotated Three-Factor Solution for T-CSUQ
(13 Items)
Factor
Item (Old No.) Item (New No.) 1 2 3
11.608 .228 .270
22.672 .234 .197
33.651 .230 .364
64.551 .360 .235
75.857 .051 .072
86.686 .063 .286
9 7 .334 .162 .499
11 8 .249 .166 .551
13 9 .195 .252 .720
16 10 .164 .697 .327
17 11 .174 .628 .260
18 12 .083 .630 .095
19 13 .178 .684 .068
Note. T-C S U Q =Turkish Computer System Usability
Questionnaire. Numbers in bold indicate the strongest factor
loading for an item.
Downloaded by [James Lewis] at 11:14 10 April 2013
324 O. ERDINÇ AND J. R. LEWIS
FIG. 1. Version 1 of the Turkish Computer System Usability Questionnaire,
Short Version.
intermediate 19- and 16-item versions were similar, ranging
from .73 to .92 (Erdinç, 2011).
3.4. Sensitivity
Consistent with the results of sensitivity analyses of the
English CSUQ (Lewis, 2002), ttests indicated no evidence of
an effect of gender on ratings for any of the scales (all p>
.9). There were marginal effects of the type of work (preparing
vs. reading/signing documents) on System Usefulness, t(95) =
1.6, p=.10, and Overall, t(95) =1.75, p=.08, but not for
Information Quality, t(95) =.58, p=.56, or Interface Quality,
t(95) =.95, p=.35.
Analyses of frequency of use and duration of daily use pro-
vided strong evidence of scale sensitivity. Because the sample
sizes for some frequency and duration groups were relatively
small, we dichotomized these variables to get the number of
participants in each group as equal as possible (Frequency of
Use: n=56 “not every day” vs. n=38 “every day”; Duration
of Use: n=58 “<2 hr per day” vs. n=35 “>2 hr per day”).
As shown in Figure 2, a mixed-model analysis of variance
(using T-CSUQ-SV scales as a within-subjects variable) indi-
cated a significant main effect of frequency of use, F(1, 92) =
10.3, p=.002; a significant main effect of scale, F(2, 2) =3.7,
p=.026; and a significant interaction, F(2, 2) =3.7, p=.026.
Figure 3 shows the results of a similar analysis of duration of
use, which had significant main effects of duration, F(1, 91) =
30.8, p=.0000003, and scale, F(2, 2) =3.8, p=.025, but no
significant interaction, F(2, 2) =2.1, p=.12. Participants who
used the system more frequently or for longer daily durations
tended to rate the system more favorably.
3.5. Extreme Response Tendency
Of particular interest was whether the Turkish sample would
show evidence of a tendency toward an extreme response bias.
Investigation of this for the English version of the CSUQ indi-
cated no extreme response tendency (Lewis, 2002), but there is
FIG. 2. Turkish Computer System Usability Questionnaire, Short Version
scale by frequency of use. Note. SysUse =System Usefulness; InfoQual =
Information Quality; IntQual =Interface Quality.
FIG. 3. Turkish Computer System Usability Questionnaire, Short Version
scale by daily duration of use. Note. SysUse =System Usefulness; InfoQual =
Information Quality; IntQual =Interface Quality.
some evidence that members of different cultures exhibit dif-
ferent levels of the extreme response tendency, although these
differences do not always appear (Grimm & Church, 1999).
Nunnally’s (1978) method (described in Lewis, 2002) for
assessing the extreme response tendency indicated no evidence
for such a response bias. All ratios of squared correlations to
products of coefficient alpha for deviation and dichotomous
scores were greater than 1 (ratios less than .8 indicate an
extreme response tendency). The extreme response tendency
results for the intermediate 19- and 16-item versions also indi-
cated no evidence of extreme responding, with all ratios greater
than 1 (Erdinç, 2011).
4. DISCUSSION
The primary goal of this research was to develop a stan-
dardized usability questionnaire for use by speakers of the
Turkish language. Through a careful multistage translation pro-
cess and psychometric evaluation, this article presents such
Downloaded by [James Lewis] at 11:14 10 April 2013
PSYCHOMETRIC EVALUATION OF THE T-CSUQ 325
a questionnaire, the T-CSUQ-SV, based on the well-known
CSUQ (Figure 1). It is important to note that the T-CSUQ-SV is
not a simple item-for-item translation of the CSUQ. Only 13 of
the original 19 items survived the process of the transformation
of the CSUQ into the T-CSUQ-SV.
Psychometric evaluation of the T-CSUQ-SV indicated an
acceptable level of reliability for the overall scale and sub-
scales (all coefficient αs>.70). Factor analysis indicated
appropriate construct validity, with the items for each subscale
loading on separate factors. Consistent with psychometric eval-
uations of the English CSUQ, the T-CSUQ-SV was insensitive
to gender, moderately sensitive to type of usage, and highly
sensitive to frequency and duration of use. Also consistent
with the English version of the CSUQ, there was no evidence
that respondents exhibited an extreme response tendency, allay-
ing potential concerns with cross-cultural use of questionnaires
raised by Baumgartner and Steenkamp (2001), Cheung and
Rensvold (2000), Clarke (2001), and van de Vijver and Leung
(2001). These results indicate that Turkish usability practition-
ers should be able to use the T-CSUQ-SV with confidence when
conducting user research.
Future work to improve the questionnaire should focus
on three areas. The first is to improve the reliability of the
Information Quality and Interface Quality subscales through the
development and assessment of additional items. The reliability
of these subscales is adequate for research purposes but would
benefit from the inclusion of one to two more items per sub-
scale that align appropriately with the factor. The second is to
increase the number of psychometrically qualified subscales to
include constructs that are important for the evaluation of web-
sites, such as effective browsing and search, self-service, and
trust (Safar & Turner, 2005; Sauro & Lewis, 2012), possibly
drawing items from the literature on the assessment of perceived
web usability (e.g., Aladwani & Palvia, 2002; Bargas-Avila,
Lötscher, Orsini, & Opwis, 2009; Kirakowski & Cierlik, 1998;
Lascu & Clow, 2008; Sauro, 2011b; Wang & Senecal, 2007).
The third is to build and publish normative databases. One of the
key advantages of licensed questionnaires such as the SUPR-
Q (Sauro, 2011b), WAMMI (Kirakowski & Cierlik, 1998), and
SUMI (Kirakowski, 1996) is the interpretation of scores against
their proprietary normative databases. In recent years, some
researchers have published databases of nonproprietary stan-
dardized usability questionnaires, including the CSUQ (Lewis,
2002, 2012) and the SUS (Bangor, Kortum, & Miller, 2008;
Lewis & Sauro, 2009; Sauro, 2011a; Sauro & Lewis, 2012).
Over time, it would be advantageous for Turkish practitioners
who use the questionnaire to periodically publish their findings
to support the creation of similar normative databases for the
T-CSUQ-SV.
Another potential topic for future research is to investigate
whether there are Turkish-specific drivers of perceived usability.
The goal of our research was to develop an instrument for
Turkish usability practitioners and researchers based on the
items and factors of the CSUQ. As we conducted this work,
the data from factor and sensitivity analyses indicated that the
CSUQ factors as instantiated in the T-CSUQ-SV are applica-
ble to a Turkish population. We did not, however, conduct any
studies of differences between English- and Turkish-speaking
cultures to investigate whether cultural issues of the Turkish
population have any effect on the variables that the CSUQ mea-
sures. Due to the empirical success of our transformation effort,
we suspect that cultural similarities in the perception of usability
far outweigh the effects of cultural differences. We do, however,
acknowledge that there may be value in conducting studies to
see if there are special usability factors associated with Turkish
(but not non-Turkish) users.
REFERENCES
Aladwani, A. M., & Palvia, P. C. (2002). Developing and validating an
instrument for measuring user perceived Web quality. Information &
Management,39, 467–476.
Bagheri, E., & Ghorbani, A. A. (2009). A belief-theoretic framework for the
collaborative development and integration of para-consistent models. The
Journal of Systems and Software,82, 707–729.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation
of the System Usability Scale. International Journal of Human–Computer
Interaction,24, 574–594.
Bargas-Avila, J. A., Lötscher, J., Orsini, S., & Opwis, K. (2009). Intranet
Satisfaction Questionnaire: Development and validation of a question-
naire to measure user satisfaction with the Intranet. Computers in Human
Behavior,25, 1241–1250.
Baumgartner, H., & Steenkamp, J. B. E. M. (2001). Response styles in market-
ing research: A cross-national investigation. Journal of Marketing Research,
38, 143–156.
Cheung, G. W., & Rensvold, R. B. (2000). Assessing extreme and acquiescence
response sets in cross-cultural research using structural equations modeling.
Journal of Cross-Cultural Psychology,31, 187–212.
Chow, M., & Chan, L., (2010). Development and evaluation of a compart-
mental picture archiving and communications system model for integration
and visualization of multidisciplinary biomedical data to facilitate student
learning health clinic. Computers & Education,54, 733–741.
Christophersen, R., & Konradt, U. (2011). Reliability, validity, and sensitivity
of a single-item measure of online store usability. International Journal of
Human–Computer Studies,69, 269–280.
Clarke, I. (2001). Extreme response style in cross-cultural research.
International Marketing Review,18, 301–324.
Erdinç, O. (2011). Turkish version of the IBM Computer System Usability
Questionnaire (CSUQ): Adaptation and psychometric evaluation (Extended
research report). Istanbul, Turkey: Department of Industrial Engineering,
Turkish Air Force Academy. (Available from the first author: E-mail:
erdinc.oguzhan@gmail.com)
Finstad, K. (2010). The usability metric for user experience. Interacting with
Computers,22, 323–327.
Frias-Martinez, E., Chen, S.Y., & Liu, X. (2009). Evaluation of a personal-
ized digital library based on cognitive styles: Adaptivity vs. adaptability.
International Journal of Information Management,29, 48–56.
Grimm, S. D., & Church, A. T. (1999). A cross-cultural study of response
biases in personality measures. Journal of Research in Personality,33,
415–441.
Kartakis, S., & Stephanidis, C. (2010). A design-and-play approach to acces-
sible user interface development in Ambient Intelligence environments.
Computers in Industry,61, 318–328.
Kirakowski, J. (1996). The Software Usability Measurement Inventory:
Background and usage. In P. Jordan, B. Thomas, & B. Weerdmeester (Eds.),
Usability evaluation in industry (pp. 169–178). London, UK: Taylor &
Francis. Available from http://www.ucc.ie/hfrg/questionnaires/sumi/index.
html
Downloaded by [James Lewis] at 11:14 10 April 2013
326 O. ERDINÇ AND J. R. LEWIS
Kirakowski, J., & Cierlik, B. (1998). Measuring the usability of websites. In
Proceedings of the Human Factors and Ergonomics Society 42nd Annual
Meeting (pp. 424–428). Santa Monica, CA: HFES.
Landauer, T. K. (1997). Behavioral research methods in human-computer inter-
action. In M. Helander, T. K. Landauer, and P. Prabhu (Eds.) Handbook
of human–computer interaction, 2nd ed. (pp. 203–227). Amsterdam:
Elsevier.
Lascu, D., & Clow, K. E. (2008). Web site interaction satisfaction:
Scale development consideration. Journal of Internet Commerce,7,
359–378.
Lewis, J. R. (1991). User satisfaction questionnaires for usability studies:
1991 manual of directions for the ASQ and PSSUQ (Tech. Rep. No. 54.609).
Boca Raton, FL: IBM Corporation.
Lewis, J. R. (1992a). Psychometric evaluation of the computer system usability
questionnaire: The CSUQ (Tech. Rep. No. 54.723), Boca Raton, FL: IBM
Corporation.
Lewis, J. R. (1992b). Psychometric evaluation of the post-study system usability
questionnaire: The PSSUQ. In Proceedings of the Human Factors Society
36th Annual Meeting (pp. 1259–1263). Santa Monica, CA: Human Factors
Society.
Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires:
Psychometric evaluation and instructions for use. International Journal of
Human–Computer Interaction,7, 57–78.
Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from
five years of usability studies. International Journal of Human–Computer
Interaction,14, 463–488.
Lewis, J. R. (2012). Usability testing. In G. Salvendy (Ed.), Handbook of
human factors and ergonomics (4th ed., pp. 1267–1312). New York, NY:
Wiley.
Lewis, J. R., & Sauro, J. (2009). The factor structure of the System Usability
Scale. In M. Kurosu (Ed.), Human centered design, HCII 2009 (pp. 94–103).
Heidelberg, Germany: Springer-Verlag.
MacDorman, K. F., Whalen, T. J., Ho, C., & Patel, H. (2011). An improved
usability measure based on novice and expert performance. International
Journal of Human–Computer Interaction,27, 280–302.
McNamara, N., & Kirakowski, J. (2011). Measuring user-satisfaction with
electronic consumer products: The Consumer Products Questionnaire .
International Journal of Human–Computer Studies,69, 375–386.
Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.
Pereira, J. A., Quach, S., Hamid, J. S., Heidebrecht, C. L., Quan, S. D., Nassif,
J., ...Kwong, J. C. (2012). Exploring the feasibility of integrating barcode
scanning technology into vaccine inventory recording in seasonal influenza
vaccination clinics. Vaccine,30, 794–802.
Safar, J. A., & Turner, C. W. (2005). Validation of a two factor structure of
system trust. In Proceedings of the Human Factors and Ergonomics Society
49th Annual Meeting (pp. 497–501). Santa Monica, CA: HFES.
Saleem, J. J., Haggstrom, D. A., Militello, L. G., Flanagan, M., Kiess, C. L.,
Arbuckle, N., & Doebbling, B. N. (2011). Redesign of a computerized clin-
ical reminder for colorectal cancer screening: A human computer interaction
evaluation. BMC Medical Informatics and Decision Making,11, 1–11.
Sauro, J. (2011a). A practical guide to the System Usability Scale (SUS):
Background, benchmarks & best practices. Denver, CO: Measuring
Usability LLC.
Sauro, J. (2011b). The Standardized Universal Percentile Rank Questionnaire
(SUPR–Q). Retrieved from http://www.suprq.com/
Sauro, J., & Lewis, J. R. (2009). Correlations among prototypical usability
metrics: Evidence for the construct of usability. In Proceedings of CHI 2009
(pp. 1609–1618). Boston, MA: ACM.
Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical
statistics for user research. Burlington, MA: Morgan-Kaufmann.
Savidis, A., Grammenos, D., & Stephanidis, C. (2006). Developing inclusive
e-learning systems. Universal Access in the Information Society,5, 51–72.
Tullis, T. S., & Stetson, J. N. (2004). A comparison of questionnaires for
assessing website usability. Paper presented at the Usability Professionals
Association Annual Conference, June. UPA, Minneapolis, MN.
van de Vijver, F. J. R., & Leung, K. (2001). Personality in cultural context:
Methodological issues. Journal of Personality,69, 1007–1031.
van Lammeren, R., Houtkamp, J., Colijn, S., Hilferink, M., & Bouwman,
A. (2010). Affective appraisal of 3D land use visualization. Computers,
Environment and Urban Systems,34, 465–475.
Wang, J., & Senecal, S. (2007). Measuring perceived website usability. Journal
of Internet Commerce,6, 97–112.
ABOUT THE AUTHORS
O˘
guzhan Erdinç received his Ph.D. in Industrial Engineering
from Marmara University and teaches ergonomics in the
Turkish Air Force Academy. His research interests include
applied industrial and office ergonomics, ergonomics assess-
ment tools, and usability of computer systems. His current
research focus is adaptation and application of usability assess-
ment tools and methods.
James R. Lewis is a senior human factors engineer (at IBM
since 1981), focusing on the design/evaluation of speech appli-
cations. He has published influential papers in the areas of
usability testing and measurement. His books include Practical
Speech User Interface Design and (with Jeff Sauro) Quantifying
the User Experience.
Downloaded by [James Lewis] at 11:14 10 April 2013
