A Usability Evaluation of Academic Virtual Reference Services

Abstract

This study examined the usability of five virtual reference services-instant messenger chat, e-mail, telephone, text messaging, and Skype videoconferencing-by having 31 undergraduate and graduate students evaluate the usability of the virtual reference services of two different universities. The study's results suggest that user preference and satisfaction for virtual reference service are highly correlated with the service's overall usability in terms of effectiveness and efficiency. Online chat was rated highest across all measures including satisfaction and seven different usability factors. Major implications of the study suggest that online chat is the virtual reference of choice for university students and that usability metrics are a good predictor of user preferences centered on high return on investment, speed of transaction, convenience, and minimal effort.

Full text

309
A Usability Evaluation of Academic
Virtual Reference Services
Anthony S. Chow and Rebecca A. Croxton
Anthony S. Chow is Associate Professor in the Department of Library and Information Studies of the School
of Education at The University of North Carolina at Greensboro; e-mail: aschow@uncg.edu. Rebecca A.
Croxton is a Doctoral Student and Instructor at The University of North Carolina at Greensboro; e-mail:
racroxto@uncg.edu. © 2014 Anthony S. Chow and Rebecca A. Croxton, Aribution-NonCommercial
(hp://creativecommons.org/licenses/by-nc/3.0/) CC BY-NC
This study examined the usability of five virtual reference services—in-
stant messenger chat, e-mail, telephone, text messaging, and Skype
videoconferencing—by having 31 undergraduate and graduate students
evaluate the usability of the virtual reference services of two different uni-
versities. The study’s results suggest that user preference and satisfaction
for virtual reference service are highly correlated with the service’s overall
usability in terms of effectiveness and efficiency. Online chat was rated
highest across all measures including satisfaction and seven different
usability factors. Major implications of the study suggest that online chat
is the virtual reference of choice for university students and that usability
metrics are a good predictor of user preferences centered on high return
on investment, speed of transaction, convenience, and minimal effort.
o stay relevant in this rapidly
evolving information age, li-
braries across the world are
racing to reinvent themselves.
They must do so in an environment
where information seekers have “…many
options, little patience, and use many
dierent types of information and com-
munication technology.”1 While many
libraries are increasingly making their
content available digitally over the Inter-
net, they are also taking steps to make their
reference services available virtually. Some
virtual reference services, including e-
mail, instant messaging (online chat), and
telephone have been around for decades,
while other services such as text messaging
and videoconferencing are just beginning
to emerge. Sloan reported that e-mail ref-
erence has been going strong for 20 years
and live chat-based reference for 10 years.2
Virtual reference services are becoming
commonplace in both academic and pub-
lic libraries across the United States. In a
recent survey study of 1,226 academic and
public libraries, 54.95 percent of survey
respondents indicated they oer reference
services virtually. In a separate phase of
this study, the researchers found that 84.6
percent (n = 203) of library websites evalu-
ated oered virtual reference services.3 The
most common type of virtual reference
was e-mail reference, followed by chat,
then text, then “other.” Both surveys found
the least used form of virtual reference was
Skype or video conferencing.
While many libraries now oer a suite
of virtual reference services, the prolif-
eration of these services is not always
matched by actual use.4 Many virtual
crl13-408

310 College & Research Libraries May 2014
reference projects initiated in recent years
have been suspended or discontinued
altogether. Several studies suggest the
success of virtual reference services does
not depend solely on the quality of the
service but may also be dependent upon
funding, volume of reference questions,
stang levels, hardware and soware
issues, and institutional culture.5 Further,
many library patrons are not aware that
virtual reference services are available.6
While many elements contribute to a
person’s experience using a virtual refer-
ence service, a signicant yet oen over-
looked factor is the overall usability of the
digital services. Chow explains, “…digital
environments are for human use, and
taking a human centered design (HCI)
approach with an emphasis on pervasive
usability with representative users … will
help ensure that the digital environment
is high on utility and ease-of-use.”7
The International Organization for
Standardization (ISO) formally denes
usability as “the extent to which the
product can be used by specied users to
achieve specied goals with eectiveness,
eciency and satisfaction in a specied
context of use.”8 Analyses of the usability
of virtual reference services must reach
beyond the actual interface design of the
tools. Mu et al. found “…there is no sig-
nicant dierence in users’ opinions of a
VRS [virtual reference service] and their
willingness to use it that is caused by the
design of the interface, provided … users
are aware of its existence and the link is
clearly labeled with its function…”9
Looking beyond the interface design
of virtual reference tools and ensuring
awareness of services, usability evalua-
tions must consider what factors make
virtual services eective, ecient, and
satisfying. Several studies have found
that the popularity and usability of vir-
tual reference services may be dependent
upon the type of question with which a
patron presents.10 Other factors that may
play a role in perceptions of usability may
include the type of user, age, and gender
of library patrons.11
A frequent oversight in the develop-
ment of highly usable digital information
environments (including virtual reference
services) is this:
…designers frequently develop
digital information spaces utilizing
their own paradigms—what they
deem to be important, organized in
a fashion that makes the most sense
to them—in absentia of the people
that will actually be using it. This
disconnect creates a gap between
the designer and the user.12
While it is ideal to engage representa-
tive users as design partners from the
very beginning of a project, many librar-
ies bypass this vital step. A key element
of application and implementation of a
usable digital information environment
is to continuously improve, rene, and
collect representative feedback.13 There-
fore, usability of virtual reference services
can continue to be improved upon, even
if already developed and implemented.
Understanding how the usability
of virtual reference mediums aects a
library patron’s information-seeking
experiences, preferences, and overall
success will allow libraries to design and
improve upon virtual reference services
with greater precision, eectiveness, and
eciency. Usability must be a develop-
mental keystone of a successful library
experience; for, if users cannot access
information easily, there is lile point in
expending the time, eort, and expense it
takes to provide digital reference.14
The authors published an article exam-
ining the usability of virtual user services
from the perspectives of university faculty,
sta, students, and librarians.15 A central
limitation of this study was self-reporting
from participants, many of whom had
never used the virtual reference services
they were rating. This study reects the
results of a more traditional usability evalu-
ation with an emphasis on user testing,
direct interaction with the services being
tested, and the collection of specic metrics

A Usability Evaluation of Academic Virtual Reference Services 311
and data points representing an industry
standard operational denition composed
of specic measures for how a user views
a service’s effectiveness, efficiency, and
satisfaction. Are particular virtual refer-
ence services more usable than others? Are
there factors that make particular reference
services more usable than others?
Whereas previous studies have ex-
amined usability from an opinion and
largely nonempirical (without usability
testing with actual users) perspective,
this study seeks to answer these questions
empirically by examining the usability
of ve virtual reference services (e-mail,
telephone, online chat, text messaging ref-
erence, and Skype videoconferencing) of-
fered in two doctoral-granting institutions
in the southeastern United States through
the perspectives and usability results of
undergraduate and graduate students
who evaluated each of the services.
Literature Review
When considering the usability of digital
information environments, one must
look closely at factors of eectiveness,
eciency, and satisfaction.16 While many
studies have examined various aspects of
the usability of e-mail and chat reference
services, literature regarding the overall
usability of the suite of virtual reference
media oered by many libraries today
(including e-mail, telephone, online chat,
text messaging, and videoconferencing)
is sparse. It does not reflect the more
traditional usability practice of conduct-
ing formal usability tests involving users
aempting to complete major tasks. Such
formal testing allows more specic feed-
back through quantitative and qualitative
test data; rather than relying solely on
participant self-report opinions, it is based
on their explicit ability to actually use a
service or system.
Do Characteristics of Reference
Mediums Affect the Usability of
Virtual Reference Services?
Many studies have indicated that users
value virtual reference services for their
eectiveness, quality, convenience, speed,
eciency, and immediacy with which
answers were received.17 However, dier-
ing characteristics of particular reference
media may aect the overall usability,
satisfaction with, and preferences for us-
ing individual virtual reference media. In
a recent survey study and factor analysis
of 936 university library users and their
virtual reference preferences (e-mail, tele-
phone, online chat, text messaging, and
videoconferencing), Chow and Croxton
found that, across all study respondents,
e-mail was the most preferred (56.6%)
virtual reference medium over all other
choices.18 In this same study, the most
frequently cited reasons for e-mail prefer-
ence were convenience, familiarity, ease
of use, and a written record that was
both precise and in depth. However, in
a separate study using semistructured
interviews with undergraduates, gradu-
ate students, and faculty members,
Connaway, Radford, and Dickey found
that, while the majority of respondents
indicated that subject expertise was very
important to them, only 42 percent would
be willing to wait for that expertise, as is
oen required when using e-mail.19
Chow and Croxton further noted
library patrons choose online chat, tele-
phone, and text messaging reference
services for their speed and quickness
of response rst, ease of use second, fol-
lowed by multitasking (for using online
chat). Texting and videoconferencing ref-
erence services received consistently low
preference scores throughout this same
study.20 Similarly, additional studies have
found that individuals choose virtual
reference services (such as online chat)
for convenience, immediacy, eciency,
remote access, price, availability 24/7/365,
anonymity, eectiveness, and quality.21
While e-mail, online chat, and tele-
phone reference services are common-
place in libraries today, text and video
reference are beginning to emerge as
virtual reference media in academic
libraries. However, lile data currently
exist in the literature regarding either

312 College & Research Libraries May 2014
the usage or usability of these services.
Current research suggests usage of these
emerging reference media is limited. In
a review of digital reference activity for
two semesters at Southeastern Louisiana
University, Hill, Hill, and Sherman found
text messaging constituted only a small
portion of digital reference activity. Of
1,447 requests for information via digital
reference, 66 percent (n = 954) were via
chat, 28 percent (n = 410) were via e-
mail, and 6 percent (n = 83) were via text
messaging.22 The researchers postulated,
“With more convenient reference options
available such as e-mail and 24/7 chat,
patrons may choose these more robust
forms of communication that are beer
suited for college level research ques-
tions.”23 Research further suggests that
online videoconferencing services may
not be as usable as other forms of virtual
reference. At Ohio University Libraries,
Booth found, “The near impossibility of
maintaining eye contact via a video chat
interface is …disconcerting and some-
what reduces one’s ability to engage in an
eective, traditional face-to-face reference
interview.”24
Despite the generally positive ndings
regarding the usability of virtual refer-
ence services, other studies have brought
to light negative factors that can aect
the overall usability of virtual reference
services. Dee and Allen suggested that
problems accessing digital reference
services and diculty using it were two
major barriers.25 Thomsett-Scott found
that students dislike having to wait for
responses from e-mail and chat.26
These ndings may be explained, in
part, by the concept of “strategic satisc-
ing,” introduced by Warwick et al., who
noted that their subjects “used the exper-
tise that they had … gained in information
seeking to create time-saving strategies
to complete coursework with minimum
eort.”27 In this study, the researchers
noted their subjects chose both sources
and strategies within a well-known com-
fort zone in information seeking. These
ndings suggest that convenience may
play a role in judging the usability of a
reference medium.
This current study sought to deter-
mine which virtual reference medium
is considered to be the most usable for
library patrons.
Does Type of Question Play a Role
in the Usability of Virtual Reference
Services?
Several studies have been conducted that
examine the types of questions users bring
to virtual reference service and the suit-
ability of particular virtual reference media
for dierent types of questions. A general
perception is that online chat reference
is suitable mostly for simple factual and
directional but not reference questions.28
Other researchers believe chat reference
is capable of going beyond basic ready
reference questions.29 Ward reported that
78 percent of chat reference transactions
in an academic library showed some
indication of bibliographic instruction
or question negotiation, which typically
occurs in subject-based research ques-
tions.30 In a review of survey data for 415
virtual reference transactions (such as
online chat), Kwon found user satisfaction
was the highest in subject-based research
questions, while simple factual questions
were the second highest, followed by local
library information, circulation-related
questions, and resource access questions.31
Bringing e-mail into the conversation,
Lee found many similarities among the
questions asked using e-mail and chat.32
Both chat and e-mail virtual reference
received approximately the same propor-
tion of questions about nding known
items, research, and reference. E-mail
received a small number of questions
about referencing and citing, while chat
received none of these questions.
According to the ndings of Hill et
al., text messaging reference is a unique
virtual reference service well suited for
short answer questions. They noted,
Most of the questions [via text
messaging] have been of the short-

A Usability Evaluation of Academic Virtual Reference Services 313
answer variety…. However, the
service also prompted short-answer
reference questions … that are atyp-
ical of reference questions received
via phone, email, and chat.33
Chow and Croxton found the prefer-
ence for using both text messaging and
Skype videoconferencing to be very
low.34 Though studies have shown the
use of text messaging reference to be low,
current trends reported by the Cellular
Telecommunications Industry Associa-
tion (CTIA) suggest text messaging refer-
ence may soon become a more prevalent
medium, regardless of type of question.
Between June 2007 and June 2008, CTIA
reported a 160 percent increase in the
number of text messages sent (from 28.8
billion to 75 billion).35
At this time there are no concrete data
as to whether online videoconferencing
is perceived to be more usable for either
a quick, factual question or a research-
related question. In a pilot study using
Skype reference in an academic library,
Booth noted, “overall trends conrmed
our hypothesis that … [Skype] … would
serve as a source for basic information
rather than an in-depth point.”36 The
limited research currently available about
preferences for and/or usability of provid-
ing online videoconferencing services in
libraries suggests users are simply not yet
ready to use these services on a regular
basis, regardless of the type of question
with which they present.37
Does Age or Patron Status Play a
Role in the Perceived Usability of
Virtual Reference Services?
The type of patron may also play a role
in perceptions of usability of virtual refer-
ence media. In a survey study of 345 chat
users, Ward found that undergraduates
saw the “chat” service as being appli-
cable for most situations, while graduate
students recommended the service notice-
ably more for ready reference questions.38
In their study of 936 university faculty,
sta, and students, Chow and Croxton
noted that, based on university status,
age, gender, and race, specic users have
particular reference service preferences
based on type of question. They found
this:
Faculty, sta, and graduate students
ranked telephone for factual ques-
tions at signicantly higher levels
than undergraduate students. Fac-
ulty and sta prefer using e-mail
for factual questions at signicantly
higher levels than both undergradu-
ate and graduate students while
students have a signicantly higher
preference for using text messag-
ing.… All user groups rated online
chat relatively high for seeking
help with a factual question.… For
research questions …faculty, sta,
and graduate students preferred
e-mail at statistically higher levels
than undergraduates, while under-
graduate students preferred online
chat at signicantly higher levels
than faculty and sta.39
If user preferences for particular refer-
ence services are an indicator of the actual
usability of these services, one may expect
the usability ratings to mirror that which
is preferred. The ndings of Chow and
Croxton suggest users select their virtual
reference medium according to the type
of question they have. Role and age also
appear to be inuencing factors in deter-
mining reference preferences.40
Regardless of type of question, age,
or role of the patron, information seek-
ing appears to follow the Principle of
Least Eort. Poole noted this principle
to be the prominent result in a review of
a dozen information-seeking studies.41
Overall, Poole found that information
seekers aempt to minimize the overall
work associated with something both
now and in the future. Rubin further
explains that people will seek the most
convenient source available to meet their
information needs. In a focus group study
of 33 university faculty, undergraduates,

314 College & Research Libraries May 2014
and graduate students, Young and Von
Seggern found that when considering
criteria for information seeking, concern
for time was brought up most oen.42
Study participants would oen accept
inappropriate information or informa-
tion of lower quality if nding it took
less time. Therefore, the ndings of Poole
and Young and Von Seggern suggest
library virtual reference services patrons
may choose the reference medium that
requires the least eort, that which is
convenient, quick, and easy to use.
This study aempts to determine the
overall usability of the suite of virtual
reference media oered by many libraries
today. The study further examines the fac-
tors that make particular virtual reference
media more usable than others. Within
this study, the type of question and role
of the user and how they are related to
usability ratings will be explored. There-
fore, the purpose of the study is to seek
answers to three research questions:
RQ1: Which virtual reference medium
is most usable?
RQ2: Does the type of question play a
factor in the usability of virtual reference
media?
RQ3: Do perceptions of usability for
virtual reference media dier depending
on the role of the patron?
Method
To assess the usability of library vir-
tual references services, this study used a
mixed method quantitative and qualita-
tive design to conduct a usability evalua-
tion that analyzed and compared patrons’
perceptions of usability of ve virtual ref-
erence support services (e-mail, telephone,
instant messaging [online chat], text mes-
saging, and Skype videoconferencing)
oered at two separate midsized public
university libraries in the Southeast, each
with a student population of approxi-
mately 18,500. The study was conducted
over ve months in the spring of 2011.
Participants
A randomly selected group of under-
graduates (n = 13) and graduate students
(n = 18) participated in this study. Though
31 individuals participated in the study,
there were occasions where fewer indi-
viduals responded to particular survey
questions. Of the study participants, 74.2
percent were female and 25.8 percent
were male. Age range was primarily
concentrated under age 34, with the fol-
lowing breakdown: 48.3 percent under
24, 37.9 percent ages 25–34, 10.3 percent
ages 35–44, and 3.4 percent ages 55–64.
In terms of prior experience with the
technologies used in the study, partici-
pants were experienced with e-mail, chat,
and telephone, respectively, moderately
experienced with text messaging, and
inexperienced with using Skype.
Materials, Instrumentation, and Data
Analysis
To properly conduct a usability evalu-
ation for each of the virtual reference
services, each study participant was e-
mailed a set of detailed instructions that
included 10 different questions—five
quick, factual questions and ve proce-
dural or research-based questions that
required a more detailed response by
the librarian (see Appendix A for sample
instructions). The questions assigned
to study participants were of similar
subject matter and depth. The actual
questions, however, diered throughout
the study to control for familiarity with
the questions received on the part of the
reference librarian (see Appendix B for
a list of questions). Participants were
assigned a particular university library
(University A or University B) to which
to direct their preassigned questions.
Fifteen participants asked questions
virtually at University A, while 16 partici-
pants asked virtual reference questions
at University B. Questions were asked
using the ve virtual reference services
(e-mail, telephone, online chat, text mes-
saging, and online videoconferencing) as
illustrated in table 1. An exception was
that University B did not oer online
videoconferencing.

A Usability Evaluation of Academic Virtual Reference Services 315
Subjects completed online surveys of
their experiences—rating measures of
usability and satisfaction aer evaluat-
ing each virtual reference medium (see
Appendices C–G for survey instruments).
Preferences for reference formats were
ranked at the completion of the study
(see Appendix H). Data regarding study
participants’ prior experience with virtual
reference media or tools were gathered
at the beginning of participation in the
study.
The instructions and survey instru-
ments were pilot tested with graduate
students for preliminary face and con-
struct validity, rened accordingly, and
then administered via e-mail to study
participants.
Virtual Reference Services Tested
Five virtual reference mediums were
tested in this study. Consistent across
each of the media, study participants
were instructed to ask the rst, preas-
signed quick, factual question, wait
four hours, and then ask the second,
preassigned procedural, research-based
question. Once replies to both questions
were received, study participants were
instructed to complete online surveys
of their experiences aer each medium
was tested. E-mail reference was used by
directing study participants (through a
link in their e-mailed instructions) to a
university library’s Ask Us! page. Sub-
jects were then instructed to move on to
Telephone reference. They were given toll-
free phone numbers to use to call their
assigned library’s reference desk. At the
completion of telephone reference evalu-
ation, participants were then instructed
to contact the Online Chat reference ser-
vices and ask both factual and procedural
research-based questions. Participants
were given a link in their instruction
set that directed them to the online chat
window. Study participants then were
instructed to contact the library using
text messaging reference via their cell
phones/mobile devices. They were given
a specic number to contact via text, along
with a particular number and/or word to
initiate their reference inquiries. Finally,
study participants contacting University
A were instructed to contact the Reference
Department via Skype videoconferencing
and ask their two assigned questions,
waiting at least four hours between each
inquiry. (University B does not oer vid-
eoconferencing reference at this time.)
Participants were given specic instruc-
tions about how to contact the library us-
ing a given Skype username. Participants
needed to have a computer with the free
TABLE 1
Participant Frequency Counts x Type of Question
University Student Type Participants Reference Service and Type & Number of
Questions
E-mail Phone Chat Text Skype*
Quick Fact
Research
Quick Fact
Research
Quick Fact
Research
Quick Fact
Research
Quick Fact
Research
University A Grad Student 9 9 8 9 8 9 9 9 6 8 7
Undergraduate 6 6 6 6 6 6 6 6 6 5 5
University B* Grad Student 9 9 8 9 9 9 9 8 8 0 0
Undergraduate 7 7 7 7 7 6 6 6 5 0 0
Total 31 31 29 31 30 30 30 29 25 13 12
*University B did not offer Skype reference services.

316 College & Research Libraries May 2014
Skype soware downloaded as well as a
web camera. Aer participants completed
this portion of the study, they completed
online surveys of their experiences with
this reference medium.
Results
“…chat has the speed and eciency of
phone usage and the in-depth eective-
ness of email” (research participant)
Instant Messenger Chat the Clear
Choice
The results found a perfect relationship
between participant satisfaction and
their usability ratings for each refer-
ence medium tested during the study.
Undergraduate and graduate students
were most satised with chat reference,
for both research and factual questions,
followed by telephone, e-mail, text, and
Skype, respectively. See table 2.
Table 3 shows the combined cumula-
tive scores of both graduate and under-
graduate students.
Chat was also rated the highest across
all usability factors by participants: more
eective (9.6 out of 10), ecient (9.4),
and satisfying to use (9.6) than any other
reference service. Participant usability
ratings for the remaining four reference
services—telephone, e-mail, text, and
Skype—also exactly paralleled partici-
pant satisfaction. See gure 1.
Chat was rated the most usable service
because it scored the highest for all refer-
ence services across all seven usability
factors for both research and factual
questions. One-way ANOVA, a statistical
test used to compare mean scores within
and between groups, showed that the
usability ratings for chat across partici-
pants were statistically the same for six
of seven usability factors, which sug-
gests consistently high ratings for chat
across participants. Overall, chat had
statistically signicant higher usability
TABLE 2
Undergraduate and Graduate Reference Service Preferences
Undergraduate and Graduate Student Usability Ratings
E-mail Telephone Chat Skype Text
UGGUGGUGGUGGUGG
Effectiveness 8.4 8.1 8.9 8.5 9.5 9.7 5.8 6.4 5.8 6.3
Task Completion 8.9 7.9 9.0 8.6 9.5 9.5 5.8 6.6 6.1 6.3
Quality of Output 7.8 8.2 8.8 8.4 9.5 9.8 5.8 6.2 5.5 6.3
Efciency 8.7 8.0 8.9 8.5 9.2 9.4 5.8 5.0 6.5 6.3
Deviations 9.3 8.3 9.2 9.0 9.5 9.4 5.4 4.9 6.5 6.4
Error Rate 9.1 8.6 9.2 9.0 9.5 9.5 5.6 5.4 6.9 6.3
Time on Task 7.3 7.3 8.5 7.7 8.9 9.6 6.4 4.7 5.7 6.2
Mental Effort 9.1 7.9 8.7 8.4 9.1 9.4 5.8 4.9 7.0 6.2
Satisfaction 8.4 7.8 9.0 8.5 9.4 9.7 5.8 5.6 6.4 6.4
Usability (Grand Mean) 8.5 8.0 8.9 8.5 9.3 9.5 5.8 5.5 6.3 6.3
TABLE 3
Combined Undergraduate and
Graduate Ratings for Reference
Service Preferences
Reference
Medium
Research
Question
Factual
Question
Chat 9.1 8.9
Telephone 9 7.5
E-mail 7.5 7.1
Text 5.9 5.8
Skype 5.8 6.3

A Usability Evaluation of Academic Virtual Reference Services 317
ratings across factors in comparison to
other media with its grand mean (mean
of means for all factors) being statisti-
cally signicantly higher than e-mail and
Skype. See table 2.
Face-to-face Versus Virtual Reference
When participants were asked to select
only one reference medium including
face-to-face, participants still preferred
chat reference more frequently for both
research (53%) and factual (60%) reference
questions than all other reference options
combined. See tables 5 and 6.
Online chat and telephone
combined represented the prefer-
ences of 73 percent of the study’s
participants for seeking help with
a basic factual question. See table 6.
Time a Factor
Four of the five reference ser-
vices—chat, telephone, text, and
Skype—represent potentially
quick responses to participants’
questions. For factual questions,
chat had the quickest response
time, with 96 percent of all transac-
tions taking less than ve minutes
and with 23 percent taking less
than a minute. The second-high-
est rated reference service was
telephone, with 90 percent of all
transactions taking less than ve
minutes and with 42 percent taking less
than a minute to complete. Both chat and
telephone were rated the rst and second
most preferred, respectively.
In contrast, only 52 percent of text mes-
sage and 67 percent of Skype transactions
were completed within ve minutes. For
e-mail, more of an asynchronous mode of
communication, only 52 percent of transac-
tions were resolved within one hour, and
39 percent took over 12 hours to resolve.
The paern was similar for research
questions. Chat transactions had 100
FIGURE 1
Reference Service Usability Ratings
9.68.78.26.16.2
9.48.78.36.45.3
9.68.78.0
6.45.7
9.58.78.2
6.35.7
Virtual Medium by Usability Factors
EffectivenessEfficiency SatisfactionGrand Mean
TABLE 4
Reference Service by Individual Usability Factors
Factor Chat Tel. E-mail Text Skype
Task Completion 9.5* 8.8** 8.3*** 6.2* 6.3*
Quality of Output 9.7 8.5* 8.1** 6.0* 6.1
Deviations 9.4* 9.1** 8.7*,*** 6.5* 5.1
Errors 9.5* 9.1*,*** 8.8*,*** 6.5* 5.5*
Time on Task 9.3* 8.0 7.3** 6.0 5.3*
Mental Effort 9.3* 8.5 8.4* 6.5 5.2*
Satisfaction 9.6* 8.7 8.0*** 6.4 5.7
Grand Mean 9.5* 8.7 8.2** 6.3 5.7**
*Homogeneity of Variance is signicant at the p<.05 level.
**Statistically signicant at the p<.05 level.
*** Statistically signicant at the p<.01 level.

318 College & Research Libraries May 2014
percent resolved and telephone resolved
97 percent of all transactions within 15
minutes or less. Skype had 15 percent and
text had 31 percent of their transactions
take longer than one hour. For e-mail, 48
percent of all transactions took longer
than two hours and 30 percent took 12
hours or longer.
Chat Favored Independently of All
Factors
Analysis of various factors found no
signicant dierences for gender, age,
race, library, or prior experience. This
suggests that the majority of participants
consistently rated chat and the four other
media independent of other factors.
Discussion
The results of our study support our pre-
vious study, which found undergraduate
students preferred online chat over all
other reference services.43 One limita-
tion of that study was the majority of
respondents to our survey had not ac-
tually used the reference services. This
study represents one of the rst usabil-
ity evaluations, which had participants
specically test each reference service
by using each of them to answer a set
of tasks. Participants tested each service
for both research and factual questions
and rated each across seven estab-
lished usability factors.44 The results
help operationally dene participant
preferences based on usability ratings
emphasizing eectiveness, eciency,
and satisfaction.
Online chat was rated highest in eec-
tiveness, eciency, and satisfaction. One
reason for this was because of how fast
the transactions were in comparison to all
other services. In addition, participants
noted other prominent factors such as
real-time interaction, the ability to mul-
titask while chaing, the convenience of
“already being on the computer,” and the
visual and archival aspects of text-based
chat that also allows for the sharing of
links. As one participant noted:
Chat was more user friendly than
the telephone. I think that seeing the
response in a wrien form helps me
to crystallize my understanding of
the answer. I also liked the fact that
the librarian could easily send me
links via the chat; it would be more
dicult to do so over the telephone.
I also think that the chat was a lot
more to the point, without being
curt; on the telephone you actually
have to converse with the other
person which may not be ideal if a
quick answer is needed.
Telephone was the next most preferred
reference medium and, similar to chat,
had an extremely fast transaction time
for participants. In fact, an immediate re-
sponse was the primary factor mentioned
by participants for why they liked using
the telephone. The other prominent fac-
tor was the real-time interaction. As one
TABLE 5
Reference Service Preferences
for Research Questions
Medium f %
Online Chat Reference 16 53.3%
Face-to-Face Consultation 8 26.7%
E-mail Reference 5 16.7%
Telephone Consultation 1 3.3%
Skype Video Reference 0 0.0%
Text 0 0.0%
TABLE 6
Reference Service Preferences for
Factual Questions
Medium f %
Online Chat Reference 18 60.0%
Telephone Consultation 4 13.3%
Text-a-Librarian Reference 3 10.0%
Face-to-Face Consultation 2 6.7%
Other 2 6.7%
E-mail Reference 1 3.3%
Skype Video Reference 0 0.0%

A Usability Evaluation of Academic Virtual Reference Services 319
connections. Second, while some lauded
the ability to see and hear the reference
librarian, others did not feel that visual
cues added anything to the process and at
times made them feel uncomfortable. One
participant noted the dierences between
chat and Skype: “Chat was very fast and
simple to use—I didn’t have to download
a new program onto my computer to get
answers. Also, being able to see and hear
the person made me feel a lile weird
because I don’t usually communicate with
people via video chat.”
High Usability, High Satisfaction
The International Organization for Stan-
dardization dened usability as a person’s
ability to use a product or service with
maximum eectiveness, eciency, and
satisfaction. Using this framework, our
study found a perfect positive relation-
ship between user preferences and how
they rated each reference service across
usability factors—the higher the usabil-
ity, the more highly rated the reference
service was. In other words, the higher
the participant’s return-on-investment in
terms of time and mental eort, the more
highly rated the service. This supports the
concept from Warwick et al. of strategic
satisficing, where students choose the
information-seeking paths that are most
comfortable and convenient for them.45
The results of the study suggest that
each of its three research questions can
be answered: online chat was found to be
the most usable and participant prefer-
ence was independent of all other factors
examined including type of question,
university service tested, gender, race,
academic status, or age.
Study Limitations
The study had four primary limitations.
First, the study conducted a usability
evaluation instead of a traditional us-
ability test with participants in a lab
seing; outside of this controlled testing
environment, the study relied on the self-
reports of students under the assumption
that they actually completed all tasks and
participant put it, “I like the telephone
for when I need information quickly and
right then.” Another noted, “Through
telephone you can again ask questions or
recheck something with the person and
would receive more accurate response
in a timely manner.” One negative for
telephone service is that it may cost stu-
dents money by using limited cell phone
minutes.
E-mail was the third-rated reference
service. Unlike the other four services, the
majority of e-mail transactions took over
12 hours to complete. While it is much
slower compared to the other services,
strong points include the ability to ask a
detailed question and receive a similarly
detailed response with web links. In ad-
dition, it serves as a wrien record that
can be referred back to anytime. One par-
ticipant noted, “[E-mail] communication
allows for the responder to provided [sic]
detailed information along with links,
examples, and aachments.”
Text messaging was rated much lower
than expected given the prevalence in
which students use text messaging for
social communication. Participants noted
that one of the reasons text was rated so
low was that, although it is very similar
to online chat, they had to remember a
phone number and use a cell phone as
opposed to just opening a web browser
and engaging in an online chat from
their computer. As one participant noted,
“I liked chat better than text messag-
ing because it was all on the computer.
The chat told me when the person was
responding so I liked that. I wasn’t just
waiting around.” Another issue was the
quality of support for text messaging
reference services as several participants
noted they never received a response at
all and gave up.
Skype was the lowest-rated reference
service. Analysis of participant comments
suggests that this low rating was due to
a host of problems. First was technical in
nature—some could not get the soware
to work properly, some did not have a
functional webcam, and others had slow

320 College & Research Libraries May 2014
completed each of the usability surveys in
a valid, authentic manner. Second , a low
sample size of only 31 student partici-
pants evaluating two university reference
services limits the study’s internal and
external validity and overall generaliz-
ability. The third major limitation was
that Skype reference services were not
oered at both reference departments,
and therefore only half of the participants
were able to evaluate and test this service.
The last primary limitation had to do with
the quality of scenarios used to test the
reference services. Feedback from one li-
brary’s reference service sta was that the
redundant questions being asked allowed
them to recognize participants in the
study and may have slightly inuenced
the quality of responses they received.
Conclusion
The study’s implications center on the
fact that it represents one of the first
formal usability evaluations of library
virtual reference services using a usability
framework. Five types of virtual reference
services were examined at two dierent
university reference departments and the
results clearly suggest that participant
satisfaction and preferences were guided
by time of response, convenience, eec-
tiveness, and eciency.
Online chat was clearly preferred by
the majority of participants and rated the
highest for time of response and across
all seven usability factors on a statisti-
cally signicant basis—task completion,
quality of output, deviations, errors, time-
on-task, mental eort, and overall satis-
faction. These ndings support Poole’s
Principle of Least Eort where informa-
tion seekers want access to information
with the least amount of eort possible.46
Online chat appears to combine the
strengths of other reference services into
one service: the immediacy of response and
real-time interaction of telephone, the con-
venience, wrien record and content shar-
ing of e-mail, and the instant messaging
of text-messaging and Skype. The study’s
results suggest that, with the increased
proliferation of wireless technologies
such as smartphones and tablets, instant
messaging chat may continue to grow
in popularity as the younger population
entering college will be even more familiar
and comfortable with this technology.
Given the rapid changes brought on
through current and emerging technolo-
gies, there is a need for more agile meth-
ods of measuring usability in the eld.
Making use of the study’s instruments
and protocol, the study represents a pro-
cess for how to conduct similar usability
evaluations that have users complete a
set of tasks so that they have actually
“tested” the service before completing
a scale about the overall eectiveness,
eciency, and satisfaction of their user
experience. This provides organizations
with both qualitative and quantitative
data to ensure at a more precise level
that services are indeed high in usability.
Furthermore, the quantitative data can be
analyzed statistically to see more clearly
what specic factors of usability might
represent opportunities for improvement.
Such usability evaluations, while
less comprehensive and valid than tra-
ditional usability testing in controlled
environments, may replace the use of
the standard controlled usability test in
terms of frequency and choice because it
is much quicker, cost eective, and viable
to conduct. This may lead to usability
evaluations occurring more oen. These
will complement rather than completely
replace controlled tests, which should
still take place but less oen and for only
those who are prepared to do so with
the appropriate time, participants, and
hardware and soware.
Future research involves seeking to
replicate this study at other universities
as well as to examine faculty and sta
usability evaluations for the same virtual
reference services, who have been found
to prefer e-mail over chat. In addition, fur-
ther examination of whether these nd-
ings can generalize to the virtual reference
service preferences of public and special
library patrons need to also be explored.

A Usability Evaluation of Academic Virtual Reference Services 321
Appendix A.
A Comparison Study of Library Reference Services
Revised Research Study Procedures
The goal of the study is to compare the overall usability of four types of reference
services—telephone, e-mail, online chat, and text messaging.
As a participant in our study you will ask two reference questions in each of these
four formats to the library reference sta at the XXXXX University Library. Aer
receiving a response to your questions in each format we want you to complete an
online survey regarding your overall experience—include a detailed description of
the general process you followed to seek and receive help, approximately how long
the entire transaction took from start-to-nish, and your general thoughts and reac-
tions to the experience. When possible, please try to ask your reference questions at
dierent times throughout the day.
Aer you have sought and received help in all four mediums, please complete a nal
debrieng survey that will collect basic demographic information, assess each of the
reference environments on major usability factors, and ask you to rank which of the
mediums you preferred and why.
Note: The XXXXX Library Reference Desk is staed during the following hours. For
optimal results, we recommend you contact the library during the following hours:
Research & Instructional Services
Sunday 1 p.m.–5 p.m.
Mon.–Thu. 9 a.m.–9 p.m.
Friday 9 a.m.–5 p.m.
Saturday 10 a.m.–5 p.m.
Here is a step-by-step procedure you can follow to complete the study:
Format 1: Email
Note: Please keep track of how long it took from start-to-nish to have each question answered.
Also, take notes of any thoughts, observations, or opinions you have so that you can share these
with us at the end of the project.
1. Go to the XXXX Library Contact Us web page and complete the “Email a Ques-
tion” form at: (hyperlink to university ask us email page). Under the drop down box
for “Where would you like your question sent?” select XXXXX Library.)
2. Complete the form along with your email address and enter this rst question
in the form provided:
“What are the hours and phone number for the Writing Center on campus?”
3. If the answer was satisfactory, please repeat steps 1 and 2 but instead ask this
second question:
“How do I cite a blog in APA format?”
4. Try out the response to make sure it is satisfactory.
5. When you feel the answer was satisfactory (if not, please ask a follow-up question
via e-mail) please complete this evaluation form for receiving email assistance
(link to online survey) on your experience.

322 College & Research Libraries May 2014
Format 2: Telephone
Note: Please keep track of how long it took from start-to-nish to have each question answered.
Also, take notes of any thoughts, observations, or opinions you have so that you can share these
with us at the end of the project.
Note: It is preferable if you complete your email questions/interactions before asking the follow-
ing questions via telephone. If possible, please wait at least four hours from the time you have
completed your email questions/interactions before asking the questions via telephone.
1. Call the XXXXX University Library Library Reference Desk at 1-800-xxx-xxxx
and ask following question:
“I’m looking for the phone number and hours for the Student Recreation Center. Can
you help?”
2. If the answer was satisfactory, please repeat step 1 but instead ask this second
question:
“How do I cite a book that has three authors using APA format?”
3. Try out the response to make sure it is satisfactory
4. When you feel the answer was satisfactory (if not, please ask a follow up ques-
tion via email) please complete this general evaluation form for the telephone
format (hyperlink to online survey) on your experience.
Format 3: Chat
Note: Please keep track of how long it took from start to nish to have each question answered.
Also, take notes of any thoughts, observations, or opinions you have so that you can share these
with us at the end of the project.
Note: It is preferable if you complete your telephone questions/interactions before asking these
same questions via online chat. If possible, please wait at least four hours from the time you have
completed your telephone questions/interactions before asking the questions via chat.
1. Go to the XXXX University Library Contact Us Web page at (hyperlink to
library contact us page)
2. Use the imbedded chat box provided on the le side of the page under Ask a
Librarian.
3. Ask the following question:
“Can you help me nd the phone and hours for the Financial Aid Oce?”
Note: Please copy and save the full text of the response that you receive to your question
here so that you can include it in the debrieng survey.
4. If the response was satisfactory, please repeat steps 1–3 but instead ask this
second question.
“How do I cite a newspaper article from an online newspaper using APA format?”
Note: Please copy and save the full text of the response that you receive to your question
here so that you can include it in the debrieng survey.
5. Try out the response to make sure it is satisfactory.
6. When you feel the answer was satisfactory (if not, please ask a follow-up ques-
tion via chat) please complete this general evaluation form on your experience
with chat (hyperlink to online survey).

A Usability Evaluation of Academic Virtual Reference Services 323
Format 4: Skype
Note: Please keep track of how long it took from start to nish to have each question answered.
Also, take notes of any thoughts, observations, or opinions you have so that you can share these
with us at the end of the project.
Note: It is preferable if you complete your online chat questions/interactions before asking
these same questions via Skype. If possible, please wait at least four hours from the time you
have completed your online chat questions/interactions before asking the questions via Skype.
1. Open up Skype on your computer and please test to make sure both your audio
and video are working before contacting the reference desk.
2. Once you are condent everything is working, please contact the Library Web
reference desk through Skype by going to (hyperlink to Library Skype Desk)
Ask the following question in the manner provided:
“I’m trying to nd the hours and phone number for the Cashier’s Oce. Can you
help?”
3. If the response was satisfactory, please repeat steps 1–3 but instead ask this
second question.
“How do I cite an electronic book in APA format?”
4. Try out the response to make sure it is satisfactory.
5. When you feel the answer was satisfactory (if not, please ask a follow up ques-
tion via Skype) please complete this general evaluation form on your experi-
ence using Skype (hyperlink to online survey).
Format 5: Text Messaging
Note: Please keep track of how long it took from start to nish to have each question answered.
Also, take notes of any thoughts, observations, or opinions you have so that you can share these
with us at the end of the project.
Note: It is preferable if you complete your Skype questions/interactions before asking these same
questions via Text Messaging. If possible, please wait at least four hours from the time you have
completed your Skype questions/interactions before asking the questions via Text Messaging.
1. Using your cell phone or mobile device, ask the following question by sending a
text message to the Text-a-Librarian service. Send your question to (800) xxx-xxxx.
“Can you help me gure out the hours and phone number for the Academic Success
Program?”
2. If the response was satisfactory, please repeat step 1 but instead ask this second
question.
“How do you cite an article from a magazine in APA format?”
3. Try out the response to make sure it is satisfactory.
4. When you feel the answer was satisfactory (if not, please ask a follow-up ques-
tion via text), please complete this general evaluation form on your experience
using Text-a-Librarian (hyperlink to online survey).
Note: The nal question of this survey asks you to compare text messaging to Skype. In
the comments eld, please indicate that you did not evaluate Skype.
Evaluation of Each Format
1. Now that you have had the questions answered in each of the four formats, we
want you to complete a nal survey that assesses the general usability of each
technology along with some general thoughts and observations.

324 College & Research Libraries May 2014
2. Only aer you have had your questions answered satisfactorily in each of the four
formats above please complete this nal survey.
Notes:
• On the nal survey form you will encounter a page relating to Skype video conferencing.
You may either skip those questions or answer NA when appropriate.
• One the nal page of the nal survey, you will encounter some questions relating to Rating
Your Preferences of Formats. Please go ahead and answer these questions (even though
they include Skype as an option).
APPENDIX B
Questions Used for this Study
Quick, Factual Questions Research/Procedural Reference Questions
• Can you help me gure out the
hours and phone number for the
Student Success Center?
• Can you help me nd out who to
contact to get some information for
tutoring in math?
• Can you help me nd the address
for the International Students &
Scholar Program?
• Can you help me nd the phone and
hours for the Financial Aid Ofce?
• Can you help me nd the street
address for the Family Research
Center?
• Can you help me nd the street
address for the Global Research
Institute?
• Can you help me nd the street ad-
dress for the parking ofce?
• Can you tell me where the book-
store is on campus?
• Can you tell me where the Writing
Center is located?
• Do you know the dates we’re off for
Winter Break?
• Do you know the hours and phone
number for the Student Health
Center?
• Do you know what day classes start
for the Summer Session?
• Can you help me nd some biographical
information on Cesar Ritz, the founder of
the Ritz hotels? I’m supposed to try to use a
database or nd a book.
• Can you help me nd some biographical
information on Conrad Hilton? I’m not sup-
posed to use Google.
• Can you help me nd some biographical in-
formation on former rst lady, Laura Bush?
I’m supposed to try to use something other
than Google.
• Can you help me nd some biographical
information on Julia Child. I’m supposed to
use a database or nd a book.
• Can you show me how to cite a magazine
article using MLA format?
• Can you tell me how to cite a book with two
authors using APA format?”
• Can you tell me how to cite a book with two
authors using MLA format?
• Can you tell me how to cite a chapter from a
book using APA format?
• How can I nd some biographical infor-
mation on Michael Jordan, the basketball
player, without using Google?
• How can I nd some biographical informa-
tion on Ray Kroc, the founder of McDon-
ald’s, without using Google?
• How can I limit my search in the library
catalog so I see only books available in
Spanish?
• How can I limit my search in the library
catalog so that only books come up?
• How can I limit my search in the library
catalog so that only books written in Span-
ish come up?
• How can I limit my search in the library
catalog to online resources?

A Usability Evaluation of Academic Virtual Reference Services 325
APPENDIX B
Questions Used for this Study
Quick, Factual Questions Research/Procedural Reference Questions
• Do you know what day classes start
in January?
• Do you know what day Summer
Break starts?
• Do you know what day the fall
semester begins this year?
• Do you know what days we are off
for Thanksgiving?
• How late are you open tonight?
• I was just wondering what the
last day of classes is for the fall
semester.
• I’m looking for the phone number
and hours for the Student Recreation
Center. Can you help?
• What are the hours and phone
number for the Speaking Center on
campus?
• What are the hours and phone num-
ber for the Writing Center?
• What are your hours during the
week?
• What are your hours over the
weekend?
• What day is fall graduation this
year?
• What day is spring graduation this
year?
• What is the street address for the
Music Library?
• What time are you open to tonight?
• What time do you close on Satur-
days?
• What time do you close on Sun-
days?
• What time do you close on week-
ends?
• What time do you close tonight?
• What time do you open in the morn-
ings?
• What time do you open on Satur-
days?
• What time do you open tomorrow
morning?
• What time does the library close on
Saturday and Sunday?
• Where is the student Health Center
located on campus?
• How do I cite a blog in APA format?
• How do I cite a blog in MLA format?
• How do I cite a book that has three authors
using APA format?
• How do I cite a chapter from a book using
MLA format?
• How do I cite a journal article in MLA for-
mat that has four authors? I’m not sure how
to list their names.
• How do I cite a newspaper article from an
online newspaper using APA format?
• How do I cite a newspaper article from an
online newspaper using MLA format?
• How do I cite a pamphlet using MLA
format?
• How do I cite a personal interview I con-
ducted for a paper in APA format?
• How do I cite a poem from an anthology
using MLA format?
• How do I cite a website using MLA format?
• How do I cite an article from an online
database using MLA format?
• How do I nd biographical information on
Maya Angelou without using Google?
• How do I limit my search in the library
catalog so I only get online resources?
• How do I limit my search in the library
catalog so I see only electronic books?
• How do I limit my search in the library
catalog so I see only electronic resources?
• How do I limit my search in the library
catalog so I see only movies?
• How do I limit my search in the library
catalog so I see only videos?
• How do you cite a magazine article using
MLA format?
• How do you cite a photo downloaded from
the Web using MLA format?
• How do you cite a website using MLA
format?
• How do you cite an article from a magazine
in APA format?
• How would I nd biographical information
on Maya Angelou without using Google?
• Is there a way to search the catalog for
books that are less than ve years old?
• Is there a way to limit my search in the li-
brary catalog so I see what is only available
in the Music Library?

326 College & Research Libraries May 2014
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
Please give us your feedback on receiving assistance via e-mail by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please copy and paste the full text of your e-mail response to question 1 in the form
below:
1.E-mail Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
5. Approximately how long, from start
to
finish, did it take for your question to be
answered?
6. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 hour
nmlkj
1–5 hours
nmlkj
6–10 hours
nmlkj
12–23 hours
nmlkj
1 day
nmlkj
2 days
nmlkj
3 or more days
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
Appendix C.
E-mail Online Survey

A Usability Evaluation of Academic Virtual Reference Services 327
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
5. Approximately how long, from start
to
finish, did it take for your question to be
answered?
6. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 hour
nmlkj
1–5 hours
nmlkj
6–10 hours
nmlkj
12–23 hours
nmlkj
1 day
nmlkj
2 days
nmlkj
3 or more days
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
Appendix C.
E-mail Online Survey
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please copy and paste the full text of your e-mail response to question 2 in the form
below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 hour
nmlkj
1–5 hours
nmlkj
6–10 hours
nmlkj
12–23 hours
nmlkj
1 day
nmlkj
2 days
nmlkj
3 or more days
nmlkj
Please elaborate if you would like to:
5
5
6
6
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
Please give us your feedback on receiving assistance via e-mail by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please copy and paste the full text of your e-mail response to question 1 in the form
below:
1.E-mail Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

328 College & Research Libraries May 2014
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
10. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
–
10 (1=lowest, 10=highest satisfaction), rate how satisfied
you were with the overall reference assistance you received using e-mail.
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did.
5
5
6
6
Appendix C.
E-mail Online Survey
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please copy and paste the full text of your e-mail response to question 2 in the form
below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 hour
nmlkj
1–5 hours
nmlkj
6–10 hours
nmlkj
12–23 hours
nmlkj
1 day
nmlkj
2 days
nmlkj
3 or more days
nmlkj
Please elaborate if you would like to:
5
5
6
6
Email Survey (Master)Email Survey (Master)Email Survey (Master)Email Survey (Master)
Please give us your feedback on receiving assistance via e-mail by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please copy and paste the full text of your e-mail response to question 1 in the form
below:
1.E-mail Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

A Usability Evaluation of Academic Virtual Reference Services 329
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
Please give us your feedback on receiving assistance via telephone by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Telephone Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Appendix D.
Telephone Online Survey
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
5. Approximately how long, from start
-
to
-
finish, did it take for your question to be
answered?
6. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 minute
nmlkj
1
–
5 minutes
nmlkj
6
–
10 minutes
nmlkj
11
–
15 minutes
nmlkj
16
–
30 minutes
nmlkj
31
–
60 minutes
nmlkj
1
–
2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6

330 College & Research Libraries May 2014
Appendix D.
Telephone Online Survey
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please provide a general description of the answer you received to question 2 in the
form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1
–
5 minutes
nmlkj
6
–
10 minutes
nmlkj
11
–
15 minutes
nmlkj
16
–
30 minutes
nmlkj
31
–
60 minutes
nmlkj
1
–
2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
5. Approximately how long, from start
-
to
-
finish, did it take for your question to be
answered?
6. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 minute
nmlkj
1
–
5 minutes
nmlkj
6
–
10 minutes
nmlkj
11
–
15 minutes
nmlkj
16
–
30 minutes
nmlkj
31
–
60 minutes
nmlkj
1
–
2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
Please give us your feedback on receiving assistance via telephone by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Telephone Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

A Usability Evaluation of Academic Virtual Reference Services 331
Appendix D.
Telephone Online Survey
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
Please give us your feedback on receiving assistance via telephone by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Telephone Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please provide a general description of the answer you received to question 2 in the
form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1
–
5 minutes
nmlkj
6
–
10 minutes
nmlkj
11
–
15 minutes
nmlkj
16
–
30 minutes
nmlkj
31
–
60 minutes
nmlkj
1
–
2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
10. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using the telephone.
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6

332 College & Research Libraries May 2014
Appendix D.
Telephone Online Survey
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
12. How did the assistance you received via telephone compare overall compare to the
e-mail format?
Overall, I liked telephone more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked telephone less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
10. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using the telephone.
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6
Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)Telephone Survey (Master)
Please give us your feedback on receiving assistance via telephone by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Telephone Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

A Usability Evaluation of Academic Virtual Reference Services 333
Appendix E.
Chat Online Survey
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
Please give us your feedback on receiving assistance via chat by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 1 in the form below:
1.Chat Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

334 College & Research Libraries May 2014
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
Please give us your feedback on receiving assistance via chat by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 1 in the form below:
1.Chat Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
5. Approximately how long, from start
to
finish, did it take for your question to be
answered?
6. On a scale from 1–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
Appendix E.
Chat Online Survey

A Usability Evaluation of Academic Virtual Reference Services 335
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
Please give us your feedback on receiving assistance via chat by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 1 in the form below:
1.Chat Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 2 in the form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
Appendix E.
Chat Online Survey

336 College & Research Libraries May 2014
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
Please give us your feedback on receiving assistance via chat by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 1 in the form below:
1.Chat Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
10. On a scale from 1–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1–
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using chat.
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6
Appendix E.
Chat Online Survey

A Usability Evaluation of Academic Virtual Reference Services 337
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
Please give us your feedback on receiving assistance via chat by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide the full text of the response you received via chat and/or a general
description of the answer you received to question 1 in the form below:
1.Chat Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)Chat Survey (Master)
12. How did the assistance you received via chat compare overall to e-mail?
13. How did the assistance you received via chat compare overall to telephone?
Overall, I liked chat more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked chat less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Overall, I liked chat more than telephone
nmlkj
About the same
nmlkj
Overall, I liked chat less than telephone
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Appendix E.
Chat Online Survey

338 College & Research Libraries May 2014
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
Please give us your feedback on receiving assistance via Skype by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Skype Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Appendix F.
Skype Online Survey

A Usability Evaluation of Academic Virtual Reference Services 339
Appendix F.
Skype Online Survey
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
Please give us your feedback on receiving assistance via Skype by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Skype Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
5. Approximately how long, from start
to
finish, did it take for your question to be
answered?
6. On a scale from 1—
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6

340 College & Research Libraries May 2014
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
7. Please copy and paste the second question that you asked.
8. Please provide a general description of the answer you received to question 2 in the
form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
Appendix F.
Skype Online Survey
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
Please give us your feedback on receiving assistance via Skype by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Skype Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

A Usability Evaluation of Academic Virtual Reference Services 341
Appendix F.
Skype Online Survey
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
Please give us your feedback on receiving assistance via Skype by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Skype Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
10. On a scale from 1—
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
—
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using Skype.
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
12. How did the assistance you received via Skype compare overall to e-mail?
13. How did the assistance you received via Skype compare overall to telephone?
Overall, I liked Skype more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked Skype less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Overall, I liked Skype more than telephone
nmlkj
About the same
nmlkj
Overall, I liked Skype less than telephone
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6

342 College & Research Libraries May 2014
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
12. How did the assistance you received via Skype compare overall to e-mail?
13. How did the assistance you received via Skype compare overall to telephone?
Overall, I liked Skype more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked Skype less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Overall, I liked Skype more than telephone
nmlkj
About the same
nmlkj
Overall, I liked Skype less than telephone
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Appendix F.
Skype Online Survey
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
Please give us your feedback on receiving assistance via Skype by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Skype Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)Skype Survey (Master)
14. How did the assistance you received via Skype compare overall to chat?
Overall, I liked Skype more than chat
nmlkj
About the same
nmlkj
Overall, I liked Skype less than chat
nmlkj
Please elaborate on your comparison between the two (i.e. Did being able to see and hear the person help or hinder the process?):
5
5
6
6

A Usability Evaluation of Academic Virtual Reference Services 343
Please give us your feedback on receiving assistance via Text Messaging by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Text Messaging Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
Appendix G.
Text Online Survey

344 College & Research Libraries May 2014
5. Approximately how long, from start
to
finish, did it take for your question to be
answered?
6. On a scale from 1
–
10 (1=lowest, 10=highest satisfaction), rate how satisfied you were
with the overall experience in
getting your first question answered.
Less than 1 minute
nmlkj
1
–
5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
Appendix G.
Text Online Survey
Please give us your feedback on receiving assistance via Text Messaging by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Text Messaging Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
7. Please copy and paste the second question that you asked.
8. Please provide a general description of the answer you received to question 2 in the
form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6

A Usability Evaluation of Academic Virtual Reference Services 345
7. Please copy and paste the second question that you asked.
8. Please provide a general description of the answer you received to question 2 in the
form below:
9. Approximately how long, from start
to
finish, did it take for your question to be
answered?
*
5
5
6
6
5
5
6
6
Less than 1 minute
nmlkj
1–5 minutes
nmlkj
6–10 minutes
nmlkj
11–15 minutes
nmlkj
16–30 minutes
nmlkj
31–60 minutes
nmlkj
1–2 hours
nmlkj
2+ hours
nmlkj
Please elaborate if you would like to:
5
5
6
6
Appendix G.
Text Online Survey
10. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using Text Messaging.
12. How did the assistance you received via Text Messaging compare overall to e-mail?
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6
Overall, I liked Text Messaging more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Please give us your feedback on receiving assistance via Text Messaging by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Text Messaging Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

346 College & Research Libraries May 2014
10. On a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied you were
with the overall experience in
getting your second question answered.
11. Overall, using a scale from 1
–
10 (1=lowest, 10=highest satsifaction), rate how satisfied
you were with the overall reference assistance you received using Text Messaging.
12. How did the assistance you received via Text Messaging compare overall to e-mail?
1
(lowest)
nmlkj
2
nmlkj
3
nmlkj
4
nmlkj
5
nmlkj
6
nmlkj
7
nmlkj
8
nmlkj
9
nmlkj
10
(highest)
nmlkj
Please elaborate on why you rated as you did (quicker/slower response than expected; response was adequate; general comfort level in asking
question, etc.):
5
5
6
6
1
(lowest)
nmlkj2
nmlkj3
nmlkj4
nmlkj5
nmlkj6
nmlkj7
nmlkj8
nmlkj9
nmlkj10
(highest)
nmlkj
Please elaborate on why you rated as you did:
5
5
6
6
Overall, I liked Text Messaging more than e-mail
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than e-mail
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
13. How did the assistance you received via Text Messaging compare overall to
telephone?
14. How did the assistance you received via Text Messaging compare overall to chat?
Overall, I liked Text Messaging more than telephone
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than telephone
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Overall, I liked Text Messaging more than chat
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than chat
nmlkj
Please elaborate on your comparison between the two (i.e. Did being able to see and hear the person help or hinder the process?):
5
5
6
6
Appendix G.
Text Online Survey
Please give us your feedback on receiving assistance via Text Messaging by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Text Messaging Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj

A Usability Evaluation of Academic Virtual Reference Services 347
15. How did the assistance you received via Text Messaging compare overall to Skype?
Overall, I liked Text Messaging more than Skype
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than Skype
nmlkj
Please elaborate on your comparison between the two (i.e. Did being able to see and hear the person help or hinder the process?):
5
5
6
6
Appendix G.
Text Online Survey
Please give us your feedback on receiving assistance via Text Messaging by answering the following questions.
1. Your Name:
2. To which university library did you direct your questions for this study?
3. Please copy and paste the first question that you asked.
4. Please provide a general description of the answer you received to question 1 in the
form below:
1.Text Messaging Format
*
*
5
5
6
6
5
5
6
6
University A
nmlkj
University B
nmlkj
13. How did the assistance you received via Text Messaging compare overall to
telephone?
14. How did the assistance you received via Text Messaging compare overall to chat?
Overall, I liked Text Messaging more than telephone
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than telephone
nmlkj
Please elaborate on your comparison between the two:
5
5
6
6
Overall, I liked Text Messaging more than chat
nmlkj
About the same
nmlkj
Overall, I liked Text Messaging less than chat
nmlkj
Please elaborate on your comparison between the two (i.e. Did being able to see and hear the person help or hinder the process?):
5
5
6
6

348 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
1. Your Name:
2. Please indicate your academic status.
3. To which university library did you direct your questions for this study?
4. What is your gender?
5. What is your age?
6. What is your ethnicity?
*
*
Undergraduate
nmlkj
Graduate
nmlkj
University A
nmlkj
University B
nmlkj
Male
nmlkj
Female
nmlkj
Under 24
nmlkj
25–34
nmlkj
35
–
44
nmlkj
45–54
nmlkj
55–64
nmlkj
65 and up
nmlkj
White
nmlkj
Black
nmlkj
Hispanic/Latino
nmlkj
Asian/Pacific Islander
nmlkj
Native American
nmlkj
Multiracial
nmlkj
Appendix H.
Debrieng Usability Survey

A Usability Evaluation of Academic Virtual Reference Services 349
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
7. Please rate your overall experience with each of the five formats (in general).
Please rate the use of e-mail on the following usability factors.
1. Effectiveness is defined as the extent to which a goal or task is reached. Rate your
overall experience with both e-mail reference questions/answers on the following factors:
No Previous Experience
Inexperienced (used less than 5
times) Moderate (used 6–10 times) Experienced (used more than 10
times)
E-mail nmlkjnmlkjnmlkjnmlkj
Telephone nmlkjnmlkjnmlkjnmlkj
Chat nmlkjnmlkjnmlkjnmlkj
Skype nmlkjnmlkjnmlkjnmlkj
Text
Messaging
nmlkjnmlkjnmlkjnmlkj
Satisfaction Rating
Task Completion (Were you able to successfully use e-mail to ask your questions and have them answered?) 6
Quality of Output (How satisfied were you with the overall quality of the e-mail responses and the extent to which
they answered your questions?)
6
Please feel free to elaborate:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
Appendix H.
Debrieng Usability Survey

350 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both e-mail reference questions/answers on the following factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both e-mail reference questions/answers:
4. In having the two reference questions answered across e-mail, what was your PRIMARY
SOURCE for access to computer technology?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to send and retrieve your e-mails without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?) 6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?) 6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using e-mail to ask reference questions? 6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
Personal computer at home
gfedc
Computer access at work
gfedc
Computer access at the department/library/university at my site
gfedc
Feel free to elaborate:
Appendix H.
Debrieng Usability Survey

A Usability Evaluation of Academic Virtual Reference Services 351
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
5. Please add any further comments about your perceptions of using e-mail.
Please rate the use of the telephone on the following usability factors.
1. Effectiveness is defined as the extent to which a goal or task is reached. Rate your
overall experience with both telephone reference questions/answers on the following
factors:
5
5
6
6
Satisfaction Rating
Task Completion (Were you able to successfully use telephone to ask your questions and have them answered?) 6
Quality of Output (How satisfied were you with the overall quality of the telephone responses and the extent to
which they answered your questions?)
6
Please elaborate on why you rated each as you did:
5
5
6
6
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both telephone reference questions/answers on the following
factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both telephone reference questions/answers:
4. What type of telephone did you use to ask the reference questions?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use telephone reference without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?) 6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?) 6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using telephone to ask reference questions?
6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
My cell phone
gfedc
My home telephone
gfedc
My work telephone
gfedc
Other Wireless Device
gfedc
Other
Appendix H.
Debrieng Usability Survey

352 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both telephone reference questions/answers on the following
factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both telephone reference questions/answers:
4. What type of telephone did you use to ask the reference questions?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use telephone reference without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?)
6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?)
6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?)
6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using telephone to ask reference questions?
6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
My cell phone
gfedc
My home telephone
gfedc
My work telephone
gfedc
Other Wireless Device
gfedc
Other
Appendix H.
Debrieng Usability Survey
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both telephone reference questions/answers on the following
factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both telephone reference questions/answers:
4. What type of telephone did you use to ask the reference questions?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use telephone reference without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?)
6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?)
6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?)
6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using telephone to ask reference questions? 6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
My cell phone
gfedc
My home telephone
gfedc
My work telephone
gfedc
Other Wireless Device
gfedc
Other
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
5. Please add any further comments about your perceptions of using the telephone.
Please rate the use of Chat on the following usability factors.
1. Effectiveness is defined as the extent to which a goal or task is reached. Rate your
overall experience with both chat reference questions/answers on the following factors:
5
5
6
6
Satisfaction Rating
Task Completion (Were you able to successfully use chat to ask your questions and have them answered?) 6
Quality of Output (How satisfied were you with the overall quality of the chat responses and the extent to which they
answered your questions?)
6
Please elaborate on why you rated each as you did:
5
5
6
6

A Usability Evaluation of Academic Virtual Reference Services 353
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both chat reference questions/answers on the following factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both your chat reference questions/answers:
4. In having the two reference questions answered in chat, what was your PRIMARY
SOURCE for access to computer technology?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use chat without any problems?) 6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?) 6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?) 6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using chat to ask reference questions? 6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
Personal computer at home
gfedc
Computer access at work
gfedc
Computer access at the department/library/university at my site
gfedc
Feel free to elaborate:
Appendix H.
Debrieng Usability Survey

354 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both Skype reference questions/answers on the following factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both your Skype reference questions/answers:
4. In having the two reference questions answered in chat, what was your PRIMARY
SOURCE for access to computer technology?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use Skype without any problems?) 6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?) 6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?) 6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using Skype to ask reference questions?
6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
Personal computer at home
gfedc
Computer access at work
gfedc
Computer access at the department/library/university at my site
gfedc
Feel free to elaborate:
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
5. Please add any further comments about your perceptions of using chat.
Please rate the use of Skype on the following usability factors.
1. Effectiveness is defined as the extent to which a goal or task is reached. Rate your
overall experience with both Skype reference questions/answers on the following factors:
5
5
6
6
Satisfaction Rating
Task Completion (Were you able to successfully use Skype to ask your questions and have them answered?) 6
Quality of Output (How satisfied were you with the overall quality of the Skype responses and the extent to which
they answered your questions?)
6
Please elaborate on why you rated each as you did:
5
5
6
6
Appendix H.
Debrieng Usability Survey

A Usability Evaluation of Academic Virtual Reference Services 355
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
5. Please add any further comments about your perceptions of using Skype.
Please rate the use of Text Messaging on the following usability factors.
1. Effectiveness is defined as the extent to which a goal or task is reached.
Rate your
overall experience with both Text Messaging reference questions/answers on the
following factors.
5
5
6
6
Satisfaction
Task Completion (Were you able to successfully use Text Messaging to ask your questions and have them
answered?)
6
Quality of Output (How satisfied were you with the overall quality of the Text Messaging responses and the extent to
which they answered your questions?)
6
Please elaborate on why you rated each as you did.
5
5
6
6
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both Skype reference questions/answers on the following factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both your Skype reference questions/answers:
4. In having the two reference questions answered in chat, what was your PRIMARY
SOURCE for access to computer technology?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use Skype without any problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?)
6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?)
6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using Skype to ask reference questions? 6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
Personal computer at home
gfedc
Computer access at work
gfedc
Computer access at the department/library/university at my site
gfedc
Feel free to elaborate:
Appendix H.
Debrieng Usability Survey
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal. Rate your
overall satisfaction with both telephone reference questions/answers on the following
factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals. Rate your
overall satisfaction with both telephone reference questions/answers:
4. What type of telephone did you use to ask the reference questions?
Satisfaction Rating
Deviations from critical path (How satisfied are you that you were able to use telephone reference without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?)
6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?)
6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?)
6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using telephone to ask reference questions?
6
Please elaborate on why you rated each as you did:
5
5
6
6
Please elaborate on why you rated each as you did:
5
5
6
6
My cell phone
gfedc
My home telephone
gfedc
My work telephone
gfedc
Other Wireless Device
gfedc
Other

356 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Efficiency is defined as the amount of effort required to reach your goal.
Rate your
overall satisfaction with both Text Messaging reference questions/answers on the
following factors:
3. Satisfaction is your overall level of comfort in being able to attain your goals.
Rate your
overall satisfaction with both your Text Messaging reference questions/answers.
4. What type of device did you use to ask the reference questions?
Satisfaction Rating
Deviations from the critical path (How satisfied are you that you were able to use Text Messaging without any
problems?)
6
Error Rate (How satisfied were you that you accomplished your goal with very few or no errors?) 6
Time on Task (How satisfied are you that you accomplished your goal with minimal time wasted?) 6
Mental Effort (How satisfied are you that you accomplished your goal with minimal mental effort?) 6
Satisfaction Rating
Overall, how satisfied were you with the general usability of using Text Messaging to ask reference questions? 6
Please elaborate on why you rated each as you did.
5
5
6
6
Please elaborate on why you rated each as you did.
5
5
6
6
My cell phone
gfedc
Other wireless device
gfedc
Appendix H.
Debrieng Usability Survey

A Usability Evaluation of Academic Virtual Reference Services 357
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
5. Please add any further comments about your perceptions of using Text Messaging.
Please rank order your preference of formats based on Question 1 and Question 2.
1. Based on Question 1 only, rank-order (1=highest, 5=lowest) the five formats based on
your preference.
5
5
6
6
1
2
3
4
5
E-mail nmlkjnmlkjnmlkjnmlkjnmlkj
Telephone nmlkjnmlkjnmlkjnmlkjnmlkj
Chat nmlkjnmlkjnmlkjnmlkjnmlkj
Skype nmlkjnmlkjnmlkjnmlkjnmlkj
Text Messaging nmlkjnmlkjnmlkjnmlkjnmlkj
Please elaborate on the rationale for your rankings:
5
5
6
6
Appendix H.
Debrieng Usability Survey

358 College & Research Libraries May 2014
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
2. Based on Question 2 only, rank-order (1=highest, 5=lowest) the five formats based on
your preference.
3. If you decided to get reference help locating materials for a research project (example:
locating biographical information for an author without using Google), which ONE of the
following options for assistance would you most likely choose first?
1
2
3
4
5
E-mail nmlkjnmlkjnmlkjnmlkjnmlkj
Telephone nmlkjnmlkjnmlkjnmlkjnmlkj
Chat nmlkjnmlkjnmlkjnmlkjnmlkj
Skype nmlkjnmlkjnmlkjnmlkjnmlkj
Text Messaging nmlkjnmlkjnmlkjnmlkjnmlkj
Please elaborate on the rationale for your rankings:
5
5
6
6
Face
-
to
-
Face Consultation
nmlkj
Telephone Consultation
nmlkj
E
-m
ail Reference
nmlkj
Online Chat Reference
nmlkj
Skype Video Reference
nmlkj
Text
-
a
-
Librarian Reference
nmlkj
Other
nmlkj
Please feel free to elaborate:
5
5
6
6
Appendix H.
Debrieng Usability Survey

A Usability Evaluation of Academic Virtual Reference Services 359
Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)Reference Services Usability and Satisfaction Survey (Master)
4. If you needed help finding a quick fact (example: finding the hours for the library for the
week), which ONE of the following options for assistance would you most likely choose
first?
5. Please feel free to leave us any other comments or thoughts you may have about this
project.
Thank you very much for participating in our project. For more information please contact Dr. Anthony Chow at
aschow@uncg.edu.
5
5
6
6
Face
-
to
-
Face Consultation
nmlkj
Telephone Consultation
nmlkj
E
-m
ail Reference
nmlkj
Online Chat Reference
nmlkj
Skype Video Reference
nmlkj
Text
-
a
-
Librarian Reference
nmlkj
Other
nmlkj
Please feel free to elaborate:
5
5
6
6
Appendix H.
Debrieng Usability Survey

360 College & Research Libraries May 2014
Notes
1. Anthony S. Chow and Rebecca A. Croxton, “Information-Seeking Behavior and Reference
Medium Preferences: Dierences between Faculty, Sta, and Students,” Reference & User Services
Quarterly 51, no. 33 (2012): 246.
2. Bernie Sloan, “Twenty Years of Virtual Reference,” Internet Reference Services Quarterly 11,
no. 2 (2006): 91–95.
3. Anthony S. Chow et al., “What Is a Usable Library Website? Results from a Nationwide
Survey” (under review), Reference and User Services Quarterly.
4. Laura K. Probst, “Digital Reference Management: A Penn State Case Study,” Internet Refer-
ence Services Quarterly 10, no. 2 (2005): 43–59.
5. Michel C. Atlas, “Library Anxiety in the Electronic Era, or Why Won’t Anybody Talk to
Me Anymore?” Reference & User Services Quarterly 44, no. 4 (2005): 314–19; R. David Lankes,
“Virtual Reference to Participatory Librarianship: Expanding the Conversation,” Bulletin of the
American Society for Information Science and Technology 34, no. 2 (2008): 11–14; Marie L. Radford
and M. Kathleen Kern, “A Multiple-Case Study Investigation of the Discontinuation of Nine Chat
Reference Services,” Library & Information Science Research 28 (2006): 521–47.
6. Lynn S. Connaway, Marie L. Radford, and Timothy J. Dickey, “On the Trail of the Elusive
Non-user: What Research in Virtual Reference Environments Reveals,” ASIS&T Bulletin 34, no.
2 (2008): 25–28.
7. Anthony S. Chow, “The Usability of Digital Information Environments: Planning, Design,
and Assessment,” in Chandos Digital Information Review, eds. W. Evans and D. Baker (Cambridge:
Chandos, 2012).
8. The International Organization for Standardization 9241-11, “Ergonomic Requirements
for Oce Work with Visual Display Terminals (VDTs), Part II: Guidance on Usability” (Geneva,
Switzerland, 1998).
9. Xiangming Mu et al., “A Survey and Empirical Study of Virtual Reference Service in Aca-
demic Libraries,” Journal of Academic Librarianship 37, no. 2 (Mar. 2011): 126.
10. Char Booth, “Developing Skype-Based Reference Services,” Internet Reference Services
Quarterly 13, no. 2 & 3 (2008): 148; Kelly M. Broughton, “Usage and User Analysis of a Real-
Time Digital Reference Services,” The Reference Librarian 38, no. 29 (2002); Chow and Croxton,
“Information-Seeking Behavior and Reference Medium Preferences”; Joel Cummings, Lara
Cummings, and Linda Frederiksen, “User Preferences in Reference Services: Virtual Reference
and Academic Libraries,” portal: Libraries and the Academy, 7, no. 1 (Jan. 2007): 81–96; Christina M.
Desai, “Instant Messaging Reference: How Does It Compare?” Electronic Library 21, no. 1 (2003);
J.B. Hill, Cherie M. Hill, and Dayne Sherman, “Text Messaging in an Academic Library: Integrat-
ing SMS into Digital Reference,” The Reference Librarian 47, no. 1 (2007); Nora Hillyer and Linda
L. Parker, “Video Reference—It’s Not Your Typical Virtual Reference: Video Reference Services at
the University of Nebraska at Omaha,” Internet Reference Services Quarterly 11, no. 4 (2006): 41–54.
11. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences.”
12. Chow, “The Usability of Digital Information Environments.”
13. Jesse J. Garre, The Elements of the User Experience: User-centered Design for the Web and
Beyond, 2nd ed. (Berkeley, CA.: New Riders, 2011).
14. Cheryl Dee and Maryellen Allen, “A Survey of the Usability of Digital Reference Services
on Academic Health Science Library Websites,” The Journal of Academic Librarianship 32, no. 1
(2006): 69–78.
15. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences.”
16. Jakob Nielsen, “Usability 101: Introduction to Usability,” www.useit.com/alertbox/20030825.
html (accessed June 18, 2012, last modied 2012).
17. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences”;
Jerey Pomerantz and Lili Luo, “Motivation and Uses: Evaluating Virtual Reference from the Users’
Perspective,” Library & Information Research 28 (2006): 5–29; Marie L. Radford et al., “Evaluating
E-Reference: Transforming Digital Reference through Research and Evaluation,” Proceedings of the
American Society for Information Science and Technology 45, no. 1 (2009): 1–11; David Ward, “Why
Users Choose Chat: A Survey of Behavior and Motivations,” Internet Reference Services Quarterly
10, no. 1 (2005): 29–46.
18. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences,”
251.
19. Lynn S. Connaway, Timothy J. Dickey, and Marie L. Radford, “If It Is Too Inconvenient I’m
Not Going aer It: Convenience as a Critical Factor in Information-Seeking Behaviors,” Library
& Information Science Research 33 (2011): 185.
20. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences,” 253.

A Usability Evaluation of Academic Virtual Reference Services 361
21. Desai, “Instant Messaging Reference”; Pomerantz and Luo, “Motivation and Uses”; Radford
et al., “E-valuating E-Reference.”
22. Hill, Hill, and Sherman, “Text Messaging in an Academic Library,” 24.
23. Ibid., 26.
24. Booth, “Developing Skype-Based Reference Services,” 160.
25. Dee and Allen, “A Survey of the Usability of Digital Reference Services.”
26. Beth Thomse-Sco, “If You Ask, I Will Tell You: Future Users of Virtual Reference Share
Their Thoughts on the Design, Operation, and Marketing of Virtual Reference,” in The Virtual
Experience: Integrating Theory into Practice, eds. R. David Lankes, Joseph Janes, Linda C. Smith,
and Christina M. Finneran. (New York: Neal-Schuman, 2004): 63–86.
27. Claire Warwick et al., “Cognitive Economy and Satiscing in Information Seeking: A
Longitudinal Study of Undergraduate Information Behavior,” Journal of the American Society for
Information Science & Technology 60 (2009): 2409.
28. Joseph Janes, “Digital Reference: Reference Librarians’ Experiences and Aitudes,” Journal
of the American Society for Information Science and Technology 53 (2002): 549–66; I.J. Lee, “Do Virtual
Reference Librarians Dream of Virtual Reference Questions? A Qualitative and Quantitative
Analysis of Email and Chat Reference,” Australian Academic and Research Libraries 35 (2004): 95–110;
Joseph E. Straw, “A Virtual Understanding: The Reference Interview and Question Negotiation
in the Digital Age,” Reference & User Services Quarterly 39 (2000): 376–79; Jo Kibbee, David Ward,
and Wei Ma, “Virtual Reference, Real Data: Results of a Pilot Study,” Reference Services Review
30, no. 1 (2002): 35–36; Kirsti Nilsen, “The Library Visit Study: User Experiences at the Virtual
Reference Desk,” Information Research 9, no. 2 (2004), available online at hp://informationr.net/
ir/9-2/paper171.html [accessed 7 January 2013 ].
29. Wendy Diamond and Barbara Pease, “Digital Reference: A Case Study of Question Types
in an Academic Library,” Reference Services Review 29 (2001): 210–19; S.M. Gray, “Virtual Reference
Services: Directions and Agendas,” Reference & User Services Quarterly 39 (2000): 367–75.
30. David Ward, “Measuring the Completeness of Reference Transactions in Online Chats:
Results of an Unobtrusive Study,” Reference & User Services Quarterly 44 (2004): 46–56.
31. Nahyun Kwon, “Public Library Patrons’ Use of Collaborative Chat Reference Service: The
Eectiveness of Question Answering by Question Type,” Library & Information Science Research
29 (2007): 70–91.
32. Lee, “Do Virtual Reference Librarians Dream of Virtual Reference Questions?”
33. Hill, Hill, and Sherman, “Text Messaging in an Academic Library,” 25.
34. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences.”
35. Marguerite Reardon, “Text Messaging Explodes in America,” CBS News, available online
at www.cbsnews.com/stories/2008/09/23/teh/cneechnews/main4471183.shtml [accessed 20 June
2012, last modied 2 April 2009].
36. Booth, “Developing Skype-Based Reference Services,” 157.
37. Ibid.; Chow and Croxton, “Information-Seeking Behavior and Reference Medium Prefer-
ences.”
38. Ward, “Why Users Choose Chat,” 37.
39. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences,”
257.
40. Ibid.
41. Herbert Poole, Theories of the Middle Range (Norwood, N.J.: Ablex, 1985).
42. Nancy J. Young and Marilyn Von Seggern, “General Information Seeking in Changing
Times: A Focus Group Study,” Reference & User Services Quarterly 41, no. 2 (Winter 2001): 159–69.
43. Chow and Croxton, “Information-Seeking Behavior and Reference Medium Preferences.”
44. Patrick Jordan, An Introduction to Usability (Philadelphia, Pa.: Taylor & Francis, 1998).
45. Warwick et al., “Cognitive Economy and Satiscing in Information Seeking.”
46. Poole, Theories of the Middle Range.