ChapterPDF Available

Usability Testing

  • February 2006
DOI:10.1002/0470048204.ch49
  • In book: Handbook of Human Factors and Ergonomics, Third Edition (pp.1275 - 1316)
  • Edition: 3rd
  • Chapter: 49
  • Publisher: John Wiley
Authors:
James R. Lewis at MeasuringU
James R. Lewis
  • MeasuringU
Download full-text PDFDownload full-text PDF
Read full-text
Download citation

Abstract and Figures

Usability testing is an essential skill for usability practitioners -- professionals whose primary goal is to provide guidance to product developers for the purpose of improving the ease of use of their products. It is by no means the only skill with which usability practitioners must have proficiency, but it is an important one. Surveys of experienced usability practitioners have indicated that usability testing is a very frequently used method, second only to the use of iterative design. One goal of this chapter is to provide an introduction to the practice of usability testing. This includes some discussion of the concept of usability and the history of usability testing, various goals of usability testing, and running usability tests. A second goal is to cover more advanced topics, such as sample size estimation for usability tests, computation of confidence intervals, and the use of standardized usability questionnaires.
Figures - uploaded by James R. Lewis
Author content
All figure content in this area was uploaded by James R. Lewis
Content may be subject to copyright.
5a18fd424585155c26a95e
a1.pdf
Content uploaded by James R. Lewis
Author content
All content in this area was uploaded by James R. Lewis on Nov 25, 2017
Content may be subject to copyright.
5a18fd410f7e9be37f9766
2b.pdf
Content uploaded by James R. Lewis
Author content
All content in this area was uploaded by James R. Lewis on Nov 25, 2017
Content may be subject to copyright.
... Usability testing for a particular, already existing, components in the DEMETER platform were conducted. Usability testing is a frequently used method to assess a product's ease-of-use (Lewis, 2006). The assessment was based on DIN ISO 9241-11, which specifies the requirements for the usability of interactive products according to the international standard. ...
Social and technological potential of on-farm experiments and demonstrations to promote innovative farming practices
Conference Paper
Full-text available
  • Oct 2021
Research-based on-farm trials and demonstrations are becoming a norm in agricultural research and extension involving collaboration between scientists, advisors, and farmers. There is, however, a rather large variation in the engagement models underlying this farmer participatory research and extension depending on the form and intensity of farmer involvement. While the very idea of farmer engagement is laudable and is increasingly promoted by various policy instruments, it is important also to critically look at these multi-actor processes of knowledge co-production and the existing areas of contention not to over-romanticize these collaborative arrangements by identifying, acknowledging and addressing the various challenges that come with it. Collaborative arrangements involved in on-farm trials and demonstrations can be demanding in terms of aligning and integrating the different needs, expectations, skills, experiences and knowledge resources possessed by the involved parties, not least in terms of specifying the research problem and in ensuring a common and shared framework for the practical implementation of the trial, data gathering and communication. Yet, there are still many benefits that compensate for these hardships when considering the social and technological potential of on-farm experiments and demonstrations to promote innovative farming practices.
View
... III. Test de Usabilidad para usuarios finales: Cuestionario que pregunta las mismas preguntas a un grupo de gente para obtener datos relacionados con sus experiencias, opiniones, intereses, etc. (Borg et al, 2003). También provee objetividad, replicabilidad, cuantificación, economía, comunicación, y genera-lización científica (Lewis, 2006). Este cuestionario basado en respuesta de tipo Likert fue aplicado a dos tipos de usuarios finales, estudiantes, y un profesor. ...
Aprendizagem Baseada em Projetos e Simulação Computacional para o Ensino de Engenharia
Conference Paper
Full-text available
  • Dec 2022
As novas Diretrizes Curriculares Nacionais de Engenharia e as demandas do mundo do trabalho apresentam-se como desafios às Instituições de Ensino Superior, que precisam rever seus processos de ensino. Nesse contexto, identifica-se a necessidade de implementar uma abordagem metodológica para o desenvolvimento de competências em um curso de Engenharia de Produção em uma universidade pública, mais especificamente no componente curricular Simulação. A proposta tem como base a Aprendizagem Baseada em Projetos e o uso de um software de Simulação Computacional, os quais possibilitaram a aproximação dos estudantes a problemas reais e o desenvolvimento de competências para a Modelagem e Simulação. Foi utilizada a avaliação por Rubricas, cujos resultados indicam que os objetivos de aprendizagem foram atingidos. Ainda, observam-se relatos dos estudantes que avaliam a abordagem como uma oportunidade para praticar seus conhecimentos.
View
View
Examining the Usability of e-Content in Different Forms for Increasing Digital Parenting Competencies
Article
Full-text available
  • Jan 2023
A usability test during the development of learning material can reduce many usability problems. This can increase the quality and efficiency of the material. In future studies, design principles that can guide e-content development can be established. More efficient learning material can be created by developing e-contents in the context of these design principles. The purpose of this study is to evaluate the usability of e-contents developed to increase parents’ digital parenting competencies and awareness. In this study, a usability test was created for online content. Materials for video and visual aided written material usability were designed and used for this study, including an online content usability questionnaire. Field experts’ opinions were taken and usability tests were conducted with real users. These usability tests were developed by Alexander (2013). This study follows usability principles to determine the usability of various online videos and visually supported materials to increase parents’ digital parenting competencies and awareness. Usability test results were revealed through descriptive statistics. This research helps parents better understand the possibilities, potentials, and limitations in designing educational online content on digital parenting. On the other hand, this usability test has the potential to detect the usability of educational videos in different fields.
View
View
Effects of design features of train electronic guidance interface on visual search, behavior, and usability
Article
In urban rail trains, traditional printed guide information interfaces are being replaced by digital interfaces. An electronic guide screen is an important device for passengers to obtain information about the ride because the train is running underground. In particular, when the train is crowded, passengers must obtain information in advance and make appropriate decisions. Therefore, it is important to efficiently search for guidance interfaces. In this study, use scenarios of the guidance interface were simulated. The typical information layout and display mode in the current electronic guidance interface were the experimental variables. Through eye tracking and subjective evaluation, the effects of design features on user search performance, visual behavior, and usability satisfaction were discussed. The experimental results showed that the information layout and display mode of the guidance interface had an interactive effect on the task completion time, fixation counts, and usability score; they also had a significant effect on the recall accuracy and saccade ratio. Task completion time had a strong positive correlation with usability satisfaction and was correlated with visual behavior. The study confirmed that the split-screen display mode combined with interlaced layout could improve search performance and subjective satisfaction. Relevance to industry Printed guide signs in subway cars are being replaced by electronic guide screens of different styles. Passengers must search for the required information within a limited period. In this study, the effects of different interface layouts and display modes on search performance, visual behavior, and usability were studied. This had a positive significance in improving user experience.
View
Archives in action. The impact of digital technology on archaeological recording strategies and ensuing open research archives
Article
Full-text available
  • Sep 2022
Digital-born research archives, data re-use, participation and the inclusion of academic and lay stakeholders in archaeological knowledge production. These are important topics that are increasingly addressed but often overlooked in the creative stages of archiving, be it data collection or the reproduction of an archiving practice. This creative practice is affected and changing due to the implementation of digital technology. These practices are reproduced in the design of the research archive and, as such, the impact of technology can potentially be scrutinised and traced reversely by analysing the uses of the archive. In addition, digital technology is believed to prompt greater inclusivity of diverse audiences. But how to reach that audience, and who is this “audience”? In this paper, emphasis is placed to reflect upon the practice of archiving of ongoing, post-excavation archaeological research with an audience, as opposed to well-established reflexive research into excavation and museum practices. As such, the concept of archiving as research process, rather than the traditional approach towards archives as data repositories is introduced here. As a case study to identify and assess potential change in a particular archaeological practice, the paper describes and analyses the archiving practice of the team of the Tracing the Potter's Wheel-project, from its inception in offices and storerooms to the archive's targeted manifold use as a place of knowledge production, data sharing and learning.
View
View
View
Usability Evaluation Using Unmoderated Remote Usability Testing on Angkasa LMS Website Case Study
Chapter
  • Jan 2023
The pandemic has made digital transformation faster, and Indonesia is no exception, especially in education. Learning management system (LMS) is a learning media that is widely used in educational institutions. However, not all educational institutions have sufficient resources to build an LMS from scratch. “Angkasa LMS” Web is a Web that allows educational institutions to order a ready-to-use LMS easily. The target users of this service are pretty varied because they consist of education management and educational management foundations, especially in areas of Indonesia that have Internet access. To be used easily by these varied target users, this Website must have good usability in its user interface. For this reason, usability testing needs to be carried out in order to get feedback for improving the user interface design before the application is delivered to the public. However, the developer has obstacles related to the COVID-19 pandemic policy to carry out usability testing, which limits direct interaction with people. Moreover, the project time is quite narrow, and the schedule is quite tight for the developer team. Based on these limitations, this study conducted usability testing of the “Angkasa LMS” Web case study using the unmoderated remote usability testing method. The experimental results show that this method can be used to obtain insightful feedback from the participants, with additional treatment such as the use of convenience sampling, periodic reminders for participants, and the increasing number of participants exceeding the target.
View
Evaluation of Graphic Symbols for Phone and Line
Technical Report
Full-text available
  • Feb 2023
Two studies were conducted by the Design Center/Human Factors department to evaluate the effectiveness of graphic symbols for phone and phone line connection for attaching telephony equipment to a computer. In the first study, computer-naive participants were asked to attach a telephone to a computer with only the proposed graphic symbols to guide the installation. Nine of eleven installations (82 percent) were incorrect. The 95-percent confidence interval for this percentage ranged from 48 percent to 98 percent, and was considered to be unacceptably high. The second study was identical to the first, except that a wordless setup illustration was provided that showed how to attache the telephone to the computer correctly. With this sheet as a performance aid, ten out of ten installations were correct. The 95-percent confidence interval for incorrect installations derived from this result ranged from 0 percent to 25 percent. Although it is possible due to the small sample size of this study that the failure rate could be as high as 25 percent, it is extremely unlikely. Including the setup sheet during the installation significantly improved the successful installation rate, the time to install, and the user satisfaction with the procedure. It is recommended, if these symbols are used with computers, that an installation sheet be provided that clearly illustrates the correct connection of phone and line. Also, the documentation should repeat the illustration in a section describing system relocation.
View
Sample Sizes for Usability Studies: Additional Considerations
Article
Full-text available
Recently, Virzi (1992) presented data that support three claims regarding sample sizes for usability studies: (1) observing four or five participants will allow a us-ability practitioner to discover 80% of a product's usability problems, (2) observing additional participants will reveal fewer and fewer new usability problems, and (3) more severe usability problems are easier to detect with the first few participants. Results from an independent usability study clearly support the second claim, partially support the first, but fail to support the third. Problem discovery shows diminishing retums as a function of sample size. Observing four to five participants will uncover about 80% of a product's usability problems as long as the average likelihood of problem detection ranges between 0.32 and 0.42, as in Virzi. If the average likelihood of problem detection is lower, then a practitioner will need to observe more than five participants to discover 80% of the problems. Using behavioral categories for problem severity (or impact), these data showed no correlation between problem severity (impact) and rate of discovery. The data provided evidence that the binomial probability formula may provide a good model for predicting problem discovery curves, given an estimate of the average likelihood of problem detection. Finally, data from economic simulations that estimated return on investment (ROI) under a variety of settings showed that only the average likelihood of problem detection strongly influenced the range of sample sizes for maximum ROI.
View
Reaping the Benefits of Modern Usability Evaluation: The Simon Story
Conference Paper
Full-text available
  • May 1996
Simon (TM-Bellsouth Corp.) is a commercially available personal communicator (PC) combining features of a PDA (personal digital assistant) with a full suite of communications features. This paper describes the involvement of human factors engineering in the development of Simon, and summarizes the various approaches to usability evaluation employed during its development. Simon has received a considerable amount of praise from the industry and won several industry awards, with recognition both for its innovative engineering and its usability.
View
View
Response Styles in Marketing Research: A Cross-National Investigation
Article
Full-text available
  • May 2001
Response styles are a source of contamination in questionnaire ratings, and therefore they threaten the validity of conclusions drawn from marketing research data. In this article, the authors examine five forms of stylistic responding (acquiescence and disacquiescence response styles, extreme response style/response range, midpoint responding, and noncontingent responding) and discuss their biasing effects on scale scores and correlations between scales. Using data from large, representative samples of consumers from 11 countries of the European Union, the authors find systematic effects of response styles on scale scores as a function of two scale characteristics (the proportion of reverse-scored items and the extent of deviation of the scale mean from the midpoint of the response scale) and show that correlations between scales can be biased upward or downward depending on the correlation between the response style components. In combination with the apparent lack of concern with response styles evidenced in a secondary analysis of commonly used marketing scales, these findings suggest that marketing researchers should pay greater attention to the phenomenon of stylistic responding when constructing and using measurement instruments.
View
View
View
View
Chapter 29. Usability Inspection Methods
Chapter
  • Dec 1997
This chapter describes the costs and benefits associated with a collection of non-empirical methods for evaluating user interfaces, collectively called usability inspection methods. These methods are considered non-empirical because rather than collecting data or observing users interacting with a system, they rely on the ability of judges who attempt to predict the kinds of problems users will experience with a user interface. While on the face of it this may seem to be pure folly, in practice variants of the technique have been demonstrated to be both cost-effective and reliable ways of evaluating usability in some cases. This chapter presents the variety of techniques that fall under the umbrella term of usability inspection. Following this, research reports are reviewed. The chapter presents the considerations for the practitioners considering using one of these techniques.
View
View