Content uploaded by James R. Lewis
Author content
All content in this area was uploaded by James R. Lewis on Feb 16, 2023
Content may be subject to copyright.
IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use
International Journal of Human-Computer Interaction
Abstract
This paper describes recent research in subjective usability measurement at IBM. The focus of the research was the application of psychometric methods to the development and evaluation of questionnaires that measure user satisfaction with system usability. The primary goals of this paper are to (1) discuss the psychometric characteristics of four IBM questionnaires that measure user satisfaction with computer system usability, and (2) provide the questionnaires, with administration and scoring instructions. Usability practitioners can use these questionnaires with confidence to help them measure users' satisfaction with the usability of computer systems.
... To evaluate the usability of Build_PC, a survey based on the IBM-CSUQ tool was used, which employs a 7-point Likert scale, where 1 indicates strong disagreement or very unlikely, and 7 indicates strong agreement or very likely. [61,62]. This questionnaire consists of 19 questions designed to measure user satisfaction with the developed application [61]. ...
The proliferation of mobile applications for educational purposes has highlighted the need to evaluate their usability, especially in diverse international contexts. This study addresses the problem of insufficient engagement and effectiveness of educational tools related to PC hardware training, a problem exacerbated by cultural and contextual differences between regions. Understanding the importance of this issue is crucial, as effective educational tools can improve learning outcomes on a global scale. Previous research has explored various educational technologies but often failed to comprehensively address usability across different cultural contexts, limiting the generalization and impact of the results. This gap underscores the need for robust evaluation of educational applications in diverse populations. In this context, our research proposes the analysis of Build_PC, a mobile augmented reality (MAR) application designed to teach PC hardware, using the IBM Computer System Usability Questionnaire (CSUQ) to assess user satisfaction. This study was conducted in three universities from three countries—Ecuador, Indonesia, and Lebanon—covering a variety of cultural and educational settings. The results indicate remarkably high levels of user satisfaction with the augmented reality (AR) application across the three participating universities. Positive feedback suggests that the application effectively engages students and improves their understanding of PC hardware training, regardless of regional differences. The implications of these findings are significant, as they suggest that augmented reality applications may be a viable solution for overcoming educational barriers related to PC hardware training on an international scale. This study highlights the potential of such technology to enhance educational outcomes and provides a framework for future research in the global deployment of educational technologies. Doi: 10.28991/ESJ-2025-09-02-024 Full Text: PDF
... The need for understanding the contribution of narrative to user experience was both inspired by literature surveyed as well as outcomes from the design workshop where the scenarios were developed and tested. The structure and wording of these questions were inspired by related questions in the NASA TLX, 45 the PQ, 35 the CSUQ, 46 and work by Slater and Wilbur 36 around importance of plot. These additional questions were not empirically validated, as they were used primarily to interpret the qualitative data. ...
Objectives
Disability support workers play a critical role in supporting people living with disabilities, but may have limited formal training as time, resources, and funding constraints can make access to training difficult for this workforce. To prepare them for challenging work situations, virtual reality (VR) offers a safe, self-paced, and practice-based way to do self-motivated training to learn and practice relevant workplace skills. The aim of this study was to identify those aspects of a VR training experience that engages disability support workers, motivating them to choose it to build the skills and capabilities needed when working with individuals with disability who require positive behavior support (PBS).
Methods
To understand what creates an engaging user experience for disability support workers, we studied two virtual learning environments in use, both designed to teach PBS skills, with 21 health industry students and 29 employed disability support workers. Observations, a survey, and structured interviews were used to measure user experience and collect user perceptions.
Results
Analysis of collected data identified four key aspects affecting user experience in virtual learning environments for this particular cohort, narrative, environment, interaction, and immersion, finding that narrative, both relevance of the storyline and empathy with the virtual character, was key to engaging disability support workers in the VR training.
Conclusions
VR can be an effective tool for motivating disability support workers to do personal training if it is designed so that they find the virtual client engaging and the storyline meaningful and relevant to their workplace experiences.
... Una de las aristas evaluadas se relacionó con la usabilidad del software. Para ello, se utilizó el "Cuestionario posterior-al escenario", propuesto por Lewis (1995), que incluye las siguientes preguntas: ...
El contexto actual de la educación superior exige una actualización constante de los planes de estudio para satisfacer las demandas sociales. Esto requiere procesos eficaces en el diseño y modificación curricular. Además, la incorporación de las tecnologías de información facilita la gestión académica y administrativa de los planes de estudio al permitir un análisis adecuado para la toma de decisiones. Entre 2019 y 2023, se desarrolló un software de mapeo curricular basado en resultados de aprendizaje (RA), con el objetivo de apoyar el diseño y la actualización de los planes de estudio. Este software utiliza la taxonomía revisada de Bloom (TRB), está orientado a programas de grado y posgrado; y, facilita el alineamiento de los componentes del currículo con base en RA. Su desarrollo siguió una metodología de ciencias del diseño, con enfoque mixto, y fue evaluado por tres personas expertas, quienes destacaron resultados positivos sobre su usabilidad, su utilidad y su pertinencia. Es importante aclarar que el software no está destinado para utilizarse en el aula, por lo que el personal docente y el estudiantado no son usuarios directos. En cambio, su propósito es apoyar a los equipos de asesoría y comisiones curriculares en la creación y actualización de planes de estudio. Se concluyó que el software integra funcionalidades esenciales que permiten implementar el mapeo curricular y comprobar el alineamiento entre los componentes del currículo y los resultados de aprendizaje (RA) en los planes de estudio. Además, le ofrece a la persona usuaria funcionalidades para la correcta redacción de los RA tanto del plan de estudio como de cada asignatura.
... Secondly, we used the Computer System Usability Questionnaire (CSUQ) [25] to test the application's perceived quality in terms of functionality and information. The full version of the CSUQ is shown in Table 3. ...
Micro-learning has been widely adopted in recent years as an approach to improve online learning, particularly on mobile applications. Micro-learning is a method of dividing learning content into smaller lessons that can be completed within a short period (e.g., from ten to 15 minutes). While micro-learning is often seen as a way to boost user engagement, strategies to optimize engagement within these applications remain underexplored. This study aims to examine the impact of notification delay times to enhance user engagement in language learning activities within micro-learning platforms. Specifically, we developed a notification system designed to encourage users to transition from social media usage to educational activities: in this case, the Duolingo application. Over a three-week period, we conducted a within-subject experiment with 10 different users, with each week dedicated to one of three notification delay times - the period of time our proposed system waits before sending a notification once a social media usage activity is recognized: immediate (0 minutes), 15 minutes, and 30 minutes. Our findings reveal that notifications sent after a delay of 15 and 30 minutes - significantly increase both the response rate and the duration of learning sessions compared to immediate notifications. This suggests that a strategic delay in sending study reminders can substantially enhance engagement and prolong the duration of learning, offering valuable insights into designing educational technologies that foster sustained learning habits.
... Lastly, in each session, the overall usability of the device was evaluated through the validated System Usability Scale (SUS), which is one of the most commonly used usability assessment questionnaires, 17,18 and satisfaction was assessed using the After-Scenario Questionnaire (ASQ), in which test participants responded using Likert Scales from 1-5 and 1-7, respectively. 19 Written feedback was obtained. Oral interview questions were administered after the interface use if the test leader or observer noticed that the participant had completed a task incorrectly, nearly made an error, or showed use di culties. ...
Importance The integration of clinical decision support tools in medical practice is challenging and must be carefully undertaken, especially in cancer management. Noninvasive Lymph Node Status (NILS) is a web-based tool designed to estimate the probability of healthy axillary lymph nodes in female patients with breast cancer scheduled for primary surgery. Objective To identify barriers to NILS adoption in a clinical setting, and assess whether intended users can operate the tool without signi cant errors or di culties. Design This mixed-methods qualitative study used simulated clinical cases, the System Usability Scale (SUS), and the After-Scenario Questionnaire (ASQ) to evaluate usability and satisfaction. An oral interview was conducted after the evaluations. The study followed a structured protocol, with distinct roles assigned to the test participants, leader, and observer. Setting A multicenter on-site usability study was conducted in a simulated clinical environment, replicating both real-world physical and digital conditions. Participants Based on the identi ed target user population for NILS, twenty physicians comprised the cohort. Exposure Web interface interaction using simulated clinical cases. Main Outcomes and Measures Physicians' perceptions of the NILS as a decision-support tool for determining whether to perform or abstain from sentinel lymph node biopsy in a simulated clinical setting. Results Twenty physicians (15 [65%] female; 15 [75%] surgeons; 5 [25%] oncologists; median 9.5 years in specialist-practice) working in four different hospitals participated. Usability scores were high, with a mean SUS score of 89.5 ("excellent") and a mean ASQ score of 6.3 (Likert scale 1-7). Several interface challenges were identi ed, including di culty in locating the reset and information buttons, the ability to enter values outside valid ranges (age and tumor size), and the risk of unintentionally modifying the entered values while scrolling. Conclusions and Relevance This study identi ed the key usability factors, barriers, and facilitators affecting NILS implementation. Physicians found the NILS interface easy to use and valued the presented risk estimates of healthy axillary lymph node status in their decisions to perform or abstain from sentinel lymph node biopsy. Redesign initiatives are ongoing. An iterative process of addressing usability in a clinical setting is crucial for successful implementation. Trial Registration ISRCTN99301435 Key Points Question How do physicians perceive the NILS tool as a clinical decision support tool that provides a noninvasive probability of healthy axillary lymph nodes in early breast cancer? Findings Surgeons and oncologists identi ed several actionable items to enhance usability and work ow integration of the NILS tool. Most physicians found the tool useful for decisions on performing or abstaining from sentinel lymph node biopsy and that it was ready for clinical implementation. Meaning Physicians have identi ed key improvements to facilitate the NILS model implementation while recognizing its value in estimating the healthy axillary lymph node status in early breast cancer.
Background and Aim
Telemedicine has gained prominence as a remote healthcare delivery method. In regions with limited access to medical services, telemedicine bridges the gap by providing remote consultations. In India, the government introduced the e-Sanjeevani platform for delivering telemedicine services in two modalities: cloud-based as well as assisted telemedicine systems. This study introduces the Indian Telemedicine Assessment Scale (I-TASC), a tool designed to assess the usability and effectiveness of the assisted telemedicine platform from the perspective of healthcare provider as the end user.
Materials and Methods
The development of I-TASC consisted of four phases: literature review, construct development, item formulation, and reliability assessment. Existing telemedicine questionnaires were reviewed for developing I-TASC. The scale encompasses five constructs: ease of use, learning, interface quality, usefulness, and satisfaction. A 15-item questionnaire was formed and validated with 56 healthcare providers. Cronbach’s alpha assessed reliability.
Results
The instrument’s reliability, measured by Cronbach’s alpha score, demonstrated strong internal consistency within each of the five domains (α >0.8), with excellent overall reliability (α >0.9).
Conclusion
I-TASC emerges as a reliable tool for assessing assisted telemedicine systems. Leveraging established telehealth measures, it exhibits strong content validity. The I-TASC is a robust tool, promising substantial value for assessing healthcare service delivery using an assisted telemedicine system.
Captchas are used as Human interaction proof mechanisms during the authentication process on software applications. They should provide resistance to various attacks to increase security but also be understood easily to ensure usability. Increasing the security generally reduces usability, so it is necessary to use captchas that will meet both the security and usability needs of users balanced. Within the scope of this study, a text‐based captcha scheme that end‐users commonly encounter during their daily interactions in mobile applications is selected and investigated to determine both a more robust and usable one for users. Six different text‐based captcha types, which were distortion‐based, non‐distortion‐based, dictionary‐based, random‐based, low contrast, and full contrast, were compared in terms of security and usability. Initially, security tests were applied. Afterwards, user tests were conducted with 30 participants. According to security test results, distortion, low contrast, and random‐based captcha types were determined to be more robust, respectively. The most usable captcha type among the secure captcha types was determined as a random string captcha based on the user test results. Thus, it has been found that a balanced level of security and usability can be achieved when mobile application developers choose to use a random string captcha when designing interfaces. Recommendations to guide mobile interface developers were provided based on the findings obtained both from the user study and previous relevant literature.
Usability evaluators used an 18-item, post-study questionnaire in three related usability tests. I conducted an exploratory factor analysis to investigate statistical justification to combine items into subscales. The factor analysis indicated that three factors accounted for 87 percent of the total variance. Coefficient alpha analyses showed that the reliability of the overall summative scale was .97, and ranged from .91 to .96 for the three subscales. In the sensitivity analyses, the overall scale and all three subscales detected significant differences among the user groups; and one subscale indicated a significant system effect. Correlation analyses support the validity of the scales. The overall scale correlated highly with the sum of the After-Scenario Questionnaire ratings that participants gave after each scenario. The overall scale also correlated moderately with the percentage of successful scenario completion. These results are consistent with the hypothesis that these alternative measurements tap into a common underlying construct. This construct is probably usability, based on the content of the questionnaire items and the measurement context.
This study is a part of a research effort to develop the Questionnaire for User Interface Satisfaction (QUIS). Participants, 150 PC user group members, rated familiar software products. Two pairs of software categories were compared: 1) software that was liked and disliked, and 2) a standard command line system (CLS) and a menu
driven application (MDA). The reliability of the questionnaire was high, Cronbach’s alpha=.94. The overall reaction ratings yielded significantly higher ratings for liked software and MDA over
disliked software and a CLS, respectively. Frequent and sophisticated PC users rated MDA more satisfying, powerful and flexible than CLS. Future applications of the QUIS on computers are
discussed.
The Subjective Workload Assessment Technique (SWAT) has been under development for approximately five years. This measure is under a systematic development program to define its' strengths and weaknesses. Both laboratory research and field applications are being employed in this evaluation and some of the findings are presented. Current research on refinements to the procedure are discussed.
This chapter discusses the conduct of research to guide the development of more useful and usable computer systems. Experimental research in human-computer interaction involves varying the design or deployment of systems, observing the consequences, and inferring from observations what to do differently. For such research to be effective, it must be owned—instituted, trusted and heeded—by those who control the development of new systems. Thus, managers, marketers, systems engineers, project leaders, and designers as well as human factors specialists are important participants in behavioral human-computer interaction research. This chapter is intended as much for those with backgrounds in computer science, engineering, or management as for human factors researchers and cognitive systems designers. It is argued in this chapter that the special goals and difficulties of human-computer interaction research make it different from most psychological research as well as from traditional computer engineering research. The main goal, the improvement of complex, interacting human-computer systems, requires behavioral research but is not sufficiently served by the standard tools of experimental psychology such as factorial controlled experiments on pre-planned variables. The chapter contains about equal quantities of criticism of inappropriate general research methods, description of valuable methods, and prescription of specific useful techniques.
Determining the number of common factors is one of the most important decisions which must be made in the application of factor analysis. Several different approaches and techniques are reviewed here along with associated strengths and weaknesses. It is argued that a combination of approaches will lead to the best judgment regarding the number of factors to retain. A computer program is available which presents the number of factors to retain as suggested by both discontinuity and parallel analyses. Utilization of the program removes the negative aspect associated with the use of each technique.