James R. Lewis

James R. Lewis
MeasuringU

PhD

About

185
Publications
207,318
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
9,213
Citations
Citations since 2016
30 Research Items
6191 Citations
201620172018201920202021202202004006008001,0001,200
201620172018201920202021202202004006008001,0001,200
201620172018201920202021202202004006008001,0001,200
201620172018201920202021202202004006008001,0001,200
Introduction
James R. (Jim) Lewis, PhD, is a Distinguished User Experience Researcher at MeasuringU. He has published influential papers on the assessment of perceived usability and sample size estimation for usability studies, and has product experience in the design and evaluation of user interfaces (graphical, spoken, mobile), including having been the lead interaction designer for the product widely regarded as the first smart phone, the Simon. He is on >90 US patents and has >100 publications.
Additional affiliations
January 2020 - present
MeasuringU
Position
  • Researcher
Description
  • Primary responsibilities are to conduct applied and fundamental research in the measurement of user experience, write/publish articles and books, conduct advanced statistical analyses of UX data, develop webinar and training material.
April 1981 - December 2019
IBM
Position
  • Engineer
Description
  • At IBM, Jim's primary focus was on the design and evaluation of user interfaces (graphical, spoken, mobile). He became an internationally recognized expert in usability testing and measurement after publishing influential papers in the assessment of satisfaction with computer systems and sample size estimation for usability studies. He has decades of experience in all aspects of speech system UX design, and was the lead interaction designer for the Simon, recognized as the first smart phone.
Education
September 1993 - December 1996
Florida Atlantic University
Field of study
  • Experimental Psychology (Psycholinguistics)
September 1978 - March 1981
New Mexico State University
Field of study
  • Experimental Psychology (Engineering Psychology, Applied Statistics)
September 1975 - August 1978
New Mexico State University
Field of study
  • Psychology

Publications

Publications (185)
Article
Full text available at: https://uxpajournal.org/comparison-of-select-all-forced-choice-items/ | Lewis, J. R., & Sauro, J. (2021). Comparison of select-all-that-apply items with yes/no forced choice items. Journal of Usability Studies, 17(1), 21-30. | Prior research has suggested that people tend to select more items when presented with a forced cho...
Article
In response to recent criticism of the usefulness of the construct of usability, we investigated the relationships between measures of perceived usability and the components of a modified version of the Technology Acceptance Model (mTAM) – Perceived Usefulness (PU) and Perceived Ease-of-Use (PEU). In three surveys, respondents used SUS, UMUX-LITE a...
Article
Objective To assess versions of the shorter form variant of Usability Metric for User Experience (UMUX-LITE) questionnaire differing in the number of response options for the items (3, 5, 7, and 11). Background The UMUX-LITE is an efficient (two-item) standardized questionnaire that measures perceived usability. A growing body of evidence shows it...
Book
Using the PSSUQ and CSUQ in User Experience Research and Practice <https://www.amazon.com/USING-PSSUQ-CSUQ-Experience-Research/dp/1733339205> is a book is for anyone who has a vested interest in assessing and improving user experiences, including UX researchers, designers, and product owners. This book draws upon my experience in the late 1980s dev...
Article
The purpose of this research was to investigate the effect of manipulating item formats for a revised version of the Technology Acceptance Model (TAM) questionnaire. The TAM has 12 items, six assessing perceived usefulness (PU) and six assessing perceived ease of use (PEU). Its original purpose was to quantify likelihood of technology acceptance, u...
Article
Full-text available
ABSTRACT This article aims to investigate the Grand Challenges which arise in the current and emerging landscape of rapid technological evolution towards more intelligent interactive technologies, coupled with increased and widened societal needs, as well as individual and collective expectations that HCI, as a discipline, is called upon to address...
Article
This research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gm...
Article
Full-text available
The current study was an investigation of the extent to which items constructed with an agreement format were affected by acquiescence bias relative to responses to a matched set of items designed with item-specific formats. If an acquiescence bias existed, the expectation was that the magnitude of responses to agreement items would be consistently...
Article
Full-text available
In 2018, Noam Tractinsky published a provocative paper entitled, “The usability construct: A dead end?” He argued the following: • Usability is an umbrella concept. • There is a mismatch between the construct of usability and its empirical measurements. • Scientific progress requires unbundling the usability construct and replacing it with well-...
Article
Full-text available
From humble beginnings, the SUS has become a valuable tool in the toolkits of usability and user experience practitioners and researchers. We have developed regression equations that compute benchmarks for SUS items based on an overall SUS score. A review of the SUS literature on published benchmarks for the means of overall SUS scores from usabili...
Article
The System Usability Scale (SUS) is the most widely used standardized questionnaire for the assessment of perceived usability. This review of the SUS covers its early history from inception in the 1980s through recent research and its future prospects. From relatively inauspicious beginnings, when its originator described it as a “quick and dirty u...
Article
Full-text available
The objectives of this research were to (1) evaluate and compare two versions of the expanded Mean Opinion Scale, one using the original 15 items (MOS-X) and the other a four-item version with one item for each of the four factors of the MOS-X (the MOS-X2), and (2) establish preliminary benchmarks for the interpretation of ratings collected using t...
Article
The primary purpose of this research was to investigate the relationship between two widely used questionnaires designed to measure perceived usability: the Computer System Usability Questionnaire (CSUQ) and the System Usability Scale (SUS). The correlation between concurrently collected CSUQ and SUS scores was 0.76 (over 50% shared variance). Afte...
Article
Full-text available
There are times when user experience practitioners might consider using the System Usability Scale (SUS), but there is an item that just doesn't work in their context of measurement. For example, the first item is "I think I would like to use this system frequently." If the system under study is one that would only be used infrequently, then there...
Article
Full-text available
There is a large body of work on the topic of the optimal number of response options to use in multipoint items. The takeaways from the literature are not completely consistent, most likely due to variation in measurement contexts (e.g., clinical, market research, psychology) and optimization criteria (e.g., reliability, validity, sensitivity, ease...
Article
Full-text available
In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather than being a unidimensional measure of perceived usability, the System Usability Scale apparently had two factors: Usability (all items except 4 and 10) and Learnability (Items 4 and 10). In that paper, we called for other researchers to re...
Chapter
To assess the precision of an estimate, compute its confidence interval. Use confidence intervals around all point estimates to understand the plausible range of the unknown population mean or proportion. Computing a confidence interval requires four things: an estimate of the mean, an estimate of the variability (derived from the sample standard d...
Chapter
Because many usability practitioners deeply depend on the use of measurement and statistics to guide their design recommendations, they inherit these controversies. In this chapter we summarize both sides of each issue and discuss what we, as pragmatic usability practitioners, recommend.
Chapter
When planning user studies, practitioners should estimate the sample sizes required to meet their goals. This is most critical when the cost of adding to a sample is high. When estimating sample sizes, the appropriate method depends on the type of study. This chapter covers methods for user studies (such as summative usability studies) that use mea...
Chapter
This chapter covers methods for determining whether a design has met or exceeded a specific goal. The statistical test you use for completion rates depends on the sample size: A sample size is considered small unless you have more than 15 successes and 15 failures. For determining whether a certain percentage of users can complete a task for small...
Chapter
The previous chapters have covered the fundamentals (information and analytical methods) that practitioners need to conduct usability testing and other user research. There is a world of more advanced statistical methods that can inform user research. This chapter provides an introduction to correlation, regression, and ANOVA.
Chapter
The primary purpose of this book is to provide a statistical resource for those who measure the behavior and attitudes of people as they interact with interfaces. Our focus is on methods applicable to practical user research, based on our experience, investigations, and reviews of the latest statistical literature. As an aid to the persistent probl...
Chapter
User research is a broad term that encompasses many methodologies, such as usability testing, surveys, questionnaires, and site visits, that generate quantifiable outcomes. Usability testing is a central activity in user research and typically generates the metrics of completion rates, task times, errors, satisfaction data, and user interface probl...
Chapter
The goal of this chapter is to cover methods for determining if a difference is statistically significant and how large or small of a difference likely exists in the untested population. It is important to account for chance differences when comparing two designs or products. To do this, you need to find a p-value from the appropriate statistical t...
Chapter
Please note that this is a book chapter that I am contractually prohibited from distributing. You can see a preview from the second edition of Quantifying the User Experience at https://www.amazon.com/Quantifying-User-Experience-Second-Statistics/dp/0128023082/ -- that said, here is the chapter's abstract -- Standardized usability questionnaires a...
Chapter
This chapter contains a summary of the contents of the book plus information on where to get more information about other, typically more advanced, statistical methods applied to user research. It also provides a list of resources for monitoring future developments in practical statistics for user research.
Book
A step-by-step guide for calculating the most common statistical procedures in user research using Excel or R. With over 100 examples pulled from the 2nd Edition of Quantifying the User Experience (Morgan-Kaufmann) we provide the reader with the right methods for comparing means and proportions, generating confidence intervals, finding the right sa...
Book
Quantifying the User Experience: Practical Statistics for User Research, Second Edition, provides practitioners and researchers with the information they need to confidently quantify, qualify, and justify their data. The book presents a practical guide on how to use statistics to solve common quantitative problems that arise in user research. It ad...
Article
The Speech User Interface Service Quality (SUISQ) questionnaire is a standardized instrument for the assessment of the usability of interactive voice response (IVR) applications, developed by Polkosky (Toward a social-cognitive psychology of speech technology: affective responses to speech-based e-service, 2005; Mediated interpersonal communication...
Conference Paper
Full-text available
This paper presents perspectives from both academia and practice on how both groups can collaborate and work together to create synergy in the development and advancement of human-computer interaction (HCI). Issues and challenges are highlighted, success cases are offered as examples, and suggestions are provided to further such collaborations.
Conference Paper
Full-text available
The UMUX-LITE is a two-item questionnaire that assesses perceived usability. In previous research it correlated highly with the System Usability Scale (SUS) and, with appropriate adjustment using a regression formula, had close correspondence to the magnitude of SUS scores, enabling its comparison with emerging SUS norms. Those results, however, we...
Article
Full-text available
This special issue focuses on the evolution of development and assessment methodologies related to usability and user experience. The five articles have a diverse range of topics, including comparison of moderated and unmoderated think-aloud usability sessions, a new usability inspection method based on concept mapping, analysis of the fitness of s...
Article
The purpose of this research was to investigate various measurements of perceived usability, in particular, to assess (a) whether a regression formula developed previously to bring Usability Metric for User Experience LITE (UMUX-LITE) scores into correspondence with System Usability Scale (SUS) scores would continue to do so accurately with an inde...
Article
This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and...
Article
Full-text available
This special issue on usability and the user experience focuses on the application of psychometrics to the development and evaluation of standardized usability and user experience questionnaires. The six articles cover advances in the measurement of perceived usability (assessment of the instruments themselves and their application to measuring the...
Article
The System Usability Scale (SUS) is a widely adopted and studied questionnaire for usability evaluation. It is technology independent and has been used to evaluate the perceived usability of a broad range of products, including hardware, software, and websites. In this article we present a Slovene translation of the SUS (the SUS-SI) along with the...
Article
Researchers performed a usability study of a digital-rights-management sharing (DRMS) application with which users protect and share digital files. Besides the standard goal of identifying usability problems, the study investigated how expertise affects objective and perceived usability, the correlations among the usability metrics, and how the usa...
Article
The philosopher of science J. W. Grove (1989) once wrote, “There is, of course, nothing strange or scandalous about divisions of opinion among scientists. This is a condition for scientific progress” (p. 133). Over the past 30 years, usability, both as a practice and as an emerging science, has had its share of controversies. It has inherited some...
Article
This article describes the development and psychometric evaluation of the Emotional Metric Outcomes (EMO) questionnaire—a new questionnaire designed to assess the emotional outcomes of interaction, especially the interaction of customers with service-provider personnel or software. The EMO is a concise multifactor standardized questionnaire that pr...
Patent
Full-text available
A method and system for defining standard catch styles used in generating speech application code for managing catch events, in which a style-selection menu that allows for selection of one or more catch styles is presented. Each catch style represents a system response to a catch event. A catch style can be selected from the style-selection menu....
Article
Full-text available
Personal Health Records (PHRs) that are tethered to electronic medical health systems are applications that can significantly enhance patients' health and health care. The primary aim of this research was to examine the roles of health literacy, numeracy ability, and graph literacy in enabling a group of veterans to perform health-management tasks...
Patent
Full-text available
A computer-implemented method for dynamically generating a speech recognition grammar is provided. The method includes determining a target entry, and accessing a plurality of potential distracters. The method also includes selecting one or more distracters from the plurality of potential distracters. More particularly, each potential distracter se...
Article
In 2009, Bargas-Avila, Lötscher, Orsini and Opwis published The Intranet Satisfaction Questionnaire (ISQ). In their research, Bargas-Avila et al. (2009, Intranet satisfaction questionnaire: development and validation of a questionnaire to measure user satisfaction with the intranet. Comput. Hum. Behav., 25, 1241–1250.) identified an important aspec...
Article
In 2010, Kraig Finstad published (in this journal) ‘The Usability Metric for User Experience’—the UMUX. The UMUX is a standardized usability questionnaire designed to produce scores similar to the System Usability Scale (SUS), but with 4 rather than 10 items. The development of the questionnaire followed standard psychometric practice. Psychometric...
Conference Paper
Full-text available
In this paper we present the UMUX-LITE, a two-item questionnaire based on the Usability Metric for User Experience (UMUX) [6]. The UMUX-LITE items are This system's capabilities meet my requirements and This system is easy to use." Data from two independent surveys demonstrated adequate psychometric quality of the questionnaire. Estimates of reliab...
Patent
Full-text available
A method for alleviating ambiguity issues of new user-defined speech commands. An original command for a user-defined speech command can be received. It can then be determined if the original command is likely to be confused with a set of existing speech commands. When confusion is unlikely, the original command can be automatically stored. When co...
Patent
Full-text available
A method (10) in a speech recognition application callflow can include the steps of assigning (11) an individual option and a pre-built grammar to a same prompt, treating (15) the individual option as a valid output of the pre-built grammar if the individual option is a potential valid match to a recognition phrase (12) or an annotation (13) in the...
Chapter
Standardized usability questionnaires are questionnaires designed for the assessment of perceived usability, typically with a specific set of questions presented in a specified order using a specified format with specific rules for producing scores based on the answers of respondents. For usability testing, standardized questionnaires are available...
Chapter
To assess the precision of an estimate, compute its confidence interval. Use confidence intervals around all point estimates to understand the most likely range of the unknown population mean or proportion. Computing a confidence interval requires four things: an estimate of the mean, an estimate of the variability (derived from the sample standard...
Chapter
This goal of this chapter is to cover methods for determining if a difference is statistically significant and how large or small of a difference likely exists in the untested population. It is important to account for chance differences when comparing two designs or products. To do this, you need to find a p -value from the appropriate statistical...
Chapter
User research is a broad term that encompasses many methodologies, such as usability testing, surveys, questionnaires, and site visits, that generate quantifiable outcomes. Usability testing is a central activity in user research and typically generates the metrics of completion rates, task times, errors, satisfaction data, and user interface probl...
Article
Full-text available
In The Voice in the Machine, Roberto Pieraccini takes the reader on a journey through the history of speech technologies, from the speaking machine of von Kempelen in 1804 through the current uses of statistical machine learning that make possible speech recognition, artificial speech production, speaker recognition, and dialog management. Although...
Chapter
Full-text available
Covers the basics of usability testing plus some statistical topics (sample size estimation, confidence intervals, and standardized usability questionnaires).
Book
A step-by-step guide for calculating the most common statistical procedures in user research using Excel or R. With 100 examples pulled from Quantifying the User Experience (Morgan-Kaufmann) we provide the reader with the right methods for comparing means and proportions, generating confidence intervals, finding the right sample size as well as man...
Article
In this paper we describe the development of a standardized computer satisfaction usability questionnaire for use with speakers of the Turkish language, the Turkish Computer System Usability Questionnaire, Short Version (T-CSUQ-SV). This new questionnaire, based on the English-language CSUQ, underwent careful translation and transformation through...
Book
You're being asked to quantify your usability improvements with statistics. But even with a background in statistics, you are hesitant to statistically analyze their data, as they are often unsure which statistical tests to use and have trouble defending the use of small test sample sizes. The book is about providing a practical guide on how to sol...
Conference Paper
Full-text available
When designing questionnaires there is a tradition of including items with both positive and negative wording to minimize acquiescence and extreme response biases. Two disadvantages of this approach are respondents accidentally agreeing with negative items (mistakes) and researchers forgetting to reverse the scales (miscoding). The original System...
Chapter
Full-text available
Human factors engineering (also known as ergonomics) is a body of knowledge and a collection of analytical and empirical methods used to guide the development of systems to ensure suitability for human use. Human factors engineering is a component of user-centered design, and encompasses the disciplines of human-computer interaction and usability e...
Book
Although speech is the most natural form of communication between humans, most people find using speech to communicate with machines anything but natural. Drawing from psychology, human-computer interaction, linguistics, and communication theory, Practical Speech User Interface Design provides a comprehensive yet concise survey of practical speech...
Conference Paper
Full-text available
The distribution of task time data in usability studies is positively skewed. Practitioners who are aware of this positive skew tend to report the sample median. Monte Carlo simulations using data from 61 large-sample usability tasks showed that the sample median is a biased estimate of the population median. Using the geometric mean to estimate th...
Article
Full-text available
An emerging area of research in engineering psychology is the evaluation of text entry for mobile devices using a small number of keys for the control of cursor direction and character selection from a matrix of characters (i.e., selection-based data entry). The present article describes a software tool designed to reduce time and effort in the dev...
Conference Paper
Full-text available
Since its introduction in 1986, the 10-item System Usability Scale (SUS) has been assumed to be unidimensional. Factor analysis of two independent SUS data sets reveals that the SUS actually has two factors - Usability (8 items) and Learnability (2 items). These new scales have reasonable reliability (coefficient alpha of .91 and .70, respectively)...
Conference Paper
Full-text available
Correlations between prototypical usability metrics from 90 distinct usability tests were strong when measured at the task-level (r between .44 and .60). Using test-level satisfaction ratings instead of task-level ratings attenuated the correlations (r between .16 and .24). The method of aggregating data from a usability test had a significant effe...
Chapter
Full-text available
From PDAs to cell phones to MP3 players, handheld electronic devices are ubiquitous. Human factors engineers and designers have a need to remain informed about advances in research on user interface design for this class of devices. This review provides human factors research summaries and research-based guidelines for the design of handheld device...
Conference Paper
Full-text available
Five-key text entry methods are useful for limited text entry on small devices. They use four directional keys to move a selector over an on-screen keyboard and an Enter key for selection. Although other researchers have described five-key character layouts using alphabetical order and predictive layouts based on digraph frequencies, there is consi...
Article
The primary purpose of this experiment was to gain a greater understanding of the utilization of working memory when interacting with a speech-enabled interactive voice response (IVR) system. A widely promoted guideline advises limiting IVR menus to five or fewer items because of constraints of the human memory system, commonly citing Miller's (195...
Patent
In a speech recognition system, a method of nonvisual enrollment comprising playing an audio representation of an enrollment script. As the enrollment is playing, shadowed speech from a user can be received, wherein the shadowed speech can lag the enrollment script. The received shadowed speech can be recorded for enrolling the user into the speech...
Article
Full-text available
"Really, how many users do you need to test? Three answers, all different." ---User Experience, Vol. 4, Issue 4, 2005
Conference Paper
Full-text available
Abstract A current controversy in the interactive voice response (IVR) community,is whether and under which conditions designers should use recorded audio when portions of the interface must be generated by text-to-speech (TTS). The purpose of this study was to examine user preferences for a very extreme case—a prompt that incorporates multiple uni...
Conference Paper
Full-text available
Guidelines for speech user interfaces generally promote the use of delayed confirmation in speech recognition applications that require users to provide multiple elements of information. In our initial investigation of a simple delayed confirmation method, we discovered a significant design flaw (requiring users to review a fairly large amount of c...
Conference Paper
Full-text available
I examined samples from a number of Companys' user publications using several automated reading measures and a graphics/text ratio. The goal was to answer two questions: Were there reliable differences in writing style among the competitors? If so, were these differences related to their rank position in published surveys of user satisfaction with...
Conference Paper
The primary purpose of this analysis was to investigate the effect of changing the level of description of usability problems on the estimate of the problem discovery rate (p). A secondary purpose was to describe a method for using p to estimate the number of problems remaining available for discovery given the constraints associated with a particu...
Article
Full-text available
Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to co...
Chapter
Full-text available
Usability testing is an essential skill for usability practitioners -- professionals whose primary goal is to provide guidance to product developers for the purpose of improving the ease of use of their products. It is by no means the only skill with which usability practitioners must have proficiency, but it is an important one. Surveys of experie...
Article
Full-text available
"Really, how many users do you need to test? Three answers, all different."---User Experience, Vol. 4, Issue 4, 2005
Chapter
Full-text available
The cumulative binomial probability formula (given appropriate adjustment of p when estimated from small samples) provides a quick and robust means of estimating problem discovery rates (p). This estimate can be used to estimate usability test sample size requirements (for studies that are underway) and to evaluate usability test sample size adequa...
Conference Paper
Full-text available
The completion rate – the proportion of participants who successfully complete a task – is a common usability measurement. As is true for any point measurement, practitioners should compute appropriate confidence intervals for completion rate data. For proportions such as the completion rate, the appropriate interval is a binomial confidence interv...
Conference Paper
Full-text available
This paper describes a number of experiments and analyses conducted to gain an understanding of the typical properties of error spans - the number of consecutive incorrect words generated when producing text with recognition technologies. The data were drawn from three studies of recognition applications: one study of an IBM dictation system, one o...
Conference Paper
Full-text available
It is important to help users navigate speech-based interactive voice response systems (IVRs) by presenting the global navigation commands at any potential task end point. This paper presents an analysis we conducted to determine the optimal pause between presentation of a menu at a task terminal point and the presentation of the global navigation...
Patent
A method for discriminating between an instance of a voice command and an instance of speech dictation can include identifying a focus point in a user interface; defining a surrounding region about the focus point; identifying user interface objects in the surrounding region; further identifying among the identified user interface objects those use...