Phil Kortum

• Doctor of Philosophy
• Professor (Full) at Rice University

Free, fair, and secure elections are the foundation of a democracy. However, the idea of voting is simple, whereas its execution is complex. There are over 3000 counties in the United States, representing 10,000 election jurisdictions with varying laws, demographics, logistics, and levels of technology integration. There are numerous open challenge...

Medical devices designed for use in low-resource settings require unique and thoughtful design considerations to facilitate successful implementation and adoption. Usability testing is critical for good design but conducting research in international locations with intended target users is challenging and costly. Proxy user groups have been propose...

Findings from previous research that assessed the usability of single-page and multipage digital interfaces in purely digital interactions indicated that the single-page format is more efficient than its multipage counterpart. This research expands on previous work by applying the findings from these digital-only interactions to a paper-digital int...

Families making medical decisions for an incapacitated loved one need to process medical information for various care pathways while balancing different perspectives and experiencing distress. A decision aid tool to help family members make medical decisions on the patient’s behalf should be easy to use and not create additional burdens. A formativ...

Absentee voting presents a unique challenge for U.S. uniformed service members, as they often struggle to request and return absentee ballots while deployed, sometimes stationed far from their registered voting area. This research evaluates the usability of a proposed absentee voting system for military voters, which allows instant ballot requests...

Background Millions of newborns die annually from preventable causes, with the highest rates occurring in Africa. Reducing neonatal mortality requires investment to scale hospital care, which includes providing hospitals with appropriate technology to care for small and sick newborns. Expensive medical devices designed for high-resource settings of...

Self-driving vehicles (SDVs) are an emerging technology in which consumers have low levels of trust. Researchers/designers can understand and improve consumer trust through research and iterative design, but doing so effectively requires reliable measures. Although general trust-in-automation measures exist, a measure tailored to SDVs may provide a...

This study evaluated people’s perceived usability of warning signs and labels using the System Usability Scale (SUS), to understand how textual and visual factors of warnings contribute to SUS scores. 30 warning signs/labels across various industries were selected and 135 participants evaluated these warnings on their usability and familiarity. Eac...

Ranked choice voting (RCV) is a method of voting where individuals rank their choices for each race in an election, rather than selecting a single candidate. For paper ballot implementations, there are three basic formats: bubble grid, column, and handwritten. This study aimed to understand which format produces the best outcomes with the fewest er...

The System Usability Scale (SUS) is a tool widely used in industry for measuring the usability of products and systems. Users are often asked to complete two or more tasks of varying complexity before evaluating a product using the SUS. However, task order effects may influence the overall usability rating of a product, but previous literature has...

Medical device implementation in global health requires careful considerations around usability and the use of proxy user groups. Working with appropriate proxy users can address the cost burden of conducting international studies with target users. This study evaluated whether proxy users are a practical substitution for conducting usability testi...

Purpose: To reduce the burden of Alzheimer's disease, the use of assistive technologies for patients and their informal caregivers is considered essential. However, these technologies are made as "one size fits all" instead of being tailored to accommodate people with varying degrees of cognitive impairment and those with diverse races/ethnicities...

At most universities, students are required to complete end-of-semester teacher evaluation forms, which may have undesirable effects on teacher ratings to the extent that students feel inconvenienced, annoyed, or disinterested when completing these forms. Addressing these concerns, we designed the Teaching and Instructional Measure of Effectiveness...

During the COVID-19 pandemic, usability practitioners and researchers had to find new approaches to product testing. In-person, contact-intensive product testing became a safety concern, resulting in the need for more remote testing practices. An underexplored and promising method to capture subjective usability measures is Watching Others Using Vi...

The Usefulness, Satisfaction, and Ease of Use Questionnaire (USE) is a 30-item measure of subjective usability. The content of the USE allows the usability of a product to be interpreted along four important dimensions—Usefulness, Ease of Use, Ease of Learning, and Satisfaction—rather than as a global construct. Although the USE has been a valuable...

There is still debate on whether voters can detect malicious changes in their printed ballot after making their selections on a Ballot Marking Device (BMD). In this study, we altered votes on a voter's ballot after they had made their selections on a BMD. We then required them to examine their ballots for any changes from the slate they used to vot...

It is essential to democracy that voters trust voting systems enough to participate in elections and use these systems. Unfortunately, voter trust has been found to be low in many situations, which could detrimentally impact human-computer interactions in voting. Therefore, it is important to understand the degree to which voters trust any specific...

Understanding why developers continue to misuse security tools is critical to designing safer software, yet the underlying reasons developers fail to write secure code are not well understood. In order to better understand how to teach these skills, we conducted two comparatively large-scale usability studies with undergraduate CS students to asses...

Objective To describe the perceived usability and usability problems associated with face coverings used to prevent the spread of COVID-19. Background Since public health experts have now identified the appropriate use of facemasks as one of the critical elements in an effective COVID mitigation strategy, understanding how people use and care for...

The purpose of this study is to compare presentation methods for use in the validation of the Trust in Selfdriving Vehicle Scale (TSDV), a questionnaire designed to assess user trust in self-driving cars. Previous studies have validated trust instruments using traditional videos wherein participants watch a scenario involving an automated system bu...

With the proliferation of mobile touchscreen computers, password entry no longer takes place exclusively on physical keyboards. Entering a strong password on a mobile device requires a person to navigate through multiple keyboard depths to access each character, while entering the same password on a desktop keyboard only requires a user to press ke...

Developing good psychological measures benefits from the input of content experts. For many constructs or domains, however, who constitutes an ‘expert’ might be ill-defined. Novices—such as students, customers, or co-workers—may possess the same knowledge as experts. Moreover, as convenience samples, novices are more readily available and less cost...

Previous work has investigated the need for domain specific heuristics. Nielsen’s ten heuristics offer a general list of principles, but those principles may not capture usability issues specific to a given interface. Studies have demonstrated methods to establish a domain specific heuristic set, but very little research has been conducted on inter...

Within the domain of subjective usability assessment, several potential discount methods exist. However, there is little or no prior research investigating how these methods compare in their impact on subjective usability ratings. This study compared four methods of collecting subjective usability data with the System Usability Scale (SUS). Users w...

The System Usability Scale (SUS) is a widely used instrument that measures the subjective usability of products and systems. Although past research has demonstrated the psychometric reliability and criterion-related validity of the SUS in specific languages, the approach and methodology of validating the translations has been somewhat inconsistent....

Objective To describe user-centered voting systems that would support the safe conduct of voting in a pandemic environment. Background The COVID-19 pandemic has complicated our democratic processes. Voters and poll workers feel threatened by the potential dangers of voting in business-as-usual polling stations. Indeed, significant problems were en...

The question of whether or not voters actually verify ballots produced by ballot marking devices (BMDs) is presently the subject of some controversy. Recent studies (e.g., Bernhard, et al. 2020) suggest the verification rate is low. What is not clear from previous research is whether this is more a result of voters being unable to do so accurately...

Objective The goal of the research presented in this paper was to determine if the positively worded System Usability Scale (SUS) can be used in place of the positively and negatively worded standard SUS instrument for the subjective assessment of usability, and whether the results found here replicate those of Sauro and Lewis. Background Sauro an...

Previous work has examined whether keyboards that have multiple levels of entry (e.g., soft keyboards, found on mobile devices) increase users’ perceptions of password security. That work found that passwords that required greater numbers of keyboard transitions had higher perceived strength. Unfortunately, it was impossible to determine whether th...

Usability researchers and product designers stand to benefit by minimizing costs (e.g., time, overhead, recruitment costs) related to usability research. Several potential discount methods for collecting subjective data exist. This study compared four methods of collecting satisfaction data using the After-Scenario Questionnaire (ASQ). Use-Then-Mea...

HTTPS and TLS are the backbone of Internet security, however setting up web servers to run these protocols is a notoriously difficult process. In this paper, we perform two live subjects usability studies on the deployment of HTTPS in a real-world setting. Study 1 is a within subjects comparison between traditional HTTPS configuration (purchasing a...

Researchers recently demonstrated that subjective usability assessments of common products do not vary across geographic locations. That study did not directly address mobile applications, which are some of the most ubiquitous and geographically diverse systems in use in the United States today. To address this shortcoming, this article examined wh...

Background: From the project's inception, STAR-Vote was intended to be one of the first usable, end-to-end (e2e) voting systems with sophisticated security. To realize STAR-Vote, computer security experts, statistical auditors, human factors (HF)/human-computer interaction (HCI) researchers, and election officials collaborated throughout the proje...

The Usefulness, Satisfaction, and Ease of Use Questionnaire (USE, Lund, 2001) measures the subjective usability of a product or service. It is a 30-item survey that examines four dimensions of usability: usefulness, ease of use, ease of learning, and satisfaction. This metric can be applied to various scenarios of usability assessment because it is...

Computer security experts recommend that people use two-factor authentication (2FA) on password protected systems to help keep hackers out. Providing two pieces of information to verify a person’s identity adds extra security to an account. However, it is not clear if the added security and procedures impact system usability. This paper aims to ans...

Previous work has examined if cognitive fatigue caused subjects to negatively evaluate products on subjective usability measures but failed to find an effect. The authors suggested that methodological issues may have caused variance, which masked any experimental effect. This study tested that claim by eliminating the extra source of variance. Twen...

This panel will discuss the System Usability Scale. Panelists all have extensive experience using the SUS within a broad range of contexts: diverse people (e.g., abilities, languages); different types of products; and different testing scenarios. Members of the audience will have the opportunity to ask questions about new research on the validity o...

Marketing researchers use geography to identify specific user groups for studies to more effectively describe their potential customer base. Since usability professionals often recruit users employing similar selection criteria as their marketing peers, the use of geographic information might also be relevant when selecting usability test participa...

A literature review confirms that evaluations of and responses to an environment are similar whether the presentation medium is the physical environment; a photograph of it; or a dynamic, virtual reality simulation. However, studies have not been conducted to determine if photorealistic renderings would receive similar assessments as a physical spa...

Cognitive fatigue is a known factor in errors and major accidents. What is unknown is the impact that cognitive fatigue might have on the subjective assessment of usability. If users are regularly cognitively depleted when using a given system, and fatigue makes a difference in their usability assessment, then a true measure of usability would be d...

For the first time, the system usability of the eSlate–one of the most ubiquitous electronic voting machines used in US elections–was assessed in the field on election day 2016 using ISO 9241-11 metrics. Effectiveness and satisfaction measures were collected immediately after voters used the eSlate to vote at their assigned polling location. The fi...

As the trend of home-delivered healthcare grows, the number of healthcare devices being utilized in the home setting also increases greatly, but the usability of all these devices has not been systematically examined. While traditional in-lab usability testing is too time-consuming to make broad assessments of a large number of home healthcare devi...

The System Usability Scale (SUS) has been widely employed in both the field and the laboratory as a valid and reliable measure of system usability. Although its psychometric properties are relatively well understood, the impact that differences in users’ personality traits have on their perceived usability of products, services, and systems has not...

Verifying a ballot for correctness in an election is a critical task for the voter. Previous work has shown that up to 30% of the ballot can be changed without being noticed by more than half of the voters. In response to this ballot weakness, this study evaluated the usability and viability of alternative ballot verification methods in an electron...

Ensuring the integrity of elections is one of the most important elements in maintaining democracy. While it is commonly believed that threats to election integrity are primarily due to security issues, the reality is that voting systems that are not designed to support human perceptual and cognitive limitations also pose a serious and immediate th...

Posters With Fellows is an opportunity to interact with some of the best scholars the Human Factors and Ergonomics Society has to offer in a low-key, intellectual setting. Fellows will present posters describing cutting-edge research from their current work, retrospectives of important work they have conducted over their careers, and descriptions o...

The aim of this study was to understand how voters’ perceptions of voting system security might be impacted by the addition of actual security features and fake security features (i.e., security theater) to a voting method. In the context of voting, positive perceptions of security might be more important than security itself because voters must fe...

Objective metrics, such as effectiveness and efficiency, are often considered to be the best website usability measurements. User performance metrics that can be collected remotely, such as mouse clicks and the distance the mouse has traveled show particular promise. However, no studies have demonstrated a direct relationship between subjective usa...

One of the main goals of the Help America Vote Act (HAVA) was to ensure that voters with disabilities could vote independently. However, the current state of most voting methods does not allow for independent voting for everyone. In response to this issue, we tested a remote IVR voting system developed by Holmes and Kortum (2013), with an added aud...

Background: There are many effective methods for decreasing the likelihood of repetitive strain injury (RSI) for those who work at a computer in an office environment. This study is focused on the highly repetitive task of interpreting seismic data. The skilled geoscientists who perform this work are very well compensated, and their work is vital...

Home healthcare devices have gained attention recently as more and more care is being delivered in the patients’ home rather than in a hospital setting. This trend is accelerating as healthcare systems work to restrain costs and provide patients with the tangible benefits of receiving care in their own home. Because the user population of these dev...

This study sought to understand if voting equipment layouts impacted anticipated voting system usability. Thirty-five participants viewed photorealistic rendered images of twelve different voting system configurations within a polling station. After viewing each condition, participants completed a survey to assess anticipated system usability and r...

The use of applications on mobile devices has reached historic levels. Using the System Usability Scale (SUS), data were collected on the usability of applications used on two kinds of mobile platforms—phones and tablets—across two general classes of operating systems, iOS and Android. Over 4 experiments, 3,575 users rated the usability of 10 appli...

This study sought to understand voter’s mental models for three endto- end (e2e) voting systems: Helios, Prêt à Voter, and Scantegrity II. To study voters’ mental models of e2e systems, 16 Houston area voters participated in mock elections that required them to vote first with a paper ballot and then with the three e2e systems. After using each sys...

Mobile web browsing is highly recurrent, in that a large proportion of a user's page requests are to a small set of websites. Despite this, most mobile browsers do not provide an efficient means for revisiting sites. Although significant research exists on prediction in the personal computer realm, little work has been done in the mobile realm wher...

Background An increasing amount of health care is now performed in a home setting, away from the hospital. While there is growing anecdotal evidence about the difficulty patients and caregivers have using increasingly complex health care devices in the home, there has been little systematic scientific study to quantify the global nature of home hea...

This study examined smartphone user behaviors and their relation to self-reported smartphone addiction. Thirty-four users who did not own smartphones were given instrumented iPhones that logged all phone use over the course of the year-long study. At the conclusion of the study, users were asked to rate their level of addiction to the device. Sixty...

This study examined whether the average usability score for a series of tasks was the same as the usability score for the product if usability was measured only after all the tasks had been completed. Fifty participants completed a set of tasks for five websites and fourteen mock voting ballots. Subjective usability assessment was made with the Sys...

The assessment of usability is a common task for human factors professionals. The System Usability Scale (SUS) is a survey tool that has been supported empirically as a psychometrically robust, valid and highly adaptive way to measure subjective usability. One area in which the SUS has not been applied yet, but could have direct and important appli...

Recently developed secure, end-to-end voting systems are designed to allow voters to confidentially verify on the internet that their ballot selections are cast and counted correctly. In response, this research paper characterizes both voters’ desire to actually use vote verification methods and the format of verification that users expected. 767 r...

Objective: The goal of this research was to assess the usability of a voting system designed for smart-phones. Background: Smartphones offer remote participation in elections through the use of pervasive technology. Voting on these devices could, among other benefits, increase voter participation while allowing voters to use familiar technology....

Smartphone technology is penetrating world markets and becoming ubiquitous in most college settings. This study takes a naturalistic approach to explore the use of these devices to support student learning. Students that had never used a smartphone were recruited to participate and reported on their expectations of the value of smartphones to achie...

This article examines the relationship between users’ subjective usability assessments, as measured using the System Usability Scale (SUS), and the ISO metric of effectiveness, using task success as the measure. The article reports the results of two studies designed to explore the relationship between SUS scores and user success rates for a variet...

Evaluation and usability as a practice area has diversified its approaches, broadened the spectrum of UX issues it addresses, and extended its contribution into deeper levels of product-development decision making. This forum addresses conceptual, methodological, and professional issues that arise in the field's continuing effort to contribute robu...

The system usability scale (SUS; Brooke, 1996) is an instrument commonly utilized in usability testing of commercial products. The goal of this symposium is to discuss the validity of the SUS in usability tests and beyond. This article serves as an introduction to the symposium. Specifically, it provides an overview of the SUS and discusses researc...

There are a myriad of voting technologies available today, but a noted lack of remote voting methods. In this paper we explore a novel remote voting method that allows users to vote-by-phone. This method used an Interactive Voice Response (IVR) system to allow users to vote using a touch-tone telephone. The IVR voting system has several advantages...

Data from previous research that investigates whether or not voters fail to notice changes to their electronic ballot selections indicate that high percentages of people do not notice if their votes were altered, even after inspecting a final review screen. This paper questions if voters really do not notice changes as the previous studies found, o...

The System Usability Scale (SUS) is a ten-point assessment tool developed as a reliable low-cost subjective usability scale that can be applied to systems in any number of contexts. Research has demonstrated higher usability ratings from users who claim greater experience with an interface than from those who rate themselves as having less experien...

Providing patient information to physicians in usable form is of high importance. Electronic presentation of patient data may have benefits in efficiency and error rate reduction for these physician facing interfaces. Using a cancer symptom measurement tool (the MD Anderson Symptom Inventory (MDASI)) we assessed the usability of patient data in its...

STAR-Vote is a collaboration between a number of academics and the Travis County (Austin), Texas elections office, which currently uses a DRE voting system and previously used an optical scan voting system. STAR-Vote represents a rare opportunity for a variety of sophisticated technologies, such as end-to-end cryptography and risk limiting audits,...

This paper describes the results of two studies undertaken to better understand the strong positive relationship between the content of a video clip and viewers' subjective ratings of the video quality of that clip. Experiment 1 evaluated the effect of training on video quality ratings. Experiment 2 assessed the impact of attention in these assessm...

In her 2011 EVT/WOTE keynote, Travis County, Texas County Clerk Dana DeBeauvoir described the qualities she wanted in her ideal election system to replace their existing DREs. In response, in April of 2012, the authors, working with DeBeauvoir and her staff, jointly architected STAR-Vote, a voting system with a DRE-style human interface and a "belt...

There is a large body of literature on trust, which focuses on interpersonal relationships. In recent years the scope of this research has expanded, as some studies have begun to explore trust between people and systems. Trust in these systems is critical, because if someone does not trust a system they will not use it. Another factor that heavily...

This paper reports the results of a study that examined the impact of aesthetic changes to a web site on user performance when a single critical link in the main navigation of the page also changed. 102 participants performed 4 search tasks on a web site. They were then redirected back to the site immediately or waited approximately 14 days to perf...

Interactive voice response systems (IVRs) are ubiquitous user interfaces that allow customers to gather information and execute transactions. Because the entire user interaction is auditory-based, the voice used in the IVR is of high importance. Entities that field IVRs go to great lengths to select appropriate voices, usually on the basis of aesth...

The present report is an empirical analysis of smartphone personalisation. We collected data from two groups of users to measure how they adapt the content, interface and physical appearance of their devices. This user-driven personalisation is measured with a simple heuristic approach to quantify the behaviour. Using these scores, we explore how u...

Previous studies have found that smartphone users differ by orders of magnitude. We explore this variability to understand how users install and use native applications in ecologically-valid environments. A quasi-experimental approach is applied to compare how users in different socio-economic status (SES) groups adopt new smartphone technology alo...

Context information brings new opportunities for efficient and effective applications and services on mobile devices. A wide range of research has exploited context dependency, i.e., the relations between context(s) and the outcome, to achieve significant, quantified, performance gains for a variety of applications. These works often have to deal w...

The usability of a product or service can mean the difference between product success or failure in both the field and the marketplace. Successful products are easy to use and intuitive for the user. There are a set of well-defined methods used to determine if a product, service or system is usable and meets the needs of the end user. This paper de...

This paper contributes an intentionally naturalistic methodology using smartphone logging technology to study communications in the wild. Smartphone logging can provide tremendous access to communications data from real environments. However, researchers must consider how it is employed to preserve naturalistic behaviors. Nine considerations are pr...

The current paper establishes empirical patterns associated with mobile internet use on smartphones and explores user differences in these behaviors. We apply a naturalistic and longitudinal logs-based approach to collect real usage data from 24 iPhone users in the wild. These data are used to describe smartphone usage and analyze revisitation patt...

This paper characterizes the usability of 14 common, everyday products using the System Usability Scale (SUS). Over 1,000 users were queried about the usability of these products using an online survey methodology. The study employed two novel applications of the SUS. First, participants were not asked to perform specific tasks on these products be...

This study investigated performance and preference differences for three different extensible Auditory Progress Bar (APB) designs. Four durations (30s, 60s, 120s, 240s) of the three APBs (Sine, Cello and Electronic) were used in the study. 105 participants listened to all durations of a single stimulus type and were asked to determine the length of...

The iPod Touch provides portable and personalized information, entertainment, and communication resources for users. The goal of this study was to assess the influence of user diversity on how these handheld mobile computers are employed in real environments. Using an unobtrusive, longitudinal methodology to collect data, we explored how different...

The goal of this research was to assess the usability of a voting system designed for current-generation smartphones. Smartphones offer remote participation in elections through the use of pervasive technology and voting on these devices could, among other benefits, increase voter participation while allowing voters to use familiar technology. We d...

The research in this paper examines the effect of web site familiarity on the impact of small changes in a site's primary navigation structure on user performance. Ninety-two participants performed multiple tasks on a web site, and then returned to the site either immediately or after a three-week delay to perform one of the original tasks again. H...

Auditory Progress Bars (APB) were originally intended to augment Visual Progress Bars (VPB) to create multimodal displays. More recently, APBs have been tested in absence of VPBs for use in the on-hold telephone setting. In this setting, APBs are a viable option for communicating the probable time remaining in the on-hold wait. However, past studie...

We present results from a longitudinal study of 34 iPh-one 3GS users, called LiveLab. LiveLab collected unprecedented usage data through an in-device, programmable logger and several structured interviews with the participants throughout the study. We have four objectives in writing this paper: (i) share the findings with the research community; (i...

The ubiquitous use of workstation and laptop-based geophysical applications for seismic interpretation presents a risk for injuries associated with computer use. While work has been done to decrease ergonomic risk for geophysical field personnel (Pearce and Shackel, 1979) the risk to office personnel is still high. Specifically, over the last decad...

This paper addresses the following question: Is it possible to migrate TCP/IP flows between different networks on modern mobile devices, without infrastructure support or protocol changes? To answer this question, we make three research contributions. (i) We report a comprehensive characterization of IP traffic on smartphones using traces collected...

System voice within interactive voice response systems (IVRs) was investigated in order to determine if voice impacts a user’s input responses. In a medical setting, it is possible that a particular voice personality and/or gender may induce more or less disclosure, thus driving a patient to relay more or less sensitive information. In the IVR sett...

A study of 105 university students examined whether the length or type of Auditory Progress Bar (APB) used in a telephone on-hold situation had an impact on users' subjective mental workload. Auditory cues can be designed into APBs to provide information to the caller regarding the approximate duration of the hold time. Previous research has found...