Bonnie E. John

IBM · Computer Science, T.J. Watson Center

Systems and computer program products for providing usability predictions are set forth herein. A system includes a test case collector for obtaining test cases for analyzing usability of a graphical user interface of a given software application, the test case collector having at least a memory storing the test cases. The system further includes a...

A system and method for evaluating interfaces includes computing a reference script for a task from a reference interface design and translating the reference script into one or more target action scripts based on a target design. The one or more target action scripts on the target design are executed to produce target metrics. The target metrics a...

Pinching and spreading gestures are prevalent in mobile applications today, but these gestures have not yet been studied extensively. We conducted an exploratory study of pinch and spread gestures with seated participants on a phone and a tablet device. We found device orientation did not have a significant effect on gesture performance, most pinch...

As software systems evolve, new interface features such as keyboard shortcuts and toolbars are introduced. While it is common to regression test the new features for functional correctness, there has been less focus on systematic regression testing for usability, due to the effort and time involved in human studies. Cognitive modeling tools such as...

Several theories of how novices explore a new interface have arisen in HCI and have led to methods and tools for UI practitioners to predict users' behavior on proposed interfaces and improve their design ideas before implementation. These tools depend on obtaining quantitative estimates of information scent, which usually requires a large corpus o...

In an age where HCI research is embracing softer approaches, we must not neglect the practical value of engineering research for HCI. This video challenges the assumption that engineering interactive systems takes a lot of effort with little pay-back or reward. We take CANCEL as an example. Not many people think about the challenges of implementing...

Many devices are designed to allow skilled users to complete routine tasks quickly, often within a specified amount of time. Predictive human performance modeling has long been able to predict the mean time to accomplish a task, making it possible to compare device designs before building them. However, estimates of the variability of performance a...

We present the Log Analyzer for generating CPM-GOMS models from human performance data. Built on top of the SANLab tool for stochastic CPM-GOMS modeling (Patton & Gray, 2010), the Log Analyzer uses event-driven parsing to map experimental log files into SANLab interactive routines used to generate CPM-GOMS activity networks. Identical models within...

In this study, we explore the role of age and fluid intelligence on the behavior of people looking for information in a real-world search space. Analyses of mouse moves, clicks, and eye movements provide a window into possible differences in both task strategy and performance, and allow us to begin to separate the influence of age from the correlat...

Information Foraging Theory (IFT) has established itself as an important theory to explain how people seek information, but most work has focused more on the theory itself than on how best to apply it. In this paper, we investigate how to apply a reactive variant of IFT (Reactive IFT) to design IFT-based tools, with a special focus on such tools fo...

With the rise of tools for predictive human performance modeling in HCI comes a need to model legacy applications. Models of legacy systems are used to compare products to competitors, or new proposed design ideas to the existing version of an application. We present CogTool-Helper, an exemplar of a tool that results from joining this HCI need to r...

Designers often struggle to create interfaces that are optimal for both younger and older adults, as they may interact differently with the same interface. Human-performance models have been used to aid designers in evaluating the efficiency of user interfaces. Can we create age-specific models to help designers create interfaces that are efficient...

CogTool-Explorer 1.2 (CTE1.2) predicts novice exploration behavior and how it varies with different user-interface (UI) layouts. CTE1.2 improves upon previous models of information foraging by adding a model of hierarchical visual search to guide foraging behavior. Built within CogTool so it is easy to represent UI layouts, run the model, and prese...

Although predictive human performance modeling has been researched for 30 years in HCI, to our knowledge modeling has been conducted as a solitary task of one modeler or, occasionally, two modelers working in tight face-to-face collaboration. In contrast, we used predictive human performance modeling in a collaborative, distributed mode and reflect...

Software developers frequently need to perform code maintenance tasks, but doing so requires time-consuming navigation through code. A variety of tools are aimed at easing this navigation by using models to identify places in the code that a developer might want to visit, and then providing shortcuts so that the developer can quickly navigate to th...

This paper reports the results of usability evaluation based on the predictive human performance models applied to any products, and introduces application of a tool for predicting operational time to an IP phone and an electronic health record system UI consulting. We assessed effectiveness of this tool using efficiency estimation technique and ex...

Usability is recognized as a critically important quality of software, as evidenced by its inclusion in SE 2004: Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering. However, many techniques for designing useful, usable and desirable software depend on knowledge and skills in the behavioral sciences, making them difficul...

Usability concerns are often difficult to integrate into real-world software development processes. To remedy this situation, IBM research and development, partnering with Carnegie Mellon University, has begun to employ a repeatable and quantifiable usability analysis method, embodied in CogTool, in its development practice. CogTool analyzes tasks...

We studied the extent to which time-on-task is correlated with perception of usability for people who are familiar with a phone model and for those who are not. Our controlled experiment, conducted in Japan, correlated subjective usability assessments with time-on-task for expert and novice users on three different mobile phone models. We found tha...

Predictive human performance modeling has traditionally been used to make quantitative comparisons between alternative designs (e.g., task execution time for skilled users) instead of identifying UI problems or making design recommendations. This note investigates how reliably novice modelers can extract design recommendations from their models. Ma...

vSked is an interactive and collaborative assistive technology for students with autism, combining visual schedules, choice boards, and a token-based reward system into an integrated classroom system. In this paper, we present the results of a study ...

We explore the match of a computational information foraging model to participant data on multi-page web search tasks and find its correlation on several important metrics to be too low to be used with confidence in the evaluation of user-interface designs. We examine the points of mismatch to inspire changes to the model in how it calculates infor...

Designers often have no access to individuals who use screen reading software, and may have little understanding of how their design choices impact these users. We explore here whether cog-nitive models of auditory interaction could provide insight into screen reader usability. By comparing human data with a tool-generated model of a practiced task...

In this paper, we describe the use of CogTool, a tool that enables non-psychologists to create cognitive models of user tasks from which reliable estimates of skilled user task times can be derived. We show how CogTool was used to compare a new parallel programming toolkit built on Eclipse, with Vim, a programming editor typically used in command l...

Low-end mobile phones are becoming generally available throughout the world and the main window onto the Web. A large constituency for Web access is being created whose experience Web designers are not familiar with, particularly because of the central role of auditory, rather than visual access. This is analogous to problems faced by visually impa...

UI designers use a variety of prototyping tools, from paper and pencil sketching, to drag-and-drop mock-up tools (e.g., Balsamiq Mockups), to sophisticated suites of modeling tools and toolkits (e.g., iRise or dijit, the dojo GUI toolkit ). Many projects would benefit from quickly analyzing prototypes at an early stage without the effort of bringin...

Human performance models based on information foraging theory have proved capable of predicting navigation behavior on the Web. They can therefore provide a useful tool for Web site design. They may also be effective for modeling auditory navigation within a single Web page. Designers often struggle to accommodate this sort of access, different as...

The variation between novice modelers has not been extensively studied, but it is important to organizations wishing to employ predictive human performance models in their system design process. This paper reports on the statistically-significant reduction in variation between novice modelers achieved by CogTool over the previously-established by-h...

Human performance modeling promises to be a valuable tool for early evaluation of user interface designs, predicting different performance for different design alternatives and, recently, different performance on a single design between younger and older adults (Jastrzembski & Charness, 2007; Jastrzembski, et al., 2010). When using modeling in the...

We present our experience using CogTool, a tool originally designed for ease of use and learning by non-psychologist design practitioners, as a means for rapid theory exploration. We created seven iterations of a “model prototype” of an aviation task where each iteration produced errors that pointed to additional theory or device knowledge that sho...

Historically, predictive human performance modeling has been successful at predicting the task execution time of skilled users on a desktop computer. More recent work has predicted novice behavior in web searches. This paper reports on a collaborative effort between industry and academia to expand the scope of predictive modeling to the mobile phon...

Usability-supporting architectural patterns (USAPs) were developed as a way to explicitly connect the needs of architecturally-sensitive usability concerns to the design of software architecture. In laboratory studies, the Cancellation USAP was shown to significantly improve the quality of architecture designs for supporting the ability to cancel a...

In this paper, we report a study that examines the relationship between image-based computational analyses of web pages and users' aesthetic judgments about the same image material. Web pages were iteratively decomposed into quadrants of minimum entropy ...

This paper addresses the problem of supporting usability in the early stages of a product line architecture design. The product line used as an example is intended to support a variety of different products each with a radically different user interface. The development cycles for new products varies between three years and five years and usability...

In recent years, research predicting search through webpages has begun to be successful. However, existing tools ignore the order in which on-screen options are evaluated and therefore might make inaccurate predictions. We developed CogTool-Explorer and used it to model a previously published web-based experiment. Its predictions were better than t...

Modern models of motor control and performance provide significant advances over the tried and true Fitts' Law on which Human Factors professionals still depend. However, the topic of motor control and performance is complex and the issues surrounding almost every aspect of the topic at times seem confusing or confused. To cut through this confusio...

We examine how ambient displays can augment social television. Social TV 2 is an interactive television solution that incorporates two ambient displays to convey to participants an aggregate view of their friends' current TV-watching status. Social TV ...

In recent years, research predicting search through webpages has begun to be successful. However, existing tools ignore the order in which on-screen options are evaluated and therefore might make inaccurate predictions. We developed CogTool-Explorer and used it to model a previously published web-based experiment. Its predictions were better than t...

Usability supporting architectural patterns (USAPs) have been shown to provide developers with useful guidance for producing a software architecture design that supports usability in a laboratory setting (7). In close collaboration between researchers and software developers in the real world, the concepts were proven useful (2). However, this proc...

As part of the ongoing project, Project Halo, our goal is to build a system capable of answering questions posed by novice users to a formal knowledge base. In our current context, the knowledge base covers selected topics in phys- ics, chemistry, and biology, and our question set consists of AP (advanced high-school) level examination questions. T...

The long-term goal of Project Halo is to build an applica- tion called Digital Aristotle that can answer questions on a wide variety of science topics and provide user- and domain-appropriate explanations. As a near-term goal, we are focusing on enabling subject matter experts (SMEs) to construct declarative knowledge bases (KBs) from 50 pages of a...

The use of embodied agents, defined as visual human-like representations accompanying a computer interface, is becoming prevalent in applications ranging from educational software to advertisements. In the current work, we assimilate previous empirical ...

The use of embodied agents, defined as visual human-like representations accompanying a computer interface, is becoming prevalent in applications ranging from educational software to advertisements. In the current work, we assimilate previous empirical ...

CogTool-Explorer is a tool to predict user exploration choices given a user interface and task. We describe the integration of components that make up CogTool-Explorer, and how it interprets the text of both the task description and interface elements to make its exploration choices. We present CogTool-Explorer's performance on a web navigational s...

The long-term goal of Project Halo is to build an application called Digital Aristotle that can answer questions on a variety of science topics and provide user and domain appropriate explanations. As a near-term goal, we are focusing on enabling Subject Matter Experts (SMEs) to construct declarative knowledge bases (KBs) from 50 pages of a science...

Comparison of model prediction against observed data is an investigative step used in cognitive modeling research for human-computer interaction. In this paper we describe comparisons between Keystroke-Level Model (KLM) predictions and user behavior by total duration, aggregated events and Cohen's Kappa. Our preliminary investigations support the v...

The Engineering Community may be the least populated and the least understood of the new communities at CHI, perhaps because so many definitions of engineering exist (as evidenced by the quotes in the margins). This SIG will provide a forum for people interested in bringing the best of the field of engineering to the field of HCI.

Prototypes' different purposes give them different requirements. Purposes for UI prototypes in pervasive computing systems development include communicating design ideas to diverse team members, supporting formative user tests, and providing summative evaluations of skilled user performance. CogTool is a system for rapidly building HTML storyboards...

The purpose of this research contract was to perform multidisciplinary research between CMU psychologists, computer scientists and NASA researchers to design a next generation collaborative system to support a team of human experts and intelligent agents. To achieve robust performance enhancement of such a system, we had proposed to perform task an...

It will take a lot of work to create a usable world full of usable products. In this Connections column, Russell Beale proposes we target a message to the public: Demand more from your products.---Manfred Tscheligi

A priori prediction of skilled human performance has the potential to be of great practical value but is difficult to carry out. This article reports on an approach that facilitates modeling of human behavior at the level of cognitive, perceptual, and motor operations, following the CPM-GOMS method (John, 1990). CPM-GOMS is a powerful modeling meth...

At the onset of evidence-based practice in software engineering, prospective disciples of this approach should inspect and learn from similar attempts in other disciplines. Having participated in the National Cancer Institute's multi-year effort compiling evidence-based guidelines for information-rich web-site design, I bring my personal experience...

The Keystroke-Level Model (KLM) has been shown to predict skilled use of desktop systems, but has not been validated on a handheld device that uses a stylus instead of a keyboard. This paper investigates the accuracy of KLM predictions for user interface tasks running on a Palm OS based handheld device. The models were produced using a recently dev...

A controlled experiment was performed to assess the usefulness of portions of a usability-supporting architectural pattern (USAP) in modifying the design of software architectures to support a specific usability concern. Results showed that participants using a complete USAP produced modified designs of significantly higher quality than participant...

Design patterns have been claimed to facilitate modification and improve understanding in software design. A controlled experiment was performed to assess the usefulness of portions of a usability-supporting architectural pattern (USAP) in modifying the design of software architectures to support a specific usability concern. Software engineering a...

Architecturally-sensitive usability scenarios are important usability concerns that require early consideration in software design so that architectural support can render them easy and cost-effective to implement. Examples include providing the ability to cancel a command, undo commands, aggregate data, etc. This chapter reports on our experiences...

Interactive products have definitely improved from the users' perspective in the 20 years since the HCI field emerged. Prior to the 1980s, only a small portion of the population of the western world used computers, primarily scientists, engineers, and financial analysts. Today, almost everyone in developed nations use dozens of computers each day:...

Software architects have techniques to deal with many quality attributes such as performance, reliability, and maintainability. Usability, however, has traditionally been concerned primarily with presentation and not been a concern of software architects beyond separating the user interface from the remainder of the application. In this paper, we i...

Software architects have techniques to deal with many quality attributes such as performance, reliability, and maintainability. Usability, however, has traditionally been concerned primarily with presentation and not been a concern of software architects beyond separating the user interface from the remainder of the application. In this paper, we p...

Although engineering models of user behavior have enjoyed a rich history in HCI, they have yet to have a widespread impact due to the complexities of the modeling process. In this paper we describe a development system in which designers generate predictive cognitive models of user behavior simply by demonstrating tasks on HTML mock-ups of new inte...

This demonstration reports results from the EU-funded project Ambient Agoras, investigating future applications of ubiquitous and ambient computing in workspaces. Instead of presenting underlying system technologies or evaluation findings, this demonstration ...

The integration of cognitive models offers interesting theoretical and practical benefits to the development of complex models, but at the same time brings up new challenges that the modeling community is only beginning to address. In this paper we present our experiences integrating models and tools for modeling,. Specifically, on the modeling end...

The Second International Workshop on the Relationships between Software Engineering and Human-Computer Interaction was held on May 24-25, 2004 as part of the 2004 International Conference on Software Engineering, in Edinburgh, Scotland This workshop was the second at ICSE and the fourth in a series held at international conferences in the past two...

This document is intended for researchers who will be using the MORSE simulatorfor their team-oriented experiments. It specifies some initial criteria that can be usedto evaluate the performance of the test subjects. This document also specifies the logevents that are generated by MORSE and that can be used for such evaluations.

Usability is an important quality attribute to be considered during software architecture design. Up to this point, usability has been served only by separating a system’s user interface from its functionality to support iterative design. However, this has the effect of pushing revisions to achieve usability toward the end of the software developme...

A system that supports the user's ability to cancel a command should be designed to achieve particular results. These results include the responses the system should make to the user, such as providing feedback to the user about the command's receipt, predicting the time the cancellation should take (for long-running cancellations), and indicating...

Computational modeling plays a central role in cognitive science. This book provides a comprehensive introduction to computational models of human cognition. It covers major approaches and architectures, both neural network and symbolic; major theoretical issues; and specific computational models of a variety of cognitive processes, ranging from lo...

An integrative cognitive architecture provides a computational framework within which to model a broad range of human behavior. It encompasses behavior from perception through cognition to motor action, and integrates performance, learning, and eventually moderator variables like cultural values or fatigue. To model any one behavioral data set usin...

Mobile computing has been an active area of research for the past decade, but its importance will increase substantially in the decade to come. One problem faced by designers of mobile systems is that of maintaining the illusion of connectivity even when network performance is poor or non-existent. The Coda file system uses its cache to maintain th...

This article investigates how and why people remember the existence of hidden information. To obtain data on this kind of memory phenomenon, we observed an experienced programmer doing her own work at her own computer. The programmer's interaction with the computer generates much more information than fits on the screen at once. Most of this inform...

CPM-GOMS is a modeling method that combines the task decomposition of a GOMS analysis with a model of human resource usage at the level of cognitive, perceptual, and motor operations. CPM-GOMS models have made accurate predictions about skilled user behavior in routine tasks, but developing such models is tedious and error-prone. We describe a proc...

In this panel, we will explore the impact that emerging new wireless technologies have on the way children learn, communicate and play. The challenge of interface design for children's wireless technologies will be discussed along with the opportunities these new technologies afford for social learning experiences. Panelists will discuss a range of...

Current computational modeling of human performance can benefit from reusable building blocks of human behavior. Using CPM-GOMS, a cognitively-based task analysis method used in HCI, we have been exploring the concept of reusable templates of common behaviors and their efficacy for generating zero-parameter a priori predictions of complex human beh...

Software engineers should consider usability as a quality attribute in their architectural a designs. Usability determines how effectively and comfortably an end-user can achieve the goals that gave rise to an interactive system. It is an important attribute to consider during all phases of software design, but especially during architectural desig...

Despite experience, many users do not make efficient use of complex computer applications. We argue that this is caused by a lack of strategic knowledge that is difficult to acquire just by knowing how to use commands. To address this problem, we present efficient and general strategies for using computer applications, and identify the components o...

The role of software architecture with respect to usability has evolved over the past 20 years. The architectures of the 1980s and early 1990s assumed that usability was primarily a property of the presentation of information. Therefore, simply separating the presentation from the dialogue and application made it easy to modify that presentation af...

For the last twenty years, techniques to design software architectures for interactive systems that support usability have been a concern of both researchers and practitioners. Recently, in the context of performing architecture evaluations, we were reminded that the techniques developed thus far are of limited utility when evaluating the usability...

Design decisions at the architecture level can have far-reaching effects on the qualities of a computer system. Recent developments in software engineering link architectural styles to quality attribute analysis techniques to predict the effects of architectural design decisions on the eventual manifestation of quality. An Attribute-Based Architect...

Many sources of empirical data can be used to evaluate an interface (e.g., time to learn, time to perform benchmark tasks, number of errors on benchmark tasks, answers on questionnaires, comments made in verbal protocols). This paper examines the relative contributions of both quanti�ta�tive and qualitative data gathered during a usability study. F...

Previous HCI-studies have compared usability evaluation methods quantitatively without supplementing these data with detailed qualitative data about how analysts actually learn and use methods. In contrast, we present two diary-based case studies that describe the processes of two novice analysts who learned about and applied the Cognitive Walkthro...

We compare three tools for creating GOMS models, QGOMS [2), CATHCI (17) and GLEAN3 [12], along several dimensions. We examine the representation and available constructs in each tool, the qualitative and quantitative design information provided, the support for building cognitively plausible models, and pragmatics about using each tool (e.g., how e...

To be effective in many HCI uses, a computational cognitive model must behave like a human, not simply get the job done with the least effort or in the least time. Building computational models that emulate human behavior on complex tasks is an ambitious undertaking, as they must include perception and motor behavior as well as problem-solving, lea...

Since the seminal Card, Moran, & Newell (1983) book, The psychology of human-computer interaction, the GOMS model has been one of the few widely known theoretical concepts in human-computer interaction. This concept has spawned much research to verify and extend the original work and has been used in real-world design and evaluation situations. Thi...

We videotaped ten volunteers as they used the World-Wide Web during a normal work day, asking them to think-aloud as they used the Web. We analyzed these videotapes at the level of user-intentions to form a taxonomy of tasks these people performed. We also coded these tapes at the level of user actions. The data reveal that several previous claims...

A central goal of many user interface development tools has been to make the construction of high quality interfaces easy enough that iterative design approaches could be a practical reality. In the last 15 years significant advances in this regard have been achieved. However, the evaluation portion of the iterative design process has received rela...