Thomas Tullis

• PhD
• Bentley University

Two-state toggle buttons (e.g., On/Off, Yes/No, etc) are very common on web sites and in mobile apps. Both the Android and iOS toolkits have standard (different) versions. In the old days, we used two radio buttons for this. In doing some usability testing of pages that contained toggle buttons (using a standard we had adopted), we noticed that som...

Research suggests that a main image on a homepage creates a distinct visual hierarchy that maybe inherently appealing to people. This in turn suggests that main images may play an important role in shaping user aesthetic reactions to a homepage. Research also shows that images of people are effective in drawing attention and that these images are p...

This chapter describes a variety of behavioral and physiological measures that might be helpful in usability testing as additional ways of learning about the users' experiences with a product and their reactions to it. Some of these can be detected by careful observation, and some require specialized equipment. A structured approach to collecting o...

Performance metrics are powerful tools to evaluate the usability of any product. Performance metrics are always based on participants' behavior rather than what they say. There are five general types of performance metrics. Task success metrics are used when one is interested in whether participants are able to complete tasks using the product; in...

Figure 5.1. Severity rating scale taking into account problem frequency and impact on the user experience.

Usability data are building blocks and each piece of usability data can be used to create new metrics. There are two ways to derive new usability metrics from existing data, namely, by combining more than one metric into a single usability measure, and by comparing existing usability data to expert or ideal results. The easiest way to combine diffe...

This chapter introduces a number of topics related to the measurement or analysis of usability data but not traditionally thought of as part of “mainstream” usability data. These include information from live data on a production website, data from card-sorting studies, data related to the accessibility of a website, the topic of Six Sigma and how...

This chapter presents a set of seven case studies. Each case study illustrates how online usability studies have been used in a wide variety of situations, including important lessons learned and impacts to the bottom line. These case studies also describe how online usability testing has been used to evaluate a wide variety of products, utilizing...

This chapter brings all the key pieces of information together in the form of 10 keys for carrying out a successful online usability study. These keys can help get the most out of the study, save money and time, and even enhance the reputations of the user and the team. These keys are not set in stone for every researcher or for every study. They a...

This chapter reviews some basic statistical concepts, a few common statistical procedures, and discusses how to design a well-thought-out usability study. A well thought-out study can save time and effort and answer research questions clearly. To design a well-thought-out usability study, one needs to answer a few questions, such as what types of p...

A previous study had shown that users tend to associate certain elements of a web page with ads, and tend to avoid them. The current study revisited that data to see if this "ad blindness" varies with the age or gender of the users. We found that younger adults (20s and 30s) exhibited significantly more ad blindness than older adults. We also found...

To be successful, websites must not only contain useful information, but provide that information in a quickly and easily accessible manner. One method of delivering this experience is to design websites that effectively guide users' attention to key information on the page. Grounded in the model of visual hierarchy, this study examines several att...

Empirical evidence suggests that users often exhibit a viewing pattern that favors the top and left sides of web pages. This viewing pattern may cause users to miss a great deal of information. Grounded in the model of visual hierarchy, this study examines the impact of visual complexity on how users view a page. The results show that users’ viewin...

The baby boom generation, born from 1946 to 1964, and Generation Y, born from 1977 to 1990, form two important groups of online users in terms of numbers and economic impact. Understanding their Web preferences is of great importance to companies, particularly because the literature suggests that these generations may differ in how they view Web pa...

Previous research had shown that pictorial passwords, where users recognize their target images among distractors, have potential for improving the usability of authentication systems. A method using personal photos provided by the users as their targets, shown among highly similar distractors, showed the most promise for both accuracy and security...

This chapter provides a comprehensive review of some of the out-of-the-box technical solutions such as Keynote's WebEffective, RelevantView, UserZoom, and Loop 11. It reviews the basic elements common to each technology and unique features to consider when choosing a vendor. These are the tools that support the kinds of large-scale online usability...

This chapter provides a set of step-by-step directions for building a personal online usability study using standard online survey tools and a little bit of HTML and JavaScript. Techniques that allow users to collect task success data, task times, task-based ratings and comments, and posttask ratings and comments, including ratings using the system...

This chapter provides all the tools needed to analyze data. Ways to handle verbatim responses, interpret task success and efficiency metrics, understand click path behavior, identifying the most salient usability issues, and presenting usability data in a compelling way are reviewed. The chapter is all about making sense of data from the online stu...

This chapter reviews the basic steps in data preparation: downloading data, performing data quality checks, removing participants, removing tasks, recoding variables, and creating new variables. It also focuses on what one needs to do once all data comes in. Techniques for cleaning up data, such as detecting fraudulent participants, running consist...

This chapter reviews important decisions one needs to make about setting up an online usability study. Issues such as study types, budgets, and time frames are also discussed. These high-level tactical decisions drive a host of other study characteristics, such as sample sizes, recruiting strategy, and participant compensation. High-level study goa...

This chapter is an aid to design the study, including introducing it, writing effective screener questions, designing tasks, and choosing the right posttask questions. It walks the users through the steps in designing an online usability study and covers all the major sections of a study in the order a participant would likely encounter them: intro...

This chapter includes material on piloting, singular and phased launches, and monitoring results. The various aspects of the study to validate through piloting are discussed, including technical checks, usability checks, and data checks. Factors around timing and phasing the study launch and monitoring the results are also reviewed. Piloting is esp...

This chapter introduces online usability studies. The background information needed to plan an online usability study is provided. The usability lab is a great resource for learning about the usability of some product based on a relatively small number of users. Online usability studies allow users to reach other users all over the world, without w...

This study examined seven methods for date entry on the web, including text input, selection using dropdown menus or radio buttons, and combination approaches. The focus was on the entry of dates that may be many years in the past, such as a date of birth. A total of 776 participants, randomly assigned to one of the seven methods, each entered ten...

Generation Y (age 18–31) is a very large and economically powerful generation, containing eighty-two million people and spending $200 billion annually. It is not surprising that companies are interested in gaining the patronage of this group, particularly via the web. Surprisingly, very little research into making web pages appealing to this import...

Creating a positive user experience is a fundamental element of producing a successful web page. Three important components of user experience are visual appeal, trust, and efficiency. This study extends past research by examining the effect of images of faces on the visual appeal, efficiency, and trustworthiness of a page. Data is collected using...

Two types of link treatments, underlined or non- underlined, were investigated in the context of three web pages. Over 1,000 participants completed tasks for which the answers were found either on the pages or by clicking a link. Task accuracy, speed, and ratings were collected in an online study. Conflicting findings suggest that primarily navigat...

Usability testing and user experience research typically take place in a controlled lab with small groups. While this type of testing is essential to user experience design, more companies are also looking to test large sample sizes to be able compare data according to specific user populations and see how their experiences differ across user group...

Usability testing and user experience research typically take place in a controlled lab with small groups. While this type of testing is essential to user experience design, more companies are also looking to test large sample sizes to be able compare data according to specific user populations and see how their experiences differ across user group...

Three studies were conducted to investigate the effects of faces on webpages. In Study I, eye-tracking data showed that users were clearly drawn to faces when asked to look at pages and report what they remember. In Study II, the presence of a face next to a message on a webpage caused users to have a harder time finding that message. In Study III,...

Self-reported data gives the most important information about users' perception of the system and their interaction with it. The most efficient way to capture self-reported data in a usability test is with the rating scale. Many different techniques are available for getting usability metrics from self-reported data. The best times to collect self-...

This chapter elaborates the most important ten steps to establish a culture of usability metrics within any organization. These are: sell usability and the power of metrics, start small and work your way up, make sure you have the time and money, plan early and often, benchmark your products, explore your data, speak the language of business, show...

This chapter reviews some simple metrics around usability issues, different ways of identifying usability issues, prioritizing the importance of different types of issues, and factors one needs to think about when measuring usability issues. Measuring usability issues helps to answer some fundamental questions about how good the design is and where...

Running a usability study including metrics requires some planning. The first decision to make when planning a usability study is how the data will ultimately be used within the product development life cycle. There are essentially two general ways to use data: formative and summative. A formative approach involves collecting data to help improve t...

This chapter describes the usability metrics as a powerful tool for successfully evaluating the user experience for any product. It serves as a practical guide to answer the question of how to measure the usability of any product. Usability plays a much wider role in our lives than most people realize. It cuts across cultures, age, gender, and econ...

At eighty-two million people, Generation Y (18-31) is the largest generation since the Baby Boomers, spending $200 billion yearly. Little past research investigates the web preferences of Generation Y, although some oblique studies have been conducted. This study extends past research by looking specifically at the web design preferences of this im...

Measuring the User Experience was the first book that focused on how to quantify the user experience. Now in the second edition, the authors include new material on how recent technologies have made it easier and more effective to collect a broader range of data about the user experience. As more UX and web professionals need to justify their desig...

Effectively measuring the usability of any product requires choosing the right metric, applying it, and effectively using the information it reveals. Measuring the User Experience provides the first single source of practical information to enable usability professionals and product developers to do just that. Authors Tullis and Albert organize doz...

Though the Web and those who use it have changed considerably in the last decade, a digital divide between older and younger users persists. Older users still use the Web less than younger users, and more commonly experience significant usability issues when they do. With the emergence of Web 2.0 technologies, we have the ability to close that divi...

An eye-tracking study of a prototype website was conducted with 10 younger adults (ages 20-39) and 10 older adults (ages 50-69) to determine if there are differences in how they scan webpages. They performed the same tasks on the website. On the average, the older adults spent 42% more time looking at the content of the pages than did the younger a...

A successful w ebsite not only contains useful information but also displays this information in a way that is easily recognized by its users. One way to accomplish this is by investigating users' visual preferences when using a web page. While each user may have unique tastes, they all can be organized into demographic groups with associated visua...

Five questionnaires for assessing the usability of a website were compared in a study with 123 participants. The questionnaires studied were SUS, QUIS, CSUQ, a variant of Microsoft's Product Reaction Cards, and one that we have used in our Usability Lab for several years. Each participant performed two tasks on each of two websites: finance.yahoo.c...

The Technology Acceptance Model (TAM) is one of the most widely used models in the MIS literature. Verified by many studies, TAM asserts that the perceived ease of use and perceived usefulness of an information technology is instrumental in its adoption. While previous research has been valuable in explaining how and why the perception of ease of u...

Five methods for eliciting subjective ratings after each task in a usability test were evaluated. The methods included simple Likert scales as well as a technique derived from Usability Magnitude Estimation. They were tested in a large-scale online study in which participants performed six tasks on an Intranet site. Performance data for the tasks r...

E-commerce companies must strive to understand the changing physical, cognitive, and social requirements of their customers. The older demographic is the fastest growing worldwide [10, 15, 16, 20] and also the fastest growing online [8, 13]. In many countries, this aging trend is driven by the increase in birth rate that occurred after World War II...

Pictorial passwords, where the user recognizes "target" images among "distractors", appear to have potential for improving the usability of authentication systems. We conducted three exploratory studies on the use of personal photos for authentication over a three-month period. Participants provided 8-20 photos of personal significance to them but...

While usability research focuses mainly on sighted users and somewhat on blind users with assistive devices, little is known about the needs of low vision users. The goal of this study was to determine if we could design an interface that would specifically address the needs of low vision users and improve both their performance and overall user ex...

The goal of our study was to learn if providing older adults with tactile feedback would improve their performance on the Web. The study examined the effects on performance of older users completing tasks on a prototype website using both a normal mouse and a mouse that provides haptic, or tactile, feedback. A total of 18 participants with low and...

This study explored the effect of displaying sideways navigation and content in graphical applications for handheld devices. Because these devices can be easily rotated during use, alternate orientations for screen elements may be acceptable. Participants performed tasks on prototypes emulating a wireless brokerage application presented on a Pocket...

A study was conducted to assess the minimum number of participants needed for a card-sorting study. Similarity matrices and tree structures from various sample sizes were compared to those based on a set of 168 participants. Results indicate that reasonable structures are obtained from 20-30 participants.

Level of Web experience is often a factor for which researchers attempt to control while conducting experimental studies on Web usability. It is typically measured by some means of self-assessment that often includes questions regarding long-term usage, frequency of use, and the types of activities performed while using the Web. A common assumption...

Users of different ages and gender were asked to physically "build" their ideal financial services homepage by choosing pieces of actual Web page content that were printed on magnetic paper. By sticking the pieces of content within a simulated Web browser that was fixed on to a magnetic white board, users could build their homepage. Both age and ge...

We conducted two usability studies that included a total of 49 participants ranging in age from 20 to 82. The goal of Study 1 was to learn whether there were differences in how older adults interact with the Web and whether changes in text size would affect performance. Users completed tasks on a prototype employee/retiree benefits site using vario...

Many usability specialists practicing today have not had the benefit of formal education in the field, instead bringing unique value from their various backgrounds. This panel will address how (if at all) individuals' backgrounds contribute to their approach to usability. The panel will also investigate potential career paths and connections betwee...

This paper presents the results of two studies comparing traditional lab-based usability testing and remote Web-based usability testing of Web sites. Two sites were tested: an employee benefits site and a financial information site. The remote tests used an automated technique whereby the users participated from their normal work locations using th...

We conducted two usability studies that included a total of 49 participants ranging in age from 20 to 82. The goal of Study 1 was to learn whether there were differences in how older adults interact with the Web and whether changes in text size would affect performance. Users completed tasks on a prototype employee/retiree benefits site using vario...

Web usability is a somewhat elusive concept. We think most people would agree that there is no magic "formula" or set of rules to follow that will guarantee a usable web site in all circumstances. Consequently, it's critical to have a comprehensive "bag of tricks" for evaluating the usability of any given web site. The purposes of this SIG are to s...

In redesigning the Intranet at Fidelity Investments, we used a paper-based technique for getting user feedback on initial design concepts for the web pages. The technique involved color printouts of "Greeked" versions of five different candidate web page designs. Users had to try to identify nine standard elements that appeared on each page (e.g.,...

This poster describes a study conducted by the Human Interface Design department at Fidelity Investments Systems Company. The purpose of this study was to obtain performance and preference data about various methods of displaying data definitions in online help. The four methods studied were an alphabetical list of data elements with pop-up definit...

This paper describes a study comparing different techniques for visually distingishing required from optional input fields in a form-filling application. Seven techniques were studied: no indication, bold field labels, chevrons in front of the labels, check marks to the right of the input fields, a different background color, grouping them separate...

This presentation describes the Human Interface Design department at Fidelity Investments. Although not in the computer hardware or software business, Fidelity develops an amazingly wide variety of systems in support of our business. The Human Interface Design department, which is composed of people from a variety of backgrounds, provides several k...

The readability of twelve different fonts and sizes in the Microsoft Windows environment was studied. The specific fonts were Arial, MS Sans Serif, MS Serif, and Small Fonts. Their sizes ranged from 6.0 to 9.75 points. These were presented using black text on either a white or gray background and either bold or non-bold style. There were significan...

A common type of display in many computer systems is a table that shows values of several different properties for a list of files. In designing such a display for a forthcoming product, we thought it would be helpful to allow the user to change the order in which the fields appear in the columns of the table. The specific fields that the user coul...

This panel will explore the varied uses of prototyping in the user interface design process. We expect to show that there is no single thing called “user interface prototyping” and that the differences are, in many ways, greater than the similarities. Panelists have been chosen to represent a wide cross section of user interface design tasks. Colle...

This chapter discusses the concept of screen design. There is a large base of knowledge about screen design that is applicable to a wide variety of technologies. This knowledge is derived from several sources: (1) basic psychological research, (2) human factor studies, (3) experiences of designers and users, and (4) graphic design experience. Much...

A systems engineering process is developed to assist Space Station designers to understand the underlying operational system of the facility so that it can be physically arranged and configured to support crew productivity. The study analyzes the operational system proposed for the Space Station in terms of mission functions, crew activities, and f...

In previous research, six parameters for objectively evaluating screen formats were identified. Based upon analyses of usability data from 520 displays, regression equations were developed for predicting user's search times to extract information from the displays as well as their subjective ratings of the displays. Recently, a program for measurin...

The method or input device by which a user indicates a selection to the computer is an important element in the design of that system. Recent advances in the technology of human-computer interfaces suggest that a "direct-manipulation" style of interface provides many advantages over traditional human-computer interfaces (e.g. Hutching, Hollan and N...

Researchers and developers generally agree that user interface problems are more easily solved at the beginning of the design process than toward the end. It is much less costly to make changes to the user interface before significant effort has been put into the development of software. One way to evaluate a user interface is to ask representative...

This paper presents a comparison of experimental results with predictions obtained from Tullis' (1984) model of search times for tabular displays. Three levels of information density for displays with and without highlighting were used in a series of experiments. The highlighting of information was done by adding graphic boundaries (lines). Two lev...

In previous research, six parameters for objectively evaluating screen formats were identified. Based upon analyses of usability data from 520 displays, regression equations were developed for predicting user's search times to extract information from the displays as well as their subjective ratings of the displays. Recently, a program for measurin...

A user interface evaluation technique has been developed that: a) accurately documents user inputs and system responses; b) follows the logic behind user actions; c) obtains behavior that is not biased by the evaluation technique; d) does not cause the participants to feel uncomfortable; and, e) easily illustrates user interface problems to others....

Before an optimum layout of the facilities for the proposed Space Station can be designed, it is necessary to understand the functions that will be performed by the Space Station crew and the relationships among those functions. Five criteria for assessing functional relationships were identified. For each of these criteria, a matrix representing t...

The Space Station must be designed to facilitate all of the functions that its crew will perform, both on-duty and off-duty, as efficiently and comfortably as possible. This paper examines the functions to be performed by the Space Station crew in order to make inferences about the design of an interior layout that optimizes crew productivity. Twen...

The development of a large menu-based interface to an operating system posed a number of interesting user interface questions. Among those were how to determine the user's view of the relationships among the myriad of functions in the system, and how to reflect those relationships in a menu hierarchy. An experiment utilizing a sorting technique and...

A review of the literature on alphanumeric displays, especially computer-generated displays, suggests that four basic characteristics of display formats affect how well users can extract information from the displays: (1) Overall density--the number of characters displayed, expressed as a percentage of the total spaces available; (2) Local density-...

This paper surveys the literature on alphanumeric display formatting, focusing primarily on computer-generated displays. Two general areas are included: guidelines addressing display design and empirical studies of displays. Four characteristics of display formats are described: overall density, local density, grouping, and layout complexity. Objec...

A review of the literature on alphanumeric displays, especially computer-generated displays, suggests that four basic characteristics of display formats affect how well users can extract information from the displays: (1) Overall density- the number of characters displayed, expressed as a percentage of the total spaces available; (2) Local density-...

Four different types of CRT display formats were evaluated in the context of a computer-based telephone line testing system. The formats considered were narrative, which used complete words and phrases; structured, which used a tabular format; black-and-white graphics, which used a schematic of the telephone line; and color graphics, which also use...

A study was conducted to determine the effectiveness of four alternative CRT display formats for diagnostic data. The formats were narrative text, structured text, black and white graphics, and color graphics. The evaluation measured subjects' speed and accuracy of response to a question requiring interpretation of a display. The results indicated...

Mean Computer Subsystem (CSS) response times and task type were studied to determine their effect on operator performance in an interactive computer system. The mean CSS response times were categorized into four groups; “zero” seconds, “low” (4.0 to 7.9 sec.), “medium” (16.8 to 17.3 sec.), and “high” (24.2 to 30.3 sec.). A simple data entry task an...

In the development of a computer-based system for automated testing of telephone lines and improved processing of customer trouble reports, a specific subsystem was developed for the personnel who will work in the automated repair bureau. This subsystem, the Personnel Subsystem (PSS), consists of (1) the people within the system, (2) the criteria b...

Using a test-study recognition task, stimulus variability was manipulated by pairing a given critical noun with either one or four different adjectives on successive presentations. This defined two levels of variability, low and high, respectively. Yes-No recognition latencies to the fifth presentation of critical items indicated that low variabili...

The usability profession is experiencing the current economic downturn just like everyone else. This article offers ten tips for usability professionals trying to weather this economic storm: