James D. Hollan

University of California, San Diego | UCSD · Cognitive Science & Computer Science and Engineering

The historical moment when a person worked in front of a single computer has passed. Computers are now ubiquitous and embedded in virtually every new device and system, connecting our personal and professional activities to ever-expanding information resources with previously unimaginable computational power. Yet with all the increases in capacity,...

Computational notebooks aim to support collaborative data analysis by combining code, visualizations, and text in a single easily shared document. Yet, as notebooks evolve and grow they often become difficult to navigate or understand, discouraging sharing and reuse. We present the design and evaluation of a Jupyter Notebook extension providing fac...

Computational notebooks combine code, visualizations, and text in a single document. Researchers, data analysts, and even journalists are rapidly adopting this new medium. We present three studies of how they are using notebooks to document and share exploratory data analyses. In the first, we analyzed over 1 million computational notebooks on GitH...

This study aimed to understand how college students participating in a 2-year randomized controlled trial (Project SMART: Social and Mobile Approach to Reduce Weight; N = 404) engaged their social networks and used social and mobile technologies to try and lose weight. Participants in the present study (n = 20 treatment, n = 18 control) were approa...

We think with things, with our bodies, with marks on paper, and with other people. Thinking is a distributed, situated, and social activity that exploits the extraordinary facilities of language, representational media, and embodied interaction with the world. Today we increasingly think with computers. This includes desktop and laptop computers bu...

This article introduces audio-enhanced paper photos enabled by digital pen technology as an interaction paradigm for supporting and understanding emotional wellbeing in late adulthood, particularly in the context of communication therapy activities and related social interactions. We describe the development of a multimodal pen-based system that en...

In this paper, we address the challenges of occlusion created by physical objects on interactive tabletops. We contribute an integrated set of interaction techniques designed to cope with the physical occlusion problem as well as facilitate organizing objects in hybrid settings. These techniques are implemented in ObjecTop, a system to support tabl...

The recent push towards patient-centered health care has put a greater emphasis on patient health literacy. Health literacy is influenced by communication between physicians and patients. We conducted research at a local health clinic to examine communication between physicians, patients, and interpreters that were present to assist patients with l...

In this paper we describe the development of a novel paper-digital interface for recording movement trajectories, designed to assist ethnographers and ethologists in analysis of social movement. While we focus on development of a system to aid analysis of elephant movement, the resulting interaction techniques and facilities are quite general. The...

Prototyping gestural and multitouch applications for desktop and mid-air interaction still requires expensive non-portable equipment, complex setup and calibration, and often reimplementation of tracking algorithms. We present Interactive Space, a prototyping framework designed to ease exploration of touch and gestural interaction in real-world set...

Photos are important for a wide variety of documentation purposes, and they play a special role in documenting physical, cognitive, and social changes in young children. This research focuses on the use of photos as evidence of development in preschool children and as a communication medium for current and future interactions. Through field observa...

Documentation of development is a critical aspect of the work of early childhood education teachers. Through field observations and interviews, we detail the process and tools used in documenting development for children age three months to five-years-old at one school. Teachers use paper forms and printed photos to record and analyze observations...

Photographs are powerful and pervasive media that facilitate communication and support reminiscence. Adding audio narrations to traditional paper photographs combines the familiarity of paper photos with the nuanced richness of voice. We enable creation of and interaction with audio-enhanced paper photographs through custom software deployed on a d...

Patient-centered health care and increased efficency are major goals of modern medicine. Research has shown that higher patient health literacy is linked to more successful health outcomes and the shift to electronic medical records (EMR) is hoped to increase efficency. Although EMR systems are designed to support physicians' clinical decision maki...

Understanding interaction with Electronic Health Records (EHR), often means to understand the multi modal nature of the physician-patient interaction, as well as the interaction with other materials (e.g. paper charts), in addition to analyze the tasks fulfilled by the doctor on his computerized system. Recent approaches started to analyze and quan...

Hybrid paper-digital desktop workspaces have long been of interest in HCI, yet their design remains challenging. One continuing challenge is to support fluid interaction with both paper and digital media, while taking advantage of established practices with each. Today researchers are exploiting depth cameras and computer vision to capture activity...

Ubiquitous data capture and visualization technologies make it possible to exploit histories of computer-mediated activity to develop activity-enriched computing.

We conducted a study in a pediatric trauma center to elicit design requirements for the TraumaPen system-a mixed paper-digital interface using a digital pen and a wall display-to support situation awareness during trauma resuscitation. In this paper, we describe the field research that informed the initial system prototype and then present findings...

Researchers from many disciplines are taking advantage of increasingly inexpensive digital video to capture extensive records of human activity in real-world settings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. While recent work is beginning to develop techniques for vis...

Hybrid paper-digital interfaces are a promising approach for supporting language activities. The familiarity of pen and paper makes it a particularly attractive media for many user groups, including young children. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures...

Multitouch tabletop displays are a promising technology for supporting small group discussion and learning activities. Based on a view of human activity as multimodal, distributed, and embodied, we examine the pedagogical affordances of interactive tabletop technology, how it supports small group scientific discussion, and compare it with use of tr...

Large-scale display walls, and the high-resolution visualizations they support, promise to become ubiquitous. Natural interaction with them, especially in collaborative environments, is increasingly important and yet remains an on-going challenge. Part of the problem is a resolution mismatch between low-resolution input devices and high-resolution...

Hybrid paper-digital interfaces enabled by digital pens are a promising approach for supporting collaborative language, communication, and socialization activities. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures. Currently, generating custom interactive paper d...

Phones with integrated pico projectors are starting to be marketed as devices for business presenta-tions and media viewing, and researchers are beginning to design projection-specific applications and interaction techniques to explore a broader array of possible uses. To begin to document how people use projector phones out-side the laboratory, we...

We conducted a study in a pediatric trauma center to elicit design requirements for the TraumaPen system-a mixed paper-digital interface using a digital pen and a wall display-to support situation awareness during trauma resuscitation. In this paper, we describe the field research that informed the initial system prototype and then present findings...

Aphasia is characterized by a reduced ability to understand and/or generate speech and language. Speech-language therapy helps individuals with aphasia regain language and cope with changes in their communication abilities. The therapy process is largely paper-based, making multimodal digital pen technology a promising tool for supporting therapy a...

The miniaturization and commoditization of computing components is enabling a world in which computing is ubiquitous. The most recent trend is the unbundling of the monolithic “computer ” into fragmentary, appliancelike components that provide specialized functions for data storage, transmission, processing, entry, and display. Critical to the visi...

Digital social media have transformed how we communicate and manage our relationships. Despite its portability, sketching as a social medium has been largely left behind. Given sketching's unique affordances for visual communication this absence is a real loss. Sketches convey visuo-spatial ideas directly, require minimal detail to render concepts,...

Objective: Increasingly health care facilities are adopting electronic medical record systems and installing computer workstations in patient exam rooms. The introduction of computer workstations into the medical interview process makes it important to consider the impact of such technology on older patients as well as new types of interfaces that...

We present ChronoViz, a system to aid annotation, visualization, navigation, and analysis of multimodal time-coded data. Exploiting interactive paper technology, ChronoViz also integrates researcher's paper notes into the composite data set. Researchers can navigate data in multiple ways, taking advantage of synchronized visualizations and annotati...

PLink is a system for integrating physical and computer desktops by creating paper links to digital resources. PLink leverages diverse formats of physical paper, ranging from tiny stickers that can be easily incorporated into traditional paper media to very large deskpad sheets that make the physical desktop partially interactive. We present PLink...

The intertwining of everyday life and computation, along with a new generation of inexpensive digital recording devices and storage facilities, is revolutionizing our ability to collect and analyze human activity data. Such ubiquitous data collection has an exciting potential to augment human cognition and radically improve information-intensive wo...

From vocational education to neurosurgery residencies, apprenticeship training is characteristic of how people are acculturated to a profession. One of the primary goals of professional education is to develop skillful performance. Expert skill includes an integrated set of perceptual and motor abilities. During practical teaching, instructors depl...

Multi-touch surfaces enable multi-hand and multi-person direct manipulation interfaces. However, they lack haptic feedback. Tangible interfaces (TUIs) are a promising approach to this issue but most are special-purpose with fixed physical and visual appearance. This chapter provides an overview of recent work to add haptic feedback to interactive s...

After a stroke or brain injury, it may be more difficult to understand language and communicate with others. Speech-language therapy may help an individual regain language and cope with changes in their communication abilities. Our research examines the process of speech-language therapy with an emphasis on the practices of therapists working with...

"Timelines" provides perspectives on HCI history, glancing back at a road that sometimes took unexpected branches and turns. History is not a dry list of events; it is about points of view and differing interpretations. Jonathan Grudin, Editor | jgrudin@microsoft.com

While mobile phones have become ubiquitous instruments of communication and social interaction, they still require explicit interaction, placing high demands on attention. Engaging the periphery of users' attention offers opportunities for awareness and interaction while reducing demands on attention and risks of disruption. We explore the mobile p...

This paper examines accessibility issues of surface computing with older adults and explores the appeal of surface computing for health care support. We present results from a study involving 20 older adults (age 60 to 88) performing gesture-based interactions on a multitouch surface. Older adults were able to successfully perform all actions on th...

Concurrent interaction with physical and digital media is ubiquitous in knowledge work. Although tabletop systems increasingly support activities involving both physical and digital media, patterns of use have not been systematically assessed. This paper contributes the results of a study of spatial usage patterns when physical and digital items ar...

Although mobile phones can help us stay connected with friends, family, and colleagues, these personal communication devices can also isolate us from the world around us by drawing our attention into a private digital realm. In contrast, public displays can catalyze conversation with people nearby. Mobile phones with integrated pico projectors have...

We present UbiSketch, a tool for ubiquitous real-time sketch-based communication. We describe the UbiSketch system, which enables people to create doodles, drawings, and notes with digital pens and paper and publish them quickly and easily via their mobile phones to social communication chan-nels, such as Facebook, Twitter, and email. The natural p...

We present DataPrism, a new interactive visualization tool to aid analysis of multimodal activity data. DataPrism enables analysts to visualize, annotate, and link multiple time-based data streams, including video, log files, and paper-based digital notes. Automated analysis is supported through a plug-in mechanism. A usage scenario is employed to...

While recent advances in graphics, display, and computer hardware support ultra-scale visualizations of a tremendous amount of data sets, mechanisms for interacting with this information on large high-resolution wall displays are still under investigation. Different issues in terms of user interface, ergonomics, multi-user interaction, and system f...

We introduce HIPerPaper, a novel digital pen and paper interface that enables natural interaction with a 31.8 by 7.5 foot tiled wall display of 268,720,000 pixels. HIPerPaper provides a flexible, portable, and inexpensive medium for interacting with large high-resolution wall displays. While the size and resolution of such displays allow visualizat...

The unbundling of the monolithic computer and its movement off the desktop and into the world are creating new work and living practices. For good and for ill, our professional, personal, and social activities are increasingly dependent on and intertwined with digital computation and communication facilities. Visual interfaces through which we typi...

Currently a new generation of inexpensive digital recording devices and storage facilities is revolutionizing data collection in behavioral science, extending it into situations that have not been typically accessible and enabling the examination of the fine details of action captured in meaningful settings. The ability to record and share such dat...

In this paper we compare the affordances of presenting educational material on a tabletop display with presenting the same material using traditional paper handouts. Ten pairs of undergraduate students used digital or paper materials to prepare for exams during four one-hour study sessions over the course of a term. Students studying with the table...

We present Silicone iLluminated Active Peripherals (SLAP), a system of tangible, transparent widgets for use on vision-based multi-touch tabletops. SLAP Widgets are cast from silicone or made of acrylic and include sliders, knobs, keyboards, and keypads. They add tactile feedback to multi-touch tables and can be dynamically relabeled with rear proj...

In this paper, we discuss the need for an effective representation of video data to aid analysis of large datasets of video clips and describe a prototype developed to explore the use of spatio-temporal interest points for action recognition. Our focus is on ways that computation can assist analysis.

We present Silicone iLluminated Active Peripherals (SLAP), a system of tangible, translucent widgets for use on multitouch tabletops. SLAP Widgets are cast from silicone or made of acrylic, and include sliders, knobs, keyboards, and buttons. They add tactile feedback to multi-touch tables, improving input accuracy. Using rear projection, SLAP Widge...

ABSTRACT We present SLAPbook, an application using SLAP, translu- cent and tangible widgets for use on vision-based multi-touch tabletops in Single Display Groupware (SDG) environments. SLAP stands for Silicone ILluminated Active Peripherals and includes widgets such as sliders, knobs, keyboards, and buttons. The widgets add tactile feedback to mul...

Communication between people is inherently multimodal. People employ speech, facial expressions, eye gaze, and gesture, among other facilities, to support communication and cooperative activity. Complexity of communication increases when a person is without a modality such as hearing, often resulting in dependence on another person or an assistive...

Paper persists as an integral component of active reading and other knowledge-worker tasks because it provides ease of use unmatched by digital alternatives. Paper documents are light to carry, easy to annotate, rapid to navigate, flexible to manipulate, and robust to use in varied environments. Interactions with paper documents create rich webs of...

Tactile feedback allows devices to communicate with users when visual and auditory feedback are inappropriate. Un- fortunately, current vibrotactile feedback is abstract and not related to the content of the message. This often clashes with the nature of the message, for example, when sending a comforting message. We propose addressing this by exte...

This paper describes the design and evaluation of Shared Speech Interface (SSI), an application for an interactive multitouch tabletop display designed to facilitate medical conversations between a deaf patient and a hearing, non- signing physician. We employ a participatory design process involving members of the deaf community as well as medical...

Being mobile influences not only the types of information people seek but also the ways they attempt to access it. Mo- bile contexts present challenges of changing location and social context, restricted time for information access, and the need to share attentional resources among concurrent activities. Understanding mobile information needs and a...

This research examines how representational gestures (Kita 2000), made by scientists during collaborative discussion in a biochemistry lab, are used in formulating scientific theory. By analyzing digital video of lab meetings and interviews, we find that representational gestures are frequently used to reference, modify, and embody portions of exis...

We present SLAP, a system of tangible, translucent wid-gets for use on FTIR-based multi-touch tabletops. Our ini-tial widgets include a SLAP keyboard, button, slider, knob, and lens. SLAP widgets add tactile feedback to multi-touch tables, improving input accuracy. They can be relabeled dynamically via table rear projection, and provide inexpen-siv...

This work describes a multi-year ethnography-for-design study in a dental hygiene training program in San Diego, USA. We performed an ethnographic analysis of instructional practices used in clinical instruction, and helped design a new training laboratory, equipped with digital media technology, through which students and instructors could access...

This paper discusses the redesign of PeopleSoft's Enterprise Expenses product from a product that was notorious for it's complexity into a product that was both usable and one of PeopleSoft's best selling products. The process used was a combination ...

Dynapad is a development environment designed to support research prototyping of multiscale workspaces. In this paper we describe applications designed to facilitate visual access to and spatial organization of digital photo collections and personal libraries of PDF documents. The research objective is to explore a generalization of the notion of a...

This research examines how representational gestures (Kita 2000) made by scientists during collaborative discussions in a biochemistry lab, are used as cognitive artifacts for formulating scientific theory. By analyzing digital video of lab meetings and interviews with scientists, we find that representational gestures serve to reference and animat...

During shared viewing printed photos frequently serve as anchors for conversations. Conversational interactions are less common when images are not printed. People do not commonly gather around a monitor to look at digital images, although the passing around of cell phones and digital cameras displaying images is increasingly common. The congeniali...

This paper describes an interdisciplinary research collaboration to design a human-centered driver assistance system. Driving behavior is captured using a novel intelligent vehicle test bed. The synchronized capture of driver behavior and driving context provides an empirical basis for the design and evaluation.

We explore and extend the metaphor of "piles" to include computationally-enriched piles, portable regions of automation in Dynapad, a multiscale workspace. We describe these collection-management tools and how their design was informed by observing people organizing collections of personal digital photographs

Scheduling access to people and selectively sharing the state of one’s activities are essential elements of collaborating with others. These processes are fundamental to arranging face-to-face meetings and coordinating access to shared information. Examples include scheduling meetings, contacting others in real time (by phone or instant messaging),...

IRYS is a tool for the replay and analysis of gaze and touch behavior during on-line activities. Essentially a "multimodal VCR", it can record and replay computer screen activity and overlay this video with a synchronized "spotlight" of the user's attention, as measured by an eye-tracking and/or touch-tracking system. This cross-platform tool is pa...

Historically, the dominant paradigm in HCI, when it appeared as a field in early 80s, was information processing ("cognitivist") psychology. In recent decades, as the focus of research moved beyond information processing to include how the use of technology emerges in social, cultural and organizational contexts, a variety of conceptual frameworks...

The web is enormous and constantly growing. User-interfaces for web-based applications need to make it easy for people to access relevant information without becoming overwhelmed or disoriented. Today's interfaces employ textual representations almost exclusively, typically organized in lists and hierarchies of web-page titles or URL taxonomies. Gi...

The web is enormous and constantly growing. User-interfaces for web-based applications need to make it easy for people to access relevant information without becoming overwhelmed or disoriented. Today's interfaces employ textual representations almost exclusively, typically organized in lists and hierarchies of web-page titles or URL taxonomies. Gi...

We are quickly passing through the historical moment when people work in front of a single computer, dominated by a small CRT and focused on tasks involving only local information. Networked computers are becoming ubiquitous and are playing increasingly significant roles in our lives and in the basic infrastructures of science, business, and social...

Introduction to HITS: Human Interface Tool Suite, in Intelligent User Interfaces, (Sullivan & Tyler, Eds.), 1991, pp. 293-337. [15] James D. Hollan and Scott Stornetta, Beyond Being There, Proceedings of ACM SIGCHI'92, pp. 119-125. (also appeared as a chapter in Readings in Groupware and Computer Supported Cooperative Work (Becker, Ed.), 1993, 842-...