Shumin Zhai

Shumin Zhai
Google Inc. | Google

PhD

About

181
Publications
162,377
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
10,344
Citations
Introduction
Principal Scientist at Google; Current areas of research: Human-Computer Interaction, mobile user interfaces, haptics, text input / smart keyboard , human performance modelling, AI-based user interfaces, shape writing / gesture typing / swiping keyboard; Past research: eye-tracking, 6 DOF input control and 3D user interfaces, VR &AR, control theory and systems
Additional affiliations
January 2011 - April 2018
Google Inc.
Position
  • Researcher

Publications

Publications (181)
Conference Paper
Full-text available
We study the effects of haptic augmentation on tapping, path following, and drag & drop tasks based on a recent flagship smartphone with refined touch sensing and haptic actuator technologies. Results show actuated haptic confirmation on tapping targets was subjectively appreciated by some users but did not improve tapping speed or accuracy. For dr...
Conference Paper
Full-text available
Entering text without having to pay attention to the keyboard is compelling but challenging due to the lack of visual guidance. We propose i'sFree to enable eyes-free gesture typing on a distant display from a touch-enabled remote control. i'sFree does not display the keyboard or gesture trace but decodes gestures drawn on the remote control into t...
Conference Paper
Full-text available
Active Edge is a feature of Google Pixel 2 smartphone devices that creates a force-sensitive interaction surface along their sides, allowing users to perform gestures by holding and squeezing their device. Supported by strain gauge elements adhered to the inner sidewalls of the device chassis, these gestures can be more natural and ergonomic than o...
Conference Paper
Full-text available
As an interdisciplinary field, CHI embraces diverse research practices ranging from controlled lab experiments to field studies in homes and communities. While quantitative research in the lab emphasizes the scientific rigor for testing hypotheses, qualitative research in the wild maximizes the understanding of contexts of using technologies. Furth...
Conference Paper
Full-text available
Human-computer input performance inherently involves speed-accuracy tradeoffs---the faster users act, the more inaccurate those actions are. Therefore, comparing speeds and accuracies separately can result in ambiguous outcomes: Does a fast but inaccurate technique perform better or worse overall than a slow but accurate one? For pointing, speed an...
Conference Paper
Full-text available
Despite their learning advantages in theory, marking menus have faced adoption challenges in practice, even on today's touchscreen-based mobile devices. We address these challenges by designing, implementing, and evaluating multiple versions of M3 Gesture Menu (M3), a reimagination of marking menus targeted at mobile interfaces. M3 is defined on a...
Conference Paper
Full-text available
In contrast to the extensive studies on static target pointing, much less formal understanding of moving target acquisition can be found in the HCI literature. We designed a set of experiments to identify regularities in 1D unidirectional moving target selection, and found a Ternary-Gaussian model to be descriptive of the endpoint distribution in s...
Conference Paper
Full-text available
A virtual keyboard takes a large portion of precious screen real estate. We have investigated whether an invisible keyboard is a feasible design option, how to support it, and how well it performs. Our study showed users could correctly recall relative key positions even when keys were invisible, although with greater absolute errors and overlaps b...
Conference Paper
Full-text available
Essential to mobile communication, the touchscreen keyboard is the most ubiquitous intelligent user interface on modern mobile phones. Developing smarter, more efficient, easy to learn, and fun to use keyboards has presented many fascinating IUI research and design questions. Some have been addressed by academic research and practitioners in indust...
Conference Paper
Full-text available
Accurately predicting the accuracy of finger-touch target acquisition is crucial for designing touchscreen UI and for modeling complex and higher level touch interaction behaviors. Despite its importance, there has been little theoretical work on creating such models. Building on the Dual Gaussian Distribution Model[3], we derived an accuracy model...
Article
Full-text available
Word–Gesture keyboards allow users to enter text using continuous input strokes (also known as gesture typing or shape writing). We developed a production model of gesture typing input based on a human motor control theory of optimal control (specifically, modelling human drawing movements as a minimisation of jerk—the third derivative of position)...
Conference Paper
Full-text available
We present WatchWriter, a finger operated keyboard that supports both touch and gesture typing with statistical decoding on a smartwatch. Just like on modern smartphones, users type one letter per tap or one word per gesture stroke on WatchWriter but in a much smaller spatial scale. WatchWriter demonstrates that human motor control adaptability, co...
Conference Paper
Full-text available
Despite of a significant body of research in optimizing the virtual keyboard layout, none of them has gained large adoption, primarily due to the steep learning curve. To address this learning problem, we introduced three types of Qwerty constraints, Qwerty1, QwertyH1, and One-Swap bounds in layout optimization, and investigated their effects on la...
Conference Paper
Full-text available
Mobile keyboards often present error corrections and word completions (suggestions) as candidates for anticipated user input. However, these suggestions are not cognitively free: they require users to attend, evaluate, and act upon them. To understand this trade-off between suggestion savings and interaction costs, we conducted a text transcription...
Article
Full-text available
This study investigated differences and similarities between finger and pen gestures on stationary devices (sitting posture) and mobile devices (sitting and walking postures). The recorded gestures were analyzed according to multiple gesture features. We found (1) pen and index finger gestures were different in features like size ratio but similar...
Patent
Full-text available
In one example, a method comprises outputting, by a computing device and for display at an output device, a graphical keyboard comprising a plurality of keys, and receiving, by the computing device, an indication of a multi-touch gesture detected at a presence-sensitive display, the multi-touch gesture comprising a first sub-gesture that traverses...
Conference Paper
Full-text available
There has been a longstanding concern within HCI that even though we are accumulating great innovations in the field, we rarely see these innovations develop into products. Our panel brings together HCI researchers from academia and industry who have been directly involved in technology transfer of one or more HCI innovations. They will share their...
Conference Paper
Full-text available
We propose a workshop on the rapidly emerging topic of optimization and computational design in human-computer interaction (HCI). The workshop will tackle the following perspectives: defining and eliciting optimality criteria, optimizing at scale, optimization and user models, optimization for safety, optimization and design practice, optimization...
Conference Paper
Full-text available
Despite its growing popularity, gesture typing suffers from a major problem not present in touch typing: gesture ambiguity on the Qwerty keyboard. By applying rigorous mathematical optimization methods, this paper systematically investigates the optimization space related to the accuracy, speed, and Qwerty similarity of a gesture typing keyboard. O...
Conference Paper
Full-text available
Modern smartphones correct typing errors and learn user-specific words (such as proper names). Both techniques are useful, yet little has been published about their technical specifics and concrete benefits. One reason is that typing accuracy is difficult to measure empirically on a large scale. We describe a closed-loop, smart touch keyboard (STK)...
Conference Paper
Full-text available
We study the performance and user experience of two popular mainstream mobile text entry methods: the Smart Touch Keyboard (STK) and the Smart Gesture Keyboard (SGK). Our first study is a lab-based ten-session text entry experiment. In our second study we use a new text entry evaluation methodology based on the experience sampling method (ESM). In...
Patent
In one example, a method includes outputting, by a computing device and for display, a graphical user interface comprising a first graphical keyboard comprising a first plurality of keys. The method further includes determining, based at least in part on an input context, to output a second graphical keyboard comprising a second plurality of keys,...
Patent
Full-text available
In one example, a method includes detecting, by a computing device, at least one user contact with a presence-sensitive screen of the computing device to input one or more characters of an input string. The method also includes detecting, by the computing device, a subsequent user contact with the presence-sensitive screen. The method also includes...
Patent
Full-text available
A graphical keyboard including a number of keys is output for display at a display device. The computing device receives an indication of a gesture to select at least two of the keys based at least in part on detecting an input unit at locations of a presence-sensitive input device. In response to the detecting and while the input unit is detected...
Article
Full-text available
Correcting erroneous input (i.e., correction) and completing a word based on partial input (i.e., completion) are two important "smart" capabilities of a modern intelligent touchscreen keyboard. However little is known whether these two capabilities are conflicting or compatible with each other in the keyboard parameter tuning. Applying computation...
Patent
Full-text available
A computing device can be configured to receive an indication of a first input gesture, a first portion of the first input gesture indicating a first character key of a plurality of character keys of a graphical keyboard and a second portion of the first input gesture indicating a second character key of the plurality of character keys. The computi...
Patent
Full-text available
A computing device is operatively coupled to a presence-sensitive surface associated with a virtual keyboard. The computing device receives an indication of a gesture input on a portion of the presence-sensitive surface associated with the virtual keyboard. In response, the computing device defines a series of input points that represent a path of...
Patent
Full-text available
Embodiments relate to systems for, and methods of, detecting attempted space key activations on a touchscreen. Such systems and methods allow for error-tolerant data input on a touchscreen. The systems and methods may be adaptive and grow progressively more accurate as additional user data is received.
Patent
Full-text available
A graphical user interface (GUI) displayed at a user interface device includes a character string that includes a substring. The computing device detects a movement of an input object from a first location on the user interface device to a second location on the user interface device. The first location corresponds to a second handle element and th...
Article
Full-text available
Older adults are a notable group among the exponentially growing population of online health information consumers. In order to better support older adults’ health-related information seeking on the Internet, it is important to understand how they ...
Patent
Full-text available
In response to determining that a first series of user inputs corresponds to a first character string, the computing device outputs, for display at a display device, the first character string. In response to determining that the first character string does not match a word in a lexicon and in response to determining that the first character string...
Patent
Full-text available
In one example, a method includes outputting, by a computing device and for display, a text display region comprising one or more characters. The method may further include outputting, in response to receiving an indication of an input unit detected at a first location of a presence-sensitive input device, a graphical indicator comprising a represe...
Conference Paper
Full-text available
To improve the accuracy of target selection for finger touch, we conceptualize finger touch input as an uncertain process, and derive a statistical target selection criterion, Bayesian Touch Criterion, by combining the basic Bayes' rule of probability with the generalized dual Gaussian distribution hypothesis of finger touch. The Bayesian Touch Cri...
Patent
Full-text available
A method, and program product for optimizing soft keyboards with multiple languages is disclosed. The text of multiple languages may be analyzed for optimization. Constraints may be chosen for key positions on the keyboard. A mean time of inputting a character in each of the multiple languages may be calculated, which may result in mean times for i...
Patent
Full-text available
Disclosed is a method and corresponding apparatus or system for disambiguating touch-input based on variation in a characteristic such as speed or pressure along a touch-trail. A computing system may detect a variation in a characteristic of a touch-trail, such as variation in speed of movement of touch along the touch-trail and/or a variation in p...
Patent
A computerized system sends a series of touchscreen keyboard touch data to a touchscreen keyboard device that receives the touchscreen keyboard touch data and processes the received string of touchscreen keyboard touch data to simulate touches to a touchscreen of the touchscreen keyboard device. A touchscreen keyboard algorithm is applied to the si...
Conference Paper
Full-text available
The time and labor demanded by a typical laboratory-based keyboard evaluation are limiting resources for algorithmic adjustment and optimization. We propose Remulation, a complementary method for evaluating touchscreen keyboard correction and recognition algorithms. It replicates prior user study data through real-time, on-device simulation. We hav...
Conference Paper
Full-text available
With the ongoing growth of digital media, academic presses and journals have had to answer some hard questions about the role of publishing in a world of blogs, social media, on-demand video and social networking. In this panel we bring together some of the top editors and publishers in HCI to explore and address these questions in a public forum.
Conference Paper
Full-text available
Fitts' law has proven to be a strong predictor of pointing performance under a wide range of conditions. However, it has been insufficient in modeling small-target acquisition with finger-touch based input on screens. We propose a dual-distribution hypothesis to interpret the distribution of the endpoints in finger touch input. We hypothesize the m...
Conference Paper
Full-text available
We propose a new approach for improving text entry accuracy on touchscreen keyboards by adapting the underlying spatial model to factors such as input hand postures, individuals, and target key positions. To combine these factors together, we introduce a hierarchical spatial backoff model (SBM) that consists of submodels with different levels of co...
Patent
Full-text available
A touch-sensitive display of a computing device may concurrently output a first virtual keyboard at a first region and a second virtual keyboard at a second region. The first region may be different from the second region, and the first and second virtual keyboards may display a substantially identical set of virtual keys. The touch-sensitive displ...
Article
Entering non-alphabetic text for languages such as Japanese and Chinese into a computer typically consists of typing Roman character-based phonemes and selecting the intended Japanese or Chinese character from a list of homophonic candidates. This paper presents a study of four candidate display styles. Three of them, Vertical, Horizontal, and Comp...
Conference Paper
Full-text available
Gesture keyboards represent an increasingly popular way to input text on mobile devices today. However, current gesture keyboards are exclusively unimanual. To take advantage of the capability of modern multi-touch screens, we created a novel bimanual gesture text entry system, extending the gesture keyboard paradigm from one finger to multiple fin...
Article
Full-text available
A keyboard design, once adopted, tends to have a long lasting and world wide impact on daily user experience. There is a substantial body of research on touch screen stylus keyboard optimization. Most of it has focused on English only. Applying rigorous mathematical optimization methods and addressing diacritic character design issues, this paper e...
Conference Paper
Full-text available
Text entry on smartphones is far slower and more error-prone than on traditional desktop keyboards, despite sophisticated detection and auto-correct algorithms. To strengthen the empirical and modeling foundation of smartphone text input improvements, we explore touch behavior on soft QWERTY keyboards when used with two thumbs, an index finger, and...
Article
Full-text available
The basic concepts and initial prototype of a word-gesture keyboard are discussed. In early 1980's, Montgomery conceived the idea of using sliding gestures on a touch keyboard to enter characters. He designed a wipe activated keyboard with a flat touch sensitive surface. The positions of the letter keys were carefully arranged to make consecutive l...
Article
Full-text available
Steering law is a fundamental model for steering tasks. Many researchers have investigated it according to different input devices, difficulty of task, subjective bias and scale effect etc. However, there is little study about the effect of surface environments especially on the texture of the interaction surface. In this paper, we experimentally i...
Conference Paper
Full-text available
This paper reports an empirical investigation in which participants produced a set of stroke gestures with varying degrees of complexity and in different target sizes using both the finger and the pen. The recorded gestures were then analyzed according to multiple measures characterizing many aspects of stroke gestures. Our findings were as follows...
Conference Paper
Full-text available
Browsing a collection of information on a mobile device is a common task, yet it can be difficult due to the small size of mobile displays. A common trade-off offered by many current mobile interfaces is to allow users to switch between an overview and detailed views of particular items. An open question is how much preview of each item to include...
Conference Paper
Full-text available
The rapid increase in smart phone capabilities has introduced new opportunities for mobile information access and computing. However, smart phone use may still be constrained by both device affordances and work environments. To understand how current business users employ smart phones and to identify opportunities for improving business smart phone...
Article
Full-text available
Facing the ever growing complexity and usability problems of the PC, some propose specialized computers as a solution, while others argue that such "information appliances" are unnecessary. Rather than pitting information appliances and PCs against each other, we argue for instead exploring the design space for using them together. We experimented...
Article
Full-text available
We investigate the use of auditory feedback in pen-gesture interfaces in a series of informal and formal experiments. Initial iterative exploration showed that gaining performance advantage with auditory feedback was possible using absolute cues and state feedback after the gesture was produced and recognized. However, gaining learning or performan...
Article
Full-text available
Traditional graphical user interfaces have been designed with the desktop mouse in mind, a device well characterized by Fitts' law. Yet in recent years, hand-held devices and tablet personal computers using a pen (or fingers) as the primary mean of interaction have become more and more popular. These new interaction modalities have pushed the tradi...
Article
This study presents a series of three experiments that evaluate human capabilities and limitations in using pen-tip pressure as an additional channel of control information in carrying out trajectory tasks such as drawing, writing and gesturing on computer screen. The first experiment measured the natural range of force used in regular drawing and...
Article
Full-text available
We investigate the use of auditory feedback in pen-gesture interfaces in a series of informal and formal experiments. Initial iterative exploration showed that gaining performance advantage with auditory feedback was possible using absolute cues and state feedback after the gesture was produced and recognized. However, gaining learning or performan...
Article
Full-text available
Traditional graphical user interfaces have been designed with the desktop mouse in mind, a device well characterized by Fitts' law. Yet in recent years, hand-held devices and tablet personal computers using a pen (or fingers) as the primary mean of interaction have become more and more popular. These new interaction modalities have pushed the tradi...
Conference Paper
Full-text available
It has been well understood that optimized soft keyboard layouts improve motor movement efficiency over the standard Qwerty layouts, but have the drawback of long initial visual search time for novice users. To ease the initial searching time on optimized soft keyboards, we explored "Quasi-Qwerty optimization" so that the resulting layouts are clos...
Conference Paper
Full-text available
Dictionary-based disambiguation (DBD) is a very popular solution for text entry on mobile phone keypads but suffers from two problems: 1. the resolution of encoding collision (two or more words sharing the same numeric key sequence) and 2. entering out-of-vocabulary (OOV) words. In this paper, we present SHRIMP, a system and method that addresses t...
Conference Paper
Full-text available
The introduction of a new generation of attractive touch screen-based devices raises many basic usability questions whose answers may influence future design and market direction. With a set of current mobile devices, we conducted three experiments focusing on one of the most basic interaction actions on touch screens: the operation of soft buttons...
Conference Paper
Full-text available
This paper investigates using stroke gestures as shortcuts to menu selection. We first experimentally measured the per- formance and ease of learning of stroke shortcuts in com- parison to keyboard shortcuts when there is no mnemonic link between the shortcut and the command. While both types of shortcuts had the same level of performance with enou...
Conference Paper
Full-text available
We present our experience in bringing ShapeWrite r, a novel HCI research product, from the laboratory to real world users through iPhone's App Store.
Chapter
Eye-gaze plays an important role in face-to-face communication. This chapter presents research on exploiting the rich information contained in human eye-gaze for two types of applications. The first is to enhance computer mediated human-human communication by overlaying eye-gaze movement onto the shared visual spatial discussion material such as a...
Conference Paper
Full-text available
Ease and efficiency are two critical qualities of human-computer interfaces. In this address, I will first examine various contributing factors to these two qualities, including recognition and recall, open and closed loop control, controlled and automatic process [Schneider and Shiffrin, 1977; Shiffrin and Schneider, 1977; Schneider and Chein, 200...
Conference Paper
Full-text available
ABSTRACT Shape writing is an ,input technology ,for touch-screen mobile phones and pen-tablets. To shape write text, the user spells out word patterns by sliding a finger or stylus over a graphical keyboard. The user’s trace is then recognized by a pattern recognizer. In this paper we analyze ,and evaluate various keyboard layouts, including alphab...