Gilles Bailly

• Researcher at Sorbonne University

Haptic interactions with objects induce complex vibrotactile signals that are central to tactile perception. Despite the broad literature on vibrotactile perception, surprisingly little is known about the sensory processing of complex tactile signals made of multiple pure tones. To fill this gap, the study reported here investigated the impact of t...

Capturing content of fabrication activities is the first step for producing knowledge resources and an integral part of maker culture. As a secondary task, it conflicts though with the fabrication activity, and thus it is often forgotten and knowledge resources end up incomplete. In this article, we investigate different dimensions of content captu...

Hand redirection is a visuo-haptic illusion offering rich haptic feedback in Virtual Reality while using a limited number of physical objects. This technique relies on a redirection function which determines the virtual hand position depending on the physical hand position. In this paper, we extend the design space of the Hand-redirection technique...

Despite the benefits of expert interaction techniques, many users do not learn them and continue to use novice ones. This article aims at better understanding if, when and how users decide to learn and ultimately adopt expert interaction techniques. This dynamic learning process is a complex skill-acquisition and decision-making problem. We first p...

Encountered-type of Haptic devices (ETHD) are robotic interfaces physically overlaying virtual counterparts prior to a user interaction in Virtual Reality. They theoretically reliably provide haptics in Virtual environments, yet they raise several intrinsic design challenges to properly display rich haptic feedback and interactions in VR applicatio...

In Virtual Reality, visuo-haptic illusions such as hand redirection introduce a discrepancy between the user’s hand and its virtual avatar. This visual shift can be used, for instance, to provide multiple virtual haptic objects through a single physical proxy object. This low-cost approach improves the sense of presence, however, it is unclear how...

Adapting an interface requires taking into account both the positive and negative effects that changes may have on the user. A carelessly picked adaptation may impose high costs to the user -- for example, due to surprise or relearning effort -- or "trap" the process to a suboptimal design immaturely. However, effects on users are hard to predict a...

Haptic feedback has become crucial to enhance the user experiences in Virtual Reality (VR). This justifies the sudden burst of novel haptic solutions proposed these past years in the HCI community. This article is a survey of Virtual Reality interactions, relying on haptic devices. We propose two dimensions to describe and compare the current hapti...

We present CoVR, a novel robotic interface providing strong kinesthetic feedback (100 N) in a room-scale VR arena. It consists of a physical column mounted on a 2D Cartesian ceiling robot (XY displacements) with the capacity of (1) resisting to body-scaled users’ actions such as pushing or leaning; (2) acting on the users by pulling or transporting...

We present CoVR, a novel robotic interface providing strong kinesthetic feedback (100 N) in a room-scale VR arena. It consists of a physical column mounted on a 2D Cartesian ceiling robot (XY displacements) with the capacity of (1) resisting to body-scaled users' actions such as pushing or leaning; (2) acting on the users by pulling or transporting...

Humans have the ability to convey an array of emotions through complex and rich touch gestures. However, it is not clear how these touch gestures can be reproduced through interactive systems and devices in a remote mediated communication context. In this paper, we explore the design space of device-initiated touch for conveying emotions with an in...

We investigated retroactive transfer when users alternate between different interfaces. Retroactive transfer is the influence of a newly learned interface on users' performance with a previously learned interface. In an interview study, participants described their experiences when alternating between different interfaces, e.g. different operating...

We propose a paradigm called Skin-On interfaces, in which interactive devices have their own (artificial) skin, thus enabling new forms of input gestures for end-users (e.g. twist, scratch). Our work explores the design space of Skin-On interfaces by following a bio-driven approach: (1) From a sensory point of view, we study how to reproduce the lo...

Robotic-assisted Minimally Invasive Surgery (MIS) is adopted more and more as it overcomes the shortcomings of classic MIS for surgeons while keeping the benefits of small incisions for patients. However, introducing new technology oftentimes affects the work of skilled practitioners. Our goals are to investigate the impacts of telemanipulated surg...

In this demo, we showcase SAM [3], a modular and extensible JavaScript framework for s elf- a dapting m enus on webpages. SAM allows control of two elementary aspects of adaptating web menus: (1) the target policy, which assigns scores to menu items for adaptation, and (2) the adaptation style, which specifies how they are adapted on display. We hi...

This paper presents SAM, a modular and extensible JavaScript framework for self-adapting menus on webpages. SAM allows control of two elementary aspects for adapting web menus: (1) the target policy, which assigns scores to menu items for adaptation, and (2) the adaptation style, which specifies how they are adapted on display. By decoupling them,...

The usability of small devices such as smartphones or interactive watches is often hampered by the limited size of command vocabularies. This paper is an attempt at better understanding how finger identification may help users invoke commands on touch screens, even without recourse to multi-touch input. We describe how finger identification can inc...

This paper aims at better understanding the transition from menus to shortcuts. We first discuss the limitations and the opportunities of the theoretical and empirical characterizations of this transition. We then consider keyboard shortcuts as a case study and manually annotate empirical data to estimate several behavioral markers such as the init...

In this paper, we explore the interaction space of MobiLimb, a small 5-DOF serial robotic manipulator attached to a mobile device. It (1) overcomes some limitations of mobile devices (static, passive, motionless); (2) preserves their form factor and I/O capabilities; (3) can be easily attached to or removed from the device; (4) offers additional I/...

Marginalised children are uniquely vulnerable within western societies. Conducting participatory design research with them comes with particular ethical challenges, some of which we illustrate in this paper. Through several examples across two different participatory design projects (one with autistic children, another with visually impaired childr...

Human decisions are prone to biases, and this is no less true for decisions made within data visualizations. Bias mitigation strategies often focus on the person, by educating people about their biases, typically with little success. We focus instead on the system, presenting the first evidence that altering the design of an interactive visualizati...

This paper reports an empirical study of a multisensory map used by visually impaired primary school pupils, to study human habitats and differences between urban, suburban and rural areas using a local example. Using multimodal analysis, we propose to examine how the use of smell and taste shape pupils’ engagement and the development of a non-visu...

This paper argues for designing geo-technologies supporting non-visual sensory knowledge. Sensory knowledge refers to the implicit and explicit knowledge guiding our uses of our senses to understand the world. To support our argument, we build on an 18 months field-study on geography classes for primary school children with visual impairments. Our...

Predicting human performance in interaction tasks allows designers or developers to understand the expected performance of a target interface without actually testing it with real users. In this work, we present a deep neural net to model and predict human performance in performing a sequence of UI tasks. In particular, we focus on a dominant class...

Predicting human performance in interaction tasks allows designers or developers to understand the expected performance of a target interface without actually testing it with real users. In this work, we present a deep neural net to model and predict human performance in performing a sequence of UI tasks. In particular, we focus on a dominant class...

Touch is one of the primary channel of non-verbal communication. It is used to convey emotions and to establish bounds between people. Its use has already been considered in HCI to interact with devices, but it has rarely been used for direct emotional communication between individuals. This article presents a literature review of social touch in h...

Watches benefit from a long design history. Designers and engineers have successfully built devices using rotary physical inputs such as crowns, bezels, and wheels, separately or combined. Smart watch designers have explored the use of some of these inputs for interactions. However, a systematic exploration of their combinations has yet to be done....

We introduce and demonstrate VersaPen, a modular pen for expanding input capabilities. Users can create their own digital pens by stacking different input/output modules that define both the look and feel of the customized device. VersaPen investigate the benefits of adaptable devices and enriches interaction by providing multimodal capabilities, a...

While software often allows user customization, most physical devices remain mainly static. We introduce VersaPen, an adaptable, multimodal, hot-pluggable pen for expanding input capabilities. Users can create their own pens by stacking different input/output modules that define both the look and feel of the customized device. VersaPen offers multi...

While it is well known that menu usage follows a Zipfian distribution, there has been little interest in the impact of menu item frequency distribution on user's behavior. In this note, we explore the effects of frequency distribution on average menu performance as well as individual item performance. We compare three frequency distributions of men...

TouchTokens are passive tokens that can be recognized on any capacitive surface based on the spatial configuration of the fingers that hold them. However, interaction with these tokens is confined to the basic two-state model of touch interaction as the system only knows the tokens' position and cannot detect tokens that are not touched. We increas...

We propose a novel perspective on the design of toolbar buttons that aims to increase keyboard shortcut accessibility. IconHK implements this perspective by blending visual cues that convey keyboard shortcut information into toolbar buttons without denaturing the pictorial representation of their command. We introduce three design strategies to emb...

Menus are used for exploring and selecting commands in interactive applications. They are widespread in current systems and used by a large variety of users. As a consequence, they have motivated many studies in Human-Computer Interaction (HCI). Facing the large variety of menus, it is difficult to have a clear understanding of the design possibili...

In this paper, we aim at better understanding users interaction with trackpads and provide recommendations for creating novel or revisiting existing interactions. To achieve this goal, we conducted a three-steps field study targeted at Apple Macbook's multi-touch trackpads. We collected (1) system preferences of 80 users; (2) low-level interaction...

Various guidance techniques have been proposed to help users to quickly and effectively locate objects in large and dense environments such as supermarkets, libraries, or control rooms. Little research, however, has focused on their impact on learning. These techniques generally transfer control from the user to the system, making the user more pas...

In this paper, we investigate how head movements can serve to change the viewpoint in 3D applications, especially when the viewpoint needs to be changed quickly and temporarily to disambiguate the view. We study how to use yaw and roll head movements to perform orbital camera control, i.e., to rotate the camera around a specific point in the scene....

We report on the design process leading to the creation of MapSense, a multi-sensory interactive map for visually impaired children. We conducted a formative study in a specialized institute to understand children’s educational needs, their context of care and their preferences regarding interactive technologies. The findings (1) outline the needs...

We investigate the potential benefits of actuated devices for the desktop workstation which remains the most used environment for daily office works. A formative study reveals that the desktop workstation is not a fixed environment because users manually change the position and the orientation of their devices. Based on these findings, we present t...

TouchTokens make it possible to easily build interfaces that combine tangible and gestural input using passive tokens and a regular multi-touch surface. The tokens constrain users' grasp, and thus, the relative spatial configuration of fingers on the surface, theoretically making it possible to design algorithms that can recognize the resulting tou...

This paper explores the quality of engagement between human and agent in large displays. We designed a musical application and introduced the illusion of touch as a novel interaction concept where the user and the agent interact through mediated virtual touch. We conducted a lab study to better understand the users' behaviors and engagement with ou...

We propose InspectorWidget, an opensource application to track and analyze users' behaviors in interactive software. The key contributions of our application are: 1) it works with closed applications that do not provide source code nor scripting capabilities; 2) it covers the whole pipeline of software analysis from logging input events to visual s...

In this paper, we report the results of a study in the field, in a specialized institute, aiming to identify the specific needs of children living with visual impairment. The principal results show (1) that there are numerous uses of digital technologies by visually impaired children; (2) how reflective and ludic artefacts limit their frustrations...

Dans cet article nous explorons l'espace de conception des techniques de raccourcis gestuels sur pavé tactile à travers quatre nouvelles techniques d'interaction : SpotPad et LociPad illustrent un vocabulaire gestuel qui repose sur des gestes statiques en un point, mais diffèrent par leur représentation graphique (grille vs. loci). ChordPad illustr...

La conception de modèles en trois dimensions nécessite un changement fréquent du point de vue lors de la manipulation d'objets 3D. Afin de réduire le coût de l'alternance entre tâches primaires (création d'objets) et tâches secondaires (contrôle du point de vue), nous proposons l'utilisation des mouvements de la tête et des épaules pour le contrôle...

System schematics, such as those used for electrical or hydraulic systems, can be large and complex. Fisheye techniques can help navigate such large documents by maintaining the context around a focus region, but the distortion introduced by traditional fisheye techniques can impair the readability of the diagram. We present SchemeLens, a vector-ba...

Locating an object in an unfamiliar and dense physical environment, such as a control room, supermarket, or warehouse, can be challenging. In this paper, we present the Projection-Augmented Arm (PAA), a motorized robotic arm augmented with a pico-projector to help users to localize targets in such environments. The arm moves and displays a projecte...

The usability of small devices such as smartphones or interactive watches is often hampered by the limited size of command vocabularies. This paper is an attempt at better understanding how finger identification may help users invoke commands on touch screens, even without recourse to multi-touch input. We describe how finger identification can inc...

We propose iSkin, a novel class of skin-worn sensors for touch input on the body. iSkin is a very thin sensor overlay, made of biocompatible materials, and is flexible and stretchable. It can be produced in different shapes and sizes to suit various locations of the body such as the finger, forearm, or ear. Integrating capacitive and resistive touc...

In this paper, we propose SuperVision, a new interaction technique for distant control of objects in a smart home. This technique aims at enabling users to point towards an object, visualize its current state and select a desired functionality as well. To achieve this: 1) we present a new remote control that contains a pico-projector and a slider;...

Designing and editing 3D curves is often involved in a wide array of applications such as CAD, multimedia content edition or landscape and road generation. This diversity resulted in a spread of 3D-related works across different communities such as SIGCHI or SIGGRAPH. In this article, we introduce a design space to gather existing techniques in the...

Finding an object in a physical environment is difficult if the environment contains many objects, especially if it is large and dense. We propose a design space that describes and compares existing guidance techniques according to four dimensions: output modality, physicality, granularity and spatial information. Output modality can be visual, aud...

Modifier keys of the keyboard such as Ctrl or Cmd are used to delimit text entry and command selection (keyboard shortcuts). \\In this paper we study the impact of the position of these modifier keys on performance and muscular load by duplicating them on the mouse. We derive two interaction techniques: CtrlMouse duplicates the Ctrl and Shift keys...

Most of today's public displays only show predefined con-tents that passers-by are not able to change. We argue that interactive public displays would benefit from immedi-ately usable mid-air techniques for choosing options, express-ing opinions or more generally selecting one among several items. We propose a design space for hand-gesture based mi...

Most of today's public displays only show predefined contents that passers-by are not able to change. We argue that interactive public displays would benefit from immediately usable mid-air techniques for choosing options, expressing opinions or more generally selecting one among several items. We propose a design space for hand-gesture based mid-a...

In this paper we present a series of three field studies on the integration of multiple modalities (touch and mid-air gestures) in a public display. We analyze our field studies using Conversion Diagrams, an approach to model and evaluate usage of multimodal public displays. Conversion diagrams highlight the transitions inherent in a multimodal sys...

Menu design can be formulated as an optimization problem and a growing number of menu design problems can be solved algorithmically. Challenges to research on menu optimization include the design space, multiplicity of objectives, users, and constraints, predictive models, and optimization. Designing great menus is very challenging solely from a co...

This paper presents a novel mathematical model for visual search and selection time in linear menus. Assuming two visual search strategies, serial and directed, and a pointing sub-task, it captures the change of performance with five fac- tors: 1) menu length, 2) menu organization, 3) target position, 4) absence/presence of target, and 5) practice....

This paper presents a novel mathematical model for visual search and selection time in linear menus. Assuming two visual search strategies, serial and directed, and a pointing sub-task, it captures the change of performance with five factors: 1) menu length, 2) menu organization, 3) target position, 4) absence/presence of target, and 5) practice. T...

Les menus sont utilisés dans la plupart de nos applications pour présenter, organiser et sélectionner des commandes. Un grand nombre de techniques d'interaction a été proposé et de nombreuses évaluations expérimentales ont été conduites. Pourtant, peu d'études ont visé à analyser les menus existants. Dans cet article, nous proposons et mettons à di...

Menu systems are challenging to design because design spaces are immense, and several human factors affect user behavior. This paper contributes to the design of menus with the goal of interactively assisting designers with an optimizer in the loop. To reach this goal, 1) we extend a predictive model of user performance to account for expectations...

We investigate how to reveal an initial mid-air gesture on interactive public displays. This initial gesture can serve as gesture registration for advanced operations. We propose three strategies to reveal the initial gesture: spatial division, temporal division and integration. Spatial division permanently shows the gesture on a dedicated screen a...

We propose and validate a model of the perception area of content on public displays in order to predict from where users can read. From this model, we derive Screenfinity, a technique to rotate, translate, and zoom content in order to enable reading while passing by very large displays. Screenfinity is comfortable to read when close, supports diff...

Keyboard shortcuts allow fast interaction, but they are known to be infrequently used, with most users relying heavily on traditional pointer-based selection for most commands. We describe the goals, design, and evaluation of ExposeHK, a new interface mechanism that aims to increase hotkey use. ExposeHK's four key design goals are: 1) enable users...

Selecting commands on multi-touch displays is still a challenging problem. While a number of gestural vocabularies have been proposed, these are generally restricted to one or two fingers or can be difficult to learn. We introduce Finger-Count gestures, a coherent set of multi-finger and two-handed gestures. Finger-Count gestures are simple, robust...

Rotating 3D objects is a difficult task on mobile devices, be-cause the task requires 3 degrees of freedom and (multi-)touch input only allows for an indirect mapping. We propose a novel style of mobile interaction based on mid-air gestures in proximity of the device to increase the number of DOFs and alleviate the limitations of touch interaction...

In this paper we present our findings from a lab and a field study investigating how passers-by notice the interactivity of public displays. We designed an interactive installation that uses visual feedback to the incidental movements of passers-by to communicate its interactivity. The lab study reveals: (1) Mirrored user silhouettes and images are...

Most interactive public displays currently rely on flat screens. This form factor impacts how users (1) notice the public display (2) develop motivation and (3) (socially) interact with the public display. In this paper, we present Chained Displays, a combination of several screens to create different form factors for interactive public displays. W...

When the user is engaged with a real-world task it can be inappropriate or difficult to use a smartphone. To address this concern, we developed ShoeSense, a wearable system consisting in part of a shoe-mounted depth sensor pointing upward at the wearer. ShoeSense recognizes relaxed and discreet as well as large and demonstrative hand gestures. In p...

In this paper we present our findings from a lab and a field study investigating how passers-by notice the interactivity of public displays. We designed an interactive installation that uses visual feedback to the incidental movements of passers-by to communicate its interactivity. The lab study reveals: (1) Mirrored user silhouettes and images are...

Interaction with TV sets, set-top boxes or media centers strongly differs from interaction with personal computers: not only does a typical remote control suffer strong form factor limitations but the user may well be slouching in a sofa. In the face of more and more data, features, and services made available on interactive televisions, we propose...

Initially designed in the nineteen seventies as a mere zapping tool, obviously the traditional device known as the TV remote control no longer responds to the multifarious needs of today's interactive television. Designing new remote-control devices is a challenge that the HCI community has started to face. The paper indicates the various direction...

The miniaturization of electronic components paves the way for new interaction techniques for wearable computing. We briefly review interactive digital jewelry, an emerging subfield. We report the data of a limited poll about the way people perceive the prospect of digital jewelry. We then consider the constraints and the promise of digital jewelry...

Distant displays such as interactive public displays (IPD) or interactive television (ITV) require new interaction techniques as traditional input devices may be limited or missing in these contexts. Free hand interaction, as sensed with computer vision techniques, presents a promising interaction technique. This paper presents the adaptation of th...

This paper presents the design and evaluation of the Wavelet menu and its implementation on the iPhone. The Wavelet menu consists of a concentric hierarchical Marking menu using simple gestures. The novice mode, i.e. when the menu is displayed, is well adapted to the limited screen space of handheld devices because the representation of the menu hi...

We propose Radial-Stroke and Finger-Count Shortcuts, two techniques aimed at augmenting the menubar on multi-touch surfaces. We designed these multi-finger two-handed interaction techniques in an attempt to overcome the limitations of direct pointing on interactive surfaces, while maintaining compatibility with traditional interaction techniques. W...

This paper presents the design and evaluation of the Wavelet menu and its implementation on the iPhone. The Wavelet menu consists of a concentric hierarchical Marking menu using simple gestures. The novice mode, i.e. when the menu is displayed, is well adapted to the limited screen space of handheld devices because the representation of the menu hi...

Exploration and navigation in multimedia data hierarchies (e.g., photos, music) are frequent tasks on mobile devices. However, visualization and interaction are impoverished due to the limited size of the screen and the lack of precise input devices. As a result, menus on mobile devices do not provide efficient navigation as compared to many innova...