No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Remote Control by Body Movement in Synchrony with Orbiting Widgets: an Evaluation of TraceMatch
Abstract
In this work we consider how users can use body movement for remote control with minimal effort and maximum flexibility. TraceMatch is a novel technique where the interface displays available controls as circular widgets with orbiting targets, and where users can trigger a control by mimicking the displayed motion. The technique uses computer vision to detect circular motion as a uniform type of input, but is highly appropriable as users can produce matching motion with any part of their body. We present three studies that investigate input performance with different parts of the body, user preferences, and spontaneous choice of movements for input in realistic application scenarios. The results show that users can provide effective input with their head, hands and while holding objects, that multiple controls can be effectively distinguished by the difference in presented phase and direction of movement, and that users choose and switch modes of input seamlessly.
... Motion correlation allows for the control of interactive objects by using body motion to mimic an observed movement [1]. One commonly used technique is the use of orbital controls [2][3][4][5][6], where a user must imitate the motion of an orbital target to activate it. These orbital targets can be (1) displayed visually through screens when these are available (e.g., in smart TV sets) [2,5], (2) projected onto objects [6], or (3) embodied in artifacts specifically designed to make orbital movements [4,7], although the last two solutions are impractical and expensive. ...
... One commonly used technique is the use of orbital controls [2][3][4][5][6], where a user must imitate the motion of an orbital target to activate it. These orbital targets can be (1) displayed visually through screens when these are available (e.g., in smart TV sets) [2,5], (2) projected onto objects [6], or (3) embodied in artifacts specifically designed to make orbital movements [4,7], although the last two solutions are impractical and expensive. ...
... To investigate this aspect, we assessed synchronization abilities at different orbit speeds of 2.4 and 3.6 s. Similarly to other works (e.g., [4,5]), the results indicate that users exhibit better performance at a lower speed (3.6 s). ...
SoundOrbit is a novel input technique that uses motion correlation to control smart devices. The technique associates controls with specific orbital sounds, made of cyclically increasing/decreasing musical scales, and the user can activate a control by mimicking the corresponding sound by body motion. Unlike previous movement-correlation techniques based on visual displays, SoundOrbit operates independent of visual perception, enabling the development of cost-effective smart devices that do not require visual displays. We investigated SoundOrbit by conducting two user studies. The first study evaluated the effectiveness of binaural sound spatialization to create a distinct orbiting sound. In comparison to a cyclic musical scale that is fixed in the apparent auditory space, we found that spatial effects did not improve users’ ability to follow the sound orbit. In the second study, we aimed at determining the optimal system parameters, and discovered that users synchronize better with slower speeds. The technique was found to be feasible and reliable for one and two orbits simultaneously, each orbit using a distinct sound timbre, but not for three orbits due to a high error rate.
... Rhythmic-synchronization-based techniques originated somehow from movementcorrelation techniques [5][6][7][8][9][10], where controls show different motion patterns and the user can select one of them by mimicking the corresponding motion. ...
... From the point of view of the input device, rhythmic synchronization techniques can be simpler compared to those based on movement correlation. While motion-correlation techniques require sensors that detect movement (e.g., cameras [6,7] or kinect [9]), rhythmic synchronization techniques require sensors as simple as a button, e.g., [1,4]. As matter of fact, previous studies showed that these techniques can support a wide variety of sensors [1,3]; in particular, any sensor capable of generating a binary input through which users can perform the required rhythm. ...
... As a result, these techniques are quite flexible. In addition, motion correlation techniques are not suitable for continuous control because "they require the user to continuously follow the target for prolonged periods" [7]. On the contrary, it was shown that rhythmic synchronization techniques can be used for continuous control [1,4] without major drawbacks. ...
Rhythmic-synchronization-based interaction is an emerging interaction technique where multiple controls with different rhythms are displayed in visual form, and the user can select one of them by matching the corresponding rhythm. These techniques can be used to control smart objects in environments where there may be interfering auditory stimuli that contrast with the visual rhythm (e.g., to control Smart TVs while playing music), and this could compromise users’ ability to synchronize. Moreover, these techniques require certain reflex skills to properly synchronize with the displayed rhythm, and these skills may vary depending on the age and gender of the users. To determine the impact of interfering auditory stimuli, age, and gender on users’ ability to synchronize, we conducted a user study with 103 participants. Our results show that there are no significant differences between the conditions of interfering and noninterfering auditory stimuli and that synchronization ability decreases with age, with males performing better than females—at least as far as younger users are concerned. As a result, two implications emerge: first, users are capable of focusing only on visual rhythm ignoring the auditory interfering rhythm, so listening to an interfering rhythm should not be a major concern for synchronization; second, as age and gender have an impact, these systems may be designed to allow for customization of rhythm speed so that different users can choose the speed that best suits their reflex skills.
... Related work has developed and studied a variety of technical implementations of motion matching interaction, using webcams [11,12], depth-sensors [9], eye-trackers [18,25,37,48,52], magnets [39], and inertial measurement units (IMUs) embedded in smart-watches [50], phones [4], and AR headsets [19]. These implementations are supplemented with work on further algorithmic developments and novel deployments [10,16,21,27,29,47]. Taken together, these laboratory studies have shown that people are able to accurately interact with motion matching interfaces after a very short learning period. ...
... Both the Smart Classroom and Public Display demos were rated high in the SUS which was qualitatively confirmed in the discussions with the participants. The two demos for the home elicited mixed responses (F4, 6,8,10), highlighted by a low SUS. Participants reported they would not use the home demos due to their complexity (for their purpose), and because they were seen as too fatiguing or distracting. ...
... The aforementioned challenges become more apparent in the relaxed atmosphere of the home, in which participants preferred more straightforward interactions, even if this, for example, meant getting up and walking towards the light-switch (F4, 8,10). Noticeably, participants' examples of other home control systems that could benefit from motion matching were based on these being out-of-reach, and enabling a smaller number of actions. ...
Amongst the variety of (multi-modal) interaction techniques that are being developed and explored, the Motion Matching paradigm provides a novel approach to selection and control. In motion matching, users interact by rhythmically moving their bodies to track the continuous movements of different interface targets. This paper builds upon the current algorithmic and usability focused body of work by exploring the product possibilities and implications of motion matching. Through the development and qualitative study of four novel and different real-world motion matching applications --- with 20 participants --- we elaborate on the suitability of motion matching in different multi-user scenarios, the less pertinent use in home environments and the necessity for multi-modal interaction. Based on these learnings, we developed three novel motion matching based interactive lamps, which report on clear paths for further dissemination of the embodied interaction technique's experience. This paper hereby informs the design of future motion matching interfaces and products.
... SEQUENCE bypasses both Midas Touch and mapping issues by matching the user's input against a corresponding rhythm, in contrast to corresponding position. PathSync [12] and TraceMatch [14,15], instead, bypass the same issues by matching the user's input against the corresponding motion. ...
... However, the use of rhythm may be advantageous for continuous control, e.g., changing volume. In fact, as explained by the authors of TraceMatch, movement correlation techniques are not generally suited for continuous controls Bbecause they require the user to continuously follow the target for prolonged periods^ [15]. Using SEQUENCE, instead, once a control was triggered through its rhythmic pattern, it can remain active for a predetermined amount of time (e.g., 2 s) waiting for new active event (e.g., a tap on smartphone, eye blink, or beat of lips), and each new active event triggers again the control one time (similarly to buttons of a remote controls). ...
... As highlighted by other researches [15,32], smart home interactions should be designed to offer users instant control Bright here^and Bright now^with minimal action and without going out of their ways. SEQUENCE concurs in this direction providing a Blazy^input technique for activating controls at a distance and has several advantages in certain contexts. ...
We present SEQUENCE, a novel interaction technique for selecting objects from a distance. Objects display different rhythmic patterns by means of animated dots, and users can select one of them by matching the pattern through a sequence of taps on a smartphone. The technique works by exploiting the temporal coincidences between patterns displayed by objects and sequences of taps performed on a smartphone: if a sequence matches with the pattern displayed by an object, the latter is selected. We propose two different alternatives for displaying rhythmic sequences associated with objects: the first one uses fixed dots (FD), the second one rotating dots (RD). Moreover, we performed two evaluations on such alternatives. The first evaluation, carried out with five participants, was aimed to discover the most appropriate speed for displaying animated rhythmic patterns. The second evaluation, carried out on 12 participants, was aimed to discover errors (i.e., activation of unwanted objects), missed activations (within a certain time), and time of activations. Overall, the proposed design alternatives perform in similar ways (errors, 2.8% for FD and 3.7% for RD; missed, 1.3% for FD and 0.9% for RD; time of activation, 3862 ms for FD and 3789 ms for RD).
... Motion-matching is an alternative selection mechanism to pointing, relying on the ability of users to couple with motion displayed at the interface [57]. First explored by Williamson and Murray-Smith [63], motion-matching has been used with a variety of different input modalities, including the mouse [63,24], eye gaze [47,59,22], and recently touchless interaction [13,17,16]. PathSync demonstrated the discoverability, intuitiveness and multi-user capacity of motion-matching for hand-based gestures [13], while TraceMatch showed users' capacity to synchronise using different input modalities [16]. ...
... First explored by Williamson and Murray-Smith [63], motion-matching has been used with a variety of different input modalities, including the mouse [63,24], eye gaze [47,59,22], and recently touchless interaction [13,17,16]. PathSync demonstrated the discoverability, intuitiveness and multi-user capacity of motion-matching for hand-based gestures [13], while TraceMatch showed users' capacity to synchronise using different input modalities [16]. TraceMatch also introduced a webcam-based implementation of motion-matching that accepts any form of movement as input [17], an approach we adopt for the motion-matching phase in MatchPoint. ...
Pointing is a fundamental interaction technique where user movement is translated to spatial input on a display. Conventionally, this is based on a rigid configuration of a display coupled with a pointing device that determines the types of movement that can be sensed, and the specific ways users can affect pointer input. Spontaneous spatial coupling is a novel input technique that instead allows any body movement, or movement of tangible objects, to be appropriated for touchless pointing on an ad hoc basis. Pointer acquisition is facilitated by the display presenting graphical objects in motion, to which users can synchronise to define a temporary spatial coupling with the body part or tangible object they used in the process. The technique can be deployed using minimal hardware, as demonstrated by MatchPoint, a generic computer vision-based implementation of the technique that requires only a webcam. We explore the design space of spontaneous spatial coupling, demonstrate the versatility of the technique with application examples, and evaluate MatchPoint performance using a multi-directional pointing task.
... An interesting future implication is to investigate the potential of using additional modalities to control or automate the boundary of diminished reality with speech, gesture, or proximity cues [11,17,102]. Designers could consider some semi-autonomous or gestural detection to initiate the control of an object being diminished or shown that can immediately respond to their behavior, e.g., detecting movement of tangible objects to reveal/hide it to enable touchless and remote pointing like [14,15]. 7.1.2 ...
... However, as one of the most basic interaction tasks, target selection can be challenging on these new interfaces. There are three reasons: 1) cross-device interaction for a large number of devices calls for association-free target selection techniques [6,10,23]. It is not practical to have a designated controller for each individual device or require users to associate with the devices each time before usage, especially when there is a large number of devices; 2) the interaction expressivity (e.g., audio, gesture) and form factor (e.g. ...
Temporal synchronous target selection is an association-free selection technique: users select a target by generating signals (e.g., finger taps and hand claps) in sync with its unique temporal pattern. However, classical pattern set design and input recognition algorithm of such techniques did not leverage users' behavioral information, which limits their robustness to imprecise inputs. In this paper, we improve these two key components by modeling users' interaction behavior. In the first user study, we asked users to tap a finger in sync with blinking patterns with various period and delay, and modeled their finger tapping ability using Gaussian distribution. Based on the results, we generated pattern sets for up to 22 targets that minimized the possibility of confusion due to imprecise inputs. In the second user study, we validated that the optimized pattern sets could reduce error rate from 23% to 7% for the classical Correlation recognizer. We also tested a novel Bayesian, which achieved higher selection accuracy than the Correlation recognizer when the input sequence is short. The informal evaluation results show that the selection technique can be effectively scaled to different modalities and sensing techniques.
... Furthermore, researchers and practitioners could apply and evaluate the slope method beyond gaze, e.g. motion matching for body movements [4][5][6] and mid-air gestures [2]. ...
In this paper we introduce a novel approach for smooth pursuits eye movement detection and demonstrate that it allows up to 160 targets to be distinguished. With this work we advance the well-established smooth pursuits technique, which allows gaze interaction without calibration. The approach is valuable for researchers and practitioners, since it enables novel user interfaces and applications to be created that employ a large number of targets, for example, a pursuits-based keyboard or a smart home where many different objects can be controlled using gaze. We present findings from two studies. In particular, we compare our novel detection algorithm based on linear regression with the correlation method. We quantify its accuracy for around 20 targets on a single circle and up to 160 targets on multiple circles. Finally, we implemented a pursuits-based keyboard app with 108 targets as proof-of-concept.
... It was tested on circular orbiting targets only as in [Esteves et al., 2015]. In a follow-up evaluation [Clarke et al., 2017], they stated that a more reined matching process would be required for non-circular targets. It is therefore surprising that optical low has attracted little attention in eye movement analysis. ...
We investigate the smooth pursuit eye movement based interaction using an unmodified off-the-shelf RGB camera. In each pair of sequential video frames, we compute the indicative direction of the eye movement by analyzing flow vectors obtained using the Lucas-Kanade optical flow algorithm. We discuss how carefully selected low vectors could replace the traditional pupil centers detection in smooth pursuit interaction. We examine implications of unused features in the eye camera imaging frame as potential elements for detecting gaze gestures. This simple approach is easy to implement and abstains from many of the complexities of pupil based approaches. In particular, EyeFlow does not call for either a 3D pupil model or 2D pupil detection to track the pupil center location. We compare this method to state-of-the-art approaches and ind that this can enable pursuit interactions with standard cameras. Results from the evaluation with 12 users data yield an accuracy that compares to previous studies. In addition, the benefit of this work is that the approach does not necessitate highly matured computer vision algorithms and expensive IR-pass cameras.
... First, we describe the implementation and affordances of Wattom, a highly interactive ambient eco-feedback device that is tightly coupled to users' energy-related activities. And second, we study the effectiveness of motion matching in the context of a smart home, where users not always perceive the moving stimulus directly ahead of them (as in a traditional lab setting [9,10,35]), or are not able to comfortably track interface motions by pointing directly at the device (e.g., while laying on the sofa). ...
This paper presents Wattom, a highly interactive ambient eco-feedback smart plug that aims to support a more sustainable use of electricity by being tightly coupled to users' energy-related activities. We describe three use cases of the system: using Wattom to power connected appliances and understand the environmental impact of their use in real time; scheduling these power events; and presenting users with personal consumption data desegregated by device. We conclude with a user study in which the effectiveness of the plug's novel interactive capabilities is assessed (mid-air, hand-based motion matching). The study explores the effectiveness of Wattom and motion matching input in a realistic setup, where the user is not always directly ahead of the interface, and not always willing to point straight at the device (e.g., when the plug is at an uncomfortable angle). Despite not using a graphical display, our results demonstrate that our motion matching implementation was effective in line with previous work, and that participants' pointing angle did not significantly affect their performance. On the other hand, participants were more effective while pointing straight at Wattom, but reported not to finding this significantly more strenuating then when pointing to a comfortable position of their choice.
... Gestural interaction has been extensively studied in the literature [4,6,9,20]. However, most studies focus on performance, and there is comparatively much less work on the memorization of gestures [1,22,27]. ...
Various techniques have been proposed to faster command selection. Many of them either rely on directional gestures (e.g. Marking menus) or pointing gestures using a spatially-stable arrangement of items (e.g. FastTap). Both types of techniques are known to leverage memorization, but not necessarily for the same reasons. In this paper, we investigate whether using directions or positions affects gesture learning. Our study shows that, while recall rates are not significantly different, participants used the novice mode more often and spent more time while learning commands with directional gestures, and they also reported more physical and mental efforts. Moreover, this study also highlights the importance of semantic relationships between gestural commands and reports on the memorization strategies that were elaborated by the participants.
... Researchers could investigate, how quickly users adapt to such interfaces and whether the need to strongly focus on the target decreases over time. Furthermore, researchers and practitioners could apply and evaluate the slope-based method in domains other than gaze, such as motion matching for body movements [4,5,6], and mid-air gestures [2]. ...
We introduce and evaluate a novel approach for detecting smooth pursuit eye movements that increases the number of distinguishable targets and is more robust against false positives. Being natural and calibration-free, Pursuits has been gaining popularity in the past years. At the same time, current implementations show poor performance when more than eight on-screen targets are being used, thus limiting its applicability. Our approach (1) leverages the slope of a regression line, and (2) introduces a minimum signal duration that improves both the new and the traditional detection method. After introducing the approach as well as the implementation, we compare it to the traditional correlation-based Pursuits detection method. We tested the approach up to 24 targets and show that, if accepting a similar error rate, nearly twice as many targets can be distinguished compared to state of the art. For fewer targets, accuracy increases significantly. We believe our approach will enable more robust pursuit-based user interfaces, thus making it valuable for both researchers and practitioners.
We introduce a novel one-handed input technique for mobile devices that is not based on pointing, but on motion matching -where users select a target by mimicking its unique animation. Our work is motivated by the findings of a survey (N=201) on current mobile use, from which we identify lingering opportunities for one-handed input techniques. We follow by expanding on current motion matching implementations - previously developed in the context of gaze or mid-air input - so these take advantage of the affordances of touch-input devices. We validate the technique by characterizing user performance via a standard selection task (N=24) where we report success rates (>95%), selection times (~1.6 s), input footprint, grip stability, usability, and subjective workload - in both phone and tablet conditions. Finally, we present a design space that illustrates six ways in which motion matching can be embedded into mobile interfaces via a camera prototype application.
Head movements are a common input modality on VR/AR headsets. However, although they enable users to control a cursor, they lack an integrated method to trigger actions. Many approaches exist to fill this gap: dedicated ”clickers”, on-device buttons, mid-air gestures, dwell, speech and new input techniques based on matching head motions to those of visually presented targets. These proposals are diverse and there is a current lack of empirical data on the performance of, experience of, and preference for these different techniques. This hampers the ability of designers to select appropriate input techniques to deploy. We conduct two studies that address this problem. A Fitts’ Law study compares five traditional selection techniques and concludes that clicker (hands-on) and dwell (hands-free) provide optimal combinations of precision, speed and physical load. A follow-up study compares clicker and dwell to a motion matching implementation. While clicker remains fastest and dwell most accurate, motion matching may provide a valuable compromise between these two poles.
Swarms of Unmanned Aerial Vehicles (drones) could provide great benefit when used for disaster response and indoor search and rescue scenarios. In these harsh environments where GPS availability cannot be ensured, prior work often relies on cameras for control and localization. This creates the challenge of identifying each drone, i.e., finding the association between each physical ID (such as their radio address) and their visual ID (such as an object tracker output). To address this problem, prior work relies on visual cues such as LEDs or colored markers to provide unique information for identification. However, these methods often increase deployment difficulty, are sensitive to environmental changes, not robust to distance and might require hardware changes.
In this paper, we present IDrone, a robust physical drone identification system through motion matching and actuation feedback. The intuition is to (1) identify each drone by matching the motion detected through their inertial sensors and from an external camera, and (2) control the drones so they move in unique patterns that allow for fast identification, while minimizing the risk of collision involved in controlling drones with uncertain identification. To validate our approach, we conduct both simulation and real experimentation with autonomous drones for the simplified case of a stationary Spotter (powerful drone equipped with the camera). Overall, our initial results show that our approach offers a great tradeoff between fast identification and small collision probability. In particular, IDrone achieves faster identification time than safety-based baseline actuation (one-at-a-time), and significantly higher survival rate compared to fast, non-safety-based baseline actuation (random motion).
Recently, interaction techniques in which the user selects screen targets by matching their movement with the input device have been gaining popularity, particularly in the context of gaze interaction (e.g. Pursuits, Orbits, AmbiGaze, etc.). However, though many algorithms for enabling such interaction techniques have been proposed , we still lack an understanding of how they compare to each other. In this paper, we introduce two new algorithms for matching eye movements: Profile Matching and 2D Correlation, and present a systematic comparison of these algorithms with two other state-of-the-art algorithms: the Basic Correlation algorithm used in Pursuits and the Rotated Correlation algorithm used in PathSync. We also examine the effects of two thresholding techniques and post-hoc filtering. We evaluated the algorithms on a user dataset and found the 2D Correlation with one-level thresholding and post-hoc filtering to be the best performing algorithm.
SynchroWatch is a one-handed interaction technique for smartwatches that uses rhythmic correlation between a user's thumb movement and on-screen blinking controls. Our technique uses magnetic sensing to track the synchronous extension and reposition of the thumb, augmented with a passive magnetic ring. The system measures the relative changes in the magnetic field induced by the required thumb movement and uses a time-shifted correlation approach with a reference waveform for detection of synchrony. We evaluated the technique during three distraction tasks with varying degrees of hand and finger movement: active walking, browsing on a computer, and relaxing while watching online videos. Our initial offline results suggest that intentional synchronous gestures can be distinguished from other movement. A second evaluation using a live implementation of the system running on a smartwatch suggests that this technique is viable for gestures used to respond to notifications or issue commands. Finally, we present three demonstration applications that highlight the technique running in real-time on the smartwatch.
We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.
This paper presents the concept of Resonant Bits, an interaction technique for encouraging engaging, slow and skilful interaction with tangible, mobile and ubiquitous devices. The technique is based on the resonant excitation of harmonic oscillators and allows the exploration of a number of novel types of tangible interaction including: ideomotor control, where subliminal micro-movements accumulate over time to produce a visible outcome; indirect tangible interaction, where a number of devices can be controlled simultaneously through an intermediary object such as a table; and slow interaction, with meditative and repetitive gestures being used for control. The Resonant Bits concept is tested as an interaction method in a study where participants resonate with virtual pendulums on a mobile device. The Harmonic Tuner, a resonance-based music player, is presented as a simple example of using resonant bits. Overall, our ambition in proposing the Resonant Bits concept is to promote skilful, engaging and ultimately rewarding forms of interaction with tangible devices that takes time and patience to learn and master.
Touchless mid-air gestural interaction has gained mainstream attention with the emergence of off-the-shelf commodity devices such as the Leap Motion and the Xbox Kinect. One of the issues with this form of interaction is fatigue, a problem colloquially known as the ``Gorilla Arm Syndrome.' However, by allowing interaction from a rested position, whereby the elbow is rested on a surface, this problem can be limited in its effect. In this paper we evaluate 3 possible methods for performing touchless mid-air gestural interaction from a rested position: a basic rested interaction, a simple calibrated interaction which models palm positions onto a hyperplane, and a more complex calibration which models the arm's interaction space using the angles of the forearm as input. The results of this work found that the two modeled interactions conform to Fitts's law and also demonstrated that implementing a simple model can improve interaction by improving performance and accuracy.
Consumed Endurance (CE) [8] is a metric that captures the degree of arm fatigue during mid-air interactions. Research has shown that CE can assist with the design of new and minimally fatiguing gestural interfaces. We introduce the Consumed Endurance Workbench, an open source application that calculates CE in real time using an off-the-shelf skeleton tracking system. The CE Workbench tracks a person’s arm as it is moved in mid-air, determining the forces involved and calculating CE over the length of the interaction. Our demonstration focuses on how to use the CE Workbench to evaluate alternative mid-air gesture designs, how to integrate the CE Workbench with existing applications, and how to prepare the CE data for statistical analysis. We also demonstrate a mid-air text-entry layout, SEATO, which we created taking CE as the main design factor.
As researchers and industry alike are proposing TV interfaces that use gestures in their designs, understanding users' preferences for gesture commands becomes an important problem. However, no rules or guidelines currently exist to assist designers and practitioners of such interfaces. The paper presents the results of the first study investigating users' preferences for free-hand gestures when controlling the TV set. By conducting an agreement analysis on user-elicited gestures, a set of gesture commands is proposed for basic TV control tasks. Also, guidelines and recommendations issued from observed user behavior are provided to assist practitioners interested in prototyping free-hand gestural designs for the interactive TV.
Determining who is interacting with a multi-user interactive touch display is challenging. We describe a technique for associating multi-touch interactions to individual users and their accelerometer-equipped mobile devices. Real-time device accelerometer data and depth camera-based body tracking are compared to associate each phone with a particular user, while body tracking and touch contacts positions are compared to associate a touch contact with a specific user. It is then possible to associate touch contacts with devices, allowing for more seamless device-display multi-user interactions. We detail the technique and present a user study to validate and demonstrate a content exchange application using this approach.
Natural forms of interaction have evolved for personal devices that we carry with us (mobiles) as well as for shared interactive displays around us (surfaces) but interaction across the two remains cumbersome in practice. We propose a novel cross-device interaction style for mobiles and surfaces that uses the mobile for tangible input on the surface in a stylus-like fashion. Building on the direct manipulation that we can perform on either device, it facilitates fluid and seamless interaction spanning across device boundaries. We provide a characterization of the combined interaction style in terms of input, output, and contextual attributes, and demonstrate its versatility by implementation of a range of novel interaction techniques for mobile devices on interactive surfaces.
Ubiquitous computing is associated with a vision of everything being connected to everything. However, for successful applications to emerge, it will not be the quantity but the quality and usefulness of connections that will matter. Our concern is how qualitative relations and more selective connections can be established between smart artefacts, and how users can retain control over artefact interconnection. We propose context proximity for selective artefact communication, using the context of artefacts for matchmaking. We further suggest to empower users with simple but effective means to impose the same context on a number of artefacts. To prove our point we have implemented Smart-Its Friends, small embedded devices that become connected when a user holds them together and shakes them.
Guessability is essential for symbolic input, in which users enter gestures or keywords to indicate characters or commands, or rely on labels or icons to access features. We present a unified approach to both maximizing and evaluating the guessability of symbolic input. This approach can be used by anyone wishing to design a symbol set with high guessability, or to evaluate the guessability of an existing symbol set. We also present formulae for quantifying guessability and agreement among guesses. An example is offered in which the guessability of the EdgeWrite unistroke alphabet was improved by users from 51.0% to 80.1% without designer intervention. The original and improved alphabets were then tested for their immediate usability with the procedure used by MacKenzie and Zhang (1997). Users entered the original alphabet with 78.8% and 90.2% accuracy after 1 and 5 minutes of learning, respectively. The improved alphabet bettered this to 81.6% and 94.2%. These improved results were competitive with prior results for Graffiti, which were 81.8% and 95.8% for the same measures.
This paper introduces two novel navigation techniques, CycloPan, for clutch-free 2D panning and browsing, and CycloZoom+, for integrated 2D panning and zooming. These techniques instantiate a more generic concept which we call Cyclo* (CycloStar). The basic idea is that users can exert closed-loop control over several continuous variables by voluntarily modulating the parameters of a sustained oscillation. Touch-sensitive surfaces tend to offer impoverished input resources. Cyclo* techniques seem particularly promising on these surfaces because oscillations have multiple geometrical and kinematic parameters many of which may be used as controls. While CycloPan and CycloZoom+ are compatible with each other and with much of the state of the art, our experimental evaluations suggest that these two novel techniques outperform flicking and rubbing techniques.
We present a novel method calledmotion-pointingfor select- ing a set of visual items such as push-buttons without actu- ally pointing to them. Instead, each potential target displays a rhythmically animated point we call the driver. To select a specific item, the user only has to imitate the motion of its driver using the input device. Once the motion has been recognized by the system, the user can confirm the selection to trigger the action. We consider cyclic motions on an el- liptic trajectory with a specific period, and study the most effective methods for real-time matching such a trajectory, as well as the range of parameters a human can reliably re- produce. We then show how to implement motion-pointing in real applications using an interaction technique we call move-and-stroke. Finally, we measure the throughput and error rate of move-and-stroke in a controlled experiment. We show that the selection time is linearly proportional to the number of input bits conveyed up to 6 bits, confirming that motion-pointing is a practical input method.
Powerful mobile devices with minimal I/O capabilities increase the likelihood that we will want to annex these devices to I/O resources we encounter in the local environment. This opportunistic annexing will require authentication. We present a sensor-based authentication mechanism for mobile devices that relies on physical possession instead of knowledge to setup the initial connection to a public terminal. Our solution provides a simple mechanism for shaking a device to authenticate with the public infrastructure, making few assumptions about the surrounding infrastructure while also maintaining a reasonable level of security.
PhoneTouch is a novel technique for integration of mobile phones and interactive surfaces. The technique enables use of phones to select targets on the surface by direct touch, facilitating for instance pick&drop-style transfer of objects between phone and surface. The technique is based on separate detection of phone touch events by the surface, which determines location of the touch, and by the phone, which contributes device identity. The device-level observations are merged based on correlation in time. We describe a proof-of-concept implementation of the technique, using vision for touch detection on the surface (including discrimination of finger versus phone touch) and acceleration features for detection by the phone.
A challenge in facilitating spontaneous mobile interactions is to provide pairing methods that are both intuitive and secure. Simultaneous shaking is proposed as a novel and easy-to-use mechanism for pairing of small mobile devices. The underlying principle is to use common movement as a secret that the involved devices share for mutual authentication. We present two concrete methods, ShaVe and ShaCK, in which sensing and analysis of shaking movement is combined with cryptographic protocols for secure authentication. ShaVe is based on initial key exchange followed by exchange and comparison of sensor data for verification of key authenticity. ShaCK, in contrast, is based on matching features extracted from the sensor data to construct a cryptographic key. The classification algorithms used in our approach are shown to robustly separate simultaneous shaking of two devices from other concurrent movement of a pair of devices, with a false negative rate of under 12 percent. A user study confirms that the method is intuitive and easy to use, as users can shake devices in an arbitrary pattern.
This paper presents an application that uses hand gesture input to control a computer while giving a presentation. In order to develop a prototype of this application, we have defined an interaction model, a notation for gestures, and a set of guidelines to design gestural command sets. This works aims to define interaction styles that work in computerized reality environments. In our application, gestures are used for interacting with the computer as well as for communicating with other people or operating other devices.
John Leslie King's interest in history was evident at the first CSCW conference in 1986. His review of 15 years of research with technology to support real-time collocated interaction, then called Group Decision Support Systems, revealed that we sometimes ...
We present a method for performing selection tasks based on continuous control of multiple, competing agents who try to determine the user's intentions from their control behaviour without requiring an explicit pointer. The entropy in the selection process decreases in a continuous fashion -- we provide experimental evidence of selection from 500 initial targets. The approach allows adaptation over time to best make use of the multimodal communication channel between the human and the system. This general approach is well suited to mobile and wearable applications, shared displays and security conscious settings.
Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design.
This paper describes the design and evaluation of our Media Ribbon, a large public interactive display for browsing hierarchical information, with mid-air gestures. Browsing a hierarchical information space is a fundamental form of interaction. Designing learnable mid-air gestures is a current challenge for large display interaction. Our in-the-wild evaluation draws on 41 days of quantitative log data, with 4484 gestures detected, and qualitative data from 15 interviews, and associated video. We explored: whether our design enabled people to learn the gestures; how our tutorial and feedback mechanisms supported learning; and the effectiveness of support for browsing hierarchical information. Our contributions are: (1) design of large public display for browsing of hierarchical information; (2) with its gesture set; (3) insights into the ways people learn and use this interface in our context; and (4) guidelines for designing learnable mid-air gestures.
Recent works have explored the concept of movement correlation interfaces, in which moving objects can be selected by matching the movement of the input device to that of the desired object. Previous techniques relied on a single modality (e.g. gaze or mid-air gestures) and specific hardware to issue commands. TraceMatch is a computer vision technique that enables input by movement correlation while abstracting from any particular input modality. The technique relies only on a conventional webcam to enable users to produce matching gestures with any given body parts, even whilst holding objects. We describe an implementation of the technique for acquisition of orbiting targets, evaluate algorithm performance for different target sizes and frequencies, and demonstrate use of the technique for remote control of graphical as well as physical objects with different body parts.
Eye tracking offers many opportunities for direct device control in smart environments, but issues such as the need for calibration and the Midas touch problem make it impractical. In this paper, we propose AmbiGaze, a smart environment that employs the animation of targets to provide users with direct control of devices by gaze only through smooth pursuit tracking. We propose a design space of means of exposing functionality through movement and illustrate the concept through four prototypes. We evaluated the system in a user study and found that AmbiGaze enables robust gaze-only interaction with many devices, from multiple positions in the environment, in a spontaneous and comfortable manner.
In this paper, we present PathSync, a novel, distal and multi-user mid-air gestural technique based on the principle of rhythmic path mimicry; by replicating the movement of a screen-represented pattern with their hand, users can intuitively interact with digital objects quickly, and with a high level of accuracy. We present three studies that each contribute (1) improvements to how correlation is calculated in path-mimicry techniques necessary for touchless interaction, (2) a validation of its efficiency in comparison to existing techniques, and (3) a demonstration of its intuitiveness and multi-user capacity 'in the wild'. Our studies consequently demonstrate PathSync's potential as an immediately legitimate alternative to existing techniques, with key advantages for public display and multi-user applications.
We introduce Orbits, a novel gaze interaction technique that enables hands-free input on smart watches. The technique relies on moving controls to leverage the smooth pursuit movements of the eyes and detect whether and at which control the user is looking at. In Orbits, controls include targets that move in a circular trajectory in the face of the watch, and can be selected by following the desired one for a small amount of time. We conducted two user studies to assess the technique's recognition and robustness, which demonstrated how Orbits is robust against false positives triggered by natural eye movements and how it presents a hands-free, high accuracy way of interacting with smart watches using off-the-shelf devices. Finally, we developed three example interfaces built with Orbits: a music player, a notifications face plate and a missed call menu. Despite relying on moving controls – very unusual in current HCI interfaces – these were generally well received by participants in a third and final study.
We introduce our initial design of gesture interface for the TV control of visually impaired users, with its implementation and user study. Two bare-hand gesture sets were designed using simple linear and circular motions of arms. For the two gesture sets, a rule-based recognition system was developed using a Kinect sensor. In a user study, the linear and circular gesture sets on our recognition system scored similarly and showed equal or better subjective ratings compared to the commercial pointer-type gesture interface. The users were negative at pointer-type gestures and showed their concerns about large motions of linear gestures and difficult matching of circular gestures.
Consumed Endurance (CE) [8] is a metric that captures the degree of arm fatigue during mid-air interactions. Research has shown that CE can assist with the design of new and minimally fatiguing gestural interfaces. We introduce the Consumed Endurance Workbench, an open source application that calculates CE in real time using an off-the-shelf skeleton tracking system. The CE Workbench tracks a person's arm as it is moved in mid-air, determining the forces involved and calculating CE over the length of the interaction. Our demonstration focuses on how to use the CE Workbench to evaluate alternative mid-air gesture designs, how to integrate the CE Workbench with existing applications, and how to prepare the CE data for statistical analysis. We also demonstrate a mid-air text-entry layout, SEATO, which we created taking CE as the main design factor.
Eye gaze is a compelling interaction modality but requires user calibration before interaction can commence. State of the art procedures require the user to fixate on a succession of calibration markers, a task that is often experienced as difficult and tedious. We present pursuit calibration, a novel approach that, unlike existing methods, is able to detect the user's attention to a calibration target. This is achieved by using moving targets, and correlation of eye movement and target trajectory, implicitly exploiting smooth pursuit eye movement. Data for calibration is then only sampled when the user is attending to the target. Because of its ability to detect user attention, pursuit calibration can be performed implicitly, which enables more flexible designs of the calibration task. We demonstrate this in application examples and user studies, and show that pursuit calibration is tolerant to interruption, can blend naturally with applications and is able to calibrate users without their awareness.
Although gaze is an attractive modality for pervasive interactions, the real-world implementation of eye-based interfaces poses significant challenges, such as calibration. We present Pursuits, an innovative interaction technique that enables truly spontaneous interaction with eye-based interfaces. A user can simply walk up to the screen and readily interact with moving targets. Instead of being based on gaze location, Pursuits correlates eye pursuit movements with objects dynamically moving on the interface. We evaluate the influence of target speed, number and trajectory and develop guidelines for designing Pursuits-based interfaces. We then describe six realistic usage scenarios and implement three of them to evaluate the method in a usability study and a field study. Our results show that Pursuits is a versatile and robust technique and that users can interact with Pursuits-based interfaces without prior knowledge or preparation phase.
As gesture recognition technology has improved, it has been applied to various consumer appliances to provide natural interfaces. Television is a representative system for this kind of interface because of its complex requirements for various functions. Icon-based interfaces are the general method used. However, according to the performed survey, various functions were not required for watching TV. This work found most viewers mainly tend to use some specific functions such as channel or volume. These functions should be controlled easily and quickly since they are frequently used. However, the iconbased interface has difficulty fulfilling this requirement since it is affected by Fitts' law. Hence, this paper proposes a gesturebased method for easy and quick control. User hand gestures are recognized by capturing the motion path when the user draws different symbols in the air. These gestures are used to interact with the TV. It is implemented using a single-camera dedicated hardware system. This system is based on a real time parallel processing structure. The analysis shows that subjects enjoy this TV control system.
As the consumer electronics industry mourns the passing of the last of the coinventors of the wireless remote control for television, IEEE hits the pause button to review the colorful history behind the remote control to reflect on the impact of this innovation that has transformed the lives of generations of TV viewers and to celebrate the lives and contributions of these consumer electronic industry pioneers.
We investigate how to reveal an initial mid-air gesture on interactive public displays. This initial gesture can serve as gesture registration for advanced operations. We propose three strategies to reveal the initial gesture: spatial division, temporal division and integration. Spatial division permanently shows the gesture on a dedicated screen area. Temporal division interrupts the application to reveal the gesture. Integration embeds gesture hints directly in the application. We also propose a novel initial gesture called Teapot to illustrate our strategies. We report on a laboratory and field study. Our main findings are: A large percentage of all users execute the gesture, especially with spatial division (56%). Users intuitively discover a gesture vocabulary by exploring variations of the Teapot gesture by themselves, as well as by imitating and extending other users' variations.
Vision-based user interfaces enable natural interaction modalities such as gestures. Such interfaces require computationally intensive video processing at low latency. We demonstrate an application that recognizes gestures to control TV operations. Accurate recognition is achieved by using a new descriptor called MoSIFT, which explicitly encodes optical flow with appearance features. MoSIFT is computationally expensive — a sequential implementation runs 100 times slower than real time. To reduce latency sufficiently for interaction, the application is implemented on a runtime system that exploits the parallelism inherent in video understanding applications.
This research explores distributed sensing techniques for mobile devices using synchronous gestures. These are patterns of activity, contributed by multiple users (or one user with multiple devices), which take on a new meaning when they occur together in time, or in a specific sequence in time. To explore this new area of inquiry, this work uses tablet computers augmented with touch sensors and two-axis linear accelerometers (tilt sensors). The devices are connected via an 802.11 wireless network and synchronize their time-stamped sensor data. This paper describes a few practical examples of interaction techniques using synchronous gestures such as dynamically tiling together displays by physically bumping them together, discusses implementation issues, and speculates on further possibilities for synchronous gestures.
Smart home environments have evolved to the point where everyday objects and devices at home can be networked to give the inhabitants new means to control them. Familiar information appliances can be used as user interfaces (UIs) to home functions to achieve a more convenient user experience. This paper reports an ethnographic study of smart home usability and living experience. The purpose of the research was to evaluate three UIs—a PC, a media terminal, and a mobile phone—for smart home environments. The results show two main types of activity patterns, pattern control and instant control, which require different UI solutions. The results suggest that a PC can act as a central unit to control functions for activity patterns that can be planned and determined in advance. The mobile phone, on the other hand, is well suited for instant control. The mobile phone turned out to be the primary and most frequently used UI during the 6-month trial period in the smart apartment.
Thesis (Ph.D.) - University of Glasgow, 2006. Includes bibliographical references.
Can discrete, single-shot movements and continuous, cyclical movements be reduced to a single concept? In the classical, computational approach to human motor behaviour, cyclical aimed movement has generally been considered to derive from discrete primitives through a concatenation mechanism. Much importance, accordingly, has been attached to discrete-movement paradigms and to techniques allowing the segmentation of continuous data. An alternative approach, suggested by the nonlinear dynamical systems theory, views discreteness as a limiting case of cyclicity. Although attempts have been made recently to account for discrete movements in dynamical terms, cyclical paradigms have been favoured. The concatenation interpretation of cyclical aimed movement is criticized on the ground that it implies a complete waste of mechanical energy once in every half-cycle. Some kinematic data from a one-dimensional reciprocal (i.e., cyclical) aiming experiment are reported, suggesting that human subjects do save muscular efforts from one movement to the next in upper-limb cyclical aiming. The experiment demonstrated convergence on simple harmonic motion as aiming tolerance was increased, an outcome interpreted with reference to Hooke's law, in terms of the muscles' capability of storing potential, elastic energy across movement reversals. Not only is the concatenation concept problematic for understanding cyclical aimed movements, but the very reality of discrete movements is questionable too. It is pointed out that discrete motor acts of real life are composed of complete cycles, rather than half-cycles.
Choose a Different Ending. Video
- Amv Bbdo
AMV BBDO. 2011. Choose a Different Ending. Video. (23 November 2011). Retrieved September 10, 2016 from https://www.youtube.
com/watch?v=Ig5OUQcqLTg.
- T William
- Craig D Freeman
- Weissman
William T. Freeman and Craig D. Weissman. 1994. Television Control by Hand Gestures. (1994).
Real-time human pose recognition in parts from single depth images
- Jamie Shotton
- Toby Sharp
- Alex Kipman
- Andrew Fitzgibbon
- Mark Finocchio
- Andrew Blake
- Mat Cook
- Richard Moore
Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. 2013.
Real-time human pose recognition in parts from single depth images. Commun. ACM 56, 1 (2013), 116-124.