No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Skinput: Appropriating the Body as an Input Surface
Abstract
We present Skinput, a technology that appropriates the hu- man body for acoustic transmission, allowing the skin to be used as an input surface. In particular, we resolve the loca- tion of finger taps on the arm and hand by analyzing me- chanical vibrations that propagate through the body. We collect these signals using a novel array of sensors worn as an armband. This approach provides an always available, naturally portable, and on-body finger input system. We assess the capabilities, accuracy and limitations of our tech- nique through a two-part, twenty-participant user study. To further illustrate the utility of our approach, we conclude with several proof-of-concept applications we developed. Author Keywords Bio-acoustics, finger input, buttons, gestures, on-body inte- raction, projected displays, audio interfaces.
... The selection of the solid surfaces has been influenced, for example, by Han et al. [2], the Scratch Input technique [54], Skinput [53], and Touché [55]. Figure 1a illustrates the considered on-body and solid surfaces. ...
... According to Harrison et al. [51], on-body interfaces offer novel interactive possibilities, transforming our hands into input/output devices. The rationale behind on-body systems is as follows: (a) they are socially more acceptable than speech interfaces and ergonomically superior to gestural interfaces; (b) the skin provides a larger interaction area compared with mobile devices; (c) tactile feedback is delivered to users through their own body; (d) they tap into muscle memory, hand-eye coordination, and familiarity with one's own body [52]; and (e) they facilitate eyes-free interaction [51,53]. ...
... The selection of the solid surfaces has been influenced, for example, by Han et al. [2], the Scratch Input technique [54], Skinput [53], and Touché [55]. Figure 1a illustrates the considered on-body and solid surfaces. The key factor in our laboratory user studies is the relationship between the bumps and the spaces on the swipe surfaces [2]. Figure 1b illustrates the location of space and bump on surfaces in a schematic representation. ...
This work examines swipe-based interactions on smart devices, like smartphones and smartwatches, that detect vibration signals through defined swipe surfaces. We investigate how these devices, held in users’ hands or worn on their wrists, process vibration signals from swipe interactions and ambient noise using a support vector machine (SVM). The work details the signal processing workflow involving filters, sliding windows, feature vectors, SVM kernels, and ambient noise management. It includes how we separate the vibration signal from a potential swipe surface and ambient noise. We explore both software and human factors influencing the signals: the former includes the computational techniques mentioned, while the latter encompasses swipe orientation, contact, and movement. Our findings show that the SVM classifies swipe surface signals with an accuracy of 69.61% when both devices are used, 97.59% with only the smartphone, and 99.79% with only the smartwatch. However, the classification accuracy drops to about 50% in field user studies simulating real-world conditions such as phone calls, typing, walking, and other undirected movements throughout the day. The decline in performance under these conditions suggests challenges in ambient noise discrimination, which this work discusses, along with potential strategies for improvement in future research.
... OmniTouch [13] utilizes a depth-sensing projection system worn on the shoulder to facilitate interactive multitouch applications on everyday surfaces, including the user's body and the surrounding environment. Skinput [15], involving a projection system enhanced by a sensing armband worn on the upper arm, relied on bioacoustic signals generated by skin touches on pre-learned locations on the arm and hand to track interactive interfaces. SixthSense [27], a wearable gestural interface that incorporates a pico-projector worn around the neck, projects interfaces onto the body or surroundings, allowing users to interact with them using natural hand gestures tracked through color markers worn on the fngers. ...
... Challenges Visual Information (77) Mobility/Portability (54) Hands-free use for multitasking (37) Improved accessibility (29) Convenience (16) Ease of use (15) Better accuracy and efciency (12) Out-of-Pocket (11) Inclusive Technology (11) Larger display sizes (7) Futuristic technology (6) Other (5) Cost prohibitive (29) Privacy Concerns (26) Device form factor (16) Physical discomfort (15) Blocks feld of vision (smart glasses) (14) Distractive (14) Learning curve (14) Finding surfaces for projection (13) Safety concerns (13) Compatibility with Rx glasses and HAs (10) Usage in bright environments (8) Other (39) 4 Study 2 ...
... Challenges Visual Information (77) Mobility/Portability (54) Hands-free use for multitasking (37) Improved accessibility (29) Convenience (16) Ease of use (15) Better accuracy and efciency (12) Out-of-Pocket (11) Inclusive Technology (11) Larger display sizes (7) Futuristic technology (6) Other (5) Cost prohibitive (29) Privacy Concerns (26) Device form factor (16) Physical discomfort (15) Blocks feld of vision (smart glasses) (14) Distractive (14) Learning curve (14) Finding surfaces for projection (13) Safety concerns (13) Compatibility with Rx glasses and HAs (10) Usage in bright environments (8) Other (39) 4 Study 2 ...
The continued evolution of voice recognition technology has led to its integration into many smart devices as the primary mode of user interaction. Smart speakers are among the most popular smart devices that utilize voice recognition to ofer interactive functions and features to serve as a personal assistant and a control hub for smart homes. However, smart speakers rely primarily on voice recognition technology and are often inaccessible to Deaf and hard of hearing (DHH) individuals. While smart speakers such as the Amazon Echo Show have a built-in screen to provide visual interaction for DHH users through features such as "Tap to Alexa," these devices still require users to be positioned next to them. Though features such as "Tap to Alexa" improve the accessibility of smart speakers for DHH users, they are not functionally comparable solutions as they restrict DHH users from benefting the same user freedom hearing users have in interacting with them from across the room or while performing another hands-on activity. To bridge this gap, we explore alternative approaches such as augmented reality (AR) wearables and various projection systems. We conducted a mixed-method study involving surveys and Wizard of Oz evaluations to investigate the proposed research objectives. The study's fndings provide a deeper insight into the potential of AR projection interfaces as novel interaction methods to improve the accessibility of smart speakers for DHH people.
... Skin is the most user-friendly biological interface for sensing and communicating with the outside world. In recent years, there has been increasing interest in developing skin interfaces for userfriendly interactions [1][2] [3]. Skin-on interfaces mimic human skin and augment our skin surface with artificial skin to provide useful interactions like gesture recognition, haptic feedback etc. ...
... This paper proposes the integration of an AI compute engine directly on artificial skins, focusing on the computing rather than the sensing aspects. It is thus orthogonal to the specific artificial skin sensor used and can interface to any artificial skin sensor [1][2][3][4][5][6][7][8][9][10][11]. In this section, we first summarize state-of-the-art artificial skin sensors and survey the artificial skin applications that have used AI to process touch data from artificial skin sensors. ...
... Thereafter, a 12X21 matrix which indicates which specific electrodes were touched is generated. For example, if the electrode at location (1,2) is touched, the corresponding value at location (1,2) in the matrix is set to 1. Finally, the 12X21 matrix is serialized into a 252-length 1D buffer and fed to the accelerator, which performs the neural network inference, as will next be explained. ...
Existing artificial skin interfaces lack on-skin AI compute that can provide fast neural network inference for time-critical applications. In this paper, we propose AI-on-skin - a wearable artificial skin interface integrated with a neural network hardware accelerator that can be reconfigured to run diverse neural network models and applications. AI-on-skin is designed to scale to the entire body, comprising tiny, low-power, accelerators distributed across the body. We built 7 AI-on-Skin application prototypes and our user trials show AI-On-Skin achieving 20X and 50X speedup over off-body inference via Bluetooth and on-body centralized microprocessor based inference approach respectively. We also project the power performance of AI-on-skin with our accelerator fabricated as silicon chips instead of emulated on FPGAs and show 10X further power savings. To the best of our knowledge, AI-on-Skin is the first ever wearable prototype to demonstrate skin interfaces with on-body AI inference.
... [12][13][14][15] Impressive results were also obtained from sound, magnetic fields, conduction, or pressure sensors. [16][17][18][19] However, these rely on cumbersome devices, or an additional emitting device, worn on the finger, for example, or even on smart textile or skin-based sensors. [20][21][22] Some of the most advanced results were obtained with a wrist-worn device combined with strap-based infrared sensors, 23,24 impedance tomography, 25 force-sensitive sensors, 26 or photodiodes and LED measuring wrist contour. ...
... Several other studies have managed to report good results using more cumbersome devices or tailored and limited gestures. [16][17][18][19] The current method improves on these and adds the challenge of achieving similar results solely with a wrist-worn device. George Nokas et al. proposed an initial approach for eye rubbing detection using machine learning methods, that was encouraging as a proof-of-concept study but had several limitations. ...
Purpose
In this work, we present a new machine learning method based on the transformer neural network to detect eye rubbing using a smartwatch in a real-life setting. In ophthalmology, the accurate detection and prevention of eye rubbing could reduce incidence and progression of ectasic disorders, such as keratoconus, and to prevent blindness.
Methods
Our approach leverages the state-of-the-art capabilities of the transformer network, widely recognized for its success in the field of natural language processing (NLP). We evaluate our method against several baselines using a newly collected dataset, which consist of data from smartwatch sensors associated with various hand-face interactions.
Results
The current algorithm achieves an eye-rubbing detection accuracy greater than 80% with minimal (20 minutes) and up to 97% with moderate (3 hours) user-specific fine-tuning.
Conclusions
This research contributes to advancing eye-rubbing detection and establishes the groundwork for further studies in hand-face interactions monitoring using smartwatches.
Translational Relevance
This experiment is a proof-of-concept that eye-rubbing detection is effectively detectable and distinguishable from other similar hand gestures, solely through a wrist-worn device and could lead to further studies and patient education in keratoconus management.
... Wearables like glasses, earphones, rings, watches, and pendants are now equipped with processors and sensors and can communicate with other smart devices. These integrations have enabled smart devices to facilitate quick microinteractions [5,40,54,60,68], more accessible interactions [3,14,15,19,35,37,81,83], and even eyes-free interactions [42,74,106]. These interactions are often used across different devices for various reasons, including convenience, efficiency, and social acceptability. ...
... While some systems detect hand gestures using devices that are not worn on the arm or the hand, such as head-mounted devices (e.g., Meta Quest VR headset) and shoulder-mounted devices (e.g., FingerInput [85]), many use hand-worn and arm-worn devices that can detect the gestures at a closer distance. A wide range of sensors can be embedded or attached to a wristband or armband for hand gesture recognition, such as infrared (IR) ranging sensors [25,26,29,48,63], cameras [41,49,96,99,102], electromyography (EMG) sensors [8,80,81], acoustic sensors [35,43,44,53], and stretch sensors [88]. The mostly used sensor for detecting hand gestures might be the Inertial Measurement Unit (IMU) [2,28,50,54,94,98,100], due to its ability to accurately and responsively detect motions, small sensor size, low cost, and being already deployed in many wearable devices possibly because of the aforementioned benefits. ...
Wearable devices allow quick and convenient interactions for controlling mobile computers. However, these interactions are often device-dependent, and users cannot control devices in a way they are familiar with if they do not wear the same wearable device. This paper proposes a new method, UnifiedSense, to enable device-dependent gestures even when the device that detects such gestures is missing by utilizing sensors on other wearable devices. UnifiedSense achieves this without explicit gesture training for different devices, by training its recognition model while users naturally perform gestures. The recognizer uses the gestures detected on the primary device (i.e., a device that reliably detects gestures) as labels for training samples and collects sensor data from all other available devices on the user. We conducted a technical evaluation with data collected from 15 participants with four types of wearable devices. It showed that UnifiedSense could correctly recognize 5 gestures (5 gestures × 5 configurations) with an accuracy of 90.9% (SD = 1.9%) without the primary device present.
... The key benefits include the users' sense of proprioception that allows some operations to be performed eyes-freely in the user's personal space without reliance on the visual feedback provided by a display and the direct use of body cues such as facial expression and posture to help manage interpersonal social coordination. This model has inspired HCI researchers to exploit the benefits of body-centric interaction, such as extending the interaction space of a mobile device [6,36] or providing on-body UIs that leverage the body skin [25,40], body landmark [23,54], muscle activation [50], body coordinate [22], and hand gestures [2,10] for HCI. ...
... Apparatus. Eight relays of two square coil sizes (w c = [25,40]mm) and four transmission lines lengths (l t =[100, 200, 400, 800]mm) of our implementation were used in the study. We evaluated the performance of the relays by measuring the reading distance between a 30mm-diameter NTAG213 NFC tag and a relay's transmitter coil, which is connected to the back of an iPhone 11 at the location of its NFC reader antenna. ...
NFC (Near-Field Communication) has been widely applied for human-computer interaction (HCI). However, the short sensing distance of NFC requires the users to initiate the tasks with extra effort mostly using their hands, so it is inconvenient to use NFC in hands-busy scenarios. This paper presents an investigation of body-centric interactions between the NFC device users and their surroundings. The exploration is based on the recent development of near-field enabled clothing, which can passively extend an NFC-enabled device's reading distance to users' body landmarks. We present an accessible method for fabricating flexible, extensible, and scalable NFC extenders on clothing pieces, and an easy-to-use toolkit for facilitating designers to realize the interactive experiences. The method and toolkit were tested in technical experiments and in a co-creation workshop. The elicited design outcomes and the further exploratory makings generated knowledge for future research and embodied interaction design opportunities. CCS CONCEPTS • Human-centered computing → User interface toolkits; Ubiquitous and mobile computing systems and tools; Interaction design theory, concepts and paradigms. Figure 1: Body-centric interaction with NFC devices through a piece of near-field enabled clothing, which extends the reading range of a mobile device's NFC to the user's protruding body landmarks (e.g., elbows) for touch interaction.
... Sinapov et al. [32] investigated using sound emitted from an object to predict its object category, classifying small objects based on sounds generated when interacting with a robotic arm (including the sound of the robot's motors). Harrison [15] observed that the sound emitted from an object may be used to predict its material category when exploring three objects (hand, paper, LCD) in the context of human-computer interaction. Inspired by these approaches, we make an alternate use of sound for semantic image segmentation, firstly to help predict the materials of everyday objects, and secondly to use these material predictions to refine our overall predictions of object categories. ...
It is not always possible to recognise objects and infer material properties for a scene from visual cues alone, since objects can look visually similar whilst being made of very different materials. In this paper, we therefore present an approach that augments the available dense visual cues with sparse auditory cues in order to estimate dense object and material labels. Since estimates of object class and material properties are mutually informative, we optimise our multi-output labelling jointly using a random-field framework. We evaluate our system on a new dataset with paired visual and auditory data that we make publicly available. We demonstrate that this joint estimation of object and material labels significantly outperforms the estimation of either category in isolation.
... To address smartwatch usability challenges from limited screen space, gesture recognition approaches have extended the input area to nearby skin surfaces using camera or RF-based sensing [11,13,18,58,62,63]. However, these systems typically focus on wrist-adjacent skin surfaces and do not support broader hand-face interactions. ...
Hand-face interactions play a key role in many everyday tasks, providing insights into user habits, behaviors, intentions, and expressions. However, existing wearable sensing systems often struggle to track these interactions in daily settings due to their reliance on multiple sensors or privacy-sensitive, vision-based approaches. To address these challenges, we propose WristSonic, a wrist-worn active acoustic sensing system that uses speakers and microphones to capture ultrasonic reflections from hand, arm, and face movements, enabling fine-grained detection of hand-face interactions with minimal intrusion. By transmitting and analyzing ultrasonic waves, WristSonic distinguishes a wide range of gestures, such as tapping the temple, brushing teeth, and nodding, using a Transformer-based neural network architecture. This approach achieves robust recognition of 21 distinct actions with a single, low-power, privacy-conscious wearable. Through two user studies with 15 participants in controlled and semi-in-the-wild settings, WristSonic demonstrates high efficacy, achieving macro F1-scores of 93.08% and 82.65%, respectively.
... In the past, a wide variety of approaches have been considered to enable on-skin touch input for on-body interactions, such as wearing a conventional trackpad on the body [65], acoustic sensing [27,50], depth sensing camera [26,59,62], touch-enabled textile sleeve [57], an array of infrared sensors [51,66,67], RF and capacitive sensing [56,81,82,84]. ...
... Several studies have examined using different body parts, such as the arms, palms, and skin, as input surfaces for VR interactions [11,16,38]. Skinput [25] and Touché [51], for instance, explore using the body as an input surface by sensing acoustic and capacitive signals to recognize gestures on the skin. ...
On-body menus present a novel interaction paradigm within Virtual Reality (VR) environments by embedding virtual interfaces directly onto the user's body. Unlike traditional screen-based interfaces, on-body menus enable users to interact with virtual options or icons visually attached to their physical form. In this paper, We investigated the impact of the creation process on the effectiveness of on-body menus, comparing first-person, third-person, and mirror perspectives. Our first study (N = 12) revealed that the mirror perspective led to faster creation times and more accurate recall compared to the other two perspectives. To further explore user preferences, we conducted a second study (N = 18) utilizing a VR system with integrated body tracking. By combining distributions of icons from both studies (N = 30), we confirmed significant preferences in on-body menu placement based on icon category (e.g., Social Media icons were consistently placed on forearms). We also discovered associations between categories, such as Leisure and Social Media icons frequently co-occurring. Our findings highlight the importance of the creation process, uncover user preferences for on-body menu organization, and provide insights to guide the development of intuitive and effective on-body interactions within virtual environments.
... This was done with the help of fabric antennas and a 10x10 grid of RFID ICs (integrated circuits) with their own codes. Human-machine interaction always requires touch or body movement, and the most popular on-body interfaces, like trackpads and tapping buttons [86,87,88,89], are often built around the arm to recognize hand movement. Skin electronics [90] are a new idea for a bendable technology that can recognize touch and gestures on the body. ...
In this work, an overview of implementing machine learning (ML) models in antenna design and optimisation has been proposed. This includes deep learning on ML structure, categories, and frameworks to obtain useful and general insights about methods of predicting, collecting, and analysing high throughput fast data using ML techniques. An in-depth overview on the various published research works related to designing and optimising of antennas using ML is proposed, including the different ML- techniques and algorithms that have been used to generate antenna parameters such as S-parameters, radiation pattern, and gain values. However, the designing of modern antennae is still complicated regarding structure, variables, and environmental factors. Moreover, the cost of time and computational resources are unavoidable and unacceptable for most users. To address these challenges, ML methods-based antennas have been developed and applied to improve the reduction in the efficiency and accuracy of antenna modelling. This can be involved methods to rain models on data that can be utilised to predict the antenna performance for a given set of antenna design variables. This work summarises the developed and applied MLs that have been proposed to improve the efficiency and accuracy of antenna modelling
... E-textiles and wearables could overcome challenges associated with using current AAC systems, such as carrying a device or folder and impaired fine motor skills. For example, new technologies can be used to detect body and limb movements, touches made on the surface of clothing, and to measure things such as heart rate or pressure [37][38][39][40]. Recent research and new developments have focused on gesture-sensing gloves [35,36,41], where the technologies used do not overcome previously presented challenges, such as the need for a separate energy source. ...
Cerebral palsy (CP) is a neurological disorder that primarily affects body movement, muscle coordination, and muscle tone. Non-progressive brain injury or abnormal brain development that occurs while the child’s brain is under development causes CP. It is estimated that 40–85% of children with CP have communication difficulties. Children with communication difficulties can benefit from augmentative and alternative communication (AAC). However, studies have shown that several challenges in using AAC exist. Developing existing AAC methods and creating new AAC solutions are important to enable everyone to communicate and express themselves without barriers. This qualitative study aims to investigate how parents of children with CP would use a wireless and battery-free, passive radio-frequency identification (RFID)-based e-textile—AACloth—as an AAC solution. The research was conducted via an online survey. Parents with a child under 15 with CP and communication difficulties were included. Parents were recruited by distributing the survey invitation via the Finnish CP Association’s monthly newsletter, Facebook page, and social media groups. Nine parents participated. Based on parents’ views, the AACloth could solve some of the challenges associated with existing AAC methods. This research provides perspectives on what kinds of factors should be considered when developing existing and new AAC aids.
... BACKGROUND This work builds upon several prior research studies, in addition to the authors' previously reported results cited above. The mechanical transmission of vibrations in the body has attracted attention in many research fields, including perception [23], [24], [25], [26], occupational health [27], [28], [29], and human-computer interaction [30], [31], [32]. Many researchers have investigated relations between the transmitted vibrations and vibrotactile perception, including studies by von Békésy, who investigated similarities between mechanical signals underlying tactile sensations in the skin and auditory sensations in the ear [33]. ...
The importance of interpersonal touch for social well-being is widely recognized, and haptic technology offers a promising avenue for augmenting these interactions. We presented smart bracelets that use vibrotactile feedback to augment social interactions, such as handshakes, by transmitting vibrations between two people. This work conducts mechanical and perceptual experiments to investigate key factors affecting the delivery of interpersonal vibrotactile feedback via bracelets. Our results show that low-frequency vibrations elicited through tangential actuation are efficiently transmitted from the wrist to the hand, with amplitude varying based on distance, frequency, and actuation direction. We also found that vibrations transmitted to different locations on the hand can be felt by a second person, with perceptual intensity correlated with oscillation magnitude at the touched location. Additionally, we demonstrate how wrist-interfaced devices can elicit spatial vibration patterns throughout the hand surface, which can be manipulated by the frequency and direction of actuation at the wrist. Our experiments provide important insights into the human factors associated with interpersonal vibrotactile feedback and have significant implications for the design of technologies that promote social well-being.
... 2.2.1 Passive sensing. For hand gesture recognition, researchers utilized the passive sound signal transmitted through bone conduction propagation from the skin vibration [10,15]. However, Each context has a more specific type with a tap and swipe gesture. ...
... When elements that cannot be discretized, such as images, 2D maps or large text segments are considered, they are still displayed in a rectilinear plane. Designs that consider the hand as a fat surface also mostly follow this convention where a rectilinear element is ftted on the space on the hand when it is held fat [4,24,32]. As content will need to be displayed on the regions of the fngers, using such rectilinear elements would not be ideal. ...
... Textile antennas and a 10 Â 10 array of RFID ICs (integrated circuits) with a unique code were employed in this. Human-machine interaction is always reliant on touch or body movement, and the most prevalent on-body interfaces, such as trackpads and tapping buttons [86,87,88,89], are typically blended around the arm to identify hand movement. Skin electronics [90] have recently been proposed as a flexible technology for on-body touch and gesture recognition. ...
The next generation of wireless communication networks will rely heavily on machine learning and deep learning. In comparison to traditional ground-based systems, the development of various communication-based applications is projected to increase coverage and spectrum efficiency. Machine learning and deep learning can be used to optimize solutions in a variety of applications, including antennas. The latter have grown popular for obtaining effective solutions due to high computational processing, clean data, and large data storage capability. In this research, machine learning and deep learning for various antenna design applications have been discussed in detail. The general concept of machine learning and deep learning is introduced. However, the main focus is on various antenna applications, such as millimeter wave, body-centric, terahertz, satellite, unmanned aerial vehicle, global positioning system, and textiles. The feasibility of antenna applications with respect to conventional methods, acceleration of the antenna design process, reduced number of simulations, and better computational feasibility features are highlighted. Overall, machine learning and deep learning provide satisfactory results for antenna design.
... Our working hypothesis was that we could use machine learning to identify the vibrational patterns produced at the distal end of the forearm by active flexion and extension of the fingers and wrist. Previous studies have shown that vibrations induced by tapping the forearm can be read out using sensors in an armband, highlighting the fact that there is informational content in vibrations that propagate through the forearm [10]. Here, we studied, for the first time to our knowledge, whether finger/wrist movements produced vibrations at the wrist that could be used to identify the occurrence of finger/wrist movements. ...
The ability to count finger and wrist movements throughout the day with a nonobtrusive, wearable sensor could be useful for hand-related healthcare applications, including rehabilitation after a stroke, carpal tunnel syndrome, or hand surgery. Previous approaches have required the user to wear a ring with an embedded magnet or inertial measurement unit (IMU). Here, we demonstrate that it is possible to identify the occurrence of finger and wrist flexion/extension movements based on vibrations detected by a wrist-worn IMU. We developed an approach we call “Hand Activity Recognition through using a Convolutional neural network with Spectrograms” (HARCS) that trains a CNN based on the velocity/acceleration spectrograms that finger/wrist movements create. We validated HARCS with the wrist-worn IMU recordings obtained from twenty stroke survivors during their daily life, where the occurrence of finger/wrist movements was labeled using a previously validated algorithm called HAND using magnetic sensing. The daily number of finger/wrist movements identified by HARCS had a strong positive correlation to the daily number identified by HAND (R2 = 0.76, p < 0.001). HARCS was also 75% accurate when we labeled the finger/wrist movements performed by unimpaired participants using optical motion capture. Overall, the ringless sensing of finger/wrist movement occurrence is feasible, although real-world applications may require further accuracy improvements.
We present Ring-a-Pose, a single untethered ring that tracks continuous 3D hand poses. Located in the center of the hand, the ring emits an inaudible acoustic signal that each hand pose reflects differently. Ring-a-Pose imposes minimal obtrusions on the hand, unlike multi-ring or glove systems. It is not affected by the choice of clothing that may cover wrist-worn systems. In a series of three user studies with a total of 36 participants, we evaluate Ring-a-Pose's performance on pose tracking and micro-finger gesture recognition. Without collecting any training data from a user, Ring-a-Pose tracks continuous hand poses with a joint error of 14.1mm. The joint error decreases to 10.3mm for fine-tuned user-dependent models. Ring-a-Pose recognizes 7-class micro-gestures with a 90.60% and 99.27% accuracy for user-independent and user-dependent models, respectively. Furthermore, the ring exhibits promising performance when worn on any finger. Ring-a-Pose enables the future of smart rings to track and recognize hand poses using relatively low-power acoustic sensing.
On-body menus present a novel interaction paradigm within Virtual Reality (VR) environments by embedding virtual interfaces directly onto the user’s body. Unlike traditional screen-based interfaces, on-body menus enable users to interact with virtual options or icons visually attached to their physical form. In this paper, We investigated the impact of the creation process on the effectiveness of on-body menus, comparing first-person, third-person, and mirror perspectives. Our first study ( N = 12) revealed that the mirror perspective led to faster creation times and more accurate recall compared to the other two perspectives. To further explore user preferences, we conducted a second study ( N = 18) utilizing a VR system with integrated body tracking. By combining distributions of icons from both studies ( N = 30), we confirmed significant preferences in on-body menu placement based on icon category ( e.g. , Social Media icons were consistently placed on forearms). We also discovered associations between categories, such as Leisure and Social Media icons frequently co-occurring. Our findings highlight the importance of the creation process, uncover user preferences for on-body menu organization, and provide insights to guide the development of intuitive and effective on-body interactions within virtual environments.
Hand microgestures are promising for mobile interaction with wearable devices. However, they will not be adopted if practitioners cannot communicate to users the microgestures associated with the commands of their applications. This requires unambiguous representations that simultaneously show the multiple microgestures available to control an application. Using a systematic approach, we evaluate how these representations should be designed and contrast 4 conditions depending on the microgestures (tap-swipe and tap-hold) and fingers (index and index-middle) considered. Based on the results, we design a simultaneous representation of microgestures for a given set of 14 application commands. We then evaluate the usability of the representation for novice users and the suitability of the representation for small screens compared with a baseline. Finally, we formulate 8 recommendations based on the results of all the experiments. In particular, redundant graphical and textual representations of microgestures should only be displayed for novice users.
In this chapter, I review how the fields of Human Computer Interaction (HCI) and Computer Science have come together over the last 60 years to create and support novel forms of interaction. We see how interaction models have progressively incorporated human skills and abilities, as well as the physical and social properties taken from the real world. I organize this evolution into three periods—pre-HCI, seminal HCI, ubiquitous HCI. Each of these periods is illustrated with key reference works that have influenced my own research as well as contributions of French researchers to the field.
Knowing the object grabbed by a hand can offer essential contextual information for interaction between the human and the physical world. This paper presents a novel system, ViObject, for passive object recognition that uses accelerometer and gyroscope sensor data from commodity smartwatches to identify untagged everyday objects. The system relies on the vibrations caused by grabbing objects and does not require additional hardware or human effort. ViObject's ability to recognize objects passively can have important implications for a wide range of applications, from smart home automation to healthcare and assistive technologies. In this paper, we present the design and implementation of ViObject, to address challenges such as motion interference, different object-touching positions, different grasp speeds/pressure, and model customization to new users and new objects. We evaluate the system's performance using a dataset of 20 objects from 20 participants and show that ViObject achieves an average accuracy of 86.4%. We also customize models for new users and new objects, achieving an average accuracy of 90.1%. Overall, ViObject demonstrates a novel technology concept of passive object recognition using commodity smartwatches and opens up new avenues for research and innovation in this area.
This paper presents a study on the touch precision of an eye-free, body-based interface using on-body and near-body touch methods with and without skin contact. We evaluate user touch accuracy on four different button layouts. These layouts progressively increase the number of buttons between adjacent body joints, resulting in 12, 20, 28, and 36 touch buttons distributed across the body. Our study indicates that the on-body method achieved an accuracy beyond 95% for the 12- and 20-button layouts, whereas the near-body method only for the 12-button layout. Investigating user touch patterns, we applied SVM classifiers, which boost both the on-body and near-body methods to support up to the 28-button layouts by learning individual touch patterns. However, using generalized touch patterns did not significantly improve accuracy for more complex layouts, highlighting considerable differences in individual touch habits. When evaluating user experience metrics such as workload perception, confidence, convenience, and willingness-to-use, users consistently favored the 20-button layout regardless of the touch technique used. Remarkably, the 20-button layout, when applied to on-body touch methods, does not necessitate personal touch patterns, showcasing an optimal balance of practicality, effectiveness, and user experience without the need for trained models. In contrast, the near-body touch targeting the 20-button layout needs a personalized model; otherwise, the 12-button layout offers the best immediate practicality.
This chapter reviews the different sensory modalities that are impacted in autism. Each modality is introduced with related technology projects for each described as well. The diversity in sensory processing become the user requirements for technology systems that are customized to unique combinations of strengths and weaknesses that occur on an individual basis as well as in unique ways across the autism spectrum. An overview of design implications to address hypo- and hyper-sensitivities across sensory modalities which provided with further details on combinations of these sensory accommodations. Lastly, a complex system temporal processing is mentioned yet discussion occurs in a future chapter.
We introduce RadarHand, a wrist-worn wearable with millimetre wave radar that detects on-skin touch-based proprioceptive hand gestures. Radars are robust, private, small, penetrate materials, and require low computation costs. We first evaluated the proprioceptive and tactile perception nature of the back of the hand and found that tapping on the thumb is the least proprioceptive error of all the finger joints, followed by the index finger, middle finger, ring finger, and pinky finger in the eyes-free and high cognitive load situation. Next, we trained deep-learning models for gesture classification. We introduce two types of gestures based on the locations of the back of the hand: generic gestures and discrete gestures. Discrete gestures are gestures that start at specific locations and end at specific locations at the back of the hand, in contrast to generic gestures, which can start anywhere and end anywhere on the back of the hand. Out of 27 gesture group possibilities, we achieved 92% accuracy for a set of seven gestures and 93% accuracy for the set of eight discrete gestures. Finally, we evaluated RadarHand’s performance in real-time under two interaction modes: Active interaction and Reactive interaction. Active interaction is where the user initiates input to achieve the desired output, and reactive interaction is where the device initiates interaction and requires the user to react. We obtained an accuracy of 87% and 74% for active generic and discrete gestures, respectively, as well as 91% and 81.7% for reactive generic and discrete gestures, respectively. We discuss the implications of RadarHand for gesture recognition and directions for future works.
Nod and shake of one's head are intuitive and universal gestures in communication. As smartwatches become increasingly intelligent through advances in user activity sensing technologies, many use scenarios of smartwatches demand quick responses from users in confirmation dialogs, to accept or dismiss proposed actions. Such proposed actions include making emergency calls, taking service recommendations, and starting or stopping exercise timers. Head gestures in these scenarios could be preferable to touch interactions for being hands-free and easy to perform. We propose Headar to recognize these gestures on smartwatches using wearable millimeter wave sensing. We first surveyed head gestures to understand how they are performed in conversational settings. We then investigated positions and orientations to which users raise their smartwatches. Insights from these studies guided the implementation of Headar. Additionally, we conducted modeling and simulation to verify our sensing principle. We developed a real-time sensing and inference pipeline using contemporary deep learning techniques, and proved the feasibility of our proposed approach with a user study (n=15) and a live test (n=8). Our evaluation yielded an average accuracy of 84.0% in the user study across 9 classes including nod and shake as well as seven other signals -- still, speech, touch interaction, and four non-gestural head motions (i.e., head up, left, right, and down). Furthermore, we obtained an accuracy of 72.6% in the live test which reveals rich insights into the performance of our approach in various realistic conditions.
Use of virtual reality while seated is common, but studies on seated interaction beyond the use of controllers or hand gestures have been sparse. This work present LapTouch, which makes use of the lap as a touch interface and includes two user studies to inform the design of direct and indirect touch interaction using the lap with visual feedback that guides the user touch, as well as eye-free interaction in which users are not provided with such visual feedback. The first study suggests that direct interaction can provide effective layouts with 95% accuracy with up to a 4×4 layout and a shorter completion time, while indirect interaction can provide effective layouts with up to a 4×5 layout but a longer completion time. Considering user experience, which revealed that 4-row and 5-column layouts are not preferred, it is recommended to use both direct and indirect interaction with a maximum of a 3×4 layout. According to the second study, increasing the eye-free interaction with support vector machine (SVM) allows for a 2×2 layout with a generalized model and 2×2, 2×3 and 3×2 layouts with personalized models.
The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.
Direct use of the hand as an input device is an attractive method for providing natural human–computer interaction (HCI). Currently, the only technology that satisfies the advanced requirements of hand-based input for HCI is glove-based sensing. This technology, however, has several drawbacks including that it hinders the ease and naturalness with which the user can interact with the computer-controlled environment, and it requires long calibration and setup procedures. Computer vision (CV) has the potential to provide more natural, non-contact solutions. As a result, there have been considerable research efforts to use the hand as an input device for HCI. In particular, two types of research directions have emerged. One is based on gesture classification and aims to extract high-level abstract information corresponding to motion patterns or postures of the hand. The second is based on pose estimation systems and aims to capture the real 3D motion of the hand. This paper presents a literature review on the latter research direction, which is a very challenging problem in the context of HCI.
A reliable and unobtrusive measurement of working mem- ory load could be used to evaluate the efficacy of interfaces and to provide real-time user-state information to adaptive systems. In this paper, we describe an experiment we con- ducted to explore some of the issues around using an elec- troencephalograph (EEG) for classifying working memory load. Within this experiment, we present our classification methodology, including a novel feature selection scheme that seems to alleviate the need for complex drift modeling and artifact rejection. We demonstrate classification accu- racies of up to 99% for 2 memory load levels and up to 88% for 4 levels. We also present results suggesting that we can do this with shorter windows, much less training data, and a smaller number of EEG channels, than reported pre- viously. Finally, we show results suggesting that the models we construct transfer across variants of the task, implying some level of generality. We believe these findings extend prior work and bring us a step closer to the use of such technologies in HCI research.
Two hundred and fifty years ago the Japanese Zen master Hakuin asked the question, "What is the Sound of the Single Hand?" This koan has long served as an aid to meditation but it also describes our new interaction techinique. We discovered that gentle fingertip gestures such as tapping, rubbing, and flicking make quiet sounds that travel by bone conduction throughout the hand. A small wristband-mounted contact microphone can reliably and inexpensively sense these sounds. We harnessed this "sound in the hand" phenomenon to build a wristband-mounted bio-acoustic fingertip gesture interface. The bio-acoustic interface recognizes some common gestures that state-of-the-art glove and image-processing techniques capture but in a smaller, mobile package.
A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneiderman's theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.
This paper presents a vision-based interface for controlling a computer mouse via 2D and 3D hand gestures. The proposed interface builds upon our previous work that permits the detection and tracking of multiple hands that can move freely in the field of view of a potentially moving camera system. Dependable hand tracking, combined with finger- tip detection, facilitates the definition of simple and, therefore, robustly interpretable vocabularies of hand gestures that are subsequently used to enable a human operator convey control information to a computer sys- tem. Two such vocabularies are defined, implemented and validated. The first one depends only on 2D hand tracking results while the second also makes use of 3D information. As confirmed by several experiments, the proposed interface achieves accurate mouse positioning, smooth cursor movement and reliable recognition of gestures activating button events. Owing to these properties, our interface can be used as a virtual mouse for controlling any Windows application.
Mobile input technologies can be bulky, obtrusive, or difficult to use while performing other tasks. In this paper, we present Hambone, a lightweight, unobtrusive system that affords quick access, subtlety, and multitasking capabilities for gesture-based mobile device interaction. Hambone uses two small piezoelectric sensors placed on either the wrist or ankle. When a user moves his hands or feet, the sounds generated by the movement travel to Hambone via bone conduction. Hambone then transmits the signals digitally to a mobile device or computer. The signals are recognized using hidden Markov models (HMMs) and are mapped to a set of commands controlling an application. In this paper, we present the hardware and software implementation of Hambone, a preliminary evaluation, and a discussion of future opportunities in bio-acoustic gesture-based interfaces.
The widespread adoption of mobile electronic devices and the advent of wearable computing have encouraged the development of compact alternatives to the keyboard and mouse. These include one-handed keyboards, digitizing tablets, and glove-based devices. This paper describes a combination pointer position and non-chorded keystroke input device that relies on miniature wrist-worn wireless video cameras that track finger position. A hidden Markov model is used to correlate finger movements to keystrokes during a brief training phase, after which the user can type in the air or above a flat surface as if typing on a standard keyboard. Language statistics are used to help disambiguate keystrokes, allowing the assignment of multiple unique keys to each finger and obviating chorded input. In addition, the system can be trained to recognize certain finger positions for switching between input modes; for example, from typing mode to pointer movement mode. In the latter mode of operation, the position of the mouse pointer is controlled by hand movement. The camera motion is estimated by tracking environmental features and is used to control pointer position. This allows fast switching between keystroke mode and pointer control mode
Wearable computers can now merge seamlessly into ordinary clothing. Using various conductive textiles, data and power distribution as well as sensing circuitry can be incorporated directly into wash-and-wear clothing. This paper describes some of the techniques used to build circuits from commercially available fabrics, yarns, fasteners, and components.
In this paper, we explore the concept of dual--purpose speech: speech that is socially appropriate in the context of a human-- to--human conversation which also provides meaningful input to a computer. We motivate the use of dual--purpose speech and explore issues of privacy and technological challenges related to mobile speech recognition. We present three applications that utilize dual--purpose speech to assist a user in conversational tasks: the Calendar Navigator Agent, DialogTabs, and Speech Courier. The Calendar Navigator Agent navigates a user's calendar based on socially appropriate speech used while scheduling appointments. DialogTabs allows a user to postpone cognitive processing of conversational material by proving short--term capture of transient information. Finally, Speech Courier allows asynchronous delivery of relevant conversational information to a third party. Additional Keywords and Phrases: Speech user interfaces, dual--purpose speech, mobile computing 1
We describe a novel interface that locates and characterizes knocks and taps atop a large glass window. Our current setup uses four contact piezoelectric pickups located near the sheet's corners to record the acoustic wavefront coming from the knocks. A digital signal processor extracts relevant characteristics from these signals, such as amplitudes, frequency components, and differential timings, which are used to estimate the location of the hit and provide other parameters, including the rough accuracy of this estimate, the nature of each hit (e.g., knuckle knock, metal tap, or fist bang), and the strike intensity. This system requires only simple hardware, needs no special adaptation of the glass pane, and allows all transducers to be mounted on the inner surface, hence it is quite easy to deploy as a retrofit to existing windows. This opens many applications, such as an interactive storefront, with projected content controlled by knocks on the display window.
This paper introduces a novel interface for digitallyaugmented cooperative play. We present the concept of the "athletic-tangible interface," a new class of interaction which uses tangible objects and full-body motion in physical spaces with digital augmentation. We detail the implementation of PingPongPlus, a "reactive ping-pong table", which features a novel sound-based ball tracking technology. The game is augmented and transformed with dynamic graphics and sound, determined by the position of impact, and the rhythm and style of play. A variety of different modes of play and initial experiences with PingPongPlus are also described. Keywords tangible interface, enhanced reality, augmented reality, interactive surface, athletic interaction, kinesthetic interaction, computer-supported cooperative play. INTRODUCTION When an expert plays ping-pong, a well-used paddle becomes transparent, and allows a player to concentrate on the task -- playing ping-pong. The good fit of grasp is vit...
The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.
Biometric input devices can provide assistive technology access to people who have little or no motor control. We explore a biometric control interface based on the Galvanic Skin Response, to determine its effectiveness as a non-muscular channel of input. This paper presents data from several online studies of a locked-in subject using a Galvanic Skin Response system for communication and control. We present issues with GSR control, and approaches that may improve accuracy and information transfer rate.
We explore the feasibility of muscle-computer interfaces (muCIs): an interaction methodology that directly senses and decodes human muscular activity rather than relying on physical device actuation or user actions that are externally visible or audible. As a first step towards realizing the mu- CI concept, we conducted an experiment to explore the potential of exploiting muscular sensing and processing technologies for muCIs. We present results demonstrating accurate gesture classification with an off-the-shelf elec- tromyography (EMG) device. Specifically, using 10 sensors worn in a narrow band around the upper forearm, we were able to differentiate position and pressure of finger presses, as well as classify tapping and lifting gestures across all five fingers. We conclude with discussion of the implica- tions of our results for future muCI designs. Author Keywords: Electromyography (EMG), Muscle-
Information is traditionally confined to paper or digitally to a screen. In this paper, we introduce WUW, a wearable gestural interface, which attempts to bring information out into the tangible world. By using a tiny projector and a camera mounted on a hat or coupled in a pendant like wearable device, WUW sees what the user sees and visually augments surfaces or physical objects the user is interacting with. WUW projects information onto surfaces, walls, and physical objects around us, and lets the user interact with the projected information through natural hand gestures, arm movements or interaction with the object itself.
We present a computer vision technique to detect when the user brings their thumb and forefinger together (a pinch gesture) for close-range and relatively controlled viewing circumstances. The technique avoids complex and fragile hand tracking algorithms by detecting the hole formed when the thumb and forefinger are touching; this hole is found by simple analysis of the connected components of the back- ground segmented against the hand. Our Thumb and Fore- Finger Interface (TAFFI) demonstrates the technique for cursor control as well as map navigation using one and two- handed interactions.
Current asynchronous voice messaging interfaces, like voicemail, fail to take advantage of our conversational skills. TalkBack restores conversational turn-taking to voicemail retrieval by dividing voice messages into smaller sections based on the most significant silent and filled pauses and pausing after each to record a response. The responses are composed into a reply, alternating with snippets of the original message for context. TalkBack is built into a digital picture frame; the recipient touches a picture of the caller to hear each segment of the message in turn. The minimal interface models synchronous interaction and facilitates asynchronous voice messaging. TalkBack can also present a voice-annotated slide show which it receives over the Internet.
We introduce PlayAnywhere, a front-projected computer vision-based interactive table system which uses a new commercially available projection technology to obtain a compact, self-contained form factor. PlayAnywhere's configuration addresses installation, calibration, and portability issues that are typical of most vision-based table systems, and thereby is particularly motivated in consumer applications. PlayAnywhere also makes a number of contributions related to image processing techniques for front-projected vision-based table systems, including a shadow-based touch detection algorithm, a fast, simple visual bar code scheme tailored to projection-vision table systems, the ability to continuously track sheets of paper, and an optical flow-based algorithm for the manipulation of onscreen objects that does not rely on fragile tracking algorithms.
We present Scratch Input, an acoustic-based input tech- nique that relies on the unique sound produced when a fin- gernail is dragged over the surface of a textured material, such as wood, fabric, or wall paint. We employ a simple sensor that can be easily coupled with existing surfaces, such as walls and tables, turning them into large, unpow- ered and ad hoc finger input surfaces. Our sensor is suffi- ciently small that it could be incorporated into a mobile device, allowing any suitable surface on which it rests to be appropriated as a gestural input surface. Several example applications were developed to demonstrate possible inter- actions. We conclude with a study that shows users can perform six Scratch Input gestures at about 90% accuracy with less than five minutes of training and on wide variety of surfaces.
Modern brain sensing technologies provide a variety of methods for detecting specific forms of brain activity. In this paper, we present an initial step in exploring how these technologies may be used to perform task classification and applied in a relevant manner to HCI research. We describe two experiments showing successful classification between tasks using a low-cost off-the-shelf electroencephalograph (EEG) system. In the first study, we achieved a mean classi- fication accuracy of 84.0% in subjects performing one of three cognitive tasks - rest, mental arithmetic, and mental rotation - while sitting in a controlled posture. In the second study, conducted in more ecologically valid setting for HCI research, we attained a mean classification accuracy of 92.4% using three tasks that included non-cognitive fea- tures: a relaxation task, playing a PC based game without opponents, and engaging opponents within the game. Throughout the paper, we provide lessons learned and dis- cuss how HCI researchers may utilize these technologies in their work.
Emerging technologies offer exciting new ways of using entertainment technology to create fantastic play experiences and foster interactions between players. Evaluating entertainment technology is challenging because success isn't defined in terms of productivity and performance, but in terms of enjoyment and interaction. Current subjective methods of evaluating entertainment technology aren't sufficiently robust. This paper describes two experiments designed to test the efficacy of physiological measures as evaluators of user experience with entertainment technologies. We found evidence that there is a different physiological response in the body when playing against a computer versus playing against a friend. These physiological results are mirrored in the subjective reports provided by the participants. In addition, we provide guidelines for collecting physiological data for user experience analysis, which were informed by our empirical investigations. This research provides an initial step towards using physiological responses to objectively evaluate a user's experience with entertainment technology.
The popularity of computer games has exploded in recent years, yet methods of evaluating user emotional state during play experiences lag far behind. There are few methods of assessing emotional state, and even fewer methods of quantifying emotion during play. This paper presents a novel method for continuously modeling emotion using physiological data. A fuzzy logic model transformed four physiological signals into arousal and valence. A second fuzzy logic model transformed arousal and valence into five emotional states relevant to computer game play: boredom, challenge, excitement, frustration, and fun. Modeled emotions compared favorably with a manual approach, and the means were also evaluated with subjective self-reports, exhibiting the same trends as reported emotions for fun, boredom, and excitement. This approach provides a method for quantifying emotional states continuously during a play experience.
People can learn to control mu (8 /12 Hz) or beta (18 /25 Hz) rhythm amplitude in the EEG recorded over sensorimotor cortex and use it to move a cursor to a target on a video screen. In the present version of the cursor movement task, vertical cursor movement is a linear function of mu or beta rhythm amplitude. At the same time the cursor moves horizontally from left to right at a fixed rate. A target occupies 50% (2-target task) to 20% (5-target task) of the right edge of the screen. The user’s task is to move the cursor vertically so that it hits the target when it reaches the right edge. The goal of the present study was to optimize system performance. To accomplish this, we evaluated the impact on system performance of number of targets (i.e. 2 /5) and trial duration (i.e. horizontal movement time from 1 to 4 s). Performance was measured as accuracy (percent of targets selected correctly) and also as bit rate (bits/min) (which incorporates, in addition to accuracy, speed and the number of possible targets). Accuracy declined as target number increased. At the same time, for six of eight users, four targets yielded the maximum bit rate. Accuracy increased as movement time increased. At the same time, the movement time with the highest bit rate varied across users from 2 to 4 s. These results indicate that task parameters such as target number and trial duration can markedly affect system performance. They also indicate that optimal parameter values vary across users. Selection of parameters suited both to the specific user and the requirements of the specific application is likely to be a key factor in maximizing the success of EEG-based communication and control.
People can learn to control mu (8-12 Hz) or beta (18-25 Hz) rhythm amplitude in the EEG recorded over sensorimotor cortex and use it to move a cursor to a target on a video screen. In the present version of the cursor movement task, vertical cursor movement is a linear function of mu or beta rhythm amplitude. At the same time the cursor moves horizontally from left to right at a fixed rate. A target occupies 50% (2-target task) to 20% (5-target task) of the right edge of the screen. The user's task is to move the cursor vertically so that it hits the target when it reaches the right edge. The goal of the present study was to optimize system performance. To accomplish this, we evaluated the impact on system performance of number of targets (i.e. 2-5) and trial duration (i.e. horizontal movement time from 1 to 4 s). Performance was measured as accuracy (percent of targets selected correctly) and also as bit rate (bits/min) (which incorporates, in addition to accuracy, speed and the number of possible targets). Accuracy declined as target number increased. At the same time, for six of eight users, four targets yielded the maximum bit rate. Accuracy increased as movement time increased. At the same time, the movement time with the highest bit rate varied across users from 2 to 4 s. These results indicate that task parameters such as target number and trial duration can markedly affect system performance. They also indicate that optimal parameter values vary across users. Selection of parameters suited both to the specific user and the requirements of the specific application is likely to be a key factor in maximizing the success of EEG-based communication and control.
The Wadsworth electroencephalogram (EEG)-based brain-computer interface (BCI) uses amplitude in mu or beta frequency bands over sensorimotor cortex to control cursor movement. Trained users can move the cursor in one or two dimensions. The primary goal of this research is to provide a new communication and control option for people with severe motor disabilities. Currently, cursor movements in each dimension are determined 10 times/s by an empirically derived linear function of one or two EEG features (i.e., spectral bands from different electrode locations). This study used offline analysis of data collected during system operation to explore methods for improving the accuracy of cursor movement. The data were gathered while users selected among three possible targets by controlling vertical [i.e., one-dimensional (1-D)] cursor movement. The three methods analyzed differ in the dimensionality of the cursor movement [1-D versus two-dimensional (2-D)] and in the type of the underlying function (linear versus nonlinear). We addressed two questions: Which method is best for classification (i.e., to determine from the EEG which target the user wants to hit)? How does the number of EEG features affect the performance of each method? All methods reached their optimal performance with 10-20 features. In offline simulation, the 2-D linear method and the 1-D nonlinear method improved performance significantly over the 1-D linear method. The 1-D linear method did not do so. These offline results suggest that the 1-D nonlinear or the 2-D linear cursor function will improve online operation of the BCI system.
The Biofeedback Pointer is a graphic input device controlled by
wrist motion. Moving the wrist causes the pointer to move in that
direction. The pointer detects the electromyograms of four of the
muscles used to move the wrist. These are interpreted by a neural
network which is trained for each user. The network takes about half a
minute to train. The Biofeedback Pointer was found to perform 14% as
well as the mouse in simple pointing tasks, which is about half of the
lowest performance of a common graphic input device. Using a more
sophisticated neural network or better training may improve the device
to a more comparable level
Clumsy intermediary devices constrain our interaction with computers and their applications. Glove-based input devices let us apply our manual dexterity to the task. We provide a basis for understanding the field by describing key hand-tracking technologies and applications using glove-based input. The bulk of development in glove-based input has taken place very recently, and not all of it is easily accessible in the literature. We present a cross-section of the field to date. Hand-tracking devices may use the following technologies: position tracking, optical tracking, marker systems, silhouette analysis, magnetic tracking or acoustic tracking. Actual glove technologies on the market include: Sayre glove, MIT LED glove, Digital Data Entry Glove, DataGlove, Dexterous HandMaster, Power Glove, CyberGlove and Space Glove. Various applications of glove technologies include projects into the pursuit of natural interfaces, systems for understanding signed languages, teleoperation and robotic control, computer-based puppetry, and musical performance.< >