Article

Multimodal user interfaces: Designing media for the auditory and the tactile channel

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The last few years have seen a rapid growth of both theoretical and commercial interest in the design of multimodal interfaces (e.g., Hempel & Altinsoy, 2005;Oviatt, 2002;Spence & Driver, 1997;Tannen, Nelson, Bolia, Warm, & Dember, 2004). Research on the topic of user interface design has gradually transitioned from the study of how individual sensory modalities (predominantly vision) interact with a system to the study of multimodal (or multisensory) human-computer interaction. ...
... Auditory icons have the advantage over auditory earcons in that their meaning should be more immediately apparent to an interface operator and so people should require less time in order to learn the appropriate behavioural responses to such signals (e.g. Begault 1994, Lucas 1995, Hempel and Altinsoy 2005, see also Keller and Stevens 2004). However, despite the fact that research has shown that people do indeed tend to respond more rapidly to auditory warning signals as the perceived level of urgency increases (e.g. ...
Article
This article reviews various different approaches to the design of unimodal and multisensory warning signals, in particular warning signals for use in alerting drivers to potentially dangerous situations. The design of optimal warning signals that are maximally effective in terms of the limitations that constrain human information processing are discussed in light of the latest findings emerging from cognitive neuroscience research. A new approach to the design of multisensory warning signals, involving the presentation of warning signals in different regions of space around a driver, is then critically examined.
... Multimodal interfaces are increasingly being used for a variety of purposes because they have the potential to facilitate effective and efficient interactions between humans and computers (Hempel & Altınsoy, 2005;Sorkin, 1987). The use of multiple display and control modalities enables different ways of presenting and responding to information, the incorporation of redundancy into displays, and emulation of real-life environments. ...
Article
Full-text available
Studies of stimulus-response compatibility show that certain stimulus modalities go more naturally with particular response modalities than others. For example, it is more natural to say the word "left" to a spoken word LEFT than to move a control left, whereas it is more natural to move a control left to a light onset to the left than it is to say the word "left". Such modality relations determine not only how quickly a person can respond to relevant information but also the amount of interference that they may get from irrelevant distracting information. Cues that signal the spatial location of an upcoming target stimulus facilitate processing not only when the two stimuli are presented in the same modality but also when they occur in different modalities. In general, these crossmodal cuing effects occur when the cue and target stimulus are spatially co-located. In this paper, we review the findings on compatibility and cuing studies across stimulus modalities and discuss their implications for multimodal interface design.
... That is, just how much information can be transmitted to the " visually overloaded " driver (see [87]) by means of tactile, auditory, and/or multisensory displays? While there have been some interesting developments in this area recently (e.g., see [34], [50], and [107]), research from our laboratory has shown that at least in the absence of prolonged training (once again, not a practical option for normal drivers), tactile information processing across the body surface is quite limited (see [27] for a recent review). For example, without extensive training, people simply cannot count more than two or three tactile stimuli when presented simultaneously across their body surface (or hands; see [25] and [28]). ...
Article
The last few years have seen many exciting developments in the area of tactile and multisensory interface design. One of the most rapidly-moving practical application areas for these findings is in the development of warning signals and information displays for drivers. For instance, tactile displays can be used to awaken sleepy drivers, to capture the attention of distracted drivers, and even to present more complex information to drivers who may be visually-overloaded. This review highlights the most important potential costs and benefits associated with the use of tactile and multisensory information displays in a vehicular setting. Multisensory displays that are based on the latest cognitive neuroscience research findings can capture driver attention significantly more effective than their unimodal (i.e., tactile) counterparts. Multisensory displays can also be used to transmit information more efficiently, as well as to reduce driver workload. Finally, we highlight the key research questions currently awaiting further research, including questions such as: Are tactile warning signals really intuitive? Are there certain regions of the body (or in the space surrounding the body) where tactile/multisensory warning signals are particularly effective? To what extent is the spatial coincidence and temporal synchrony of the individual sensory signals critical to determining the effectiveness of multisensory displays? And, finally, how does the issue of compliance vs. reliance (or the 'cry wolf' phenomenon associated with the presentation of signals that are perceived as false alarms) influence the effectiveness of tactile and/or multisensory warning signals?
Article
Full-text available
Visual-tactile carry-over effects of global/local processing (attention to the whole, versus the details) have been reported under active touch conditions. We investigated whether carry-over effects of global/local processing also occur for passive touch and whether global/local processing has differential effects on affective and discriminative aspects of touch. Participants completed two tactile tasks involving pleasantness rating and discrimination of a set of tactile vibrations before and after completing a version of the Navon task that encouraged a focus on the global (n = 30), local (n = 30), or both (n = 30) features of a series of visual stimuli. In line with previous research suggesting a link between global processing and positive emotion, global processing increased pleasantness ratings of high (but not low) frequency tactile vibrations. Local processing did not improve the ability to discriminate between vibrations of different frequencies, however. There was some evidence of a tactile-visual carry-over effect; prior local processing of tactile vibrations reduced global precedence during the Navon task in the control group. We have shown carry-over effects of global versus local processing on passive touch perception. These findings provide further evidence suggesting that a common perceptual mechanism determines processing level across modalities and show for the first time that prior global processing affects the pleasantness of touch.
Article
The purpose of this paper is to present a versatile clamping system to apply controlled pressure during the consolidation of objects or when adhering parts into position. Known as shimbari in Japan, this technique uses flexible fiberglass or bamboo sticks braced against a frame to provide clamping pressure. This paper describes readily available equipment that can be used and discusses how the principles can be adapted to solve a wide range of clamping problems. Additionally, the paper presents case studies to illustrate the use of the shimbari technique along with several variations. This method works well for objects that do not allow for conventional clamping and can be applied to a range of lacquered, japanned, painted, veneered, or inlaid surfaces.
Article
The continued advancement in computer interfaces to support 3D tasks requires a better understanding of how users will interact with 3D user interfaces in a virtual workspace. This article presents two studies that investigated the effect of visual, auditory, and haptic sensory feedback modalities presented by a virtual button in a 3D environment on task performance (time on task and task errors) and user rating. Although we expected task performance to improve for conditions that combined two or three feedback modalities over a single modality, we instead found a significant emergent behavior that decreased performance in the trimodal condition. We found a significant increase in the number of presses when a user released the button before closing the virtual switch, suggesting that the combined visual, auditory, and haptic feedback led participants to prematurely believe they actuated a button. This suggests that in the design of virtual buttons, considering the effect of each feedback modality independently is not sufficient to predict performance, and unexpected effects may emerge when feedback modalities are combined.
Article
The majority of multimodal systems on research level as well as on product level involve speech input. With speech as an eyes-free hands-free input modality, these systems enable the user to interact more effectively across a large range of tasks and environments. But the advantages of multimodal user interfaces will only show up if they are designed to support the abilities and characteristics of the human users. Thus, it is necessary to integrate research results from cognitive sciences in the development process. This paper discusses several experimental findings that demonstrate this necessity. User-centered design methods and user testing will further improve the usability of multimodal systems. However, compared to voice-only interfaces the design, development and usability testing of multimodal systems are far more complicated. A process model shows how the interplay between the development of system components, user-centered evaluation and the integration of knowledge from cognitive sciences can be organized.
Article
The sense of touch can provide an effective, but at present underutilized, alternative to vision for information presentation to drivers. A series of empirical studies was designed to investigate the possibility of improving driver responses to potential critical emergency situations by the presentation of directional haptic warning signals. The results demonstrated the potential utility of haptic (and combined auditory and haptic) warning signals in capturing the visual attention of drivers to the direction requiring their immediate attention. These results have important implications for the increasingly popular multisensory interfaces.
Article
Full-text available
The suggestion that the body surface might be used as an additional means of presenting information to human-machine operators has been around in the literature for nearly 50 years. Although recent technological advances have made the possibility of using the body as a receptive surface much more realistic, the fundamental limitations on the human information processing of tactile stimuli presented across the body surface are, however, still largely unknown. This literature review provides an overview of studies that have attempted to use vibrotactile interfaces to convey information to human operators. The importance of investigating any possible central cognitive limitations (i.e., rather than the peripheral limitations, such as related to sensory masking, that were typically addressed in earlier research) on tactile processing for the most effective design of body interfaces is highlighted. The applicability of the constraints emerging from studies of tactile processing under conditions of unisensory (i.e., purely tactile) stimulus presentation, to more ecologically valid conditions of multisensory stimulation, is also discussed. Finally, the results obtained from recent studies of tactile information processing under conditions of multisensory stimulation are described, and their implications for haptic/tactile interface design elucidated.
Article
A driving simulator study was conducted in order to assess the relative utility of unimodal auditory, unimodal vibrotactile, and combined audiotactile (i.e., multisensory) in-car warning signals to alert and inform drivers of likely front-to-rear-end collision events in a situation modeled on real-world driving. The implementation of nonvisual in-car warning signals may have important safety implications in lessening any visual overload during driving. Multisensory integration can provide synergistic facilitation effects. The participants drove along a rural road in a car-following scenario in either the presence or absence of a radio program in the background. The brake light signals of the lead vehicle were also unpredictably either enabled or disabled on a trial-by-trial basis. The results showed that the participants initiated their braking responses significantly more rapidly following the presentation of audiotactile warning signals than following the presentation of either unimodal auditory or unimodal vibrotactile warning signals. Multisensory warning signals offer a particularly effective means of capturing driver attention in demanding situations such as driving. The potential value of such multisensory in-car warning signals is explained with reference to recent cognitive neuroscience research.
Article
This paper addresses the temporal factors involved in the integration of auditory and tactile information. Perceptual threshold values for auditory-tactile asynchrony were measured using a broadband noise for the auditory stimulus (presented to the subjects via headphones) and a sinusoidal wave for the tactile stimulus (presented at the tip of the index finger via a shaker). In the first experiment, subjects were asked to make a three-alternative forced-choice judgment whether the audio stimulus and the tactile stimulus were synchronous, the audio stimulus preceded the tactile stimulus, or the tactile stimulus preceded the audio stimulus. Stimuli with audio delays in the range of –26 to 51 ms were judged synchronous. In the second experiment, the judgement was whether the auditory stimulus and the tactile stimulus were synchronous or asynchronous. The results showed that stimuli with audio delays in the range of –24 to 50 ms were judged synchronous. In the third experiment the subjects had to judge whether the audio stimulus preceded the tactile stimulus, or the tactile stimulus preceded the audio stimulus. Stimuli with audio delays in the range of –13 to 28 ms were judged synchronous.
Article
This article presents a model of character recognition and the experiments used to develop and test it. The model applies to foveal viewing of blurred or unblurred characters and to tactile sensing of raised characters using the fingerpad. The primary goal of the model is to account for variations in legibility across character sets; a secondary goal is to account for variations in the cell entries of the confusion matrix for a given character set. The model consists of two distinct processing stages. The first involves transformation of each stimulus into an internal representation; this transformation consists of linear low-pass spatial filtering followed by nonlinear compression of stimulus intensity. The second stage involves both template matching of the transformed test stimulus with each of the stored internal representations of the characters within the set and response selection, which is assumed to conform to the unbiased choice model of Luce (1963). Though purely stimulus driven, the model accounts quite well for differences in the legibility of character sets differing in character type, size of character, and number of characters within the set; it is somewhat less successful in accounting for the details of each confusion matrix.