Serena Planera’s research while affiliated with Osnabrück University and other places

A large number of studies suggest that the integration of multisensory signals by humans is well-described by Bayesian principles. However, there are very few reports about cue combination between a native and an augmented sense. In particular, we asked the question whether adult participants are able to integrate an augmented sensory cue with existing native sensory information. Hence for the purpose of this study, we build a tactile augmentation device. Consequently, we compared different hypotheses of how untrained adult participants combine information from a native and an augmented sense. In a two-interval forced choice (2 IFC) task, while subjects were blindfolded and seated on a rotating platform, our sensory augmentation device translated information on whole body yaw rotation to tactile stimulation. Three conditions were realized: tactile stimulation only (augmented condition), rotation only (native condition), and both augmented and native information (bimodal condition). Participants had to choose one out of two consecutive rotations with higher angular rotation. For the analysis, we fitted the participants' responses with a probit model and calculated the just notable difference (JND). Then, we compared several models for predicting bimodal from unimodal responses. An objective Bayesian alternation model yielded a better prediction (χred2 = 1.67) than the Bayesian integration model (χred2 = 4.34). Slightly higher accuracy showed a non-Bayesian winner takes all (WTA) model (χred2 = 1.64), which either used only native or only augmented values per subject for prediction. However, the performance of the Bayesian alternation model could be substantially improved (χred2 = 1.09) utilizing subjective weights obtained by a questionnaire. As a result, the subjective Bayesian alternation model predicted bimodal performance most accurately among all tested models. These results suggest that information from augmented and existing sensory modalities in untrained humans is combined via a subjective Bayesian alternation process. Therefore, we conclude that behavior in our bimodal condition is explained better by top down-subjective weighting than by bottom-up weighting based upon objective cue reliability.

Background / Purpose: We developed a sensory augmentation device (feelSpace belt), which transfers heading direction into tactile stimulation onto the waist of the participants. Therefore we aim to investigate the psychophysics underpinning multimodal integration between an augmented sense and an innate sensory modality, the vestibular one, for heading direction in humans. Main conclusion: Our preliminary results show that participants in both unimodal conditions, reliably used the information from vestibular and belt cues for the estimation of heading orientation. However, the same pattern did not show up in the bimodal condition. Nevertheless, further investigations with a bigger sample size need to be done in order to truly test our hypothesis of Bayesian integration. In summary, we hereby demonstrate that we are able to test and analyse multimodal cue integration in the domain of sensory augmentation.