ArticlePDF Available

Augmented Cross-modality: Translating the Physiological Responses, Knowledge and Impression to Audio-visual Information in Virtual Reality

Article

Augmented Cross-modality: Translating the Physiological Responses, Knowledge and Impression to Audio-visual Information in Virtual Reality

Abstract and Figures

This paper proposes the method of interaction design to present haptic experience as intended in virtual reality (VR). The method that we named “Augmented Cross-Modality” is to translate the physiological responses, knowledge and impression about the experience in real world into audio-visual stimuli and add them to the interaction in VR. In this study, as expressions for presenting a haptic experience of gripping an object strongly and lifting a heavy object, we design hand tremor, strong gripping and increasing heart rate in VR. The objective is, at first, to enhance a sense of strain of a body with these augmented cross-modal expressions and then, change the quality of the total haptic experience and as a result, make it closer to the experience of lifting a heavy object. This method is evaluated by several rating scales, interviews and force sensors attached to a VR controller. The result suggests that the expressions of this method enhancing a haptic experience of strong gripping in almost all participants and the effectiveness were confirmed. c 2018 Society for Imaging Science and Technology.
Content may be subject to copyright.
JOURNAL OF
IMAGING SCIENCE AND TECHNOLOGY
JIST
VOL. 62, NO. 6
|
NOVEMBER/DECEMBER 2018
JOURNAL OF IMAGING SCIENCE AND TECHNOLOGY VOL. 62, NO. 6
NOVEMBER/DECEMBER 2018
AutoSens
Conference
2019
A conference created
by engineers for engineers
IS&T journal readers receive
a 15% discount code to attend,
simply visit
go.auto-sens.com/IST
Before cars can be connected, engineers must first connect.
The most influential vehicle perception event, invites 1,000
of the world’s leading minds in ADAS and autonomous vehicles
to meet again throughout 2019 in both Europe and the USA.
Each event is a week of discussions, collaborations, debates,
business meetings, interactive learning, autonomous driving
demonstrations and technology showcasing.
17 – 19
Sept 2019
Brussels,
Belgium
14 – 16
May 2019
Detroit,
USA
REPRINTED FROM:
Journal of Imaging Science and Technology R
62(6): 060402-1–060402-8, 2018.
c
Society for Imaging Science and Technology 2018
Augmented Cross-modality: Translating the Physiological
Responses, Knowledge and Impression to Audio-visual
Information in Virtual Reality
Yutaro Hirao and Takashi Kawai
Department of Intermedia Art and Science, Waseda University, 3-4-1 Okubo, Shinjuku, Tokyo, 169-0072, Japan
E-mail: yutaro-hirao@suou.waseda.jp
Abstract. This paper proposes the method of interaction design to
present haptic experience as intended in virtual reality (VR). The
method that we named “Augmented Cross-Modality” is to translate
the physiological responses, knowledge and impression about the
experience in real world into audio-visual stimuli and add them to
the interaction in VR. In this study, as expressions for presenting a
haptic experience of gripping an object strongly and lifting a heavy
object, we design hand tremor, strong gripping and increasing heart
rate in VR. The objective is, at first, to enhance a sense of strain
of a body with these augmented cross-modal expressions and
then, change the quality of the total haptic experience and as a
result, make it closer to the experience of lifting a heavy object.
This method is evaluated by several rating scales, interviews and
force sensors attached to a VR controller. The result suggests that
the expressions of this method enhancing a haptic experience of
strong gripping in almost all participants and the effectiveness were
confirmed. c
2018 Society for Imaging Science and Technology.
[DOI: 10.2352/J.ImagingSci.Technol.2018.62.6.060402]
1. INTRODUCTION
In recent years, we can see and touch digital information with
reality due to the development of Head Mounted Display
(HMD), stereophonic technology. Thus, there is a problem of
how to make the quality of experience in the interaction with
virtual objects more realistic. This becomes an active area of
research especially in the field of force feedback and tactile
sensation, called Haptics [1].
For example, grounded force feedback display typified
by PHANToM [2] and SPIDAR [3], Cybergrasp [4] and
exoskeleton force feedback display such as Nagawara’s
one [5] are based on the approach that aims to present
force feedback by calculating and mechanically reproducing
physical force which is produced in the interaction with
virtual objects. There is another approach that aims to
present target feedback that is not physically the same
output as real one by using perceptual illusion from
human cognitive characteristics. There are various studies
in this approach such as using Pseudo-haptics [611], using
electrical stimulation [12] and using vibrator [13].
Among these approaches, many recent studies are
being done especially using the method of cross-modal
Received June 19, 2018; accepted for publication Oct. 28, 2018; published
online Dec. 7, 2018. Associate Editor: Hang-Bong Kang.
1062-3701/2018/62(6)/060402/8/$25.00
integration. The reason is that its structure is not overly
complex and it provides the benefit of allowing users
to move freely and comparatively simple system without
hardware problems such as maximum output of motor,
weight, latency and spatial resolution. For instance, both
studies of Taima et al. [7] and Rietzler et al. [8,9] propose
the method presenting weight sense or reaction force by
making a gap between the movement of real body and
virtual body. Furthermore, the studies of Azmandian [14],
Matsumoto [15] and Nagao [16] enable a user various
actions in virtual reality (VR) though space or resources
in real are limited by changing the correspondence relation
between actual body movement and virtual one. Thus, the
effectiveness of presenting a haptic experience by integrating
embodiment into VR and changing it in VR is becoming
clearer. In addition, it is assumed that this approach is feasible
for immersive VR using HMD because of its high flexibility
of audio and visual expression.
1.1 Problem
As one of the problems in the method presenting haptic
feedback by using cross-modality described above, the
problem lies in the fact that the haptic experience may
have large individual differences depending on the degree
of expressions and make users feel discomfort or other
unintended sensation. For example, this is shown in our
preliminary experiment in which we evaluated a haptic illu-
sion in VR using cross-modality. The method is presenting
various weight senses when a user lifting a virtual object in
VR by delaying virtual object (and hand model) following his
or her hand or by increasing the distance between the hand
and the virtual object. The result suggests that participants
significantly felt heavier as the gap became larger and can
perceive multistage weight of the virtual object by both
approaches. However, the larger the gap is, the larger the
individual differences are. Furthermore, some participants
commented that they felt inertial force, viscosity or the
speed of movements itself instead of the sense of lifting a
heavy object. It is assumed that one of the factors of this
phenomenon is that participants who could not feel the sense
of lifting a heavy object by cross-modality recognize the
expression of delay or offset as different from the intended
context.
J. Imaging Sci. Technol. 060402-1 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
Figure 1. The conceptual diagram of augmented cross-modal system.
2. OBJECTIVE
It is suggested by several experiments [1719] that personal
experience and knowledge tend to affect haptic experience
presented by cross-modality. Therefore, there is a possibility
that the quality of the haptic experience is changed and
enhanced, and becomes closer to one of the real experiences
by translating the physiological responses, knowledge and
impression about the target experience in real world into
audio-visual information and adding it to the interaction in
VR. Whereas conventional approaches using cross-modality
aim to make the visual expression in VR closer to the real one,
the method proposed here focused on making users evoke
their past experience, knowledge and impression related to
the target experience through designed expressions in VR.
Thus, it does not matter how real the expression is and the
expression can be unique in VR. We named the cross-modal
illusion by this approach ‘‘Augmented Cross-Modality.’’
In this research, as expressions for presenting the
experience of gripping an object strongly and lifting a
heavy object, we design a hand tremor, strong gripping and
increasing heart rate. Only with the conventional pseudo-
haptic method using a gap between virtual body movements
and actual movements, it is possible that haptic experience
might have diversity such as inertial force, viscosity or
just slow. In contrast, by adding the augmented cross-
modal expressions described above, the haptic experience is
expected to be unified into the experience of strong gripping
and lifting a heavy object more than with just delay.
Therefore, in this paper, the main objective is to evaluate
the effect of augmented cross-modal expressions on the
quality of the haptic experience in VR. More specifically,
the first objective is to evaluate whether the augmented
cross-modal expressions of strain while lifting enhance the
sense of strong gripping of the object, and the second is
to evaluate if these expressions change the quality of the
total haptic experience and as a result, make it closer to
the experience of lifting a heavy object than just delay
expression. This evaluation is performed by measuring an
objective index of force sensor attached to a VR controller
and subjective indexes of rating scales and interviews.
3. METHOD
3.1 Approach
Figure 1shows the conceptual diagram of augmented
cross-modality system. This system aims not to reproduce
the way to interact with objects in real world or stimuli
received from the interaction, but to design expressions
for VR by translating physiological responses, knowledge
and impression about the experience of the interaction in
real world into audio-visual stimuli. With these augmented
cross-modal expressions, two effects are expected. The first
one is synchronization of actual physiological responses with
expressions of them and the second one is evoking the
knowledge and impression related to the target experience
due to the designed expressions. Through this approach, it
is expected that though the way to interact with objects or
the stimuli received from the interaction is different from the
real one, the VR experience resembles the characteristics of
real-life experience.
There are two main advantages in the proposed method.
The first one is that the proposed method has good
compatibility with immersive VR using HMD which has
a high degree of freedom of audio and visual expressions.
Due to the high degree of freedom of expressions, the
possibility of audio-visual expressions in VR expands further
than in conventional 2-dimensional display or mixed reality
(MR) technology. Thus, it is assumed that it is possible to
present haptic information with more variety and wider
scalability. The second is that the examination of the design
of audio and visual expressions is a main task to do for
this method. Therefore, there is an advantage that devices
tend to be comparably simple and can be realized with a
relatively simple system without limiting the degree of how
participants can move freely. In addition, another advantage
is that hardware problem such as maximum output of
J. Imaging Sci. Technol. 060402-2 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
motor, weight, latency and spatial resolution does not occur
frequently in the present study.
3.2 Stimuli
As the stimulus of the proposed method, physiological
responses, knowledge and impression about the experience
of a user lifting a heavy object were translated into visual–
audio expressions. Specifically, a hand model tremor and a
skin color of the hand model turning red (an expression
for strong gripping) were presented as visual expressions,
and heartbeat getting louder and faster was presented as an
audio expression. In addition, these expressions were given
in stages over time while a participant was lifting a virtual
object in VR. These expressions are added to the condition of
delaying the speed at which the virtual object and the hand
model follow the actual hand movement (delay expression).
Figure 2shows the behavioral image of the proposed method
in time series. The detail of each expression is described
below.
In the delay expression, the virtual object (and a
hand model) moves to the coordinates which is linearly
complemented by the parameter k(0k1,kR) between
the virtual object and the VR controller for each frame. Now
defining the position of the hand of a participant (same as one
of the VR controllers) in the f frame as yfand the position of
the virtual object as Yf, the expression is as follows.
Yf=Yf1+(yfYf1)·k.(1)
In this experiment, the delay parameter was purposely
set ‘‘very slow’’ (k=0.0005) because of two reasons. One
reason is that the objective of the augmented cross-modal
expressions is to present a sense of strain of body while
lifting a heavy object. Therefore, the ‘‘heavy’’ expression is
needed in this case and the delay parameter ‘‘0.0005’’ was
confirmed to present a heavy sensation in our preliminary
experiment. The other reason is that we wanted to know
whether the augmented cross-modal expressions eased the
unnaturalness of the large gap between the actual body and
virtual body. As we discussed in ‘‘A. Problem,’’ the more the
delay was, the heavier the weight sense of the participant
was. At the same time, however, the individual differences
of weight sense were larger and participants felt stronger
unnaturalness. The delay parameter ‘‘0.0005’’ is the heaviest
parameter in the parameters used in our past study but most
participants commented they felt strongest unnaturalness
with it. From above, we set the parameter as 0.0005 in the
present experiment.
Next, various augmented cross-modal expressions are
explained below in time series. First, as the initial condition,
heartbeat sound whose volume is around the detection limit
at a rate of 0.5 times/sec is presented—namely, participant
is less conscious of the heartbeat sound in a normal state
though it is noticeable if consciousness is given. Then, lateral
vibration whose frequency is 70 rad/sec and amplitude is 1
mm is started as an expression of hand tremor 1 second after
a participant lifted up the virtual object. After that, 2 seconds
after the start of lifting, the animation starts where the skin
color of the hand model turns red as an expression of the
strong gripping and it is completed in 2 seconds. The heart
rate monotonically increases to 1 time/sec and the heartbeat
sound monotonically increases to 5 times the first volume
(which is sufficiently loud for being noticeable and is under
the allowable limit) within 6 seconds from 4 seconds after the
start of lifting to 10 seconds. Each value of these expressions
was decided by carrying out user tests in the preliminary
experiment. In this experiment, the three conditions below
are evaluated.
A virtual object and a hand model follow the user’s actual
hand
with completely synchronized (control condition).
with delay of which the parameter is 0.0005 (delay
condition).
with delay (k=0.0005, same as delay condition)
and augmented cross-modal expressions (complex
condition).
3.3 Equipment
Participants are equipped with HMD for VR and headphones
and have a VR controller on the right hand. Force sensors
are attached to the trigger and the grip of the VR controller,
and the data from these sensors is transmitted to the personal
computer (PC) by Bluetooth from the Analog to Digital
Converter (ADC) mounted on the waist of the participant.
The equipment used in this experiment is as follows.
HMD for VR: HTC VIVE Headset
VR controller: HTC VIVE Controller
Force sensor: Tekscan Flexi Force
ADC: Plux biosignalsplux ADC
In the VR space, a virtual dumbbell model is placed on
a table whose height is 50 cm from the floor. In addition,
the local position and rotation of the hand model in VR
corresponds to that of VR controller, and it is possible to
perform operations such as gripping and releasing virtual
objects with triggers of the controller. The experimental
system and operation image are shown in Figure 3.
3.4 Measurements
Evaluation was carried out by four questionnaires with
seven-point scales on the lifting experience of the virtual
object, a force sensor and a free description interview. Details
are shown below.
Participants are asked to answer the four questionnaires
that ‘‘Did you feel the weight?’’, ‘‘Did you feel a sense
of strong gripping?’’, ‘‘Did you feel that you lifted up the
dumbbell?’’ and ‘‘Were you aware of your actual body during
the operation?’’ by 7-point scales from ‘‘Strongly agree’’ to
‘‘Strongly disagree.’’ About the questionnaire of whether
a participant felt lifting up the dumbbell, participants are
taught to judge the reality of the experience of lifting up an
object. In addition, about the questionnaires of whether a
participant was conscious of themselves during operation,
J. Imaging Sci. Technol. 060402-3 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
Figure 2. Detailed description of stimuli in time series.
participants were taught to judge whether the position or
posture of their actual body were conscious of.
Force sensors are used to measure the strength of
gripping the controller while lifting the virtual object as
an objective index of illusion intensity and quality of
experience. The reason why the force sensor is used is
that in our preliminary experiment which evaluated the
illusion intensity and the quality of the experience with the
force sensor in similar haptic illusion presentation, a certain
tendency was found. The force sensor measurement was
performed at sampling rate of 1 [kHz] and resolution of 216
[bit]. Conversion from the acquired digital value D to force F
[kgf] is calculated according to the following equations. Now,
3 [v] of the numerator in the equation (2) is the maximum
output voltage of the biosignals Plux ADC, and 6 [v] and 47
[] of the denominator in equation (3) represent the applied
voltage and the reference resistance (GND side) respectively
when the force sensor Flexi Force is pressured, and 0.000331
[mS / kgf] of the denominator in equation (4) is the rate of
change of the force sensor Flexi Force.
Vout[V] = 3D
216 (2)
G[mS]=Vout
((6Vout)·47)(3)
F[kgf] = G
0.000331 .(4)
3.5 Procedure
The experiment participants were 20 university students in
their 20s (18 males, 2 females). They were explained the
purpose, method and flow of the experiment beforehand and
asked to fill in the consent form and the attribute sheet. Next,
they mounted the HMD to see the experimental environment
in the VR space, and then were explained the tasks and
precautions in detail. After that, in order to familiarize
the operation and the environment in the VR space, they
operated the dumbbell object of the control condition freely
as a practice trial. This practice trial ended when they were
able to operate as they wanted and did not feel the sense of
incongruity with hand model. After the practice trial, main
experiment is carried out in accordance with the following
flow.
1. The experimenter randomly selects one condition of the
three conditions.
2. With the signal of the experimenter the participant lifts
the dumbbell object toward ones chest height.
3. After ten seconds from the start of lifting, the experi-
menter sends a signal and the participant releases the
dumbbell object at the height.
4. The participant removes the HMD and answers to the
questionnaires.
5. Repeat steps 1 to 4 for each condition.
6. Set steps 1 to 5 as one set and carry out 4 sets in total.
4. RESULT
For each question item, each result from ‘‘Strongly disagree’
to ‘‘Strongly agree’’ was corresponded from 1 to 7 points as a
rating point. The results of each question item are described
in order below.
In the question ‘‘Did you feel the weight?’, there is a
tendency that participants felt heavier with the delay and
complex conditions than the control condition, and the
dispersion of the evaluation tended to be relatively large
particularly in the delay condition. A one-way ANOVA
analysis was conducted on the rating points with the
conditions as factors and a significant difference was found
(F (2, 237) =57.19,p<0.001). Therefore, TukeyHSD
test was processed as a subordinate test, and a significant
difference was found between both control–delay condition
and control–complex condition with p<0.001 (Fig. 4).
In the interview, 14 of the 20 participants mentioned the
difference in a weight feeling between the delay condition
and the complex condition. According to the comments, 7
participants felt heavier with complex condition, 6 felt about
the same weight between the conditions and 1 participant felt
heavier with delay condition.
In the question ‘‘Did you feel a sense of strong gripping?’’,
the feeling of strong gripping tended to increase, followed
J. Imaging Sci. Technol. 060402-4 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
Figure 3. The experiment system.
Figure 4. Boxplot of the result of the questionnaire on a sense of
weight; boxplots show the median, 25th and 75th percentiles; error bar
indicates minima and maxima; cross indicates mean; ***: p<0.001,
** p<0.01, *: p<0.05, :p<0.1.
in order by control, delay and complex conditions. Similarly,
a one-way ANOVA analysis was conducted with conditions
as a factor, and a significant difference was observed (F
(2,237) =59.48,p<0.001). Thus, a TukeyHSD test was
processed as a subordinate test and as a result, a significant
difference was found with p<0.001 among all pairs of
the conditions (Fig. 5). In the interview, many participants
commented on the influence on the sense of strong gripping
especially under the complex condition and 17 of the 20
participants commented on the relationship between a sense
of strong gripping and augmented cross-modal expressions.
15 of the 17 participants responded that the sense of strong
gripping was enhanced by the expressions and in particular
Figure 5. Boxplot of the result of the questionnaire on a sense of
strain; boxplots show the median, 25th and 75th percentiles; error bar
indicates minima and maxima; cross indicates mean; ***: p<0.001,
** p<0.01, *: p<0.05, :p<0.1.
2 participants commented on hand tremor, 1 participant on
heartbeat, 2 participants on both the change of skin color
and hand tremor and 10 participants on entire augmented
cross-modal expressions.
In the question ‘‘Did you feel that you lifted the
dumbbell?’’, The rating point tended to be the highest
in the control condition and be relatively higher in the
complex condition compared with the delay condition.
Analyzed in the same way, a significant difference was found
by analysis of variance (F(2, 237) =19.12,p<0.001),
and a significant difference was found with p<0.001
between both control–delay condition and control–complex
condition by TukeyHSD test (Fig. 6). In the interview,
most of the participants commented that they felt the act
J. Imaging Sci. Technol. 060402-5 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
Figure 6. Boxplot of the result of the questionnaire on a sense of lifting a
dumbbell; boxplots show the median, 25th and 75th percentiles; error bar
indicates minima and maxima; cross indicates mean; ***: p<0.001,
** p<0.01, *: p<0.05, :p<0.1.
of lifting was real because the dumbbell object and hand
model completely followed the movement of their body and
they could move them as desired in the control condition.
In addition, as comments on the delayed expression, 6
participants commented that the sense of lifting and reality
were lost because they felt a sense of incompatibility on the
situation that the dumbbell object and hand model did not
follow immediately and they could not move them as they
desired. Furthermore, 5 participants commented that they
felt the sense of manipulating or pulling something rather
than lifting. On the other hand, as comments on the complex
condition, 7 participants commented that they felt they were
actually lifting a dumbbell by the augmented cross-modal
expressions or the expressions match the experience of lifting
a heavy object. In addition, 2 participants commented that
they felt something that is not a part of their body is moving.
In the question ‘Were you aware of your actual body
during the operation?’, the rating point tended to be high
in delay condition and complex condition. Analyzed in the
same way, a significant difference was observed by analysis
of variance (F (2,237) =4.96,p<0.01), and as a result
of TukeyHSD test, a significant difference was found with
p<0.01 between control–delay condition and a significant
trend with p<0.1between control–complex condition
(Fig. 7). In the interview, 9 participants commented that they
felt discomfort due to the awareness of the separation or
gap between the actual body and the hand model by the
delay expression. Additionally, there were comments on the
augmented cross-modal expressions that their actual body
is being conscious of due to the awareness of the difference
between their actual heartbeat and the presented heartbeat,
or that there was a feeling that the hand model is not a part of
their own body due to expressions. On the other hand, also
there were positive comments on the expressions that a sense
of immersion to the contents increased, participants were less
Figure 7. Boxplot of the result of the questionnaire on consciousness
of actual body; boxplots show the median, 25th and 75th percentiles;
error bar indicates minima and maxima; cross indicates mean; ***:
p<0.001, ** p<0.01, *: p<0.05, :p<0.1.
conscious of the gap and so they did not feel discomfort due
to the gap.
About the force sensor data, there was a tendency to grip
the controller more strongly, followed in order by the control
condition, the delay condition and the complex condition.
Furthermore, especially under the complex condition, it
seems that participants grabbed not only the trigger part
but also the whole hand. Because data of two participants
were partially missing due to communication failure between
ADC and PC, the statistical processing was carried out
with data of 18 people whose all data were accurately
acquired. As a processing of raw data, the digital data was
converted to force (kgf) by Eqs. (2)–(4) and the average
value for 10 seconds during proceeding lifting task was
calculated for each trial. Analysis of variance was similarly
conducted in each of the trigger part and the grip part. As a
result, a significant difference was found in the trigger part
(F (2,230) =8.34,p<0.001) and in the grip part (F (2,230)
=3.21,p<0.05). Thus, TukeyHSD test was processed and as
a result, a significant difference was found between both the
control–delay condition and the control–complex condition
in the trigger part with p<0.01 (Fig. 8) and between the
control–complex condition in the grip part with p<0.05
(Fig. 9).
As other comments of interviews, there was a comment
that the enjoyment and the reality are improved or there
was a feeling that the sense on actual body is changed by
augmented cross-modal expressions.
5. DISCUSSION
First of all, with regard to weight sensation, it was confirmed
that the sense of weight was significantly increased through
the delayed expression by the result of the rating scales,
and from the comments, the tendency is observed that
the sense of weight is further increased by the augmented
cross-modal expressions. In addition, in this experiment, the
J. Imaging Sci. Technol. 060402-6 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
Figure 8. Boxplot of the result of the force sensor of trigger part; boxplots
show the median, 25th and 75th percentiles; error bar indicates minima
and maxima; cross indicates mean; ***: p<0.001, ** p<0.01, *:
p<0.05, :p<0.1.
Figure 9. Boxplot of the result of the force sensor of grip part; boxplots
show the median, 25th and 75th percentiles; error bar indicates minima
and maxima; cross indicates mean; ***: p<0.001, ** p<0.01, *:
p<0.05, :p<0.1.
degree of delay was large and individual differences were
observed in the occurrence of weight sensation by delay
expression. Moreover, there were participants who hardly
felt the weight sense. These individual differences with large
delay expression were also observed in our preliminary
experiment. However, from the comments, the individual
differences tended to be eased by augmented cross-modal
expressions.
As one of the factors of the individual differences
with the delay expression, it is considered that due to
the expression in which the positions or behaviors of the
actual body and the virtual body are extremely separated,
some participants were strongly conscious of the separation.
Participants said, ‘‘Feeling of something coming along with
my movement,’’ ‘‘Feeling just slow against my movement’’
or ‘‘A sense of ownership to the hand model disappears’
with regard to the incompatibility on the delay expression.
In addition, it can be also one of the factors that the delay
expressions were understood in different contexts from the
sense of weight or lifting, such as a sense of slow, pulling or
manipulating an object.
On the other hand, the reason why individual differences
or incompatibility with the delay expression tended to
be eased in the complex condition was considered as
follows. The augmented cross-modal expressions increased
the participants’ sense of immersion and concentration in
the content, and it is assumed that the sense of separation
between the virtual and real body was relatively unconscious.
In addition, participants commented that they felt the
augmented cross-modal expressions matched the experience
of lifting a heavy object and the haptic experience was real.
Therefore, it is considered that the experience in VR got
closer to the experience of lifting a heavy object with the
augmented cross-modal expressions than only with delay
expressions.
This tendency is also found in the result of the rating
scales of ‘‘Were you aware of your actual body during the
operation?’’. However, there were some participants who felt
uncomfortable in the augmented cross-modal expressions
and not all participants were able to feel the sense of
weight even under the complex condition. Therefore, further
consideration is necessary for expressions in order to present
weight sense to all users in common.
In addition, with regard to the force of hand and a sense
of strong gripping under the experience, the questionnaire
of rating scales resulted in a significantly higher score,
followed in the order by the control, the delay and the
complex condition (Fig. 5). The trend can be seen also in
the objective index. The value of force sensor of trigger part
is significantly larger in delay and complex conditions than
control condition, and that of grip part is significantly larger
in complex condition than in control condition (figures 8,
9). Particularly with regard to the augmented cross-modal
expressions, there are the comments that most participants
feel an enhancement of a sense of strong gripping by the
expressions, and at the same time, the comments that ‘‘Fun,’
‘‘Reality is improved,’’ ‘‘Feeling like a hand getting hot’’ or
‘‘Get a feeling of tension.’’ Therefore, it was suggested that
the quality of experiences was changed and enhanced by
the augmented cross-modal expressions. Furthermore, it was
also considered that the augmented cross-modality has a
physical or behavioral influence such as the way to grip
and the strength of gripping because the value of the force
sensor of the grip part was significantly high in the complex
condition.
6. CONCLUSION
In this paper, we proposed a method of enhancing and
changing the quality of total haptic experiences as intended
in a VR space by translating physiological responses, knowl-
edge and impressions related to the targeted experience
into audio-visual expressions, as a method of presenting
cross-modal haptic illusion. The conventional pseudo-haptic
method of generating a sense of weight uses delay expression
while a user is lifting a virtual object. In addition to that,
our method presented the expressions such as ‘‘Visual hand
J. Imaging Sci. Technol. 060402-7 Nov.-Dec. 2018
Hirao and Kawai: Cross-modal approach using physiological responses, knowledge and impression
trembling,’’ ‘‘Skin color of a hand model turning red’’ and
‘‘Increasing of the heart rate and the volume of heartbeat’
as augmented cross-modal expressions. This method was
evaluated by the experiment of comparing the condition
using only delay expression (delay condition), the condition
using delay expression with the augmented cross-modal
expressions (complex condition) and the condition without
these expressions (control condition).
Based on the results of the experiment, it is confirmed
that due to the augmented cross-modal expressions of
strain, most participants experienced an enhancement of the
sense of strong gripping. In addition, several participants
commented that this enhancement leads to change the
quality of participants’ total haptic experiences in VR and
makes it closer to the experience of lifting a heavy object
more than only delay. Furthermore, due to the result that
the expression of strain increased the gripping force, it
is suggested that the actual physiological responses and
behavior could be induced by augmented cross-modal
expressions.
We consider that the effectiveness of the proposed
method was confirmed at a certain level. In the future,
we will explore other augmented cross-modal expressions
presenting haptic experiences other than weight sense with
large scalability in VR.
ACKNOWLEDGMENT
This research was partially supported by the Japan Society
for the Promotion of Science, Grant-in-Aid for Scientific
Research (C) 2018–2020 (ID18995328, Takashi Kawai).
REFERENCES
1M. A. Srinivasan and C. Basdogan, ‘‘Haptics in virtual environments:
taxonomy, research status, and challenges,’’ Comput. Graph. 21, 393–404
(1997).
2T. H. Massie and J. K. Salisbury, ‘‘PHANToM haptic interface: a device for
probing virtual objects,’’ Proc. ASME Winter Annual Meeting Symposium
on Haptic Interfaces for Virtual Environment and Teleoperator Systems
(American Society of Mechanical Engineers, New York, NY, 1994), 55-1,
pp. 295–299.
3M. Sato, ‘‘SPIDAR and virtual reality,’’ Proc. 5th Biannual World
Automation Congress (IEEE, Piscataway, NJ, 2002), pp. 17–23.
4CyberGlove Systems Inc., ‘CyberGrasp’, http://www.cyberglovesystems.
com/cybergrasp (accessed 2018).
5S. Nagawara, H. Kajimoto, N. Kawakami, S. Tachi, and I. Kawabuchi,
‘‘An encounter-type multi fingered master hand using circuitous joints,’’
Proc. IEEE Int’l. Conf. on Robotics and Automation (IEEE, Piscataway, NJ,
2005), pp. 2667–2672.
6A. Lécuyer, ‘‘Simulating haptic feedback using vision: A survey of
research and applications of pseudo-haptic feedback,’Presence: Teleope
rators and Virtual Environments 18, 39–53 (2009).
7Y. Taira, Y. Ban, T. Narumi, T. Tanikawa, and M. Hirose, ‘‘Controlling
fatigue while lifting objects using Pseudo-haptics in a mixed reality space,’’
2014 IEEE Haptics (HAPTICS) (IEEE, Piscataway, NJ, 2014), pp. 175–180.
8M. Rietzler, F. Geiselhart, J. Gugenheimer, and E. Rukzio, ‘‘Breaking the
tracking: enabling weight perception using perceivable tracking offsets,
Proc. 2018 CHI Conf. on Human Factors in Computing Systems (ACM,
New York, NY, 2018).
9M. Rietzler, F. Geiselhart, J. Frommel, and E. Rukzio, ‘‘Conveying the
perception of kinesthetic feedback in virtual reality using state-of-the-art
hardware,’’ Proc. 2018 CHI Conf. on Human Factors in Computing Systems
(ACM, New York, NY, 2018).
10 P. Issartel, F. Gueniat, S. Coquillart, and M. Ammi, ‘‘Perceiving mass in
mixed reality through pseudo-haptic rendering of Newton’s third law,’
IEEE Virtual Reality Conf., VR 2015 – Proceedings (IEEE, Piscataway, NJ,
2015), pp. 41–46.
11 S. Hashiguchi, Y. Sano, F. Shibata, and A. Kimura, ‘‘R-V Dynamics
illusion: psychophysical influence on sense of weight by mixed-reality
visual stimulation of moving objects,’’ VAMR 2014: Virtual, Augmented
and Mixed Reality, Designing and Developing Virtual and Augmented
Environments (Springer, Cham, New York, NY, 2014), pp. 55–64.
12 A. Takahashi, K. Tanabe, and H. Kajimoto, ‘‘Relationship between force
sensation and stimulationparameters in tendon electrical stimulation,’’
AsiaHaptics’16 (Springer, Singapore, 2016).
13 K. Tanabe, S. Takei,and H. Kajimoto, ‘‘The whole hand haptic glove using
numerous linear resonantactuators,’IEEE World Haptics Conference2015
(IEEE, Piscataway, NJ, 2015).
14 M. Azmandian, M. Hancock, H. Benko, E. Ofek, and A. D. Wilson,
‘‘Haptic retargeting: dynamic repurposing of passive hapticsfor enhanced
virtual reality,Proc. 2016 CHI Conf. on Human Factors in Computing
Systems (ACM, New York, NY, 2016), pp. 1968–1979.
15 K. Matsumoto, Y. Ban, T. Narumi, Y. Ynase, T. Tanikawa, and M. Hirose,
‘‘Unlimited corridor: redirected walking techniques using visuo haptic
interaction,ACM SIGGRAPH 2016 Emerging Technologies (ACM, New
York, NY, 2016).
16 R. Nagao, K. Matsumoto, T. Narumi, T. Tanikawa, and M. Hirose, ‘‘As-
cending and descending in virtual reality: simple and safe system using
passive haptics,IEEE Trans. Vis. Comput. Graphics 24, 1584–1593
(2018).
17 C. M. Davis and W. Roberts, ‘‘Lifting movements in the size-weight
illusion,Attention Perception & Psychophysics 20, 33–36 (1976).
18 G. Buckingham, N. S. Ranger, and M. A. Goodale, ‘‘The material-weight
illusion induced by expectations alone,’’ Attention Perception & Psy-
chophysics 73, 36–41 (2011).
19 R. R. Ellis and S. J. Lederman, ‘‘The golf-ball illusion; evidence for
top-down processing in weight perception,Perception-London 27,
193–303 (1998).
J. Imaging Sci. Technol. 060402-8 Nov.-Dec. 2018
©2018 Society for Imaging Science and Technology (IS&T)
All rights reserved. This paper, or parts thereof, may not be reproduced in any form
without the written permission of IS&T, the sole copyright owner of
The Journal of Imaging Science and Technology.
For information on reprints or reproduction contact
Donna Smith
Production Editor
The Journal of Imaging Science and Technology
Society for Imaging Science and Technology
7003 Kilworth Lane
Springfield, Virginia 22151 USA
703/642-9090 extension 107
703/642-9094 (fax)
dsmith@imaging.org
www.imaging.org
... C/D ratio [7], [115], [126] [137]- [145] Includes approaches that adjusting the translation and rotation displacement of the object, or the movement offset between virtual hand and actual hand to create an illusion of weight perception Velocity [128], [138], [142], [146]- [149] Includes approaches that alter the moving speed of the object/hand to create the illusion of weight Passive Force [86], [137], [150]- [152] Includes approaches where trigger the user to generate and control the force feedback himself or herself by increasing or decreasing the force exerted on the static interface to give an impression of weight Audio-Visual and Others [83], [146], [148], [149], [152]- [155] Includes approaches that changing the audio-visual representation of the object and others such as manipulating the avatar posture or body gesture, measuring the finger distance, or displaying the liquid movement to create an illusion of weight The visual representation of the pseudo-haptic techniques discussed in this section is shown in Figure 8, grouped by the pseudo-haptic cues. ...
... C/D ratio [7], [115], [126] [137]- [145] Includes approaches that adjusting the translation and rotation displacement of the object, or the movement offset between virtual hand and actual hand to create an illusion of weight perception Velocity [128], [138], [142], [146]- [149] Includes approaches that alter the moving speed of the object/hand to create the illusion of weight Passive Force [86], [137], [150]- [152] Includes approaches where trigger the user to generate and control the force feedback himself or herself by increasing or decreasing the force exerted on the static interface to give an impression of weight Audio-Visual and Others [83], [146], [148], [149], [152]- [155] Includes approaches that changing the audio-visual representation of the object and others such as manipulating the avatar posture or body gesture, measuring the finger distance, or displaying the liquid movement to create an illusion of weight The visual representation of the pseudo-haptic techniques discussed in this section is shown in Figure 8, grouped by the pseudo-haptic cues. ...
... Humans always relate the moving speed of an object to its perceived weight, and the faster an object moves, the lighter they seem. Earlier studies [128], [146] have seen to use the visually represented speed of an object in combination with other stimuli to influence the perception of weight in VR. In Rosa et al. [128] experiments, with the HMD on, the participants were shown with an avatar sitting on a chair with the arm on the table while wearing a vibrotactile sleeve, the simulated falling speed of a virtual object onto the arm has changed the perceived weight of the object and the intensity of the vibration felt on the skin. ...
Article
Full-text available
Weight is perceived through the combination of multiple sensory systems, and many factors can influence the perception of heaviness, including touch, visual, and force senses. There have been significant advancements in the development of haptic interfaces throughout the years. However, many challenges limit the progression to enable humans to sense the weight in virtual reality (VR). This article presents an overview of the factors that influence how weight is perceived and the phenomenon that contributes to various types of weight illusions. A systematic review has been undertaken to study the development of weight perception in VR, underlying haptic technology that renders the mass of a virtual object, and the creation of weight perception through pseudo-haptic. We summarize the approaches from the perspective of haptic and pseudo-haptic cues that exhibit the sense of weight such as force, skin deformation, vibration, inertia, control-display (C/D) ratio, velocity, body gestures, and audio-visual representation. The design challenges are highlighted, and research gaps are discussed, including accuracy and precision, light weight discrimination, heavy weight rendering, and absolute weight simulation.
... Dans le cas contraire, l'information perçue n'est alors que partielle. Cette interface correspond à une interface de rendu réaliste où plusieurs modalités permettent de recréer des sensations réelles [98]. C'est ce que propose la combinaison de plusieurs modalités tactiles et visuelles pour la discrimination de matériau en environnement téléopéré. ...
... L'utilisation des perceptions intermodales peut aussi remplacer des sensations absentes. Hirao et al. en font la démonstration en créant l'illusion de l'effort à fournir pour déplacer un objet lourd sans avoir recours à un dispositif haptique [98]. Pour cela, ils reproduisent un ensemble de sensations éprouvées par un utilisateur dans le monde réel à l'exception des sensations haptiques. ...
Thesis
Cette thèse montre que « l’interaction directe d’un opérateur au cours de la mise en œuvre du procédé industriel de fabrication additive pour améliorer la fabrication est possible ». Aujourd’hui, la mise en œuvre de la Chaîne Numérique ne permet pas de réagir naturellement à l’apparition d’un aléa de fabrication. Or, il apparaît que la fabrication à la demande de produits engendre souvent des aléas durant le processus. La première section du mémoire est consacrée à un état de l’art sur la Fabrication Interactive à partir duquel nous proposons une boucle d’interaction Humain-Robot. Cette boucle permet l’interruption du processus automatique pour apporter une modification locale, directement depuis la cellule de fabrication. Un des verrous à lever est de permettre la perception des valeurs des critères de la fabrication par l’opérateur en temps réel et en mobilité. Pour cela, nous avons exploré les retours vibrotactiles. Ainsi, dans la seconde section du mémoire est présentée TacTool, une application web conçue et réalisée afin d’aider la conception de systèmes vibrotactiles. A partir d’une combinaison de 21 caractéristiques, sélectionnées interactivement et par catégories, l’utilisateur se voit proposer des recommandations sur la conception du dispositif de communication vibrotactile caractérisé. Ces recommandations sont sélectionnées dans un ensemble de 181 recommandations issues de la revue de 97 références comprises entre 1960 et 2020. Dans la dernière section, grâce à la conception d’un bracelet vibrotactile appelé ViMA (Vibrotactile Monitoring Armband), nous avons conçu 20 items vibrotactiles, dont nous avons évalué la capacité à être reconnus. Ainsi, cette évaluation et une enquête auprès de 27 experts de la fabrication additive, ont permis de définir 18 symboles vibrotactiles permettant d’informer un opérateur pour la surveillance des procédés automatisés de fabrication additive.
... Due to these limitations, researchers have proposed that in addition to the haptic sensation, people can also perceive information from a virtual environment through other modalities such as audio and visual cues (Kataoka 2018). For example, Hirao and Kawai (2019) have implemented audio and visual messages in VR to lead users to develop a sense of weight while lifting virtual objects. Thus, both audio and visual cues were added to VR scenarios to raise the sensation of heavy lifting. ...
Article
Full-text available
In recent years, research has found that people have stable predispositions to engage in certain behavioural patterns to work safely or unsafely, which vary among individuals as a function of their personality features. In this regard, an innovative machine learning model has been recently developed to predict workers’ behavioural tendency based on personality factors. This paper presents an empirical evaluation of the model’s prediction performance (i.e. the degree to which the model can generate similar results compared to reality) to address the issue of the model’s usability before it is implemented in real situations. As virtual reality allows a good grip on fidelity resembling real-world situations, it can stimulate more natural behaviour responses from participants to increase ecological validity of experimental results. Thus, we implemented a virtual reality experimentation environment to assess workers’ safety behaviour. The model’s prediction capability was then evaluated by comparing the model prediction results and workers’ safety behaviour as assessed in virtual reality. The comparison results showed that the model predictions on two dimensions of workers’ safety behaviour (i.e. task and contextual performance) were in good agreement with the virtual reality experimental results, with Spearman correlation coefficients of 79.7% and 87.8%, respectively. The machine learning model thus proved to have good prediction capability, which allows the model to help identify vulnerable workers who are prone to undertake unsafe behaviours. The findings also suggest that virtual reality is a promising method for measuring workers’ safety behaviour as it can provide a realistic and safe environment for experimentation.
... These weight illusions can be further strengthened through modification of the User Representation's visual appearance and based on the inclusion of biofeedback. In a system from Hirao and Kawai (2018), users are provided with visual feedback of their heart rate, and the virtual hands are shown trembling and turning red with the aim of reducing fatigue during the lifting of objects. ...
Article
Full-text available
Cursors, avatars, virtual hands or tools, and other rendered graphical objects, enable users to interact with computers such as PCs, game consoles or virtual reality systems. We analyze the role of these various objects from a user perspective under the unifying concept of “User Representations”. These representations are virtual objects that artificially extend the users’ physical bodies, enabling them to manipulate the virtual environment by performing motor actions that are continuously mapped to their User Representations. In this paper, we identify a set of concepts that are relevant for different User Representations, and provide a multidisciplinary review of the multisensory and cognitive factors underlying the control and subjective experience of User Representations. These concepts include visual appearance, multimodal feedback, sense of agency, input methods, peripersonal space, visual perspective, and body ownership. We further suggest a research agenda for these concepts, which can lead the human computer interaction community towards a wider perspective of how users perceive and interact through their User Representations.
Conference Paper
Full-text available
Virtual reality (VR) technology strives to enable a highly immersive experience for the user by including a wide variety of modalities (e.g. visuals, haptics). Current VR hardware however lacks a sufficient way of communicating the perception of weight of an object, resulting in scenarios where users can not distinguish between lifting a bowling ball or a feather. We propose a solely software based approach of simulating weight in VR by deliberately using perceivable tracking offsets. These tracking offsets nudge users to lift their arm higher and result in a visual and haptic perception of weight. We conducted two user studies showing that participants intuitively associated them with the sensation of weight and accept them as part of the virtual world. We further show that compared to no weight simulation, our approach led to significantly higher levels of presence, immersion and enjoyment. Finally, we report perceptional thresholds and offset boundaries as design guidelines for practitioners.
Article
Full-text available
Haptic displays are emerging as effective interaction aids for improving the realism of virtual worlds. Being able to touch, feel, and manipulate objects in virtual environments has a large number of exciting applications. The underlying technology, both in terms of electromechanical hardware and computer software, is becoming mature and has opened up novel and interesting research areas. In this paper, we clarify the terminology of human and machine haptics and provide a brief overview of the progress recently achieved in these fields, based on our investigations as well as other studies. We describe the major advances in a new discipline, Computer Haptics (analogous to computer graphics), that is concerned with the techniques and processes associated with generating and displaying haptic stimuli to the human user. We also summarize the issues and some of our results in integrating haptics into multimodal and distributed virtual environments, and speculate on the challenges for the future.
Article
Full-text available
The occurrence of the size-weight illusion is related to the manner in which-objects are lifted. When the SWI occurs, the larger of two objects of equal objective weight is usually lifted with greater acceleration, deceleration, and maximum velocity than the smaller one, but to approximately the same height. These differences are not present when the cans feel equally heavy. The relationship of lifting movements to judgments is consistent with the known behavior of proprioceptors which provide sensory input about muscular and movement events.
Article
Full-text available
ABSTRACT This paper presents a survey of the main results obtained in the field of “pseudo-haptic feedback”: a technique meant to simulate haptic sensations in virtual environments using visual feedback and properties of human visuo-haptic perception. Pseudo-haptic feedback uses vision to distort haptic perception and verges on haptic illusions. Pseudo-haptic feedback has been used to simulate various haptic properties such as the stiffness of a virtual spring, the texture of an image, or the mass of a virtual object. This paper describes the several experiments inwhich,these haptic properties were simulated. It assesses the definition and the properties of pseudo-hapticfeedback. It also describes several virtual reality applications in which pseudo-haptic feedback has been successfully implemented, such as a virtual environment for vocational training of milling machine operations, or a medical simulator for training in regional anesthesia procedures. Pseudo-Haptic Feedback 3
Conference Paper
Including haptic feedback in current consumer VR applications is frequently challenging, since technical possibilities to create haptic feedback in consumer-grade VR are limited. While most systems include and make use of the possibility to create tactile feedback through vibration, kinesthetic feedback systems almost exclusively rely on external mechanical hardware to induce actual sensations so far. In this paper, we describe an approach to create a feeling of such sensations by using unmodified off-the-shelf hardware and a software solution for a multi-modal pseudo-haptics approach. We first explore this design space by applying user-elicited methods, and afterwards evaluate our refined solution in a user study. The results show that it is indeed possible to communicate kinesthetic feedback by visual and tactile cues only and even induce its perception. While visual clipping was generally unappreciated, our approach led to significant increases of enjoyment and presence.
Article
This paper presents a novel interactive system that provides users with virtual reality (VR) experiences, wherein users feel as if they are ascending/descending stairs through passive haptic feedback. The passive haptic stimuli are provided by small bumps under the feet of users; these stimuli are provided to represent the edges of the stairs in the virtual environment. The visual stimuli of the stairs and shoes, provided by head-mounted displays, evoke a visuo-haptic interaction that modifies a user's perception of the floor shape. Our system enables users to experience all types of stairs, such as half-turn and spiral stairs, in a VR setting. We conducted a preliminary user study and two experiments to evaluate the proposed technique. The preliminary user study investigated the effectiveness of the basic idea associated with the proposed technique for the case of a user ascending stairs. The results demonstrated that the passive haptic feedback produced by the small bumps enhanced the user's feeling of presence and sense of ascending. We subsequently performed an experiment to investigate an improved viewpoint manipulation method and the interaction of the manipulation and haptics for both the ascending and descending cases. The experimental results demonstrated that the participants had a feeling of presence and felt a steep stair gradient under the condition of haptic feedback and viewpoint manipulation based on the characteristics of actual stair walking data. However, these results also indicated that the proposed system may not be as effective in providing a sense of descending stairs without an optimization of the haptic stimuli. We then redesigned the shape of the small bumps, and evaluated the design in a second experiment. The results indicated that the best shape to present haptic stimuli is a right triangle cross section in both the ascending and descending cases. Although it is necessary to install small protrusions in the determined direction, by using this optimized shape the users feeling of presence of the stairs and the sensation of walking up and down was enhanced.
Conference Paper
Most haptic devices have the common problem of requiring a large hardware setup, because they must present actual force. To cope with this issue, we have proposed a method to present force sensation using tendon electrical stimulation. In this paper, we investigated whether it is possible to present force sensation by electrically stimulating the tendon through the skin surface at the wrist. We also investigated the relationship between the amount of sensation and the stimulation parameters. We found that a force sensation can be generated by electrical stimulation to the wrist, and the direction of the force sensation is opposite to the motion elicited by muscle electrical stimulation. We also found that it is possible to control the amount of the sensation by varying both pulse height and pulse frequency.
Conference Paper
Manipulating a virtual object with appropriate passive haptic cues provides a satisfying sense of presence in virtual reality. However, scaling such experiences to support multiple virtual objects is a challenge as each one needs to be accompanied with a precisely-located haptic proxy object. We propose a solution that overcomes this limitation by hacking human perception. We have created a framework for repurposing passive haptics, called haptic retargeting, that leverages the dominance of vision when our senses conflict. With haptic retargeting, a single physical prop can provide passive haptics for multiple virtual objects. We introduce three approaches for dynamically aligning physical and virtual objects: world manipulation, body manipulation and a hybrid technique which combines both world and body manipulation. Our study results indicate that all our haptic retargeting techniques improve the sense of presence when compared to typical wand-based 3D control of virtual objects. Furthermore, our hybrid haptic retargeting achieved the highest satisfaction and presence scores while limiting the visible side-effects during interaction.
Conference Paper
This paper describes the PHANTOM haptic interface - a device which measures a user's finger tip position and exerts a precisely controlled force vector on the finger tip. The device has enabled users to interact with and feel a wide variety of virtual objects and will be used for control of remote manipulators. This paper discusses the design rationale, novel kinematics and mechanics of the PHANTOM. A brief description of the programming of basic shape elements and contact interactions is also given.
Conference Paper
We have developed a new type of master hand to control a dexterous slave robot hand for telexistence. Our master hand has two features. One is a compact exoskeleton mechanism called “circuitous joint,” which covers wide workspace of an operator’s finger. Another is the encounter-type force feedback that enables unconstrained motion of the operator’s finger and natural contact sensation. In this paper, the mechanism and control method of our master hand are introduced and experimental master-slave finger control is conducted.