Conference PaperPDF Available

Exploring Rock Climbing in Mixed Reality Environments


Abstract and Figures

While current consumer virtual reality headsets can convey a strong feeling of immersion, one drawback is still the missing haptic feedback when interacting with virtual objects. In this work, we investigate the use of a artificial climbing wall as a haptic feedback device in a virtual rock climbing environment. It enables the users to wear a head-mounted display and actually climb on the physical climbing wall which conveys the feeling of climbing on a large mountain face.
Content may be subject to copyright.
Exploring Rock Climbing in Mixed
Reality Environments
Felix Kosmalla
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
Florian Daiber
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
André Zenner
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
Nico Herbig
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
Marco Speicher
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
Antonio Krüger
German Research Center for
Artificial Intelligence (DFKI)
Saarland Informatics Campus
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHI’17 Extended Abstracts, May 06-11, 2017, Denver, CO, USA
ACM 978-1-4503-4656-6/17/05.
While current consumer virtual reality headsets can con-
vey a strong feeling of immersion, one drawback is still
the missing haptic feedback when interacting with virtual
objects. In this work, we investigate the use of a artificial
climbing wall as a haptic feedback device in a virtual rock
climbing environment. It enables the users to wear a head-
mounted display and actually climb on the physical climbing
wall which conveys the feeling of climbing on a large moun-
tain face.
Author Keywords
Virtual Reality; Passive Haptic Feedback; Mixed Reality;
Rock Climbing.
ACM Classification Keywords
H.5.1 [Information interfaces and presentation (e.g., HCI)]:
Multimedia Information Systems - Artificial, augmented, and
virtual realities; H.5.2 [Information interfaces and presenta-
tion (e.g., HCI)]: User Interfaces - Haptic I/O
Rock climbing exists as a sport since the 1890s and was
founded in the Saxon Switzerland region [5]. Since then
climbing has undergone a broad development from a dan-
gerous extreme sport to an easily accessible leisure activity
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
with 25 million people climbing regularly 1. Climbing gyms
with artificial walls are on the rise and can be found in ev-
ery major city. They offer the climber a safe opportunity for
some hours of after-work climbing without much prepara-
tion. The walls are equipped with holds of different shapes
and colors, some being as easy to grab as a rung of a lad-
der, others which only fit a single finger. This allows the
creators of climbing routes to set routes for different levels
of difficulty. Lately, artificial rock climbing was extended by
computer systems which aim to enhance the experience
of the climber. This includes augmenting artificial climb-
ing walls with projections that allow for interactive games
or training assistance [4, 14]. Rock climbing also found its
place in computer games ranging from simple smart-phone
games to more sophisticated virtual reality (VR) experi-
ences like The Climb2, which enables the climber to explore
remote areas with a scenic view in an immersive virtual en-
vironment (VE) through a head-mounted display.
The increasing power and decreasing price of powerful
graphics cards, display technology, and tracking solutions
made immersive VR available and affordable for the masses
and thus paves the way for new game concepts and novel
types of applications and interactions. However, the degree
of immersion is determined by the expression of the three
feedback dimensions: visual, auditory and haptic. While
today, the visual and auditory feedback are already very so-
phisticated, realistic haptic feedback is still one of the next
big challenges for VR and a very active field of research. In-
sko et al. [3] presented a concept of passive haptics, a low-
cost approach which can provide realistic haptic impres-
sions. Here, physical objects in the user’s real environment,
called proxies, are spatially registered with virtual counter-
parts and provide natural haptic feedback when the user
Figure 1: Visualization of the virtual reality climbing system. The
integrated field camera of the HTC Vive was used to capture the
view that the climber would have seen.
touches them. This feedback can drastically increase the
user’s sense of presence and enhance spatial learning [3].
While The Climb comes closest to climbing in VR, it is lack-
ing exertion and haptic feedback. In this work we explore
rock climbing in VR and discuss potential applications.
For this, we developed a prototype of a VR climbing sys-
tem, using a 3x4 m2artificial climbing wall and a HTC Vive
equipped with a Leap Motion for hand tracking. We propose
to use the artificial climbing wall as a haptic proxy for a vir-
tual rock, which enables the user to climb in a VE while still
having both realistic haptic feedback and exertion (see Fig-
ure 1). To allow bystanders to participate and observe the
climber in the VE, we additionally integrated a projection-
based mixed reality feature. Therefore, the virtual rock en-
vironment is projected onto the physical climbing wall (see
Figure 6).
Our system facilitates the creation of several applications:
training for dangerous situations like falling rocks or sud-
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
den appearances of wildlife in a safe environment or giving
climbers with a disability or those who suffer from altitude
sickness the feeling of climbing outside while climbing an
easier route inside. Another application would be to map
different VEs to a single climbing wall, which would be es-
pecially useful for climbing treadmills, which present the
climber with the same holds every couple meters of climb-
With this work, we contribute a) a VR climbing system that
provides haptic feedback and exertion, b) a collaborative
projection-based mixed reality experience, and c) an easy
calibration method for incorporating a virtual model of a
physical scene into a virtual environment to allow for realis-
tic haptic interaction.
Related Work
The Reality - Virtuality Continuum
Milgram and Colquhoun [9] introduced the Reality - Virtual-
ity continuum, a taxonomy that classifies systems based on
their real and virtual aspects. It spans a continuous space
of Mixed Reality between the two poles Reality and Virtu-
ality. The experience we introduce here can be classified
as Augmented Virtuality, as the user’s stimulation is primar-
ily virtual (visuals & audio), augmented with passive haptic
Passive Haptic Feedback in VR
The concept of passive haptic feedback is an established
way of introducing haptic feedback to immersive VEs [1,
3, 7, 8, 11]. Here, proxies (i.e. physical counterparts rep-
resenting virtual objects) are used to provide tactile and
kinesthetic feedback. Often, proxies are low-fidelity props
made out of cheap and available material such as styro-
foam, cardboard, or wood. However, concepts that utilize
existing objects in the real surrounding as proxies, exist
as well, e.g. the concept of Substitutional Reality [11].
Through spatial registration with virtual counterparts, these
proxies provide tangibility for objects in the VE. Insko [3]
showed that passive haptics can enhance the user’s sense
of presence. We utilize this to enhance climbing on an artifi-
cial climbing wall.
Interactive Climbing Walls
In the past, rock climbing has been investigated in the con-
text of human computer interaction. Liljedahl et al. [6] pre-
sented Digiwall, a climbing wall which featured translucent,
instrumented holds that incorporated LEDs and capacitive
sensors, enabling different interaction models. Ouchi et al.
also used sensor-embedded climbing holds to model play
behavior on an instrumented climbing wall [10]. A similar
system that augments the climbing wall by instrumenta-
tion was presented by Fiess and Hundhausen [2]. In their
work they build a climbing wall out of translucent material. A
LED-wall placed behind the climbing wall surface acted as a
display that covered the whole climbing wall. As in the work
of Liljedahl et al. [6], the authors also included capacitive
sensors to allow for interactive climbing games.
While the systems described above rely on heavy instru-
mentation of the climbing wall, including wiring or replac-
ing the surface with translucent material, some research
projects approach the augmentation of climbing walls by
projections from behind the climber. Kajastilla et al. intro-
duced The Augmented Climbing Wall [4], which is a com-
bination of a normal climbing wall, a depth-camera, and a
projector. With this, the authors implemented several in-
teractive games and new ways to highlight custom routes
which are usually indicated by the color of the holds. Sim-
ilar to that, Wiehr et al. proposed a mobile, self-calibrating
camera-projection unit which could be placed in front of an
arbitrary bouldering wall [14]. Besides playing games and
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
creating new routes via an augmented reality smart-phone
app, the system allows for the recording and playback of
in-place video, projected directly on the climbing wall.
In contrast to the direct instrumentation and the augmenta-
tion via projection, the system proposed in this work aug-
ments the climbing wall in three dimensions by utilizing the
possibilities of current VR headsets. The use of VR entails
a lot of opportunities that go beyond illumination of holds or
projections on the climbing wall. The main reasons for that
is that the augmentation is not only limited to the surface of
the climbing wall, but can span areas as far as the eye can
reach. Furthermore, it gives the opportunity to alter the rep-
resentation of the reality by removing elements like holds
from the wall or adding new details like weather conditions,
animals, or falling rocks in the VE.
In the following, we describe the general concept of our
VR climbing system and its theoretical classification. We
extended Milgram and Colquhoun’s [9] Reality - Virtuality
continuum, by introducing the Climbing Reality Continuum
that classifies climbing activities and applications based on
the authenticity of the climbing experience. Figure 2 shows
the resulting two-dimensional design space combining the
Reality - Virtuality continuum and the Climbing Reality con-
The Climbing Reality continuum spans from pure virtual
climbing on one end to classical rock climbing on the op-
posite end. Due to the similarity to classical rock climbing,
artificial climbing walls are located in between both ends,
close to the classical rock climbing end. On the opposite,
games that require the user to use a standard game con-
troller (like a Xbox controller) to climb, are located close to
the virtual climbing end as the interaction strongly abstracts
virtuality (AV)
reality (AR)
The Climb
Rock Climbing
betaCube [14]
The Augmented
Climbing Wall [4]
VR Climbing
The Climb
Figure 2: 2D continuum spanned by the Climbing Reality
Continuum (vertical axis) and the Reality - Virtuality Continuum [9]
(horizontal axis) with some examples. Our VR climbing system is
located in the upper right quadrant.
from classical rock climbing. With regard to the VR locomo-
tion techniques investigated by Usoh et al. [13], this type of
interaction is closely related to push-button-fly. Thus we call
it push-button-climb (e.g. playing The Climb with a game
controller). If the game, however, allows the user to climb by
grabbing virtual grips with motion-tracked controllers, and
moving the arms to virtually pull up her body, the interac-
tion is more related to actual climbing. Due to the similarity
to the VR locomotion technique walking-in-place, we call
this type of interaction climbing-in-place (e.g. playing The
Climb with motion-tracked controllers). Climbing-in-place is
located between push-button-climb and climbing on artificial
climbing walls in our continuum.
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
Previous work and existing concepts can be categorized
in this design space as depicted. The upper right and the
lower left quadrant of the design space are so far left un-
derstudied. While applications that involve a form of vir-
tual climbing in real environments are rather abstract, the
combination of real physical climbing and immersive VEs
promises interesting applications. Thus our goal is to ex-
plore this part. For this, our prototype system has to meet
two main criteria: The user should be immersed in a VE
while having the realistic physical impression of climbing.
To meet these requirements, our system includes a head-
mounted display (HMD) and an artificial climbing wall.
As an optional addition, the system can utilize a projector
placed in front of the climbing wall to generate a projection-
based Augmented Reality experience for bystanders watch-
ing the climber. By projecting the virtual rock, we intend to
help climbers to get familiar with the VE (see [12]) and to
provide a smooth transition into it. In addition, the projection
allows bystanders to observe and guide the climber (see
Figure 6).
Figure 3: To register the physical
wall with its virtual counterpart, the
Vive controllers are used to set the
position of the calibration points.
Figure 4: The physical climbing
wall used in our prototype. The
marked calibration points are
analog to the ones seen in
Figure 5.
Figure 5: 3D Model of the climbing
wall within the virtual environment.
The spheres depict the calibration
The setup of our VR climbing prototype consists of an arti-
ficial climbing wall with 4min width and 3min height that
included an overhanging panel and three volumes (see
Figure 1, lower right image). To catch the climber in case
of a sudden fall, a thick mat covers the floor in front of the
climbing wall. For the VR headset, we opted for the HTC
Vive, since it allows for free movement within a certain area
and high quality position tracking. The HTC Vive system
3consists of the headset itself, two controllers, and two
lighthouses that are needed for tracking. In our setup, we
placed the lighthouses to the right and left of the climbing
wall in approximately 2.5mheight. For the hand tracking,
we mounted a Leap Motion controller on the headset. The
current prototype does not feature a tracking for feet yet. In
the future, the tracking could be integrated by using addi-
tional optical tracking (e.g. OptiTrack) or Vive tracking pucks
which were not available at the time of this work. Unity was
used as development environment which runs on a ZOTAC
NEN Steam Machine.
Virtual Environment
As a first step, we scanned the climbing wall with the help
of a Kinect v1 and the Skanect4software. After cleaning
up the model in Meshlab, we imported the model in a Unity
scene. For our prototype, we used a height map of the Mat-
terhorn that was applied to a terrain. The 3D model of the
climbing wall was placed at the top of the mountain to give
a feeling of height. As an ”entry point“ for the climber, we
added a virtual wooden ledge on which the climber would
find herself when putting on the HMD.
As opposed to common VR games where a registration
in the physical playing environment is not necessary (ex-
cept for the position and orientation of the floor), the cali-
bration of the virtual and the physical environment is crit-
ical for climbing in VR. The positions of the virtual holds
have to match the physical holds exactly, otherwise the
climber would grab into the air or hit the climbing wall with
her hands.
For this, we implemented a simple calibration method: In
the first step we defined four calibration points on the climb-
ing wall that are easily identifiable in both the 3D model and
the physical wall. In our case, we chose the tips of the three
volumes (see Figure 4) and one additional hold. These four
points were then registered in the Unity scene by placing
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
invisible game objects at the respective positions on the 3D
model of the climbing wall. In a second step we used one
of the Vive controllers to register the positions of the cali-
bration points in Vive / physical space. As a result we had
four coordinate pairs. In a last step, we calculated the opti-
mal rotation and translation applied to the Vive space using
single value decomposition to match the model of the climb-
ing wall. We provide a sample Unity project with a manual
on how to create custom virtual environments matching a
physical scene on GitHub5.
As an optional feature we used the Microsoft Room Alive
Toolkit6and the corresponding Unity Plugin7to create a
mixed reality experience. For this, we used a camera pro-
jection unit as in the work by Wiehr et al. [14], which was
placed in front of the climbing wall. The Unity plugin was
used to place a virtual camera in front of the 3D model of
the climbing wall, with intrinsic parameters equivalent to
those of the used projector. Using the same calibration
method as above, we rotated the complete virtual scene
so that it matched up with the model of the climbing wall. A
network connection between the computer connected to the
Vive and a second laptop used for the projection was estab-
lished. It was used to synchronize the game states, so that
the projection would reflect the VE seen by the climber.
Figure 6: Using a projector, the
virtual environment is projected
back on the physical climbing wall
to reflect the interactions of the
player in the VE.
Climbing Game
We presented our prototype at our institution’s Christmas
event. Since we did not expect every user to be a climber,
we implemented a small game in which the player had to
collect as many presents as possible by touching them
with their hands. The player had to step on two marked
foot holds and as soon as she grabbed a third hold with
her hand, the game started. To include the audience, we
used the projected mixed reality as described above (see
Figure 6).
In total, 27 guests tried out the game and from most of
them we received positive feedback. A formal evaluation
will be conducted as part of future work.
Conclusion & Future Work
In this work, we presented a VR climbing prototype that
utilizes a 3D model of an artificial climbing wall and a HMD
equipped with a Leap Motion Controller to allow for physical
climbing in different virtual environments.
Based on this work, a large number of extensions are pos-
sible. Climbing treadmills could be made less monotone by
mapping a virtual environment on the revolving belt to con-
vey the impression of climbing a large mountain face, by
displaying only a subset of holds that are actually mounted
on the treadmill.
A different application could aim at training scenarios where
a 3D representation of a climber could be placed in the VE
to demonstrates a difficult climbing move. As opposed to
2D projection, this would have the advantage of depth per-
ception and no occlusion when climbing the same problem
simultaneously. Other applications could focus on training
for dangerous situations such as the simulation of falling
rocks. This could also be applied to other domains e.g. a
training system for firemen.
For the future, we plan to integrate the tracking of feet,
followed by a formal user study. Albeit we did not experi-
ence any tangling of the wires of the HMD by the climbers,
counter measures like the use of a wireless headset should
be investigated.
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
[1] Dru Clark and Michael J. Bailey. 2002. Virtual-virtual
haptic feedback and why it wasn’t enough. Proc. SPIE
4665 (2002), 308–318. DOI:
[2] Fabian Fiess and Felix Hundhausen. 2015. Interaktive
Kletterwand. (2015), 429–432.
[3] Brent Edward Insko. 2001. Passive haptics signifi-
cantly enhances virtual environments. Ph.D. Disserta-
tion. University of North Carolina at Chapel Hill, USA.
[4] Raine Kajastila, Leo Holsti, and Perttu Hämäläi-
nen. 2016. The Augmented Climbing Wall. In Pro-
ceedings of the 2016 CHI Conference on Human
Factors in Computing Systems - CHI ’16. ACM
Press, New York, New York, USA, 758–769. DOI:
[5] Timothy W. Kidd. and Jennifer Hazelrigs. Rock Climb-
[6] Mats Liljedahl, Stefan Lindberg, and Jan Berg. 2005.
Digiwall - An Interactive Climbing Wall. In Proceedings
of the 2005 ACM SIGCHI International Conference
on Advances in computer entertainment technology -
ACE ’05 (ACE ’05). ACM, New York, NY, USA, 225–
228. DOI:
[7] Robert William Lindeman. 1999. Bimanual Interaction,
Passive-haptic Feedback, 3D Widget Representation,
and Simulated Surface Constraints for Interaction in
Immersive Virtual Environments. Ph.D. Dissertation.
[8] Néstor Andrés Arteaga Martin, Victor Mittelstädt,
Michael Prieur, Rainer Stark, and Thomas Bär. 2013.
Passive Haptic Feedback for Manual Assembly Sim-
ulation. Procedia CIRP 7 (2013), 509 – 514. DOI:
[9] Paul Milgram and Herman Colquhoun. 1999. A taxon-
omy of real and virtual world display integration. Mixed
Reality: Merging Real and Virtual Worlds (1999), 5–
[10] Hisakazu Ouchi, Yoshifumi Nishida, Ilwoong Kim,
Yoichi Motomura, and Hiroshi Mizoguchi. 2010. De-
tecting and modeling play behavior using sensor-
embedded rock-climbing equipment. Proceedings
of the 9th International Conference on Interaction
Design and Children - IDC ’10 (2010), 118. DOI:
[11] Adalberto L. Simeone, Eduardo Velloso, and Hans
Gellersen. 2015. Substitutional Reality: Using the
Physical Environment to Design Virtual Reality Experi-
ences. In Proc. CHI. ACM, New York, NY, USA, 3307–
3316. DOI:
[12] Frank Steinicke, Gerd Bruder, Anthony Steed,
Klaus H. Hinrichs, and Alexander Gerlach. 2009. Does
a Gradual Transition to the Virtual World increase
Presence? (2009). http://basilic.informatik.uni- hamburg.
[13] Martin Usoh, Kevin Arthur, Mary C. Whitton, Rui
Bastos, Anthony Steed, Mel Slater, and Frederick P.
Brooks, Jr. 1999. Walking > Walking-in-place >
Flying, in Virtual Environments. In Proceedings of
the 26th Annual Conference on Computer Graphics
and Interactive Techniques (SIGGRAPH ’99). ACM
Press/Addison-Wesley Publishing Co., New York, NY,
USA, 359–364. DOI:
[14] Frederik Wiehr, Felix Kosmalla, Florian Daiber, and
Antonio Krüger. 2016. betaCube: Enhancing Training
for Climbing by a Self-Calibrating Camera-Projection
Unit. In Proceedings of the 2016 CHI Conference Ex-
tended Abstracts on Human Factors in Computing
Systems. ACM, 1998–2004.
Late-Breaking Work
CHI 2017, May 6–11, 2017, Denver, CO, USA
... Climbing systems that combine physical climbing with the feeling of presence that can ensue in immersive virtual environments (IVEs), promise to be suitable instruments for climbing preparations and have been a focus of past research [11,15,17]. With such virtual reality (VR) systems, users can climb on a physical wall while being immersed visually in a virtual scene through head-mounted displays (HMDs). ...
... As climbers in the real world can always see their hands and feet, it is necessary to integrate an accurate tracking of the hands and feet, as well as a suitable virtual representation to allow for the same visual reference points in VR climbing. This work is motivated by previous observations showing that users could successfully climb in VR without seeing any representations of their hands and/or feet [11], just relying on their proprioception. To investigate the importance of integrating virtual representations of hands and feet in VR climbing systems, we propose a basic method to track, calibrate, and represent hands and feet in VR. ...
... A first approach to VR climbing experiences have been proposed by Kosmalla et al. [11]. Their system consists of a real climbing wall, a head-mounted display, and a tracking system. ...
... With the recent decline in the prices of displays, tracking technologies and graphics cards, and the competitive nature of the consumer market, new VR applications are being developed for a wide range of fields like entertainment, health care, education, etc. These applications are often multimodal, highly realistic and enable users to experience some of the most extreme activities in VR with multi-sensory stimuli, for example, underwater VR exploration [4], rock climbing [26], etc. A combination of these factors have brought forth a new era of virtual realism ready to be exploited by researchers and businesses alike. ...
... Operating mouse and keyboard are considered just as complex as wielding a joystick or racing wheel. When the game-input requires users to move their body through space, the task complexity further increases [e.g., as in: (Fogtmann et al., 2011;Jensen et al., 2014Jensen et al., , 2015Kajastila and Hämäläinen, 2015;Sano et al., 2016;Kosmalla et al., 2017Kosmalla et al., , 2018Postma et al., 2019)] 2 . Arcadia 3 is a prime example of an eSport that occupies the bottom-right quadrant of Gentile's taxonomy. ...
Full-text available
In 2021, the International Olympic Committee ventured virtual space by launching their first ever Olympic Virtual Series – featuring virtual baseball, cycling, rowing, sailing and motor racing. Interestingly, all these virtual events take strongly after their physical counterparts. Which begs the question: Where are the massively popular esports games like Fortnite, League of Legends, and Dota?–What do the Olympic Virtual Series have that these popular video games do not? Here, we argue for the inclusion of esports within the Olympic program. In many respects, esports “act” and “behave” just like traditional sports. We argue that esports and traditional sports share many of the same values, like the values of meritocracy, competition, fair play, and the value of having a “level playing field”. Yet, in esports, many of these values remain underappreciated, losing out to negative values such as physical inactivity and game-addiction. To preserve what is worth preserving, we borrow from Value Sensitive Design to ameliorate the design-tensions that are foregrounded in esports. Thereby, paving possible ways toward the inclusion of esports in the Olympic program. Ultimately, the question for the IOC should not be “does it look like ‘real sport’, as we know it?”, but rather: are they sporting, rule-led, and fair activities worth preserving and setting an example for a new digitally savvy generation?
To provide immersive haptic experiences, proxy-based haptic feedback systems for virtual reality (VR) face two central challenges: (1) similarity, and (2) colocation. While to solve challenge (1), physical proxy objects need to be sufficiently similar to their virtual counterparts in terms of haptic properties, for challenge (2), proxies and virtual counterparts need to be sufficiently colocated to allow for seamless interactions. To solve these challenges, past research introduced, among others, two successful techniques: (a) Dynamic Passive Haptic Feedback (DPHF), a hardware-based technique that leverages actuated props adapting their physical state during the VR experience, and (b) Haptic Retargeting, a software-based technique leveraging hand redirection to bridge spatial offsets between real and virtual objects. Both concepts have, up to now, not ever been studied in combination. This paper proposes to combine both techniques and reports on the results of a perceptual and a psychophysical experiment situated in a proof-of-concept scenario focused on the perception of virtual weight distribution. We show that users in VR overestimate weight shifts and that, when DPHF and HR are combined, significantly greater shifts can be rendered, compared to using only a weight-shifting prop or unnoticeable hand redirection. Moreover, we find the combination of DPHF and HR to let significantly larger spatial dislocations of proxy and virtual counterpart go unnoticed by users. Our investigation is the first to show the value of combining DPHF and HR in practice, validating that their combination can better solve the challenges of similarity and colocation than the individual techniques can do alone.
Conference Paper
Full-text available
In rock climbing, discussing climbing techniques with others to master a specific route and getting practical advice from more experienced climbers is an inherent part of the culture and tradition of the sport. Spatial information, such as the position of holds, as well as learning complex body postures plays a major role in this process. A typical problem that occurs during advising is an alignment effect when trying to picture orientation-specific knowledge, e.g. explaining how to perform a certain self-climbed move to others. We propose betaCube, a self-calibrating camera-projection unit that features 3D tracking and distortion-free projection. The system enables a life-sized video replay and climbing route creation using augmented reality. We contribute an interface for automatic setup of mobile distortion-free projection, blob detection for climbing holds, as well as an automatic method for extracting planar trackables from artificial climbing walls.
Conference Paper
Full-text available
Experiencing Virtual Reality in domestic and other uncontrolled settings is challenging due to the presence of physical objects and furniture that are not usually defined in the Virtual Environment. To address this challenge, we explore the concept of Substitutional Reality in the context of Virtual Reality: a class of Virtual Environments where every physical object surrounding a user is paired, with some degree of discrepancy, to a virtual counterpart. We present a model of potential substitutions and validate it in two user studies. In the first study we investigated factors that affect participants' suspension of disbelief and ease of use. We systematically altered the virtual representation of a physical object and recorded responses from 20 participants. The second study investigated users' levels of engagement as the physical proxy for a virtual object varied. From the results, we derive a set of guidelines for the design of future Substitutional Reality experiences.
Full-text available
In this study movable props representing an object were used to try to achieve realistic movements of a worker during completion of a manipulation task in an assembly-like scenario recreated in a mixed reality environment. The form and the weight of the movable props were adjusted to different resemblance levels. The effect of these levels on the experiential fidelity of the worker and on the action fidelity of process observers was studied. Results showed that increasing the fidelity of the movable props does not necessarily result on better performance, e.g. simulating the weight of the actual object only seems to improve the experience if the prop also resembles the form of the actual object. Further study is suggested to better understand the effect of other factors.
Conference Paper
Full-text available
In order to increase a user's sense of presence in an artificial environment some researchers propose a gradual transition from reality to the virtual world instead of immersing users into the virtual world directly. One approach is to start the VR experience in a virtual replica of the physical space to accustom users to the characteristics of VR, e.g., latency, reduced field of view or tracking errors, in a known environment. Although this procedure is already applied in VR demonstrations, until now it has not been verified whether the usage of such a transitional environment - as transition between real and virtual environment - increases someone's sense of presence. We have observed subjective, physiological and behavioral reactions of subjects during a fully-immersive flight phobia experiment under two different conditions: the virtual flight environment was displayed immediately, or subjects visited a transitional environment before entering the virtual flight environment. We have quantified to what extent a gradual transition to the VE via a transitional environment increases the level of presence. We have found that subjective responses show significantly higher scores for the user's sense of presence, and that subjects' behavioral reactions change when a transitional environment is shown first. Considering physiological reactions, no significant difference could be found.
Conference Paper
Full-text available
Many injures during childhood are related to the use of playground equipment. Until recently, scientific data of how children actually use playground equipment were scarce. Childhood injury cases were not examined thoroughly from the perspective of how equipment can be modified for improving safety without ruining its attraction to children. To design age-appropriate and safer playground equipment, it is essential that scientific data on the interaction between children and this equipment be accumulated. Herein we report on studies to develop new playground equipment by applying sensor technology to examine the science behind children's interaction with playground equipment. We developed a rock-climbing wall equipped with force sensors to record the physical behavior of children while on the wall, thus allowing measurement of these behaviors in a more natural environment. Fifty force sensors installed in the developed rock-climbing wall are able to collect a large amount of data while children are playing with the equipment. The behavior data of 623 children were recorded in the present study. Herein, we also report on a child behavior prediction model created from the collected data.
Conference Paper
We present the design and evaluation of the Augmented Climbing Wall (ACW). The system combines computer vision and interactive projected graphics for motivating and instructing indoor wall climbing. We have installed the system in a commercial climbing center, where it has been successfully used by hundreds of climbers, including both children and adults. Our primary contribution is a novel movement-based game system that can inform the design of future games and augmented sports. We evaluate ACW based on three user studies (N=50, N=10, N=10) and further observations and interviews. We highlight three central themes of how digital augmentation can contribute to a sport: increasing diversity of movement and challenges, enabling user-created content in an otherwise risky environment, and enabling procedurally generated content. We further discuss how ACW represents an underexplored class of interactive systems, i.e., proximity interaction on wall-sized interactive surfaces, which presents novel human-computer interaction challenges.
Traditional interference detection for visualization has taken a virtual-virtual approach, that is, both the intersector and the intersectee are virtual geometries. But, we have learned that there are advantages in combining both physical models and virtual models in the same space. The physical model has many properties that are difficult to mimic in an all-virtual environment. A realistic interaction is achieved by casting the physical model as a twin to the virtual model. The virtual twin has the ability to interact with other virtual models in software. The two combined into a single system allow for a more effective haptic visualization environment than virtual interaction alone.
BRENT EDWARD INSKO: Passive Haptics Significantly Enhances Virtual Environments (Under the direction of Frederick P. Brooks, Jr.) One of the most disconcertingly unnatural properties of most virtual environments (VEs) is the ability of the user to pass through objects. I hypothesize that passive haptics, augmenting a high-fidelity visual virtual environment with low-fidelity physical objects, will markedly improve both sense of presence and spatial knowledge training transfer. The low-fidelity physical models can be constructed from cheap, easy- to-assemble materials such as styrofoam, plywood, and particle board.
Conference Paper
Digiwall is a climbing wall enhanced with hardware and software. It combines the computer game with sport climbing, and extends both concepts with new features. Digiwall frees the user from focusing on a computer screen. Instead sound and music are used to convey the gaming experience. The Digiwall concept is designed to support a large number of games, competitions, challenges and even aesthetic experiences. It is an example of how technology can promote physical activity and engage people's senses and capabilities in a way that traditional computer gaming and sport climbing do not.