Content uploaded by Rodrigo L. S. Silva
Author content
All content in this area was uploaded by Rodrigo L. S. Silva on Dec 14, 2023
Content may be subject to copyright.
OrBI – A Hands-free Virtual Reality User Interface for
Mobile Devices
Journal of Mobile Multimedia
Paulo Victor de Magalh˜
aes Rozatto
Andr´
e Luiz Cunha de Oliveira
Rodrigo Luis de Souza da Silva
This is a summary of the aforementioned article. The full version of this article
can be accessed at this link.
Use the following bibtex to cite the main article.
@article{jmm2022orbi,
title={OrBI--A Hands-free Virtual Reality User Interface for Mobile
Devices},
author={Rozatto, Paulo Victor and Oliveira, Andr{\’e} Luiz Cunha and
Silva, Rodrigo Luis},
journal={Journal of Mobile Multimedia},
volume={18},
pages={1599--1616},
year={2022}
}
OrBI – A Hands-free Virtual Reality User Interface for
Mobile Devices
Paulo Victor de Magalh˜
aes Rozatto, Andr´
e Luiz C. de Oliveira, Rodrigo Luis de Souza da Silva
Abstract: The purpose of this paper is to present OrBI, a framework for developing hands-free user interfaces
for interacting with low-cost Virtual Reality systems through mobile devices. The proposed implementation was
developed and used in various Virtual Reality apps. To assess OrBI’s precision and ease to use, we submit the
framework to a quantitative and qualitative evaluation. The findings indicate that the proposed interface model
is easy to use and have a suitable precision. The findings also showed that the framework might be challenging
for older people to use and, in some cases, tiresome.
Keywords: Virtual Reality — User Interface — Hands-Free
Computer Science Department, Federal University of Juiz de Fora
1. Introduction
Immersive virtual reality (VR) commonly uses visual-
ization devices as Head-Mounted Displays (HMDs) ca-
pable of generating 3D scenes and tracking the user’s
head movement. The Oculus Quest, Oculus Rift, and
PlayStation VR are examples of such equipment. De-
spite significant advancements in HMDs and other VR
technologies over the last decade, specialist hardware
remains financially out of reach for a large portion of
the world’s population. As a result, the short- and
medium-term popularization of virtual reality is depen-
dent on lower-cost alternatives. Assuming that mobile
devices are already widely adopted, the usage of low-
cost HMDs that use mobile devices to produce 3D apps
and keep head-tracking, such as Google Cardboard,
may be a key to the democratization of virtual reality.
The problem of interaction is a barrier experienced
while using these low-cost options. In contrast to the
previously mentioned high-end HMDs, which contain
sophisticated six degrees of freedom hand controls that
the user may use to interact with virtual environments,
low-cost HMDs often have at best a few standard phys-
ical buttons that are limited in what they can offer.
The creation of effective in-app controls can help
compensating the lack of physical controls. That in-app
control could take form of a User Interface (UI), which
mediates the user’s interaction with the virtual world.
Ideally, the UI should be straightforward, simple to use,
and adaptable to many applications. From the devel-
oper’s point of view, creating applications that are inde-
pendent of the HMD model to be used by users is an
advantage.
The main contribution of this work is a novel orbital
based virtual reality UI, focusing on mobile devices.
The UI is composed by a positioning system created
relative to the user, similar to an orbital system, in which
the user is at the center of a virtual sphere and the UI
may be moved around using the mobile device’s head
tracking and some auxiliary virtual buttons added to it.
2. Fundamentals
According to [1], the main sources of hands-free inter-
action with virtual reality studied in the literature are
voice recognition, eye tracking, and head tracking. Usu-
ally, voice recognition is accomplished by the recogni-
tion of keywords (e.g., select, start, pause) or through
natural language processing.
Eye tracking can be used to assess where the user
is looking in the virtual environment without the user
having to turn her head. As a result, the user can in-
teract with virtual reality menus and objects by simply
looking at them. The disadvantages of this approach
are non-intentional interactions, which can be resolved
by incorporating more steps (e.g., the user must look
and speak a command to interact with an object), and
HMDs with eye tracking sensors may not be accessible.
Head tracking is fairly common in VR systems. Vir-
tually all HMDs support it, and mobile devices typically
have gyroscopes that can be used to maintain head
tracking. Its primary application is to look around in a
virtual world, but it can also be used to make selec-
tions: a cursor is positioned in the middle of the screen
and can be used to select objects as the user moves
his/her head.
Besides the sources of interaction, there are mainly
three types of UIs for virtual reality: 2D, 3D, and speech
interfaces. In 2D UIs, the user interacts with flat sur-
faces such as buttons. On the other hand, 3D UIs at-
tempt to abstractly imitate real-life activities in order to
replace 2D UIs.
The OrBI (Orbit-Based Interface) framework ex-
tends the hands-free approach, which employs most
frequently 2D UIs with head tracking, by including
movement components for these types of UIs. Our
framework was used to build the VR mode of some
apps available in the VRTools platform [2]. You can ac-
cess a summary of this paper in this link. and see some
examples of the application of this tool in Figure 1.
1
(a) (b) (c)
Figure 1. OrBI framework applied to VR apps. (a) Virtual Travel, (b) Inclined Plane and (c) Vascular Diseases.
3. Related Work
In [3], the three types of user interface are investigated.
Their results suggest that 3D UIs are more pleasant
and immersive, 2D UIs perform better when many items
must be manipulated quickly and precisely, and speech
interfaces are the easiest to learn and the best choice
when several texts must be entered.
In [4] the cell phone accelerometer is used to detect
false steps and translate them to movement in the vir-
tual world. Interaction with objects would take place
through gaze, where a cursor can be placed on the
screen indicating where the user is looking. After look-
ing at an object for a certain time, some action is per-
formed. This technique is known as Dwelling.
The sound of the user’s hand passing over the sur-
faces of a Google Cardboard can be used to recognize
interactive gestures [5], the mobile device’s camera can
be used to recognize the user’s hand gestures [6][7]
and voice commands can be used to toggle between
interaction modes like translation or rotation [8]. A user
interface based on body tilt can improve user experi-
ence and the precision of their movements [9].
According to [10], using Dwelling can be stressful,
exhausting and less efficient than using eye blinking.
Low-cost HMDs, however, do not include eye-blinking
sensors so far. In [11], a circular UI made of an inner
empty circle contained by an outer ring containing char-
acters is proposed, in which selecting a character is as
simple as glancing at the chosen letter and then gazing
to the inner circle for confirmation, requiring no dwelling
time.
Recent researches use non-conventional hands-
free interfaces like facial expression [12][13] and brain-
waves [14][15][16] interpretation, but both classes re-
quire special equipment to be used.
The related work presented above does not address
how to move a user interface around the environment.
In fact, some approaches, such as the swipe over the
HMD surface [5] and the body lean [9] do not render
user interfaces within the application, but they each
have their own limitations. The former may need spe-
cific training for each type of HMD, while the latter is
application specific.
4. OrBI framework
The OrBI framework approach considers the user to be
in the center of an imaginary sphere, with the UI po-
sitioned at some point on the sphere’s surface. This
allows for the creation of a simple way to move the
UI using head tracking: the user triggers the mode to
re-position the UI, and it is positioned in the imaginary
sphere at the point where the user is looking. The prin-
ciple of adjusting the distance between the user and the
UI is also straightforward; simply increase or decrease
the radius of the imaginary sphere.
The tools used to create the framework were
Javascript, WebXR and Threejs. With OrBI it is pos-
sible to build VR content for virtually any smartphone
with a gyroscope and a browser that supports WebXR.
The approach for building the UI was to create 2D
buttons that were presented as a matrix (Figures 2.a
and 2.b). The buttons are code-configurable as they
can be changed in size, background picture, border,
distance between them, how they are displayed and,
of course, their functionality.
A side bar was added to help implement the move-
ment commands. It has three buttons, as shown in
Figure 2.c. The first button increases the radius of
the imaginary sphere, the second activates horizontal
movement, and the third activates vertical movement.
The sphere’s possible radius are determined by code
and can vary from application to application. When the
user clicks the button, the radius increases, and if it
reaches the limit, it returns to the first value. When the
horizontal movement button is pressed, the UI will fol-
low the user’s head motions to the right and left, and
a stop button will appear in the middle top of the in-
terface, allowing the user to stop horizontal movement
by looking up (Figure 2.d). Similarly, when the verti-
cal movement button is pressed, the UI will follow the
user’s head motion up and down, and a stop button will
appear in the middle of the interface, allowing the user
to stop moving by looking right (Figure 2.e). In addition,
a text-display panel was included and it is located on
the right side of the interface. When the displayed text
is difficult to see, the panel can be rotated to allow a
clear view (Figure 2.f).
2
(a) (b) (c)
(d) (e) (f)
Figure 2. An example of the UI displayed in a couple matrix shapes. (a) 2x3 display, (b) 3x2 display, (c) UI with the
movement bar, (d) and (e) UI appearance while being moved and (f) Side panel rotated by 30 degrees
(a) (b) (c)
Figure 3. Evaluation - From left to right, a modified version of the Virtual Travel app followed by the images from
the evaluation circuit.
5. Evaluation
To start our evaluation, volunteers were first introduced
to Virtual Reality and the concepts of head tracking
and dwell time for interaction before being introduced to
OrBI and instructed to perform a small circuit in which
they had to move and place the interface in 3 indepen-
dent targets.
In the introduction to VR and to the other concepts,
the participants used an application identical to the Vir-
tual Travel (Figure 3.a), except for the movement but-
tons and the side panel, which were hidden. The par-
ticipants used the application for about a minute each,
and they were instructed to gaze all around the virtual
environment and use the interface to switch from one
360-degree image to another.
Following the introduction to Virtual Reality, the par-
ticipants watched a short video that introduced OrBI
and explained the circuit they would do.
5.1 The Evaluation Circuit
The evaluation circuit app is a simple 3D room in which
the OrBI is initially placed in front of the user. In this ap-
plication, OrBI has four placeholder buttons in addition
to the movement buttons.
A red square with the same size as the interface
shows where the participant has to position the inter-
face, and an outer square concentric to the first one in-
dicates the allowable margin of error, changing to green
when the interface was inside the allowed boundaries
(Figures 3.b and 3.c).
3
The first target is positioned to the user’s right and
at the same height as the interface, so the user simply
needs to slide OrBI horizontally to fit it in the first target.
The second target differs from the first just in height
and in distance, thus the user must increase the orbit
radius and shift the interface upwards. The final target
is positioned at the same coordinates as OrBI at the
start, but at a greater distance, so the user needs to
move the interface horizontally and vertically to fit it in.
5.2 The Volunteers
In total, 20 volunteers participated in this study, with
80% male and 20% female. The age range of the vol-
unteers is from 17 to 63 years old, with the majority of
them being young (50% aged from 17 to 20 years old).
In terms of education level, 10% have a complete or
incomplete middle school education, 40% have a com-
plete or incomplete high school education, 25% have
an incomplete higher education, and 25% have a com-
plete higher education.
6. Analyses and Discussion
In this section we will present the qualitative and quan-
titative analysis of our work including the discussion of
the results. You can find the detailed analysis in the full
version of this article available at this link.
6.1 Quantitative analysis
To perform our quantitative analysis we developed a
protocol in which we evaluate the time a volunteer took
to complete the circuit and the distance to the center of
each target. The time unit is seconds, and the distance
unit is virtual world unit, but the application was built so
that one virtual world unit represents one meter. The
measured time and the average between the three dis-
tances corresponding to the three targets used in the
analysis.
The average execution time was approximately
94.29s, but the standard deviation was 1×102s. An
explanation for such high deviation may lie in the differ-
ence of the ages groups. Despite not being a strongly
linear relationship, a high level of correlation (R2=
0.742) between age and time performance was found. It
may be a plausible hypothesis that older people strug-
gle more with OrBI because they are less experienced
with technology in general, but this conclusion requires
comparative data across different technologies to be
more assertive.
The average distance from the center of the target
was 0.12m and the standard deviation was 0.1m. It is
important mentioning that the final position of the in-
terface has to be less than 0.4m to the target to be
accepted and the users had only one try for each tar-
get (the first valid position where OrBI was stopped
was counted and the target was removed). This time
there is no correlation between age and measurements
(R2=0.066).
6.2 Qualitative Analysis
Following the completion of the evaluation circuit, the
participants were requested to answer a survey with
five questions about their perceptions of OrBI, with each
question having five options: fully agree, agree, neutral,
disagree, or totally disagree.
The findings were generally positive: 95% of the col-
laborators agree or totally agree that it is easy to learn
how to use OrBI, 90% agree or totally agree that it is
easy to position OrBI wherever they want, 95% agree
or totally agree in the utility of moving interfaces, and
95% agree or totally agree that they had a overall good
impression of OrBI.
However, the results regarding how tiring it is to
move the interface are not as largely positive as the
others. Despite the fact that 60% of participants dis-
agreed or fully disagreed that moving OrBI is taxing, the
fact that 25% were indifferent and 15% agreed or totally
agreed may cause concerns. Those statistics appear
to support the assertions in the related work that the
Dwelling technique may be exhausting to the user in
some instances.
7. Conclusion and future works
The goal of this paper was to present a framework for
creating mobile-based user interfaces for low-cost Vir-
tual Reality that do not require physical controllers to
interact with a virtual world. Several VR apps for the
Web were created using an OrBI framework implemen-
tation that employs the Dwelling approach for triggering
interactions. Then, two specific apps were created to
evaluate the ease of use and precision of OrBI.
The overall evaluation was positive, with 95% of
participants finding it easy to learn how to use OrBI
and 90% finding it simple to position the UI anywhere
they wanted. However, quantitative statistics reveal that
older people have a more difficult time utilizing OrBI, al-
though age has no effect on precision. Furthermore,
only 60% of participants believe that using the interface
is not tiring, while 25% are indifferent and 15% agree
that it is exhausting someway. Due to the existence of
some related works claiming that Dwelling is exhaust-
ing for the user, the Dwelling approach was speculated
to be the reason of the tiring issue.
In the future, it is planned to investigate alternatives
to the Dwelling technique, primarily by employing light-
weight approaches for hand-tracking and gesture de-
tection using mobile devices’ cameras.
4
References
1 MONTEIRO, P. et al. Hands-free interaction in
immersive virtual reality: A systematic review. IEEE
Trans. Vis. Comput. Graph., v. 27, n. 5, p. 2702–2713,
2021.
2 COSTA, L. D. da et al. VR Tools - A free
open-source platform for virtual and augmented reality
applications on the web. Journal of Mobile Multimedia,
p. 27–42, 2022.
3 WEISS, Y. et al. What user interface to use for
virtual reality? 2d, 3d or speech–a user study. In:
IEEE. 2018 International Conference on Cyberworlds
(CW). [S.l.], 2018. p. 50–57.
4 SHANMUGAM, S.; VELLORE, V. R.; SARASVATHI,
V. Vrnav: A framework for navigation and object
interaction in virtual reality. In: IEEE. 2017 2nd
International Conference on Computational Systems
and Information Technology for Sustainable Solution
(CSITSS). [S.l.], 2017. p. 1–5.
5 CHEN, T. et al. Gestonhmd: Enabling gesture-
based interaction on low-cost vr head-mounted display.
IEEE Transactions on Visualization and Computer
Graphics, IEEE, v. 27, n. 5, p. 2597–2607, 2021.
6 LUO, S.; TEATHER, R. J. Camera-based selection
with cardboard hmds. In: IEEE. 2019 IEEE Conference
on Virtual Reality and 3D User Interfaces (VR). [S.l.],
2019. p. 1066–1067.
7 DANI, M. et al. Mid-air fingertip-based user
interaction in mixed reality. In: IEEE. 2018 IEEE
International Symposium on Mixed and Augmented
Reality Adjunct (ISMAR-Adjunct). [S.l.], 2018. p.
174–178.
8 GRINSHPOON, A. et al. Hands-free interaction for
augmented reality in vascular interventions. In: IEEE.
2018 IEEE Conference on Virtual Reality and 3D User
Interfaces (VR). [S.l.], 2018. p. 751–752.
9 HASHEMIAN, A. et al. Headjoystick: Improving
flying in vr using a novel leaning-based interface. IEEE
Transactions on Visualization and Computer Graphics,
IEEE, 2020.
10 LU, X. et al. Exploration of hands-free text entry
techniques for virtual reality. In: IEEE. 2020 IEEE
International Symposium on Mixed and Augmented
Reality (ISMAR). [S.l.], 2020. p. 344–349.
11 XU, W. et al. Ringtext: Dwell-free and hands-free
text entry for mobile head-mounted displays using
head motions. IEEE transactions on visualization and
computer graphics, IEEE, v. 25, n. 5, p. 1991–2001,
2019.
12 DEY, A. et al. Effects of interacting with
facial expressions and controllers in different
virtual environments on presence, usability, affect,
and neurophysiological signals. International
Journal of Human-Computer Studies, v. 160, p.
102762, 2022. ISSN 1071-5819. Dispon´
ıvel em:
⟨https://www.sciencedirect.com/science/article/pii/
S1071581921001804⟩.
13 WEN, L. et al. A survey of facial capture for virtual
reality. IEEE Access, v. 10, p. 6042–6052, 2022.
14 SUHAIMI, N. S.; MOUNTSTEPHENS, J.;
TEO, J. A dataset for emotion recognition using
virtual reality and eeg (der-vreeg): Emotional
state classification using low-cost wearable vr-eeg
headsets. Big Data and Cognitive Computing,
v. 6, n. 1, 2022. ISSN 2504-2289. Dispon´
ıvel em:
⟨https://www.mdpi.com/2504-2289/6/1/16⟩.
15 ROUILLARD, J.; SI-MOHAMMED, H.;
CABESTAING, F. A pilot study for a more Immersive
Virtual Reality Brain-Computer Interface. In:
International Conference on Applied Human
Factors and Ergonomics (AHFE). New-York,
United States: [s.n.], 2022. Dispon´
ıvel em:
⟨https://hal.archives-ouvertes.fr/hal-03589578⟩.
16 NUGROHO, E. W.; HARNADI, B. The method of
integrating virtual reality with brainwave sensor for an
interactive math’s game. In: 2019 16th International
Joint Conference on Computer Science and Software
Engineering (JCSSE). [S.l.: s.n.], 2019. p. 359–363.
5
