No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Applications of networked virtual reality for tele-operation and tele-assistance systems in the mining industry
Abstract
The mining industry is interested in tele-operation systems to remove mining operators from hazardous or inconvenient environments without losing efficiency. Technologies to enhance the operator's experience are advancing but there is a lack of evidence supporting the extent to which these emerging technologies positively affect user experience. In this paper, we describe three applications that make use of networked virtual and mixed reality. These prototype systems each represent a step towards new VR based technologies that will increase the efficiency and safety of the mining industry.
... Despite massive investments in autonomous driving with certain technological progress, fully autonomous driving (SAE level 5 [19]) does not seem achievable in the foreseeable future. Hence, until the introduction of robust level-5 driving, autonomous vehicles (AVs) will require human supervision and direct intervention in certain situations (SAE level 4), as required by law in some countries 1 . However, one-on-one supervision-no matter if in the car or remotely-is costly and might hinder broad application of the technology. ...
... Offering options for flexible virtual display setup and allowing users to immerse themselves in remote locations, virtual reality (VR) could provide relevant solutions for these challenges. VR has already been proposed for the teleoperation of various sorts of vehicles and machines, including robots [38], vessels [21], mine-site vehicles [1], and road vehicles [11,12,18,36]. Likewise, solutions exist that use VR in monitoring scenarios, for instance, in the maritime industry [37]. ...
While the promise of autonomous vehicles has led to significant scientific and industrial progress, fully automated, SAE level 5 conform cars will likely not see mass adoption anytime soon. Instead, in many applications, human supervision, such as remote monitoring and teleoperation, will be required for the foreseeable future. While Virtual Reality (VR) has been proposed as one potential interface for teleoperation, its benefits and drawbacks over physical monitoring and teleoperation solutions have not been thoroughly investigated. To this end, we contribute two user studies, comparing the performance of and subjective feedback for a VR-based system with an existing monitoring and teleoperation system, which is in industrial use today. The results of our first user study (n=16), indicate that a VR interface replicating the physical interface does not outperform the physical interface. It also quantifies the negative effects that combined monitoring and teleoperating tasks have on users irrespectively of the interface being used. The results of the second user study (n=24), indicate that the perceptual and ergonomic issues caused by VR outweigh its benefits like better concentration through isolation, which is critical for monitoring tasks. Through these two user studies, we contribute to a better understanding of future virtual monitoring and teleoperation solutions for autonomous vehicles.
... Based on Dale's Cone theory [128], [129], [130], immersive training where users can physically enact some of the activities should provide a superior training experience over 2D or 3D counterparts. It is still being determined if an immersive system can provide better training than non-immersive environments in other areas [63], [180]. However, for firefighting training, recent studies that use immersive 3D interactions show considerably improved learning outcomes over 2D systems [151], [181], [182]. ...
Wildfires affect many regions across the world. The accelerated progression of global warming has amplified their frequency and scale, deepening their impact on human life, the economy, and the environment. The temperature rise has been driving wildfires to behave unpredictably compared to those previously observed, challenging researchers and fire management agencies to understand the factors behind this behavioral change. Furthermore, this change has rendered fire personnel training outdated and lost its ability to adequately prepare personnel to respond to these new fires. Immersive visualization can play a key role in tackling the growing issue of wildfires. Therefore, this survey reviews various studies that use immersive and non-immersive data visualization techniques to depict wildfire behavior and train first responders and planners. This paper identifies the most useful characteristics of these systems. While these studies support knowledge creation for certain situations, there is still scope to comprehensively improve immersive systems to address the unforeseen dynamics of wildfires.
... Other specific areas of application include the field of navigation [9,44,61], education [34,77,84], and sports [30]. For example, ObserVAR Fig. 8: Comparison between the types of input modality enabled for the expert to interact with the trainee's space (green) and for the trainee to interact with the expert's space (blue). ...
The recent pandemic, war, and oil crises have caused many to reconsider their need to travel for education, training, and meetings. Providing assistance and training remotely has thus gained importance for many applications, from industrial maintenance to surgical telemonitoring. Current solutions such as video conferencing platforms lack essential communication cues such as spatial referencing, which negatively impacts both time completion and task performance. Mixed Reality (MR) offers opportunities to improve remote assistance and training, as it opens the way to increased spatial clarity and large interaction space. We contribute a survey of remote assistance and training in MR environments through a systematic literature review to provide a deeper understanding of current approaches, benefits and challenges. We analyze 62 articles and contextualize our findings along a taxonomy based on degree of collaboration, perspective sharing, MR space symmetry, time, input and output modality, visual display, and application domain. We identify the main gaps and opportunities in this research area, such as exploring collaboration scenarios beyond one-expert-to-one-trainee, enabling users to move across the reality-virtuality spectrum during a task, or exploring advanced interaction techniques that resort to hand or eye tracking. Our survey informs and helps researchers in different domains, including maintenance, medicine, engineering, or education, build and evaluate novel MR approaches to remote training and assistance. All supplemental materials are available at
https://augmented-perception.org/publications/2023-training-survey.html
.
... It is important to note that the term "digital twin" is still relatively young in literature, and in this review, we have focused on works that use this term to present their research. However, there are relevant works in the past that already take up and describe similar concepts without using this term directly, such as [4,24,34,46]. In addition, it must be mentioned that the term digital twin is strongly influenced by industry and that there are many implementations in the industrial sector that, however, do not appear in the academic literature. ...
... different head gesture controls, and the use of hand gestures such as data gloves and position trackers, have shown favourable results too (Zhu, Gedeon, & Taylor, 2011). Most frequently, some information is shown on-screen displays on live feed camera footage (Bednarz et al., 2011). ...
In mine production scheduling problems are well-known, where most are categorised into deterministic and non-deterministic NP-hard problems because of their complexity. Many researchers have attempted to solve the problem by applying various optimization techniques. While using traditional methods they observed extreme difficulty in solving highly complex problems. In the 1990s, many researchers addressed long-term production scheduling (LTPS) by using simulated annealing, the genetic algorithm, the tabu search algorithm, and the ant colony algorithm. These algorithms are known as the meta-heuristic algorithms, and have proved to be the most efficient algorithms for solving LTPS thus far. The firefly algorithm is one of the best methods for visualising problems related to LTPS, and provides the best possible optimisation. It is one of the simplest methods, and easy to apply to any NP-hard problem. The experimental, and comparison with other recently published algorithms shows that the proposed algorithm is feasible, and is an effective approach for long-term production scheduling problems. The results indicated that a better solution is obtained by the proposed method, compared to other methods in terms of cumulative net present value, and average ore grade. Additionally, the CPU time via the proposed model is roughly 2.52% higher than the other method.
... Healthcare [1][2][3][4][5][6][7][8][9][10]61,101] Industry [13][14][15][16][17][18][19][20][21]30,31,36,38,40,41,92,108,112] Education [11,12,26,29,34,39,47,53,95,96,99,110,111,121,125] Assembly Task [28,35,37,42,48- Design [59,118,124] To summarize the current literature review, as seen in Figure 6, we depict the trend for devices used for displaying the working context, and the development of the industry is surveyed during the periods 1999-2006, 2007-2010, 2011-2015, and 2016-2020. We observe an increase in the production of device types over all periods for all the studied technologies-HMCs, HMDs, video cameras, and smart devices and displays-with the exception of robotic devices and projectors, which are not as widely examined over time, perhaps because of their complexity and automation needs. ...
Due to the appearance of COVID-19 in 2019, person-to-person interactions were drastically reduced. The impact of these restrictions on the economic environment was significant. For example, technical assistance for commissioning or adjusting the parameters of some complex machines/installations had to be postponed. Economic operators became interested in the possibility of remote collaboration, depending on the manufactured products and the performance of the production lines that they owned. This bibliographic research was undertaken to address these needs. The purpose of this review was to analyze the current solutions, approaches, and technologies that workers and specialists can implement to obtain a reliable remote collaboration system. This survey focuses on techniques, devices, and tools that are being used in different contexts to provide remote guidance. We present communication cues and methods being employed, the implemented technological support, and the areas that benefit from remote collaboration. We hope that our effort will be useful to those who develop such systems and people who want to learn about the existence of collaborative solutions, and that it will increase awareness about the applications and the importance of the domain. We are convinced that, with the development of communication systems, the advancement of remote support systems will be a goal for many economic operators.
... Augmented Reality, the term attributed to Tom Caudell in 1990, provides an interface between humans and computers that allows real-time interaction with virtual information in the real world, therefore enhancing our view of the physical real world environment [Akcayir and Akcayir 2017;Bednarz et al. 2011]. ...
The HoloSensor project aims to enhance the visualisation and visual analytics data sourced from various sensors through use of Augmented Reality (AR) technology, allowing users to anchor information throughout various locations inside a building. The output of the project is an application, that connects networked sensors (Arduino with temperature, humidity, light sensors) communicating its data though a Python-based server. Users are able to interact with this data on holograms in real-time through the Microsoft HoloLens. This integration of the immersive AR technology with Internet of Things (IoT) shows its versatile usage in all three areas of human experience: health, home and entertainment.
... During the last few decades, robots have been used with success in various domains ranging from manufacturing (Merzouki et al., 2010), space exploration (Ambrose et al., 2010), and surgery (McMahan et al., 2011) to mining (Bednarz et al., 2011) and military assistance (Wooden et al., 2010). Developing robotic software is difficult and time-consuming, especially when the same functionality must be developed separately for robots with different physical dimensions, hardware control protocols, mechanical configurations, or actuators and sensors. ...
This article presents the core elements of a cross-platform tactile capabilities interface (TCI) for humanoid arms. The aim of the interface is to reduce the cost of developing humanoid robot capabilities by supporting reuse through cross-platform deployment. The article presents a comparative analysis of existing robot middleware frameworks, as well as the technical details of the TCI framework that builds on the the existing YARP platform. The TCI framework currently includes robot arm actuators with robot skin sensors. It presents such hardware in a platform independent manner, making it possible to write robot control software that can be executed on different robots through the TCI frameworks. The TCI framework supports multiple humanoid platforms and this article also presents a case study of a cross-platform implementation of a set of tactile protective withdrawal reflexes that have been realised on both the Nao and iCub humanoid robot platforms using the same high-level source code.
This paper provides an overview of how cognitive load theory applies to web site design and then considers three different methods of measuring cognitive load. NASA TLX is the standard method, but is a post-event measure. The Sternberg Memory Test and a tapping test both provide a means of measuring cognitive load while the person performs a usability test. By knowing the areas of high cognitive load, the designer can then consider what factors are causing the high load and potentially redesign the web site to reduce it.
Projector-camera systems drive applications in many fields such as measurement and spatial augmented reality. When needed, we can find their internal and external parameters via geometric calibration. For this process, we have to use both a printed pattern and a projector pattern, but they can easily interfere with each other. Current methods compensate by decoupling their calibrations or by leveraging structured light and color channels, but the required manipulations are not user-friendly. Therefore, we cannot expect normal users to execute the procedure, which can also become a burden for researchers. Although not always required, knowledge of the geometric parameters can often facilitate development of new systems. To make the calibration process easier, we propose a method that uses fiducial markers, from which we can easily derive a prewarp that, once applied to the projector calibration pattern, prevents its interference. Using our method, we confirmed that users can easily calibrate a projector-camera system in less than one minute, which we consider to be user-friendly, while still achieving typical subpixel accuracy.
We have created an immersive application for statistical graphics
and have investigated what benefits it offers over more traditional data
analysis tools. We present a description of both the traditional data
analysis tools and our virtual environment, and results of an experiment
designed to determine if an immersive environment based on the XGobi
desktop system provides advantages over XGobi for analysis of
high-dimensional statistical data. The experiment included two aspects
of each environment: three structure detection (visualization) tasks and
one ease of interaction task. The subjects were given these tasks in
both the C2 virtual environment and a workstation running XGobi. The
experiment results showed an improvement in participants' ability to
perform structure detection tasks in the C2 to their performance in the
desktop environment. However, participants were more comfortable with
the interaction tools in the desktop system
This paper presents a system that will support a re mote guidance collaboration, in which a local expert guides a rem otely located assistant to perform physical, three-dimensional ta sks. The system supports this remote guidance by allowing th e expert to annotate, point at and draw upon objects in the rem ote location using a pen and tablet-based interface to control a laser projection device. The specific design criteria for this syst em are drawn from a tele-health scenario involving remote medica l examination of patients and the paper presents the software arc hitecture and implementation details of the associated hardware. In particular, the algorithm for aligning the representation of th e laser projection over the video display of the remote sce ne is described. Early evaluations by medical specialists are presen ted, the usability of the system in laboratory experiments i s discussed and ideas for future developments are outlined.
The present paper is a review of the role of individual factors in Spatial Presence. If Spatial Presence is a subjective mental phenomenon psychological factors must have an important role on it. Our review shows that, even though many authors claim about the need for a better understanding about this relation, empirical evidence is still very limited. Personality-related factors as absorption, and the capability to be immersed show to have an influence on the sense of Presence. Additional evidence is needed for the role of such factors as extraversion/introversion. Evidence of the impact of cognitive abilities on Presence in complex media environments is greatly indirect, and based on studies investigating the effect of those cognitive abilities on situation awareness and task performance. The role of practice and demographic factors is also considered.
The Augmented Representation of Cultural Objects (ARCO) system, developed as a part of an EU ICT project, provides museum curators with software and interface tools to develop web-based virtual museum exhibitions by integrating augmented reality (AR) and 3D computer graphics. ARCO technologies could also be deployed in order to implement educational kiosks placed in real-world museums. The main purpose of the system is to offer an entertaining, informative and enjoyable experience to virtual museum visitors.
This paper presents a formal usability study that has been undertaken in order to explore participants’ perceived ‘sense of being there’and enjoyment while exposed to a virtual museum exhibition in relation to real-world visits. The virtual museum implemented was based on an existing gallery in the Victoria and Albert Museum, London, UK. It is of interest to determine whether a high level of presence results in enhanced enjoyment. After exposure to the system, participants completed standardized presence questionnaires related to the perceived realism of cultural artifacts referred to as AR objects’ presence, as well as to participants’ generic perceived presence in the virtual museum referred to as VR presence. The studies conducted indicate that previous experience with ICTs (Information and Communication Technologies) did not correlate with perceived AR objects’ presence or VR presence while exposed to a virtual heritage environment. Enjoyment and both AR objects’ presence and VR presence were found to be positively correlated. Therefore, a high level of perceived presence could be closely associated with satisfaction and gratification which contribute towards an appealing experience while interacting with a museum simulation system.
In this paper we present ARToolKitPlus, a successor to the popular ARToolKit pose tracking library. ARToolKitPlus has been optimized and extended for the usage on mobile devices such as smartphones, PDAs and Ultra Mobile PCs (UMPCs). We explain the need and specific requirements of pose tracking on mobile devices and how we met those requirements. To prove the applicability we performed an extensive benchmark series on a braod range of off-the-shelf handhelds.
We propose a wearable active camera with laser pointer (WACL) as a human interface device for use in telecommunications. The WACL laser pointer is attached to the active camera-head and it can point a laser spot while elevating and panning the camera-head. In our system, a remote instructor can observe around the worker who is wearing the WACL, independently of the worker's motion, and can clearly and naturally instruct the worker in tasks by pointing the laser spot at real objects. This paper describes the outline of a telecommunication support system using the WACL and a method to stabilize the camera and laser pointer independently of the wearer's motion. We have implemented and demonstrated an example of an application using our system.
We describe a new paradigm for three-dimensional computer graphics, using projectors to graphically animate physical objects in the real world. The idea is to replace a physical object---with its inherent color, texture, and material properties---with a neutral object and projected imagery, reproducing the original (or alternative) appearance directly on the object. Because the approach is to effectively "lift" the visual properties of the object into the projector, we call the projectors shader lamps. We address the central issue of complete and continuous illumination of non-trivial physical objects using multiple projectors and present a set of new techniques that makes the process of illumination practical. We demonstrate the viability of these techniques through a variety of table-top applications, and describe preliminary results to reproduce life-sized virtual spaces.
Markerless highly-detailed performance capture is an emerging technology in vision-based graphics and 3D video. For instance, a framework for generating mesh animations from multi-view silhouettes is presented in [Vlasic et al. 2008]. Achieving inverse ...
Although telepresence and a sense of connectedness with others are frequently mentioned in media space studies, as far as
we know, none of these studies report attempts at assessing this critical aspect of user experience. While some attempts have
been made to measure presence in virtual reality or augmented reality, (a comprehensive review of existing measures is available
in Baren and Ijsselsteijn [2004]), very little work has been reported in measuring presence in video-mediated collaboration
systems. Traditional studies of video-mediated collaboration have mostly focused their evaluation on measures of task performance
and user satisfaction. Videoconferencing systems can be seen as a type of media space; they rely on technologies of audio,
video, and computing put together to create an environment extending the embodied mind. This chapter reports on a set of video-mediated
collaboration studies conducted at CSIRO in which different aspects of presence are being investigated. The first study reports
the sense of physical presence a specialist doctor experiences when engaged in a remote consultation of a patient using the
virtual critical care unit (Alem et al., 2006). The Viccu system is an “always-on” system connecting two hospitals (Li et
al., 2006). The presence measure focuses on the extent to which users of videoconferencing systems feel physically present
in the remote location. The second study reports the sense of social presence users experience when playing a game of charades
with remote partners using a video conference link (Kougianous et al., 2006). In this study the presence measure focuses on
the extent to which users feel connected with their remote partners. The third study reports the sense of copresence users
experience when building collaboratively a piece of Lego toy (Melo and Alem, 2007). The sense of copresence is the extent
to which users feel present with their remote partner. In this final study the sense of copresence is investigated by looking
at the word used by users when referring to the physical objects they are manipulating during their interaction as well as
when referring to locations in the collaborative workspace. We believe that such efforts provide a solid stepping stone for
evaluating and analyzing future media spaces.
The concepts of telerobotics, and more generally teleoperation—extension of human sensing and manipulating capability by coupling through communication means to artificial sensors and actuators—are discussed, along with related concepts of telepresence, robotics and supervisory control. Relevant historical developments are reviewed. Current and future applications to space, undersea, nuclear power, the handicapped, surgery, terrestrial mining, construction and maintenance, warehousing, firefighting, policing and military operations are briefly described. Current and new research problems are presented in telesensing, teleactuating, computer-aided supervisory control, and the interactions between the human, computer, teleoperator and task. Finally, some future social implications are discussed. The emphasis throughout is on human-machine interaction rather than on teleoperator hardware or software.
Traditionally, the main goal of teleoperation has been to successfully achieve a given task as if performing the task in local space. An emerging and related requirement is to also match the subjective sensation or the user experience of the remote environment, while maintaining reasonable task performance. This concept is often called “presence” or “(experiential) telepresence,” which is informally defined as “the sense of being in a mediated environment.” In this paper, haptic feedback is considered as an important element for providing improved presence and reasonable task performance in remote navigation. An approach for using haptic information to “experientially” teleoperate a mobile robot is described. Haptic feedback is computed from the range information obtained by a sonar array attached to the robot, and delivered to a user's hand via a haptic probe. This haptic feedback is provided to the user, in addition to stereoscopic images from a forward-facing stereo camera mounted on the mobile robot. The experiment with a user population in a real-world environment showed that haptic feedback significantly improved both task performance and user-felt presence. When considering user-felt presence, no interaction among haptic feedback, image resolution, and stereoscopy was observed, that is, haptic feedback was effective, regardless of the fidelity of visual elements. Stereoscopic images also significantly improved both task performance and user-felt presence, but high-resolution images only significantly improved user-felt presence. When considering task performance, however, it was found that there was an interaction between haptic feedback and stereoscopy, that is, stereoscopic images were only effective when no force feedback was applied. According to the multiple regression analysis, haptic feedback was a higher contributing factor to the improvement in performance and presence than image resolution and stereoscopy.
Designing usable and effective interactive virtual environment (VE) systems is a new challenge for system developers and human factors specialists. In particular, traditional usability principles do not consider characteristics unique to VE systems, such as the design of wayfinding and navigational techniques, object selection and manipulation, as well as integration of visual, auditory and haptic system outputs. VE designers must enhance presence, immersion, and system comfort, while minimizing sickness and deleterious after effects. Through the development of a multi-criteria assessment technique, the current effort categorizes and integrates these VE attributes into a systematic approach to designing and evaluating VE usability. Validation exercises suggest this technique, the Multicriteria Assessment of Usability for Virtual Environments (MAUVE) system, provides a structured approach for achieving usability in VE system design and evaluation. Applications for this research include military, entertainment, and any other interactive system that seeks to provide an enjoyable and effective user experience.
Two experiments investigated components of participants’ spatial knowledge when they navigated large-scale ‘‘virtual buildings’’ using ‘‘desk-top’’ (i.e., nonimmersive) virtual
environments (VEs). Experiment 1 showed that participants could estimate directions with reasonable accuracy when they traveled along paths that contained one or two turns (changes of direction), but participants’ estimates were significantly less accurate when the paths contained three turns. In Experiment 2 participants repeatedly navigated two more complex virtual buildings, one with and the other without a compass. The accuracy of participants’ route-finding and their direction and relative straight-line distance estimates improved with experience, but there were no significant differences between the two compass conditions. However, participants did develop significantly more accurate spatial knowledge as they became more familiar with navigating VEs in general.
Engineering education benefits from the knowledge and experience of topic “experts” both in the technical perspective of understanding the material and the educational perspective of understanding the challenges that students commonly encounter in learning a topic. Classically, these experts have published material as textbooks and perhaps web pages. Through distance learning, these experts can interact with students and instructors beyond the normal confines of a classroom or campus. Classroom-based, distance learning does not provide personalized, individual interaction outside the classroom setting. This situation can be critical in the context of courses with laboratory content. In this paper, the authors discuss how individualized interactions across a wide range of educators and learners can be supported. This support is facilitated by the deployment of telecollaboration stations (TCS). The evolution and use of these stations are described, as are some of the challenges and educational benefits that result.
The effectiveness of virtual environments (VEs) has often been linked to the sense of presence reported by users of those VEs. (Presence is defined as the subjective experience of being in one place or environment, even when one is physically situated in another.) We believe that presence is a normal awareness phenomenon that requires directed attention and is based in the interaction between sensory stimulation, environmental factors that encourage involvement and enable immersion, and internal tendencies to become involved. Factors believed to underlie presence were described in the premier issue of Presence: Teleoperators and Virtual Environments. We used these factors and others as the basis for a presence questionnaire (PQ) to measure presence in VEs. In addition we developed an immersive tendencies questionnaire (ITQ) to measure differences in the tendencies of individuals to experience presence. These questionnaires are being used to evaluate relationships among reported presenc...
Measuring Presence in Virtual Environments: A Presence Questionnaire Hand gestures to operate VR Use of iPhone's touch screen
- B Witmer
- M Singer
WITMER, B., SINGER, M. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence 7, 225-240. Hand gestures to operate VR Use of iPhone's touch screen
Annotating with 'sticky' light for remote guidance. SIGGRAPH Asia 2010, Sketches and Ap-plications Presence in video mediated interaction: case stu-dies. CSIRO in Media space: 20+ years of mediated Life
- M Adcock
- C And Gunn
- Korea
- L Alem
ADCOCK, M., AND GUNN, C. 2010. Annotating with 'sticky' light for remote guidance. SIGGRAPH Asia 2010, Sketches and Ap-plications, Seoul, South Korea. ALEM, L. 2009. Presence in video mediated interaction: case stu-dies. CSIRO in Media space: 20+ years of mediated Life. Ed. London: Springer-Verlag, 369-392.
Annotating with light for remote guid-ance. OZCHI
- D Palmer
- Et Al
PALMER, D., ET AL. 2007. Annotating with light for remote guid-ance. OZCHI 2007.
Shader lamps: animating real objects with image-based illumination. Eurographics Rendering Work-shop 2001 Influence of individual factors on presence in Computers
- R Et Al
- A Sacau
- J Laarni
- T And Hartmann
RASKAR, R.,ET AL. 2001. Shader lamps: animating real objects with image-based illumination. Eurographics Rendering Work-shop 2001. SACAU, A., LAARNI, J., AND HARTMANN, T. 2008. Influence of individual factors on presence in Computers. Human Behavior 24, 2255-2273.
WACL: supporting telecommunications using wearable active camera with laser pointer
- N Sakata
- Et Al
SAKATA, N., ET AL. 2003. WACL: supporting telecommunications
using wearable active camera with laser pointer. Int Symposium
on Wearable Computers.
Shader lamps: animating real objects with image-based illumination
- R Raskar
- Et Al
RASKAR, R.,ET AL. 2001. Shader lamps: animating real objects
with image-based illumination. Eurographics Rendering Workshop 2001.