Conference Paper
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

We present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user's gaze direction and body gestures while it transforms in size and orientation to stay within the AR user's field of view. A user study was conducted to evaluate Mini-Me in two collaborative scenarios: an asymmetric remote expert in VR assisting a local worker in AR, and a symmetric collaboration in urban planning. We found that the presence of the Mini-Me significantly improved Social Presence and the overall experience of MR collaboration.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Due to the versatility of avatars in AR and VR, different kinds of collaborative telexistence systems with virtual avatars have been proposed by prior research (e.g. [2,7,17,27,30,37,38,43,44,45]). ...
... People can be also presented in telexistense systems via real-time video streaming [42]. Piumsomboon et al. [43] present an adaptive avatar called Mini-Me for mixed reality remote collaboration between a local AR user and a remote VR user. Mini-Me is shown as a realistic yet simplified full body avatar in VR, whereas the AR avatar is shown as a hologram. ...
... Mini-Me is shown as a realistic yet simplified full body avatar in VR, whereas the AR avatar is shown as a hologram. In VR, Mini-Me also shows user's gaze direction and body gestures while it transforms in size and orientation to stay within the AR user's field of view [43]. In later study, Piumsomboon et al. [44] combine video with AR and VR interaction with an adaptive photorealistic full body avatar. ...
Article
This paper presents two studies investigating how physically remote telexistence users wish to see other users visualized as virtual avatars in a) augmented reality, and b) immersive virtual reality while conducting a collaborative task. To answer this research question, a telexistence system was designed and implemented with simple avatar designs. After that, visual examples of alternative avatar representations for both use cases were designed by thoughtfully altering the visual parameters of 36 virtual avatar examples. The avatar designs were first evaluated in a user study with 16 participants in conjunction with using an implemented telexistence system. As a follow-up an online survey with 43 respondents was used to record their preferences regarding virtual avatar appearance. The results suggest that users prefer the other user to be represented in a photorealistic full-body human avatar in both augmented reality and virtual reality due to its humanlike representation and affordances for interaction. In augmented reality, the choice for a hologram full body avatar was also popular due to its see-through appearance, which prevents a mix-up with a real person in the physical space.
... Recently, multimodal cues are used in collaborative environments, typically using auditory and visual elements. For example, virtual avatars have been explored to represent each collaborator and to provide an increased awareness of others in the shared environment [21,30,46]. In AR environments, researchers explored the use of virtual arrows to represent collaborators' head directions [11] and miniature virtual avatars to show collaborators' gaze directions and body gestures [46]. ...
... For example, virtual avatars have been explored to represent each collaborator and to provide an increased awareness of others in the shared environment [21,30,46]. In AR environments, researchers explored the use of virtual arrows to represent collaborators' head directions [11] and miniature virtual avatars to show collaborators' gaze directions and body gestures [46]. Although using avatars can contribute to users' perceived social presence, it adds extra visual elements to the limited display and field-of-view of current AR HMDs. ...
... In addition to gaze pointing cues, augmenting hand pointing cues via gestures was shown to be able to facilitate users' perception of others' actions. For example, Piumsomboon et al. [46] reported that redirected gaze and gestures in their Mini-Me system improved users' awareness of the partner in a collaborative AR interface. Yang et al. [59] stated that visual head frustum and hand gestures intuitively demonstrated the remote user's movements and target positions. ...
Preprint
Full-text available
Visual cues are essential in computer-mediated communication. It is especially important when communication happens in a collaboration scenario that requires focusing several users' attention on aspecific object among other similar ones. This paper explores the effect of visual cues on pointing tasks in co-located Augmented Reality (AR) collaboration. A user study (N = 32, 16 pairs) was conducted to compare two types of visual cues: Pointing Line (PL)and Moving Track (MT). Both are head-based visual techniques.Through a series of collaborative pointing tasks on objects with different states (static and dynamic) and density levels (low, mediumand high), the results showed that PL was better on task performance and usability, but MT was rated higher on social presenceand user preference. Based on our results, some design implicationsare provided for pointing tasks in co-located AR collaboration.
... This section intends to describe related works following the taxonomy presented in section 2. It is our objective to focus exclusively on research or industrial attempts that move forward from the traditional video conferencing systems by enhancing some of their limitations. However, this overview does not claim Lawrence et al. (2021) for Face-to-face window, Horizon Workrooms by Meta for Shared Virtual Reality, Remote Collaboration by Gao et al. (2018) for Remote Assistance, VR Together by Prins et al. (2018) for Social Extended Reality, The Owl by Kachach et al. (2021) for Immersive Telepresence, and Mini me by Piumsomboon et al. (2018) for Digital Twin. ...
... Some academic works have also explored this type of systems, such as Pan and Steed (2017) In more advanced implementations, such as the aforementioned Holoportation and VR Together, users are represented using 3D point clouds obtained from volumetric video capture. Should there are spatial constrains for full-size avatar placement, the use of miniature versions is becoming a common practise (Piumsomboon et al., 2018), yet eye contact needs to be carefully designed (Anjos et al., 2019). ...
... In this work, apart from audio communication, the host might has further visual cues such as cursor pointer, head pointer or eye gaze from the visitor. , c) AR glasses depicting laser pointing cues (Gao et al., 2020), d) AR glasses depicting hand pointing (Sasikumar et al., 2019), e) AR glasses depicting area of Lee et al. (2018), f) AR glasses depicting a mini representation of the visitor pointing to some object (Piumsomboon et al., 2018), and g) avatar in a display (left), upper body avatar on top of the robot (right) resulting from the combination of 3D reconstructed environment and live 360 • video panorama camera (Teo et al., 2019b(Teo et al., , 2020. In turn, all these systems share the use of AR Host archetype with AR googles as the mainstream system for the reverse side. ...
Preprint
Full-text available
Several technological and scientific advances have been achieved recently in the fields of immersive systems, which are offering new possibilities to applications and services in different communication domains, such as entertainment, virtual conferencing, working meetings, social relations, healthcare, and industry. Users of these immersive technologies can explore and experience the stimuli in a more interactive and personalized way than previous technologies. Thus, considering the new technological challenges related to these systems and the new perceptual dimensions and interaction behaviors involved, a deep understanding of the users' Quality of Experience is required to satisfy their demands and expectations. In this sense, it is essential to foster the research on evaluating the QoE of immersive communication systems, since this will provide useful outcomes to optimize them and to identify the factors that can deteriorate the user experience. With this aim, subjective tests are usually performed following standard methodologies, which are designed for specific technologies and services. Although numerous user studies have been already published, there are no recommendations or standards that define common testing methodologies to be applied to evaluate immersive communication systems, such as those developed for images and video. Therefore, a revision of the QoE evaluation methods designed for previous technologies is required to develop robust and reliable methodologies for immersive communication systems. Thus, the objective of this paper is to provide an overview of existing immersive communication systems and related user studies, which can help on the definition of basic guidelines and testing methodologies to be used when performing user tests of immersive communication systems, such as 360-degree video-based telepresence, avatar-based social VR, cooperative AR, etc.
... They showed that direct manipulations were faster for assembly tasks than drawing annotations. Piumsomboon et al.'s [217] Mini-Me displays an adaptive remote VR user's avatar always visible to an AR user, either as a mini or as a life-size avatar. The avatar conveys pointing actions, is surrounded by a halo notifying the mini avatar entering of leaving the local user's FoV, has a blowing ring location cue for the actual life-size avatar, and has a graphical distinction between the mini and the life-size avatars. ...
... Embodiment in remote systems provides information about the users' locations. The general focus of a collaborator can be provided through FoV representation, and eye-and head-gaze cues [215,[217][218][219]. FoV cues work best and eye-gaze cues are not sufficient [215]. ...
... The general focus of a collaborator can be provided through FoV representation, and eye-and head-gaze cues [215,[217][218][219]. FoV cues work best and eye-gaze cues are not sufficient [215]. Embodiment can rely on hands-only [131], head-only [178], exocentric life-size avatars, or egocentric miniavatars [217] which allow workers to keep the remote collaborators' expressions in sight. In co-located scenarios with a shared exocentric view, most of the listed cues come for free and can even be removed. ...
Thesis
I studied the benefits and limitations of Augmented Reality (AR) Head-Mounted Displays (AR-HMDs) for collaborative 3D data exploration. Prior of conducting any projects, I saw in AR-HMDs benefits concerning their immersive features: AR-HMDs merge the interactive, visualization, collaborative, and users' physical spaces together. Multiple collaborators can then see and interact directly with 3D visuals anchored within the users' physical space. AR-HMDs usually rely on stereoscopic 3D displays which provide additional depth cues compared to 2D screens, supporting users at understanding 3D datasets better. As AR-HMDs allow users to see each other within the workspace, seamless switches between discussion and exploration phases are possible. Interacting within those visualizations allow for fast and intuitive 3D direct interactions, which yields cues about one's intentions to others, e.g., moving an object by grabbing it is a strong cue about what a person intends to do with that object. Those cues are important for everyone to understand what is currently going on. Finally, by not occluding the users' physical space, usual but important tools such as billboards and workstations performing simulations are still easily accessible within this environment without wearing off the headsets. That being said, and while AR-HMDs are being studied for decades, their computing power before the recent release of the HoloLens in 2016 was not enough for an efficient exploration of 3D data such as ocean datasets. Moreover, previous researchers were more interested in how to make AR possible as opposed to how to use AR. Then, despite all those qualities one may think prior of working with AR-HMDs, there were almost no work that discusses the exploration of such 3D datasets. Moreover AR-HMDs are not suitable for 2D input which are however commonly used with usual explorative tools such as ParaView or CAD software, where users such as scientists and engineers are already efficient with. I then theorize in what situations are AR-HMDs preferable. They seem preferable when the purpose is to share insights with multiple collaborators and to explore patterns together, and where explorative tools can be minimal compared to what workstations provide as most of the prior work and simulations can be done before hand. I am thus combining AR-HMDs with multi-touch tablets, where I use AR-HMDs to merge the visualizations, some 3D interactions, and the collaborative spaces within the users' physical space, and I use the tablets for 2D input and usual Graphical User Interfaces that most software provides (e.g., buttons and menus). I then studied low-level interactions necessary for data exploration which concern the selection of points and regions inside datasets using this new hybrid system. The techniques my co-authors and I have chosen possess different level of directness that we investigated. As this PhD aims at studying AR-HMDs within collaborative environments, I also studied their capacities to adapt the visual to each collaborator for a given anchored 3D object. This is similar to the relaxed "What-You-See-Is-What-I-See" that allows, e.g., multiple users to see different parts of a shared document that remote users can edit simultaneously. Finally, I am currently (i.e., is not finished by the time I am writing this PhD) studying the use of this new system for the collaborative 3D data exploration of ocean datasets that my collaborators at Helmholtz-Zentrum Geesthacht, Germany, are working on. This PhD provides a state of the art of AR used within collaborative environments. It also gives insights about the impacts of 3D interaction directness for 3D data exploration. This PhD finally gives designers insights about the use of AR for collaborative scientific data exploration, with a focus on oceanography.
... For instance, collaborators may not be able to see or hear each other, which may impede natural interaction among them. To overcome these barriers, previous research proposed several features to support communication and coordination in co-located [12,13] or distributed [14][15][16][17][18][19][20][21][22] scenarios, using HHDs [12,13], HMDs [14][15][16][17][18][20][21][22], or both [19,23]. ...
... For instance, collaborators may not be able to see or hear each other, which may impede natural interaction among them. To overcome these barriers, previous research proposed several features to support communication and coordination in co-located [12,13] or distributed [14][15][16][17][18][19][20][21][22] scenarios, using HHDs [12,13], HMDs [14][15][16][17][18][20][21][22], or both [19,23]. ...
... Such collaboration support features include awareness cues indicating where collaborators are or what they do. To represent absent collaborators in space, previous research considered different kinds of avatars: While some used human-like avatars [18,20,21,24], others only visualized parts of the collaborator using virtual replications of the respective XR device [18,19,25], view frustums [14,21,25], or further virtual objects as abstract representations [19,22,25]. While most avatars were created based on hand and head movements, some approaches tracked the entire body. ...
Article
Full-text available
Extensive research has outlined the potential of augmented, mixed, and virtual reality applications. However, little attention has been paid to scalability enhancements fostering practical adoption. In this paper, we introduce the concept of scalable extended reality (XRS), i.e., spaces scaling between different displays and degrees of virtuality that can be entered by multiple, possibly distributed users. The development of such XRS spaces concerns several research fields. To provide bidirectional interaction and maintain consistency with the real environment, virtual reconstructions of physical scenes need to be segmented semantically and adapted dynamically. Moreover, scalable interaction techniques for selection, manipulation, and navigation as well as a world-stabilized rendering of 2D annotations in 3D space are needed to let users intuitively switch between handheld and head-mounted displays. Collaborative settings should further integrate access control and awareness cues indicating the collaborators’ locations and actions. While many of these topics were investigated by previous research, very few have considered their integration to enhance scalability. Addressing this gap, we review related previous research, list current barriers to the development of XRS spaces, and highlight dependencies between them.
... To solve the problems mentioned above, a straightforward thought is to integrate the 1PP and 3PP to reap the advantages of the two perspectives. Technologies of multiple perspectives integration have been extensively studied to improve user's navigation [29], collaboration [34], and manipulation performance [23]. World-in-miniature (WIM), which can integrate the 1PP and 3PP simultaneously into a single interface with considerable interaction performance [14], is one of the most sound integration technologies. ...
... We implemented the MPI using a handheld WIM which enables adjusting the assisted view in natural habits of human, but further improvements on MPI's control are needed. Possible improvements could be reducing the controlling complexity of the MPI, such as the adaptive WIM locating and adjusting [34] to further relieve the interference of the WIM model control on bimanual manipulation. Another possible improvement could be providing collision cues in MPI to strengthen user's collision awareness. ...
Article
Full-text available
Virtual reality (VR) has been proved as a promising tool for industrial design, but the traditional VR interface of first-person perspective (1PP) is not efficient to support assemblability assessment in narrow assembly spaces. In this paper, we proposed the multi-perspectives interface (MPI) which integrates the 1PP and the third-person perspective (3PP) using the handheld world-in-miniature (WIM). The MPI allows users to simulate the assembly operations in a natural manner similar to 1PP, while providing users with an overview of the assembly status through the WIM to assess the assemblability with superior spatial awareness. Two studies were conducted to test the performance of the proposed MPI. The first study tested user’s interaction performance in MPI using a common interaction task, which reveals stronger spatial awareness in MPI than in 1PP without the cost of losing natural interaction. Based on the results of the first study, the second study tested the performance, usability, and workload of MPI in an assemblability assessment task. The results show the advantages of MPI in the reachability evaluation in the narrow spaces. The main contribution of this paper is improving the interface and user-interface interaction in VR-aided assembly assessment system to improve user’s interaction performance and assessment ability in narrow assembly spaces.
... They can manipulate the virtual on-site room and objects in it by hand-gestures or additional control devices. Areas in which such settings have been successfully applied include industrial maintenance [4,24], the provision of advice and finding objects [38,50], manual assembly tasks [18,26], crime scene investigation [17], remote furniture consultations [15,37] as well as support and training in healthcare [9,40,44] as discussed in this paper. In most of these scenarios, the remote user is some sort of helper or holds additional knowledge. ...
... Datcu et al. [17] showed how pointers used by the remote helper enable the on-site user to act based on the helper's advice. Kim et al. [28] and Piumsomboon et al. [38] showed that pointing gestures improve the cooperation by referencing areas to be focused on. Norman et al. [37] and Teo et al. [48] showed that drawing gestures effectively enable the remote user to depict areas of the on-site user's environment. ...
Article
Professional caregivers often face complex situations in which they need the support of a colleague or a specialist. This is especially necessary for caregivers during their training or with less practice in certain tasks. Due to time and space restrictions, colleagues or specialists are not always available for local support. Remote support by streaming videos to a remote helper has been discussed in healthcare domains as a so-called teleconsultation. However, little is known about how to apply teleconsultation in care. We conducted a study with caregivers and remote helpers to compare head mounted devices (HMDs) for teleconsultation in care to two alternative solutions using smartphones. We found that despite lacking familiarity, HMDs have good potential for remote support in care, and that creating video streams with smartphones is not preferable for teleconsultations. We suggest that ideal support needs to balance freedom and guidance and suggest how such support needs to be further explored.
... Most AR-based shared spaces are confined to the local space itself, which limits the number of remote participants present owing to spatial limitations. One possibility is to use miniature avatars [40], which we believe will not provide a sufficient co-presence or realism or apply a large empty hall as the local space (and augment the audience to all seats), which is too costly or impractical. A classroom or performance hall with a full audience televised to the remote lecturer would be incompatible with the COVID-19 pandemic. ...
Article
Full-text available
AudienceMR is designed as a multi-user mixed reality space that seamlessly extends the local user space to become a large, shared classroom where some of the audience members are seen seated in a real space, and more members are seen through an extended portal. AudienceMR can provide a sense of the presence of a large-scale crowd/audience with the associated spatial context. In contrast to virtual reality (VR), however, with mixed reality (MR), a lecturer can deliver content or conduct a performance from a real, actual, comfortable, and familiar local space, while interacting directly with real nearby objects, such as a desk, podium, educational props, instruments, and office materials. Such a design will elicit a realistic user experience closer to an actual classroom, which is currently prohibitive owing to the COVID-19 pandemic. This paper validated our hypothesis by conducting a comparative experiment assessing the lecturer’s experience with two independent variables: (1) an online classroom platform type, i.e., a 2D desktop video teleconference, a 2D video screen grid in VR, 3D VR, and AudienceMR, and (2) a student depiction, i.e., a 2D upper-body video screen and a 3D full-body avatar. Our experiment validated that AudienceMR exhibits a level of anxiety and fear of public speaking closer to that of a real classroom situation, and a higher social and spatial presence than 2D video grid-based solutions and even 3D VR. Compared to 3D VR, AudienceMR offers a more natural and easily usable real object-based interaction. Most subjects preferred AudienceMR over the alternatives despite the nuisance of having to wear a video see-through headset. Such qualities will result in information conveyance and an educational efficacy comparable to those of a real classroom, and better than those achieved through popular 2D desktop teleconferencing or immersive 3D VR solutions.
... Vive has a FOV around 110° and HoloLens has a FOV around 35° [Piumsomboon, 2018]. During the experiment with HoloLens, students could see only a part of the hint at a glance because of the narrow FOV. ...
Article
Full-text available
Video/audio conferencing systems have been used extensively for remote collaboration over many years. Recently, virtual and mixed reality (VR/MR) systems have started to show great potential as communication media for remote collaboration. Prior studies revealed that the creation of common ground between discourse participants is crucial for collaboration and that grounding techniques change with the communication medium. However, it is difficult to find previous research that compares VR and MR communication system performances with video conferencing systems regarding the creation of common ground for collaborative problem solving. On the other hand, prior studies have found that display fidelity and interaction fidelity had significant effects on performance-intensive individual tasks in virtual reality. Fidelity in VR can be defined as the degree of objective accuracy with which the real-world is represented by the virtual world. However, to date, fidelity for collaborative tasks in VR/MR has not been defined or studied much. In this paper, we compare five different communication media for the establishment of common ground in collaborative problem-solving tasks: Webcam, headband camera, VR, MR, and audio-only conferencing systems. We analyzed these communication media with respect to collaborative fidelity components which we defined. For the experiments, we utilized two different types of collaborative tasks: a 2D Tangram puzzle and a 3D Soma cube puzzle. The experimental results show that the traditional Webcam performed better than the other media in the 2D task, while the headband camera performed better in the 3D task. In terms of collaboration fidelity, these results were somehow predictable, although there was a little difference between our expectations and the results.
... The concept of scaling the size of the user in MR collaboration is explored in recent studies, by utilizing the miniaturized avatar [2], the giant avatar [1]. Our multi-scale user function allows the AR user to change the size of the VR user which enables the VR user to see the space from a different viewpoint. ...
... Avatars are often deployed to create a sence of presence and to mediate non-verbal communication [11,31,32]. Such collaborative digital environments have been presented in literature and can take place in a virtual environment ( [19,28,30,35]), in augmented environments ( [4,24,25]), or in an asymmetric combination of virtual and augmented reality ( [5,17,20,23,26,27,36,41]). Immediate environment can be captured and represented by a point cloud using real-time reconstruction [21,37,38]. ...
Conference Paper
Full-text available
Current teleconsultation solutions for preclinical emergencies can transmit knowledge from a remote expert to a local paramedic using audio and 2D video channels. Such technology lacks precision and efficiency for medical diagnostic tasks, and visual feedback is often missing between participants. We investigate a mixed reality 3D teleconsultation solution for preclinical use, which provides a 3D reconstruction of the local scene to a remote expert, displayed in Virtual Reality. A remote expert can join the local scene virtually as an Augmented Reality avatar. The remote expert can annotate the local scene and guide the local paramedics through the procedure. We explored our system in a user study within a preclinical scenario on a collaborative task of attaching chest lead electrodes of a 12 channel electrocardiogram on a mannequin. We compared the 3D teleconsultation system to a 2D video-mediated teleconsultation via a top-mounted camera and report results from the consultee side in AR. Based on our empirical user study with 10 paramedics with an average of 17 years experience, we observe an improvement in the electrode placement quality using the 3D teleconsultation system. Results indicate no significant difference in the cognitive task-load between conditions. Participants perceived the video-mediated consultation as more usable due to their unfamiliarity with the 3D teleconsultation system. However, participants acknowledge the potential of 3D teleconsultation and believe such a system can significantly improve the preclinical treatment.
... XRDirector mixes several VR and AR realities as a role based immersive authoring tool [21]. Piumsomboon et al. investigated hybrid XR environments for remote collaboration, [26][27][28][29], where one user in VR and another user is in AR. ...
... XRDirector mixes several VR and AR realities as a role based immersive authoring tool [21]. Piumsomboon et al. investigated hybrid XR environments for remote collaboration, [26][27][28][29], where one user in VR and another user is in AR. ...
... As mentioned by the authors, the limited FoV of their HoloLens (34°diagonal) may have an impact on a user's sense of (co-)presence and affect an agent's perceived capabilities. For a similar reason, although not for a virtual human but an avatar representing a VR user, in [32] a miniature avatar was introduced to reposition the VR user and its communication cues in the user's FoV. From experiments with a HoloLens device and more and less realistic virtual humans, it was concluded in [33] that more realistic agents in AR were rated higher from the point of view of usability and attractiveness. ...
Conference Paper
In augmented reality, we have a virtual layer superimposed on reality. This virtual layer can contain multisensory content, that is, content that stimulates each of our senses: sight, hearing, taste, touch, and smell, and many interoceptive senses such as pain, heat, time, and balance. The virtual layer can contain virtual humans that have virtual senses to explore the augmented reality environment and to interact with real humans. These virtual humans can take on a role in social, marketing, retail, training, education, and entertainment augmented reality applications. In the human-like behavior of virtual humans, we expect awareness, use, and display of awareness of these senses. We present a view on the existing literature on virtual humans in augmented reality with a focus on sense-related presence issues.
... Misawa et al. [8] proposed a telepresence method using a robot equipped with a simulated human face to act as the avatar of remote users in support of nonverbal communication. In addition, Piumsomboon et al. [15] developed a remote collaboration system that visualizes the gaze direction and body gestures of a remote user wearing a VR headset; using avatar representation, the size and orientation of the remote user avatar are changed to stay in the AR user's field of sight. Ruvimova et al. proposed a platform that aims to improve business performance by building a virtual environment away from the office using virtual reality (VR) [16]. ...
Preprint
Full-text available
Due to the influence of the new coronavirus disease (COVID-19), teleworking has been expanding rapidly. Although existing interactive remote working systems are convenient, they do not allow users to adjust their spatial distance to team members at will, %"Arbitrarily" is probably not the best word here. It means without apparent reason. A better expression might be "at will." and they ignore the discomfort caused by different levels of intimacy. To solve this issue, we propose a telework support system using spatial augmented reality technology. This system calibrates the space in which videos are projected with real space and adjusts the spatial distance between users by changing the position of projections. Users can switch the projection position of the video using hand-wave gestures. We also synchronize audio according to distance to further emphasize the sense of space within the remote interaction: the distance between projection position and user is inversely proportional to the audio volume. We conducted a telework experiment and a questionnaire survey to evaluate our system. The results show that the system enables users to adjust distance according to intimacy and thus improve the users' comfort.
... We recruited 20 workers currently living in Taiwan 1 during the study period (12 women and 8 men). Prior research on remote work and remote collaboration suggests that it is possible to derive insights on remote work interactions with approximately 20 participants [30,52,53]. We approached participants by posting the call-for-participants registration form on several Facebook groups where people share their work experiences. ...
Article
Workers tend to make inferences about one another's commitment and dedication to work depending on what cues are available to them, affecting worker relationships and collaboration outcomes. In this work, we investigate how remote work affects workers' perceptions of their colleagues with different levels of social connectivity, commonly referred to as strong ties and weak ties. When working remotely, workers' perceptions of weak ties may suffer due to the lack of in-person interaction. On the other hand, workers' inferences about their strong ties may also be impacted by losing richer communication cues, even though they had more connections with their strong ties than weak ties. This study explores how remote workers make inferences about engagement levels of and willingness to collaborate with weak ties compared to strong ties. We used a mixed-methods approach involving survey data, experience sampling, and in-depth interviews with 20 workers from different companies in Taiwan. Results showed that workers depended on one-on-one synchronous tools to infer the engagement level of strong ties but used group-based communication tools to infer the engagement level of weak ties. Interestingly, the absence of cues in remote workplaces exacerbated prior impressions formed in the physical office. Furthermore, remote work led workers to develop polarized perceptions of their respective ties. We discuss how characteristics of computer-mediated communication tools and interaction types interplay to affect workers' perceptions of remote colleagues and identify design opportunities for helping remote workers maintain awareness of weak ties.
... Another lens to view collaborative interaction is through the mechanics of shared understanding, such as the task, person, and reference spaces suggested by Buxton [30]. This kind of stance is valuable for analyzing hybrid interaction mechanics in comparison and contrast with findings of more co-located (see e.g., [106,119]) or remote collaboration (see e.g., [91,149,150]). ...
Preprint
Interest in hybrid collaboration and meetings (HCM), where several co-located participants engage in coordinated work with remote participants, is gaining unprecedented momentum after the rapid shift in working from home due to the COVID-19 pandemic. However, while the interest is new, researchers have been exploring HCM phenomena for decades, albeit dispersed across diverse research traditions, using different terms, definitions, and frameworks. In this article, we present a systematic literature review of the contexts and tools of HCM in the ACM Digital Library. We obtained approximately 1,200 results, which were narrowed down to 62 key articles. We report on the terms, citations, venues, authors, domains, study types, and data of these publications and present a taxonomic overview based on their reported hybrid settings' actual characteristics. We discuss why the SLR resulted in a relatively small number of publications, and then as a corollary, discuss how some excluded high-profile publications flesh out the SLR findings to provide important additional concepts. The SLR itself covers the ACM until November 2019, so our discussion also includes relevant 2020 and 2021 publications. The end result is a baseline that researchers and designers can use in shaping the post-COVID-19 future of HCM systems.
... Since COVID-19, remote collaboration has become commonplace, and an avatar-based telepresence system using Virtual Reality (VR) and Augmented Reality (AR) Head-Mounted Displays (HMDs) is drawing attention as a means to provide immersion and copresence between users (Kim et al., 2021b, Orts-Escolano et al., 2016, Piumsomboon et al., 2018. In most cases, the virtual space accessed by the VR client is more expansive and different from the tracked space in which the user physically exists. ...
Preprint
In this study, we explore how spatial configurations can be reflected in determining the threshold range of Relative Translation Gains (RTGs), a translation gain-based Redirected Walking (RDW) technique that scales the user's movement in Virtual Reality (VR) in different ratios for width and depth. While previous works have shown that various cognitive factors or individual differences influence the RDW threshold, constructive studies investigating the impact of the environmental composition on the RDW threshold with regard to the user's visual perception were lacking. Therefore, we examined the effect of spatial configurations on the RTG threshold by analyzing the participant's responses and gaze distribution data in two user studies. The first study concerned the size of the virtual room and the existence of objects within it, and the second study focused on the combined impact of room size and the spatial layout. Our results show that three compositions of spatial configuration (size, object existence, spatial layout) significantly affect the RTG threshold range. Based on our findings, we proposed virtual space rescaling guidelines to increase the range of adjustable movable space with RTGs for developers: placing distractors in the room, setting the perceived movable space to be larger than the adjusted movable space if it's an empty room, and avoid placing objects together as centered layout. Our findings can be used to adaptively rescale VR users' space according to the target virtual space's configuration with a unified coordinate system that enables the utilization of physical objects in a virtual scene.
... Asymmetry for immersive collaboration is created by capturing the scene of a local user in AR through external cameras or transmitting the egocentric view directly to the remote user. Teleconsultation is one of the typical usecases for which such asymmetry is favourable [21,22,32,49,50,53,78], equally as for time-asynchronous consultation [28]. A disadvantage of asymmetric teleconsultation is the rigid role distribution. ...
Article
Full-text available
When two or more users attempt to collaborate in the same space with Augmented Reality, they often encounter conflicting intentions regarding the occupation of the same working area and self-positioning around such without mutual interference. Augmented Reality is a powerful tool for communicating ideas and intentions during a co-assisting task that requires multi-disciplinary expertise. To relax the constraint of physical co-location, we propose the concept of Duplicated Reality, where a digital copy of a 3D region of interest of the users’ environment is reconstructed in real-time and visualized in-situ through an Augmented Reality user interface. This enables users to remotely annotate the region of interest while being co-located with others in Augmented Reality. We perform a user study to gain an in-depth understanding of the proposed method compared to an in-situ augmentation, including collaboration, effort, awareness, usability, and the quality of the task. The result indicates almost identical objective and subjective results, except a decrease in the consulting user’s awareness of co-located users when using our method. The added benefit from duplicating the working area into a designated consulting area opens up new interaction paradigms to be further investigated for future co-located Augmented Reality collaboration systems.
... As AR-based smartglasses are evolving, they are unlocking exciting new forms of immersive communication. Prior research has explored the utility of AR mainly as a means of annotating environments [22,35,43,54], communicating with avatars [68], relaying remote instructions and guiding [30,79], visualizing biosignals [69], and rendering visual alterations [72]. However, there is a lack of research in understanding how AR serves in the context of informal communication using modern AR smartglasses. ...
Preprint
Full-text available
Augmented Reality (AR) smartglasses are increasingly regarded as the next generation personal computing platform. However, there is a lack of understanding about how to design communication systems using them. We present ARcall, a novel Augmented Reality-based real-time communication system that enables an immersive, delightful, and privacy-preserving experience between a smartphone user and a smartglasses wearer. ARcall allows a remote friend (Friend) to send and project AR content to a smartglasses wearer (Wearer). The ARcall system was designed with the practical limits of existing AR glasses in mind, including shorter battery life and a reduced field of view. We conduct a qualitative evaluation of the three main components of ARcall: Drop-In, ARaction, and Micro-Chat. Our results provide novel insights for building future AR-based communication methods, including, the importance of context priming, user control over AR content placement, and the feeling of co-presence while conversing.
... In Mixed Reality, Piumsomboon et al. (2018b) demonstrated an asymmetric collaboration between AR and VR users in a multi-scale reconstruction of the AR user's physical environment. In this research, the VR user could scale themselves into a giant and manipulate the larger virtual objects such as furniture or scale down into a miniature to interact with tabletop objects. ...
Article
Full-text available
This research explores augmented perception by investigating the effects of spatial scale manipulation in Virtual Reality (VR) to simulate multiple levels of virtual eye height (EH) and virtual interpupillary distance (IPD) of the VR users in the design context. We have developed a multiscale VR system for design applications, which supports a dynamic scaling of the VR user’s EH and IPD to simulate different perspectives of multiple user’s groups such as children or persons with disabilities. We strongly believe that VR can improve the empathy of VR users toward the individual sharing or simulating the experience. We conducted a user study comprising two within-subjects designed experiments for design-related tasks with seventeen participants who took on a designer’s role. In the first experiment, the participants performed hazards identification and risks assessment tasks in a virtual environment (VE) while experiencing four different end-user perspectives: a two-year-old child, an eight-year-old child, an adult, and an adult in a wheelchair. We hypothesized that experiencing different perspectives would lead to different design outcomes and found significant differences in the perceived level of risks, the number of identified hazards, and the average height of hazards found. The second experiment had the participants scale six virtual chairs to a suitable scale for different target end-user groups. The participants experienced three perspectives: a two-year-old child, an eight-year-old child, and an adult. We found that when the designer’s perspective matched that of the intended end-user of the product, it yielded significantly lower variance among the designs across participants and more precise scales suitable for the end-user. We also found that the EH and IPD positively correlate with the resulting scales. The key contribution of this work is the evidence to support that spatial scale manipulation of EH and IPD could be a critical tool in the design process to improve the designer’s empathy by allowing them to experience the end-user perspectives. This could influence their design, making a safer or functionally suitable design for various end-user groups with different needs.
... Piumsomboon et al. [73] developed a system with an adaptive avatar Mini-Me which uses redirected gaze and gestures to enhance remote collaboration with improved social presence. To assess the usefulness of the avatar, a scenario where a remote expert in VR assists a local worker in AR was used. ...
Article
Remote collaboration systems have become increasingly important in today’s society, especially during times where physical distancing is advised. Industry, research and individuals face the challenging task of collaborating and networking over long distances. While video and teleconferencing are already widespread, collaboration systems in augmented, virtual, and mixed reality are still a niche technology. We provide an overview of recent developments of synchronous remote collaboration systems and create a taxonomy by dividing them into three main components that form such systems: Environment , Avatars , and Interaction . A thorough overview of existing systems is given, categorising their main contributions in order to help researchers working in different fields by providing concise information about specific topics such as avatars, virtual environment, visualisation styles and interaction. The focus of this work is clearly on synchronised collaboration from a distance. A total of 87 unique systems for remote collaboration are discussed, including more than 100 publications and 25 commercial systems.
... One common use of virtual humans in AR is to allow remote communication and collaboration between people when it is not possible or is otherwise inconvenient to meet in person. While this domain has been more extensively explored with combinations of AR and virtual reality (VR) displays [6,23,37,51], several recent works have explored this context solely using AR displays [57,59,61]. For instance, the AR telepresence project Holoportation reconstructed a user's appearance in real time and presented it to other remote AR users [48]. ...
Article
Full-text available
Current optical see-through displays in the field of augmented reality are limited in their ability to display colors with low lightness in the hue, saturation, lightness (HSL) color space, causing such colors to appear transparent. This hardware limitation may add unintended bias into scenarios with virtual humans. Humans have varying skin tones including HSL colors with low lightness. When virtual humans are displayed with optical see-through devices, people with low lightness skin tones may be displayed semi-transparently while those with high lightness skin tones will be displayed more opaquely. For example, a Black avatar may appear semi-transparent in the same scene as a White avatar who will appear more opaque. We present an exploratory user study ( N=160) investigating whether differing opacity levels result in dehumanizing avatar and human faces. Results support that dehumanization occurs as opacity decreases. This suggests that in similar lighting, low lightness skin tones (e.g., Black faces) will be viewed as less human than high lightness skin tones (e.g., White faces). Additionally, the perceived emotionality of virtual human faces also predicts perceived humanness. Angry faces were seen overall as less human, and at lower opacity levels happy faces were seen as more human. Our results suggest that additional research is needed to understand the effects and interactions of emotionality and opacity on dehumanization. Further, we provide evidence that unintentional racial bias may be added when developing for optical see-through devices using virtual humans. We highlight the potential bias and discuss implications and directions for future research.
Chapter
With the COVID-19 pandemic, distributed and remote working became a necessity but in agile project management, social interactions like daily standup meetings in Scrum are vital for the project success. Mixed reality can provide a new way of combining remote collaboration with innovative 3D visualizations to analyze the project status. In this paper, we present a visual immersive analytics framework for project management (VIAProMa). It imports data from project management tools like the GitHub issue tracker for open-source projects. With these task data as the basis, it can generate three-dimensional visualizations, e.g. about the overall progress or the competences of individual developers. Developers, stakeholders and end users can meet in the collaborative virtual environment as avatars and establish a spatial structure with the task cards and visualizations. Therefore, VIAProMa with its adapted and customized mixed reality project management features supports both the shared meetings and the information flow in the project. The shared environment makes it a suitable tool for DevOpsUseXR, an extension to the DevOps workflow, where end users are able to participate in the development process in mixed reality. The resulting implementation is available as an open-source project with cross-platform capabilities targeting the Microsoft HoloLens, HTC VIVE and Android smartphones, as well as tablets. The framework is applied in university teaching classes to convey agile methodology in mixed reality programming practices.
Article
Full-text available
Cross-virtuality analytics (XVA) is a novel field of research within immersive analytics and visual analytics. A broad range of heterogeneous devices across the reality-virtuality continuum, along with respective visual metaphors and analysis techniques, are currently becoming available. The goal of XVA is to enable visual analytics that use transitional and collaborative interfaces to seamlessly integrate different devices and support multiple users. In this work, we take a closer look at XVA and analyse the existing body of work for an overview of its current state. We classify the related literature regarding ways of establishing cross-virtuality by interconnecting different stages in the reality-virtuality continuum, as well as techniques for transitioning and collaborating between the different stages. We provide insights into visualization and interaction techniques employed in current XVA systems. We report on ways of evaluating such systems, and analyse the domains where such systems are becoming available. Finally, we discuss open challenges in XVA, giving directions for future research.
Article
Full-text available
[Background] Virtual reality (VR) technology can provide unique immersive experiences for group users, and especially for analytics tasks with visual information in learning. Providing a shared control/view may improve the task performance and enhance the user experience during VR collaboration. [Objectives] Therefore, this research explores the effect of collaborative modes and user position arrangements on task performance, user engagement, and collaboration behaviors and patterns in a VR learning environment that supports immersive collaborative tasks. [Method] The study involved two collaborative modes (shared and non-shared view and control) and three position arrangements (side-by-side, corner-to-corner, and back-to-back). A user study was conducted with 30 participants divided into three groups (Single, Shared, and Non-Shared) using a VR application that allowed users to explore the structural and transformational properties of 3D geometric shapes. [Results] The results showed that the shared mode would lead to higher task performance than single users for learning analytics tasks in VR. Besides, the side-by-side position got a higher score and more favor for enhancing the collaborative experience. [Conclusion] The shared view would be more suitable for improving task performance in collaborative VR. In addition, the side-by-side position may provide a higher user experience when collaborating in learning VR. From these results, a set of guidelines for the design of collaborative visualizations for VR environments are distilled and presented at the end of the paper. All in all, although our experiment is based on a colocated setting with two users, the results are applicable to both colocated and distributed collaborative scenarios with two or more users.
Article
Virtual teachers embedded into reality enable the transfer of skills and knowledge to students by demonstration. However, authoring the animation of virtual characters aligned with real environments is difficult for educators. In this study, a novel animation authoring framework for demonstration teaching is proposed based on human–computer interaction and intelligent algorithm, and its application is demonstrated in mixed reality experimental education. We developed a simulation experiment environment; users could operate equipment using gestures by a Leap Motion controller. The interaction trajectory can be adaptively adjusted by the dynamic motion primitives algorithm to generate a virtual teacher demonstration trajectory aligned with real environments. For the details of the virtual teacher animation, the deep reinforcement learning algorithm performed the adaptive grabbing for objects of different shapes and sizes. Our experimental results showed that the framework is easy to use; users are able to easily author a natural animation of a virtual teacher performing a chemical experiment aligned with real environments. Also, the user feedback showed that the virtual teacher's demonstration animation in MR is impressive, and students can learn from the animation by observation and imitation.
Conference Paper
Full-text available
Augmented Reality (AR) embeds virtual content in physical spaces, including virtual agents that are known to exert a social presence on users. Existing design guidelines for AR rarely consider the social implications of an agent's personal space (PS) and that it can impact user behavior and arousal. We report an experiment (N=54) where participants interacted with agents in an AR art gallery scenario. When participants approached six virtual agents (i.e., two males, two females, a humanoid robot, and a pillar) to ask for directions, we found that participants respected the agents' PS and modulated interpersonal distances according to the human-like agents' perceived gender. When participants were instructed to walk through the agents, we observed heightened skin-conductance levels that indicate physiological arousal. These results are discussed in terms of proxemic theory that result in design recommendations for implementing pervasive AR experiences with virtual agents.
Article
Remote guidance on physical tasks is a type of collaboration in which a local worker is guided by a remote helper to operate on a set of physical objects. It has many applications in industrial sections such as remote maintenance and how to support this type of remote collaboration has been researched for almost three decades. Although a range of different modern computing tools and systems have been proposed, developed and used to support remote guidance in different application scenarios, it is essential to provide communication cues in a shared visual space to achieve common ground for effective communication and collaboration. In this paper, we conduct a selective review to summarize communication cues, approaches that implement the cues and their effects on augmented reality based remote guidance. We also discuss challenges and propose possible future research and development directions.
Chapter
In the last two decades a huge number of interactive and collaborative applications of Virtual Environments for designing products has been proposed. Such applications have been recommended as tools to implement Human Centered Design Approach in experiments where potential users are involved in participatory design sessions before going for production. In this kind of experiments, we observe that users are mainly involved in the validation of solutions previously elaborated by designers while in the last decade the most innovative approach in the creation of solutions seems to be moving from a user centered design to a co-design or co-creation process. Thus, it is essential to have a platform where the elements of co-creation can be fulfilled in building a successful project. The purpose of this paper is to outline the concept of co-creation and the significance of co-creation platforms alongside of proposing innovative tools for building the co-creative environments. The paper aims to layout a classification of the Extended Reality (XR) tools currently available and of their functionality as valuable means to actually embed co-design and co-creation concepts in Virtual Environments evaluating the advantages that this can bring to Industry through field studies. A review of innovative solutions like Virtual, Augmented and Mixed reality technologies is examined and drawn towards the requirements of the concept through a literature research. Additionally, a co-creative environment for designing aircraft cabin interiors is conceived and discussed a with company representative.
Article
Full-text available
An immersive virtual laboratory (VL) could offer flexibility of time and space, as well as safety, for remote students to conduct laboratory activities through online experiential learning. Recording an instructor's demonstration inside a VL is an approach that allows students to learn directly from a demonstration. However, students have to learn from a recording while controlling the playback, which requires attention spent on additional spatial and temporal cues. This additional cognitive load could lead to errors during the laboratory procedure. To address these challenges, we have identified four design requirements to reduce attention load in VLs; namely, organized learning steps, improved student sense of co-presence, reduction of task-instructor split-attention, and learning independent of interpersonal distance. Based on these requirements, we have designed and implemented spatial-temporal assistive toolsets for laboratories in a virtual environment, namely XR-LIVE, to reduce cognitive load and enhance learning in an asynchronous shared-space demonstration, implemented based on the setup of a standard civil engineering laboratory. We also analyzed students' behavior in the VL demonstration to design guidelines applicable to generic VLs.
Article
Collaborative design activities are often centered around physical artifacts. Depending on the design activity, this can be the model of a building, paper crafts, carving artwork, or a new circuit to be debugged and evaluated. In a typical setting, collaborators are seated around a table and divide their attention between the design artifact under review, at least one laptop supporting measurements and information foraging, and of course their collaborators. Although these activities involve complex sets of tools and configurations, people can easily work together when they are present in the same space. This is because the physical presence of a partner affords peripheral awareness to inform where the partner's attention is and what they are doing. This peripheral awareness allows collaborators to coordinate actions and manage coupling to achieve a shared task. For example, it is quite easy to know when your partner switches their focus from a breadboard to you as a request to start a face to face discussion.
Thesis
Durant cette dernière décennie, la quantité de données numériques n'a cessé d'augmenter. Ces données provenant de l'Internet des Objets ou de smartphones permettent aux utilisateurs de comprendre et d'analyser leur santé, leur mode de vie ou encore leur consommation énergétique, et aux différentes institutions (villes, campus, promoteurs immobiliers) de gérer le flux de population, le trafic routier ou les pertes énergétiques dans les bâtiments administrés. Le domaine de la "ville intelligente" en tire particulièrement profit. Il est donc capital de mettre à disposition des visualisations interactives de ces données pour pouvoir faire émerger des connaissances et faciliter la prise de décision. Une approche récente propose de représenter ces données proches de l'objet qui les capte ou les produit. Cependant, ce domaine est encore assez récent et il reste de nombreux défis à explorer. Un des défis majeurs identifié par Thomas et collab. (Thomas et al., 2018) est le développement et la conception de techniques d'interaction pour faciliter l'analyse de données situées. L'objectif de nos travaux de thèse est de concevoir et d'évaluer des solutions interactives tirant profit du référent physique pour interagir avec des données numériques situées en se focalisant sur une tâche interactive fondamentale en IHM : la sélection. Dans ce manuscrit nous abordons cette tâche selon une approche fonctionnelle et une approche conceptuelle.
Article
Many instructors in computing and HCI disciplines use hands-on activities for teaching and training new skills. Beyond simply teaching hands-on skills like sketching and programming, instructors also use these activities so students can acquire tacit skills. Yet, current video-conferencing technologies may not effectively support hands-on activities in online teaching contexts. To develop an understanding of the inadequacies of current video-conferencing technologies for hands-on activities, we conducted 15 interviews with university-level instructors who had quickly pivoted their use of hands-on activities to an online context during the early part of the COVID-19 pandemic. Based on our analysis, we uncovered four pedagogical goals that instructors have when using hands-on activities online and how instructors were unable to adequately address them due to the technological limitations of current video-conferencing tools. Our work provides empirical data about the challenges that many instructors experienced, and in so doing, the pedagogical goals we identify provide new requirements for video-conferencing systems to better support hands-on activities.
Thesis
Remote Collaboration using Augmented Reality (AR) shows great potential to establish a common ground in physically distributed scenarios where team-members need to achieve a shared goal. However, most research efforts in this field have been devoted to experiment with the enabling technology and propose methods to support its development. As the field evolves, evaluation and characterization of the collaborative process become an essential, but difficult endeavor, to better understand the contributions of AR. In this thesis, we conducted a critical analysis to identify the main limitations and opportunities of the field, while situating its maturity and proposing a roadmap of important research actions. Next, a human-centered design methodology was adopted, involving industrial partners to probe how AR could support their needs during remote maintenance. These outcomes were combined with literature methods into an AR-prototype and its evaluation was performed through a user study. From this, it became clear the necessity to perform a deep reflection in order to better understand the dimensions that influence and must/should be considered in Collaborative AR. Hence, a conceptual model and a human- centered taxonomy were proposed to foster systematization of perspectives. Based on the model proposed, an evaluation framework for contextualized data gathering and analysis was developed, allowing support the design and performance of distributed evaluations in a more informed and complete manner. To instantiate this vision, the CAPTURE toolkit was created, providing an additional perspective based on selected dimensions of collaboration and pre-defined measurements to obtain “in situ” data about them, which can be analyzed using an integrated visualization dashboard. The toolkit successfully supported evaluations of several team-members during tasks of remote maintenance mediated by AR. Thus, showing its versatility and potential in eliciting a comprehensive characterization of the added value of AR in real-life situations, establishing itself as a general-purpose solution, potentially applicable to a wider range of collaborative scenarios.
Conference Paper
Full-text available
In this paper, we explore techniques for enhancing remote Mixed Reality (MR) collaboration in terms of communication and interaction. We created CoVAR, a MR system for remote collaboration between an Augmented Reality (AR) and Augmented Virtuality (AV) users. Awareness cues and AV-Snap-to-AR interface were proposed for enhancing communication. Collaborative natural interaction, and AV-User-Body-Scaling were implemented for enhancing interaction. We conducted an exploratory study examining the awareness cues and the collaborative gaze, and the results showed the benefits of the proposed techniques for enhancing communication and interaction.
Conference Paper
Full-text available
We present CoVAR, a novel remote collaborative system combining Augmented Reality (AR), Virtual Reality (VR) and natural communication cues to create new types of collaboration. AR user can capture and share their local environment with a remote user in VR to collaborate on spatial tasks in shared space. COVAR supports various interaction methods to enrich collaboration, including gestures, head gaze, and eye gaze input, and provides virtual cues to improve awareness of a remote collaborator. We also demonstrate collaborative enhancements in VR user's body scaling and snapping to AR perspective.
Conference Paper
Full-text available
HCI research has demonstrated Mixed Reality (MR) as being beneficial for co-located collaborative work. For remote collaboration, however, the collaborators' visual contexts do not coincide due to their individual physical environments. The problem becomes apparent when collaborators refer to physical landmarks in their individual environments to guide each other's attention. In an experimental study with 16 dyads, we investigated how the provisioning of shared virtual landmarks (SVLs) influences communication behavior and user experience. A quantitative analysis revealed that participants used significantly less ambiguous spatial expressions and reported an improved user experience when SVLs were provided. Based on these findings and a qualitative video analysis we provide implications for the design of MRs to facilitate remote collaboration.
Conference Paper
Full-text available
We present an end-to-end system for augmented and virtual reality telepresence, called Holoportation. Our system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture and objects, using a set of new depth cameras. These 3D models can also be transmitted in real-time to remote users. This allows users wearing virtual or augmented reality displays to see, hear and interact with remote participants in 3D ,almost as if they were present in the same physical space. From an audio-visual perspective, communicating and interacting with remote users edges closer to face-to-face communication. This paper describes the Holoportation technical system in full, its key interactive capabilities, the application scenarios it enables, and an initial qualitative study of using this new communication medium.
Conference Paper
Full-text available
In this work, we investigate how remote collaboration between a local worker and a remote collaborator will change if eye fixations of the collaborator are presented to the worker. We track the collaborator's points of gaze on a monitor screen displaying a physical workspace and visualize them onto the space by a projector or through an optical see-through head-mounted display. Through a series of user studies, we have found the followings: 1) Eye fixations can serve as a fast and precise pointer to objects of the collaborator's interest. 2) Eyes and other modalities, such as hand gestures and speech, are used differently for object identification and manipulation. 3) Eyes are used for explicit instructions only when they are combined with speech. 4) The worker can predict some intentions of the collaborator such as his/her current interest and next instruction.
Conference Paper
Full-text available
" In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.
Conference Paper
Full-text available
We introduce a layout manager that exploits the robust sensing capabilities of next-generation head-worn displays by embedding virtual application windows in the user's surroundings. With the aim of allowing users to find applications quickly, our approach leverages spatial memory of a known body-centric configuration. The layout manager balances multiple constraints to keep layouts consistent across environments while observing geometric and visual features specific to each locale. We compare various constraint weighting schemas and discuss outcomes of this approach applied to models of two test environments.
Conference Paper
Full-text available
There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.
Conference Paper
Full-text available
In this paper we propose a general-purpose telepresence system design that can be adapted to a wide range of scenarios and present a framework for a proof-of-concept prototype. The prototype system allows users to see remote participants and their surroundings merged into the local environment through the use of an optical see-through head-worn display. Real-time 3D acquisition and head tracking allows the remote imagery to be seen from the correct point of view and with proper occlusion. A projector-based lighting control system permits the remote imagery to appear bright and opaque even in a lit room. Immersion can be adjusted across the VR continuum. Our approach relies only on commodity hardware; we also experiment with wider field of view custom displays.
Article
Full-text available
The feeling of presence is essential for efficient interaction within Virtual Environments (VEs). When a user is fully immersed within a VE through a large immersive display system, his/her feeling of presence can be altered because of disturbing interactions with his/her physical environment, such as collision with hardware parts of the system or loss of tracking. This alteration can be avoided by taking into account the physical features of the user as well as those of the system hardware and embedding them in the VE. Moreover, the 3D abstract representation of these physical features can also be useful for collaboration between distant users because they can make a user aware of the physical limitations of the others he/she is collaborating with. In this paper we present how we use the Immersive Interactive Virtual Cabin (IIVC) model to obtain this virtual representation of the user’s physical environment and we illustrate how this representation can be used in a collaborative navigation task in a VE. We also present how we can add 3D representations of 2D interaction tools in order to cope with asymmetrical collaborative configurations, providing 3D cues for a user to understand the actions of the others even if he/she is not fully immersed in the shared VE. Last, we briefly explain how we plan to enhance 3D interaction and collaboration by embedding a symbolic 3D user representation that will give 3D information about his/her posture.
Article
Full-text available
We present a novel technique for animating self-avatar eye movements in an immersive virtual environment without the use of eye-tracking hardware, and evaluate our technique via a two-alternative, forced-choice-with-confidence experiment that compares this simulated-eye-tracking condition to a no-eye-tracking condition and a real-eye-tracking condition in which the avatar’s eyes were rotated with an eye tracker. Viewing the reflection of a tracked self-avatar is often used in virtual-embodiment scenarios to induce in the participant the illusion that the virtual body of the self-avatar belongs to them, however current tracking methods do not account for the movements of the participants eyes, potentially lessening this body-ownership illusion. The results of our experiment indicate that, although blind to the experimental conditions, participants noticed differences between eye behaviors, and found that the real and simulated conditions represented their behavior better than the no-eye-tracking condition. Additionally, no statistical difference was found when choosing between the real and simulated conditions. These results suggest that adding eye movements to selfavatars produces a subjective increase in self-identification with the avatar due to a more complete representation of the participant’s behavior, which may be beneficial for inducing virtual embodiment, and that effective results can be obtained without the need for any specialized eye-tracking hardware.
Chapter
Full-text available
Mixed presence groupware (MPG) allows collocated and distributed teams to work together on a shared visual workspace. Presence disparity arises in MPG because it is harder to maintain awareness of remote collaborators compared to collocated collaborators. We examine the role of one’s body in collaborative work and how it affects presence disparity, articulating four design implications for embodiments in mixed presence groupware to mitigate the effects of presence disparity: embodiments should provide local feedback; they should visually portray people’s interaction with the work surface using direct input mechanisms; they should display fine-grain movement and postures of hand gestures, and they should be positioned within the workspace. We realize and evaluate these implications with VideoArms, an embodiment technique that captures and reproduces people’s arms as they work over large displays.
Conference Paper
Full-text available
As users pan and zoom, display content can disappear into off-screen space, particularly on small-screen devices. The clipping of locations, such as relevant places on a map, can make spatial cognition tasks harder. Halo is a visualization technique that supports spatial cognition by showing users the location of off-screen objects. Halo accomplishes this by surrounding off-screen objects with rings that are just large enough to reach into the border region of the display window. From the portion of the ring that is visible on-screen, users can infer the off-screen location of the object at the center of the ring. We report the results of a user study comparing Halo with an arrow-based visualization technique with respect to four types of map-based route planning tasks. When using the Halo interface, users completed tasks 16-33% faster, while there were no significant differences in error rate for three out of four tasks in our study.
Conference Paper
Full-text available
Shared phy sical workspaces allow people to maintain up - to-the minute knowledge about others' interaction with the workspace. This knowledge is workspace awareness, part of the glue that allows groups to collaborate effectively. In this paper, we present the concept of workspace awareness as a key for groupware systems that wish to support the fluid interaction evident i n face-to-face collaboration. We discuss why workspace awareness is difficult to support in groupware systems, and then present a conceptual framework that groupware designers can u se a s a starting point for thinking about and supporting awareness .
Conference Paper
Full-text available
Post-task ratings of difficulty in a usability test have the potential to provide diagnostic information and be an additional measure of user satisfaction. But the ratings need to be reliable as well as easy to use for both respondents and researchers. Three one-question rating types were compared in a study with 26 participants who attempted the same five tasks with two software applications. The types were a Likert scale, a Usability Magnitude Estimation (UME) judgment, and a Subjective Mental Effort Question (SMEQ). All three types could distinguish between the applications with 26 participants, but the Likert and SMEQ types were more sensitive with small sample sizes. Both the Likert and SMEQ types were easy to learn and quick to execute. The online version of the SMEQ question was highly correlated with other measures and had equal sensitivity to the Likert question type. Author Keywords
Article
Full-text available
We describe a design approach, Tangible Augmented Reality, for developing face-to-face collaborative Augmented Reality (AR) interfaces. Tangible Augmented Reality combines Augmented Reality techniques with Tangible User Interface elements to create interfaces in which users can interact with spatial data as easily as real objects. Tangible AR interfaces remove the separation between the real and virtual worlds, and so enhance natural face-to-face communication. We present several examples of Tangible AR interfaces and results from a user study that compares communication in a collaborative AR interface to more traditional approaches. We find that in a collaborative AR interface people use behaviours that are more similar to unmediated face-to-face collaboration than in a projection screen interface.
Article
Full-text available
Projects in the area of architectural design and urban planning typically engage several architects as well as experts from other professions. While the design and review meetings thus often involve a large number of cooperating participants, the actual design is still done by the individuals in the time in between those meetings using desktop PCs and CAD applications. A real collaborative approach to architectural design and urban planning is often limited to early paper-based sketches. In order to overcome these limitations, we designed and realized the ARTHUR system, an Augmented Re- ality (AR) enhanced round table to support complex design and planning decisions for architects. While AR has been applied to this area earlier, our approach does not try to replace the use of CAD systems but rather integrates them seamlessly into the collabora- tive AR environment. The approach is enhanced by in- Digital Peer Publishing Licence Any party may pass on this Work by electronic means and make it available for download under the terms and conditions of the current version of the Digital Peer Publishing Licence (DPPL). The text of the licence may be accessed and retrieved via Internet at http://www.dipp.nrw.de/.
Article
Full-text available
This study sought to develop and test a measure of social presence. Based on review of current definitions and measures, a synthesis of the theoretical construct that meets the criteria and dimensions [1] is proposed for a broad successful measure of social presence. An experiment was conducted to test the internal consistency and criterion validity of the measures as determined by theory, specifically the ability of the measure to distinguish levels of social presence that almost all theories suggest exist between (1) face-to-face interaction and mediated interaction, and (2) different levels of mediated interaction.
Conference Paper
Full-text available
Recently augmented reality (AR) technology has been used to develop the next generation collaborative interfaces. First results have shown the value of using AR for co-located tasks based on egocentric viewpoints. In contrast, virtual reality (VR) seems to offer interesting advantages for immersive collaborative experiences with egocentric viewpoints. In this paper we focus on a new area: a mixed collaboration between AR and VR environments. We present a new conceptual model of transitional interfaces that allow users to move between AR and VR viewpoints. We then describe the results of a quantitative evaluation with an AR exocentric viewpoint and a VR egocentric viewpoint for a navigational task. We also conducted a second experiment on the impact of the relationship between the interaction and visualization space in mixed collaboration. Results of these studies can provide a better understanding of how to design interfaces for multispace and transitional collaboration.
Conference Paper
Full-text available
Proposes seamless view-mode switching, a novel technique for supporting spatial collaboration that integrates a shared virtual environment (SVE) and a shared augmented environment (SAE) in a natural way. The technique uses two view modes: a see-through mode and a blind mode, corresponding to an SAE and an SVE, respectively. In the initial state, the view mode is the see-through one. Whenever a user changes his/her location or scaling factor by using a 3D widget, the view mode is automatically switched to the blind one. In this mode, a computer-graphics human body (avatar), which includes a head and two arms, is displayed in place of a real partner. When a user chooses the see-through mode by using the widget, the position, orientation and scaling factor are gradually changed to those of the partner, and the view mode is switched back to the see-through one. In addition to this seamless view-mode switching, we also employ two techniques for collaboration support. First, the system optionally draws a line from the center of the partner's eyes in his/her viewing direction to enhance face direction information. Second, the system employs two methods for precise calibration: a look-up table-based mechanism for correcting the distorted magnetic field, and a Kalman filter for compensating the computational delay. Owing to these techniques, users of our system can smoothly perform spatial collaboration
Conference Paper
Full-text available
Real time groupware systems often display telepointers (multiple cursors) of all participants in the shared visual workspace. Through the simple mechanism of telepointers, participants can communicate their location, movement, and probable focus of attention within the document, and can gesture over the shared view. Yet telepointers can be improved. First, they can be applied to groupware where people's view of the work surface differs-through viewport, object placement, representation variation-by mapping telepointers to the underlying objects rather than to Cartesian coordinates. Second, telepointers can be overloaded with semantic information to provide participants a stronger sense of awareness of what is going on, with little consumption of screen real estate
Conference Paper
Room2Room is a life-size telepresence system that leverages projected augmented reality to enable co-present interaction between two remote participants. Our solution recreates the experience of a face-to-face conversation by performing 3D capture of the local user with color + depth cameras and projecting their virtual copy into the remote space at life-size scale. This creates an illusion of the remote person’s physical presence in the local space, as well as a shared understanding of verbal and non-verbal cues (e.g., gaze, pointing) as if they were there. In addition to the technical details of our two prototype implementations, we contribute strategies for projecting remote participants onto physically plausible seating or standing locations, such that they form a natural and consistent conversational formation with the local participant. We also present observations and feedback from an evaluation with 7 pairs of participants on the usability of our solution for solving a collaborative, physical task.
Conference Paper
Augmented Reality (AR) applications may require the precise alignment of virtual objects to the real world. We propose automatic alignment of virtual objects to physical constraints calculated from the real world in real time ("snapping to reality"). We demonstrate SnapToReality alignment techniques that allow users to position, rotate, and scale virtual content to dynamic, real world scenes. Our proof-of-concept prototype extracts 3D edge and planar surface constraints. We furthermore discuss the unique design challenges of snapping in AR, including the user's limited field of view, noise in constraint extraction, issues with changing the view in AR, visualizing constraints, and more. We also report the results of a user study evaluating SnapToReality, confirming that aligning objects to the real world is significantly faster when assisted by snapping to dynamically extracted constraints. Perhaps more importantly, we also found that snapping in AR enables a fresh and expressive form of AR content creation.
Conference Paper
ShowMe is an immersive mobile collaboration system that allows a remote user to communicate with a peer using video, audio and hand gestures. We explore the use of a Head Mounted Display and depth camera to create a system that (1) enables a remote user to be immersed in another user's first-person's point of view and (2) offers a new way for the remote expert to provide guidance through three dimensional hand gestures and voice. Using ShowMe both users are present in the same physical environment and can perceive real-time communication from one another in the form of 2-handed gestures and voice.
Conference Paper
ImmerseBoard is a system for remote collaboration through a digital whiteboard that gives participants a 3D immersive experience, enabled only by an RGBD camera (Microsoft Kinect) mounted on the side of a large touch display. Using 3D processing of the depth images, life-sized rendering, and novel visualizations, ImmerseBoard emulates writing side-by-side on a physical whiteboard, or alternatively on a mirror. User studies involving three tasks show that compared to standard video conferencing with a digital whiteboard, ImmerseBoard provides participants with a quantitatively better ability to estimate their remote partners' eye gaze direction, gesture direction, intention, and level of agreement. Moreover, these quantitative capabilities translate qualitatively into a heightened sense of being together and a more enjoyable experience. ImmerseBoard's form factor is suitable for practical and easy installation in homes and offices.
Article
The goal of visual telecommunication has been to create a sense of 'being there' or telepresence.' This paper introduces a novel shared drawing medium called ClearBoard that goes beyond 'being there' by providing virtual shared workspace. It realizes (1) a seamless integration of shared drawing space and partner's image, and (2) eye contact to support real-time and remote collaboration by two users. We devised the key metaphor: 'talking through and drawing on a transparent glass window' to design ClearBoard. A prototype, ClearBoard-1 is implemented based on the 'Drafter-Mirror' architecture. This paper first reviews previous work on shared drawing support to clarify our design goals. We then examine three metaphors that fulfill these goals. The design requirements and the two possible system architectures of ClearBoard are described. Finally, some findings gained through the experimental use of the prototype, including the feature of 'gaze awareness,' are discussed.
Thesis
In this increasingly globalized and technological world, professionals often have to collaborate remotely on 3D content, especially in fields such as urban planning or architectural design. We propose HandOn, a portable system for collaboration on virtual 3D objects. HandsOn leverages the recent advances in optical head-mounted display and finger tracking technology, allowing users to view digital content as if it were merged with the physical reality, while being able to manipulate the objects using simple hand gestures. Supporting both co-located and remote collaboration, this proof-of-concept system is designed to be portable and easy-to-setup, in contrast with many previous systems that require elaborate setup. Our findings suggests that HandsOn greatly increases awareness of remote collaborators and their actions by displaying the remote user'-s hands while interacting with the virtual objects, and that gestural hand-based interactions felt natural to the users. We also found that the illusion of the physical and the virtual being merged improved the overall experience, even though this aspect could be further improved by using better hardware.
Article
This paper presents 3D-Board, a digital whiteboard capable of capturing life-sized virtual embodiments of geographically distributed users. When using large-scale screens for remote collaboration, awareness for the distributed users' gestures and actions is of particular importance. Our work adds to the literature on remote collaborative workspaces, it facilitates intuitive remote collaboration on large scale interactive whiteboards by preserving awareness of the full-body pose and gestures of the remote collaborator. By blending the front-facing 3D embodiment of a remote collaborator with the shared workspace, an illusion is created as if the observer was looking through the transparent whiteboard into the remote user's room. The system was tested and verified in a usability assessment, showing that 3D-Board significantly improves the effectiveness of remote collaboration on a large interactive surface.
Article
In this paper we describe a system for remote collaboration using Augmented Reality (AR) that supports view independence. The system was designed to allow a remote user to assist a local user in an object placement task. The remote helper can navigate the local user’s scene independently from the local user by using a three-dimensional reconstruction of the environment. The remote user can also place virtual annotations in the scene that the local user views through a head mounted display. The system tracks the position of key physical objects and the local user’s head pose, displaying both to the remote user. A key advantage of this system compared to other collaborative AR interfaces is that it allows the remote expert to have an independent view into the shared task space. A user study was performed using this system with the users completing a simple mostly two-dimensional object placement task. It was designed to test how the amount of remote view independence affected collaboration. Four conditions were used with varying degrees of view independence. It was found that increased view independence led to faster task completion time, more confidence from users, and a decrease in the amount of time spent communicating verbally during the task. However, varying the view independence did not significantly change the accuracy of the placement of objects. The implications of the results are discussed along with directions for future research.
Conference Paper
In tabletop work with direct input, people avoid crossing each others' arms. This natural touch avoidance has important consequences for coordination: for example, people rarely grab the same item simultaneously, and negotiate access to the workspace via turn-taking. At digital tables, however, some situations require the use of indirect input (e.g., large tables or remote participants), and in these cases, people are often represented with virtual arm embodiments. There is little information about what happens to coordination and reaching when we move from physical to digital arm embodiments. To gather this information, we carried out a controlled study of tabletop behaviour with different embodiments. We found dramatic differences in moving to a digital embodiment: people touch and cross with virtual arms far more than they do with real arms, which removes a natural coordination mechanism in tabletop work. We also show that increasing the visual realism of the embodiment does not change behaviour, but that changing the thickness has a minor effect. Our study identifies important design principles for virtual embodiments in tabletop groupware, and adds to our understanding of embodied interaction in small groups.
Conference Paper
There is currently a strong need for collaborative systems with which two or more participants interact over a distance on a task involving tangible artifacts (e.g., a machine, a patient, a tool). The present paper focuses on the specific category of remote-collaboration systems where hand gestures are used by a remote helper to assist a physically distant worker to perform manual tasks. Existing systems use a combination of video capturing, 2D monitors or 2D projectors, however displaying a video of the remote workspace and allowing helpers to gesture over the video does not provide helpers with sufficient understanding of the spatial relationships between remote objects and between their hands and the remote objects. In this paper we introduce our tele-presence Mixed Reality system for remote collaboration on physical tasks based on real-time capture and rendering of the remote workspace and of the helper's hands. We improve on previous 2D systems introducing 3D capturing and rendering, and exploiting the possibility offered by the use of real 3D data to increase the feeling of immersion offered by the system using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. We performed initial usability test of our system to verify if users are satisfied with the spatial awareness the system provides.
Conference Paper
We present BeThere, a proof-of-concept system designed to explore 3D input for mobile collaborative interactions. With BeThere, we explore 3D gestures and spatial input which allow remote users to perform a variety of virtual interactions in a local user's physical environment. Our system is completely self-contained and uses depth sensors to track the location of a user's fingers as well as to capture the 3D shape of objects in front of the sensor. We illustrate the unique capabilities of our system through a series of interactions that allow users to control and manipulate 3D virtual content. We also provide qualitative feedback from a preliminary user study which confirmed that users can complete a shared collaborative task using our system.
Conference Paper
We introduce an augmented reality referencing technique for shared environments that is designed to improve the accuracy with which one user can point out a real physical object to another user. Our technique, GARDEN (Gesturing in an Augmented Reality Depth-mapped ENvironment), is intended for use in otherwise unmodeled environments in which objects in the environment, and the hand of the user performing a selection, are interactively observed by a depth camera, and users wear tracked see-through displays. We present the results of a user study that compares GARDEN against existing augmented reality referencing techniques, as well as the use of a physical laser pointer. GARDEN performed significantly more accurately than all the comparison techniques when the participating users have sufficiently different views of the scene, and significantly more accurately than one of these techniques when the participating users have similar perspectives.
Article
Preference testing refers to consumer tests in which the consumer is given a choice and asked to indicate their most liked product, usually from a pair. Although these tests appear straightforward and simple, several complications are encountered in the methods, notably how to treat replicated data and how to analyze data that include a “no-preference” option as a response. Additional methods are discussed including ranking more than two products, choosing both the best and worst from a group, and rating the degree of preference.
Article
Large surfaces such as tabletop and whiteboard displays naturally afford collocated collaboration, where multiple people work together over the shared workspace. As large digital displays become more ubiquitous, it becomes increasingly important to examine their role in supporting groups of distributed collaborators working over the digital work surface. In particular, Mixed Presence Groupware (MPG) is software that connects both collocated and distributed collaborators and their disparate displays via a common shared virtual workspace. We have built several MPG systems by connecting several distributed displays, each with multiple input devices, thereby connecting both collocated and distributed collaborators. By observing how these systems are used, we found that MPG presents a unique problem called presenc1e disparity: collaborators focus their energies on collocated collaborators at the expense of their distributed counterparts. Presence disparity arises because the physical presence of collaborators varies across the MPG workgroup: physically collocated collaborators are seen in full fid