Figure 2 - uploaded by Deepak Akkil
Content may be subject to copyright.
Source publication
We present GazeTorch, a novel interface that provides gaze awareness during remote collaboration on physical tasks. GazeTorch uses a spotlight to display gaze information of the remote helper on the physical task space of the worker. We conducted a preliminary user study to evaluate user's subjective opinion on the quality of collaboration, using G...
Citations
... For example, D'Angelo and Begel (2017) developed a system where a pair of programmers were shown what the other was looking at while they worked, and they found providing this shared gaze information aids in coordination and effective communication. Moreover, Akkil et al. (2016) developed a shared gaze interface called GazeTorch which facilitated the collaboration in physical tasks. Several other studies found that shared gaze improved performance and remote collaboration in several domains (Lee et al., 2017). ...
Cross-recurrence quantification analysis (CRQA) metrics may offer a means to provide information about the quality of collaboration in real-time. The goal of the present work is to use Area of Interest (AOI) based CRQA metrics to analyze the eye-tracking data of 10 pairs who participated in a shared unmanned aerial vehicle (UAV) command and control task. We are interested in how teams respond to workload transitions and how it affects AOI-based CRQA metrics. The results showed that as workload increased, team members spent a longer time on the same task which may indicate that they are coordinating together on a task, or they are not adapting and getting “trapped” in certain tasks. The findings suggest that CRQA AOIbased metrics are sensitive to workload changes and validate these metrics in unraveling the visual puzzle of how workload impacts scanpath patterns which contribute to quantifying the adaptation process of pairs over time. This also has the potential to inform the design of real-time technology in the future.
... D'Angelo and Begel (2017) developed a system where programmers were shown what the other was looking at while they worked, and they found providing this shared gaze information aids in coordination and effective communication. GazeTorch, a shared gaze interface developed by Akkil et al. (2016), was also found to make collaboration more effortless. Several other studies found that shared gaze improved performance and remote collaboration in teleconferencing (Gupta et al., 2016), video conferencing systems (Lee et al., 2017), problem solving (Schneider & Pea, 2013), collaborative visual search (Siirtola et al., 2019), and competitive and cooperative online gaming (Maurer et al., 2018). ...
Teamwork and collaboration form the cornerstones of organizational performance and success. It is important to understand how the attention allocation of team members is linked to performance. One approach to studying attention allocation in a team context is to compare the scanpath similarity of two people working in teams and to explore the link between scanpath similarity and team performance. In this study, participants were recruited to work in pairs on an unmanned aerial vehicle (UAV) task that included low and high workload conditions. An eye tracker was used to collect the eye movements of both participants in each team. The scanpaths of two teammates were compared in low and high workload conditions using MultiMatch, an established scanpath comparison algorithm. The obtained scanpath similarity values were correlated with performance measures of response time and accuracy. Several MultiMatch measures showed significant strong correlations across multiple dimensions, providing insight into team behavior and attention allocation. The results suggested that the more similar each team member’s scanpath is, the better their performance. Additional research and consideration of experimental variables will be necessary to further understand how best to use MultiMatch for scanpath similarity assessment in complex domains.
... This includes 2D graphic overlays on 2D views, 3D graphic overlays in MR, and projections onto the physical world in projected AR. Prior works have found improved collaborative outcomes when gestures are combined with visualizations of the helper's eye gaze [4,5], the worker's eye gaze [31,78], or viewing direction [33,63]. Other interesting communication cues include virtual replicas of task objects [61] or human avatars to provide non-verbal cues in MR [65]. ...
We investigate how robotic camera systems can offer new capabilities to computer-supported cooperative work through the design, development, and evaluation of a prototype system called Periscope. With Periscope, a local worker completes manipulation tasks with guidance from a remote helper who observes the workspace through a camera mounted on a semi-autonomous robotic arm that is co-located with the worker. Our key insight is that the helper, the worker, and the robot should all share responsibility of the camera view--an approach we call shared camera control. Using this approach, we present a set of modes that distribute the control of the camera between the human collaborators and the autonomous robot depending on task needs. We demonstrate the system's utility and the promise of shared camera control through a preliminary study where 12 dyads collaboratively worked on assembly tasks. Finally, we discuss design and research implications of our work for future robotic camera systems that facilitate remote collaboration.
... Additionally, many studies have used localized agents [23,24] and visual communication cues [25,26] with remote users. Teo et al. [1][2][3] investigated adding light pointing and drawing annotations to gestures as nonverbal communication cues in a real-time 360 panorama-based MR remote collaboration system and found that participating local users were able to perform tasks faster and with fewer errors with the help of visual annotation cues and that having a variety of visual cues significantly improved social presence. ...
In mixed-reality (MR) telecollaboration, the local environment is remotely presented to a remote user wearing a virtual reality (VR) head-mounted display (HMD) via a video capture device. However, remote users frequently face challenges in naturally and actively manipulating their viewpoints. In this paper, we propose a telepresence system with viewpoint control, which involves a robotic arm equipped with a stereo camera in the local environment. This system enables remote users to actively and flexibly observe the local environment by moving their heads to manipulate the robotic arm. Additionally, to solve the problem of the limited field of view of the stereo camera and limited movement range of the robotic arm, we propose a 3D reconstruction method combined with a stereo video field-of-view enhancement technique to guide remote users to move within the movement range of the robotic arm and provide them with a larger range of local environment perception. Finally, a mixed-reality telecollaboration prototype was built, and two user studies were conducted to evaluate the overall system. User study A evaluated the interaction efficiency, system usability, workload, copresence, and user satisfaction of our system from the remote user’s perspective, and the results showed that our system can effectively improve the interaction efficiency while achieving a better user experience than two traditional view-sharing techniques based on 360 video and based on the local user’s first-person view. User study B evaluated our MR telecollaboration system prototype from both the remote-user side and the local-user side as a whole, providing directions and suggestions for the subsequent design and improvement of our mixed-reality telecollaboration system.
... Some previous studies acknowledged this issue and even researched on finding feasible solutions for resolving this issue. For instance, Li et al. [114] introduced a trace augmentation method to trace the user's cursor movement; Prouzeau et al. [143] proposed Awareness Bards, Focus Map, and Step Map to assist the transition between personal and shared workspaces; Li et al. [116] and Akkil et al. [6] tried to introduced gaze awareness for user to better collaborate for remote physical tasks. ...
Traditional wall-sized displays mostly only support side-by-side co-located collaboration, while transparent displays naturally support face-to-face interaction. Many previous works assume transparent displays support collaboration. Yet it is unknown how exactly its afforded face-to-face interaction can support loose or close collaboration, especially compared to the side-by-side configuration offered by traditional large displays. In this paper, we used an established experimental task that operationalizes different collaboration coupling and layout locality, to compare pairs of participants collaborating side-by-side versus face-to-face in each collaborative situation. We compared quantitative measures and collected interview and observation data to further illustrate and explain our observed user behavior patterns. The results showed that the unique face-to-face collaboration brought by transparent display can result in more efficient task performance, different territorial behavior, and both positive and negative collaborative factors. Our findings provided empirical understanding about the collaborative experience supported by wall-sized transparent displays and shed light on its future design.
... In addition, XR can enable gestures and gaze-based features much more seamlessly and effectively than in other communication platforms, potentially facilitating implicit intention recognition along the way (Anton et al., 2018;Huang et al., 2015;Orts-Escolano et al., 2016;Pan et al., 2018;Tomasello et al., 2007). Even though gaze-based interactions and eye contact in XR are far from established, and their benefits are debated (Li et al., 2016), there is some early evidence that intention recognition might be already feasible (Akkil et al., 2016;Piumsomboon et al., 2017). ...
There is evidence in literature that collaborative work while using digital tools could benefit from visualizing the real time eye movements of a selected participant, or possibly, several participants. In this study, we examine alternative gaze interaction and visualization design prototypes in a digital collaboration scenario, in which assumed collaboration environment is a co-located mixed reality environment. Specifically, we implemented a virtual pointer as a baseline, and representations of gaze as a line, a cursor, and an ‘automated line’ where the line and cursor are automatically alternated based on occlusion detection. These prototypes are then evaluated in a series of usability studies with additional exploratory observations for a spatial communication scenario. In the scenario participants either describe routes to someone else or learn them from someone else for navigational planning. In this paper we describe the alternative interaction design prototypes, as well as various visualization designs for the gaze itself (continuous line and dashed line) and the point of regard (donut, dashed donut, sphere, rectangle) to guide collaboration and report our findings from several usability studies (n=6). We also interviewed our participants which allows us to make some qualitative observations on the potential function and usefulness of these visualization and interaction prototypes. Overall, the outcomes suggest that gaze visualization solutions in general are promising approaches to assist communication in collaborative XR, although, not surprisingly, how they are designed is important.
... Akkil, et al. [26] presented a novel remote guidance interface, in which the helper's eye gaze is projected onto the physical task space of the worker and displayed as spotlight. This effectively transfers gaze information into a pointer, guiding the worker into the object or an area that the gaze is pointing to. ...
Remote guidance on physical tasks is a type of collaboration in which a local worker is guided by a remote helper to operate on a set of physical objects. It has many applications in industrial sections such as remote maintenance and how to support this type of remote collaboration has been researched for almost three decades. Although a range of different modern computing tools and systems have been proposed, developed and used to support remote guidance in different application scenarios, it is essential to provide communication cues in a shared visual space to achieve common ground for effective communication and collaboration. In this paper, we conduct a selective review to summarize communication cues, approaches that implement the cues and their effects on augmented reality based remote guidance. We also discuss challenges and propose possible future research and development directions.
... Therefore, we take full advantage of combining gestures and gaze in order to develop a novel multimodal interaction for an AR remote collaborative platform. This research is motivated by earlier research [13][14][15][16] and builds upon it by combining 2.5D gestures, gaze, and SAR. Hence, the contributions of the present study fall into three areas: ...
... However, they shared the gaze-based cues from the local site. Akkil et al. [14] developed a SAR remote collaborative system called GazeTorch, which provides the remote user' gaze awareness for the local user. Recently, continuing in this exploratory direction, they investigated the influence of gaze in video-based remote collaboration for a physical task, compared with the POINTER interface (e.g., a mouse-controlled pointer) [16]. ...
... In our current study, we simplified the system to concentrate on the core elements using co-located collaboration to emulate remote collaboration, as in many prior related systems [9,10,12,14,15,19,30] and in several publications reported in the review of related work [1]. There were some technologies that we needed to further investigate. ...
Although sharing gestures and gaze can improve AR remote collaboration, most current systems only enable collaborators to share 2D or 3D gestures, and the unimodal HCI interface remains dominant. To address this problem, we describe a novel remote collaborative platform based on 2.5D gestures and gaze (2.5DGG), which supports an expert who collaborates with a worker (e.g., during assembly or training tasks). We investigate the impact of sharing the remote site’s 2.5DGG using spatial AR (SAR) remote collaboration in manufacturing. Compared to other systems, there is a key advantage that it can provide more natural and intuitive multimodal interaction based on 2.5DGG. We track the remote experts’ gestures and eye gaze using Leap Motion and aGlass, respectively, in a VR space displaying the live video stream of the local physical workspace and visualize them onto the local work scenario by a projector. The results of an exploratory user study demonstrate that 2.5DGG has a clear difference in performance time and collaborative experience, and it is better than the traditional one.
... In particular, it may be possible to handle the problem by traditional video-mediated remote collaboration when figuring out complex tasks (e.g., inspection, repair or maintenance) [35]. However, compared with video-conferencing for physical tasks, AR/MR-based remote collaboration can support more natural and intuitive interaction [10,21,36,37], improve the performance time [19,21,24,38,39], provide more engaging user experience [16,21,24,40,41], and share some important non-verbal cues of collocated work (e.g., annotations [26,28,42,43], user's emotions [17,[44][45][46], gaze [19,39,47] and gestures [10,36,48]), which have been shown to increase empathy, trust, productivity and collaboration [17,21,44,45,49]. Examples of AR/MR remote collaborative professional scenarios include remote technical support services [14,16,21,23,50,51], assembly and disassembly of complex parts [10,21,36,52], production and inspection processes [9,30,52,53], Co-design [54,55], through-life maintenance assistance [14,27,30,36,[56][57][58][59], emergency repair guidance [10,20,23,34,36,50,56], telemedicine [14,20,22,[60][61][62][63], tele-education [20,64], training [9,10,[21][22][23]57]. ...
... In particular, it may be possible to handle the problem by traditional video-mediated remote collaboration when figuring out complex tasks (e.g., inspection, repair or maintenance) [35]. However, compared with video-conferencing for physical tasks, AR/MR-based remote collaboration can support more natural and intuitive interaction [10,21,36,37], improve the performance time [19,21,24,38,39], provide more engaging user experience [16,21,24,40,41], and share some important non-verbal cues of collocated work (e.g., annotations [26,28,42,43], user's emotions [17,[44][45][46], gaze [19,39,47] and gestures [10,36,48]), which have been shown to increase empathy, trust, productivity and collaboration [17,21,44,45,49]. Examples of AR/MR remote collaborative professional scenarios include remote technical support services [14,16,21,23,50,51], assembly and disassembly of complex parts [10,21,36,52], production and inspection processes [9,30,52,53], Co-design [54,55], through-life maintenance assistance [14,27,30,36,[56][57][58][59], emergency repair guidance [10,20,23,34,36,50,56], telemedicine [14,20,22,[60][61][62][63], tele-education [20,64], training [9,10,[21][22][23]57]. ...
... These advantages of AR and MR have attracted academic researchers and industrial developers to explore remote collaborative applications for physical tasks. Thus a lot of remote collaborative architectures, models, methods, systems have been proposed and developed to improve the user experience and task performance by sharing a live panorama, ARA, CP, avatar, gestures and gaze such as JackIn [97,98], Showme [48], Remot-IO [99], 2.5DHANDS [10], RemoteFusion [100], GazeTorch [39], Mini-Me [24], BeThere [101], TeleAdvisor [23,34,50], 3D helping hands [37], HandsIn3D [36], Room2Room [93], TeleAR [54], DigiTable [102], MTA [103], SEMarbeta [104], Vishnu [4], RemoteBob [105], AlphaRead [106], ReMa [56], etc. ...
This paper provides a review of research into using Augmented Reality (AR) and Mixed Reality(MR) for remote collaboration on physical tasks. AR/MR-based remote collaboration on physical tasks has recently become more prominent in academic research and engineering applications. It has great potential in many fields, such as real-time remote medical consultation, education, training, maintenance, remote assistance in engineering, and other remote collaborative tasks. However, to the best of our knowledge there has not been any comprehensive review of research in AR/MR remote collaboration on physical tasks. Therefore, this paper presents a comprehensive survey of research between 2000 and 2018 in this domain. We collected 215 papers, more than 80% of which were published between 2010 and 2018, and all relevant works are discussed at length. Then we elaborate on the review from typical architectures, applications (e.g., industry, telemedicine, architecture, teleducation and others), and empathic computing. Next, we made an in-depth review of the papers from seven aspects: (1) collection and classification research, (2) using 3D scene reconstruction environments and live panorama, (3) periodicals and conducting research, (4) local and remote user interfaces, (5) features of user interfaces commonly used, (6) architecture and sharing non-verbal cues, (7) applications and toolkits. We find that most papers (160 articles, 74.4%) are published in conferences, using co-located collaboration to emulate remote collaboration is adopted by more than half (126, 58.6%) of the reviewed papers, the shared non-verbal cues can be mainly classified into five types (Virtual Replicas or Physical Proxy(VRP), AR Annotations or a Cursor Pointer(ARACP), avatar, gesture, and gaze), the local/remote interface is mainly divided into four categories (Head-Mounted Displays(HMD), Spatial Augmented Reality(SAR), Windows-Icon-Menu-Pointer(WIMP) and Hand-Held Displays(HHD)). From this, we can draw ten conclusions. Following this we report on issues for future works. The paper also provides an overall academic roadmap and useful insight into the state-of-the-art of AR/MR remote collaboration on physical tasks. This work will be useful for current and future researchers who are interested in collaborative AR/MR systems.
... The spotlight metaphor was inspired by current approaches in theater and cinematography, in which camera recorders often highlight audience responses to capture the most relevant moments [13,14], draw attention to elements on screen, and potentially evoke emotional responses [15]. Beyond cinematic experiences, the spotlight technique has also been used in computer applications to direct and maintain users' attention while simultaneously making surrounding context still visible [16][17][18][19]. Moreover, Khan et al. [20] showed that spotlighting can also help manage users' attention between their primary activity and peripheral information, even on small monitors. ...
The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N = 175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N = 117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.
