Article

A survey of visual, mixed, and augmented reality gaming

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

Visual mixed and augmented realities have historically been applied to the gaming application domain. This article provides a survey of visual mixed and augmented reality gaming in both the academic and commercial contexts. There is an exploration of both indoor and outdoor mixed and augmented reality gaming. The different games are presented via the three major display technologies: head-mounted display, handheld display, and spatial immersive display. A number of academic mixed and augmented reality research projects are described that provide an overview of the current state of the art. A set of example commercial games are also examined to provide the context for the state of the games on the market.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... XR is an emerging technology that simulates realistic environments for users. XR techniques have provided revolutionary user experiences in various application scenarios (e.g., training [19,21], education [38], product/architecture design [46], gaming [41], remote conferencing/tours [24,45], etc.). According to a recent report from MarketsandMarkets Research [5], the XR market size is expected to reach USD 125.2 billion by 2026 from USD 33.0 billion in 2021, at a Compound Annual Growth Rate (CAGR) of 30.6% during the forecast period. ...
Preprint
Extended Reality (XR) includes Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). XR is an emerging technology that simulates a realistic environment for users. XR techniques have provided revolutionary user experiences in various application scenarios (e.g., training, education, product/architecture design, gaming, remote conference/tour, etc.). Due to the high computational cost of rendering real-time animation in limited-resource devices and constant interaction with user activity, XR applications often face performance bottlenecks, and these bottlenecks create a negative impact on the user experience of XR software. Thus, performance optimization plays an essential role in many industry-standard XR applications. Even though identifying performance bottlenecks in traditional software (e.g., desktop applications) is a widely explored topic, those approaches cannot be directly applied within XR software due to the different nature of XR applications. Moreover, XR applications developed in different frameworks such as Unity and Unreal Engine show different performance bottleneck patterns and thus, bottleneck patterns of Unity projects can't be applied for Unreal Engine (UE)-based XR projects. To fill the knowledge gap for XR performance optimizations of Unreal Engine-based XR projects, we present the first empirical study on performance optimizations from seven UE XR projects, 78 UE XR discussion issues and three sources of UE documentation. Our analysis identified 14 types of performance bugs, including 12 types of bugs related to UE settings issues and two types of CPP source code-related issues. To further assist developers in detecting performance bugs based on the identified bug patterns, we also developed a static analyzer, UEPerfAnalyzer, that can detect performance bugs in both configuration files and source code.
... AR can enhance people's perception of and interaction with their surroundings and can also help them more easily perform real-world tasks [2]. Over the past few decades, technological advancements have significantly increased the adoption of AR technology in a wide range of application domains, including industrial maintenance [3,1,4,5], education [6][7][8], gaming [9,10], and collaborative work [11][12][13][14][15]. ...
... Differently, the optical see-through devices consist of an optical combiner or holographic waveguides, the lenses, that enable the overlay of images transmitted by a projector over the same lenses through which a normal visualization of the real world is allowed. In this way the user visualizes directly the reality augmented with the VOs overlaid onto it [7,19]. The different techniques are summarized in Figure 3. ...
Article
Full-text available
Abstract: Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used.We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.
... According to Kruse, (2008), [12] Gagné's nine events of instruction model focus on the outcomes or behaviours from training. Besides, this model has contributed significantly to instructional technology, especially in designing web-based instruction [13]. Hence, Gagne's nine instructional events are considered compatible with adult behaviours and adult learning styles Ference & Vockell (1994) [14] agreed that Gagne's events of instruction are compatible with adult learners characteristics. ...
Article
Full-text available
Game has been proven an effective approach tool to improve learning and has become a new tool for training delivery. The development of the gamification framework involves integrating design processes which are input, process, and output. The design phase is considered essential to guide the flow of the gamification framework. It offers a safer, interactive, and entertaining learning environment for construction-related workers. This paper aims to report on the approach to design a gamification framework for hazard identification training in Construction using Garris's Input-Process-Outcome game model as the basis. It focuses on the three main design elements: instructional design, game characteristic, and user characteristic. The study outlines two objectives: (1) to identify the game's attributes and Gagne's Nine Events Instructional Methods Design which supports effective learning, and (2) to determine the user's characteristics of self-directed learning. This study focused on designing the Design phase, which consists of three main elements, i.e. instructional design, game characteristics and user characteristics. Mixed methods were used to extract the attributes and elements of the Design phase. Content analysis was carried out to determine the model of instructional design and game attributes. The findings show 12 attributes of the game through content analysis methods. Gagne's Nine Events Instructional Methods Design can support effective learning. Meanwhile, a questionnaire survey was subsequently administered to determine the user's ability to self-directed learning and decision-making style, where 319 construction-related workers responded. Data were analysed using mean comparison. The results showed that construction-related workers belong to the independent learners' category and are inclined to 'vigilant' and 'brooding' types of decision-making style. Findings confirmed that construction-related workers belong to the independent learners' category and are inclined to 'vigilant' and 'brooding' types of decision-making style. Following the aim of this paper, these findings were incorporated into the design phase of the game framework.
... Since the 2010s resurgence of interest in consumer-grade virtual reality (VR) [48], VR has successfully established itself on the consumer market [47]. However, as with any emerging technology, differences can exist between how industry / academia perceive a technology and how consumers perceive it [49,50]. ...
... N order to give a user of virtual reality (VR) system highly immersion, to freely interact between a user and the virtual environment, accurate and low latency tracking is one of the most important requirements for the VR system [1]. This accurate and fast tracking is expected to be used in variety of applications not only for entertainment [2] but also for medicine [3], engineering and design [4], military [5], education [6], virtual prototyping [7], and architecture and cultural heritage [8]. To meet the high requirements of tracking, lighthouse localization system of HTC Vive use inertial measurements and light data from the inertial measurement unit (IMU) and the photodiodes in the tracked object like HMD, controller, and tracker [9]. ...
Article
Full-text available
The lighthouse localization system has recently been developed and used for localization in virtual reality (VR). Not only for VR but also for a general indoor positioning systems (IPSs) it has several advantages over existing methods, including low cost, wide detection area, and easy setup. Here, we adopt the stereo configuration of a lighthouse for improved sensing performance and propose a novel calibration method for stereo configuration. For the stereo calibration, the exact positions of sufficient corresponding points in two sensor coordinates need to be determined. A printed checkerboard is widely used for stereo camera systems because it is easy to construct and its accuracy is guaranteed owing to its printing accuracy. However, in the case of the lighthouse system, it is very difficult or impossible to construct a highly accurate calibration board similar to the checkerboard mainly because of manufacturing errors. In this study, we use a receiver sensor and a two-axis linear stage equipped with micrometers. By moving predetermined distances along the x and y directions on the linear stage, we can obtain multiple-point information with high accuracy, which can then be used for the stereo calibration of two lighthouses. In this paper, the calibration and pose estimation procedures are described in detail, and the pose estimation result of the perspective-n-points method is compared with that of the triangulation method. Finally, the pose estimation accuracy of the proposed system is compared with that of a commercial system that is widely used for highly accurate medical applications.
... This paper will then evaluate four AR apps in detail according to five criteria and highlight their educational potential, including ideas for classroom activities that embrace and enhance their use. (Milgram et al., 1995, p. 283) It is a form of mixed reality where digital graphical information is superimposed on a user's view of the physical world (Bower et al., 2014;Thomas, 2012), In essence, AR "uses displays, tracking, and other technologies to enhance (augment) the user's view of a realworld environment with synthetic objects or information" (LaViola et al., 2017, p. 8). Some of the distinguishing characteristics of AR include: ...
Article
Empirical studies of augmented reality (AR) in education have suggested a vast range of educational benefits, including deep learning (Wu et al., 2013), collaborative learning in locative storytelling (Chinthammit & Thomas, 2014), higher-order thinking skills (Bower et al., 2015), increased student engagement in play-based literacy practices (Yamada-Rice et al., 2017), 21st century skills (Wang et al., 2018), and spatial thinking (George et al., 2020). Yet, in language and literacy teaching, little is known about using AR to enhance inquiry learning, and to encourage students to experiment with affordances of new media to develop critical and creative knowledge. Although teachers are expected to support students in learning, both with and through new digital forms, there is a slow uptake of new media, such as AR, in Australian contexts. This is possibly due to a lack of mainstream understanding about what AR is and what educational affordances it offers. The lack of classroom research and a “conceptual framework regarding the implementation of technologies such as Augmented Reality system” remain an impeding factor to effective explorations of AR in education (Bower et al., 2014, p.7).
... Nonetheless, the fields that have recently boosted its potential are virtual reality (VR) and augmented reality (AR). Indeed, the potential applications of AR/VR technology to a multitude of sectors such as online education [5], healthcare [6,7], entertainment [8,9], communication [10,11] and/or gaming industry [12,13] have created an ever-growing demand for more realistic and immersive AR/VR experiences. ...
Article
Full-text available
This paper summarizes the OpenEDS 2020 Challenge dataset, the proposed baselines, and results obtained by the top three winners of each competition: (1) Gaze prediction Challenge, with the goal of predicting the gaze vector 1 to 5 frames into the future based on a sequence of previous eye images, and (2) Sparse Temporal Semantic Segmentation Challenge, with the goal of using temporal information to propagate semantic eye labels to contiguous eye image frames. Both competitions were based on the OpenEDS2020 dataset, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display with two synchronized eye-facing cameras. The dataset, which we make publicly available for the research community, consists of 87 subjects performing several gaze-elicited tasks, and is divided into 2 subsets, one for each competition task. The proposed baselines, based on deep learning approaches, obtained an average angular error of 5.37 degrees for gaze prediction, and a mean intersection over union score (mIoU) of 84.1% for semantic segmentation. The winning solutions were able to outperform the baselines, obtaining up to 3.17 degrees for the former task and 95.2% mIoU for the latter.
... This brings supplementary opportunities for appealing game-plays involving interaction with real world objects (e.g. hiding behind real walls from attackers) and people [167], [168]. Additionally, AR visualizations are not concerned with many of the challenges linked to VR gaming, including avoiding real-world obstacles (e.g. ...
Article
Full-text available
Virtual self-avatars have been increasingly used in Augmented Reality (AR) where one can see virtual content embedded into physical space. However, little is known about the perception of self-avatars in such a context. The possibility that their embodiment could be achieved in a similar way as in Virtual Reality opens the door to numerous applications in education, communication, entertainment, or the medical field. This article aims to review the literature covering the embodiment of virtual self-avatars in AR. Our goal is (i) to guide readers through the different options and challenges linked to the implementation of AR embodiment systems, (ii) to provide a better understanding of AR embodiment perception by classifying the existing knowledge, and (iii) to offer insight on future research topics and trends for AR and avatar research. To do so, we introduce a taxonomy of virtual embodiment experiences by defining a "body avatarization" continuum. The presented knowledge suggests that the sense of embodiment evolves in the same way in AR as in other settings, but this possibility has yet to be fully investigated. We suggest that, whilst it is yet to be well understood, the embodiment of avatars has a promising future in AR and conclude by discussing possible directions for research.
... Accordingly, the selected developer engine shall support as many platforms as possible. Due to the further application possibilities, it could also be used to create virtual or mixed reality applications, as well as serve simulations and other experiments [71]. The virtual model of ZalaZONE is publicly available in several data formats under MIT License for further research and evaluation. ...
Article
Full-text available
Testing self-driving vehicles is still a new and immature process; the globally harmonised procedure expected much later. The resource-demanding nature of real-world tests makes it indispensable to develop and improve the efficiency of virtual environment based testing methods. Accordingly, a novel X-in-the-Loop framework is proposed to fully exploit the recent advances in info-communication technologies, vehicle automation, and testing and validation requirements. This methodology real-time connects physical and virtual testing with high correlation while completely blurs the sharp boundaries between them. Measurement results confirm the superior performance of the 5G communication link in providing a stable, real-time connection between the real world and its virtual representation. The live demonstration proved the presented concept at the newly constructed Hungarian proving ground for automated driving. The performed investigation also includes comprehensive benchmarking, focusing on the most up-to-date automotive testing frameworks. The analysis considers the methodologies and techniques applied by the most relevant actors in the automotive testing sector worldwide. Accordingly, the newly developed testing framework is evaluated and validated in light of the state-of-the-art methods used by the automotive industry.
... With the rapid development of virtual reality technology, virtual reality systems appear in various places in life. 1 Existing consumer head-mounted displays use a completely immersive design without visual cues from the surrounding real environment. Due to the loss of the external visual information during the virtual reality (VR) experience, users are prone to a certain degree of insecurity, which affects the VR experience. ...
Article
Full-text available
During the immersive virtual reality experience, because the visual senses are completely enclosed in the virtual environment, they are unable to perceive environmental changes in the real world and produce unsafe feelings, which affects their experience in virtual reality systems. In order to improve people's sense of security in the immersive virtual reality experience, the paper has designed four ways to interact with the real world in a virtual environment, so that users can get real‐world information in the virtual environment. This paper tests users' changes in their sense of security during immersive virtual reality experiences under the influence of various interaction methods and conduct psychological analysis. Twenty‐one volunteers are recruited to participate in the experiment, and their personal safety and psychological effects are tested. The experimental results show that the method of using Yolcat neural network to segment the captured images according to categories and then fuse the segmented images into the virtual environment can improve the user's sense of security without destroying the immersion.
... In contrast to these VR examples, challenges are different for enabling co-located interaction in AR settings. AR technology has been applied to a wide variety of leisure and gaming activities [52], e.g., with the widely popular games Ingress [41] and Pokemon Go [42], for augmenting and game balancing table tennis [1,2] and foosball gameplay [47], or for enabling digitally augmented tabletop board games [25]. These approaches use AR technologies to provide additional benefts for users, but were not applied to HMD-based systems that use AR technology in general and therefore have unique challenge, such as one-sided visualization of AR content for the HMD Users resulting in exclusion of bystanders. ...
Conference Paper
Full-text available
Head-Mounted Displays (HMDs) are the dominant form of enabling Virtual Reality (VR) and Augmented Reality (AR) for personal use. One of the biggest challenges of HMDs is the exclusion of people in the vicinity, such as friends or family. While recent research on asymmetric interaction for VR HMDs has contributed to solving this problem in the VR domain, AR HMDs come with similar but also different problems, such as conflicting information in visualization through the HMD and projection. In this work, we propose ShARe, a modified AR HMD combined with a projector that can display augmented content onto planar surfaces to include the outside users (non-HMD users). To combat the challenge of conflicting visualization between augmented and projected content, ShARe visually aligns the content presented through the AR HMD with the projected content using an internal calibration procedure and a servo motor. Using marker tracking, non-HMD users are able to interact with the projected content using touch and gestures. To further explore the arising design space, we implemented three types of applications (collaborative game, competitive game, and external visualization). ShARe is a proof-of-concept system that showcases how AR HMDs can facilitate interaction with outside users to combat exclusion and instead foster rich, enjoyable social interactions.
... Head-mounted displays (HMDs) for Virtual Reality (VR) are finally available for the consumer market. Today, consumers mainly use VR for entertainment applications including 3D movies and games [16]. A large field of view (FOV), high visual fidelity as well as the visual and auditory encapsulation can create truly immersive experiences with almost unlimited opportunities. ...
Research
Full-text available
Entering text is one of the most common tasks when interacting with computing systems. Virtual Reality (VR) presents a challenge as neither the user's hands nor the physical input devices are directly visible. Hence, conventional desktop peripherals are very slow, imprecise, and cumbersome. We want to develop an apparatus that tracks the user's hands, and a physical keyboard, and visualize them in VR. In a text input study with 32 participants, Pascal Knierim [1] investigated the achievable text entry speed and the effect of hand representations and transparency on typing performance, workload, and presence. With that apparatus, experienced typists benefited from seeing their hands, and reach almost outside-VR performance. Inexperienced typists profited from semi-transparent hands, which enabled them to type just 5.6 WPM slower than with a regular desktop setup. We conclude that optimizing the visualization of hands in VR is important, especially for inexperienced typists, to enable a high typing performance.
... Our paper believes that the more natural integration of AR technology with the traditional escape room can apply more interactive and imaginative narratives to help achieve the "total immersion." As shown in Fig. 1, augmented reality could register the virtual object generated by computer through the physical world (Thomas 2012). Augmented reality makes users experience a new scene where realistic scene and virtual object are seamlessly blended together (Billinghurst et al. 2015). ...
Article
Full-text available
Escape room is a live-action adventure game, where the players search clues, solve puzzles and achieve the assigned tasks. This paper proposed a novel escape room system combining augmented reality and deep learning technology. The system adopts a client–server architecture and can be divided into the server module, the smart glasses module and the player–hardware interaction module. The player–hardware interaction module consists of subsystems each of which includes a Raspberry Pi 3. HoloLens is used as the smart glasses in the experiment of the paper. The server communicates with all the Raspberry Pis and HoloLens through TCP/IP protocol and manages all the devices to achieve the game flow by following the process timeline. The smart glasses module provides two display modes, i.e., the AR 3D models display and the 2D text clues display. In the first mode, the SDK Vuforia is used for detection and tracking of markers. In the second mode, the scene images captured by HoloLens camera are sent to the pre-trained image classifier based on deep convolutional neural network. Considering both the image category and the game status value, the server decides the text clue image to be displayed on HoloLens. The accuracy of the image classification model reaches 94.9%, which can be correctly classified for a certain rotation angle and partial occlusion. The integration of AR, deep learning, electronics and escape room games opens up exciting new directions for the development of escape room. Finally, a built mini-escape room is analyzed to prove that the proposed system can support more complicated narratives showing the potential of achieving immersion.
... A place-based analysis was presented by Graham et al. [71]. Thomas [184] surveys the area from a technical perspective. A detailed psychological study of motivations to play a successful location based AR game is presented by Zsila et al [206], who argue that a loss of a sense of reality and a competitive motivation lead to problematic behavior of players. ...
Article
Full-text available
With the CoViD-19 pandemic, location awareness technologies have seen renewed interests due to the numerous contact tracking mobile application variants developed, deployed, and discussed. For some, location-aware applications are primarily a producer of geospatial Big Data required for vital geospatial analysis and visualization of the spread of the disease in a state of emergency. For others, comprehensive tracking of citizens constitutes a dangerous violation of fundamental rights. Commercial web-based location-aware applications both collect data and-through spatial analysis and connection to services-provide value to users. This value is what motivates users to share increasingly private and comprehensive data. The willingness of users to share data in return for services has been a key concern with web-based variants of the technology since the beginning. With a focus on two privacy preserving CoViD-19 contact tracking applications, this survey walks through the key steps of developing a privacy preserving context-aware application: from types of applications and business models, through architectures and privacy strategies, to representations.
... Augmented Reality (AR) is another exciting field and there are several recent developments [3,7] that potentially transform the way people play games [31,59] or experience online shopping [55,63]. The most popular mobile platforms (e.g. ...
... Based on several studies show that AR has a better market share than virtual reality. AR applications for fields called aCommerce are the second most popular in AR applications [8] - [10]. We make AR implementations for IAP systems based on paper markers. ...
Article
Full-text available
— Different arrangement in selling freemium programming application is the In-App Purchase (IAP) technique, where clients can download and enjoy the application for free; however, there are premium highlights that must be paid. We use Augmented Reality (AR) innovation into the IAP framework in the buying system of the game application, Heroes of Surabaya (HOS). The application allows user keeps run and play on the Android mobile system, but it will require AR marker cards to enable the premium feature on it. This investigation plans to test the effectiveness of AR marker cards that proposed as a substitute for computerized cash on the purchasing system of the game application. From the experimental results, the distance of the marker and the light factor are the main factors for the success-rate of the detection of the AR marker.
... Nonetheless, the fields that have recently boosted its potential are virtual reality (VR) and augmented reality (AR). Indeed, the potential applications of AR/VR technology to a multitude of sectors such as online education [5], healthcare [9], [11], entertainment [8], [19], communication [21], [10] and/or gaming industry [22], [12] have created an evergrowing demand of more realistic and immersive AR/VR experiences. ...
Preprint
Full-text available
We present the second edition of OpenEDS dataset, OpenEDS2020, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display mounted with two synchronized eye-facing cameras. The dataset, which is anonymized to remove any personally identifiable information on participants, consists of 80 participants of varied appearance performing several gaze-elicited tasks, and is divided in two subsets: 1) Gaze Prediction Dataset, with up to 66,560 sequences containing 550,400 eye-images and respective gaze vectors, created to foster research in spatio-temporal gaze estimation and prediction approaches; and 2) Eye Segmentation Dataset, consisting of 200 sequences sampled at 5 Hz, with up to 29,500 images, of which 5% contain a semantic segmentation label, devised to encourage the use of temporal information to propagate labels to contiguous frames. Baseline experiments have been evaluated on OpenEDS2020, one for each task, with average angular error of 5.37 degrees when performing gaze prediction on 1 to 5 frames into the future, and a mean intersection over union score of 84.1% for semantic segmentation. As its predecessor, OpenEDS dataset, we anticipate that this new dataset will continue creating opportunities to researchers in eye tracking, machine learning and computer vision communities, to advance the state of the art for virtual reality applications. The dataset is available for download upon request at http://research.fb.com/programs/openeds-2020-challenge/.
... AR (Augmented Reality) VR (Virtual Reality) [1]. HMD ...
Conference Paper
Full-text available
一般的な既存のVR (Virtual Reality)用HMD (Head Mounted Display) は高い没入感や臨場感をユーザに与えるが,垂直視野角は人間の元来のものと比べると狭い.特に水平から下方向の視野角は,HMDを装着しての卓上での作業や歩行などの日常的な動作に重要であるが,人間本来の65〜70度程度の垂直視野角に対して35〜45度程度しか提示できていない.そこで我々は,既存のHMD内部の鉛直下方向に対し下方視野用のLCDを新しく追加し,下方視野を拡大したHMDの開発と評価を行った.作成したHMDには臨場感の向上を目的としたVRモードと,外部に取り付けたカメラの映像をLCDに投影することで卓上作業の作業効率化と首運動の負担軽減を目的としたVST(Video See-through)モードを実装した. 結果,下方視野を拡大したHMDは垂直視野角を約70度拡大させ,臨場感向上と首の負担軽減に成功した.一方,情報量の増加による集中の阻害や映像提示の不完全さによる作業時間増加も確認され,さらなる研究が必要であることが示唆された.
... AR has seen a remarkable evolution with applications in many areas [5,45], such as gaming, e-commerce, mobile computing and, recently, home entertainment and iTV [22,47,49,56,57]. For example, a review by Thomas [45] on AR gaming reported implementations on many platforms, such as tabletops, see-through displays, wearable computers, hand-held devices, helmets, and systems based on video projections, revealing an accelerating field of research that is moving towards ubiquitous consumer AR systems [7]. ...
... Both variants are typically interfaced to the user via a head-mounted display (HMD). Applications leveraging VR and AR mainly fall into the categories of information (e.g., in retail [151] or for tourist guidance [32]), education [152], and entertainment [153]. Common to all of them is that computationally heavy tasks need to be carried out in order to render and understand a scene. ...
Article
Full-text available
Edge computing brings computing and storage resources closer to (mobile) end users and data sources, thus bypassing expensive and slow links to distant cloud computing infrastructures. Often leveraged opportunistically, these heterogeneous resources can be used to offload data and computations, enabling upcoming demanding applications such as augmented reality and autonomous driving. Research in this direction has addressed various challenges, from architectural concerns to runtime optimizations. As of today, however, we lack a widespread availability of edge computing—partly because it remains unclear which of the promised benefits of edge computing are relevant for what types of applications. This article provides a comprehensive snapshot of the current edge computing landscape, with a focus on the application perspective. We outline the characteristics of edge computing and its postulated benefits and drawbacks. To understand the functional composition of applications, we first define common application components that are relevant w.r.t. edge computing. We then present a classification of proposed use cases and analyze them according to their expected benefits from edge computing and which components they use. Furthermore, we illustrate existing products and industry solutions that have recently surfaced and outline future research challenges.
... Over the last years, virtual reality (VR) received increased attention from various fields such as medicine, engineering, education, design, and entertainment. Head-mounted displays (HMDs) for consumers enable more and more people become interested in VR and increase the demand for applications and content [46]. One of the core challenges of VR is transporting people to another place and inducing presence -the illusion of acting and being 'there' even when one is physically situated in another place. ...
Conference Paper
Full-text available
Virtual reality (VR) is becoming more and more ubiquitous to interact with digital content and often requires renderings of avatars as they enable improved spatial localization and high levels of presence. Previous work shows that visual-haptic integration of virtual avatars depends on body ownership and spatial localization in VR. However, there are different conclusions about how and which stimuli of the own appearance are integrated into the own body scheme. In this work, we investigate if systematic changes of model and texture of a users' avatar affect the input performance measured in a two-dimensional Fitts' law target selection task. Interestingly, we found that the throughput remained constant between our conditions and that neither model nor texture of the avatar significantly affected the average duration to complete the task even when participants felt different levels of presence and body ownership. In line with previous work, we found that the illusion of virtual limb-ownership does not necessarily correlate to the degree to which vision and haptics are integrated into the own body scheme. Our work supports findings indicating that body ownership and spatial localization are potentially independent mechanisms in visual-haptic integration.
... The key idea of these markers is that they are easy to find in a captured (usually color) image, with relatively low false positives, and based on prior information we can know the three-dimensional positions that corresponds to the image points. Localizing these markers, and its dual problem, localizing the camera based on these markers, are very common challenges in augmented reality (AR) applications [2], [3]. Recently, platforms that allow for reliable markerless tracking [4] have reached wide acceptance, especially after the entrance of technological giants such as Apple ARKit [5], Google ARCore [6], or Microsoft Hololens [7]. ...
Conference Paper
Full-text available
We introduce an algorithm to refine the estimation of corners and pose of square fiducial markers, such as Arucos, with focus on mobile augmented reality applications. The idea is to reduce pixel jitter, which causes distracting artifacts such as "vibrating" objects, by exploiting information from the contour pixels of the detected markers. To achieve this, we develop a nonlinear least squares estimator that models a marker explicitly as a polygon and employs ideas from shape fitting. This provides not only a best-fitting estimate of the corners, but also a covariance matrix that can be used during further processing. We also implement a pose estimator that incorporates these covariance matrices and show how the effect of pixel jitter is greatly reduced in our approach, without increasing resource usage substantially in mobile devices.
... This was a focused review to provide a snapshot of work over this period, where we systematically searched for relevant papers from primary conference proceedings such as CHI, ISMAR and CSCW. We supplemented these with other papers we were aware of, and those cited in related literature surveys [7,66,67,68,69]. ...
Article
Full-text available
Collaborative Mixed Reality (MR) systems are at a critical point in time as they are soon to become more commonplace. However, MR technology has only recently matured to the point where researchers can focus deeply on the nuances of supporting collaboration, rather than needing to focus on creating the enabling technology. In parallel, but largely independently, the field of Computer Supported Cooperative Work (CSCW) has focused on the fundamental concerns that underlie human communication and collaboration over the past 30-plus years. Since MR research is now on the brink of moving into the real world, we reflect on three decades of collaborative MR research and try to reconcile it with existing theory from CSCW, to help position MR researchers to pursue fruitful directions for their work. To do this, we review the history of collaborative MR systems, investigating how the common taxonomies and frameworks in CSCW and MR research can be applied to existing work on collaborative MR systems, exploring where they have fallen behind, and look for new ways to describe current trends. Through identifying emergent trends, we suggest future directions for MR, and also find where CSCW researchers can explore new theory that more fully represents the future of working, playing and being with others.
... Augmented Reality (AR) is a novel technology that permits to display a virtual layer of information or graphic objects (or both) on the top or a real world captured through a vision-based device camera [56]. AR has been largely employed [57] to enhance cultural heritage experiences [58], to enrich gaming experiences [59] , to improve education performance [60], to increase the quality of life of elderly [61], as well as to handle chronic health conditions [62]. ...
Article
Full-text available
In the last decades, given the necessity of assisting fragile citizens, of which elderly represent a significant portion, a considerable research effort has been devoted to the use of information and communication technologies (ICT) in daily living to promote activity, social connections, and independence. With similar purposes, in recent years psychologists proposed the novel paradigm of Positive Psychology (PP), the scientific study of positive human functioning and flourishing on multiple levels. The joint effort between ICT and PP has led to the definition of the emerging field of Positive Technology (PT), with the aim of developing technology consciously designed to foster well-being in individuals and groups. In this paper we review PT focusing on frameworks involving computer vision and machine learning for promoting cognitive, physical, emotional and social elderly well-being. Our discussion highlights a significant gap between theoretical needs and technological systems availability, suggesting future lines of research.
... Based on several studies shows that AR has a better market share compared to virtual reality. AR-app for commercial fields which called aCommerce is the second most popular in AR-app [8]- [10]. We are inspired by aCommerce from IKEA which utilizes AR technology in their sales catalog. ...
... Augmented Reality (AR) is another exciting field and there are several recent developments [3,7] that potentially transform the way people play games [31,59] or experience online shopping [55,63]. The most popular mobile platforms (e.g. ...
Preprint
Full-text available
Shopping is difficult for people with motor impairments. This includes online shopping. Proprietary software can emulate mouse and keyboard via head tracking. However, such a solution is not common for smartphones. Unlike desktop and laptop computers, they are also much easier to carry indoors and outdoors.To address this, we implement and open source button that is sensitive to head movements tracked from the front camera of iPhone X. This allows developers to integrate in eCommerce applications easily without requiring specialized knowledge. Other applications include gaming and use in hands-free situations such as during cooking, auto-repair. We built a sample online shopping application that allows users to easily browse between items from various categories and take relevant action just by head movements. We present results of user studies on this sample application and also include sensitivity studies based on two independent tests performed at 3 different distances to the screen.
... We have seen how technological and psychological immersion has been studied for virtual environments [10], telepresence [94], telecommunications, and gaming [95]. A fundamental question for immersive analytics is how to modify these theories to identify and formalise the various kinds of immersion and engagement that might occur during data analytics and decision making. ...
... More and more people are spending a large amount of time using virtual reality systems for work and entertainment [20,21], which makes the study on working in a virtual environment (VE) become imperative [17]. In this way, a good input device should be much important for virtual reality systems based on head-mounted displays (HMDs) for both life and entertainment. ...
Conference Paper
Full-text available
Text entry is an imperative issue to be addressed in current entry systems for virtual environments (VEs). The entry method using a physical keyboard is still the most dominant choice for an efficient interaction regarding text entry. In this paper, we propose a typing system with a style of mixed reality, which is called HiKeyb, and it possesses a similar high-efficiency with the single physical keyboard in the real environment. The HiKeyb system consists of a depth camera, a pose tracking module, a head-mounted display (HMD), a QWERTY keyboard and a black table mat. This system can guarantee the entry efficiency and the amenity by not only introducing the force feedback from a movable physical keyboard, but also improving the immersion with the real hand image. In addition, the infrared absorption material helps improve the robustness of the system against different lighting environments. Experiments have proved that users wearing HMDs in Virtual Phrases session can achieve an entry rate of 23.1 words per minute and an error rate of 2.76%, and the rate ratio of virtual reality to real world is 78% when typing phrases. Besides, we find that the proposed system can provide a relatively close entry efficiency to that using a pure physical keyboard in the real environment.
... We have seen how technological and psychological immersion has been studied for virtual environments [10], telepresence [94], telecommunications, and gaming [95]. A fundamental question for immersive analytics is how to modify these theories to identify and formalise the various kinds of immersion and engagement that might occur during data analytics and decision making. ...
Chapter
Immersive Analytics is a new research initiative that aims to remove barriers between people, their data and the tools they use for analysis and decision making. Here we clarify the aims of immersive analytics research, its opportunities and historical context, as well as providing a broad research agenda for the field. In addition, we review how the term immersion has been used to refer to both technological and psychological immersion, both of which are central to immersive analytics research.
... In this way, the level of Presence is fundamental to understanding the extension of which the subject perceives the scenario as a real world experience, even though the origin and nature of this variable are still not clear. Mixed Reality (AR and VR) Presence enabling technologies (Thomas, 2012;Kapp et al., 2013;Slater, 2014;Papagiannakis et al., 2015Papagiannakis et al., , 2018 have only recently started being investigated in Serious Games applications (Kateros et al., 2015) for digital heritage (Egges et al., 2007;Kapp et al., 2013;Abrash, 2014) in terms of entertainment (Markouzis and Fessakis, 2015), education: cognition and knowledge transfer (Capuano et al., 2016), as well as learning (Pollalis et al., 2018), gamification Zikas et al., 2016), engagement (Di Pietro et al., 2017), emotions (Harley et al., 2016), and enjoyment (Sylaiou et al., 2010, Brancati et al., 2015. ...
Article
Full-text available
It is well known that the Decadent movement in European literature (fin de siècle) depends on the narrative of the antiquity, as it is revealed from the discoveries of archaeology in the second half of the 19th century. Amid the ruins of the past authors, painters and poets re-conceptualize time and history through a modernist vision based on an imaginary reconfiguration of the antiquity. In this context, the myth of a city (Pompeii) or of a woman (Salomé) offer examples that, which would illustrate in a great variety the synergy of a multi-temporal and multi-cultural memory of the myth. In this paper we identify a “content-based” shortcoming of modern Mixed Reality (MR) intangible and tangible digital heritage storytelling applications for digital humanities. It is an important problem as the very nature of these applications is has often been identified with either misguided storytelling, or non-compelling, non-engaging narratives, except the initial captivating moments due to the immersive 3D visual simulation. We propose a new concept that forthcoming MR applications can draw from: “Literature-based MR Presence”. Based on modern literature excerpts associated with the real heritage sites, digital narratives can achieve new depths of Presence (phenomenon of behaving and feeling, as if we are in the virtual/augmented world created by computerized displays). They would evoke deeper sensations if their dramaturgical plots were based on literary texts associated with the heritage sites, from users, as similar to those often associated with cognitive presence, e.g. when someone is feeling of being transported in an alternate reality when simply reading a compelling novel or poem. We examine modern MR simulations and serious games for digital heritage and propose this conceptual framework to study them under this new concept, in order to achieve heightened feeling of Presence in the virtual heritage simulations, based on recent novel h/w advances. Two cases of a tangible historical place (Pompeii) and an intangible character (Salome) are identified as cultural heritage items, with associated reconstruction examples via Mixed Reality simulations and corresponding early modern literary works.
... We witness a proliferation of Augmented Reality (AR) applications such as games [Thomas 2012] or utilities [Hartl et al. 2013] across device classes (e.g., tablets, smartphones, smartglasses) and operating systems. Software Development Kits (SDKs) such as Vuforia, Apple ARKit, Google ARCore or the Windows Mixed Reality Toolit allow for efficient creation of AR applications for individual platforms through spatial tracking components with 6 degrees of freedom (DoF) pose estimation. ...
Article
Computer Vision-based natural feature tracking is at the core of modern Augmented Reality applications. Still, Web-based Augmented Reality typically relies on location-based sensing (using GPS and orientation sensors) or marker-based approaches to solve the pose estimation problem. We present an implementation and evaluation of an efficient natural feature tracking pipeline for standard Web browsers using HTML5 and WebAssembly. Our system can track image targets at real-time frame rates tablet PCs (up to 60 Hz) and smartphones (up to 25 Hz).
... We witness a proliferation of Augmented Reality (AR) applications such as games [Thomas 2012] or utilities [Hartl et al. 2013] across device classes (e.g., tablets, smartphones, smartglasses) and operating systems. Software Development Kits (SDKs) such as Vuforia, Apple ARKit, Google ARCore or the Windows Mixed Reality Toolit allow for efficient creation of AR applications for individual platforms through spatial tracking components with 6 degrees of freedom (DoF) pose estimation. ...
Conference Paper
Full-text available
Computer Vision-based natural feature tracking is at the core of modern Augmented Reality applications. Still, Web-based Augmented Reality typically relies on location-based sensing (using GPS and orientation sensors) or marker-based approaches to solve the pose estimation problem. We present an implementation and evaluation of an efficient natural feature tracking pipeline for standard Web browsers using HTML5 and WebAssembly. Our system can track image targets at real-time frame rates tablet PCs (up to 60 Hz) and smartphones (up to 25 Hz).
... Augmented reality has been used in many different areas related to gaming [39], and has been investigated in research through AR systems for a variety of traditional sports or leisure activities. For example, AR was used in a squash court to attract a new audience to the sport, provide enhanced training capabilities, and support a novel game experience within the context of squash [19]. ...
Conference Paper
When player skill levels are not matched, games provide an unsatisfying player experience. Player balancing is used across many digital game genres to address this, but has not been studied for co-located augmented reality (AR) tabletop games, where using boosts and handicaps can adjust for different player skill levels. In the setting of an AR table football game, we studied the importance of game balancing being triggered by the game system or the players, and whether player skill should be required to trigger game balancing. We implemented projected icons to prominently display game balancing mechanics in the AR table football game. In a within-subjects study (N=24), we found players prefer skill-based control over game balancing and that different triggers are perceived as having different fairness. Further, the study showed that even game balancing that is perceived as unfair can provide enjoyable game experiences. Based on our findings, we provide suggestions for player balancing in AR tabletop games.
Article
Full-text available
The usefulness of a web site is based on the prospect of a satisfactory website user. In improving the usefulness of a university’s website, it is essential to present precise data and disseminate prompt information to students. This paper was conducted to analyse and recommend a better process for the information display (content and work processes) of university websites development. Questionnaires were distributed and feedbacks were received from 350 respondents. The aim of the study was to evaluate the effectiveness of the process of (content and processes) of the university websites based on five main aspects: i) Content and Organization Structure, ii) Linkages and Navigation, iii) Language, iv) Education Content and v) Option and Performance. Analysis showed that the percentage of question items has a positive tendency in perception regarding improvements need to enhance the intensity of the university website portal, especially in terms of links, content organization and live help desk requirements that can directly provide an easier means of that can providing information and accelerating up the operation of receiving data. In summary, this study also outlines the proposal for academic websites display to increase the serviceability of the site to users.
Chapter
Viele Mitarbeiterinnen (In diesem Kapitel wird durchgehend die grammatikalisch weibliche Bezeichnung verwendet. Damit sind alle Geschlechter gemeint) im Dienstleistungsbereich sehen sich täglich Herausforderungen gegenüber, welche durch Indoor-Navigation beeinflusst werden. Krankenschwestern müssen ihre Routen anpassen, um auf plötzliche Patientenanfragen reagieren zu können, während Wartungsarbeiterinnen oft Maschinen in unbekannten Räumlichkeiten finden müssen. Dies führt zu Stress und ineffizienten Routing-Entscheidungen. Unter Verwendung der Design-Science-Research-Methode wurde eine Augmented-Reality-Anwendung für die Microsoft HoloLens entwickelt, welche in der Lage ist Benutzerinnen in Gebäuden zu navigieren und Routing-Informationen als erweiterte virtuelle Pfeile an kritischen Punkten anzuzeigen. In einer Nutzerstudie (n = 20) zeigen wir, dass bei der Nutzung von Routing basierend auf virtuellen Pfeilen ein geringerer geistiger Aufwand als bei konventionellen 2D-Visualisierungen erforderlich ist, die vor allem durch Systeme wie Google Maps oder TomTom bekannt sind.
Chapter
Augmented Reality (AR) is a field of knowledge that emerged in the middle of the last century, and its use has been spreading because of its usefulness, but also because of mobile platforms, accessible to most users. AR characteristics are valued in several fields of human activity, and also in the field of Education and Training, being AR pointed out as useful to the learning process. In this paper we search and analyse surveys and reviews of AR. We present a AR’s definition, and we create a classification scheme of two dimensions for AR: the dimension of the fields of application of AR, and the dimension of the technologies of AR.
Article
Augmented Reality (AR) offers the possibility to enrich the real world with digital mediated content, increasing in this way the quality of many everyday experiences. While in some research areas such as cultural heritage, tourism, or medicine there is a strong technological investment, AR for game purposes struggles to become a widespread commercial application. In this article, a novel framework for AR kid games is proposed, already developed by the authors for other AR applications such as Cultural Heritage and Arts. In particular, the framework includes different layers such as the development of a series of AR kid puzzle games in an intermediate structure which can be used as a standard for different applications development, the development of a smart configuration tool, together with general guidelines and long-life usage tests and metrics. The proposed application is designed for augmenting the puzzle experience, but can be easily extended to other AR gaming applications. Once the user has assembled the real puzzle, AR functionality within the mobile application can be unlocked, bringing to life puzzle characters, creating a seamless game that merges AR interactions with the puzzle reality. The main goals and benefits of this framework can be seen in the development of a novel set of AR tests and metrics in the pre-release phase (in order to help the commercial launch and developers), and in the release phase by introducing the measures for long-life app optimization, usage tests and hint on final users together with a measure to design policy, providing a method for automatic testing of quality and popularity improvements. Moreover, smart configuration tools, as part of the general framework, enabling multi-app and eventually also multi-user development, have been proposed, facilitating the serialization of the applications. Results were obtained from a large-scale user test with about 4 million users on a set of eight gaming applications, providing the scientific community a workflow for implicit quantitative analysis in AR gaming. Different data analytics developed on the data collected by the framework prove that the proposed approach is affordable and reliable for long-life testing and optimization.
Article
Dynamic Adaptive Streaming over HTTP (DASH) has been widely used by several popular streaming services, such as YouTube, Netflix, and Facebook. Adopting DASH requires to pre-determine a set of encoding configurations, called encoding ladder, to generate a set of representations stored on the streaming server. These representations are adaptively requested by clients according to their network conditions during streaming sessions. In this article, we aim to solve the optimal laddering problem that determines the optimal encoding ladder to maximize the client viewing quality. In particular, we consider video models, viewing probability, and client distribution to formulate the mathematical problem. We use divide-and-conquer approach to decompose the problem into two subproblems: (i) per-class optimization for clients with different bandwidths and (ii) global optimization to maximize the overall viewing quality under the storage limit of the streaming server. We propose two algorithms for each of the per-class optimization and global optimization problems. Analytical analysis and real experiments are conducted to evaluate the performance of our proposed algorithms, compared to other state-of-the-art algorithms. Based on the results, we recommend a combination of the proposed algorithms to solve the optimal laddering problem. The evaluation results show the merits of our recommended algorithms, which: (i) outperform the state-of-the-art algorithms by up to 52.17 and 26.35 in Viewport Video Multi-Method Assessment Fusion (V-VMAF) in per-class optimization, (ii) outperform the state-of-the-art algorithms by up to 43.14 in V-VMAF for optimal laddering in global optimization, (iii) achieve good scalability under different storage limits and number of bandwidth classes, and (iv) run faster than the state-of-the-art algorithms.
Chapter
At the dawn of the 4th Industrial Revolution, the field of computer science-based technologies is growing continuously and rapidly, developing in both itself and towards applications of many other disciplines. The book at hand aims at exposing its reader to some of the most significant advances in core computer science-based technologies. As such, the book is directed towards professors, researchers, scientists, engineers and students in computer science-related disciplines. It is also directed towards readers who come from other disciplines and are interested in becoming versed in some of the most recent computer science-based technologies. An extensive list of bibliographic references at the end of each chapter guides the reader to probe further into the application areas of interest to him/her.
Article
Digital twin (DT) is an emerging concept that is gaining attention in various industries. It refers to the ability to clone a physical object (PO) into a software counterpart. The softwarized object, termed logical object, reflects all the important properties and characteristics of the original object within a specific application context. To fully determine the expected properties of the DT, this article surveys the state-of-the-art starting from the original definition within the manufacturing industry. It takes into account related proposals emerging in other fields, namely augmented and virtual reality (e.g., avatars), multiagent systems, and virtualization. This survey thereby allows for the identification of an extensive set of DT features that point to the "softwarization" of POs. To properly consolidate a shared DT definition, a set of foundational properties is identified and proposed as a common ground outlining the essential characteristics (must-haves) of a DT. Once the DT definition has been consolidated, its technical and business value is discussed in terms of applicability and opportunities. Four application scenarios illustrate how the DT concept can be used and how some industries are applying it. The scenarios also lead to a generic DT architectural model. This analysis is then complemented by the identification of software architecture models and guidelines in order to present a general functional framework for the DT. This article, eventually, analyses a set of possible evolution paths for the DT considering its possible usage as a major enabler for the softwarization process.
Article
Head-mounted displays and 360° videos have become increasingly more popular, delivering a more immersive viewing experience to end users. Streaming 360° videos over the best-effort Internet, however, faces tremendous challenges, because of the high resolution and the short response time requirements. This survey presents the current literature related to 360° video streaming. We start with 360° video streaming systems built for real experiments to investigate the practicality and efficiency of 360° video streaming. We then present the video and viewer datasets, which may be used to drive large-scale simulations and experiments. Different optimization tools in various stages of the 360° video streaming pipeline are discussed in detail. We also present various applications enabled by 360° video streaming. In the appendices, we review the off-the-shelf hardware available at the time of writing and the open research problems.
Article
Full-text available
In the first quarter of the new millennium, the immersive technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are only a few steps away from becoming the mainstream tools within the design industry. This study investigated the internal and external barriers of technology adoption within design-oriented businesses. A mixed method was used to collect and analyze the data from the employees of a large design firm. This research confirmed that external barriers such as funding, technical support, training, and business strategy that exist at the organizational level are interrelated with the internal barriers such as designers’ and managers’ perception and attitude toward the new technologies. The managerial applications were discussed later and the directions for future research were provided.
Article
Full-text available
Augmented reality artwork is an emerging field of art, including grand transformations of entire building facades that have been carried out by use of detailed 3D models and customised to the building's unique surface. The extension of projectors into the realm of small mobile devices affords an opportunity to extend such AR art into the personal scale of the individual, allowing the user to customise and transform their surroundings on an ad hoc basis. In this paper, a system is proposed for the modification and augmentation of a mobile user's surroundings is put forward, together with the technical challenges such a system raises.
Article
Full-text available
This demo shows BattleBoard 3D which is an Augmented Reality (AR) based game prototype featuring the use of LEGO for the physical and digital pieces. Design concepts, the physical setting and, user interface for the game is illustrated and described. Based on qualitative studies of children playing the game we illustrate design issues for AR board games.
Article
Full-text available
An abstract is not available.
Article
Full-text available
The VR Studio was founded in 1992 to explore the potential of Virtual Reality technology for theme park attractions. This paper presents an overview of the VR Studio's history, from the location-based entertainment attractions developed for DisneyQuest, to research in using virtual reality technology for theme park design. The goal is to present many of the lessons learned during 10 years of building interactive virtual worlds. In particular, the paper will focus on the challenge of creating location-based virtual reality attractions for the mass audience.
Article
Full-text available
The Historic Oakland Cemetery in downtown Atlanta provides a unique setting for exploring the challenges of location-based mixed-reality experience design. Our objective is to entertain and educate visitors about historically and culturally significant events related to the deceased inhabitants of the cemetery. We worked with the constraints and affordances of the physical environment of the cemetery to design an audio-based dramatic experience. The dramatic narrative is realized through voice actors who play the parts of cemetery residents and tell stories about the time periods in which they lived. The experience provides navigation and linearity through a main narrator who guides visitors to various gravesites. While at each grave, the visitor can choose from several categories of content using a handheld controller. Formative evaluations conducted with users in the cemetery indicate strengths of the current experience and suggest ideas for continued development.
Article
Full-text available
Human Pacman is a novel interactive entertainment system that ventures to embed the natural physical world seamlessly with a fantasy virtual playground by capitalizing on mobile computing, wireless LAN, ubiquitous computing, and motion-tracking technologies. Our human Pacman research is a physical role-playing augmented-reality computer fantasy together with real human–social and mobile gaming. It emphasizes collaboration and competition between players in a wide outdoor physical area which allows natural wide-area human–physical movements. Pacmen and Ghosts are now real human players in the real world, experiencing mixed computer graphics fantasy–reality provided by using the wearable computers. Virtual cookies and actual tangible physical objects are incorporated into the game play to provide novel experiences of seamless transitions between real and virtual worlds. We believe human Pacman is pioneering a new form of gaming that anchors on physicality, mobility, social interaction, and ubiquitous computing.
Article
Full-text available
We describe a software framework for rapidly developing and deploying self-contained, multi-user Augmented Reality applications on a variety of commercially available handheld computers (PDAs).
Article
Full-text available
Mixed Reality (MR) visual displays, a particular subset of Virtual Reality (VR) related technologies, involve the merging of real and virtual worlds somewhere along the 'virtuality continuum' which connects completely real environments to completely virtual ones. Augmented Reality (AR), probably the best known of these, refers to all cases in which the display of an otherwise real environment is augmented by means of virtual (computer graphic) objects. The converse case on the virtuality continuum is therefore Augmented Virtuality (AV). Six classes of hybrid MR display environments are identified. However quite different groupings are possible and this demonstrates the need for an efficient taxonomy, or classification framework, according to which essential differences can be identified. An approximately three-dimensional taxonomy is proposed comprising the following dimensions: extent of world knowledge, reproduction fidelity, and extent of presence metaphor.
Article
Full-text available
In this paper we present the results of a qualitative, empirical study exploring the impact of immersive technologies on presence and engagement, using the interactive drama Façade as the object of study. In this drama, players are situated in a married couple's apartment, and interact primarily through conversation with the characters and manipulation of objects in the space. We present participants' experiences across three different versions of Façade – augmented reality (AR) and two desktop computing based implementations, one where players communicate using speech and the other using typed keyboard input. Through interviews and observations of players, we find that immersive AR can create an increased sense of presence, confirming generally held expectations. However, we demonstrate that increased presence does not necessarily lead to more engagement. Rather, mediation may be necessary for some players to fully engage with certain interactive media experiences.
Article
Full-text available
This paper presents the design of pOwerball, a novel augmented reality computer game for children aged 8-14. The pOwerball was designed to bring together children with and without a physical or learning disability and to encourage social interactions surrounding the play. The contribution of this design case is two fold. From a design perspective, pOwerball exemplifies an emerging class of computer games where the interaction style and game mechanics support social interactions amongst the players. From a methodological perspective, we describe the various ways children became involved in our design process; we highlight the related difficulties and successes in the context of relevant research literature.
Article
Full-text available
Augmented reality (AR) makes it possible to create games in which virtual objects are overlaid on the real world, and real ob-jects are tracked and used to control virtual ones. We describe the development of an AR racing game created by modifying an ex-isting racing game, using an AR infrastructure that we developed for use with the XNA game development platform. In our game, the driver wears a tracked video see-through head-worn display, and controls the car with a passive tangible controller. Other players can participate by manipulating waypoints that the car must pass and obstacles with which the car can collide. We dis-cuss our AR infrastructure, which supports the creation of AR applications and games in a managed code environment, the user interface we developed for the AR racing game, the game's soft-ware and hardware architecture, and feedback and observations from early demonstrations.
Article
Full-text available
Augmented tabletops can be used to create multi-modal and collaborative environments in which natural interactions with tangible objects that represent virtual (digital) information can be performed. Such environments are considered potentially interesting for many different applications. In this paper, we address the question of whether or not it makes sense to use such environments to design learning experiences for young children. More specifically, we present the "Read-It" application that we have created to illustrate how augmented tabletops can support the development of reading skills. Children of five-to-seven-years old were actively involved in designing and testing this application. A pilot experiment was conducted with a prototype of the Read-It application, in order to confirm that it does indeed meet the a priori expectations. We hope that the Read-It application will inspire the development of more tabletop applications that are targeted at specific user groups and activities.
Article
Full-text available
In this paper we discuss a case study for which we applied a customized augmented reality display –the Virtual Showcase– as a new platform for digital storytelling. Different storytelling components are identified and examples for their specific realization are explained. Our case study focuses on communicating scientific information to a novice audience in a museum context. Addressing first user feedback, we describe our current efforts of improvement.
Article
Full-text available
In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context of a Reality-Virtuality (RV) continuum, encompassing a large class of "Mixed Reality" (MR) displays, which also includes Augmented Virtuality (AV). MR displays are defined by means of seven examples of existing display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish different Mixed Reality display systems from each other are presented, first by means of a table in which the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and then by means of a three dimensional taxonomic framework, comprising: Extent of World Knowledge (EWK), Reproduction Fidelity (RF) and Extent of Presence Metaphor (EPM). A principal objective of the taxonomy is to clarify terminology issues and to provide a framework for classifying research across different disciplines.
Article
Full-text available
By introducing virtual content onto a regular tabletop, aug-mented reality (AR) offers a compelling display medium for enhancing tabletop games. Coupled with tangible user in-terfaces, it constitutes a truly novel environment for games. Our research investigates the possibilities and constraints for succesful game design, starting with principles drawn fom tabletop and conventional computer games, and mov-ing towards designs unique to AR. Our first major game, AR Tankwar, was recently demonstrated at GenCon Indy 2005 to over 300 players and many more spectators. This paper presents a short overview of AR Tankwar along with results from questionnaires and player discussion at the con-vention.
Article
Full-text available
This paper describes a recent program for game-based learning within a mixed-reality environment, the Situated Multimedia Arts Learning Lab [SMALLab]. In the program, our research team collaborated with a 9 th grade Language Arts teacher to design and deliver a new learning game and associated curriculum. Through the process of game-design and game-play, students advance their understanding of metaphor. We outline the theoretical basis upon which design decisions were made, and describe the rationale for choosing Language Arts as the subject area for this program. Three goals structure our research: (1) to advance students' understanding of literary devices with an emphasis on metaphor; (2) to engage otherwise under-performing students through game-based learning that is student-centered, collaborative, and based in reflective practice; and (3) to demonstrate effective game-based learning using a mixed-reality platform in a conventional classroom context. Twenty-four students attending a large suburban high school in the southwest United States participated in this learning experience once a week for seven weeks during the Fall of 2007. Our data indicates that these students attained a more globally coherent model of metaphor in the course of their participation, that they found both the game-design and the game-play process stimulating and rewarding, and that, given the necessary scaffolding, a mixed-reality learning environment can be effectively employed to teach standards-based curriculum in a conventional high school classroom.
Article
Full-text available
This paper presents a novel computer entertainment system which recaptures human touch and physical interaction with the real-world environment as essential elements of the game play, whilst also maintaining the exciting fantasy features of traditional computer entertainment. Our system called ‘Touch-Space’ is an embodied (ubiquitous, tangible, and social) computing based Mixed Reality (MR) game space which regains the physical and social aspects of traditional game play. In this novel game space, the real-world environment is an essential and intrinsic game element, and the human’s physical context influences the game play. It also provides the full spectrum of game interaction experience ranging from the real physical environment (human to human and human to physical world interaction), to augmented reality, to the virtual environment. It allows tangible interactions between players and virtual objects, and collaborations between players in different levels of reality. Thus, the system re-invigorates computer entertainment systems with social human-to-human and human-to-physical touch interactions.
Conference Paper
Full-text available
In modern society, people increasingly lack social interaction, although beneficial to work and personal life. Airhockey Over a Distance addresses this issue by recreating the social experience facilitated by physical game play in a distributed environment. We networked two airhockey tables and augmented them with a videoconference. Concealed mechanics on each table allow for a physical puck to be shot back and forth between the two locations. Supporting the hitting of a fast-moving, tangible puck between the two players creates a compelling social game experience which was confirmed by about 30 players. Our preliminary findings suggest that our casual physical game supports social interactions and contributes to an increased connectedness between people who are geographically apart.
Conference Paper
Full-text available
We describe two games in which online participants collaborated with mobile participants on the city streets. In the first, the players were online and professional performers were on the streets. The second reversed this relationship. Analysis of these experiences yields new insights into the nature of context. We show how context is more socially than technically constructed. We show how players exploited (and resolved conflicts between) multiple indications of context including GPS, GPS error, audio talk, ambient audio, timing, local knowledge and trust. We recommend not overly relying on GPS, extensively using audio, and extending interfaces to represent GPS error.
Conference Paper
Full-text available
Physical and social interactions are constrained,and natural interactions are lost in most of present digital family entertainment systems [5]. Magic Cubes strive for bringing the computer storytelling, doll 's house,and board game back into reality so that the children can interact socially and physically as what we did in the old days. Magic Cubes are novel augmented reality systems that explore to use cubes to interact with three dimensional virtual fantasy world. Magic Cubes encourage discussion, idea exchange, collaboration, social and physical interactions among families.
Conference Paper
Full-text available
In this paper we present the results of a qualitative, empirical study exploring the impact of immersive technologies on presence and engagement, using the interactive drama Façade as the object of study. In this drama, players are situated in a married couple's apartment, and interact primarily through conversation with the characters and manipulation of objects in the space. We present participants' experiences across three different versions of Façade - augmented reality (AR) and two desktop computing based implementations, one where players communicate using speech and the other using typed keyboard input. Through interviews and observations of players, we find that immersive AR can create an increased sense of presence, confirming generally held expectations. However, we demonstrate that increased presence does not necessarily lead to more engagement. Rather, mediation may be necessary for some players to fully engage with certain interactive media experiences. Author Keywords
Conference Paper
Full-text available
The augurscope is a portable mixed reality interface for outdoors. A tripod-mounted display is wheeled to different locations and rotated and tilted to view a virtual environment that is aligned with the physical background. Video from an onboard camera is embedded into this virtual environment. Our design encompasses physical form, interaction and the combination of a GPS receiver, electronic compass, accelerometer and rotary encoder for tracking. An initial application involves the public exploring a medieval castle from the site of its modern replacement. Analysis of use reveals problems with lighting, movement and relating virtual and physical viewpoints, and shows how environmental factors and physical form affect interaction. We suggest that problems might be accommodated by carefully constructing virtual and physical content
Conference Paper
Full-text available
There is a growing interest in developing technologies for creating interactive dramas (13, 22). Evaluating them, however , remains an open research problem. In this paper, we present a method for evaluating the technical and design approaches employed in a conversation-centered interactive drama. This method correlates players' subjective experience during conversational breakdowns, captured using retrospective protocols, with the corresponding AI processing in the input language understanding and dialog management subsystems. The methodology is employed to analyze conversation breakdowns in the interactive drama Façade. We find that the narrative cues offered by an interactive drama, coupled with believable character performance, can allow players to interpretively bridge system limitations and avoid experiencing a conversation breakdown. Further, we find that, contrary to standard practice for task-oriented conversation systems, using shallowly understood information as part of the system output hampers the player experience in an interactive drama.
Conference Paper
Full-text available
We present Muddleware, a communication platform designed for mixed reality multi-user games for mobile, lightweight clients. An approach inspired by Tuplespaces, which provides decoupling of sender and receiver is used to address the requirements of a potentially large number of mobile clients. A hierarchical database built on XML technology allows convenient prototyping and simple, yet powerful queries. Server side-extensions address persistence and autonomous behaviors through hierarchical state machines. The architecture has been tested with a number of multi-user games and is also used by non-entertainment applications.
Conference Paper
Full-text available
It is rapidly becoming clear that entertainment will be one of the killer applications of future wireless networks. More specifically mobile gaming is predicted to be worth $1.2 billion by the year 2006 to providers in the U.S. alone [20]. The driving force behind this is the introduction of powerful feature rich handsets and ubiquitous access to high performance wireless networks. However, mobile applications face issues that are subtly different from fixed network applications, including fluctuating connectivity, network QoS and host mobility issues. To investigate the requirements of future mobile applications we have deployed a wireless MAN consisting of GPRS and IEEE 802.11 hotspots based on Mobile IPv6 around the city of Lancaster and have built an augmented reality game designed to evaluate future mobile application requirements.In this paper we introduce Real Tournament, a prototype multi-player mobile game, which uses handheld computers augmented with an array of sensors to enable true mobile interaction in a real-world environment. We then evaluate current approaches to real-time interaction and follow by outlining our own architecture more suited to wireless environments and based on the peer-to-peer approach. The approach provides adaptation, shared state, and consistency mechanisms in order to provide support for scalable, low latency, soft real time mobile applications.
Conference Paper
Full-text available
Human Pacman is an interactive ubiquitous and mobile entertainment system that is built upon position and perspective sensing via Global Positioning System and inertia sensors; and tangible human-computer interfacing with the use of Bluetooth and capacitive sensors. Although these sensing-based subsystems are weaved into the fabric of the game and are therefore translucent to players, they are nevertheless the technical enabling forces behind Human Pacman. The game strives to bring the computer gaming experience to a new level of emotional and sensory gratification by embedding the natural physical world ubiquitously and seamlessly with a fantasy virtual playground. We have progressed from the old days of 2D arcade Pacman on screens, with incremental development, to the popular 3D game console Pacman, and the recent mobile online Pacman. With our novel Human Pacman, we have a physical role-playing computer fantasy together with real human-social and mobile-gaming that emphasizes on collaboration and competition between players in a wide outdoor physical area that allows natural wide-area human-physical movements. Pacmen and Ghosts are now real human players in the real world experiencing mixed computer graphics fantasy-reality provided by using wearable computers that are equipped with GPS and inertia sensors for players' position and perspective tracking. Virtual cookies and actual tangible physical objects with Bluetooth devices and capacitive sensors are incorporated into the game play to provide novel experiences of seamless transitions between real and virtual worlds. In short, we believe Human Pacman is pioneering a new form of gaming that is based on sensing technology and anchored on physicality, mobility, social interaction, and ubiquitous computing.
Conference Paper
Full-text available
Pervasive games provide a new type of game combining new technologies with the real environment of the players. While this already poses new challenges to the game developer, requirements are even higher for pervasive Augmented Reality games, where the real environment is additionally enhanced by virtual game items. In this paper we will review the technological challenges to be met in order to realize pervasive AR games, show how they go beyond those of other pervasive games, and present how our AR framework copes with them. We will further show how these approaches are applied to three pervasive AR games and draw conclusions regarding the future requirements regarding the support of this type of games.
Conference Paper
Full-text available
In this paper we present MAR (Mobile Augmented Reality) Toolkit as an easy-to-use augmented reality toolset for building multi-user mobile phone games. It is built on top of MUPE - Nokia-developed open source mobile platform based on Java - which considers the special qualities of mobile technology. MAR Toolkit contains four components, map interface (MAP), physical object tagger (POT), public display (PUD) and silent communicator (SIC).We have successfully demonstrated MAR Toolkit by implementing a game named as Mupeland Yard based on classical board game Scotland Yard. In user testing we found that common usability issues related to mobile technology and MUPE-platform troubled the tests. However, especially POT component raised interest among developers. We found that using graphical 2D-tags for providing location information for augmented reality games is a simple and robust alternative for more technology intensive GPS and cell-ID information.
Conference Paper
Full-text available
: We introduce a local collaborative environment for gaming. In our setup multiple users can interact with the virtual game and the real surroundings at the same time. They are able to communicate with other players during the game. We describe an augmented reality setup for multiple users with see-trough head-mounted displays, allowing dedicated stereoscopic views and individualized interaction for each user. We use face-snapping for fast and precise direct object manipulation. With face snapping and the subdivision of the gaming space into spatial regions, the semantics of actions can be derived out of geometric actions of the user. Further, we introduce a layering concept allowing individual views onto the common data structure. The layer concept allows to make privacy management very easy by simply manipulating the common data structure. Moreover, assigning layers to spatial regions carefully, a special privacy management is often not necessary. Moving objects from one region into ...
Article
A new type of interactive attraction in mixed reality environment, BLADESHIPS, has been designed and developed. The BLADESHIPS is a game in which players compete with each other by controlling virtually expressed belt-shaped flying objects, "ships", with their hands in a real environment. They control their own ships by avoiding other ships and real/virtual obstacles. They try to drive enemy ships to hit the obstacles. The occlusion between real and virtual objects and the collision of virtual ships with the real environment are represented to increase the reality of the game.
Article
This paper presents a prototype developed as part of the Backseat gaming project. The aim of the project is to explore how to make use of mobile properties for developing compelling and fun game experiences. The prototype is developed for use in a highly mobile situation, that of a car passenger and is realized by the use of mobile devices and the users physical location during speed to merge the virtual content and surrounding road context into an augmented reality game.
Article
Computer games lack the social bonding and collective physical exercise benefits that sports provide. To overcome these limitations, we have been investigating how to apply the benefits of sport, in particular the workout and social bonding effect, in a distributed setting. We designed, developed, and evaluated Breakout for Two, which allows people who are miles apart to play a physically exhausting ball game together. We had over a thousand players who interacted through a life-size video-conference screen using a regular soccer ball as an input device. In an evaluative study, 56 players were interviewed and said that they got to know the other player better, had more fun, became better friends, and were happier with the transmitted audio and video quality in comparison to those who played the same game using a nonexertion keyboard interface. These results suggest that sports over a distance is an exciting new field with an "exertion interface" that encourages remote interaction, where players can achieve both a work-out and socializing.
Conference Paper
This paper presents ARCHEOGUIDE, a novelsystem offering augmented reality tours in archaeologicalsites. The system is based on wearable and mobilecomputers, networking technology and real-time computergraphics and 3D animation and visualization techniques.The user can participate tours adapted to his profile andautomatically receive information based on his positionand orientation as calculated by a hybrid technique makinguse of GPS, compass and image-based tracking. The usercan interact with the device via multi-modal interactiontechniques and request navigation and other information.ARCHEOGUIDE has been tested at the archaeologicalsite of Olympia in Greece.
Article
University of South Australia has successfully developed through-walls collaboration which allows users in the field work in real time with users indoors who have access to reference materials, a global picture, and advanced technology. The technology supports ubiquitous workspaces, augmented reality (AR), and wearable computers. AR wearable computer technology provides digital images, videos, and voice information that are geospatially mapped to the recording point, which will provide control center personnel better situational awareness. The indoor system provides appropriate visualizations to support situational awareness for control room experts. A particularly powerful feature of the through-walls systems is that both indoor and outdoor users can provide an AR overlay with multimedia data such as images, text, video, and sound for highlighting information.
Article
This paper introduces a collaborative shooting game - "RV-Border Guards," which uses Mixed Reality (MR) technologies. This system is designed to emphasize MR-specific features for entertainment. Three players wearing HMDs cooperatively battle with virtual invaders flying around them in the MR space. Each player is armed with a virtual gear such as a helmet and a gun, and can intuitively interact with the MR space using easy gestures. Total reality of the MR space is carefully tuned. This project tries to achieve a novel multi-player entertainment, which has never been realized without MR technologies.
Article
This sketch presents a prototype developed as part of the Backseat gaming project. The aim of the project is to explore how to make use of mobile properties for developing compelling and fun game experiences. The prototype is developed for use in a highly mobile situation, that of a car passenger and is realized by the use of mobile devices and the users physical location during speed to merge the virtual content and surrounding road context into an augmented reality game. In this research, in addition to location, we also introduce variables such as speed, direction, timing, changing surrounding, fast movement of manipulative objects and multiple entry and exits.