Figure 2 - uploaded by Abraham G. Campbell
Content may be subject to copyright.
Milgram’s Reality-Virtuality Continuum (Milgram and Kishino, 1994) 

Milgram’s Reality-Virtuality Continuum (Milgram and Kishino, 1994) 

Source publication
Article
Full-text available
In recent years, an increasing number of Mixed Reality (MR) applications have been developed using agent technology — both for the underlying software and as an interface metaphor. However, no unifying field or theory currently exists that can act as a common frame of reference for these varied works. As a result, much duplication of research is ev...

Contexts in source publication

Context 1
... Reality Agent recorded in scientific literature and was years ahead of similar efforts. The Invisible Person (Psik et al., 2003) is based on the ALIVE system and employs a humanoid virtual character in an effort to engage visitors at the Vienna Museum of Technology in a game of Tic Tac Toe. The game board is digitally added onto the floor, and both user and character control the game via body postures and hand gestures. The character’s internal state is based on an emotional system that directs its actions, facial expressions and manner of interaction with the user. Feedback from visitors of the exhibition commend the lifelikeness of the agent. Storytelling engines also explicitly model individual agents’ goals and motivations in order to create dynamic narratives from the interplay of these goals and the users actions. One of the first projects to apply digital narrative to Mixed Reality is the cultural heritage application GEIST (Kretschmer et al., 2001). GEIST 1 immerses the user in a thrilling adventure involving events from the Thirty Years’ War. As the user roams the old town of Heidelberg, he can enter certain ‘hotspots’ in which ghosts from the past appear in the form of virtual characters. They plead for the user’s help in solving the mystery surrounding their death, creating a quest around the city in which the user has to learn about the history of places and events in order to succeed. In the physical domain, GEIST agents are limited to sensing the user’s position and orientation, while in the virtual, which is populated with spatially aligned models of buildings and other virtual objects, interaction is much more varied and versatile. However, due to the spirit nature of the GEIST agents, corporeal presence of the ghosts is inhibited, as they appear translucent and float in midair. Another prominent example of MR storytelling is the Mixed-Reality Interactive Storytelling (MRIS) project (Charles et al., 2004), which allows a user to immerse himself into a spy thriller story in the role of the villain. It does so by capturing the user’s image in real time, extracting it from the background, and inserting it into a virtual world populated by autonomous synthetic actors with which the user then interacts using natural language and gestures. The resulting image is projected onto a large screen facing the user, who sees his own image embedded in the virtual stage alongside the synthetic actors. Notably, when viewed from the perspective of Milgram’s continuum of MR displays (see Figure 2), MRIS is a rare example exemplifying the concept of Augmented Virtuality, i.e. a virtual world with added ‘real’ components. Finally, Virtual Gunslinger (Hartholt et al., 2009), is a more recent example of a Mixed Reality storytelling experience. In it, the user plays the character of a cowboy in a Wild West saloon who gets challenged to a duel. The user is placed in an environment featuring a real bar counter and a virtual bartender and outlaw, both of which are projected onto screens placed in the room. The user can interact with the agents using natural language dialogues and gestures, e.g. moving his arm as if to pull a gun when duelling with the outlaw. Common to all these strong agents is that the agent architecture facilitates the development of agents that exhibit realistic and lifelike behaviour. But strong agent system are also often used to realise highly complex and distributed systems that deal with ...
Context 2
... (1998, 1999) and Duffy (2000), who claim that the social environment includes the agent’s interactions with other agents or human users. Within the context of this research, embodiment is seen as the strong provision of environmental context (structural coupling) with a social element included. An agent is embodied if it is situated in a particular environment, has a body , and senses and interacts with that environment, and any other individuals located therein. This definition of embodiment coincides with that of Dourish (2001), who emphasised the importance of an embodied approach to Human-Computer Interaction, in light of new developments in Ubiquitous and Social Computing and proposed a number of design guidelines for the development of Embodied Interaction. Early Artificial Intelligence (AI) research focused upon reasoning based upon search of abstract symbol structures (Newell and Simon, 1976). But this unembodied approach, sometimes referred to as ‘Good Old Fashioned AI’ (Haugeland, 1985), had a number of flaws. As noted by Steels (2000) and Dautenhahn (1999), humans have a tendency to ‘animate’ the world and are unlikely to see an unembodied agent as intelligent. Therefore embodiment has, in recent years, come to be seen as an important requirement in the development of an intelligent system (Duffy et al., 2005). The move away from the unembodied approach was triggered by a series of papers by Brooks (1991a,b), who emphasised the situatedness and embodiment of an agent. Brooks’ popularisation of the reactive approach served as a catalyst for the creation of a more embodied approach to AI, where an agent must be structurally coupled with its environment if it is to be seen as intelligent. While robot agents are embodied in a physical form, using sensors and actuators to perceive and act upon the physical world, virtual agents can also be considered embodied in their simulated environment, at least to the extent to which the simulation manages to create a structural coupling between the agent and the simulated environment. Both strands are motivated by the desire to create agents that are capable of behaving and interacting in an intelligent manner with other agents. Crucially, both robotic and virtual agents can be considered synthetic characters, although differently embodied, i.e. in physical or digital form. Key to the definition of Mixed Reality Agents, as outlined within this paper, is the idea of a Mixed Reality environment. Milgram and Kishino (1994) define MR in terms of their Reality-Virtuality Continuum (Figure 2) whereby Mixed Reality is the space between a purely physical (or ‘real’, as they describe it) environment and a purely virtual environment. Each MR environment can be seen, to a greater or ...

Similar publications

Conference Paper
Full-text available
As computing and displays become more pervasive and wireless networks are increasing the connections between people and things, humans inhabit both digital and physical realities. In this paper we describe our prototype Mirror Worlds framework, which is designed to fuse these realities: Fusality. Our goal for Fusality is to support innovative resea...

Citations

... Therefore, previous research has demonstrated that food science and AR represent a potentially useful combination. Mixed reality (MR) fuses the virtual and physical environments, thus enabling the virtual objects to interact with the physical world (Holz et al., 2011). The motivation behind developing MR systems, in general, is not only to 'see' the virtual/digital overlay or object, as in AR systems, but also to physically interact with and/or manipulate it (Maas & Hughes, 2020). ...
Article
Full-text available
Background The topic of food is broad and global, thereby representing an influential sector of the economy. Motivated by the advent of Industry 4.0, massive potential exists to implement cutting-edge technologies in the food industry. Recent years have seen a growing interest towards the applications of augmented/mixed reality (AR/MR) in the food sector. Scope and approach An extensive search of online journals focusing on Scopus was conducted using terms including ‘augmented reality’, ‘mixed reality’ and ‘food’ in the search fields of Title, Abstract, and Keywords. Full paper reading was implemented and ineligible articles (i.e., non-English-language, review articles, not peer-reviewed and without full paper) were removed. Key findings and conclusions Our systematic search resulted in 111 eligible articles, eight of which related to MR technology. There is an overall increasing trend in the number of publications appearing annually since the first relevant publication in 2010. Analysing these publications demonstrates the multidisciplinary nature of this technology which is closely linked to machine learning, computer vision, the Internet of Things (IoT), and artificial intelligence. Our findings also revealed that AR/MR technology is mainly applied in the following areas: dietary assessment, food nutrition and traceability, food sensory science, retail food chain applications, food education and learning, and precision farming. Furthermore, we highlight the limitations and analytical challenges that hinder the application of AR/MR to food-related research, such as the lack of reliable wireless connection and the difficulty in recognizing food objects in a complex environment, while also describing future research needs and directions.
... Social presence is about feeling socially connected to a virtual human or avatar in a shared space. For a virtual human in augmented reality, the notion of corporeal presence has been introduced [19,20]. It focuses on the visual appearance and the alignment of virtual objects in the augmented reality world. ...
Conference Paper
In augmented reality, we have a virtual layer superimposed on reality. This virtual layer can contain multisensory content, that is, content that stimulates each of our senses: sight, hearing, taste, touch, and smell, and many interoceptive senses such as pain, heat, time, and balance. The virtual layer can contain virtual humans that have virtual senses to explore the augmented reality environment and to interact with real humans. These virtual humans can take on a role in social, marketing, retail, training, education, and entertainment augmented reality applications. In the human-like behavior of virtual humans, we expect awareness, use, and display of awareness of these senses. We present a view on the existing literature on virtual humans in augmented reality with a focus on sense-related presence issues.
... In one of the classic definitions [36] AR combines the real space and objects with computer-generated (virtual) objects, registers them together and runs interactively in real time. The main difference between MR and AR is that in the case of the latter virtual objects are not only superimposed on the material world but they can be also interacted with [37,38]. The difference can also be seen in spatial reference frames used in the former [27]. ...
Article
Full-text available
Extended reality (XR) technology is increasingly often considered in practical applications related to urban planning and smart city management. It offers many advantages as a new visualization technique that gives its users access to places that are not available in material space and a unique perspective on existing objects. It can provide immersive multi-sensory experience that can induce emotional response in participatory planning. However, standard mode of implementation that relies on mobile phone applications and VR headsets has a disadvantage when it comes to availability and accessibility. Here we test the WebXR solution that can mitigate those problems. We have created six AR and VR environments that resembled common urban planning scenarios and conducted usability tests with people having planning and GIS background. Results indicate that WebXR can provide useful solution in urban planning when the interface and environment resemble common practices and situations encountered in real life. Environments that have introduced new digital affordances like AR measurements or semi-transparent walkable scale models were rated lower. Users evaluated presented environment as having high usability and expressed their positive attitude toward using XR in their professional practice mainly as a participatory and visualization tool.
... However, at this stage, there remains a need for clarity in the design and development process for XRI applications, with several new and emerging sub-themes. This work recognizes this convergent need and aims to contribute by applying a multidimensional methodology to the problem of interface design within the IoT, for agency in mixed reality [22], and also by the development of a new system architectural framework that blends components from 3D development, web machine learning, publish-subscribe connectivity, and conventional IoT development frameworks. An early functional prototype is presented, alongside a descriptive evaluation. ...
... Mixed Reality Agents (MiRAs) are defined as agents embodied in the Mixed Reality environment [22]. Basing this research on the definition of agency given by [45], an agent is defined as a hardware-or-software-based entity characterised by the following attributes:Autonomy: Agents can operate without the direct intervention of humans or others, and have control over their actions and internal state. ...
... To investigate Mixed Reality Agents, Holz et al [22] establishes [29]; an extension to the related work found in [28]. No. ...
... Pela pesquisa realizada, pode-se concluir que: Realidade Virtual (RV) implica a imersão num mundo artificial, gerado por computador (Carmigniani et al., 2011); Realidade Aumentada (RA), a sobreposição de objetos virtuais ao mundo real (Azuma, 1997); Realidade Mista (RM), possibilita a interação, em ambiente real, com objetos virtuais (Holz et al., 2011); Realidade Estendida (XR) cria uma experiência completa que envolve todos os sentidos (Çöltekin et al., 2020). A Figura 2, sintetiza as diferenças entre RA, RV e XR. ...
Article
Full-text available
Neste artigo apresenta-se umestudo sobre a utilização das tecnologias ExtendedReality(XR)pelos professoresde ensino básico e secundário,a finalidadedeste trabalho foi o de compreender se os professores conhecem e usam estastecnologias nas suas práticas letivas; de que forma utilizam esta estratégia de ensino na sua prática docente e ainda, se têm interesse em utilizar estas tecnologias como ferramenta pedagógica. Usou-sea metodologia positivista, de natureza quantitativa. Recolheram-seos dados, através de inquérito em que participaram 105 professores do ensino básico e secundário. Os resultados obtidos indicam que os professores que mais usam estas tecnologias são os professores licenciados, quadros de escola agrupamento e maioritariamente pertencentes ao grupo de informática. O principal contributo do trabalho apresentado é averiguar se estas tecnologias têm boa aceitação como estratégia de ensino e quais as necessidades de formação do corpo docente no que concerne a estas tecnologia. Pretendemos, como trabalho futuro,realizar ações de formação e workshop dirigidos aosprofessores do ensino básico e secundário com o intuito de os formarcientificamente e pedagogicamenteno uso destas tecnologias e incentivar à sua utilização em contexto educativo.
... Spatial mapping builds up a representation of physical surfaces, so that apps can adapt to room shape/size, and voice, gesture, and gaze interaction allow to observe and accurately control digital information projected onto a user's view of the real world. Campbell et al. define agents embodied in AR as AuRAs [7] based on the concept of MiRA, "an agent embodied in a Mixed Reality environment" [8]. They, however, do not systematically analyze the difference in features of agents in AR. ...
Chapter
Full-text available
Humanoid intelligent agents, or 'Holographic AIs', as we prefer, are trending, promising improved delivery of personalized services on smart glasses and in Augmented Reality. Lacking clarity of the concept and missing recommendations for their features, however, pose a challenge to developers of these novel, embodied agents. In this paper, we therefore conduct a comparative analysis of nine intelligent agents who can interact with both physical and virtual surroundings. We identify, select, and investigate four distinct types of non-player game characters, chatbot agents, simulation agents, and intelligent tutors in order to, subsequently, develop a framework of features and affordances for holographic AIs along the axes of appearance, behavior, intelligence, and responsiveness. Through our analysis, we derive preliminary recommendations for developers of Holographic AIs: the use case determines appearance; dialogue management is key; awareness and adaptation are equally important for successful personalization; and environmental responsiveness to events both in the virtual and digital ream is needed for a seamless experience.
... The MR IAM is both spatially and socially present in the same physical environment as those with which it converses. This sense of corporeal presence, a critical affordance of MR, gives the agent the illusion of substance in physical space [49]. While this presence is not manifested in a physical and corporeal manner, as it would be for a living person, MR affords the illusion of this for the posthumous persona. ...
Chapter
Full-text available
Beginning with the 2009 publication of Michael Massimi’s Dying, Death, and Mortality: Towards Thanatosensitivity in HCI, researchers have been exploring how to design UX and UI for end-of-life systems and memorials. This work has resulted in the design of interactive posthumous personhoods memorializing those who have lived by utilizing the data the deceased produced while they were alive. HCI designers and researchers have implemented Natural Language Processing (NLP) to create interactive posthumous personhoods in the form of chatbots. Over the last five years, Mixed Reality (MR) has been integrated by practitioners to create posthumous personhoods that they claim offer a greater fidelity with the departed when they were alive than previous technologies. How MR intersects with those previous technologies, cultural, personal heritage, and memorialization to achieve this greater fidelity requires further exploration if the media is to be used to elevate the evidentiary status of these posthumous personhoods. Insights from HCI end-of-life literature, death and dying studies, and interactive non-fiction are used to explore the ontology of these interactive posthumous personhoods to discover what MR is doing. This paper elucidates connected ethical, ontological, and thanatological issues that the HCI field needs to address as this alternative practice of mourning and memorialization becomes a mainstream practice. The paper argues that MR increases the social and spatial presence of an MR posthumous personhood’s identity and subjectivity with greater fidelity than previous technologies, and that this supports an audience’s active creation of belief in the agent as being evidentiary of a posthumous personhood.
...  MR is an extension to AR, which delivers interaction between real and virtual elements in an environment [36].  MR changes the information of a BIM model and can feed the regenerated visuals back to the model [129]  Sometimes termed as Hybrid Reality, MR combines the features of VR and AR in the best possible way by making the interactions of the user more realistic, mimicking it's perspective and behaviour with the elements in the scene [130].  MR combines real and virtual worlds to generate novel situations, environments and visualisations, creating a co-existence of physical and digital objects [131]. ...
Article
Full-text available
With the outset of Industrial Revolution 4.0 (IR 4.0), every sector is escalating to get enrichment out of it, whether they are research- or industry-oriented. The Architecture Engineering and Construction (AEC) industry lags a bit in adopting it because of its multi-faceted dependencies and unique nature of work. Despite this, a trend has been seen recently to hone the IR 4.0 multitudes in the AEC industry. The upsurge has been seen in the usage of Immersive Technologies (ImTs) as one of the disruptive techniques. This paper studies the literature based on ImTs, which are Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) integrating with Building Information Modelling (BIM) in the AEC sector. A total number of 444 articles were selected from Scopus following the preferred reporting items for systematic reviews and meta-analysis (PRISMA) protocol of reviewing the literature. Among the selected database, 64 papers are identified as the result of following the protocol, and the articles are divided into eight domains relevant to the AEC industry, namely client/stakeholder, design exploration, design analysis, construction planning, construction monitoring, construction health/safety, facility/management, and education/training. This study adopts both a scientometric analysis for bibliometrics visualization and a critical review using Strength Weakness Opportunity Threat (SWOT) analysis for finding gaps and state of play. The novelty of this paper lies in the analysis techniques used in the literature to provide an insight into the literature, and it provides directions for the future with an emphasis on developing sustainable development goals (SDGs). In addition, research directions for the future growth on the adoption of ImTs are identified and presented based on categorization in immersive devices, graphical/non-graphical data and, responsive/integrative processes. In addition, five subcategories for each direction are listed, citing the limitations and future/needs. This study presents the roadmap for the successful adoption of ImTs for industry practitioners and stakeholders in the AEC industry for various domains. The paper shows that there are studies on ImTs with or without BIM; however, future studies should focus on the usage of ImTs in various sectors such as modular integrated construction (MiC) or emerging needs such as SDGs.
... Some classifications of virtual assistants are reported in [16,24]. Moreover, [15] introduces the concepts of Mixed Reality Assistants, i.e., agents that operate in a MR environment. ...
... On similar grounds, the research presented by Anabuki et al. [6] introduces and evaluates an ECA in MR and concludes that people prefer the ECA to be of a size that can be seen in its entirety within their field of view. Furthermore, Holz et al. [16] have proposed the concept of Mixed Reality Agent (MiRA) to be considered as a distinct field of study that uses a common taxonomy. MiRAs, in general, often provide context-sensitive and spatially [8] referenced information to the users (e.g., "to your left"). ...
Article
Full-text available
In this paper, we take a first step in exploring the effect of xR devices on the experiences of users in the context of human-agent interactions. We report on the design and heuristic evaluation of an embodied conversational agent integrated into either a Virtual or Mixed Reality environment. Our evaluation with four experts show that Virtual Reality may elicit a better experience from users, while Mixed Reality may evoke a better sense of social presence due to natural user embodiment and spatial references to physical objects in the real environment.