Figure 2 - uploaded by Abraham G. Campbell
Content may be subject to copyright.
Source publication
In recent years, an increasing number of Mixed Reality (MR) applications have been developed using agent technology — both for the underlying software and as an interface metaphor. However, no unifying field or theory currently exists that can act as a common frame of reference for these varied works. As a result, much duplication of research is ev...
Contexts in source publication
Context 1
... Reality Agent recorded in scientific literature and was years ahead of similar efforts. The Invisible Person (Psik et al., 2003) is based on the ALIVE system and employs a humanoid virtual character in an effort to engage visitors at the Vienna Museum of Technology in a game of Tic Tac Toe. The game board is digitally added onto the floor, and both user and character control the game via body postures and hand gestures. The character’s internal state is based on an emotional system that directs its actions, facial expressions and manner of interaction with the user. Feedback from visitors of the exhibition commend the lifelikeness of the agent. Storytelling engines also explicitly model individual agents’ goals and motivations in order to create dynamic narratives from the interplay of these goals and the users actions. One of the first projects to apply digital narrative to Mixed Reality is the cultural heritage application GEIST (Kretschmer et al., 2001). GEIST 1 immerses the user in a thrilling adventure involving events from the Thirty Years’ War. As the user roams the old town of Heidelberg, he can enter certain ‘hotspots’ in which ghosts from the past appear in the form of virtual characters. They plead for the user’s help in solving the mystery surrounding their death, creating a quest around the city in which the user has to learn about the history of places and events in order to succeed. In the physical domain, GEIST agents are limited to sensing the user’s position and orientation, while in the virtual, which is populated with spatially aligned models of buildings and other virtual objects, interaction is much more varied and versatile. However, due to the spirit nature of the GEIST agents, corporeal presence of the ghosts is inhibited, as they appear translucent and float in midair. Another prominent example of MR storytelling is the Mixed-Reality Interactive Storytelling (MRIS) project (Charles et al., 2004), which allows a user to immerse himself into a spy thriller story in the role of the villain. It does so by capturing the user’s image in real time, extracting it from the background, and inserting it into a virtual world populated by autonomous synthetic actors with which the user then interacts using natural language and gestures. The resulting image is projected onto a large screen facing the user, who sees his own image embedded in the virtual stage alongside the synthetic actors. Notably, when viewed from the perspective of Milgram’s continuum of MR displays (see Figure 2), MRIS is a rare example exemplifying the concept of Augmented Virtuality, i.e. a virtual world with added ‘real’ components. Finally, Virtual Gunslinger (Hartholt et al., 2009), is a more recent example of a Mixed Reality storytelling experience. In it, the user plays the character of a cowboy in a Wild West saloon who gets challenged to a duel. The user is placed in an environment featuring a real bar counter and a virtual bartender and outlaw, both of which are projected onto screens placed in the room. The user can interact with the agents using natural language dialogues and gestures, e.g. moving his arm as if to pull a gun when duelling with the outlaw. Common to all these strong agents is that the agent architecture facilitates the development of agents that exhibit realistic and lifelike behaviour. But strong agent system are also often used to realise highly complex and distributed systems that deal with ...
Context 2
... (1998, 1999) and Duffy (2000), who claim that the social environment includes the agent’s interactions with other agents or human users. Within the context of this research, embodiment is seen as the strong provision of environmental context (structural coupling) with a social element included. An agent is embodied if it is situated in a particular environment, has a body , and senses and interacts with that environment, and any other individuals located therein. This definition of embodiment coincides with that of Dourish (2001), who emphasised the importance of an embodied approach to Human-Computer Interaction, in light of new developments in Ubiquitous and Social Computing and proposed a number of design guidelines for the development of Embodied Interaction. Early Artificial Intelligence (AI) research focused upon reasoning based upon search of abstract symbol structures (Newell and Simon, 1976). But this unembodied approach, sometimes referred to as ‘Good Old Fashioned AI’ (Haugeland, 1985), had a number of flaws. As noted by Steels (2000) and Dautenhahn (1999), humans have a tendency to ‘animate’ the world and are unlikely to see an unembodied agent as intelligent. Therefore embodiment has, in recent years, come to be seen as an important requirement in the development of an intelligent system (Duffy et al., 2005). The move away from the unembodied approach was triggered by a series of papers by Brooks (1991a,b), who emphasised the situatedness and embodiment of an agent. Brooks’ popularisation of the reactive approach served as a catalyst for the creation of a more embodied approach to AI, where an agent must be structurally coupled with its environment if it is to be seen as intelligent. While robot agents are embodied in a physical form, using sensors and actuators to perceive and act upon the physical world, virtual agents can also be considered embodied in their simulated environment, at least to the extent to which the simulation manages to create a structural coupling between the agent and the simulated environment. Both strands are motivated by the desire to create agents that are capable of behaving and interacting in an intelligent manner with other agents. Crucially, both robotic and virtual agents can be considered synthetic characters, although differently embodied, i.e. in physical or digital form. Key to the definition of Mixed Reality Agents, as outlined within this paper, is the idea of a Mixed Reality environment. Milgram and Kishino (1994) define MR in terms of their Reality-Virtuality Continuum (Figure 2) whereby Mixed Reality is the space between a purely physical (or ‘real’, as they describe it) environment and a purely virtual environment. Each MR environment can be seen, to a greater or ...
Similar publications
As computing and displays become more pervasive and wireless networks are increasing the connections between people and things, humans inhabit both digital and physical realities. In this paper we describe our prototype Mirror Worlds framework, which is designed to fuse these realities: Fusality. Our goal for Fusality is to support innovative resea...
Citations
... Mixed Reality, a hybrid of augmented reality and virtual reality, is the outcome of fusing both the digital and physical worlds and has lately gained popularity, overcoming the limitations of VR's absence of real-world surroundings and AR's lack of ability for interacting with threedimensional (3D) data packages [22]. MR technology integrates the virtual and physical environments in a spatially coherent manner [23]. ...
As the world faces increasing global challenges, innovation becomes crucial for creating a sustainable future. In the age of the Internet of Things (IoT) and Industry 5.0, emerging technologies such as Digital Twins and Extended Reality (XR) are showing promise in addressing these challenges. The article examines the concept of merging Digital Twins, which are virtual reproductions of actual assets, with technologies like Extended Reality (XR) and the Internet of Things (IoT), as well as its applications in optimizing processes, enhancing decision making, and decreasing waste in global enterprises. The convergence of digital twins with extended reality, IoT, and Industry 5.0 presents unprecedented opportunities for innovation in agriculture, healthcare, and industry. In healthcare, digital twins enable personalized treatments and simulations for surgical planning, while in agriculture, they optimize crop growth and resource utilization. In the industry, digital twins streamline production processes, predict maintenance needs, and reduce downtime. This comprehensive review delves into the present landscape, obstacles, and future prospects of digital twins, extended reality, and IoT in various industries, emphasizing the imperative of ongoing research and development to unlock the complete transformative potential of these technologies for a sustainable future in Industry 5.0.
... • Avatarization [11] [8] -Refers to the virtual representation and embodiment of a user, or an object. • Agency [12] [13] -Refers to the ability of users and digital agents to perceive the environment (local or remote; virtual or physical) and take actions within the environment (local or remote; virtual or physical). • Synchronous -Refers to the continuous real-time interactions between users and other agents within the environment (local or remote; virtual or physical). ...
... Figure 2 presents an XRI Agent Design Landscape, with details of the Agency design process, virtual embodiment design method, and XRI interaction with input and output methods, communication method, and interaction path. These design elements are extracted from the authors' previous research on IoT Avatars [9], and Extended Metaverse frameworks [4], building on theoretical design frameworks like [12]; as a set of three design dimensions: Virtual Embodiment Method, XRI Interaction Method, and Agency. Together, systems that account for these designs would have the foundations for an XRI metaverse experience. ...
... Agent design is related to how the intelligent agents [12] interact with the hybrid environments, humans, and each other. Prometheus methodology is presented in [15] for designing and developing intelligent agent systems with inputs (percepts), outputs (actions), and shared data sources. ...
The metaverse refers to the merger of technologies for providing a digital twin of the real world and the underlying connectivity and interactions for the many kinds of agents within. As this set of technology paradigms-involving artificial intelligence, mixed reality, the Internet of Things, and others-gains in scale, maturity, and utility there are rapidly emerging design challenges and new research opportunities. In particular is the metaverse disconnect problem, the gap in task switching that inevitably occurs when a user engages with multiple virtual and physical environments simultaneously. Addressing this gap remains an open issue that affects the user experience and must be overcome to increase overall utility of the metaverse. This article presents design frameworks that consider how to address the metaverse as a hyper-connected meta-environment that connects and expands multiple user environments, modalities, contexts, and the many objects and relationships within them. This article contributes to: a framing of the metaverse as a social XR-IoT (XRI) concept, design Considerations for XRI metaverse experiences, a design architecture for social multi-user XRI metaverse environments, and descriptive exploration of social interaction scenarios within XRI multi-user metaverses. These contribute a new design framework for metaverse researchers and creators to consider the coming wave of interconnected and immersive smart environments.
... • Avatarization [11] [8] -Refers to the virtual representation and embodiment of a user, or an object. • Agency [12] [13] -Refers to the ability of users and digital agents to perceive the environment (local or remote; virtual or physical) and take actions within the environment (local or remote; virtual or physical). • Synchronous -Refers to the continuous real-time interactions between users and other agents within the environment (local or remote; virtual or physical). ...
... Figure 2 presents an XRI Agent Design Landscape, with details of the Agency design process, virtual embodiment design method, and XRI interaction with input and output methods, communication method, and interaction path. These design elements are extracted from the authors' previous research on IoT Avatars [9], and Extended Metaverse frameworks [4], building on theoretical design frameworks like [12]; as a set of three design dimensions: Virtual Embodiment Method, XRI Interaction Method, and Agency. Together, systems that account for these designs would have the foundations for an XRI metaverse experience. ...
... Agent design is related to how the intelligent agents [12] interact with the hybrid environments, humans, and each other. Prometheus methodology is presented in [15] for designing and developing intelligent agent systems with inputs (percepts), outputs (actions), and shared data sources. ...
The metaverse refers to the merger of technologies for providing a digital twin of the real world and the underlying connectivity and interactions for the many kinds of agents within. As this set of technology paradigms - involving artificial intelligence, mixed reality, the internet-of-things and others - gains in scale, maturity, and utility there are rapidly emerging design challenges and new research opportunities. In particular is the metaverse disconnect problem, the gap in task switching that inevitably occurs when a user engages with multiple virtual and physical environments simultaneously. Addressing this gap remains an open issue that affects the user experience and must be overcome to increase overall utility of the metaverse. This article presents design frameworks that consider how to address the metaverse as a hyper-connected meta-environment that connects and expands multiple user environments, modalities, contexts, and the many objects and relationships within them. This article contributes to i) a framing of the metaverse as a social XR-IoT (XRI) concept, ii) design Considerations for XRI metaverse experiences, iii) a design architecture for social multi-user XRI metaverse environments, and iv) descriptive exploration of social interaction scenarios within XRI multi-user metaverses. These contribute a new design framework for metaverse researchers and creators to consider the coming wave of interconnected and immersive smart environments.
... However, at this stage, there remains a need for clarity in the design and development process for XRI applications, with several new and emerging sub-themes. This work recognizes this convergent need and aims to contribute by applying a multidimensional methodology to the problem of interface design within the IoT, for agency in mixed reality [22], and also by the development of a new system architectural framework that blends components from 3D development, web machine learning, publish-subscribe connectivity, and conventional IoT development frameworks. An early functional prototype is presented, alongside a descriptive evaluation. ...
... Mixed Reality Agents (MiRAs) are defined as agents embodied in the Mixed Reality environment [22]. Basing this research on the definition of agency given by [45], an agent is defined as a hardware-or-software-based entity characterised by the following attributes:Autonomy: Agents can operate without the direct intervention of humans or others, and have control over their actions and internal state. ...
... To investigate Mixed Reality Agents, Holz et al [22] establishes [29]; an extension to the related work found in [28]. No. ...
The internet-of-things (IoT) refers to the growing number of embedded interconnected devices within everyday ubiquitous objects and environments, especially their networks, edge controllers, data gathering and management, sharing, and contextual analysis capabilities. However, the IoT suffers from inherent limitations in terms of human-computer interaction. In this landscape, there is a need for interfaces that have the potential to translate the IoT more solidly into the foreground of everyday smart environments, where its users are multimodal, multifaceted, and where new forms of presentation, adaptation, and immersion are essential. This work highlights the synergetic opportunities for both IoT and XR to converge toward hybrid XR objects with strong real-world connectivity, and IoT objects with rich XR interfaces. The paper contributes i) an understanding of this multi-disciplinary domain XR-IoT (XRI); ii) a theoretical perspective on how to design XRI agents based on the literature; iii) a system design architectural framework for XRI smart environment development; and iv) an early discussion of this process. It is hoped that this research enables future researchers in both communities to better understand and deploy hybrid smart XRI environments.
... In Multi-Agent Systems (MAS), agent-to-agent communication heavily relies on agent communication languages (ACLs) such as FIPA-ACL, standardized by the Foundation for Intelligent Physical Agents(FIPA) consortium [19,[37][38][39]. However, in mixed reality environments, where bots and humans share and collaborate in digital spaces, communication cannot rely only on ACLs and APIs [40]. ...
Software bots operating in multiple virtual digital platforms must understand the platforms' affordances and behave like human users. Platform affordances or features differ from one application platform to another or through a life cycle, requiring such bots to be adaptable. Moreover, bots in such platforms could cooperate with humans or other software agents for work or to learn specific behavior patterns. However, present-day bots, particularly chatbots, other than language processing and prediction, are far from reaching a human user's behavior level within complex business information systems. They lack the cognitive capabilities to sense and act in such virtual environments, rendering their development a challenge to artificial general intelligence research. In this study, we problematize and investigate assumptions in conceptualizing software bot architecture by directing attention to significant architectural research challenges in developing cognitive bots endowed with complex behavior for operation on information systems. As an outlook, we propose alternate architectural assumptions to consider in future bot design and bot development frameworks.
... In Multi-Agent Systems (MAS), agent-to-agent communication heavily relies on agent communication languages (ACLs) such as FIPA-ACL, standardized by the Foundation for Intelligent Physical Agents(FIPA) consortium [19,[37][38][39]. However, in mixed reality environments, where bots and humans share and collaborate in digital spaces, communication cannot rely only on ACLs and APIs [40]. ...
Software bots operating in multiple virtual digital platforms must understand the platforms’ affordances and behave like human users. Platform affordances or features differ from one application platform to another or through a life cycle, requiring such bots to be adaptable. Moreover, bots in such platforms could cooperate with humans or other software agents for work or to learn specific behavior patterns. However, present-day bots, particularly chatbots, other than language processing and prediction, are far from reaching a human user’s behavior level within complex business information systems. They lack the cognitive capabilities to sense and act in such virtual environments, rendering their development a challenge to artificial general intelligence research. In this study, we problematize and investigate assumptions in conceptualizing software bot architecture by directing attention to significant architectural research challenges in developing cognitive bots endowed with complex behavior for operation on information systems. As an outlook, we propose alternate architectural assumptions to consider in future bot design and bot development frameworks.Keywordscognitive botcognitive architectureproblematization
... In terms of agents, a mixed reality agent, otherwise known as an MiRA, can be defined as hardware or software based entities that have the conventional agent systems theory attributes: Autonomy, Social Ability, Reactivity, and Pro-Activity [10] and that exist with a mixture of virtual and physical embodiment (usually with stronger virtual embodiment versus physical embodiment) [11]. In particular, the dimensions of agency can be combined with dimensions of interactive capacity, and corporeal presence, as defined in [11], providing a method to consider a spectrum of MiRA implementations. ...
... In terms of agents, a mixed reality agent, otherwise known as an MiRA, can be defined as hardware or software based entities that have the conventional agent systems theory attributes: Autonomy, Social Ability, Reactivity, and Pro-Activity [10] and that exist with a mixture of virtual and physical embodiment (usually with stronger virtual embodiment versus physical embodiment) [11]. In particular, the dimensions of agency can be combined with dimensions of interactive capacity, and corporeal presence, as defined in [11], providing a method to consider a spectrum of MiRA implementations. In this, embodiment can be considered in different ways; one way MiRAs can do this is to demonstrate structural coupling, in which a system is embodied if it is able to sense, affect and be affected by its environment [11], [12]. ...
... In particular, the dimensions of agency can be combined with dimensions of interactive capacity, and corporeal presence, as defined in [11], providing a method to consider a spectrum of MiRA implementations. In this, embodiment can be considered in different ways; one way MiRAs can do this is to demonstrate structural coupling, in which a system is embodied if it is able to sense, affect and be affected by its environment [11], [12]. They can also have social embodiment, in which an agent can interact with users and other agents within a social framework or environment [11], [13]. ...
The internet-of-things (IoT) refers to the growing field of interconnected pervasive computing devices and the networking that supports smart, embedded applications. The IoT has multiple human-computer interaction challenges due to its many formats and interlinked components, and central to these is the need to provide sensory information and situational context pertaining to users in a more human-friendly, easily understandable format. This work addresses this by applying mixed reality toward expressing the underlying behaviors and states internal to IoT devices and IoT-enabled objects. It extends the authors' previous research on IoT Avatars (mixed reality character representations of physical IoT devices), presenting a new head-mounted display framework and interconnection architecture. This contributes i) an exploration of mixed reality for smart spaces, ii) an approach toward expressive avatar behaviors using fuzzy inference, and iii) an early functional prototype of a hybrid physical and mixed reality IoT-enabled object. This approach is a step toward new information presentation, interaction, and engagement capabilities for smart devices and environments.
... Another level of the AR spectrum is mixed reality (MR), which was rst described by Milgram and Kishino in 1994 [30] as "a subclass of virtual reality (VR) related technologies that involve the merging of the real and virtual worlds". Different from assisted reality systems, MR increases the user's spatial and visual interaction possibilities [31]. MagicBook is considered one of the rst examples of an MR system [32]. ...
Modern livestock farm technologies allow operators to have access to a multitude of data thanks to the high number of mobile and fixed sensors available on both the livestock farming machinery and the animals. These data can be consulted via PC, tablet, and smartphone, which must be handheld by the operators, leading to an increase in the time needed for on-field activities. In this scenario, the use of augmented reality smart glasses could allow the visualization of data directly in the field, providing for a hands-free environment for the operator to work. Nevertheless, to visualize specific animal information, a connection between the augmented reality smart glasses and electronic animal identification is needed. Therefore, the main objective of this study was to develop and test a wearable framework, called SmartGlove that is able to link RFID animal tags and augmented reality smart glasses via a Bluetooth connection, allowing the visualization of specific animal data directly in the field. Moreover, another objective of the study was to compare different levels of augmented reality technologies (assisted reality vs. mixed reality) to assess the most suitable solution for livestock management scenarios. For this reason, the developed framework and the related augmented reality smart glasses applications were tested in the laboratory and in the field. Furthermore, the stakeholders’ point of view was analyzed using two standard questionnaires, the NASA-Task Load Index and the IBM-Post Study System Usability Questionnaire. The outcomes of the laboratory tests underlined promising results regarding the operating performances of the developed framework, showing no significant differences if compared to a commercial RFID reader. During the on-field trial, all the tested systems were capable of performing the task in a short time frame. Furthermore, the operators underlined the advantages of using the SmartGlove system coupled with the augmented reality smart glasses for the direct on-field visualization of animal data.
... Living and working in immersive hybrid virtual and physical worlds with smart devices (i.e., head-mounted displays) may become as commonplace as the mobile phone today. Within these speculative futures parameters, this work explores an extended metaverse framework to enhance embodiment, interaction, and agency [10] that could provide a seamless interface to the physical world. ...
... The project explores an extended metaverse framework for applying the metaverse layer in the physical space(s) to increase dynamic inter-connections of humans, agents, and the environment through MR and the IoT. The extended metaverse framework focuses on improving interaction, embodiment, and agency [10] in the human-in-the-loop MR space. The contributions of this include: i) an exploration of the metaverse disconnect problem and background literature, ii) an architectural framework for extending the metaverse, iii) design prototypes using the framework, (one for a hyper-connected space for time-awareness while in the metaverse; and the other for enabling users to transition between real and virtual environments where the environment helps to detect physical objects), and iv) a discussion-based evaluation of these approaches. ...
... Although this work is not comprehensively evaluated (such as via user study), the frameworks developed are a step toward future research in this direction. However, to show how these prototypes can relate to other work in this area, a subjective rating has been applied, based on a selection of factors (inspired by [10] [17] [14] [32]) considered deemed important for such systems. Table 2 presents the definitions of the factors for comparison, including embodiment, connectedness, content generation, mixed reality access, context awareness, virtual-physical agency and userinteractive level, with the meaning of the levels of very low, low, medium, high, and very high. ...
... Further concepts related to level of agency are also relevant. In terms of the agent in Metaverse, Mixed Reality Agents (MiRAs) address an agent with virtual or physical entities embodied in a Mixed Reality environment [19]. The key concepts of MiRAs are Agency, Embodiment, and Interaction Capability within Mixed Reality Environments, divided into the three-axis agency, corporeal presence, and interactive capacity. ...
... Based on a combination of theories including the MiRAs cube of [19], and the reality-virtuality continuum of Milgram [24], the domain of an Extended Metaverse Agent is depicted (see Fig. 4) extending along the dimensions of Mixed Reality Embodiment, Extended Interaction, and level of Agency. They represent how the virtual and physical entities can be presented on the virtual or physical space, and how they could interact with users and by other agent-objects. ...
... Criteria of Metaverse Agent (derived from the MiRAs taxonomy of[19]). ...
The Metaverse encompasses technologies related to the internet, virtual and augmented reality, and other domains toward smart interfaces that are hyper-connected, immersive, and engaging. However, Metaverse applications face inherent disconnects between virtual and physical components and interfaces. This work explores how an Extended Metaverse framework can be used to increase the seamless integration of interoperable agents between virtual and physical environments. It contributes an early theory and practice toward the synthesis of virtual and physical smart environments anticipating future designs and their potential for connected experiences.
