ArticlePDF Available

Abstract and Figures

A new kind of smart space is emerging in which digital, physical, and social layers are strongly intertwined. These spaces extend the classic assistive functionality of ambient intelligence toward more proactive possibilities, where the smart environment not only monitors people as they perform tasks but also influences their plans and intentions. The authors explore this concept of the smart space as a mirror world, looking in particular at how it will affect our cognitive abilities and noting some of the research challenges that will need to be addressed. This department is part of a special issue on smart spaces.
Content may be subject to copyright.
60
PERVA SIV E
computing
Publis hed by the IE EE CS n 153 6 -126 8/15/$31.0 0 © 2 015 IE EE
Spotlight
In recent years, impressive hardware
technologies have been developed
that let mobile and embedded devices
better exploit the Internet and Web
and better interact with the physical
world. Such developments are making
futuristic scenarios involving ambient
intelligence (AmI) and smart environ-
ments an everyday reality by integrat-
ing research contributions from the
elds of ubiquitous computing, sensor
networking, and articial intelligence.
Such AmI and smart environment
scenarios will be further enriched by
augmented reality (AR) and mixed-
reality technologies. We’ll be able to
create a digital layer to augment the
physical world with situated informa-
tion and services that can be perceived
by wearable devices, such as smart
glasses. Conceptually, such devices will
be able to seamlessly extend people’s
cognitive capabilities of perception and
reasoning to inuence their actions in
both the real and augmented world.
Furthermore, the scale of the envi-
sioned scenarios is no longer limited
to rooms or houses. The development
of network and cloud technologies
and the Internet of Things enables
large-scale smart spaces, scaling from
small environments to entire cities.
This implies going from mobile apps
to urban apps—that is, applications
running on city-level (software and
hardware) infrastructures—creating an
open ecology of interconnected smart
spaces and services.
All this leads to a new kind of smart
space in which digital, physical, and social
layers are strongly intertwined. These
spaces extend the classic assistive func-
tionality of AmI toward more proactive
possibilities, where the smart environment
not only monitors people as they perform
tasks, or supports them by executing their
requests, but also inuences and changes
their plans and intentions.
A main research challenge will be to
dene a conceptual foundation for these
spaces, effective enough to model open,
possibly large-scale smart environments
and their interaction with humans. The
vision of mirror worlds presents one
possibility for modeling these spaces.1
The name “mirror world” is a tribute to
David Gelernter’s book,2 which inspired
some of the main concepts behind the
idea of integrating research visions devel-
oped in the context of agents and multia-
gent systems, pervasive computing, and
augmented reality.
Here, we explore this concept of a
mirror world, looking in particular at
how it will affect our cognitive abilities
and noting some of the research chal-
lenges that will need to be addressed.
SMART SPACES AS MIRROR
WORLDS
In the mirror world vision, smart
spaces are modeled in terms of digital
cities shaped by the physical world
with which they’re coupled, inhabited
by open societies and organizations of
software agents playing the role of the
inhabitants of those cities.
Two Worlds Collide
Mirroring occurs when physical things,
which can be perceived and acted upon
by humans in the physical world, have
a digital counterpart (or extension) in
the mirror world, so that they can be
observed and acted upon by agents.
Vice versa, an entity in the mirror world
that can be perceived and acted upon
by software agents can have a physical
appearance (or extension) in the physi-
cal world—for example, through AR—
so that it can be observed and acted
upon by humans.
This implies a form of coupling, such
that an action on an object in the physi-
cal world causes some kind of changes
in entities in the mirror world, perceiv-
able by software agents. Similarly, an
action by agents on an entity in the
mirror world can have an effect on
things in the physical world, perceiv-
able by people. As mirror world citi-
zens, agents are responsible for autono-
mously performing tasks by observing
or using the mirror world things in their
environment.
So, in this scenario, we don’t just
live in our “real” world, with a parallel
second life in the virtual or simulated
The Mirror World: Preparing
for Mixed-Reality Living
Alessandro Ricci, University of Bologna
Michele Piunti, Reply, Inc.
Luca Tummolini and Cristiano Castelfranchi,
Institute of Cognitive Science and Technologies
APRIL–JUNE 2015
PERVA SIV E
computing
61
world; rather, we will live in both
worlds, which together comprise an
“augmented” reality.
A Gaming Example
As a clarifying example, consider an
extension of the mobile AR game
Ghosts in the City modeled as a mir-
ror world (see Figure 1). The mirror
world comprises a collection of trea-
sures and ghosts distributed in some
part of a real-world city (mapped out
in the mirror world). There are two
teams of human players. Their objec-
tive is to collect as many treasures as
possible—by actually walking around
in the city—without being caught by
the ghosts.
Players have smart glasses, and they
use their smartphone as a magic wand.
Ghosts are agents autonomously mov-
ing around in the mirror world—and
in the city (via devices or sensors).
Players perceive ghosts using their AR
glasses—the ghosts appear when the
player approaches. Ghosts can also
perceive nearby players. Ghosts aim
to catch human players, so they start
to follow a player soon as they per-
ceive one. A ghost catches a player by
grabbing his or her body in the mirror
world, which is physically perceived by
humans when their magic wand (smart-
phone) vibrates.
Different kinds of ghosts prefer dif-
ferent locations depending on the physi-
cal parameters—the level of humidity,
amount of light, temperature, and so on.
So ghosts can perceive the physical world
too using their own looking glasses in the
mirror world. Players can exploit this fact
to avoid being captured. For example, if a
player is being chased by a ghost known
to be intolerant to light, the player can
run under a street lamp. However, a
ghost might have the power to switch
off the physical lights (if the lights have a
counterpart in the mirror world that can
be controlled by agents).
Also, a player with enough power
can use his or her magic want to create
temporary holes in the ground, which
can absorb ghosts. Yet ghosts can
collaborate to create team strategies
to catch players by encircling them, for
example.
Although this is just a game, it
exemplifies a number of features
found in a mirror world concerning
the action and perception of human
actors and agents in the augmented,
mixed environment.
AUGMENTING HUMAN
COGNITION AND SOCIALITY
Given the tight coupling between the
physical world and its mirror, the vision
of a mirror world will have a profound
impact on human cognitive systems and
how human societies function.
Already, our brains augment our
reality. Using our imagination, we
can explore possible actions and their
effects to mentally solve problems—
with less costs and risks than actually
performing the actions. Moreover, our
brain serves to integrate our current
perception of the physical world with
our top-down memory-based predic-
tions. We “see” much more than what
impinges on our retina: we see balls and
spheres even if we just receive the stim-
uli of a semisphere, or we perceive that
an ambulance is coming even if we just
faintly hear a siren. Even if natural cog-
nition is already a form of “augmented
reality,” a mirror world will multiply
Ghost
agent
Ghost
body Street
lamp
Player
body
Player
assistant
agent
Mirror world
Physical world
8
Figure 1. The main elements of the game Ghosts in the Cit y. The game’s virtual
(mirror) world includes the following: human player bodies (coupled with the
physical ones via smartphones), perceivable by the ghost agents; ghost bodies,
controlled by ghost agents, with an extension in the physical world to make them
perceivable by human players; and items such as street lamps, which can have both
a physical and mirror world extension, each providing different kinds of (coupled)
functionalities and affordances.
62
PERVA SIV E
computing
www.computer.org/pervasive
Spotlight
Spotlight
this augmentation because, while we
act, we will have innite memory, data,
processing power, and simulation capa-
bilities offered by the parallel and inte-
grated computational world. Much of
our intelligence will be articial but in
real time, changing what we see, imag-
ine, and can do.
Moreover, once the mirror world is in
place, we will act in this augmented real-
ity with the help of (more or less hidden)
articial intelligence. What will emerge
is a new extended mindas depicted
by Andy Clark in Supersizing the Mind:
Embodiment, Action and Cognitive
Extension (Oxford University Press,
2010): a mind in which cognitive func-
tions are distributed well beyond the
limits of our skulls and brains. This
mixed articial and natural brain will
be both individual and social: a collec-
tive, cooperative intelligence that will
be experienced in several cases as our
own intelligence/mind.
Such a hybrid mind will be necessary
for dealing with the new augmented
and mixed (mirror) world, where other
eyes, senses, actions, data, and rea-
soning are needed. When will we feel
these functions as “our own” extended
mental (and practical/behavioral)
activities, and when will we perceive
them as a cooperative “team” activ-
ity? Furthermore, when might they be
perceived as an animistic presence in
the world? Will we have an external-
ized “mirror” super-ego? All these psy-
chologies will be possible in our mirror
world.
As a case in point, consider the feel-
ing of individual agency—that is, to be
the author and initiator of your own
actions. A feeling of agency is indeed
crucial to sustain our sense of control
and, ultimately, moral responsibility for
what we do. Recent research in the cog-
nitive sciences has revealed that such
feeling of agency depends on the per-
ceived congruence between predicted
and actual outcomes, both in individual
and joint action contexts.3 To avoid dis-
rupting this core cognitive process, the
extended range of agency of individuals
and groups that will be available in mir-
ror worlds should match this require-
ment by design. In particular, it should
be possible to support the simulation of
individual and combined effects in the
mirror world in a way that is shared and
observable both by human and articial
agents, is updated in runtime, and can
be matched to perceived results in the
physical world.
In addition, inhabiting a mirror world
could also profoundly affect collective
behavior. First, at a basic level, mirror
worlds allow for augmenting the design
space of interaction mechanisms—in
particular, for emergent forms of coor-
dination, such as stigmergy and behav-
ioral implicit communication.1 An easy
example is a virtual post-it, with a mes-
sage observable by nearby humans and
perceivable and modiable by software
agents in the mirror world.
Second, at a higher level, mirror
worlds can also support new solutions
to collective-action problems by mak-
ing the effect of individual behaviors
explicit. Modifying dysfunctional
collective patterns is often hampered
by specic cognitive limitations. For
example, the perceived negligence of
individual contributions to such col-
lective outcomes might downsize your
Mirror world fabric
Mirror world middleware
Mirror worlds
Agents
Things
(agent environment)
A mirror
world
- UbiqStacks
- Web of Things
- AR technology
- Agent and MAS technology
- Web and cloud
- Internet of Things
- Mobile technologies
- Wearable technologies
Web REST API
Web/cloud-based distributed
middleware & infrastructure
Figure 2. An abstract view of the stack of enabling technologies for mirror worlds,
comprising three layers: a mirror world app layer—the mirror world as the individual
applications running on top of the mirror world middleware layer (the always-on
infrastructure that functions as operating systems for the mirror world, providing
a set of core APIs and managing their execution), and a fabric layer—the low-level
distributed infrastructure including the network and every hardware/software
resource needed to run the mirror world infrastructure.
APRIL–JUNE 2015
PERVA SIV E
computing
63
Spotlight
Selected CS articles and columns
are also available for free at
http://ComputingNow.computer.org.
own personal responsibility. Also, the
complexity of anticipating any negative
consequences of present-directed deci-
sions might obscure individual and
collective interests. Additionally, well-
known cognitive biases discount the
value of larger future rewards in favor
of smaller but temporally closer ones
and thus ease myopic and poor choices.
All of these cognitive limitations
might be better addressed in a mirror
world, whose role in making these com-
bined and temporally extended effects
visible and perceivable could inuence
individual decision making in ways not
currently possible.
The concrete development of a mir-
ror world calls for integrating a
variety of existing enabling technolo-
gies—the Internet of Things and mixed
reality in particular—into a three-layer
stack (Figure 2), which closely resem-
bles existing ubiquitous computing
stacks. In addition, we’ll need to learn
how to concretely and systematically
use the mirror world to explore interdis-
ciplinary aspects concerning large-scale
mixed-reality smart spaces—from cog-
nition to interaction, cooperation, and
governance.
Finally, as Gregory Abowd has noted,4
after two decades of research, ubicomp
is going to disappear as a niche research
eld, becoming an intellectual domain
of all of computing. This requires us to
devise the next generation of comput-
ing, which should be effective enough
to conceive, design, and develop future
pervasive and smart computing apps
in-the-large, as we do with traditional
applications today. In that perspective,
a main research point is understanding
how mirror words can affect how we
think about computing, programming,
and designing systems.
REFERENCES
1. C. Castelfranchi et al., “AmI Systems
as Agent-Based Mirror Worlds: Bridg-
ing Humans and Agents through
St ig mer gy,” Agents and Ambient Intel-
ligence, T. Bosse, ed., IOS Press, 2012,
pp. 17– 31.
2. D. Gelernter, Mirror Worlds: Or the
Day Software Puts the Universe in a
Shoebox... How It Will Happen and
What It Will Mean, Oxford, 1993.
3. E. Pacherie, “Sense of Agency: Many
Facets, Multiple Sources,” Agency
and Joint Atte ntion , H.S. Terrace and
J. Metcalfe, eds., Oxford Univ. Press,
2013, pp. 321–345.
4. G.D. Abowd, “What Next, Ubicomp?
Celebrating an Intellectual Disappearing
Act,” Proc. 2012 ACM Conf. Ubiquitous
Computing (UbiComp), 2012, pp.31–40.
Alessandro Ricci is an asso-
ciate professor in the Depart-
ment of Computer Science
and Engineering at the Uni-
versity of Bologna, Italy. Con-
tact him at a.ricci@unibo.it.
Michele Piunti is a solu-
tion architect at Reply Inc.
Contact him at m.piunti@
reply.eu.
Luca Tummolini is a
researcher at the Institute of
Cognitive Sciences and Tech-
nologies, Italy. Contact him
at luca.tummolini@istc.cnr.it.
Cristiano Castelfranchi
is a professor of cognitive
psycholo gy and an associate
researcher at the Institute
of Cognitive Sciences and
Technologies, Italy. Contact
him at cristiano.castelfranchi@istc.cnr.it.
Newsletters
Stay Informed on
Hot Topics
computer.org/newsletters
... The future holds potential to be a hybrid one that seamlessly decorates our physical world with digital information (Weiser, 1999;Abowd, 2012;Ricci et al., 2015;Fuchsberger, 2019). In those environments the information from physical, social, and digital sources needs to be intertwined (Ricci et al., 2015). ...
... The future holds potential to be a hybrid one that seamlessly decorates our physical world with digital information (Weiser, 1999;Abowd, 2012;Ricci et al., 2015;Fuchsberger, 2019). In those environments the information from physical, social, and digital sources needs to be intertwined (Ricci et al., 2015). As we approach this hybrid future ( Figure 5), we need to improve our understanding on how we can make use of various context sources, including humans themselves. ...
Article
Full-text available
In recent years, everyday activities such as work and socialization have steadily shifted to more remote and virtual settings. With the COVID-19 pandemic, the switch from physical to virtual has been accelerated, which has substantially affected almost all aspects of our lives, including business, education, commerce, healthcare, and personal life. This rapid and large-scale switch from in-person to remote interactions has exacerbated the fact that our current technologies lack functionality and are limited in their ability to recreate interpersonal interactions. To help address these limitations in the future, we introduce “Telelife,” a vision for the near and far future that depicts the potential means to improve remote living and better align it with how we interact, live and work in the physical world. Telelife encompasses novel synergies of technologies and concepts such as digital twins, virtual/physical rapid prototyping, and attention and context-aware user interfaces with innovative hardware that can support ultrarealistic graphics and haptic feedback, user state detection, and more. These ideas will guide the transformation of our daily lives and routines soon, targeting the year 2035. In addition, we identify opportunities across high-impact applications in domains related to this vision of Telelife. Along with a recent survey of relevant fields such as human-computer interaction, pervasive computing, and virtual reality, we provide a meta-synthesis in this paper that will guide future research on remote living.
... By pushing to its limits the idea to have a pervasive substrate of DTs, not only mirroring physical objects and equipment, but also providing digital artefacts embodied in some way into our physical reality (e.g. via holograms), we get to the idea of mirror worlds, as originally inspired by D. Gelernter in [13], and further explored and developed in the context of agents and multi-agent systems in [32]. Following Gelernter, mirror worlds are "software models of some chunk of reality" [13], that is: "a true-to-life mirror image trapped inside a computer ", which can be then viewed, zoomed, analysed by real-world inhabitants with the help of proper software (autonomous) assistant agents. ...
Preprint
Full-text available
Digital Twins (DTs) are rapidly emerging as a fundamental brick of engineering cyber-physical systems, but their notion is still mostly bound to specific business domains (e.g. manufacturing), goals (e.g. product design), or application domains (e.g. the Internet of Things). As such, their value as general purpose engineering abstractions is yet to be fully revealed. In this paper, we relate DTs with agents and multiagent systems, as the latter are arguably the most rich abstractions available for the engineering of complex socio-technical and cyber-physical systems, and the former could both fill in some gaps in agent-oriented engineering and benefit from an agent-oriented interpretation -- in a cross-fertilisation journey.
... Today's smart environments not only play a role in monitoring and task execution, but pointedly they also directly influence and serve to inform people's decisionmaking in undertaking routine daily activities [1]. The increasing dataset scales brought by the smart environment revolution however reveals the limitations of traditional 2D-screen-based smart environment data visualization interfaces [2]- [5]: decision-makers have to spend an additional cognitive load on searching and filtering valuable Internet of Things (IoT) data from the centralized long data lists. ...
Article
Full-text available
The Internet of Things (IoT) provides unprecedented opportunities for the access to and conflation of a myriad of heterogeneous data to support real-time decision-making within smart environments. Augmented Reality (AR) is on cusp of becoming mainstream and will allow for the ubiquitous visualization of IoT derived data. Such visualization will simultaneously permit the cognitive and visual binding of information to the physical object(s) to which they pertain. Important questions exist as to how one can efficiently filter, prioritize, determine relevance and adjudicate on individual information needs in support of real-time decision making. To this end, this paper proposes a novel AR decision support framework (STARE) to support immediate decisions within a smart environment by augmenting the user’s focal objects with assemblies of semantically relevant IoT data and corresponding suggestions. In order to evaluate this technique, a remote user study was undertaken within a simulated smart home environment. The evaluation results demonstrate that the proposed Semantic Augmented Reality decision support framework leads to a reduction in information overloading and enhanced effectiveness, both in terms of IoT data interpretation and decision support.
... Some future implications of remotely performed activities may be beyond comprehension, so understanding how physical things and their digital counterparts inhabit telelife will play a major role in future research Mayer et al. (2018). Our future tends to be a hybrid one that will be composed of smart environments where information from physical, social, and digital sources is intertwined Ricci et al. (2015). The digital twin of a physical system accounts for its real-world interactions between the environment and physical models through realistic simulations to update changes accordingly Glaessgen and Stargel (2012). ...
Preprint
Full-text available
In recent years, everyday activities such as work and socialization have steadily shifted to more remote and virtual settings. With the COVID-19 pandemic, the switch from physical to virtual has been accelerated, which has substantially affected various aspects of our lives, including business, education, commerce, healthcare, and personal life. This rapid and large-scale switch from in-person to remote interactions has revealed that our current technologies lack functionality and are limited in their ability to recreate interpersonal interactions. To help address these limitations in the future, we introduce "Telelife," a vision for the near future that depicts the potential means to improve remote living better aligned with how we interact, live and work in the physical world. Telelife encompasses novel synergies of technologies and concepts such as digital twins, virtual prototyping, and attention and context-aware user interfaces with innovative hardware that can support ultrarealistic graphics, user state detection, and more. These ideas will guide the transformation of our daily lives and routines soon, targeting the year 2035. In addition, we identify opportunities across high-impact applications in domains related to this vision of Telelife. Along with a recent survey of relevant fields such as human-computer interaction, pervasive computing, and virtual reality, the directions outlined in this paper will guide future research on remote living.
... Günümüzde yaşadığımız gerçekliklerin dijital bir kopyasını oluşturmak ayna dünya olarak tanımlanmaktadır (4) . Sağlık alanında bu şekilde yapılan çalışmalar ile yenilikçi içerikler üretilebilmektedir (5) . ...
Article
Full-text available
Objective: To create contents that allow students to be experienced in 3D with 360-degree video and 3-dimensional scenes in a real environment for student/assistant training and distance online medicine education. Method: The 360 degree video recordings were made with Samsung Gear 360 in three different areas: Classroom, Operating Room and Case Meeting in the medical school. A 3D scene was designed with Autodesk 3dsMax. 360-degree videos and 3D scene were combined in Unity program. The Augmented Reality (AR) application was developed within the Unity and the mobile application was tested by different users. AR experience is now available with the mobile application. Results: As part of this study, three 360-degree videos were recorded and the augmented reality mobile application was developed. Conclusion: In this study, the pilot study was completed by producing content with 360-degree videos and augmented reality technology for use in medical education. We plan to investigate the eficacy of these mirror world contents to medical education with further studies.
Article
Full-text available
Virtual geographical environments (VGEs) are being used to represent our sense of place through the application of extended-reality or cross-reality (XR) technologies with a focus on technological advancement and immersion. In light of this, we propose that an expanded and interdisciplinary understanding of the concept of immersion is required to facilitate an understanding of spatial data to a broader audience. The potential of focusing on narrative immersion and literary placemaking in VGEs is discussed to evoke a stronger sense of place and a feeling of presence and belonging in users. This paper seeks to highlight the unique affordances and potentialities of XR narrative style GIS digital representations through an interdisciplinary theoretically lead analysis of outputs created by the Building City dashboards project towards the goal of expanding audiences of non-specialist stakeholders in urban planning processes.
Chapter
This chapter describes examples of mixed reality (MR) poetry which spans a continuum from virtual reality (VR) to augmented reality (AR). There is clearly identifiable digital poetry being created in MR platforms such as Borsuk and Bouse’s Between Page and Screen (2016). However, MR works such as Laurie Anderson and Hsin-Chien Huang’s Chalkroom (2017) and Mez Breeze and Andy Campbell’s All the Delicate Duplicates (2017) are harder to define. These are projects that incorporate considerable use of poetic text so an argument could be made to categorise them as MR poetry; however, they could also be categorised as experiential games. This chapter uses these examples of mixed reality poetry as case studies in which to examine the changing role of words in virtual spaces.
Chapter
Extended Reality (XR) refers to applications that blend the digital and the physical worlds in different ways: both by situating virtual worlds into physical environments by means of Augmented and Mixed Reality Technologies, and by exploiting smart things and devices in the physical environment connected to the Virtual World, in a pervasive computing perspective. Like in the case of Virtual Worlds and Intelligent Virtual Environments, XR applications are a relevant application domain for multi-agent systems and AI—for instance, for designing XR-based smart environments. The research question addressed by this paper is about the definition of a model for conceiving and designing agent-based XR applications, effective enough to capture essential aspects in spite of the specific implementing technologies. To this purpose, the paper describes a model based on the Mirror World conceptual framework and a concrete platform used to evaluate its strengths and weaknesses.
Conference Paper
Weiser's landmark Scientific American article inspired many researchers to explore an exciting socio-technical vision of a third generation of computing. At the 21st anniversary of that published vision, I want to assess ubicomp's maturity and explore the identity challenge it faces. Today, ubicomp as a niche research topic no longer makes sense; we must celebrate its "disappearance" as a well-scoped research agenda because it has become a profound agenda across most of computing, and beyond. This should not be surprising; the 2nd generation of computing, the personal computer revolution, experienced the same profound disappearance. In celebration of this imminent disappearance, I will highlight the unique contributions of the ubicomp community, express some remaining intellectual challenges, and speculate on how to formulate new visions of computing that might succeed this third generation.
Sense of Agency: Many Facets, Multiple Sources
  • E Pacherie
E. Pacherie, "Sense of Agency: Many Facets, Multiple Sources," Agency and Joint Attention, H.S. Terrace and J. Metcalfe, eds., Oxford Univ. Press, 2013, pp. 321-345.