Content uploaded by Maria Luce Lupetti
Author content
All content in this area was uploaded by Maria Luce Lupetti on Oct 12, 2019
Content may be subject to copyright.
Design and Semantics of Form and Movement 87
Robot Citizenship:
A Design Perspective
Maria Luce Lupetti1, Roy Bendor1, Elisa Giaccardi1,2
1Faculty of Industrial Design, Delft University of Technology, Delft, The Netherlands
2Umea Institute of Design, Umea, Sweden
{m.l.lupetti; r.bendor; e.giaccardi}@tudelft.nl
Abstract
This paper suggests robot citizenship as a design
perspective for attending to the sociality of human-
robot interactions (HRI) in the near future. First, we
review current positions regarding robot citizenship,
which we summarize as: human analogy, nonhuman
analogy and socio-relationality. Based on this review,
we then suggest an understanding of citizenship that
stresses the socio-relational implications of the concept,
and in particular its potential for rethinking the way
we approach the design of robots in practice. We
suggest that designing for robot citizenship (in the terms
suggested by this paper) has the potential of fostering a
shift from a logic of functionality to one of relationality.
To illuminate the direction of this shift in design
practice, we include and discuss three robot concepts
designed to address and rethink present HRI challenges
in the urban environment from a relational perspective.
Keywords
Citizenship; Design Practice; Urban Robotics; Human-
Robot Interaction; More-Than-Human City
1. Introduction
Current technological advancements in the fields of
Artificial Intelligence (AI) and robotics have stirred a
lively debate about the nature of human-nonhuman
relationships and how these may be regulated. In this
debate, designers often find themselves caught in
between sensationalist attributions of citizenship to
humanoid robots such as Sophia [1], and more pragmatic
initiatives that contemplate the attribution of legal
personhood to robots, such as the European Parliament
Resolution of Civil Law Rules of Robotics [2]. Attempts to
regulate human-robot relationships through the typically
human construct of citizenship as a congruent set of
rights and responsibilities can also be observed in other,
more mundane cases. The widespread and unregulated
presence of delivery robots in San Francisco, for
example, has raised a series of social concerns to
which the city has responded with a strict regulation
that limits the number of delivery robots moving freely
around the city [3]. In contrast, the state of Arizona
has responded to similar concerns by giving the delivery
robots the same rights as pedestrians [4] as a way to
make them comply with the same rules.
Although robots have yet to populate the urban
environment en masse, unresolved challenges concerning
their social desirability and responsibility [5-8] call
for a deep reflection on what attributing citizenship
to robots may lead to. This, in turn, compels us to
rethink future urban environments as more-than-human
entanglements of human and nonhuman entities and
needs [9].
Design and Semantics of Form and Movement88 Design and semantics of form and movement88
While a socio-relational perspective of human and
nonhuman coexistence has gained attention in the
field of human-computer interaction (HCI), mainly in
research concerned with animals and plants [10-13], it
has yet to make an impact when it comes to the design
of human-robot interactions. This paper responds
to this gap with a speculative investigation into the
idea of robot citizenship. The paper first reviews the
current debate on the topic of robot citizenship as it
is underlined by three different rhetorical strategies:
using human analogies, using nonhuman analogies,
and pointing to socio-relationality. The paper then
explores the implications of robot citizenship as an
instantiation of the socio-relational perspective, and
in particular its potential for rethinking the way we
approach the design of robots in the urban context.
In doing so, the paper attempts to move away from
addressing citizenship normatively as a codified set of
rights and responsibilities by proposing an alternative,
complementary design perspective [14] meant to
challenge and inspire practitioners to shift from a logic
of functionality to one of relationality.
2. Robot Citizenship
Investigations of the social implications of robots
through legal categories such as rights, personhood,
and citizenship, have become increasingly frequent
within academe [15-21]. In the following sections, we
draw upon, and extend, existing work on the topic to
describe three main rhetorical strategies through which
the concept of robot citizenship can be approached:
human analogy, nonhuman analogy, and socio-
relationality.
2.1 Human Analog y
The first perspective is grounded in the idea that in
the future, robots, especially when powered by AI, will
become so sophisticated that they will be practically
indistinguishable from humans in terms of cognitive
abilities, sentience, and self-awareness. In such Blade
Runner-like scenarios robots may be eligible for rights
and even citizenship. An extensive argument from this
perspective is provided by Marx and Tiefensee [18].
Although they remain skeptical about robots becoming
fully sentient, they envision functionally equivalent states
that would enable robots to perceive and preserve
their “wellbeing”, making them worthy of protection as
“moral patients”, as Gunkel [21] puts it. Similarly, due to
their growing complexity and sophistication, robots may
also be able to detect moral demands from other agents
and, accordingly, behave responsibly. Because of such
ability to hold both rights and responsibilities, Mark
and Tiefensee [18] argue that robots may also become
citizens.
Argumentation grounded in possible future abilities of
robots, however, is often contested [21-23] because
it relies on overvaluations of the actual capabilities of
even the most advanced robots. In this sense, such
argumentation reads more like science fiction than
plausible foresight [23].
2.2 Nonhuman Analogy
An alternative perspective on citizenship that better
accounts for current robots’ abilities is based on
attributing citizenship to “useful” nonhumans. Kymlicka
and Donaldson [24], for instance, discuss animal
citizenship by focusing on the concept of domestication
as a qualifying relationship. Domesticated animals, they
note, can be seen as citizens because of their ability to
regulate their behavior according to norms of civility
(thus respecting the rights of other members), and
because of their ability to perform their duties (thus
providing a meaningful service for the community).
Although criticized when it comes to animals [18],
this argumentation can be applied to robots who
fulfil the same criteria. There are precedents: rights
were recently granted to nonhumans by virtue of
their membership, contribution and relationship
with the human community. The Whanganui river
in New Zealand, for instance, was granted the same
legal rights as humans after a local Māori tribe fought
for its recognition as an ancestor and a contributing
member to the welfare and wellbeing of the tribe [25].
As Forlano [25] explains, “by granting the river legal
rights, crimes against the river can now be treated as
crimes against the tribe”. Similar initiatives include the
attribution of rights to the Ganges and Yamuna rivers in
India [26] and Lake Erie in the United States [27].
However, attributing human legal rights to nonhumans
may lead to open conflicts between the interests of the
two, as in is the case of Lake Erie. The attribution of
Design and Semantics of Form and Movement 89
rights to the lake, in fact, generated a protest from local
farmers who claimed that their rights (endangered by
the impossibility of fertilizing their crops because of the
protection of the lake) should be anteposed to the ones
of the lake [27]. Despite the merit of accounting for the
expanded nature of communities and of stressing the
importance of contribution to a shared good, then, this
perspective remains controversial.
2.3 Socio -Relationalit y
While both previous perspectives provide convincing,
even if controversial answers to whether and how
we could consider robots as citizens, we argue
that addressing robots through a logic of rights and
responsibilities only provides partial answers. As
Coeckelbergh [22] argues in his discussion on robots
and morality, by focusing on a robot’s individual
features, the rights approach does not account for how
relations among entities and the social context itself
contribute to changes in moral considerations. Instead,
we could approach moral considerations from a socio-
relational perspective [21-22] in which morality should
not be seen as inherent to any single entity but rather
as an extrinsic quality. What this means is that a robot
should not be addressed as a moral agent or “patient”
[28] per se, but as an object of moral consideration by
virtue of its relations within a social context.
Accordingly, the socio-relational perspective goes
beyond individual abilities, and accounts for the relations
between the individual and the whole. In other words,
attributing citizenship to robots should not be based on
the question of whether robots are “like us”, or “help
us”, but are “part of us” – a point also made by Marx
and Tiefensee [18] in their account for citizenship based
on robot sophistication, and by Kymlicka and Donaldson
[24] in their discussion of citizenship for domesticated
animals. Going beyond rights and responsibilities, a
citizen, to be qualified as such, should be engaged with
other entities in interdependent relations aimed at a
collective welfare.
By firmly shifting the emphasis from a logic of rights
and responsibilities to one of relations, then, this
perspective reveals the need for a richer vocabulary
(that would, for instance, help differentiate hard from
soft rights [22]), or a completely new one that would
enable us to account for the different forms of moral
considerations that arise from new human-robot shared
performances.
3. A Design Perspective on Robot
Citizenship
The preceding discussion hints at how the concept of
robot citizenship may help the HRI field to extend its
interest from technical concerns to social ones, shifting
its focus from pragmatic and technical challenges to
topics like relationality and ethics. Given that current
debates on robot citizenship tend to focus on normative
questions and seek resolution in policy and regulation,
a process that tends to react to technological
developments instead of anticipating them, we suggest
there is value in considering robots as citizens as a
matter of philosophical and designerly speculation. In
other words, we are not interested in offering legalistic
solutions for the more-than-human city or a critical
speculation on the near future, but in opening up a
provocative design space.
By introducing the notion of robot citizenship in this
way (in terms of relationality and not legality), we invite
designers to look at emerging human-robot interactions
not as a matter of individual robotic capabilities but
as a matter of the relations among robotic and non-
robotic entities. Discussing robot citizenship, therefore,
invites designers to approach the design of HRI from
considerations of the community, its values and shared
goals, instead of from the individual robot’s functional
capabilities. Through this conceptual shift, considering
robot citizenship may not only contribute to the
ongoing discussion about meaningful future partnerships
between humans and computational artefacts [29-33],
but also contribute to a more holistic view of HRI.
The question remains, how can this conceptual shift be
translated into actionable design strategies?
As described above, approaching design from the
perspective of robot citizenship asks us to rethink the
performance of robots interdependently, and thus
investigate the appropriateness of the relationships
between robots and other entities instead of robots on
their own. As summarized in Figure 1, the design space
that opens up in response to thinking about robots
as citizens requires that we understand how a robot
Design and Semantics of Form and Movement90
may enter into relation with both other entities and
the environment. From this initial framing, then, it is
possible to identify opportunities for meaningful and
appropriate partnerships by reflecting on the possible
roles that both human and nonhuman entities may be
asked to perform together, on the basis of what Kuijer
and Giaccardi [33] refer to as capabilities that are
“uniquely human” and “uniquely artificial”.
To this suggestion we add a specific perspective:
instead of looking only at what individual entities are
‘good at’, we suggest considering what they are not
good at. As Marenko and van Allen [31] argue, most
current approaches to interaction design tend to be
task-oriented and efficiency-driven, and therefore
tend to produce specific narratives about devices as
consistently behaving entities towards which people
often build inappropriate expectations. By recognizing
limits and coming to terms with unpredictability, and
by suggesting narratives of “dumb-smart” [34] entities,
designers can free themselves from the idea of designing
for perfection and redirect their actions towards
“ecologies of things that are mutually responsive and
interdependent” [31].
Once they identify such inabilities, designers can explore
how the same performance is successfully instantiated
by other entities, as a way to envision possible design
alternatives. Among these, we suggest focusing on
strategies that communicate interdependency and may
foster values that can be considered appropriate for
human-robot interactions. This perspective helps us not
only to pivot towards relationality and interdependence,
but also to shift our focus to the extrinsic (rather than
intrinsic) qualities of a robot that can enable appropriate
forms of interaction. To do so, we need to understand
what Coeckelbergh [22] calls “apparent features”,
according to which the features of a robot are not
appropriate or morally significant on their own, but only
by virtue of their interplay with other entities (much in
the same way that “affordances” differ from technical
features). Consequently, by addressing this socially
constructed idea of appropriateness, designers are
invited to craft robot features that account for how these
features would be experienced and judged by humans.
In what follows, we illustrate the design space that
opens up by considering socio-relationality as a key
framing for HRI. We start with a brief discussion of
urban robot challenges, and then present three robot
concepts that illustrate how addressing robot citizenship
can translate into tangible design strategies.
4. Urban Robots in Question
By approaching the design of robots from the
perspective of robot citizenship, we developed three
concepts for urban robots. These address real world
challenges faced by designers of urban HRI, that
were identified through interviews with five robotics
researchers with expertise in autonomous navigation
for unmanned ground vehicles (UGV) (a type of robot
used in urban applications, e.g., delivery of goods). We
started the interviews with a short introduction about
the project and its objectives, stressing our interest
in identifying what are the most pressing challenges
in urban robotics. We then investigated further the
emerging challenges through a focused review of related
HRI literature. Then, for each of these challenges,
Fig. 1. Overview of the
main principles involved
when approaching robot
design from the socio-
relational perspective
suggested by the
concept of citizenship.
Design and Semantics of Form and Movement 91
we suggested an alternative approach to the problem
by identifying potential cooperative strategies and
envisioning apparent features that may facilitate them.
We exemplify the concepts with illustrations that
show a robot in a specific situation, communicating an
implicit message, and provoking a desired response from
humans.
4.1 Robot Challenges and Relational
Alternatives
Our interviews with the roboticists helped us identify
a small series of current urban robot challenges (see
figure 2). These challenges include problems related
to (1) the robot’s need for adapting to non-dedicated
infrastructures and related social norms, e.g. navigating
sidewalks and adapting to pedestrians’ speed and norms;
(2) inefficiencies resulting from the need for keeping safe
distances, as a way to appropriately navigate crowded
areas and deal with the unpredictable behaviors of
other entities; (3) issues emerging from the robot’s
difficulty of being understood, which may lead people
to misjudgment and adverse feelings; and (4) inability
of being respected, which makes the robot a potential
victim of human malicious actions, e.g. hacking and
bullying.
By reflecting on some of these issues and envisioning
possible alternative approaches, we developed three
concepts: the Transparent Robot, which responds to the
issue of being (mis)understood; the Handleable Robot,
which responds to the difficulty of dealing with non-
dedicated infrastructures; and the Shapeshifting Robot,
which addresses the challenge of keeping safe distances
from others.
The Transparent Robot. This concept (Fig. 3A)
illustrates situations in which a malfunctioning robot
may be perceived as something mysterious and
potentially dangerous, thus generating adverse feelings
and attitudes in human bystanders. In fact, “if a robot
is just standing somewhere looking as a generic
box, without doing anything, people may think it’s a
bomb” stressed one of the interviewees. Common
HRI strategies address this issue by preventing and
detecting malfunctions with regular interval checks
(e.g. [35]). However, this challenge may be reframed by
considering the social environment the robot is part of,
and relying on the human perception of the situation.
Recalling existing practices (e.g., calling for assistance
if an elevator breaks; calling the owner of a lost dog to
bring it home, etc.), we suggest malfunctioning robots
may be addressed not as a matter of manufacturer
responsibility, but rather as a case of a community
member in need of care.
From this perspective, designers may shift their focus
from increasing efficiency to evoking empathy. One
feature that may help facilitate this shift is the robot’s
Fig. 2. Overview of
urban robot challenges
identified through
the interviews with
roboticists.
Design and Semantics of Form and Movement92
appearance: changing its appearance from opaque
to transparent (with or without additional symbolic
elements), the robot can communicate its situation:
“Look what happened to me! I need your help!” In this
way, the challenge of human (mis)interpretation of
the situation may become an opportunity to generate
empathy and care.
The Handleable Robot. This concept (Fig. 3B) relates
to the challenges faced by a robot when attempting to
fit its behavior to non-dedicated urban infrastructure. In
such cases, the robot’s autonomy is strongly dependent
on its ability to combine various skills like detection,
prediction and planning. Take, for instance, the case of a
robot trying to cross a street or a busy intersection on
its own – a particularly complex challenge [36]. Within
current HRI strategies, this challenge is addressed by
improving the robot’s autonomy and self-sufficiency –
creating better environmental modeling and detection,
and designing more sophisticated predictive algorithms.
However, if we look at social encounters in similar
situations, we notice that the same problem is often
dealt with interdependencies generated by affinity and
shared membership in the community. Recalling these
existing social phenomena, then, we may consider
the possibility of a robot’s lack of autonomy as an
opportunity to instantiate interdependent relationships.
Much like children, elderly, or the disabled, robots may
cross a street safely by joining a shared performance
and relying on the abilities (and kindness!) of others.
When arriving at a crosswalk or needing to cross a busy
street, the robot may communicate its need for help by
producing a gesture that mimics the way humans reach
for help by extending their hand to others. This might
be achieved by rethinking the shape and purpose of the
flagpoles that sometimes protrude from the robot’s
back. The pole, in this mode, can be used not only to
signal the presence of the robot, but also to function as
a steering device that indicates that the robot is flexible
enough to be helped. Shifting the position of the pole,
the robot implicitly says to humans: “You can help me by
handling me”.
The Shapeshifting Robot. This concept (Fig. 3C)
responds to the challenge of navigating a crowded
environment, where the robot’s difficulty to predict the
behavior of a large number of moving agents (especially
people) represents a very complex problem. Currently,
designers try to solve the problem by developing
algorithms based on a “preventive approach” in which
the environment and other entities are detected,
their behaviors are predicted, and the movements
are planned for avoiding collision. Nonetheless, the
complexity of the challenge and the insufficiency
of current modelling efforts often lead to errors in
navigation, harm to humans, or robots stopping in
their tracks in order to prevent harm. As stated by one
of our interviewees, “most of navigation algorithms
are designed to be passive […] there is too much
focus on prevention”. This highlights how current
design strategies do not relate to the social nature of
the challenge they address. When we look at crowd
behavior, however, we notice that it is often regulated
by a series of tacit norms that go far beyond the desire
to avoid bumping into others. From body gestures that
enable a mutual understanding of intentions, to gentle
physical contact, humans, as well as animals, adapt to
each other. What we suggest, then, is to look at the
robot as a constitutive part of the crowd and, as such,
an entity that can enable such mutual understanding and
gentle physical contact.
Fig. 3. Rethinking
design challenges
through relational
strategies yielded three
concepts (from left to
right): the Transparent
Robot (A), the
Handleable Robot (B),
and the Shapeshifting
Robot (C).
Design and Semantics of Form and Movement 93
Providing a robot with a flexible soft shell, for instance,
may mimic how humans acknowledge and interact in a
crowd, sleekly squeezing and slipping through the crowd
respectfully instead of parting it aggressively. Through
these nonverbal behaviors, the robot communicates to
its human surrounding: “You can touch me as I go by, I’m
safe and friendly”.
5. Discussion and Conclusions
The examples discussed above hint at practical
implications of approaching robots not as tools but
rather as members of a co-performing community.
Subsequently, designers may be able to solve
some of the challenges inherent to complex urban
environments by designing robots that would be
perceived, recognized, and tolerated as valuable
members of the community. In this mode, the three
concepts we describe above replace self-sufciency with
interdependency; autonomy with mutuality; and a tool
perspective with a civic sensibility.
By considering the community instead of the individual
robot’s functional capabilities (or lack of), designers
can gain a more holistic view of HRI, understanding
a robot according to its embeddedness in the urban
environment, its social relations and practices. In
this perspective, what usually represents a limit and
challenge for a robot, may become an opportunity
for instantiating meaningful shared performances
with humans, in which the abilities of one may
become a strategy to deal with the limits of another
[33]. Furthermore, by reframing HRI challenges as
sociotechnical and not merely technical, the concept
of citizenship helps to unfold a design space that is
much less reliant on the legal system’s catching up to
the everyday presence of robots. Thus, while effort is
being put to regulate robots from a legal perspective,
designers may already move forward by thinking of
urban robots as social actors and therefore anticipating
regulatory and behavioral responses. The design of HRI,
it follows, can become anticipatory instead of reactive.
With that said, despite its potential, this new design
space is not free from complications. First, our proposal
assumes that citizen robots, or, more accurately,
robots that behave as citizens, will elicit certain
responses from humans. But, what if nobody wants to
take the extended hand of the Handleable robot and
help it cross the street? What if nobody cares if the
Transparent robot needs help? What if the softness of
the Shapeshifting robot is only perceived aesthetically?
What if people, despite those apparent features, still
perceive robots as an obstruction, a burden, or even as
competition?
This last question introduces a second complication. We
suggested here that robots may be considered members
contributing to the community’s common good, but
who gets to decide what that means in practice, and
how? Should HRI designers be responsible for deciding
what robot uses and roles are desirable? Should they
hold public referendums on each and every proposal
for an urban robot? And even then, what if conflicting
proposals emerge from within the community?
These critical questions highlight how despite its
practical implications, our approach does not provide
solutions, but only opens up a larger space for
discussion. This very ability to raise questions and
foster further reflections, however, is what we believe
represents the very meaningful nature of citizenship as a
concept that can be used in the design and investigation
of urban robots. In fact, “in certain circumstances asking
questions is as important as solving a problem” [37].
Design can play a crucial role in this. What we suggest,
then, is to look at citizenship as a design perspective that
can be used to challenge existing norms and attitudes,
provoke discussion and question established practices
[38] – a way to question the drive for technical
efficiency that characterizes current robot design.
Through the concept of citizenship, designers are
invited to question beliefs about the role of robots in
society, and to rethink their approach to urban robotics
from the logic of autonomy and efficiency towards
relationality and interdependency.
Acknowledgments. We thank the Amsterdam
Institute for Advanced Metropolitan Solutions (AMS)
and Delft Design for Values Institute for supporting the
project.
Design and Semantics of Form and Movement94
References
1. Stone Z.: Everything you need to Know About Sophia, The
World's First Robot Citizen. Forbes, Retrieved July 10, 2018,
from: https://bit.ly/2SgY4qX (2017).
2. Hern A.: Give Robots ‘Personhood’ Status, EU Committee
Argues, The Guardian, Jan. 12, 2017. Retrieved July 19, 2018,
from: https://bit.ly/2ioJolQ (2017).
3. Wong J C.: San Francisco sours on rampant deliver y robots:
‘Not every innovation is great’, The Guardian, Dec. 10, 2017.
Retrieved July 19, 2018, from: https://bit.ly/2jPqS6v (2017).
4. Lapastora C.: Arizona law gives delivery robots same rights
as pedestrians – but they must abide by same rules. Fox
News. Retrieved July 16, 2018, from: ht tps://fxn.ws/2J6GEc8
(2018).
5. Salvini P., Ciaravella G., Yu W. L., Ferri G., Manzi A.,
Mazzolai B., Laschi C ., Kim B. O. and Dario P.: How safe
are service robots in urban environments? Bullying a
robot. In Int. Symposium on Robot and Human Interactive
Communication (RO-MAN’10), 1-7 (2010).
6. Nagenborg M.: Urban robotics and responsible urban
innovation. Ethics and Information Technology, 1-11 (2018).
7. Arnold T. and Scheutz M.: Beyond Moral Dilemmas:
Exploring the Ethical Landscape in HRI. In ACM/IEEE
International Conference on Human-Robot Interaction (HRI
'17), 445-452 (2017).
8. Boden M., Bryson J., Caldwell D., Dautenhahn K ., Edwards
L., Kember S., Newman P., Parr y V., Pegman G., Rodden T.,
Sorrell T., Wallis M., Whitby B. and Winfield A.: Principles
of robotics: regulating robots in the real world, Connection
Science, 29:2, 124-129 (2017).
9. Franklin A .: The more-than-human city. The Sociological
Review, 65(2), 202-217 (2017).
10. Mancini C.: Animal-computer interaction: a manifesto.
Interactions, 18, 4, 69-73 (2011).
11. Aspling F., Juhlin O., and Chiodo E.: Smelling, pulling, and
looking: unpacking similarities and differences in dog
and human city life. In 12th International Conference on
Advances in Computer Enter tainment Technology (ACE
'15), Article 64 (2015).
12. Bastian M.: Towards a more-than-human par ticipatory
research. In Michelle Bastian, Owain Jones, Niamh Moore,
Emma Roe (eds.) Participatory Research in More-than-
Human Worlds, 33-51, Routledge (2016).
13. Smith N., Bardzell S., and Bardzell J.: Designing for
Cohabitation: Naturecultures, Hybrids, and Decentering
the Human in Design. In Conference on Human Factors in
Computing Systems (CHI '17), 1714-1725 (2017).
14. Pierce, J., Sengers, P., Hirsch, T., Jenkins, T., Gaver, W. and
DiSalvo, C.: Expanding and ref ining design and criticality in
HCI. In 33rd Annual ACM Conference on Human Factors in
Computing Systems, 2083-2092 (2015).
15. MacDorman K. F. and Cowley S. J.: Long-term
relationships as a benchmark for robot personhood. In
15th Int. Symposium on Robot and Human Interactive
Communication (ROMAN’06), IEEE (2006).
16. Sullins J. P.: When is a robot a moral agent?. International
Review of Information Ethics, 6 (2006).
17. Robertson J.: Human rights vs. Robot rights: Forecasts from
Japan, Critical Asian Studies, 46:4, 571-598 (2014).
18. Marx J. and Tiefensee C.: Of animals, robots and men.
Historical Social Research / Historische Sozialforschung, Vol.
40, No. 4 (154), 70-91 (2015).
19. Ashrafian H.: Ar tificial intelligence and robot responsibilities:
Innovating beyond rights. Science and engineering ethics,
21(2), 317-326 (2015).
20. Rainey S.: Friends, robots, citizens? SIGCAS Comput. Soc.
45, 3, 225-233 (2016).
21. Gunkel D. J.: Robot rights. MIT Press (2018).
22. Coeckelbergh M.: Robot rights? Towards a social-relational
justification of moral consideration. Ethics and Information
Technology 12.3, 209-221 (2010).
23. Floridi L. and Taddeo M.: Romans would have denied robots
legal personhood. Nature, May 16, 2018. Retrieved July 20,
2018, from: https://go.nature.com/2UKYp20 (2018).
24. Kymlicka W. and Donaldson S.: Animals and the Frontiers of
Citizenship. Oxford Journal of Legal Studies, 34(2), 201-219
(2014).
25. Forlano L.: Posthumanism and Design. She Ji: The Journal of
Design, Economics, and Innovation, 3(1), 16-29 (2017).
26. Daley J.: India’s Ganges and Yamuna rivers are given the
rights of people. Smithsonian.com. Retrieved March 7, 2019,
from: https://bit.ly/2Hpqfx0 (2017).
27. Daley J.: Toledo, Ohio, just granted lake Erie the same legal
rights as people. Smithsonian.com. Retrieved March 7, 2019,
from: https://bit.ly/2H44zXZ (2019).
28. Floridi L. and Sanders J. W.: On the morality of artificial
agents. Minds and machines, 14(3), 349-379, (2004).
29. Breazeal C., Gray J., Hoffman G. and Berlin M. Social robots:
Beyond tools to partners. In ROMAN 2004. 13th IEEE
International Workshop on Robot and Human Interactive
Communication, 551-556 (2004).
30. Johnson M., Bradshaw J. M., Feltovich P. J., Jonker C. M., Van
Riemsdijk M. B., and Sierhuis M.: Co-active design: Designing
suppor t for interdependence in joint activity. Journal of
Design and Semantics of Form and Movement 95
Human-Robot Interaction, 3(1), 43-69 (2014).
31. Marenko B. and Van Allen P.: Animistic design: how to
reimagine digital interaction between the human and the
nonhuman. Digital Creativity, 27(1), 52-70 (2016).
32. Rozendaal M.: Objects with intent: a new paradigm for
interaction design. interactions, 23(3), 62-65 (2016).
33. Kuijer L. and Giaccardi E.: Co-performance: Conceptualizing
the role of artificial agency in the design of everyday life.
In Conference on Human Factors in Computing Systems
(CHI’18), 125 (2018).
34. Van Allen P.: Rethinking IxD. Medium. Retrieved February
22, 2019, from: https://bit.ly/2IsSAoE (2016).
35. Crestani D., Godary-Dejean K . and Lapierre L.: Enhancing
fault tolerance of autonomous mobile robots. Robotics and
Autonomous Systems, 68, 140 -155 (2015).
36. Chand A. and Yuta S. I.: Road-crossing landmarks detection
by outdoor mobile robots. Journal of Robotics and
Mechatronics, 22(6), 708-717 (2010).
37. Ozkaramanli D. and Desmet P.: Provocative design for
unprovocative designers: Strategies for triggering personal
dilemmas. In 50th Conference of Design Research Society
(2016).
38. Raptis D., Jensen R . H., Kjeldskov J. and Skov M. B.:
Aesthetic, Functional and Conceptual Provocation in
Research Through Design. In Conference on Designing
Interactive Systems (DIS’17), 29-41 (2017).
