Content uploaded by Kobi Leins
Author content
All content in this area was uploaded by Kobi Leins on Oct 15, 2020
Content may be subject to copyright.
To Chat, or Bot to Chat, Just the First Question:
Potential legal and ethical issues arising from a chatbot
case study
Kobi Leins, Marc Cheong, Simon Coghlan, Simon D’Alfonso, Piers Gooding, Reeva Lederman,
Jeannie Paterson1
Introduction
The use of audio and textual chatbots, one type of digital intermediary, is pervasive and changing
the way that we engage with computers and each other. Chatbots raise a wide range of legal and
ethical issues, including questions about suitability for the user group, data collection, security of
data storage and privacy, later linking of data, repurposing, as well as broader questions of
accountability, applicable regulatory frameworks and responsibility to report, just to name a few.
Chatbots, as benign as they first seem, raise a multitude of ethical and legal issues, questions and
potential liabilities that need to be thoughtfully addressed. Adequate chatbots can be very easy to
build – the more sophisticated the chatbot, the more technologically challenging,- but both
simple and complex chatbots require a certain level of technical and social complexity to
maintain and dispose of.
This paper will focus on one very specific example of an application of a chatbot to illustrate the
breadth and depth of issues that require contemplation prior to use: the use of chatbots for young
people at risk of homelessness as they transition through and out of high school. There is
particular interest in developing chatbots for this group as smartphones are prevalent, even where
functional human support networks may be lacking. Often individuals will suffer poverty and
mental health issues, and may even encounter many other government departments several times
over the years before they become homeless. In Australia, one tech platform developer is
contemplating how to assist teens at risk of homelessness, trying to find ways to mitigate this
happening. In fact, this research was undertaken to meet the needs of this developer who is trying
to protect its constituents and to meet the highest standard of care.2 Given that this cohort is not
only young, but often in vulnerable economic, social and psychological circumstances, safely
developing tools for this demographic will require particular care. Analysis of these issues for a
group with multiple vulnerabilities will better inform questions about the use of chatbots for
other communities.
1 Thanks to Susan Sheldrick and Rhys Ryan for their assistance with this article.
2 ‘New tech to help homeless youth connect to services’, Infoxchange (Web Page, 5 August 2019)
<https://www.infoxchange.org/au/news/2019/08/new-tech-help-homeless-youth-
connect-services>.
1
In this paper, after defining chatbots, we contemplate some of the questions that need to be raised
prior to their use and suggest a framework of questions and consultation to ensure that the use of
chatbots results in both desirable and intended outcomes, whilst minimising risks, unintended
outcomes, and potential harm. To do so, we first provide the historical context of chatbots. We
then consider potential benefits and risks of chatbots, especially for the user group in question –
teens at risk of homelessness.3 Given the frequency of use of chatbots in the mental health
context, existing research on the use of chatbots for mental health provides some guidance on
how to evaluate a chatbot for teens at risk of homelessness, but has limitations in that it does not
necessarily contemplate the user group in a particular context.4 Further, much research in this
area, without necessarily contemplating social implications, has already been undertaken and
continues apace in human computer interaction research (HCI).5
We recommend that certain questions should be asked before, during and after the use of
chatbots to provide services to individuals, especially those in groups who have traditionally
been marginalised or are otherwise vulnerable. Our paper concludes that the evaluation of
chatbots should be based not just on whether they are taken up and used, but on whether they
meet users’ needs.6 We shall stress the elementary but vital point that a generic evaluation of
digital intermediaries will not suffice. Rather, each use of intermediary technology must be
evaluated on its ability to meet the needs of the particular individuals and/or group(s) it is
designed to assist. Risks, unintended consequences and potential harm, we argue, must be
considered prior to use.
What is a ‘chatbot’?
The very first actual natural language processing computer program, or what became known as a
chatterbot (or chatbot for short), originated at MIT and was created by Joseph Weizenbaum, a
German-American computer scientist frequently referred to as one of the fathers of modern
artificial intelligence. Named ELIZA, its primary purpose was to simulate a Rogerian
psychotherapist by demonstrating ‘the responses of a non-directional psychotherapist in an initial
psychiatric interview’.7
3 John A Powell and Stephen Menendian, ‘The Problem of Othering: Towards Inclusiveness and
Belonging’, Othering and Belonging (Web Page, 29 June 2017)
<http://www.otheringandbelonging.org/the-problem-of-othering/>.
4 R R Morris et al, ‘Towards an Artificially Empathic Conversational Agent for Mental Health
Applications: System Design and User Perceptions’ (2018) 20(6) Journal Of Medical Internet Research
e10148; A Ho, J Hancock and A S Miner, ‘Psychological, Relational, and Emotional Effects of Self-
Disclosure After Conversations With a Chatbot’ (2018) 68(4) Journal of Communication 712.
5 J Lazar, J Feng and H Hochheiser, Research Methods in Human-Computer Interaction (Morgan
Kaufmann 2nd ed, 2017); M Kuniavsky, ‘User Experience and HCI’ in A Sears & J A Jacko (eds), The
Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging
Applications (Lawrence Erlbaum, 2nd ed, 2008) 897.
6 B A Shawar and E Atwell, ‘Different measurements metrics to evaluate a chatbot system’ in
Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog
Technologies (Association for Computational Linguistics, 2007) 89.
2
One limitation of ELIZA, Weizenbaum explained, was that ‘communication between man and
machine was superficial’.8 What followed was entirely unexpected – Weizenbaum’s secretary,
although fully aware of the purpose and limitations of the machine, became somewhat obsessed
with communicating with it.9 The event illustrated the possibility of a form of magical thinking
elicited by chatbots.
Subsequently, Weizenbaum wrote a series of books and articles warning about humans’
engagement with even very limited technology and that this interaction is not always entirely
rational. The term ‘ELIZA effect’ describes the tendency to anthropomorphise computers,
ascribing to them human traits and intentions which the user may know that they don’t actually
have. Despite Weizenbaum’s pessimistic intention of creating ELIZA to demonstrate the
superficiality or limitations of communication between human and machine, chatbots have
continued to be developed and have advanced to provide human-machine interaction across a
wide range of industries.
The term ‘chatbot’ has now come to encompass a very broad range of capabilities, including
textual and voice recognition. Even the use of a function like Google Duplex, which uses digital
‘assistants’ to help make a restaurant booking, may potentially fall under the definition of a
chatbot.10 The operation of chatbots without users’ knowledge raises multiple issues around
consent and human dignity that cannot be addressed in this article.
The use of chatbots as glorified search engines, such as those commonly appearing on retail and
banking websites, is widespread. Some chatbots are designed simply to enable human users to
find services more efficiently or to tailor responses to a history of use to make searches more
accurate. In the field of mental health and wellness, chatbots are more often a conduit to a real
person with whom you can professionally consult.11
Some chatbots have more ambitious aims. Microsoft’s Xiaoice (pronounced ‘Shao-ice’, meaning
‘Little Bing’), for example, is a social agent ‘with a personality modeled on that of a teenage girl,
and a dauntingly precocious skill set’ and over 660 million users.12 Unlike Amazon Alexa or
7 J Weizenbaum, Computer power and human reason: From judgment to calculation (W H Freeman &
Co, 1976) 188; Caroline Bassett, ‘The computational therapeutic: exploring Weizenbaum’s ELIZA as a
history of the present’ (2019) 34(4) AI & Society 803.
8 J Epstein and W D Klinkenberg, ‘From Eliza to Internet: A brief history of computerized assessment’
(2001) 17(3) Computers in Human Behavior 295 citing T Nadelson, ‘The inhuman computer/the too-
human psychotherapist’ (1987) 41(4) American Journal of Psychotherapy 489.
9 Weizenbaum (n 7) 2, 3, 6, 182, 189.
10 Magikmaker, ‘Google Assistant calling a restaurant for a reservation’ (YouTube, 9 May 2018)
<https://www.youtube.com/watch?v=-RHG5DFAjp8>.
11 ‘6 ways Head to Health can help you’, Australian Government Department of Health (Web Page)
<https://headtohealth.gov.au/>; Woebot Health (Web Page) <https://woebot.io>; Wysa—Your 4 am
friend and AI life coach (Web Page) <https://www.wysa.io/>; Joyable (Web Page)
<https://joyable.com/>; Talkspace (Web Page) <https://www.talkspace.com>.
12 Luke Dormehl, ‘Microsoft Xiaoice: AI That Wants To Be Your Friend’, Digital Trends (Web Page, 18
November 2018) <https://www.digitaltrends.com/cool-tech/xiaoice-microsoft-future-of-ai-
3
Google Home, which are a kind of voice-based chatbot, the interaction with Xiaoice typically
takes place by text message, and not with the spoken word. ‘Social chatbots’ like Xiaoice have a
special aim:
The primary goal of a social chatbot is not necessarily to solve all the questions the users might
have, but rather, to be a virtual companion to users. By establishing an emotional connection with
users, social chatbots can better understand them and therefore help them over a long period of
time.13
On a more cynical view, in addition to providing a service, chatbots also collect immense
amounts of data about an individual that is useful for commercial purposes. In a traditional
human sense, of course, chatbots do not ‘understand’ anything at all. Instead, chatbots are
programs that simulate human conversation; they collect data, find patterns (save for those
powered by machine learning NLP, respond with a pre-defined decision logic, and project future
patterns based on those of the past. By such means, chatbots may sometimes be said to
‘understand’, predict, and respond to users’ wishes and needs, and, in this way, to engender a
sense of emotional connection from users. The nature of interactions with certain individuals
gives rise to potential risks and benefits of chatbots.
In the 1950s, Alan Turing suggested a procedure, commonly known as the ‘Turing Test’, for
determining whether a computer could convince a human user that they were in fact interacting
with another human being.14 Now, some chatbots are designed specifically to advise the users
that they are not humans to avoid confusion, while others simulate a human without notifying
users that they are simply machines.15 Chatbots that are designed to deceive user-subjects into
believing they are speaking with a human raise a host of ethical and legal issues, particularly
regarding privacy, as well as ethical questions about human dignity, which we will not address
here.
For the purposes of this article, we are not considering broader conversational chatbots or
chatbots with assistant capabilities. We are considering an advanced search engine, in the format
of a bot, to help young people at risk of homelessness access services quickly and discreetly
where other avenues may be difficult to access. This proposed chatbot would be presented
clearly as a bot, without practising the deceit that they are a person. What we have illustrated is
the importance of terminology, and the wide suite of technologies that may be covered by the
term ‘chatbot’. Each technology may carry different social, legal, philosophical and practical
implications that require consideration prior to use.
assistants/>.
13 H Shum, X He and D Li, ‘From Eliza to XiaoIce: Challenges and opportunities with social chatbots’
(2018) 19(1) Frontiers of Information Technology & Electronic Engineering 10.
14 A M Turing, ‘Computing Machinery and Intelligence’ (1950) LIX(236) Mind 433; G Oppy and D
Dowe, ‘The Turing Test’ in E N Zalta (ed), The Stanford Encyclopedia of Philosophy (Stanford
University, 2019) <https://plato.stanford.edu/archives/spr2019/entriesuring-test/>.
15 Id 13.
4
Types of Chatbots
We shall categorise contemporary chatbots across two dimensions. First, there is complexity.
Simple chatbots are often rule-based16 and follow pre-programmed decision trees or simple ‘If-
Then-Else’ rules. More complex ‘conversational agents’ have greater flexibility: they can discern
user intentions and meanings and may rely on AI techniques such as NLP and machine learning,
including deep learning neural networks.17
Second, there is a chatbot’s interface and interaction style.18 Simpler chatbots present on-screen
textual dialogue (e.g. input via keyboard or mouse). More advanced chatbots are speech- or
voice-based; popular examples include digital voice assistants,19 like Amazon Alexa or Google
Home.
Potential benefits and risks of chatbots
The known and possible risks of chatbots are relevant to determining their ethical justification
and proper use. In professional health and social work roles, a fiduciary duty, with both ethical
and legal dimensions, binds practitioners to acting in the interests of their patients or clients.20
This includes duties to assess benefits and risks of interventions and to carefully weigh them in
order to promote client wellbeing (within the limits of respect for personal autonomy). For
example, while certain interventions may promise benefits, the risks and harms they pose may be
too great to justify those benefits. Such professional responsibilities can serve as a model for
ethically evaluating the justification and proper use of chatbots designed to advance the interests
of vulnerable individuals, such as some adolescents. Consequently, we should examine the
potential benefits and harms of using chatbots in these ways.
Chatbots do have potential advantages over people. Unlike humans, chatbots can run day and
night and do not require rest. Neither do they require salaries, or holiday pay or sick leave. If
chatbots malfunction, they can be upgraded or switched off. Some parties, such as Microsoft,
would go a step further, suggesting that ‘Social chatbots’ appeal lies not only in their ability to
respond to users’ diverse requests, but also in being able to establish an emotional connection
16 Sarabeth Lewis, ‘Ultimate Guide to Chatbots 2020 – Examples, Best Practices & More’, AppSumo
Blog (Blog Post, 29 July 2019) <https://blog.appsumo.com/ultimate-guide-to-chatbots-2020/>.
17 Michal Stojanov, ‘Prospects for Chatbots’ (2019) 8(3) Izvestia Journal of the Union of Scientists –
Varna, Economic Sciences Series 10; Eric Michael Smith et al, ‘Can You Put It All Together: Evaluating
Conversational Agents’ Ability to Blend Skills’, arXiv (Web Page, 17 April 2020)
<https://arxiv.org/abs/2004.08449>.
18 Jenny Preece, Yvonne Rogers and Helen Sharp, Interaction Design: Beyond Human-Computer
Interaction (John Wiley & Sons, 2015).
19 This is a working definition used in a UNESCO report investigating the gendering of voice assistant
technology. See I'd blush if I could: closing gender divides in digital skills through education, UNESCO,
UN Doc GEN/2019/EQUALS/1 REV 3 (2019) <https://en.unesco.org/EQUALS/voice-assistants>.
20 T L Beauchamp and J F Childress, Principles of biomedical ethics (Oxford University Press, 2001); H
Kutchins, ‘The fiduciary relationship: The legal basis for social workers’ responsibilities to clients’ (1991)
36(2) Social Work 106.
5
with users’.21 In fact, the establishment of an emotional connection by the user may be especially
crucial for assisting individuals from vulnerable groups. In healthcare, a perceived relationship
can be vital for successful therapy, which is typically dependent on trust and mutual
understanding.22 Chatbots that are adept at facilitating perceived emotional connections, and
thereby promoting trust and understanding, may, to a certain extent, be beneficial for teens facing
homelessness (and for other vulnerable groups).
Moreover, research shows that in situations which are embarrassing or carry a stigma, there may
be a greater acceptance of chatbots in the broader community.23 Users might be as likely, if not
more likely, to disclose emotional and factual information to a chatbot as they are to a human
partner, indicating that people may psychologically engage with chatbots much as they do with
people (even, at times, when they know that these tools are automated and no human is involved,
a phenomenon that recalls the ‘ELIZA effect’). It is important to note that exactly how the
ELIZA effect plays out with youths at risk of homelessness requires further research. Where
there is an indication that chatbots may provide benefits to a particular group, it may be justified
to pursue their development. But before that is done, it is essential to consider the risks arising
from that use, including potential misuse, dual use and potential harm.
Chatbots undoubtably carry potential harms and risks for adolescents. As with many areas of use
of technology, the reality does not necessarily always match the hype. While much of the
research about chatbots concerns their success, arguably more can be learned from studies that
demonstrate the failure of such technologies. The main barriers found to use by adolescents in
one study were stigma and embarrassment and difficulty in recognising their own symptoms (i.e.
poor mental health literacy),24 as well as a preference for self-reliance.25 On the other hand, one
largescale survey warned that the current mental health app landscape, of which chatbots form a
significant part, tends to over-medicalise states of distress and may over-emphasise ‘individual
responsibility for mental well-being’.26 Claims that users suffer from ‘poor mental health
literacy’, which poses a barrier to higher use of chatbots, therefore, should be regarded with
caution. Such claims could even be considered a failure to frame the needs of user-subjects in
ways that are relevant to them. It is important that any tool designed to assist a user group
actually meets the needs of its users. This is particularly important when the chatbot is designed
to provide assistance to groups of vulnerable users. In the case of teens at risk of homelessness,
21 Shum, He and Li (n 13).
22 D Roter, ‘The enduring and evolving nature of the patient–physician relationship’ (2000) 39(1)
Patient Education and Counseling 5.
23 T Nadarzynski et al, ‘Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A
mixed-methods study’ (2019) 5 Digital Health 1.
24 R Crutzen et al, ‘What can we learn from a failed trial: Insight into non-participation in a chat-based
intervention trial for adolescents with psychosocial problems’ (2014) 7(1) BMC Research Notes 824.
25 Ibid; A Gulliver, K M Griffiths and H Christensen, ‘Perceived barriers and facilitators to mental health
help-seeking in young people: A systematic review’ (2010) 10(1) BMC Psychiatry 113.
26 L Parker et al, ‘Mental Health Messages in Prominent Mental Health Apps’ (2018) 16(4) Annals of
Family Medicine 338.
6
such considerations might, for example, include meeting their needs for self-reliance and
reducing the stigma of seeking help. For this user-centred approach to be successful, it is
necessary to build in users trust that the developers of the technology will be responsive to their
feedback and that the technology itself is able to meet their needs safely and effectively.27
Research amongst college students in the United States indicates that there are specific barriers
to using chatbots for mental health, including privacy concerns, financial constraints, scepticism
about treatment effectiveness, and, in particular, a lack of willingness to engage if the user is
from a lower socioeconomic background.28 This may prove relevant in contemplating the use of
chatbots for teens at risk of homelessness in Australia, or it may be particular to a geographic and
cultural context. Research on contemplating the needs of users prior to design recommends an
agenda that promotes inclusion and engagement rather than top-down decision-making.29
Should chatbots be used at all?
Whether or not a chatbot should be developed and used depends on an overall evaluation of its
harms and benefits. If the benefits outweigh the harms, its use may be justified in some
circumstances, depending on what those potential harms are. But sometimes the risks and harms
will be too high to justify using particular chatbots for particular individuals. Again, adequate
judgments here turn on careful ethical and legal evaluations of specific cases. We must ask: What
tool most aptly fits the task? Is a non-technological solution better justified in these
circumstances? Or, if a technology appears promising, what should it look like? Solutions range
from a paperclip cartoon on your Microsoft screen (for those who remember Clippy),30 which
provides guidance without any memory of previous questions, through to an autopoietic system
that responds and grows with the user. The first step should always be to define what the problem
is that one is trying to solve to ensure that the tool fits the task. Furthermore, each time a system
is repurposed, the same process of evaluation should be repeated. If a judgment is made that a
chatbot is justified, we must next evaluate whether its benefits can and should be increased and
its harms and risks minimised. Thus, in addition to the fundamental initial questions of
justifiability raised above, more specific questions, both technical and social, should be
considered.31 We discuss these further questions below, and offer some recommendations.
27 F N Egger, “Trust me, I’m an online vendor”: towards a model of trust for e-commerce system design’
in Extended Abstracts of the Conference of Human Factors in Computing Systems (Association for
Computing Machinery, 2000) 101.
28 J Hunt and D Eisenberg, ‘Mental Health Problems and Help-Seeking Behavior Among College
Students’ (2010) 46(1) Journal of Adolescent Health 3.
29 J Waycott, ‘Ethics in Evaluating a Sociotechnical Intervention With Socially Isolated Older Adults’
(2015) 25(11) Qualitative Health Research 1518.
30 T Schamp-Bjerede, ‘What Clippy and Tux can teach us: Incorporating affective aspects into
pedagogy’ (2012) 1 Högskolepedagogisk debatt 47.
31 J Waycott et al, ‘Co-constructing Meaning and Negotiating Participation: Ethical Tensions when
‘Giving Voice’ through Digital Storytelling’ (2017) 29(2) Interacting with Computers 237; Kobi Leins,
‘AI for Better or for Worse, or AI at all?’, Future Leaders (Web Page, 2019)
<http://www.futureleaders.com.au/book_chapters/pdf/Artificial-Intelligence/Kobi-Leins.pdf>.
7
Failed Chatbots
Concrete examples of chatbot failures can be found in a number of different fields. Indeed,
chatbots can fail for a whole range of reasons. Some of these are presented in the following table.
Chatbot Reasons for failure
Lawbot: Created by Cambridge University students to
help victims of sexual assault navigate the legal
system.32
-Emotionally insensitive
-Strict checklist to determine what a crime is
-Can discourage users form seeking help
-Directs users to local police station, but not to any
support services33
Newsbots: A number of news agencies jumped on the
bot bandwagon in 2016 with the aim to create bots that
personalised content and opened up new audiences.34
-Significant resources required for maintenance
-Did not sync with existing formats, delivery or
distribution of news content
-Lacked sophistication to personalise content
effectively
-Minimal input from journalists during development35
Poncho: Weather chatbot using Facebook Messenger
with a sassy cartoon cat as the front. 36
-Sending users unrelated information
-Not understanding words it should, e.g. ‘weekend’37
Tay: Microsoft chatbot trained via crowdsourced input
on Twitter.38
-Shut down after 24 hours after it was trained to
produce racist, sexist and anti-Semitic tweets
-Public able to influence the outputs as minimal human
supervision provided39
Data – What is used to train the chatbot? What data should the chatbot collect? What data
should the chatbot store? and How should the chatbot store this data?
Often chatbots are trained on existing data. What data is used will shape the responses of the
chatbot, and where data sets are not comprehensive, blind spots, bias and unintended bias may
occur. Further, questions must be asked about what data is collected from those using the
32 A Packham, ‘Cambridge students build a “lawbot” to advise sexual assault victims’, The Guardian
(online, 9 November 2016) <https://www.theguardian.com/education/2016/nov/09/cambridge-
students-build-a-lawbot-to-advise-sexual-assault-victims>.
33 Mariya Yao, ‘5 Ways Humanitarian Bots Can Save The World’, Topbots (Web Page, 3 November
2016) <https://www.topbots.com/social-good-humanitarian-bots-can-save-world/>.
34 V Belair-Gagnon, S C Lewis and C Agur, ‘Failure to Launch: Competing Institutional Logics,
Intrapreneurship, and the Case of Chatbots’ (2020) 25(4) Journal of Computer-Mediated Communication
291.
35 Ibid.
36 ‘Worst Chatbot Fails’, Business News Daily (Web Page, 13 February 2020)
<https://www.businessnewsdaily.com/10450-funniest-chatbot-fails.html>; Emily
Cummins, ‘Conversational AI vs. Chatbots’, Netomi (Web Page, 31 October 2019)
<https://www.netomi.com/the-worst-chatbot-fails-and-how-to-avoid-them>.
37 Ibid.
38 M J Wolf, K Miller and F S Grodzinsky, ‘Why we should have seen that coming’ (2017) 47(3) ACM
SIGCAS Computers and Society.
39 Ibid.
8
chatbots, how that data is stored, where it is used, and how it is linked to other data. Raw chat
data, metadata, and even client use behaviour can be tracked and linked with other online
behavioural data. It is imperative that the use of the data is made clear to the user prior to use,
and that the terms and conditions are adhered to, not only to avoid legal liability, but also to
ensure trust by the user. We must also ask what data will be collected, and how it will be used
each and every time, as well as how long and where the data is stored, and what other data it may
be connected to, or with whom else it may be shared. These are questions that need to be
contemplated prior to the rolling out or use of any chatbot. There are not only ethical questions
concerning fundamental rights to privacy; there are also legal questions, as privacy and data
protection laws in many jurisdictions place strict limits on what data can be collected,
particularly sensitive personal data, how it is stored and the uses that can be made of it.40
Recommendations: In general, data should not be retained. If data is retained, the purpose must
be made clear.
Data collected should not be shared with third parties or kept long-term, subject to legal
requirements, such as the legal requirement of mandatory reporting of mistreatment of children
in Australia. This recommendation may pose technical challenges, as commercially available
infrastructures collect data as a matter of course. Not-for-profit providers or small businesses
may struggle with the costs of building their own infrastructure, instead having to rely on
commercial offerings.
Platforms Used to Support the Chatbot
Further, the platform used to provide the service, such as Dialogflow or Facebook, may use this
data regardless of the consent the service requires, undermining privacy and sharing intimate and
very personal information.41
Recommendations: Platforms used should not collect or use data collected during use of the
chatbot. If platforms do this, alternative platforms should be used. If alternative platforms are not
available, users should be made aware of, and consent to, the data-use of any platform that hosts
the chatbot.
Repurposing
Any time a chatbot developed for one purpose is repurposed, the parameters and risks need to be
reconsidered.
Recommendations: Consider all of the points raised in this article each time a chatbot is
repurposed.
40 See, eg, Privacy Act 1988 (Cth) and, in the EU, the Regulation (EU) No 2016/679 of the European
Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the
processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC
[2016] OJ L 119/1 (‘General Data Privacy Regulation’).
41 K Kretzschmar, ‘Can Your Phone Be Your Therapist? Young People’s Ethical Perspectives on the Use
of Fully Automated Conversational Agents (Chatbots) in Mental Health Support’ (2019) 11 Biomedical
Informatics Insights 1.
9
Supervision, Human Intervention & Resourcing
To operate 24/7, chatbots also require supervision. As real chatbots learn and develop, they can
also fail and have glitches. This means that having humans involved in their supervision is
required to ensure that they operate as programmed – and this may not always be achieved. The
absence of trained human supervisors may be problematic for another reason. In the case of
mental health chatbots, automated bots are still far from recreating the richness of a face-to-face
encounter with a mental health professional, despite their efforts to mirror real-life interactions.
Even though a minimal level of personalisation exists (e.g. different tips/strategies are given for
users presenting symptoms of depression versus anxiety), the support provided is still generic
and perhaps more akin to a self-help book. That is, as of yet, chatbots cannot grasp the nuances
of users’ life history and current circumstances that may be at the root of mental health
difficulties. As Woebot warns its users: ‘As smart as I may seem, I’m not capable of really
understanding what you need’.42
The explosion of digital technology in health and social services is premised on the reasonable
idea that some forms of automation and digital communication could assist with care. One
common aim of new technology, such as artificial intelligence, is to break down tasks into
individual components that can be repetitively undertaken. Yet care is not just tasks; it also has a
rich emotional dimension. ‘Care’ is a fundamental human part of human relationships and is a
highly complex social interaction.43 One of the criticisms of chatbots is that they are not capable
of empathy, of recognising users’ emotional states and tailoring responses to reflect these
emotions. The lack of empathy may therefore compromise engagement with health chatbots. On
the other hand, the field of affective computing has seen the emergence of work on
computing/robotic systems that can reportedly recognise, interpret, process and simulate human
affects and emotions.44
On a purely practical level, chatbot moderation may also mean increased pressure on service
providers to increase multitasking and workloads in collecting, inputting, organising and
constantly updating digital materials, which, paradoxically, may reduce time for teamwork and
face-to-face engagement. Another practical issue relates to power outages; if power goes out or
there is a national disaster, chatbots may no longer function. Furthermore, chatbots, like any
computer program can be hacked, and security systems to ensure privacy, safety and security of
data must be paramount.
Recommendations: Contemplate the resource intensity of chatbots and whether a chatbot is the
best method to meet users’ needs given the resources it requires. Be clear with users that chatbots
are being used and clear about their limitations, and have human consultation readily available as
a back-up.
42 Ibid.
43 E F Kittay, ‘The Ethics of Care, Dependence, and Disability*’ (2011) 24(1) Ratio Juris 49.
44 R W Picard, Affective Computing (MIT Press, 2000); R Calvo et al, The Oxford Handbook of Affective
Computing (Oxford University Press, 2015).
10
Reliability, Risk and the Law
At a minimum, automated systems require compliance with existing law, such as consumer
protection law, privacy and data protection and human rights. In addition to these requirements,
other legal issues may arise.
Unexpected disclosures to chatbots raise distinct issues. What happens if someone discloses that
there has been abuse of a child, which, in Australia, for example, requires mandatory reporting?
Or what if domestic violence is disclosed?45 Users can develop what they perceive as
relationships with chatbots, and disclose information that, if disclosed to a human, would bring
about certain legal obligations to report. It is not clear whether this same obligation applies to an
unsupervised chatbot, but the ethical obligations may suggest that these disclosures must be
taken seriously, and a failure to do so may raise legal risks. One option could be to develop a
system such that any user input detected to be problematic by a computational linguistic analysis
and beyond the bot’s purview, is passed on to a human content moderator.
Recommendations: Ensure all legal obligations are complied with under the jurisdiction in which
the chatbot is being used, including for unintended uses. If the chatbot is being used in more than
one jurisdiction, be aware that laws will vary.
Cybersecurity
The more data a chatbot collects, the more likely it is to be the target of hacks. Documents that
are attached to communications, the program or the database itself may be the target for attacks
or leaks. Cyber criminals may want this data either to steal and use for their own benefit, or to
block access from it and to demand payment (ransomware). Both pose threats to broader
business models using programs that are known to collect data and are therefore targets for
hackers.
Recommendation: Ensure the highest level of cybersecurity for any data transfer.
Usability vs human interaction
More general ‘mental health’ apps, purporting to support anxiety, depression and other mental
health disorders, have been used with varying levels of success. Regarding app-based mental
health intervention, Simon Leigh and Steve Flatt characterise the wide range of mental health
apps as suffering from a ‘frequent lack of an underlying evidence base, a lack of scientific
credibility and subsequent limited clinical effectiveness’.46 There are clear risks associated with
overhyping technology without a commensurate evidence base.47 These risks appear at the
individual and population-level (ranging from shaping individual users’ preferences and
expectations about service provision, to altering how national research funding is directed).
While the lack of evidential support does not imply that chatbots have no value, it does mean that
45 G White, ‘Child advice chatbots fail sex abuse test’, BBC News (11 December 2018)
<https://www.bbc.com/news/technology-46507900>.
46 S Leigh and S Flatt, ‘App-based psychological interventions: Friend or foe?’ (2015) 18(4) Evidence-
Based Mental Health 97.
47 E Anthes, ‘Pocket psychiatry: Mobile mental-health apps have exploded onto the market, but few have
been thoroughly tested’ (2016) 532 Nature 20.
11
we should exercise caution in developing chatbots for vulnerable teens and other groups, and that
research into their benefits and risks should be undertaken.48
Recommendation: Any risks that are identified must inform decision-making about legal
regulatory frameworks; for example, in determining evidentiary standards before products are
introduced. Other issues that affect implementation and regulation include insufficient
integration and failure of technologies to be adaptable to real-world complexity.
Active Involvement of Users
From a pragmatic perspective, the involvement of the affected groups (including representative
organisations) is generally agreed to increase the likelihood of ‘viable and effective – rather than
disruptive and short-lived – advances’ in technologies such as chatbots in any social or health
service provision.49
According to Rishi Duggal and colleagues, a robust regulatory framework in the mental health
context, for example, will only emerge when service users, patients, persons with disabilities,
clinicians, and providers collaborate to design a ‘forward thinking, future proof, and credible
regulatory framework that can be trusted by all parties’.50 Without it, there is a greater likelihood
of costly technologies being introduced in an unthinking manner, created to address one issue
without sufficient thought to harmful flow-on consequences. Poor user consultation also
increases the likelihood of wasted resources.
Deliberative, participatory development is also important because the convergence of services
and technology tends to emerge from a concentration of power in a myriad of ways: government
agencies, venture capital and Big Technology companies, universities with largescale
infrastructure for tech development, professional associations, and so on. To ensure greater
equity in design, development and regulation, some writers call for ‘interdisciplinary empirical
research on the implications of these technologies that centers the experiences and knowledge of
those who will be most affected’.51 Their recommendation to centre the experiences and
knowledge of the primarily affected group is not shared across the literature. Further, actually
doing participatory, community-engaged, co-designed development well is not at all
straightforward, whether it be in the development of chatbots, or in relation to any other support
practice or service provision.
End of Life
Questions regarding the retirement of a chatbot need to be considered. How will it be removed?
What, if any, of the data will be stored long-term? What will it be replaced with and how will
48 N Martinez-Martin and K Kreitmair, ‘Ethical Issues for Direct-to-Consumer Digital Psychotherapy
Apps: Addressing Accountability, Data Protection, and Consent’ (2018) 5(2) JMIR Mental Health e32.
49 D Bhugra et al, ‘The WPA-Lancet Psychiatry Commission on the Future of Psychiatry’ (2017) 4(10)
The Lancet Psychiatry 775.
50 R Duggal, I Brindle and J Bagenal, ‘Digital healthcare: Regulating the revolution’ (2018) 360 BMJ.
51 A Guta, J Voronka and M Gagnon, ‘Resisting the Digital Medicine Panopticon: Toward a Bioethics of
the Oppressed’ (2018) 18(9) The American Journal of Bioethics 62.
12
users be notified? Very little research appears to address the retirement of systems, which also
requires consideration from a whole range of angles.
Recommendation: Have a strategy and budget for maintaining, updating and retiring any planned
chatbot.
Conclusion
Before contemplating the use of a chatbot, the specific problem or need should be clearly
specified. As a second step, the specific requirements of the target audience should be
contemplated, from a user-based perspective. An evaluation should be made about whether the
balance of benefit over risk justifies a given chatbot’s use not for users in general, but for the
particular group of users it is designed to assist. A wide range of legal requirements – consumer
protection, privacy protection, and security of data storage – may arise. In addition, a range of
specific ethical considerations should be considered, with a view to reducing harms and
increasing benefits for users. These ethical considerations include: what obligations arise when
personal information is disclosed, how data should be secured, how or if data should be reused in
another context, what the cybersecurity settings are, and what back-ups are available in the case
of an emergency or lack of power or another event that would render the chatbot inoperable.
Although in this paper we have focussed on social chatbots for teens at risk of homelessness, and
more peripherally, health chatbots, the legal and ethical considerations discussed have broad
application to other groups and individuals who may potentially use chatbots.
This article has demonstrated that, for a very narrow application of a chatbot for teens at risk of
homelessness, a whole palette of questions must be asked prior to use, during use, and after
retirement of a system. A framework for asking and answering these questions, as well as
ensuring compliance with existing legal obligations and minimising risk to users should be
undertaken, regardless of what field a chatbot is being used in. This is not currently the approach
to launching chatbots, which poses unconsidered risks, breaches to privacy and further
vulnerabilities to users as data becomes increasingly linked.
13
References
‘6 ways Head to Health can help you’, Australian Government Department of Health (Web Page)
<https://headtohealth.gov.au/>
Anthes, E, ‘Pocket psychiatry: Mobile mental-health apps have exploded onto the market, but few have
been thoroughly tested’ (2016) 532 Nature 20
Bassett, Caroline, ‘The computational therapeutic: exploring Weizenbaum’s ELIZA as a history of the
present’ (2019) 34(4) AI & Society 803
Beauchamp, T L and J F Childress, Principles of biomedical ethics (Oxford University Press, 2001)
Belair-Gagnon, V, S C Lewis and C Agur, ‘Failure to Launch: Competing Institutional Logics,
Intrapreneurship, and the Case of Chatbots’ (2020) 25(4) Journal of Computer-Mediated
Communication 291
Bhugra, D et al, ‘The WPA-Lancet Psychiatry Commission on the Future of Psychiatry’ (2017) 4(10) The
Lancet Psychiatry 775
Calvo, R et al, The Oxford Handbook of Affective Computing (Oxford University Press, 2015)
Crutzen, R et al, ‘What can we learn from a failed trial: Insight into non-participation in a chat-based
intervention trial for adolescents with psychosocial problems’ (2014) 7(1) BMC Research Notes
824
Cummins, Emily, ‘Conversational AI vs. Chatbots’, Netomi (Web Page, 31 October 2019)
<https://www.netomi.com/the-worst-chatbot-fails-and-how-to-avoid-them>
Dormehl, Luke, ‘Microsoft Xiaoice: AI That Wants To Be Your Friend’, Digital Trends (Web Page, 18
November 2018) <https://www.digitaltrends.com/cool-tech/xiaoice-microsoft-future-of-ai-
assistants/>
Duggal, R, I Brindle and J Bagenal, ‘Digital healthcare: Regulating the revolution’ (2018) 360 BMJ
Egger, F N, “Trust me, I’m an online vendor”: towards a model of trust for e-commerce system design’ in
Extended Abstracts of the Conference of Human Factors in Computing Systems (Association for
Computing Machinery, 2000) 101
Epstein, J and W D Klinkenberg, ‘From Eliza to Internet: A brief history of computerized assessment’
(2001) 17(3) Computers in Human Behavior 295 citing T Nadelson, ‘The inhuman computer/the
too-human psychotherapist’ (1987) 41(4) American Journal of Psychotherapy 489
Gulliver, A, K M Griffiths and H Christensen, ‘Perceived barriers and facilitators to mental health help-
seeking in young people: A systematic review’ (2010) 10(1) BMC Psychiatry 113
Guta, A, J Voronka and M Gagnon, ‘Resisting the Digital Medicine Panopticon: Toward a Bioethics of the
Oppressed’ (2018) 18(9) The American Journal of Bioethics 62
Ho, A, J Hancock and A S Miner, ‘Psychological, Relational, and Emotional Effects of Self-Disclosure
After Conversations With a Chatbot’ (2018) 68(4) Journal of Communication 712
14
Hunt, J and D Eisenberg, ‘Mental Health Problems and Help-Seeking Behavior Among College Students’
(2010) 46(1) Journal of Adolescent Health 3
I'd blush if I could: closing gender divides in digital skills through education, UNESCO, UN Doc
GEN/2019/EQUALS/1 REV 3 (2019) <https://en.unesco.org/EQUALS/voice-assistants>
Joyable (Web Page) <https://joyable.com/>
Kittay, E F, ‘The Ethics of Care, Dependence, and Disability*’ (2011) 24(1) Ratio Juris 49
Kretzschmar, K , ‘Can Your Phone Be Your Therapist? Young People’s Ethical Perspectives on the Use of
Fully Automated Conversational Agents (Chatbots) in Mental Health Support’ (2019) 11
Biomedical Informatics Insights 1
Kuniavsky, M, ‘User Experience and HCI’ in A Sears & J A Jacko (eds), The Human–Computer
Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications
(Lawrence Erlbaum, 2nd ed, 2008) 897
Kutchins, H, ‘The fiduciary relationship: The legal basis for social workers’ responsibilities to clients’
(1991) 36(2) Social Work 106
Lazar, J, J Feng and H Hochheiser, Research Methods in Human-Computer Interaction (Morgan
Kaufmann 2nd ed, 2017)
Leigh, S and S Flatt, ‘App-based psychological interventions: Friend or foe?’ (2015) 18(4) Evidence-
Based Mental Health 97
Leins, Kobi, ‘AI for Better or for Worse, or AI at all?’, Future Leaders (Web Page, 2019)
<http://www.futureleaders.com.au/book_chapters/pdf/Artificial-Intelligence/Kobi-Leins.pdf>
Lewis, Sarabeth, ‘Ultimate Guide to Chatbots 2020 – Examples, Best Practices & More’, AppSumo Blog
(Blog Post, 29 July 2019) <https://blog.appsumo.com/ultimate-guide-to-chatbots-2020/>
Magikmaker, ‘Google Assistant calling a restaurant for a reservation’ (YouTube, 9 May 2018)
<https://www.youtube.com/watch?v=-RHG5DFAjp8>
Martinez-Martin, N and K Kreitmair, ‘Ethical Issues for Direct-to-Consumer Digital Psychotherapy Apps:
Addressing Accountability, Data Protection, and Consent’ (2018) 5(2) JMIR Mental Health e32
Morris, RR et al, ‘Towards an Artificially Empathic Conversational Agent for Mental Health
Applications: System Design and User Perceptions’ (2018) 20(6) Journal Of Medical Internet
Research e10148
Nadarzynski, T et al, ‘Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A
mixed-methods study’ (2019) 5 Digital Health 1
‘New tech to help homeless youth connect to services’, Infoxchange (Web Page, 5 August 2019)
<https://www.infoxchange.org/au/news/2019/08/new-tech-help-homeless-youth-connect-services>
Oppy, G and D Dowe, ‘The Turing Test’ in E N Zalta (ed), The Stanford Encyclopedia of Philosophy
(Stanford University, 2019) <https://plato.stanford.edu/archives/spr2019/entriesuring-test/>
15
Packham, A, ‘Cambridge students build a “lawbot” to advise sexual assault victims’, The Guardian
(online, 9 November 2016) <https://www.theguardian.com/education/2016/nov/09/cambridge-
students-build-a-lawbot-to-advise-sexual-assault-victims>
Parker, L et al, ‘Mental Health Messages in Prominent Mental Health Apps’ (2018) 16(4) Annals of
Family Medicine 338
Picard, R W, Affective Computing (MIT Press, 2000)
Powell, John A and Stephen Menendian, ‘The Problem of Othering: Towards Inclusiveness and
Belonging’, Othering and Belonging (Web Page, 29 June 2017)
<http://www.otheringandbelonging.org/the-problem-of-othering/>
Preece, Jenny, Yvonne Rogers and Helen Sharp, Interaction Design: Beyond Human-Computer
Interaction (John Wiley & Sons, 2015)
Privacy Act 1988 (Cth)
Regulation (EU) No 2016/679 of the European Parliament and of the Council of 27 April 2016 on the
protection of natural persons with regard to the processing of personal data and on the free
movement of such data, and repealing Directive 95/46/EC [2016] OJ L 119/1 (‘General Data
Protection Regulation’)
Roter, D, ‘The enduring and evolving nature of the patient–physician relationship’ (2000) 39(1) Patient
Education and Counseling 5
Schamp-Bjerede, T, ‘What Clippy and Tux can teach us: Incorporating affective aspects into pedagogy’
(2012) 1 Högskolepedagogisk debatt 47
Shawar, B A and E Atwell, ‘Different measurements metrics to evaluate a chatbot system’ in Proceedings
of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies
(Association for Computational Linguistics, 2007) 89
Shum, H, X He and D Li, ‘From Eliza to XiaoIce: Challenges and opportunities with social chatbots’
(2018) 19(1) Frontiers of Information Technology & Electronic Engineering 10
Smith, Eric Michael et al, ‘Can You Put It All Together: Evaluating Conversational Agents’ Ability to
Blend Skills’, arXiv (Web Page, 17 April 2020) <https://arxiv.org/abs/2004.08449>
Stojanov, Michal, ‘Prospects for Chatbots’ (2019) 8(3) Izvestia Journal of the Union of Scientists – Varna,
Economic Sciences Series 10
Talkspace (Web Page) <https://www.talkspace.com>
Turing, A M, ‘Computing Machinery and Intelligence’ (1950) LIX(236) Mind 433
Waycott, J et al, ‘Co-constructing Meaning and Negotiating Participation: Ethical Tensions when ‘Giving
Voice’ through Digital Storytelling’ (2017) 29(2) Interacting with Computers 237
Waycott, J, ‘Ethics in Evaluating a Sociotechnical Intervention With Socially Isolated Older Adults’
(2015) 25(11) Qualitative Health Research 1518
Weizenbaum, J, Computer power and human reason: From judgment to calculation (W H Freeman & Co,
1976)
16
White, G, ‘Child advice chatbots fail sex abuse test’, BBC News (11 December 2018)
<https://www.bbc.com/news/technology-46507900>
Woebot Health (Web Page) <https://woebot.io>
Wolf, M J, K Miller and F S Grodzinsky, ‘Why we should have seen that coming’ (2017) 47(3) ACM
SIGCAS Computers and Society
‘Worst Chatbot Fails’, Business News Daily (Web Page, 13 February 2020)
<https://www.businessnewsdaily.com/10450-funniest-chatbot-fails.html>
Wysa—Your 4 am friend and AI life coach (Web Page) <https://www.wysa.io/>
Yao, Mariya, ‘5 Ways Humanitarian Bots Can Save The World’, Topbots (Web Page, 3 November 2016)
<https://www.topbots.com/social-good-humanitarian-bots-can-save-world/>
17