ArticlePDF Available

Abstract and Figures

Background: One reason for the introduction of digital technologies into health care has been to try to improve safety and patient outcomes by providing real-time access to patient data and enhancing communication among health care professionals. However, the adoption of such technologies into clinical pathways has been less examined, and the impacts on users and the broader health system are poorly understood. We sought to address this by studying the impacts of introducing a digitally enabled care pathway for patients with acute kidney injury (AKI) at a tertiary referral hospital in the United Kingdom. A dedicated clinical response team-comprising existing nephrology and patient-at-risk and resuscitation teams-received AKI alerts in real time via Streams, a mobile app. Here, we present a qualitative evaluation of the experiences of users and other health care professionals whose work was affected by the implementation of the care pathway. Objective: The aim of this study was to qualitatively evaluate the impact of mobile results viewing and automated alerting as part of a digitally enabled care pathway on the working practices of users and their interprofessional relationships. Methods: A total of 19 semistructured interviews were conducted with members of the AKI response team and clinicians with whom they interacted across the hospital. Interviews were analyzed using inductive and deductive thematic analysis. Results: The digitally enabled care pathway improved access to patient information and expedited early specialist care. Opportunities were identified for more constructive planning of end-of-life care due to the earlier detection and alerting of deterioration. However, the shift toward early detection also highlighted resource constraints and some clinical uncertainty about the value of intervening at this stage. The real-time availability of information altered communication flows within and between clinical teams and across professional groups. Conclusions: Digital technologies allow early detection of adverse events and of patients at risk of deterioration, with the potential to improve outcomes. They may also increase the efficiency of health care professionals' working practices. However, when planning and implementing digital information innovations in health care, the following factors should also be considered: the provision of clinical training to effectively manage early detection, resources to cope with additional workload, support to manage perceived information overload, and the optimization of algorithms to minimize unnecessary alerts.
Content may be subject to copyright.
Original Paper
A Qualitative Evaluation of User Experiences of a Digitally Enabled
Care Pathway in Secondary Care
Alistair Connell1,2, MBBS; Georgia Black3, PhD; Hugh Montgomery1, MD; Peter Martin3, PhD; Claire Nightingale3,4,
PhD; Dominic King2, PhD; Alan Karthikesalingam2, PhD; Cían Hughes2, MSc; Trevor Back2, PhD; Kareem Ayoub2,
MA; Mustafa Suleyman2; Gareth Jones5, PhD; Jennifer Cross5, PhD; Sarah Stanley5, MSc; Mary Emerson5, BSc;
Charles Merrick5; Geraint Rees6, PhD; Christopher Laing5, MD; Rosalind Raine3, PhD
1Centre for Human Health and Performance, University College London, London, United Kingdom
2DeepMind Health, London, United Kingdom
3Department of Applied Health Research, University College London, London, United Kingdom
4Population Health Research Institute, St. George's, University of London, London, United Kingdom
5Royal Free London NHS Foundation Trust, London, United Kingdom
6University College London, London, United Kingdom
Corresponding Author:
Rosalind Raine, PhD
Department of Applied Health Research
University College London
1-19 Torrington Place
London, WC1E 7HB
United Kingdom
Phone: 44 (0)20 7679 1713
Email: r.raine@ucl.ac.uk
Abstract
Background: One reason for the introduction of digital technologies into health care has been to try to improve safety and
patient outcomes by providing real-time access to patient data and enhancing communication among health care professionals.
However, the adoption of such technologies into clinical pathways has been less examined, and the impacts on users and the
broader health system are poorly understood. We sought to address this by studying the impacts of introducing a digitally enabled
care pathway for patients with acute kidney injury (AKI) at a tertiary referral hospital in the United Kingdom. A dedicated clinical
response team—comprising existing nephrology and patient-at-risk and resuscitation teams—received AKI alerts in real time
via Streams, a mobile app. Here, we present a qualitative evaluation of the experiences of users and other health care professionals
whose work was affected by the implementation of the care pathway.
Objective: The aim of this study was to qualitatively evaluate the experience of mobile results viewing and automated alerting
as part of a digitally enabled care pathway on the working practices of users and their interprofessional relationships.
Methods: A total of 19 semistructured interviews were conducted with members of the AKI response team and clinicians with
whom they interacted across the hospital. Interviews were analyzed using inductive and deductive thematic analysis.
Results: The digitally enabled care pathway improved access to patient information and expedited early specialist care.
Opportunities were identified for more constructive planning of end-of-life care due to the earlier detection and alerting of
deterioration. However, the shift toward early detection also highlighted resource constraints and some clinical uncertainty about
the value of intervening at this stage. The real-time availability of information altered communication flows within and between
clinical teams and across professional groups.
Conclusions: Digital technologies allow early detection of adverse events and of patients at risk of deterioration, with the
potential to improve outcomes. They may also increase the efficiency of health care professionals’ working practices. However,
when planning and implementing digital information innovations in health care, the following factors should also be considered:
the provision of clinical training to effectively manage early detection, resources to cope with additional workload, support to
manage perceived information overload, and the optimization of algorithms to minimize unnecessary alerts.
(J Med Internet Res 2019;21(7):e13143) doi:10.2196/13143
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.1http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
KEYWORDS
nephrology; acute kidney injury
Introduction
Background
In many health systems, the ageing demographic of hospital
patients is accompanied by worsening health and a greater need
for diverse investigations and treatments. As a result, care
pathways are increasingly complex and ever more reliant on
access to relevant data and on communication between
individuals and multidisciplinary teams [1]. In this context,
although adverse events—such as acute kidney injury (AKI),
cardiac arrest, or clinical decline which necessitates
high-dependency care—are commonly a consequence of the
natural history of the underlying disease, they may also occur
because of delays in treatment pathways [2]. Evidence suggests
that patient outcomes improve where such clinical decline is
detected and acted upon early, and substantial effort has been
made worldwide in this regard, for instance, through the use of
track and trigger scoring systems to detect decline [3] or the
provision of emergency response teams [4,5]. However, such
changes have not been matched by other key components of
care delivery: the manner in which health care teams
communicate and the manner in which data are accessed and
presented. Globally, the most widely used hospital
communication system continues to be the pager [6], whereas
data are accessed from paper records and a range of disparate
and disconnected electronic data repositories. Care might be
improved if interpersonal communication were enhanced and
if it were possible to readily access data in a form that allowed
rapid assessment of the patient’s status.
The introduction of digital technologies offers one potential
solution, allowing ready detection of the deteriorating patient,
communication among health care workers, and immediate
access to patient data in a user-friendly and appropriate format,
thus improving outcomes. The embedding of digital technologies
into health care is now a priority in the United Kingdom [7] and
internationally [8]. However, the adoption of such technologies
into health care has been little studied, and the impacts on users
and the broader health system are poorly understood. We sought
to address these issues by studying the impacts of introducing
a digitally enabled care pathway for patients with AKI.
AKI—a sudden reduction in kidney function diagnosed by
changes in serum creatinine [9]—is common, appears across
multiple care pathways, and is associated with significantly
increased mortality, morbidity, and cost of health care [10-15].
However, substantial deficits exist in all key processes of AKI
care including early recognition and therapy, appropriate
escalation to specialist or critical care services, and follow-up
[16]. In an effort to expedite and standardize diagnosis, the
National Health Service (NHS) mandated the use of a new
diagnostic algorithm in all English hospitals in 2014 [17] and
provided guidance as to how the algorithm could be
implemented [18]. However, simple alerting to the presence of
AKI does not seem to improve outcome [19]. We therefore
designed a new care pathway that encompassed AKI detection,
mobile alerting of a dedicated response team comprising
multiple specialists, and the provision of protocolized care [20].
Evaluation of the implementation of the digitally enabled care
pathway with regard to impacts on processes of care, clinical
outcome, and the cost of care delivery are described elsewhere
[21,22].
Objectives
We present a qualitative evaluation of the experiences of users
and other health care professionals whose work was affected
by the implementation of the new care pathway. We sought to
characterize the impacts on staff of such automated alerting,
mobile results viewing, and ready communication, with
particular focus on their working practices and interprofessional
relationships.
Methods
Setting
The digitally enabled pathway was designed and implemented
at the Royal Free Hospital (RFH)—a large, acute, tertiary
referral hospital providing a range of acute services (including
a 34-bed intensive treatment unit and an inpatient nephrology
service) in central London, United Kingdom. The care pathway
has been described in detail elsewhere [20] and is summarized
in brief below.
The Preimplementation Care Pathway
Before making the changes to the care pathway described here,
pathology results were viewed by ordering clinicians in batch
at the end of the working day using desktop computers. A
message linking to local Web-based clinical guidelines was
appended to any creatinine result suggestive of AKI in the
electronic patient record (EPR), and such results were also
communicated to the clinical area by biochemistry staff by
telephone. In its early stages, AKI was typically managed
independently by general acute care and various specialty teams;
specialist input was requested through hospital pagers and
telephone communication at the discretion of referring clinical
teams.
The Digitally Enabled Care Pathway
Streams (DeepMind Technologies Ltd) is a mobile app deployed
on iPhone operating system–enabled smartphones. It processes
relevant routinely collected clinical and demographic data
through secure integration with hospitals’ existing information
systems; owing to the need for real-time event-driven data,
Health Level Seven (version 2, Health Level Seven
International) feeds were used for integration with the laboratory
information management system and electronic medical record.
It was first registered with the Medicines and Healthcare
Products Regulatory Agency as a Class I, nonmeasuring,
nonsterile medical device under the European Union Medical
Device Directive (1993) on August 30, 2016. Future revisions
of the device may be classified at a higher level under the new
medical device regulation.
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.2http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
Streams analyzes serum creatinine results immediately and
continuously, alerting clinicians in real time to all potential AKI
cases as defined by the NHS England AKI algorithm. The app
also provides clinicians with data relevant to AKI management,
including a graphical trend view of serum creatinine, specific
flags for the presence of life-threatening AKI complications
(such as hyperkalemia), details of any previous AKI episodes,
demographic information, and past medical history from coded
Hospital Episode Statistics data. Videos demonstrating Streams
functionality can be found on the DeepMind Health support
website [23].
AKI alerts are sent in real time to a specialist clinical response
team (henceforth, the AKI response team), comprising the RFH’s
existing patient-at-risk and resuscitation team (PARRT) and
nephrology team. The PARRT (Clinical Nurse Specialists who
review at-risk or deteriorating inpatients) receive alerts on all
patients with AKI stages 2 and 3 and are on site 24 hours a day.
The nephrology team comprises a renal consultant and specialty
registrar, both of whom receive all AKI notifications. The
registrar is on site 24 hours a day and is typically the first
responder. The consultant can triage alerts through secure remote
access if off site, providing clinical supervision and subsequent
patient review where needed. Through Streams, the AKI
response team triages alerts, communicates with other team
members, and documents the outcome of clinical reviews.
Relevant contacts and clinical guidance are available on Streams
phones.
The AKI response team prioritizes patients for review according
to the information available in Streams. Patients with
life-threatening complications or deemed at high risk are
reviewed immediately, whereas a case review within 2 hours
is suggested for all other alerts. Upon review, a care protocol
(based on existing best practice guidelines [24,25]) is annotated
and entered into the patient’s paper notes alongside an advisory
sticker for key nursing actions (Multimedia Appendix 1).
Although the nephrology team occasionally takes over patient
care, overall responsibility for care rests with patients’ primary
teams. AKI recovery is monitored remotely in-app. Realerting
for AKI that has not recovered is enabled 48 hours after the first
alert. However, worsening of AKI stage or the development of
a new complication (eg, hyperkalemia) at any time results in a
further notification. Follow-up reviews are undertaken by the
AKI response team according to clinical judgement. A diagram
outlining the pre- and postintervention care pathways is provided
in Figure 1. Although clinical guidelines and specialist response
teams existed before the new care pathway, the goals of
implementation of the care pathway were to improve the
reliability and speed at which AKI recognition and appropriate
specialist review occurred.
Implementation
Before deployment, Streams users attended training events and
accessed a video users’ guide to both the Streams app and the
clinical pathway. Feedback from the AKI response team was
gathered during a 16-week pathway optimization period
(January-May 2017), during which key adjustments to the
Streams app and associated clinical pathway were made. The
optimized care pathway has been deployed continuously at RFH
since May 2017.
Figure 1. Pre- and postintervention care pathways.
Data Collection
We undertook an exploratory case study approach to understand
the implementation of Streams within a single site. Data
collected included in-depth semistructured interviews and brief
nonparticipant observations. The interview guide (included in
the Multimedia Appendix 1) was developed by the research
team, which included physicians with extensive experience in
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.3http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
clinical nephrology and intensive care medicine and experts in
health services research. Interviews explored the impacts of the
care pathway on staff members and the care delivered to
patients, with a particular focus on working practices and
interprofessional relationships. Face-to-face interviews began
1 month after the start of the pathway optimization period and
were spaced throughout a 16-month period of implementation
and evaluation (February 2017 to May 2018). Purposive
sampling was employed following a key informant strategy [26]
that identified individuals with important roles in the
environment under study who had expert knowledge to share
impartially. A sample size of 20 is typical for a case study such
as this, in line with both international consensus guidance and
common practice in qualitative research [27-29]. Furthermore,
the total number of users involved in providing the care pathway
was small, which necessarily restricted the number of interviews
that could be conducted. A list of potential respondents was
drawn up to ensure representation from both groups in the AKI
response team (ie, PARRT and nephrology teams) and clinicians
from the wider hospital community affected by the care
pathway, as well as a diverse range of clinical experience and
level of comfort with mobile technologies. A total of 20
respondents were approached and 19 consented. In total, 8
PARRT nurses (5 band 7 and 3 band 8 or above) and 8
nephrologists (4 registrars and 4 consultants) were interviewed
from the AKI response team. Of the respondents, 3 (2 consultant
physicians and 1 medical registrar) from the wider hospital
community were selected as a result of their frequent interactions
with the AKI response team. Interviews were conducted by the
lead researcher (AC). Each respondent was interviewed once.
Respondents were informed that the interviewer was from a
university and that the research was independent of the RFH.
Field observations were undertaken during the first 4 weeks of
the pathway optimization period. For these, the lead researcher
observed user behavior in the emergency department and in
inpatient wards during day and evening shifts using extensive
note-taking to document users’ interaction with the Streams app
and impacts on working practices and interprofessional
relationships.
Data Analysis
Data were analyzed using a combination of inductive and
deductive thematic analysis techniques [30]. First, quotes from
each interview were arranged into a matrix (included in
supplementary materials) in which rows represented individual
respondents and columns represented categories that aligned to
the basic principles of the intervention pathway (for example,
the triage of AKI alerts). The matrix was populated by 2
researchers (AC and GB) who independently analyzed the entire
dataset. Researchers met regularly to critique and challenge
each other’s allocations; these were in turn reviewed with the
lead researcher (RR), a process that enabled us to compare
different professional groups’ perspectives and identify
discordant views. The group then synthesized new descriptive
codes based on emergent themes in the matrix (for example,
the impact of real-time information availability) and assigned
the quotes to these. Discordant quotes that challenged these
themes were routinely sought and discussed. Additional
independent oversight was provided by the lead researcher who
identified additional quotes of relevance and refined the final
themes. We employed the principle of keyness in our analysis
[31], drawing out novel issues that might be generalizable and
relevant to the adoption of digital health products in clinical
practice (for example, how mobile working tools impact
established clinical workflows). Our results present
representative quotes for each theme, the titles of which were
iterated through the writing process. We therefore took both a
descriptive and an interpretive approach to analysis, first
understanding how the intervention was used and how it affected
clinical practice, then considering the intervention in terms of
cultural practices and overarching meta-themes about mobile
app use.
Ethical Approval
The digitally enabled care pathway constituted a new standard
service at RFH. Plans for the evaluation of the digitally enabled
care pathway were independently reviewed by the University
College London Joint Research Office. They directed that this
study fell under the remit of service evaluation (rather than
research), as per guidance from the NHS Health Research
Authority [32]. As such, the service evaluation was registered
locally with the RFH audit lead and medical director, and no
participant consent was required. An independent data
monitoring committee (which included a patient member)
reviewed all analyses before preparation for publication. A full
list of committee members is provided in the Multimedia
Appendix 1.
Results
Although interviews focused on the deployment of a pathway
for patients with AKI, they also sought to more broadly
characterize the impacts of automated alerting, mobile results
viewing, and ready communication on staff, with a particular
focus on their working practices and interprofessional
relationships. In this respect, 3 central themes emerged.
Theme 1: The Impact of Real-Time Information
Availability
This theme relates to respondents’ experiences of the benefits
and drawbacks of real-time mobile access to patient data. The
app provided automated alerts about patients’ kidney function
and gave staff access to current and historical information, such
as previous pathology and coded diagnoses. These functions
were reported to save valuable time among participants from
both teams:
Being able to look up the blood results for anyone in
the hospital wherever you are is unparalleled. [...] it
feels almost archaic these days, to go and see a
patient and then go and sit down in front of a
computer 15 minutes later. As a doctor, you have to
integrate what you know about them at the time of
seeing them. So if you could literally have this phone,
look at the results, go and see them... Or even look
at it while you are seeing them. [...] It must save at
least - I don’t know if you could analyze it - but it
must save at least a couple of hours in a day.
[Respondent 3: Nephrology team]
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.4http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
This in turn expedited rapid intervention for deteriorating
patients, wherever they were in the hospital:
The speed at which it happened was impressive. [...]
I happened to be in A&E and got the alert of someone
with severe kidney injury. [...] The patient was
admitted to [...] a specialist renal ward [...] within 2
or 3 hours, which I don’t think would have happened
without the app. [...] I think it streamlines care and
speeds up the time in which they get a specialist renal
review. [Respondent 9: Nephrology team]
I personally have noticed [...] patients who have
flagged up on the app that the clinical management
has been poor up to that point. When we get involved,
or the renal team get involved, that management
changes [...] It has definitely saved people’s lives.
[Respondent 14: PARRT]
Being able to access all the bloods for the patients in
the hospital and to be able to be alerted to the sick
ones and already know about them before we usually
do... Sometimes you know about them before the crash
bleep comes through. You turn up and you think,
“That was actually the alert I was coming to see.
[Respondent 10: PARRT]
Participants in both teams found alerts to be particularly valuable
for patients whose lead consultant was not a physician:
The most value came from patients under [...] surgical
patients, for whom the list of priorities for their
clinicians are very different from what [physicians]
look for when they are looking after a patient. For
those [patients], getting a rapid alert about deranged
renal function is very valuable. [Respondent 6:
Nephrology team]
The provision of results and real-time clinical alerts and team
communication via mobile phones introduced workload for
clinicians in a new modality. Overall, experienced clinicians
were able to integrate AKI alerts into their existing duties,
discriminating between high- and low-priority cases and using
this information to adjust their current priorities:
I would intermittently [...] check it, like I would [...]
check emails, [...] check it every hour or so, something
like that. And within 5 minutes or so I could easily
flick through the alerts and [...] identify which ones
I needed to see. [...] I felt it was very easy to use, I
think some people when they were trying to use it
would try and respond immediately to every alert. I
wouldn’t personally, I didn’t think that was the best
way to do it. Intermittently checking it throughout the
day, I managed to keep on top of things. [Respondent
9: Nephrology team]
However, some more junior clinicians in both teams felt that
the pathway created additional workload and suggested that
clinical review might not be deliverable to all patients by the
AKI response team as configured currently:
It does increase our work. Some days [...] we can
have eight or nine referrals. But there is obviously a
huge issue about workload for many people. But if
we need to increase the size of our team because of
this then that’s a good thing. And also it highlights
[...] the acuity of our patients in our hospital. These
patients are not straightforward. [Respondent 19:
PARRT]
Others pointed out that the added burden of the app was related
to the volume of false positive alerts produced by the mandated
NHS AKI algorithm:
...if the noise of the system could be reduced it would
be a lot better. If [we were] able to get rid of all the
nonsense alerts, that would be fantastic. [Respondent
1: Nephrology team]
Some respondents from the nephrology team pointed out that
although important information was now more readily available,
this created additional anxiety because it was not clear who was
primarily responsible for delivering timely care. Once again,
the need to expand the responsive workforce was suggested as
a way to mitigate the workload-associated stress:
...as Renal [Registrars], [...] you are always now, in
the back of your head, thinking “I’ve got this other
job to do.” And I think it does create... not anxiety
that keeps you up at night... But it’s another anxiety
when you already have enough anxiety! So I think
even if it was available in the hands of more people,
or we were a bit clearer that during times of people
being unwell who are your own patients, you
shouldn’t prioritise Streams people because they are
under another team, then that’s fine. That’s one way
of dealing with it. [Respondent 3: Nephrology team]
In summary, although the digitally enabled care pathway was
widely valued for increasing efficient access to patient
information, thus facilitating better care, some respondents
reported increased workload and anxiety.
Theme 2: The Implications of Early Detection
The digitally enabled care pathway was designed to expedite
identification of AKI so as to avoid deterioration and improve
outcomes, meaning that staff who normally respond to critically
unwell patients would now also attend patients at lower risk.
There was a divergence of opinion among members of the AKI
response team about the rationale for this approach. Respondents
in both teams pointed out that the AKI algorithm identified
deteriorating patients at an earlier stage than was possible
through other means:
It’s a good thing from the point of view that I know
there are patients that are potentially sick out there.
[...] You could have an AKI and look relatively well
initially. But [...] nobody would have known about
those patients. [Respondent 8: PARRT]
Others emphasized the benefits of early recognition for patient
health and in terms of reducing the complexity of required
interventions:
I think it does a good job. We pick up patients that
would maybe sit for another day or so before we pick
them up. It’s certainly beneficial. It is more work, but
we might be saving ourselves work in a couple of
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.5http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
days, we might have to do more stuff to catch up.
[Respondent 15: PARRT]
However, some respondents in the nephrology team did not see
the point of being alerted to low-risk patients:
...you get a lot of AKI stage 1s. They build up. Looking
through those and dismissing them each time is time
consuming. The AKI stage 2 and 3 [alerts] are more
helpful for me to look at, so I tend to just look at those
and dismiss the stage 1s. [Respondent 18: Nephrology
team]
Some respondents in this team also pointed out that early
identification could not necessarily be aligned to early
intervention because of a lack of knowledge with respect to
appropriate management:
The patients you definitely need to see are the patients
that have acute renal failure with a creatinine of 300
or 400 that and it’s going up - patients you’d normally
want to see [...] The other patients [...] that have a
rising creatinine, but the creatinine is not very high
- it doesn’t mean that they don’t need to be seen
necessarily. We are not trained as doctors to look
after those sorts of patients. [Respondent 1:
Nephrology team]
Thus, the shift toward earlier detection highlights the need to
consider the resources required to manage both early and late
disease and the training needed to enable clinicians to effectively
intervene at an earlier stage.
Theme 3: Behavioral Effects of the Care Pathway
The final theme demonstrates how real-time data provision
affected relationships between users of the digital intervention
and the broader health system with which the users interacted,
together with how these changes impacted upon beliefs,
behaviors, and care delivery. These are described in 3
subcategories below.
First, the digital care pathway affected behaviors within clinical
teams in a number of ways. An immediate benefit was the use
of mobile phones for team communication, experienced by
members of both teams:
I’ve found the [mobile phone] really useful because
I’ve been able to message my team when I’m out
seeing a patient, rather than finding a phone and to
bleep them with and waiting for them to answer.
[Respondent 5: PARRT]
However, a disadvantage was also identified: junior members
of the nephrology team do not currently use the Streams app.
Unequal access to patient data occasionally reversed the usual
direction of communication of information from junior doctor
to consultant, which some consultants suggested impeded
learning opportunities:
I think it’s important for [Junior Doctors] to
understand what the decisions about their patients
are. They have to be across the data. And that’s why
I prefer getting information from them [...] We were
in the position where I was telling the Juniors what
the blood results were. It makes me uncomfortable
and it makes them uncomfortable. [Respondent 6:
Nephrology team]
In addition, the visualization of each other’s triage decisions
within the app (a feature specifically requested by users)
revealed the hitherto unrecognized variations that exist in
professional judgements. Respondents in both teams raised the
fact that knowledge of others’ decisions sometimes confused
rather than clarified the clinical decision-making processes of
colleagues:
I was quite surprised about how other people triaged
initially. I felt we’d be much more similar in our
thinking, because when we talk about other things we
do think similarly about other stuff. [...] I felt like -
probably naively - that everyone would do what I did.
And they didn’t at all. [Respondent 11: PARRT]
Second, the digital pathway had an impact on relationships
between clinical teams. Several members of the nephrology
team were uncomfortable about providing a clinical opinion
when not solicited by the clinicians primarily responsible for
that patient’s care:
So you might [...] call the team and say “we suggest
you give some fluid,” but I don’t think it’s ethical to
prescribe it yourself. After all, they might say, “Listen,
he has heart failure.” So you can’t intrude.
[Respondent 2: Nephrology team]
However, there was an indication among several members of
the PARRT team that this concern might be limited to
communication between different specialty doctors:
I think the doctors have found it more difficult because
in medicine there is this real model of, “[...] I don’t
see this patient unless I’m asked to see them”. There’s
this formality. [...] Nurses don’t think like that, people
are used to us showing up. So it’s been easier for us
to think of every patient that we see as our patient,
our problem, our sick patients. I think that’s been
easier for this team to absorb and deal with.
[Respondent 11: PARRT]
This is relevant because the type of information available also
changed the professional group with whom this team directly
liaised:
[I will] always [speak to] the nursing staff, because
you are on their ward. It’s only polite and also you
will generally be recommending frequencies of obs;
they need to know what’s going on. And someone
from the medical team. I would say the app is making
me speak to more senior doctors more. [...] I’d be
more likely to seek out a consultant and say “by the
way, this person has alerted” and show them the app.
[Respondent 5: PARRT]
Third, the care pathway had an impact on the relationship
between clinicians and their patients. In particular, several
PARRT team members described that alerts identified patients
at an earlier stage of AKI than was the case through established
clinical pathways (eg, monitoring of vital signs). This may have
led to an unexpected and evolving role for some members of
the team. Several respondents described how the care pathway
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.6http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
enabled them to help patients make informed decisions
surrounding end-of-life care. For example:
Why do we have to talk about end of life just as I’m
about to die? [...] We could plan. Every single person
we’ve been referred today has a terminal disease.
[...] Trying to move the decision making back, in a
more timely way. [...] We are getting an alert before
they have even triggered [via vital signs], so we can
probably have a sensible conversation with a patient
with capacity. [Respondent 4: PARRT]
Discussion
Principal Findings
Qualitative and quantitative results from our mixed-methods
service evaluation suggest that the digitally enabled care
pathway has positive impacts on patient care [21,22]. Here, we
demonstrated the ability to intervene in the treatment of
deteriorating patients more quickly and the opportunity for
earlier, more constructive end-of-life planning. The ability to
integrate mobile results viewing into existing clinical workflows
also appears to increase efficiency through the immediate access
to specific and contextual information. However, this comes at
a price, particularly for some junior staff, in terms of anxiety
associated with increasing numbers of priority patients and
information overload, in part exacerbated by false positive alerts.
We also highlight the hitherto unrecognized need to ensure that
relevant clinical specialties include training in prevention if we
are to optimize the value of digital innovations that promote
early intervention. These findings suggest that the true
cost-effectiveness of such innovations cannot be assessed until
the balance between early intervention leading to better
outcomes and increased workload is ascertained. Maximization
of utility also requires finding the most appropriate balance
between sensitivity and specificity for clinical alerts. This can
be difficult to achieve: although it is recognized that the NHS
England AKI algorithm produces false positives, some argue
that this is a necessary trade-off for enhanced sensitivity [33].
Strengths and Limitations
This evaluation has a number of strengths. First, it benefits from
the diversity of our respondent sample, presenting multiple
perspectives on the intervention based on cultural differences
between different professionals and teams. Second, the robust
analysis uncovered issues that are likely to be generalizable to
the implementation of other digital technologies in health care.
Finally, our analysis team includes researchers from medical,
public health, and psychology backgrounds, creating
trustworthiness by encouraging debate and multidisciplinary
interpretation.
Our evaluation has a number of limitations. First, we initiated
interviews during the pathway optimization period to enable us
to gather insights that would improve the pathway itself.
However, interviews at this early stage may have been more
prone to include negative feedback before users were used to
the changes described. The period in which interviews were
conducted did not allow us to assess whether perceptions of the
care pathway changed over time. Second, interviews were
conducted in a single clinical setting. Although our methods
allowed us to identify the active ingredients of a digital
intervention in an acute setting with communication systems
familiar to health care teams worldwide, the magnitude of the
efficiency benefits reported may vary according to the digital
maturity of the health care environment studied.
Comparison With Prior Work
Few studies of AKI alerting systems have been previously
described. Alavijeh et al [34] used a questionnaire to assess
satisfaction with a new automated AKI warning system among
physicians working in primary care. The questionnaire used
was limited to questions about respondents’ knowledge of the
existence of the alert system, perceptions of its utility, and
impact on practice, making comparisons with our findings
difficult. However, in common with our findings, many
respondents found the alert system valuable. Kanagasundam et
al [35] examined the effects of an interruptive AKI clinical
decision support system (embedded within an EPR system)
through a series of semistructured interviews with stakeholders.
Themes revealed were similar to those encountered in the
generic clinical decision support literature, namely, alert fatigue
and user dissatisfaction with mandatory interactions. Although
respondents in Kanagasundam’s evaluation believed that the
alerts led to earlier patient assessment, some clinicians found
them to be an insult to their knowledge. A major reason for
dismissing alerts was users’ need to review a comprehensive
dataset (including historical creatinine) at the point of alert. We
overcame this limitation by including a curated summary of
relevant clinical data in-app at the point of alert. Kanagasundam
et al also reported that the impression that the alert system
prioritized sensitivity over specificity limited perceived
credibility for some users. Our evaluation also demonstrated
the presence of both uncertainties and variations in professional
judgement among specialists, as to what changes in serum
creatinine were clinically significant. In addition, respondents
in our evaluation were occasionally unsure as to whether clinical
intervention was warranted, even for cases where the AKI alert
was perceived to be genuine. Bevan’s [36] exploration of the
impact of a clinical decision support system for AKI embedded
within a single hospital’s EPR found that alerts were unpopular
because of their interruption to the established workflows. We
were able to avoid this problem through the separation of alerts
from the hospital’s EPR so that mobile working allowed users
to integrate alert reviews into their routine working practice.
Our finding that mobile working tools are integrated into
clinicians’ workflow is pertinent given that junior medical staff
currently spend almost half their working day using desktop
computers [37]. Finally, the survey by Oh et al [38] of provider
acceptance of automated electronic alerts for AKI demonstrated
that approval of alerts was positively correlated with the belief
that such alerts improved patient care and negatively correlated
with the belief that alerts did not provide any novel information.
The overall odds of approval decreased over time. Thus, the
success of deploying clinical alerts is dependent on clinicians’
perceptions of their relevance. This tallies with our finding of
diverse opinions about the value of low risk alerts. A number
of mixed-methods analyses of electronic alerting systems for
AKI are still underway; results from the qualitative segments
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.7http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
of the Acute Kidney Outreach to Reduce Deterioration and
Death [39] and Tackling AKI [40] studies are awaited.
Conclusions
Our results are relevant to the design and evaluation of care
pathways that involve automated alerts, mobile working, or the
early deployment of specialist care. Such innovations will
increasingly emerge with the application of machine learning
[41,42] to early diagnosis and disease prediction. Our findings
suggest that alerting systems aiming to encourage early or
preventive action will achieve buy-in from health care
professionals if they believe in the clinical and cost-effectiveness
of the intervention, feel equipped to respond, understand clearly
what their responsibilities are, and feel empowered to act.
Training in prioritization of information is needed to balance
the planned benefits of real-time access to mobile data with the
cognitive load this will produce; digitally enabled pathways
should be designed so that the most appropriate clinician is able
to access the right data at the right time. E-alerting or the early
deployment of a specialist resource may also have an impact
on other clinical teams affected by implementation; future
evaluations should seek to further explore this. Finally, the
inevitable introduction of digital technology to health care is
more likely to improve both patient outcomes and working
practices if aligned with a commitment to proactively identify
and address concomitant, and sometimes unexpected, sequelae.
Acknowledgments
RR and GB are in part supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied
Health Research and Care North Thames at Barts Health NHS Trust, and RR is an NIHR senior investigator. GR and HM are
funded in part by the NIHR University College London Hospitals Biomedical Research Centre. OSA is in part supported by an
NIHR academic clinical fellowship. The authors wish to thank the staff and patients of the RFH and the RFH data monitoring
committee. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of
Health and Social Care.
Authors' Contributions
HM, CL, RR, CH, AK, TB, KA, DK, and MS initiated the project and the collaboration. CL conceived the care pathway. AC,
CL, CM, JC, GJ, SS, and ME supported implementation. RR led the design of the evaluation with assistance from AC, CL, GR,
HM, PM, and CN. AC and CL triaged alerts necessary for the evaluation. AC collected all necessary data that were analyzed by
AC with assistance and oversight from GB and RR. AC, HM, RR, PM, CL, OSA, GB, and GR wrote the paper. All authors read
and agreed to the final submission.
Conflicts of Interest
CL, HM, GR, and RR are paid clinical advisors to DeepMind. AC’s clinical research fellowship was part-funded by DeepMind.
CL was a member of the National Institute for Health and Care Excellence clinical guideline 169 development group referenced
in the paper. DeepMind remained independent from the collection and analysis of all data. HM coholds a patent on a fluid delivery
device that might ultimately help in preventing some (dehydration-related) cases of AKI occurring.
DeepMind was acquired by Google in 2014 and is now a part of the Alphabet group. The deployment of Streams at RFH was
the subject of an investigation by the Information Commissioner’s Office in 2017. RFH has since published an audit completed
to comply with undertakings following this investigation [43]. In November 2018, it was announced that the Streams team will
be joining Google as part of a wider health effort [44].
Multimedia Appendix 1
Care Protocol, nursing advisory sticker, interview guide, and Royal Free Hospital committee members.
[PDF File (Adobe PDF File), 402KB - jmir_v21i7e13143_app1.pdf ]
References
1. Weller J, Boyd M, Cumin D. Teams, tribes and patient safety: overcoming barriers to effective teamwork in healthcare.
Postgrad Med J 2014 Mar;90(1061):149-154. [doi: 10.1136/postgradmedj-2012-131168] [Medline: 24398594]
2. Schwendimann R, Blatter C, Dhaini S, Simon M, Ausserhofer D. The occurrence, types, consequences and preventability
of in-hospital adverse events - a scoping review. BMC Health Serv Res 2018 Dec 4;18(1):521 [FREE Full text] [doi:
10.1186/s12913-018-3335-z] [Medline: 29973258]
3. Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, et al. Systematic review and evaluation of physiological
track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007 Apr;33(4):667-679.
[doi: 10.1007/s00134-007-0532-3] [Medline: 17318499]
4. AHRQ Patient Safety Network. 2019. Rapid Response Systems URL: https://psnet.ahrq.gov/primers/primer/4 [accessed
2018-11-06]
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.8http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
5. Winters BD, Weaver SJ, Pfoh ER, Yang T, Pham JC, Dy SM. Rapid-response systems as a patient safety strategy: a
systematic review. Ann Intern Med 2013 Mar 5;158(5 Pt 2):417-425 [FREE Full text] [doi:
10.7326/0003-4819-158-5-201303051-00009] [Medline: 23460099]
6. Smith F. Who invented that bleeping thing? Br Med J 2003 Sep 27;327(7417):719 [FREE Full text] [doi:
10.1136/bmj.327.7417.719]
7. Gov.uk. 2018. The Future of Healthcare: Our Vision for Digital, Data and Technology in Health and Care URL: https:/
/www.gov.uk/government/publications/the-future-of-healthcare-our-vision-for-digital-data-and-technology-in-health-and-care/
the-future-of-healthcare-our-vision-for-digital-data-and-technology-in-health-and-care [accessed 2018-11-12]
8. Gottlieb S. US Food & Drug Administration. 2018. Transforming FDA's Approach to Digital Health URL: https://www.
fda.gov/newsevents/speeches/ucm605697.htm [accessed 2018-11-14]
9. KDIGO Working Group. KDIGO. 2012. KDIGO clinical practice guidelines for acute kidney injury URL: https://kdigo.
org/wp-content/uploads/2016/10/KDIGO-2012-AKI-Guideline-English.pdf
10. Porter CJ, Juurlink I, Bisset LH, Bavakunji R, Mehta RL, Devonald MA. A real-time electronic alert to improve detection
of acute kidney injury in a large teaching hospital. Nephrol Dial Transplant 2014 Oct;29(10):1888-1893. [doi:
10.1093/ndt/gfu082] [Medline: 24744280]
11. Kerr M, Bedford M, Matthews B, O'Donoghue D. The economic impact of acute kidney injury in England. Nephrol Dial
Transplant 2014 Jul;29(7):1362-1368. [doi: 10.1093/ndt/gfu016] [Medline: 24753459]
12. der Mesropian PJ, Kalamaras JS, Eisele G, Phelps KR, Asif A, Mathew RO. Long-term outcomes of community-acquired
versus hospital-acquired acute kidney injury: a retrospective analysis. Clin Nephrol 2014 Mar;81(3):174-184. [doi:
10.5414/CN108153] [Medline: 24361059]
13. Aitken E, Carruthers C, Gall L, Kerr L, Geddes C, Kingsmore D. Acute kidney injury: outcomes and quality of care. QJM
2013 Apr;106(4):323-332. [doi: 10.1093/qjmed/hcs237] [Medline: 23345468]
14. Wang HE, Muntner P, Chertow GM, Warnock DG. Acute kidney injury and mortality in hospitalized patients. Am J Nephrol
2012;35(4):349-355 [FREE Full text] [doi: 10.1159/000337487] [Medline: 22473149]
15. Chawla LS, Amdur RL, Amodeo S, Kimmel PL, Palant CE. The severity of acute kidney injury predicts progression to
chronic kidney disease. Kidney Int 2011 Jun;79(12):1361-1369 [FREE Full text] [doi: 10.1038/ki.2011.42] [Medline:
21430640]
16. Alleway R. National Confidential Enquiry Into Patient Outcome and Death. 2009. Acute Kidney Injury: Adding Insult to
Injury (2009) URL: http://www.ncepod.org.uk/2009aki.html [accessed 2016-03-08]
17. Selby NM, Hill R, Fluck RJ, NHS England 'Think Kidneys' AKI Programme. Standardizing the early identification of acute
kidney injury: the NHS England national patient safety alert. Nephron 2015;131(2):113-117 [FREE Full text] [doi:
10.1159/000439146] [Medline: 26351847]
18. Hill R, Selby N. Think Kidneys. 2014. Acute Kidney Injury Warning Algorithm: Best Practice Guidance URL: https://www.
thinkkidneys.nhs.uk/wp-content/uploads/2014/12/
AKI-Warning-Algorithm-Best-Practice-Guidance-final-publication-0112141.pdf [accessed 2018-11-14]
19. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, electronic alerts for acute kidney
injury: a single-blind, parallel-group, randomised controlled trial. Lancet 2015 May 16;385(9981):1966-1974 [FREE Full
text] [doi: 10.1016/S0140-6736(15)60266-5] [Medline: 25726515]
20. Connell A, Montgomery H, Morris S, Nightingale C, Stanley S, Emerson M, et al. Service evaluation of the implementation
of a digitally-enabled care pathway for the recognition and management of acute kidney injury. F1000Res 2017;6:1033
[FREE Full text] [doi: 10.12688/f1000research.11637.2] [Medline: 28751970]
21. Connell A, Montgomery H, Martin P, Nightingale C, Sadeghi-Alavijeh O, King D, et al. Evaluation of a digitally-enabled
care pathway for the management of acute kidney injury in patients admitted to hospital as an emergency (in press). NPJ
Digit Med 2019.
22. Connell A, Montgomery H, Martin P, Nightingale C, Sadeghi-Alavijeh O, King D, et al. Implementation of a digitally-enabled
intervention to detect and treat acute kidney injury arising in hospitalised patients: an evaluation of impact on clinical
outcomes and associated healthcare costs (in press). J Med Internet Res 2019.
23. DeepMind Health. How To Use Streams URL: https://support.deepmindhealth.com [accessed 2018-10-18]
24. London AKI Network. AKI Guidelines URL: http://www.londonaki.net/clinical/guidelines-pathways.html [accessed
2018-07-23]
25. National Institute for Health and Care Excellence. 2013. Acute Kidney Injury: Prevention, Detection and Management:
Guidance URL: https://www.nice.org.uk/guidance/cg169 [accessed 2018-11-14]
26. Marshall MN. The key informant technique. Fam Pract 1996 Feb;13(1):92-97. [doi: 10.1093/fampra/13.1.92] [Medline:
8671109]
27. Marshall B, Cardon P, Poddar A, Fontenot R. Does sample size matter in qualitative research?: a review of qualitative
interviews in is research. J Comput Inform Syst 2015 Dec 10;54(1):11-22. [doi: 10.1080/08874417.2013.11645667]
28. Hudelson PM. World Health Organization. 1996. Qualitative Research for Health Programmes URL: https://apps.who.int/
iris/handle/10665/62315 [accessed 2018-07-13]
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.9http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
29. Vasileiou K, Barnett J, Thorpe S, Young T. Characterising and justifying sample size sufficiency in interview-based studies:
systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol 2018 Dec 21;18(1):148
[FREE Full text] [doi: 10.1186/s12874-018-0594-7] [Medline: 30463515]
30. Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive
coding and theme development. Int J Qual Methods 2006;5(1):80-92. [doi: 10.1177/160940690600500107]
31. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [doi:
10.1191/1478088706qp063oa]
32. NHS: Health Research Authority. Is My Study Research? URL: http://www.hra-decisiontools.org.uk/research/redirect.html
[accessed 2018-03-23]
33. Sawhney S, Marks A, Ali T, Clark L, Fluck N, Prescott GJ, et al. Maximising acute kidney injury alerts--a cross-sectional
comparison with the clinical diagnosis. PLoS One 2015;10(6):e0131909 [FREE Full text] [doi: 10.1371/journal.pone.0131909]
[Medline: 26125553]
34. Alavijeh OS, Bansal J, Hadfield K, Laing C, Dawnay A. Implementation of an automated primary care acute kidney injury
warning system: a quantitative and qualitative review of 2 years of experience. Nephron 2017;135(3):189-195. [doi:
10.1159/000452928] [Medline: 28030868]
35. Kanagasundaram NS, Bevan MT, Sims AJ, Heed A, Price DA, Sheerin NS. Computerized clinical decision support for the
early recognition and management of acute kidney injury: a qualitative evaluation of end-user experience. Clin Kidney J
2016 Feb;9(1):57-62 [FREE Full text] [doi: 10.1093/ckj/sfv130] [Medline: 26798462]
36. Bevan M, Heed A, Sheerin NS, Sims A, Price DA, Kanagasundaram NS. Electronic clinical decision support for the early
recognition and management of acute kidney injury: qualitative evaluation of end-user experience. Nephrol Dial Transplant
2015;30(Suppl 3):iii443. [doi: 10.1093/ndt/gfv190.10]
37. Méan M, Garnier A, Wenger N, Castioni J, Waeber G, Marques-Vidal P. Computer usage and task-switching during
resident's working day: disruptive or not? PLoS One 2017;12(2):e0172878 [FREE Full text] [doi:
10.1371/journal.pone.0172878] [Medline: 28235078]
38. Oh J, Bia JR, Ubaid-Ullah M, Testani JM, Wilson FP. Provider acceptance of an automated electronic alert for acute kidney
injury. Clin Kidney J 2016 Aug;9(4):567-571 [FREE Full text] [doi: 10.1093/ckj/sfw054] [Medline: 27478598]
39. Abdelaziz TS, Lindenmeyer A, Baharani J, Mistry H, Sitch A, Temple RM, et al. Acute kidney outreach to reduce
deterioration and death (AKORDD) trial: the protocol for a large pilot study. BMJ Open 2016 Dec 19;6(8):e012253 [FREE
Full text] [doi: 10.1136/bmjopen-2016-012253] [Medline: 27543592]
40. Selby NM, Casula A, Lamming L, Mohammed M, Caskey F, Tackling AKI Investigators. Design and rationale of 'tackling
acute kidney injury', a multicentre quality improvement study. Nephron 2016;134(3):200-204 [FREE Full text] [doi:
10.1159/000447675] [Medline: 27376867]
41. Koyner JL, Carey KA, Edelson DP, Churpek MM. The development of a machine learning inpatient acute kidney injury
prediction model. Crit Care Med 2018 Jul;46(7):1070-1077. [doi: 10.1097/CCM.0000000000003123] [Medline: 29596073]
42. Mohamadlou H, Lynn-Palevsky A, Barton C, Chettipally U, Shieh L, Calvert J, et al. Prediction of acute kidney injury with
a machine learning algorithm using electronic health record data. Can J Kidney Health Dis 2018;5:2054358118776326
[FREE Full text] [doi: 10.1177/2054358118776326] [Medline: 30094049]
43. Royal Free Hospital. 2018. Royal Free London Publishes Audit Into Streams App URL: https://www.royalfree.nhs.uk/
news-media/news/royal-free-london-publishes-audit-into-streams-app/ [accessed 2018-10-18]
44. DeepMind Technologies. Scaling Streams with Google URL: https://deepmind.com/blog/scaling-streams-google/ [accessed
2018-11-21]
Abbreviations
AKI: acute kidney injury
EPR: electronic patient record
NHS: National Health Service
NIHR: National Institute for Health Research
PARRT: patient-at-risk and resuscitation team
RFH: Royal Free Hospital
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.10http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
Edited by G Eysenbach; submitted 14.12.18; peer-reviewed by M Devonald, J Varghese, N Selby; comments to author 17.01.19;
revised version received 29.01.19; accepted 24.03.19; published 06.07.19.
Please cite as:
Connell A, Black G, Montgomery H, Martin P, Nightingale C, King D, Karthikesalingam A, Hughes C, Back T, Ayoub K, Suleyman
M, Jones G, Cross J, Stanley S, Emerson M, Merrick C, Rees G, Laing C, Raine R
A Qualitative Evaluation of User Experiences of a Digitally Enabled Care Pathway in Secondary Care
J Med Internet Res 2019;21(7):e13143
URL: http://www.jmir.org/2019/7/e13143/
doi:10.2196/13143
PMID:
©Alistair Connell, Georgia Black, Hugh Montgomery, Peter Martin, Claire Nightingale, Dominic King, Alan Karthikesalingam,
Cían Hughes, Trevor Back, Kareem Ayoub, Mustafa Suleyman, Gareth Jones, Jennifer Cross, Sarah Stanley, Mary Emerson,
Charles Merrick, Geraint Rees, Christopher Laing, Rosalind Raine. Originally published in the Journal of Medical Internet
Research (http://www.jmir.org), 06.07.2019. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The
complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license
information must be included.
J Med Internet Res 2019 | vol. 21 | iss. 7 | e13143 | p.11http://www.jmir.org/2019/7/e13143/ (page number not for citation purposes)
Connell et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
FO
RenderX
... Organizations have only just started to adjust their pathways accordingly in recent years, and researchers have increasingly presented new approaches for pathway development considering the changes brought about by digitalization and new technological advances such as AI. Again, different terms are being used in this context: E.g., "digital health pathway" (44) or "digital care pathway" (45)(46)(47)(48), "partially digital pathway" (49), "human-centered integrated care pathways" or simply "integrated care pathways" (50), or "digitally enabled care pathway" (51)(52)(53). Table 2 provides a comprehensive, albeit not exhaustive overview of corresponding changes in selected pathway functions (traditional paper-based vs. digital pathway) along with potential benefits and drawbacks. ...
... Up until now, however, the number of examples for digital pathways that can be found in the literature is still fairly low. Examples we found refer to conditions such as arrhythmia (45), acute kidney injury (51)(52)(53), HIV (60), genetic testing for breast cancer (49) and Parkinson's Disease (53). But, as far as we know, a digital pathway for MS, or an approach for how such a pathway could be built has not been put forward yet. ...
... Up until now, however, the number of examples for digital pathways that can be found in the literature is still fairly low. Examples we found refer to conditions such as arrhythmia (45), acute kidney injury (51)(52)(53), HIV (60), genetic testing for breast cancer (49) and Parkinson's Disease (53). But, as far as we know, a digital pathway for MS, or an approach for how such a pathway could be built has not been put forward yet. ...
Article
Full-text available
Recent advances in the field of artificial intelligence (AI) could yield new insights into the potential causes of multiple sclerosis (MS) and factors influencing its course as the use of AI opens new possibilities regarding the interpretation and use of big data from not only a cross-sectional, but also a longitudinal perspective. For each patient with MS, there is a vast amount of multimodal data being accumulated over time. But for the application of AI and related technologies, these data need to be available in a machine-readable format and need to be collected in a standardized and structured manner. Through the use of mobile electronic devices and the internet it has also become possible to provide healthcare services from remote and collect information on a patient’s state of health outside of regular check-ups on site. Against this background, we argue that the concept of pathways in healthcare now could be applied to structure the collection of information across multiple devices and stakeholders in the virtual sphere, enabling us to exploit the full potential of AI technology by e.g., building digital twins. By going digital and using pathways, we can virtually link patients and their caregivers. Stakeholders then could rely on digital pathways for evidence-based guidance in the sequence of procedures and selection of therapy options based on advanced analytics supported by AI as well as for communication and education purposes. As far as we aware of, however, pathway modelling with respect to MS management and treatment has not been thoroughly investigated yet and still needs to be discussed. In this paper, we thus present our ideas for a modular-integrative framework for the development of digital patient pathways for MS treatment.
... Therefore, the study provides initial evidence that the use of the HAIL system in a clinical context could be a starting point for reducing the amount of information presented to clinical staff. With a similar aim, Connell et al. (2019) also tested the use of automated alarms in medical care and showed that patient care improved due to the alarms. However, effects on information overload were not reported. ...
... For the implementation of automated alarms, the authors recommended that users should be trained to ensure that the workload and the handling of the information received is well organized. Furthermore, such systems should be continuously optimized to reduce false alarms (Connell et al., 2019). ...
Article
Full-text available
Information overload is a problem that is being exacerbated by the ongoing digitalization of the world of work and the growing use of information and communication technologies. Therefore, the aim of this systematic literature review is to provide an insight into existing measures for prevention and intervention related to information overload. The methodological approach of the systematic review is based on the PRISMA standards. A keyword search in three interdisciplinary scientific databases and other more practice-oriented databases resulted in the identification of 87 studies, field reports, and conceptual papers that were included in the review. The results show that a considerable number of papers have been published on interventions on the behavioral prevention level. At the level of structural prevention, there are also many proposals on how to design work to reduce information overload. A further distinction can be made between work design approaches at the level of information and communication technology and at the level of teamwork and organizational regulations. Although the identified studies cover a wide range of possible interventions and design approaches to address information overload, the strength of the evidence from these studies is mixed.
... This might indicate either that the technology is not ready yet, or, considering 4 of these 5 articles were published in the last 3 years, that the technology is just starting to creep into real clinical settings. Recent evidence suggests that it is paramount to test newly developed algorithms in clinical settings before trying to deploy them [88]. Despite promising laboratory results, clinical translation is not always guaranteed. ...
... These studies showed promising results and did not report any unintended consequences after evaluation and/or validation. Notwithstanding the robust results described by the authors, as discussed before, recent evidence suggests that it is paramount to test newly developed algorithms in clinical settings to avoid adverse or unintended consequences [88,89]. Taking into account the pinnacle of importance of validating ones' results in real clinical contexts and not just "in lab", in reading their results, their generalizability has to be questioned, especially since no multi-center validations were described among the validated models. ...
Article
Full-text available
Objectives In this systematic review we aimed at assessing how artificial intelligence (AI), including machine learning (ML) techniques have been deployed to predict, diagnose, and treat chronic kidney disease (CKD). We systematically reviewed the available evidence on these innovative techniques to improve CKD diagnosis and patient management. Methods We included English language studies retrieved from PubMed. The review is therefore to be classified as a "rapid review", since it includes one database only, and has language restrictions; the novelty and importance of the issue make missing relevant papers unlikely. We extracted 16 variables, including: main aim, studied population, data source, sample size, problem type (regression, classification), predictors used, and performance metrics. We followed the Preferred Reporting Items for Systematic Reviews (PRISMA) approach; all main steps were done in duplicate. Results From a total of 648 studies initially retrieved, 68 articles met the inclusion criteria. Models, as reported by authors, performed well, but the reported metrics were not homogeneous across articles and therefore direct comparison was not feasible. The most common aim was prediction of prognosis, followed by diagnosis of CKD. Algorithm generalizability, and testing on diverse populations was rarely taken into account. Furthermore, the clinical evaluation and validation of the models/algorithms was perused; only a fraction of the included studies, 6 out of 68, were performed in a clinical context. Conclusions Machine learning is a promising tool for the prediction of risk, diagnosis, and therapy management for CKD patients. Nonetheless, future work is needed to address the interpretability, generalizability, and fairness of the models to ensure the safe application of such technologies in routine clinical practice.
... Generative AI models require continuous and accurate updates of data, and expert supervision and must address potential biases and limitations. Recent evidence stresses that the importance of the validation of their results in real clinical contexts suggests that it is paramount to test newly developed algorithms before trying to deploy them 33 . Despite the potential benefits and promising results, clinical translation is not always guaranteed and presents several issues, namely, fairness, model, and results interpretability 34 and the lack of validated models. ...
Article
Full-text available
Artificial intelligence (AI) generative models driven by the integration of AI and natural language processing technologies, such as OpenAI’s chatbot generative pre-trained transformer large language model (LLM), are receiving much public attention and have the potential to transform personalized medicine. Dialysis patients are highly dependent on technology and their treatment generates a challenging large volume of data that has to be analyzed for knowledge extraction. We argue that, by integrating the data acquired from hemodialysis treatments with the powerful conversational capabilities of LLMs, nephrologists could personalize treatments adapted to patients’ lifestyles and preferences. We also argue that this new conversational AI integrated with a personalized patient-computer interface will enhance patients’ engagement and self-care by providing them with a more personalized experience. However, generative AI models require continuous and accurate updates of data, and expert supervision and must address potential biases and limitations. Dialysis patients can also benefit from other new emerging technologies such as Digital Twins with which patients’ care can also be addressed from a personalized medicine perspective. In this paper, we will revise LLMs potential strengths in terms of their contribution to personalized medicine, and, in particular, their potential impact, and limitations in nephrology. Nephrologists’ collaboration with AI academia and companies, to develop algorithms and models that are more transparent, understandable, and trustworthy, will be crucial for the next generation of dialysis patients. The combination of technology, patient-specific data, and AI should contribute to create a more personalized and interactive dialysis process, improving patients’ quality of life.
... They assume that every model prediction gets acted upon, and thus ignore the structure and constraints of the relevant workflow (e.g. budget, staffing, data acquisition delays, human error, etc.) [6,9,21]. ...
Article
Full-text available
Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.
... It misses the totality of a patient's lived health experience and may not be the best first point of contact for all clinical conditions. Digital health pathways, on the other hand, provide an opportunity to improve health outcomes by enabling the delivery of continuous, proactive, and personalized care [9]. To enable this digital transformation, health care systems must reimagine the care delivery model and identify whether they need to buy, build, or partner to acquire the capabilities necessary to deliver this care. ...
Article
Full-text available
The digital transformation of our health care system will require not only digitization of existing tools but also a redesign of our care delivery system and collaboration with digital partners. Traditional patient journeys are reactive to symptom presentation and delayed by health care system-centric scheduling, leading to poor experience and avoidable adverse outcomes. Patient journeys will be reimagined to a digital health pathway that seamlessly integrates various care experiences from telemedicine, remote monitoring, to in-person clinic visits. Through centering the care delivery around the patients, they can have more delightful experiences and enjoy the quality of standardized condition pathways and outcomes. To design and implement digital health pathways at scale, enterprise health care systems need to develop capabilities and partnerships in human-centered design, operational workflow, clinical content management, communication channels and mechanisms, reporting and analytics, standards-based integration, security and data management, and scalability. Using a human-centered design methodology, care pathways will be built upon an understanding of the unmet needs of the patients to have a more enjoyable experience of care with improved clinical outcomes. To power this digital care pathway, enterprises will choose to build or partner for clinical content management to operationalize up-to-date, best-in-class pathways. With this clinical engine, this digital solution will engage with patients through multimodal communication modalities, including written, audio, photo, or video, throughout the patient journey. Leadership teams will review reporting and analytics functions to track that the digital care pathways will be iterated to improve patient experience, clinical metrics, and operational efficiency. On the backend, standards-based integration will allow this system to be built in conjunction with the electronic medical record and other data systems to provide safe and efficient use of the digital care solution. For protecting patient information and compliance, a security and data management strategy is critical to derisking breeches and preserving privacy. Finally, a framework of technical scalability will allow digital care pathways to proliferate throughout the enterprise and support the entire patient population. This framework empowers enterprise health care systems to avoid collecting a fragmented series of one-off solutions but develop a sustainable concerted roadmap to the future of proactive intelligent patient care.
... If the alert is via a mobile telephone, and the clinician is remote from the patient, the drug chart or the previous results, reaction may be less consistent. Training and resource are important issues and may hamper success of software [35]. We are aware of the dangers of alert fatigue [36,37] and are careful to avoid over alerting. ...
Article
Full-text available
Background Electronic alerts (e-alerts) for Acute Kidney Injury (AKI) have been implemented into a variety of different Electronic Health Records (EHR) systems worldwide in order to improve recognition and encourage early appropriate management of AKI. We were interested in the impact on patient safety, specialist referral and clinical management. Methods All patients admitted to our institution with AKI were included in the study. We studied AKI progression, dialysis dependency, length of hospital stay, emergency readmission, ICU readmission, and death, before and after the introduction of electronic alerts. The impact on prescription of high risk drugs, fluid administration, and referral to renal services was also analysed. Results After the introduction of the e-alert, progression to higher AKI stage, emergency readmission to hospital and death during admission were significantly reduced. More prescriptions were stopped for drugs that adversely affect renal function in AKI and there was a significant increase in the ICU admissions and in the number of patients having dialysis, especially in earlier stages. Longer term mortality, renal referrals, and fluid alteration did not change significantly after the AKI e-alert introduction. Conclusions AKI e-alerts can improve clinical outcomes in hospitalised patients.
Article
Objective: This study aimed to acquire insight into the perceived user friendliness, accessibility and inclusiveness of a personalized digital care pathway. Materials & methods: Usability of the tool was tested in an experimental setting. Mixed methods data collection consisted of scenario-based eye tracking tests in a web- or mobile-based prototype of the tool, followed by a questionnaire assessing user friendliness (System Usability Scale; SUS) and a structured interview. Inclusiveness was assessed by subgroup comparisons based on language proficiency, age and education level. Via purposive sampling a heterogeneous population of users (N = 24) was recruited. Eye tracking was used to measure gaze behavior. Results: Overall, participants were satisfied with the tool (scale 0-10, 7.5; SD = 1.29). User friendliness of the mobile version (68.3; SD = 21.6) was higher than the web version (50.9; SD = 17.3) measured by SUS score (0-100). With regard to accessibility, eye tracking scenarios showed that the menu bar was hard to find (17% mobile, 55% web). In all scenario's, information was found faster in the mobile version than the web version. Attention was easily drawn to images. Regarding inclusiveness of the tool, we found significantly longer completing time of the scenario tasks for low language proficiency (p-value = 0.029) and higher age subgroups (p-value = 0.049). Lower language proficiency scored a significant lower SUS score (p-value = 0.012). Conclusions: Overall, user friendliness and accessibility were positively evaluated. Assessment of inclusiveness emphasized the need for tailoring digital tools to those with low language proficiency and/or an older age. Co-creation of digital care tools with users is therefore important to match users' needs, make tools easily understandable and accessible to all users, and ultimately result in better uptake and impact.
Article
Full-text available
Background The rhetoric surrounding clinical artificial intelligence (AI) often exaggerates its effect on real-world care. Limited understanding of the factors that influence its implementation can perpetuate this. Objective In this qualitative systematic review, we aimed to identify key stakeholders, consolidate their perspectives on clinical AI implementation, and characterize the evidence gaps that future qualitative research should target. Methods Ovid-MEDLINE, EBSCO-CINAHL, ACM Digital Library, Science Citation Index-Web of Science, and Scopus were searched for primary qualitative studies on individuals’ perspectives on any application of clinical AI worldwide (January 2014-April 2021). The definition of clinical AI includes both rule-based and machine learning–enabled or non–rule-based decision support tools. The language of the reports was not an exclusion criterion. Two independent reviewers performed title, abstract, and full-text screening with a third arbiter of disagreement. Two reviewers assigned the Joanna Briggs Institute 10-point checklist for qualitative research scores for each study. A single reviewer extracted free-text data relevant to clinical AI implementation, noting the stakeholders contributing to each excerpt. The best-fit framework synthesis used the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework. To validate the data and improve accessibility, coauthors representing each emergent stakeholder group codeveloped summaries of the factors most relevant to their respective groups. Results The initial search yielded 4437 deduplicated articles, with 111 (2.5%) eligible for inclusion (median Joanna Briggs Institute 10-point checklist for qualitative research score, 8/10). Five distinct stakeholder groups emerged from the data: health care professionals (HCPs), patients, carers and other members of the public, developers, health care managers and leaders, and regulators or policy makers, contributing 1204 (70%), 196 (11.4%), 133 (7.7%), 129 (7.5%), and 59 (3.4%) of 1721 eligible excerpts, respectively. All stakeholder groups independently identified a breadth of implementation factors, with each producing data that were mapped between 17 and 24 of the 27 adapted Nonadoption, Abandonment, Scale-up, Spread, and Sustainability subdomains. Most of the factors that stakeholders found influential in the implementation of rule-based clinical AI also applied to non–rule-based clinical AI, with the exception of intellectual property, regulation, and sociocultural attitudes. Conclusions Clinical AI implementation is influenced by many interdependent factors, which are in turn influenced by at least 5 distinct stakeholder groups. This implies that effective research and practice of clinical AI implementation should consider multiple stakeholder perspectives. The current underrepresentation of perspectives from stakeholders other than HCPs in the literature may limit the anticipation and management of the factors that influence successful clinical AI implementation. Future research should not only widen the representation of tools and contexts in qualitative research but also specifically investigate the perspectives of all stakeholder HCPs and emerging aspects of non–rule-based clinical AI implementation. Trial Registration PROSPERO (International Prospective Register of Systematic Reviews) CRD42021256005; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=256005 International Registered Report Identifier (IRRID) RR2-10.2196/33145
Article
Full-text available
Objective A prerequisite for patient-centredness in healthcare organisations is offering patients access to adequate health information, which fits their needs. A personalised digital care pathway (PDCP) is a tool that facilitates the provision of tailored and timely information. Despite its potential, barriers influence the implementation of digital tools in healthcare organisations. Therefore, we investigated the perceived barriers and facilitators for implementation of the PDCP among stakeholders. Design A qualitative study was conducted to acquire insight into perceptions of the stakeholders involved in the implementation of a digital care pathway in three diverse patient groups. Setting This study is part of the PDCP research project in a large academic hospital in the Netherlands. Participants Purposive sampling was used to recruit internal stakeholders (eg, healthcare professionals, employees of the supporting departments) and external stakeholders (eg, employees of the external PDCP supplier). In addition, existing semistructured interviews with patients involved in pilot implementation (n=24) were used to verify the findings. Results We conducted 25 semistructured interviews using the Consolidated Framework for Implementation Research. Content analyses yielded four themes: (1) stakeholders’ perceptions of the PDCP (eg, perceived usefulness); (2) characteristics of the individuals involved and the implementation process (eg, individuals express resistance to change); (3) organisational readiness (eg, lack of resources); and (4) collaboration within the organisation (eg, mutual communication, multidisciplinary codesign). The main barriers mentioned by patients were duration of first activation and necessity for up-to-date content. In addition, the most facilitating factor for patients was user-friendliness. Conclusion Our findings emphasise the importance of gaining insights into the various perspectives of stakeholder groups, including patients, regarding the implementation of the PDCP. The perceived barriers and facilitators can be used to improve the PDCP implementation plan and tailor the development and improvement of other digital patient communication tools.
Article
Full-text available
Background: The development of acute kidney injury (AKI) in hospitalized patients is associated with adverse outcomes and increased health care costs. Simple automated e-alerts indicating its presence do not appear to improve outcomes, perhaps because of a lack of explicitly defined integration with a clinical response. Objective: We sought to test this hypothesis by evaluating the impact of a digitally enabled intervention on clinical outcomes and health care costs associated with AKI in hospitalized patients. Methods: We developed a care pathway comprising automated AKI detection, mobile clinician notification, in-app triage, and a protocolized specialist clinical response. We evaluated its impact by comparing data from pre- and postimplementation phases (May 2016 to January 2017 and May to September 2017, respectively) at the intervention site and another site not receiving the intervention. Clinical outcomes were analyzed using segmented regression analysis. The primary outcome was recovery of renal function to ≤120% of baseline by hospital discharge. Secondary clinical outcomes were mortality within 30 days of alert, progression of AKI stage, transfer to renal/intensive care units, hospital re-admission within 30 days of discharge, dependence on renal replacement therapy 30 days after discharge, and hospital-wide cardiac arrest rate. Time taken for specialist review of AKI alerts was measured. Impact on health care costs as defined by Patient-Level Information and Costing System data was evaluated using difference-in-differences (DID) analysis. Results: The median time to AKI alert review by a specialist was 14.0 min (interquartile range 1.0-60.0 min). There was no impact on the primary outcome (estimated odds ratio [OR] 1.00, 95% CI 0.58-1.71; P=.99). Although the hospital-wide cardiac arrest rate fell significantly at the intervention site (OR 0.55, 95% CI 0.38-0.76; P<.001), DID analysis with the comparator site was not significant (OR 1.13, 95% CI 0.63-1.99; P=.69). There was no impact on other secondary clinical outcomes. Mean health care costs per patient were reduced by £2123 (95% CI −£4024 to −£222; P=.03), not including costs of providing the technology. Conclusions: The digitally enabled clinical intervention to detect and treat AKI in hospitalized patients reduced health care costs and possibly reduced cardiac arrest rates. Its impact on other clinical outcomes and identification of the active components of the pathway requires clarification through evaluation across multiple sites.
Article
Full-text available
Background: The development of acute kidney injury (AKI) in hospitalized patients is associated with adverse outcomes and increased health care costs. Simple automated e-alerts indicating its presence do not appear to improve outcomes, perhaps because of a lack of explicitly defined integration with a clinical response. Objective: We sought to test this hypothesis by evaluating the impact of a digitally enabled intervention on clinical outcomes and health care costs associated with AKI in hospitalized patients. Methods: We developed a care pathway comprising automated AKI detection, mobile clinician notification, in-app triage, and a protocolized specialist clinical response. We evaluated its impact by comparing data from pre- and postimplementation phases (May 2016 to January 2017 and May to September 2017, respectively) at the intervention site and another site not receiving the intervention. Clinical outcomes were analyzed using segmented regression analysis. The primary outcome was recovery of renal function to ≤120% of baseline by hospital discharge. Secondary clinical outcomes were mortality within 30 days of alert, progression of AKI stage, transfer to renal/intensive care units, hospital re-admission within 30 days of discharge, dependence on renal replacement therapy 30 days after discharge, and hospital-wide cardiac arrest rate. Time taken for specialist review of AKI alerts was measured. Impact on health care costs as defined by Patient-Level Information and Costing System data was evaluated using difference-in-differences (DID) analysis. Results: The median time to AKI alert review by a specialist was 14.0 min (interquartile range 1.0-60.0 min). There was no impact on the primary outcome (estimated odds ratio [OR] 1.00, 95% CI 0.58-1.71; P=.99). Although the hospital-wide cardiac arrest rate fell significantly at the intervention site (OR 0.55, 95% CI 0.38-0.76; P<.001), DID analysis with the comparator site was not significant (OR 1.13, 95% CI 0.63-1.99; P=.69). There was no impact on other secondary clinical outcomes. Mean health care costs per patient were reduced by £2123 (95% CI -£4024 to -£222; P=.03), not including costs of providing the technology. Conclusions: The digitally enabled clinical intervention to detect and treat AKI in hospitalized patients reduced health care costs and possibly reduced cardiac arrest rates. Its impact on other clinical outcomes and identification of the active components of the pathway requires clarification through evaluation across multiple sites.
Article
Full-text available
We developed a digitally enabled care pathway for acute kidney injury (AKI) management incorporating a mobile detection application, specialist clinical response team and care protocol. Clinical outcome data were collected from adults with AKI on emergency admission before (May 2016 to January 2017) and after (May to September 2017) deployment at the intervention site and another not receiving the intervention. Changes in primary outcome (serum creatinine recovery to ≤120% baseline at hospital discharge) and secondary outcomes (30-day survival, renal replacement therapy, renal or intensive care unit (ICU) admission, worsening AKI stage and length of stay) were measured using interrupted time-series regression. Processes of care data (time to AKI recognition, time to treatment) were extracted from casenotes, and compared over two 9-month periods before and after implementation (January to September 2016 and 2017, respectively) using pre–post analysis. There was no step change in renal recovery or any of the secondary outcomes. Trends for creatinine recovery rates (estimated odds ratio (OR) = 1.04, 95% confidence interval (95% CI): 1.00–1.08, p = 0.038) and renal or ICU admission (OR = 0.95, 95% CI: 0.90–1.00, p = 0.044) improved significantly at the intervention site. However, difference-in-difference analyses between sites for creatinine recovery (estimated OR = 0.95, 95% CI: 0.90–1.00, p = 0.053) and renal or ICU admission (OR = 1.06, 95% CI: 0.98–1.16, p = 0.140) were not significant. Among process measures, time to AKI recognition and treatment of nephrotoxicity improved significantly (p < 0.001 and 0.047 respectively).
Article
Full-text available
Background: Choosing a suitable sample size in qualitative research is an area of conceptual debate and practical uncertainty. That sample size principles, guidelines and tools have been developed to enable researchers to set, and justify the acceptability of, their sample size is an indication that the issue constitutes an important marker of the quality of qualitative research. Nevertheless, research shows that sample size sufficiency reporting is often poor, if not absent, across a range of disciplinary fields. Methods: A systematic analysis of single-interview-per-participant designs within three health-related journals from the disciplines of psychology, sociology and medicine, over a 15-year period, was conducted to examine whether and how sample sizes were justified and how sample size was characterised and discussed by authors. Data pertinent to sample size were extracted and analysed using qualitative and quantitative analytic techniques. Results: Our findings demonstrate that provision of sample size justifications in qualitative health research is limited; is not contingent on the number of interviews; and relates to the journal of publication. Defence of sample size was most frequently supported across all three journals with reference to the principle of saturation and to pragmatic considerations. Qualitative sample sizes were predominantly – and often without justification – characterised as insufficient (i.e., ‘small’) and discussed in the context of study limitations. Sample size insufficiency was seen to threaten the validity and generalizability of studies’ results, with the latter being frequently conceived in nomothetic terms. Conclusions: We recommend, firstly, that qualitative health researchers be more transparent about evaluations of their sample size sufficiency, situating these within broader and more encompassing assessments of data adequacy. Secondly, we invite researchers critically to consider how saturation parameters found in prior methodological studies and sample size community norms might best inform, and apply to, their own project and encourage that data adequacy is best appraised with reference to features that are intrinsic to the study at hand. Finally, those reviewing papers have a vital role in supporting and encouraging transparent study-specific reporting.
Article
Full-text available
Background: Adverse events (AEs) seriously affect patient safety and quality of care, and remain a pressing global issue. This study had three objectives: (1) to describe the proportions of patients affected by in-hospital AEs; (2) to explore the types and consequences of observed AEs; and (3) to estimate the preventability of in-hospital AEs. Methods: We applied a scoping review method and concluded a comprehensive literature search in PubMed and CINAHL in May 2017 and in February 2018. Our target was retrospective medical record review studies applying the Harvard method-or similar methods using screening criteria-conducted in acute care hospital settings on adult patients (≥18 years). Results: We included a total of 25 studies conducted in 27 countries across six continents. Overall, a median of 10% patients were affected by at least one AE (range: 2.9-21.9%), with a median of 7.3% (range: 0.6-30%) of AEs being fatal. Between 34.3 and 83% of AEs were considered preventable (median: 51.2%). The three most common types of AEs reported in the included studies were operative/surgical related, medication or drug/fluid related, and healthcare-associated infections. Conclusions: Evidence regarding the occurrence of AEs confirms earlier estimates that a tenth of inpatient stays include adverse events, half of which are preventable. However, the incidence of in-hospital AEs varied considerably across studies, indicating methodological and contextual variations regarding this type of retrospective chart review across health care systems. For the future, automated methods for identifying AE using electronic health records have the potential to overcome various methodological issues and biases related to retrospective medical record review studies and to provide accurate data on their occurrence.
Article
Full-text available
Background A major problem in treating acute kidney injury (AKI) is that clinical criteria for recognition are markers of established kidney damage or impaired function; treatment before such damage manifests is desirable. Clinicians could intervene during what may be a crucial stage for preventing permanent kidney injury if patients with incipient AKI and those at high risk of developing AKI could be identified. Objective In this study, we evaluate a machine learning algorithm for early detection and prediction of AKI. Design We used a machine learning technique, boosted ensembles of decision trees, to train an AKI prediction tool on retrospective data taken from more than 300 000 inpatient encounters. Setting Data were collected from inpatient wards at Stanford Medical Center and intensive care unit patients at Beth Israel Deaconess Medical Center. Patients Patients older than the age of 18 whose hospital stays lasted between 5 and 1000 hours and who had at least one documented measurement of heart rate, respiratory rate, temperature, serum creatinine (SCr), and Glasgow Coma Scale (GCS). Measurements We tested the algorithm’s ability to detect AKI at onset and to predict AKI 12, 24, 48, and 72 hours before onset. Methods We tested AKI detection and prediction using the National Health Service (NHS) England AKI Algorithm as a gold standard. We additionally tested the algorithm’s ability to detect AKI as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines. We compared the algorithm’s 3-fold cross-validation performance to the Sequential Organ Failure Assessment (SOFA) score for AKI identification in terms of area under the receiver operating characteristic (AUROC). Results The algorithm demonstrated high AUROC for detecting and predicting NHS-defined AKI at all tested time points. The algorithm achieves AUROC of 0.872 (95% confidence interval [CI], 0.867-0.878) for AKI detection at time of onset. For prediction 12 hours before onset, the algorithm achieves an AUROC of 0.800 (95% CI, 0.792-0.809). For 24-hour predictions, the algorithm achieves AUROC of 0.795 (95% CI, 0.785-0.804). For 48-hour and 72-hour predictions, the algorithm achieves AUROC values of 0.761 (95% CI, 0.753-0.768) and 0.728 (95% CI, 0.719-0.737), respectively. Limitations Because of the retrospective nature of this study, we cannot draw any conclusions about the impact the algorithm’s predictions will have on patient outcomes in a clinical setting. Conclusions The results of these experiments suggest that a machine learning–based AKI prediction tool may offer important prognostic capabilities for determining which patients are likely to suffer AKI, potentially allowing clinicians to intervene before kidney damage manifests.
Article
Full-text available
Acute Kidney Injury (AKI), an abrupt deterioration in kidney function, is defined by changes in urine output or serum creatinine. AKI is common (affecting up to 20% of acute hospital admissions in the United Kingdom), associated with significant morbidity and mortality, and expensive (excess costs to the National Health Service in England alone may exceed £1 billion per year). NHS England has mandated the implementation of an automated algorithm to detect AKI based on changes in serum creatinine, and to alert clinicians. It is uncertain, however, whether ‘alerting’ alone improves care quality. We have thus developed a digitally-enabled care pathway as a clinical service to inpatients in the Royal Free Hospital (RFH), a large London hospital. This pathway incorporates a mobile software application - the “Streams-AKI” app, developed by DeepMind Health - that applies the NHS AKI algorithm to routinely collected serum creatinine data in hospital inpatients. Streams-AKI alerts clinicians to potential AKI cases, furnishing them with a trend view of kidney function alongside other relevant data, in real-time, on a mobile device. A clinical response team comprising nephrologists and critical care nurses responds to these AKI alerts by reviewing individual patients and administering interventions according to existing clinical practice guidelines. We propose a mixed methods service evaluation of the implementation of this care pathway. This evaluation will assess how the care pathway meets the health and care needs of service users (RFH inpatients), in terms of clinical outcome, processes of care, and NHS costs. It will also seek to assess acceptance of the pathway by members of the response team and wider hospital community. All analyses will be undertaken by the service evaluation team from UCL (Department of Applied Health Research) and St George’s, University of London (Population Health Research Institute).
Article
Full-text available
Acute Kidney Injury (AKI), an abrupt deterioration in kidney function, is defined by changes in urine output or serum creatinine. AKI is common (affecting up to 20% of acute hospital admissions in the United Kingdom), associated with significant morbidity and mortality, and expensive (excess costs to the National Health Service in England alone may exceed £1 billion per year). NHS England has mandated the implementation of an automated algorithm to detect AKI based on changes in serum creatinine, and to alert clinicians. It is uncertain, however, whether ‘alerting’ alone improves care quality. We have thus developed a digitally-enabled care pathway as a clinical service to inpatients in the Royal Free Hospital (RFH), a large London hospital. This pathway incorporates a mobile software application - the “Streams-AKI” app, developed by DeepMind Health - that applies the NHS AKI algorithm to routinely collected serum creatinine data in hospital inpatients. Streams-AKI alerts clinicians to potential AKI cases, furnishing them with a trend view of kidney function alongside other relevant data, in real-time, on a mobile device. A clinical response team comprising nephrologists and critical care nurses responds to these AKI alerts by reviewing individual patients and administering interventions according to existing clinical practice guidelines. We propose a mixed methods service evaluation of the implementation of this care pathway. This evaluation will assess how the care pathway meets the health and care needs of service users (RFH inpatients), in terms of clinical outcome, processes of care, and NHS costs. It will also seek to assess acceptance of the pathway by members of the response team and wider hospital community. All analyses will be undertaken by the service evaluation team from UCL (Department of Applied Health Research) and St George’s, University of London (Population Health Research Institute).
Article
Full-text available
Background Recent implementation of electronic health records (EHR) has dramatically changed medical ward organization. While residents in general internal medicine use EHR systems half of their working time, whether computer usage impacts residents’ workflow remains uncertain. We aimed to observe the frequency of task-switches occurring during resident’s work and to assess whether computer usage was associated with task-switching. Methods In a large Swiss academic university hospital, we conducted, between May 26 and July 24, 2015 a time-motion study to assess how residents in general internal medicine organize their working day. Results We observed 49 day and 17 evening shifts of 36 residents, amounting to 697 working hours. During day shifts, residents spent 5.4 hours using a computer (mean total working time: 11.6 hours per day). On average, residents switched 15 times per hour from a task to another. Task-switching peaked between 8:00–9:00 and 16:00–17:00. Task-switching was not associated with resident’s characteristics and no association was found between task-switching and extra hours (Spearman r = 0.220, p = 0.137 for day and r = 0.483, p = 0.058 for evening shifts). Computer usage occurred more frequently at the beginning or ends of day shifts and was associated with decreased overall task-switching. Conclusion Task-switching occurs very frequently during resident’s working day. Despite the fact that residents used a computer half of their working time, computer usage was associated with decreased task-switching. Whether frequent task-switches and computer usage impact the quality of patient care and resident’s work must be evaluated in further studies.
Article
Objectives: To develop an acute kidney injury risk prediction model using electronic health record data for longitudinal use in hospitalized patients. Design: Observational cohort study. Setting: Tertiary, urban, academic medical center from November 2008 to January 2016. Patients: All adult inpatients without pre-existing renal failure at admission, defined as first serum creatinine greater than or equal to 3.0 mg/dL, International Classification of Diseases, 9th Edition, code for chronic kidney disease stage 4 or higher or having received renal replacement therapy within 48 hours of first serum creatinine measurement. Interventions: None. Measurements and main results: Demographics, vital signs, diagnostics, and interventions were used in a Gradient Boosting Machine algorithm to predict serum creatinine-based Kidney Disease Improving Global Outcomes stage 2 acute kidney injury, with 60% of the data used for derivation and 40% for validation. Area under the receiver operator characteristic curve (AUC) was calculated in the validation cohort, and subgroup analyses were conducted across admission serum creatinine, acute kidney injury severity, and hospital location. Among the 121,158 included patients, 17,482 (14.4%) developed any Kidney Disease Improving Global Outcomes acute kidney injury, with 4,251 (3.5%) developing stage 2. The AUC (95% CI) was 0.90 (0.90-0.90) for predicting stage 2 acute kidney injury within 24 hours and 0.87 (0.87-0.87) within 48 hours. The AUC was 0.96 (0.96-0.96) for receipt of renal replacement therapy (n = 821) in the next 48 hours. Accuracy was similar across hospital settings (ICU, wards, and emergency department) and admitting serum creatinine groupings. At a probability threshold of greater than or equal to 0.022, the algorithm had a sensitivity of 84% and a specificity of 85% for stage 2 acute kidney injury and predicted the development of stage 2 a median of 41 hours (interquartile range, 12-141 hr) prior to the development of stage 2 acute kidney injury. Conclusions: Readily available electronic health record data can be used to predict impending acute kidney injury prior to changes in serum creatinine with excellent accuracy across different patient locations and admission serum creatinine. Real-time use of this model would allow early interventions for those at high risk of acute kidney injury.