Content uploaded by George Despotou
Author content
All content in this area was uploaded by George Despotou on Jun 17, 2019
Content may be subject to copyright.
Accepted Manuscript
A collaborative platform for Management of Chronic Diseases via
guideline-driven individualized care plans
Gokce B. Laleci Erturkmen, Mustafa Yuksel, Bunyamin Sarigul,
Theodoros N. Arvanitis, Pontus Lindman, Rong Chen, Lei Zhao,
Eric Sadou, Jacques Bouaud, Lamine Traore, Alper Teoman,
Sarah N. Lim Choi Keung, George Despotou, Esteban de Manuel,
Dolores Verdoy, Antonio de Blas, Nicolas Gonzalez, Mikael
Liljaj, Malte von Tottleben, Marie Beach, Christopher Marguerie,
Gunnar O. Klein, Dipak Kalra
PII: S2001-0370(18)30350-7
DOI: https://doi.org/10.1016/j.csbj.2019.06.003
Reference: CSBJ 335
To appear in: Computational and Structural Biotechnology Journal
Received date: 20 December 2018
Revised date: 18 March 2019
Accepted date: 4 June 2019
Please cite this article as: G.B. Laleci Erturkmen, M. Yuksel, B. Sarigul, et al., A
collaborative platform for Management of Chronic Diseases via guideline-driven
individualized care plans, Computational and Structural Biotechnology Journal,
https://doi.org/10.1016/j.csbj.2019.06.003
This is a PDF file of an unedited manuscript that has been accepted for publication. As
a service to our customers we are providing this early version of the manuscript. The
manuscript will undergo copyediting, typesetting, and review of the resulting proof before
it is published in its final form. Please note that during the production process errors may
be discovered which could affect the content, and all legal disclaimers that apply to the
journal pertain.
ACCEPTED MANUSCRIPT
A Collaborative Platform for Management of Chronic
Diseases via Guideline-Driven Individualized Care
Plans
Gokce B. Laleci Erturkmena,* gokce@srdc.com.tr, Mustafa Yuksela,
Bunyamin Sarigula, Theodoros N. Arvanitisb, Pontus Lindmanc, Rong
Chend,e, Lei Zhaob, Eric Sadoug, Jacques Bouaudf,g, Lamine Traoreg,
Alper Teomana, Sarah N. Lim Choi Keungb, George Despotoub, Esteban
de Manuelh, Dolores Verdoyh, Antonio de Blasi, Nicolas Gonzalezi,
Mikael Liljaj, Malte von Tottlebenk, Marie Beachl, Christopher
Margueriel, Gunnar O. Kleinm, Dipak Kalran
aSRDC Software Research Development and Consultancy Corp, Ankara, Turkey
bInstitute of Digital Healthcare, WMG, University of Warwick, Coventry, UK
cMedixine, Finland
dCambio Healthcare Systems, Sweden
eHealth Informatics Center, Karolinska Institutet, Sweden
fAP-HP, Delegation for Clinical Research and Innovation, Paris, Frances
gInserm, Sorbonne Université, Univ Paris 13, Laboratoire d'Informatique Médicale
et
d'Ingénierie des Connaissances pour la e-Santé, LIMICS, F-75011 Paris, France
hKronikgune, Research Center in Chronicity, Spain
iOsakidetza, Spain
jDepartment of Public Health and Clinical Medicine, Unit of Research, Education,
and Development, Östersund Hospital, Umeå University, Umeå, Sweden
kempirica Gesellschaft fÜr Kommunikations- und Technologieforschung mbH,
Bonn, Germany
lSouth Warwickshire NHS Foundation Trust, UK
mÖrebro University School of Business, Informatics, Örebro, Sweden
nEuropean Institute for Innovation through Health Data, Belgium
*Corresponding author.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Abstract
Older age is associated with an increased accumulation of multiple chronic conditions. The clinical
management of patients suffering from multiple chronic conditions is very complex, disconnected and
time-consuming with the traditional care settings. Integrated care is a means to address the growing
demand for improved patient experience and health outcomes of multimorbid and long-term care patients.
Care planning is a prevalent approach of integrated care, where the aim is to deliver more personalized
and targeted care creating shared care plans by clearly articulating the role of each provider and patient in
the care process. In this paper, we present a method and corresponding implementation of a semi-
automatic care plan management tool, integrated with clinical decision support services which can
seamlessly access and assess the electronic health records (EHRs) of the patient in comparison with
evidence based clinical guidelines to suggest personalized recommendations for goals and interventions to
be added to the individualized care plans. We also report the results of usability studies carried out in four
pilot sites by patients and clinicians.
Keywords:
integrated care chronic disease management clinical decision support systems evidence based clinical
guidelines
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
1. Introduction
A growing share of the population (15% in 2010) in OECD countries is over 65 and expected to reach 22
% by 2030 [1]. Older age is associated with an increased accumulation of multiple chronic conditions
(multi-morbidity), including a growing number of functional and cognitive impairments [2]. More than
half of all older people have at least three chronic conditions, and a significant proportion has five or
more [3]. Multi-morbidity creates diverse, and sometimes, contradictory needs, which challenge patients
and the delivery of health services [4]. The clinical management of patients suffering from multiple
chronic conditions is very complex, disconnected and time-consuming within the traditional care settings
and, hence, currently those with chronic conditions and long-term care needs experience shortcomings
and gaps in care provision.
Integrated care is seen as a means to transform health services to meet these challenges of 21st
century, by addressing the growing demand for improved patient experience and health outcomes of
multimorbid and long-term care patients [5]. Care planning is a central approach of integrated care, where
the aim is to deliver more personalized and targeted care creating shared care plans that map care
processes (care pathways) by clearly articulating the role of each provider and patient in the care process.
In the state of the art practices, the multidisciplinary teams (MDT) meet face-to-face to discuss and revise
the care plans of several patients at once, at regular time intervals; usually monthly. Individualized care
plans are created by manually going over the standard steps of care pathways, i.e. template care plans
which are documentation of the optimal management for typical, defined disease patterns. Although
implementation of integrated care via these manual processes is already an enhancement over traditional
fragmented care practices, we believe significant improvement can be achieved if intelligent collaborative
tools can be developed to support both the care teams and also patients and their care givers. In this paper,
we present the architecture and implementation of a coordinated care and cure delivery platform, as an
integrated care management tool encompassing clinical decision support services for MDTs and patient
empowerment tools for patients and their care givers (Figure 1).
Figure 1: The basic concepts of the C3-Cloud system
In particular, we address the following bottlenecks in traditional care delivery mechanisms,
through the automated tools and services we provide:
Traditionally, the adoption of clinical practice guidelines has been promoted for
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
managing chronic conditions. However, although CPGs may contain some basic
references, their scope is predominantly focused on single diseases, without sufficient
consideration of co-morbidity and multimorbidity. Following more than one clinical
guideline can result in inefficiencies for the patient and for the health system due to
duplicated and inconveniently scheduled investigations and clinic visits and, more
importantly, treatments that may adversely affect another condition [6, 7]. Care providers
are in need of clinical decision support services to detect and warn them about guideline
conflicts, to select upon most suitable treatment options in the light of evidence based
guidelines and to schedule and prioritize treatment activities. Our platform equips the
MDTs with intelligent services to suggest personalized goals and interventions for the
care plan of the patient based on the most recent context of the patient and evidence
based guidelines. The coordinated care and cure delivery platform enables the MDTs to
coordinate the execution and monitoring of the integrated care plans in close cooperation.
Managing multi-morbidity, through the current treatment methods, results in specialty
silos involving multiple health and social care providers who are not effectively
communicating and sharing information. As the number and complexity of health
conditions increase over time and episodes of acute illness are superimposed, the type
and number of care providers contributing to the care of individuals also increases. It
becomes significantly more difficult to align and coordinate care across care teams and
associated settings. This results in fragmented care, due to poor communication and
information sharing. Without secure information exchange among the actors involved in
health, social and informal care services and a process to reconcile potentially conflicting
treatment plans, it is impossible to avoid redundant and potentially harmful interventions.
Informed decision making also requires information to be shared between the
regional/institutional Electronic Health Records (EHRs), Social Care Records (SCRs) and
homecare services. C3-Cloud encompasses interoperability adapters that allow
heterogeneous data sources to share their EHR data securely, and an online collaboration
that offers MDT members a single coherent view (of that data). The interoperability
architecture also enables the clinical decision support services to seamlessly access and
assess the electronic health records (EHRs). In this way they can provide personalized
recommendations for goals and interventions to be added to the individualized care plans
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
of the patient.
Patients and their informal care givers including family members often do not have a
Rvoice in the management of their own care. WHO reports that adherence to long-term
therapy for chronic illnesses in developed countries averages only 50 %, and is even
lower in developing countries [8]. Adherence rates drop significantly in complex
treatment and care regimes compared to simpler ones [8]; multi-morbidity together with
the increased probability of poly-pharmacy reinforces non-adherence behaviour further
[9]. Adherence to treatment regimes (lifestyle and drugs) necessitates behaviour change
by the patients, which can be more difficult for the elderly. Patients and their informal
care givers, including family members, need to receive complete information about the
benefits and risks of treatments; and to be offered real opportunities for shared decision
making, expressing preferences and engaging in self-management. Our solution provides
a patient empowerment platform to ensure active participation of patients and their
informal care givers to the management of their multi-morbid chronic conditions. Patient
empowerment platform presents the care plan via quantitative and qualitative outcome
goals and action items to the patients. It continuosly collects feedback from the patient to
record their activities and problems they encounter, monitor risk factors through online
patient-reported outcome measures questionnaires, and establish a two way
communication between MDT members and the patients.
The paper is organized as follows: in Section 2, we briefly present existing research results in this
field and explain how we complement and extend these work. Section 3 provides a detailed description of
our architecture, elaborating the architectural choices and implementation strategies for each
subcomponent. Section 4 reports the results of the usability studies that have been conducted. Finally, in
Section 5 we present the planned future work, and elaborate on the advantages of our platform and
conclude the paper.
2. Related Work
Clinical guidelines are used in the healthcare domain to improve the quality of care [10]. It has been
demonstrated that clinical guidelines provided as real-time decision support systems improve patient care
significantly [11, 12, 13, 14] and decrease undesired practice variability [15]. Yet, the success of clinical
decision-support systems requires that they are seamlessly integrated with clinical workflows [16, 17].
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Several methodological approaches exist to implement clinical guidelines into operational practice.
Narrative guidelines can be formalized via computer interpretable guideline representation languages
such as Arden Syntax and PROforma [18]. These can be served as modular clinical decision support
(CDS) services, that can be utilized by hospital information systems during patient treatment to provide
alert and reminders about missing or contraindicating interventions (e.g., EBMeDS [19]). However, this
does not directly support healthcare professionals to follow a standardized plan of care for a specific
condition, as a clinical workflow. Clinical pathways are appropriate for that purpose; however clinical
guidelines and care pathways are often viewed as separate entities, their synergistic potential remaining
only partially exploited [20].
Clinical information systems have been built to automate care pathways to send reminders for
providers to enable periodical assessments, diagnostic tests and treatments; data collection on process and
outcome indicators for performance assessment; continuous monitoring of progress and information
sharing, examples include InformaCare, Medix [21, 22]. Yet, these do not include personalized clinical
decision support in the light of clinical guidelines.
In this paper, we present an approach to effectively integrate clinical guidelines and care
pathways: we show that it is possible to semi-automatically personalize care pathways to create
individualized care plans, by automatically processing knowledge in clinical guidelines and patient’s
EHRs. In this way, this will enable following the recommendations of clinical guidelines as a clinical
workflow executed via integrated care plans for addressing the demanding needs of patients suffering
from long-term chronic conditions.
3. C3-Cloud System Architecture
The C3-Cloud project [23] aims to change the currently fragmented medical care provided for the patients
suffering from multiple chronic conditions. It provides an ICT infrastructure for patients and multi-
disciplinary care teams, to coordinate the integrated care for the patients in a patient centered fashion.
We have implemented a Coordinated Care and Cure Delivery Platform (C3DP) that allows
collaborative creation and execution of personalised care plans for multi-morbid patients by a
multidisciplinary care team (MDT) including GPs, specialists, study nurses, pharmacists,
physiotherapists, geriatricians, nutritionists, social care and homecare workers. C3DP is the Web
application for collaborative and personalized care plan management by the members of MDT. In the C3-
Cloud architecture, C3DP sits at the top of the hierarchy and is directly integrated with all the other C3-
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Cloud components and indirectly with the local EHR/EMR systems of the pilot sites as presented in
Figure 2. All the patient data required for care planning are fetched from the C3-Cloud FHIR Repository,
which is continuously fed with existing EHR data of the pilot sites via our interoperability architecture
composed of the Technical and Semantic Interoperability Suites (TIS and SIS). With the help of Clinical
Decision Support Modules (CDSM) automating multiple clinical guidelines, C3DP processes electronic
health records of the individual patients and provides guidance to the multidisciplinary care team
members for i) risk prediction and stratification, ii) personalized selection of treatment goals and
interventions in the light of evidence based guidelines, iii) reconciliation of conflicting treatment options
and iv) management of polypharmacy. Active patient involvement and treatment adherence is achieved
through a Patient Empowerment Platform (PEP), ensuring patient needs are respected in decision making
and taking into account preferences and psychosocial aspects. Finally, the Security and Privacy Suite
(SPS) provides common security features for user authentication, authorization and audit logging to all of
the other components.
Figure 2: High Level System Architecture
In order to ensure wide adoption, we have chosen to build a standards-based architecture, where
widely accepted industry standards are chosen as building blocks of our implementation. In the following
sub-sections, we first briefly present our interoperability infrastructure, then outline the details of the
architecture sub-components which are integrated to implement an intelligent platform to support
integrated care by enabling the personalization of care pathways as care plans dynamically.
3.1. C3-Cloud Interoperability Architecture
Aiming to orchestrate the care across multiple care givers and treatment sites, and automatically process
patients’ EHRs to be able to recommend personalized treatment goals and interventions, inevitably
requires interoperability to exchange and seamlessly process medications, conditions, interventions,
episodic treatment plans, preferences and patient reported data including sensor measurements. We have
chosen to build our technical interoperability layer based on clinical resources and RESTful interfaces of
the HL7 Fast Healthcare Interoperability Resources (FHIR) STU3 standards framework [24]. The C3DP
accesses patient’s most recent EHRs, through FHIR based interfaces implemented on top of the
proprietary APIs provided by local EHR systems in our pilot sites.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Technical Interoperability Suite. The Technical Interoperability Suite (TIS) provides a standard-based
data exchange protocol, in order to enable information exchange between local care systems and C3-
Cloud components, such as C3DP. C3-Cloud is being piloted in three different pilot sites: a) Basque
Health Service - Osakidetza [OSAKI], Spain; b) Region Jamtland Harjedalen [RJH], Sweden and c)
South Warwickshire NHS Foundation Trust [SWFT], UK. The EHR API, data representation, and
operational environment vary amongst local care systems, which hinders the processing of a unified
record while creating the care plan. In order to provide maximum flexibility and extensibility, TIS is
implemented as an extract, transform and load (ETL) software development kit (SDK). TIS utilizes the
ETL model to pull patient data out of a local EHR system through its native API, convert the data into
selected FHIR resources compliant with C3-Cloud profiles with support of the Semantic Interoperability
Suite (SIS), and push the transformed FHIR data into the C3-Cloud FHIR repository. The core of TIS is
an ETL engine, which is able to schedule and execute ETL tasks. TIS also provides an extensible library
of functions, on top of which is easy to assemble an ETL task for integration with an EHR data source.
TIS provides both a web-based user interface for system administrator to execute or schedule an ETL
task, and a RESTful service API for other C3-Cloud components, such as C3DP, to trigger an immediate
ETL action, so as to get the latest patient data.
Figure 3: TIS interfaces to local EHR systems for fetching patient data
A set of pipelines have been developed for addressing the diverse needs of the three different C3-
Cloud pilot environments and the heteregeneous EHR system APIs provided. Figure 3 summarises the
patient data APIs that each pilot site exposes. The first pilot site (Basque Health Service - Osakidetza
[OSAKI], Spain) provides two separate Web services: one for exposing medical summary of patient as
HL7 CDA documents, and a second one to expose the lab results of the patients as a separate CDA
document. The second pilot site (Region Jamtland Harjedalen [RJH], Sweden) provides 6 different
RESTful services that expose patient data (patient demographics, diagnoses, lab results, medications,
notes and encounters) via proprietary JSON documents. Finally, the third pilot site (South Warwickshire
NHS Foundation Trust [SWFT], UK) provides patient data through two daily CSV exports; one from
primary care system, and the second from secondary care and community care encounters. All the
pipelines follow a similar pattern: retrieve patient data by patient identifiers; invoke SIS Structure
Mapping Service to transform the data into FHIR resources; combine all resources into a FHIR
transaction bundle; include an AuditEvent with timestamp in the transaction bundle; and commit the
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
transaction bundle into the C3-Cloud FHIR Repository. If it is the first data import, i.e. the patient has not
been created in the repository yet, TIS will add a FHIR Patient resource to the bundle, which includes C3-
Cloud study identifier and evaluation group assignment information, and notify C3DP by sending a
PatientCreated event through C3DP Event API. If an error occurs at any step of the pipeline execution,
TIS logs the error in the database and presents it via the control panel.
Semantic Interoperability Suite. The Semantic Interoperability Suite (SIS) handles both structural
mappings among different information models and resolves semantic mismatches due to the use of
different terminology systems and different compositional aggregations, as used to represent the same
clinical concept. Due to local implications of terminologies used, the SIS is developed in close relation
with the pilot sites. Two different types of mappings are performed in the semantic interoperability suite:
structural mappings and semantic mappings. Structural mappings are involved in the transformation
between local pilot sites data in local format and FHIR resources data format used in C3-Cloud. Semantic
mappings perform the transcoding between coding systems used in local sites and the C3-Cloud
components.
Figure 4: Semantic Interoperability Suite Architecture
The architecture of the SIS is provided in Figure 4. SIS is articulated around two main sub-
components: SIS Structural Mapper and SIS Semantic Mapper.
1. SIS Structural Mapper: The structural mapper of SIS is the internal SIS sub-component in
charge of the generation of FHIR resources, which have to be filled with data provided in
pilot site local format by TIS. To achieve its purpose, the structural mapper consists of
pilot site dedicated local format mappers. These mappers provide precise mappings to
create correspondence to every relevant data exported by the pilot site to its correct
interpretation and place in FHIR resource. FHIR resources, mapped from pilot site data,
are defined in the C3-Cloud data dictionary.
2. SIS Semantic Mapper: The semantic mapper of SIS is in charge of transcoding, using the
HeTOP service [25], the vocabulary used to describe data exported by pilot site into
standard codes that are used in the high-level components of C3-Cloud. A clinical
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
concept mapping sheet is being maintained as the source of truth, which includes all the
clinical concepts that are needed by the CDS services, in reference terminologies like
SNOMED-CT, LOINC and WHO ATC, and all the local codes (e.g., Spanish and
Swedish versions of ICD-10, completely local terminologies for laboratory tests) that are
used by the pilot sites for these concepts. In total, 218 common clinical concepts
including conditions, active ingredients of medications, procedures, lab results, vital
signs, immunizations and family member history have been identified and bound to well-
known terminology systems like SNOMED-CT, LOINC and ATC. It should be noted
that this list includes not only leaf-level but also high-level concepts such as
antihypertensive drugs (ATC:C02) and beta blockers (ATC:C07) or diabetes (SNOMED-
CT:73211009); hence the number of leaf-level concepts in effect is much higher. These
concepts have been mapped to 516 different codes from locally used terminology systems
of three pilot sites. These local systems are composed of localized versions of
international systems like Spanish and Swedish versions of ICD-10 and ATC in the case
of Basque Country and Region Jamtland Harjedalen, and national systems like DBP
codes for Spanish observations and READ codes for UK diagnoses. Mapping benefits
from the implicit hierarchical relationship between high-level and leaf-level concepts in
these local systems as well.
The Structural Mapper generates JSON encoded FHIR resources. The semantic mapping is based
on a pre-filled registry containing, for each concept, the corresponding code(s) for each site’s
terminology, and the code used as reference by C3-Cloud. The registry is continuously updated via a
dedicated service during the time of the project. Multiple codes can be specified for a single concept if the
used terminology has several codes corresponding to the concept (narrower-than relation). Multiple
terminologies are used as reference, in order to match each concept exactly. Both the Structural Mapper
and Semantic Mapper provide a REST API for integration with other C3-Cloud components. An example
mapping response, which is represented as an HL7 FHIR ConceptMap resource, is provided in Figure 5.
Figure 5: Semantic Mapping Example
In this example, the request is to map 44054006 in SNOMED-CT terminology that is used as a
reference for diagnoses in C3-Cloud to the terminology that is used by the Osakidetza pilot site, which is
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
ICD-10 Spanish (ICD-10-SE) in this case. A JSON-encoded FHIR ConceptMap resource is provided as a
response, describing the URIs of the input and output code systems (SNOMED-CT as the source and
ICD-10-SE as the target), the input code and the corresponding code in the target system (ICD-10-SE
code E11 for Diabetes mellitus type 2 in local language). It also shows the type of relation; equivalent in
this case, meaning that the two concepts are identical.
FHIR Repository. The C3-Cloud FHIR Repository acts as the centralized data repository for existing
clinical data of the patients and newly created care planning related data. It stores the data, which arrive
from EHR systems via TIS and newly created or updated care plan data from other C3-Cloud components
like C3DP and PEP, as HL7 FHIR resources. C3-Cloud FHIR Repository, onFHIR.io [26] is fully
compliant to FHIR STU3 specification and implemented on top of MongoDB noSQL database. An
authorized user or system can use native FHIR STU3 API, i.e Restful interfaces to
store/query/update/delete patient data. It is not possible to access any resource in the secured repository
without first acquiring a valid access token. The authorization flow is fully compliant with the Smart App
Authorization specification, which is based on the well-known OAuth 2.0 specification [27] as supported
by C3-Cloud SPS module. Thanks to C3-Cloud FHIR Repository’s automatic auditing functionality, audit
trail records are kept for each access and manipulation of data as FHIR AuditEvent resources to ensure
accountability. These audit resources are available from the same API for authorized users with
administrator roles as any other FHIR resource.
3.2 Clinical Decision Support Services
The Clinical Decision Support (CDS) services are supporting modules of the C3-Cloud Coordinated Care
and Cure Delivery Platform (C3DP). The CDS services enable the reconciliation of clinical guidelines for
individual diseases, risk stratification, poly-pharmacy management and care plan goal setting and
monitoring. As part of the C3DP platform, CDS services access, fuse and analyse patient data. This is
accomplished in order to: perform risk assessment and stratification of candidate elderly people for
inclusion in integrated care programmes; reconcile clinical guidelines for individual diseases to develop
personalised care plans; detect and propose resolutions for guideline clashes; detect duplicate,
unnecessary or contraindicating medications; and monitor and detect deviations from the outcome goals
set in a patient’s care plan. C3-Cloud focuses on elderly patients, who have at least 2 out of the following
4 chronic diseases:
Diabetes Mellitus type 2 (T2D)
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Renal Failure (RF) (excluding Glomerular filtration rate (GFR) or estimated (eGFR)<30)
Heart Failure (HF) (including New York Heart Association (NYHA) Functional
Classification I-II; excluding NYHA III-IV)
Depression (Dp) (mild/moderate conditions only)
The clinical expert group of the C3-Cloud project have examined the clinical literature, and have
identified four NICE (National Institute for Health and Care Excellence) guidelines to be followed for the
management of these four conditons as depicted in Table 1. NICE guidelines are already used in UK; the
clinical experts from Spanish and Swedish pilot sites have reviewed them and provided the required local
extensions. For example, while NICE guidelines suggest using ’Atorvastatin 20mg or 80mg’ for lipid
lowering, this recommendation has been modified as ’Simvastatin 20-40 mg’ in Basque localization.
Based on these selected NICE guidelines, CDS services have been implemented to support care planning
for the 4 specific diseases and their combinations, as listed in Table 1.
Table 1: CDS Services implemented for each disease
Diseases
Type 2 Diabetes
Renal Failure
Heart Failure
Depression
Selected
Guidelines
NG28: Type 2
diabetes in adults
[28]
CG182 Chronic
kidney disease in
adults [29]
CG108: Chronic
heart failure in
adults [30]
CG90: Depression
n adults [31]
CDS Services
Implemented
DM Blood
pressure
management
CKD referral
CHF vaccination
Depression
assessment
QRISK2 assesment
and Lipid
Management
CKD eGFR-control
frequency
CHF Stability
Review
Mild to moderate
depression
treatment
HbA1c targets
CKD CVD
prevention and
treatment
CHF Diuretics
Recommendation
Antidepressant
treatment
Blood glucose
management
CKD blood
pressure treatment
CHF
Pharmacological
Treatment
Diabetic foot
complication
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Diabetic
nephropathy
management
Diabetic neuropathy
management
Diabetic retinopathy
management
Generic
Services for
multiple
diseases
Life style
management
Diet management
Education
The clinical expert group examined these guidelines and have designed several flowcharts that
can be used as a guidance to ease the development of care plans addressing the individual needs of
patients [32]. As an example, 19 flowcharts have been designed by clinical experts covering the
recommendations of NICE Type 2 diabetes in adults: management clinical guideline (NG28) [28]. These
flowcharts constituted the basis for the CDS algorithms, which, in collaborations with the project
engineers, have been implemented as a real-time executable CDS services. The ones that can provide
computable suggestions have been identified and the inputs and possible outputs of these CDS services
have been specified as FHIR resources. An example annotated flowchart in Figure 6, depicts possible
CDS recommendations about the required lipid lowering goals and interventions. The specifications of
these CDS services have been validated once again by clinical experts. 7 different CDS services have
been implemented automating the 19 flowcharts extracted from the NG 28 guidelines. These 7 CDS
services implement a total of 80 different clinical rules, checking 108 different patient criteria, to
recommend 119 different personalized goal and intervention suggestions.
Figure 6: A sample flowchart for Lipid Managment CDS
An uncritical combination of clinical guidelines for separate diseases when treating multi-morbid
patients could have contradictions that would increase risk and in some cases even result in unfeasible
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
treatment. There is a need for reconciliation to support clinicians in decision making in risk assessment,
setting goals, choosing activities or pharmacologic treatment to include in a care plan of a multi-morbid
patient. The reconciliation exercise we followed aimed to analyze the Disease-Disease, Disease-Drug and
Drug-Disease interactions between the recommendations of given by the flowcharts designed to address
the needs of single conditions. The final aim is to reconcile relevant recommendations for different
chronic conditions by identifying the synergies, cautions and contradictions. The clinical expert groups
have carefully analyzed the flowcharts identified for individual conditions, and checked the interactions in
multi-morbid patients (patterns of double, triple or quadruple comorbidity). Potential conflicts were
identified such as repetition, wrong sequence or overlaps of activities, contradictory goals, location
inconsistencies alternative options, constraints, potential treatment synergies (either beneficial or
harmful), outliers, and inconsistencies within pairs of pathways. The next step was to reconcile these
recommendations to mitigate the conflict or inconsistency by modifiying them according to available
knowledge (select, removal, merge, substitution, modify with extra input and output). Reconciled rules
have been designed and integrated into existing flowcharts by modfying them where necessary.
Altogether, 52 reconciled rules were defined: 50 rules for the two-diseases combination , 1 rule for three-
diseases combination and 1 rule for four-diseases combination [33].
These guideline-based flowcharts and reconciliation rules are implemented as FHIR based the
CDS Hooks services [34], with decision logic encoded in the Guideline Definition Language (GDL)
version 2 [35]. GDL is a formal language used to express clinical rules and guidelines in a machine-
readable format by leveraging semantically interoperable EHR standards. GDLv2 supports FHIR and
CDS Hooks. The project uses the GDL2 Editor software to develop both the CDS guideline definitions
and CDS Hooks services (Figure 7).
To be able to intelligently propose individualized goals and activity suggestions for the selected
health concerns, C3DP is integrated with these external CDS services via CDS Hooks API. The electronic
health records of the patient retrieved from local care systems, are passed as input to CDS services after
the semantic mapping of the local codes into international code systems are handled as explained in
Section 3.1. Within CDS logic, processing these patient specific diagnoses, lab results, medication data
enables the selection of individualized goals and interventions for this specific patient. The response
consists of textual recommendations communicated as information cards and computable
recommendations communicated as suggestion cards in conformance to CDS Hooks API. In suggestion
cards, the recommended goals and activities are represented as FHIR resources (such as
MedicationRequest, Goal, Appointment resources) which can readily be included into the care plan
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
model. The finalized personalized care plan can be shared back with the local EHR Systems by exporting
the care plan as a FHIR CarePlan resource serialized as a JSON instance.
Figure 7: A screenshot of the Diabetic foot problem guideline in GDL2 editor
In addition to the guideline-based services, a RESTful service is developed to manage drug-drug
and drug-disease interactions. The drug interaction service provides warning of potential adverse
interactions between drugs, as well as a list of side-effects, for use by the rest of the C3-Cloud
architecture. The service implements the interactions between drugs, as specified by the National Institute
of Care Excellence’s implementation of the British National Formulary (BNF) [36]. BNF is a
pharmaceutical reference book, used by the UK NHS. The information provided by the service, is
identification of potential adverse interaction between drugs, the effects of the interaction, the severity of
the interaction, as well as the basis on which the interaction has been specified by the NICE-BNF. For
example, Acarbose is a drug active ingredient Alpha-glucosidase inhibitor, commonly used by patients
with type 2 diabetes, which reduces the effects of carbohydrates on blood sugar. Acarbose is listed as
having a pharmacokinetic interaction with the active ingredient Digoxin used in patients with Congestive
Heart Failure to improve quality of life and prevent hospitalisation. The interaction is listed as moderate
in criticality, having an effect as decreasing the concentration of Digoxin. In addition to interactions, the
service provides a list of side-effects for each substance, along with their frequency (i.e., common,
uncommon and rare).
The information collected by the NICE BNF is encoded as a database defining the relationships
between the main concepts e.g., interactions and drugs, and contains instances for each active ingredient.
The current database contains 108,600 instances of interactions and 26,403 instances of side-effects for
1,009 substances. The service is designed to receive a list of active ingredients of the drugs a patient may
be taking. The list is checked against the database and returns a data object containing the interaction
information as well as the list of side-effects. The information can be shown to the C3DP dashboard
notifying the MDT members of potential risks.
3.3 Patient Empowerment Platform
The objective of the Patient Empowerment Platform (PEP) is to provide patients with access to the
published care plan and its information and thus increase patient and informal caregiver participation in
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
decision making [37, 38]. PEP aims to provide computerised means to improve the interaction between
patients and health professionals and to digitally collect relevant data and information to enable
monitoring of care plan related activity status and progress. It directly interacts with C3DP to receive new
and updated care plans, and to send back patient reported observations. It allows MDT members to assign
questionnaires to patients to collect patient reported outcome measures, which are later filled in by the
patient via PEP interfaces and shared back with C3DP through standard based interfaces based on HL7
FHIR. It, also, directly communicates with the supported set of sensor devices to record patient
measurements. The core user functionalities and features provided to PEP users are:
Make published care plans available to the users.
Send reminders to patients to help them comply and stay on track with the interventions
and activities included in the care plan.
Allow patients to actively collect data related to the care plan activities.
Allow health professionals and patients to communicate with each other using either
messages or video appointments.
Provide patients with access to relevant self-management material.
Allow patients to provide feedback to MDT members about the care plan activities (such
as reporting probable side effects of medications)
Figure 8: Patient Empowerment Platform - Care Plan Details Screen
The PEP user application is a modern web application, which allows PEP Users (patients and
their informal caregivers) to access all the functionality via web browsers (see Figure 8). PEP is built on
top of the Medixine Suite product. The Medixine Suite technology stack follows traditional logic for web-
based services and consists of an OS (Microsoft Windows Server), data storage (Microsoft SQL Server),
Web Server (IIS) and Programming platform (.NET). The database layer contains some supporting
functionalities, but the main business logic is built into the application layer. The application logic is
based on modern resource-based thinking and is accessed through a REST API that is structured around
those resources. The core logic includes role-based access configuration for different operations, full audit
trails of operations performed in the system, an application ecosystem model, dynamic and extensible
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
data modelling tools, event subscription model for integrations and extensible support for multiple
languages and cultures.
3.4. Security and Privacy Suite
The Security and Privacy Suite (SPS) is responsible for authentication and authorisation of the care team
members, while they are managing personalised care plans of patients and ensuring that all data
exchanged within and across C3-Cloud software components is encrypted and properly auditable. In the
C3-Cloud architecture, the patient’s electronic health records received from the local EHR systems via the
TIS, patient reported observations from the PEP, and the care plan of the patient managed through C3DP,
are all managed in the C3-Cloud FHIR Repository. Hence, each of these client apps, i.e. TIS, PEP and
C3DP needs to be authenticated and authorized to access (read, write, and update) patient data to the C3-
Cloud FHIR Repository, via the functionalities provided by SPS. All such operations need to be logged
for ensuring accountability via SPS. SPS enables authentication of the care team members into the C3-
Cloud applications in two ways: i) via their already existing accounts (e.g., username-password) provided
by the local authorities by integrating with the existing Identity Provider (IdP) systems of the pilot sites;
and ii) by creating C3-Cloud specific user accounts for those users whose IdP’s cannot be integrated with
the SPS, for example, the social care workers. The SPS has three sub-components:
C3-Cloud SPS Server provides services for user registration, privacy policy management
and endpoints defined in the OpenID Connect 1.0 standard to perform authentication and
authorization (Authorization Endpoint, Token Endpoint, etc.). By implementing the
OpenID Connect API, it serves C3-Cloud Identity Provider (IdP), which is the default
IdP when the IdP of some users (e.g., social care workers) of the pilot sites cannot be
integrated within the scope of the project. The SPS Server also manages the C3-Cloud
Access Control Policy Store.
C3-Cloud SPS Manager is a web application for representing the functionalities of C3-
Cloud SPS Server with the following user interfaces: single sign on UIs, policy
management UI, client registration UI, user registration UI and audit viewer UI.
Audit Record Repository is a FHIR repository that maintains audit trail records
implemented as FHIR AuditEvent resource. In C3-Cloud architecture, the C3-Cloud
FHIR Repository is used as the Audit Record Repository. An extra instance of the same
repository is not created for practical reasons.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
3.5. Coordinated Care and Cure Delivery Platform (C3DP)
As depicted in Figure 9, the Coordinated Care and Cure Delivery Platform (C3DP) is implemented as a
Web application for collaborative and personalized care plan management by the members of a
multidisciplinary team of care (MDT). All the patient data required for care planning are fetched from the
C3-Cloud FHIR Repository, which is continuously updated with EHR data from the pilot sites via TIS
and SIS. C3DP visualizes these data and helps the health professionals to easily manage the integrated
care coordination process for multi-morbid elderly patients. C3DP implements the HL7 Care Plan DAM,
and enables health professionals to design a care plan for a patient from scratch by selecting health
concerns to be addressed from the EHR of the patient, and setting goals and activities to address the needs
of this health concern. This process is formalized as a FHIR Care Plan resource, which consists of
building blocks like Goal and different types of Activity resources (Figure 10).
Figure 9: Care Plan Summary Screen of C3DP
Figure 10: Building blocks of a care plan
The major functionalities enabled by the C3DP are:
Review of medical summary. C3DP provides a mechanism to review a complete view of patient’s medical
record. All the patient data that are provided by the local EHR/EMR systems, and also by the patient via
Patient Empowerment Platform (PEP), including conditions, medications, allergies, lab results, vital
signs, procedures and social history are presented to the care team members in a single location.
Preparation of an individualised care plan based on evidence based clinical guidelines. In addition to the
addressed health concerns and risk factors, an integrated care plan is mainly composed of goals to manage
the health concerns and the activities (i.e., interventions) to achieve the identified goals and improve the
associated health concerns. In C3-Cloud, education materials for the empowerment of the patients are also
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
part of the care plans, while they are treated separately from the rest of the activities. There are three ways
of adding a goal / activity / education material to a care plan:
Manual entry from scratch: Care team members can, at any time, create a goal / activity /
education material themselves via the C3DP user interface.
Recommendations from the CDS services: Personalised goal, activity and education
material suggestions, provided by the CDS services according to patient data can be
directly added to the care plan of a patient by the care team members, or after some
modifications. Figure 11 depicts a snapshot where personalized suggestions based on
CDS recommendations are presented to MDT members.
Transfer from the older care plan: When provided by the local systems of the pilot sites,
it is possible to transfer existing goals and activities from a treatment plan of a patient
into an integrated care plan during its initialisation.
Figure 11: A snapshot from C3DP presenting personalized activity suggestions
Cross-check of all patient data that are needed as input by the CDS services. The CDS services process
patient EHR data to recommend personalized goal and interventions. However the missing or incomplete
data can affect the correctness of a CDS recommendation. In order to address this challenge, all clinical
concepts that may affect the CDS recommendation and result in adverse events are presented to the care
team members along with contextual information that will help the user flag potential errors. For
example, the lipid management CDS service checks the existence of three diseases (and some other
parameters such as lab results and specific medications) in its decision tree: type 2 diabetes, chronic
kidney disease and cardiovascular disease. The patient records retrieved from the local EHR system show
that the patient has type 2 diabetes, but there is no information about the existence of chronic kidney
disease and cardiovascular disease. The GP of the patient can declare that this patient has also a
cardiovascular disease, which was somehow missing in the patient’s EHR system records. Such newly
provided patient data, via the C3DP interface, can be provided back to the original EHR/EMR systems.
Execution of a care plan. Integrated care planning is a continuous process. Ideally, an integrated care plan
lives with the patient and is adjusted to the most recent patient context. It is updated during planned and
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
unplanned encounters of the patient with health professionals and social care workers, and also with
patient provided feedback via the Patient Empowerment Platform. All updates can be shared with the
local EHR/EMR systems as well. Hence, execution of a care plan refers to the continuous follow-up and
update of an integrated care plan. This can happen in a number of ways in C3-Cloud:
Updating the progress of goals and activities: The status of any goal or activity can be
updated (e.g., a goal can be set as achieved or on-target) by a care team member. The
patient can also provide feedback on their progress.
Re-execution of CDS services during planned and unplanned encounters: This is akin to
the CDS service usage for the first time during initialization of a care plan. Relevant
progress in the patient status is reflected in the recommendations of the CDS services.
Display of patient provided data: Patient and his informal care giver are active
participants of the care planning process. Goals and activities are decided with his active
involvement, and for an activity that is assigned to themselves, the patient is able to
provide update via the Patient Empowerment Platform (PEP). Patient provided data
includes questionnaire responses, medical device measurements (e.g., blood glucose,
blood pressure), daily meal photographs and more. All patient provided data are matched
with the corresponding care plan items and shown to the care team members.
Commenting on the care plan items: It is also possible to comment on specific goals and
activities of a care plan, which are visible to the care team members.
Management of the care team. It is possible to invite new care team members to a care plan, during
initialization or at any time. An invitation is subject to the confirmation of the invited care team member,
who is informed via a notification in the system and an email depending on the preference of the pilot
sites. The care team manager, who is always the GP of the patient in all 3 pilot sites of C3-Cloud, can also
remove a professional from a care team, or a member may want to leave a care team. It is also possible
for a health professional or social care worker to request joining an existing care team for a specific
patient. Different roles can have different rights in the care team; for example, a nurse assistant or a social
care worker can see a care plan but not modify it.
Communication among care team members and with the patient / informal care giver. C3DP has its own
messaging module that enables safe messaging among all care team members, and also with the patients
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
due to the integration between C3DP and PEP. HL7 FHIR Communication resource is used for
messaging.
Dashboard view. Dashboard view enables a signed in care team member to quickly go over the important
updates in the care plans of all her patients since the previous login, such as new messages received,
awaiting appointments, new system notifications.
Patient provided data screen. This view collates all the patient provided data such as vital sign
measurements, meal photos, feedback on the care plan and messages to the care team members in a single
dashboard.
Activity calendar. It enables view and update of scheduled activities of a care team member on a calendar.
Real-time system notifications. Real-time system notifications are implemented for several events (e.g.,
for care plan update, new patient feedback, new message, invitation to a care team, etc.). When the user is
already logged in to the system, such notifications are displayed in real time. It is also possible to access
care team members via email for offline scenarios. SMS option was dropped by the pilot sites for real-
time clinical notifications.
4. Usability Studies
To ensure an iterative and holistic approach, the evaluation and impact assessment of the C3-Cloud
project has been split into four layers in accordance with the different stakeholder groups it effects and the
different stages of development and deployment:
Evaluation layer 1 targeted C3-Cloud software component and application tests along
defined protocols by making use of 5 health ICT experts from the University of Warwick,
26 patients and 22 MDT members in the pilot sites.
Evaluation layer 2 included a heuristic evaluation, a Nielsen walkthrough [?] and a
questionnaire, in preparation for layer 3 evaluation. Layer 2 involves 5 health ICT experts
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
from the University of Warwick, 27 patients and 20 MDT members in the 3 pilot sites.
Evaluation layer 3 will employ an exploratory technology trial that uses baseline and
closure patient observations. Approximately 150 intervention patients and 52 MDT
members will be involved in layer 3 evaluations by answering questionnaires and being
involved in interviews.
Evaluation layer 4 will employ a predictive modelling tool to model the C3-Cloud impact
when scaled up, using intervention and control patient data. Approximately 526
intervention patients and 62 members of the multidisciplinary teams will be involved in
layer 4 evaluation by answering questionnaires and giving access to their anonymized
EHR data. In addition, the data of 526 control patients will be used for data analysis.
Among these, we have completed the execution of first two layers, and evaluation layers 3 and 4
will be initiated in early 2019.
We aimed to ensure continuous, open feedback loops to the development team in evaluation
layers 1 and 2 for the software improvement. In the first evaluation layer, we first carried out tests
specified based on functional requirements of the software and performed application testing before
deployment to ensure that all software components work well together. Results of this study is reported in
[39]. In this section we will focus on reporting the usability studies realized in evaluation layer 2.
Usability testing is a process of tracking real users via a carefully designed protocol to test a
system before and during deployment. This is useful to avoid technology-induced errors; identify issues
and validate and improve the performance of a final product [40]. Kushniruk and colleagues argue that
both commercial vendor based testing and in-situ testing are needed to ensure system usability [41].
The research objective of the presented approach is to identify and categorize early usability
issues of the C3-Cloud components that are used by MDT members and patients. This objective served to
answer the following research questions in accordance with the C3-Cloud research protocol [42].
How usable is the C3-Cloud application perceived by experts, patients and MDT
members?
What usability issues can be identified that must be improved before deployment of the
solution?
The following four methods were selected for our usability testing approach:
Method 1: A heuristic evaluation with health IT experts following the Nielsen
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
walkthrough (Health ICT Experts). Heuristic evaluation is a usability engineering method
which was comprehensively discussed by Jakob Nielsen in 1994 in his book, “Usability
inspection methods” [?]. According to Nielsen, it is “a usability engineering method for
finding the usability problems in a user interface design so that they can be attended to as
part of an iterative design process”. In Nielsen walkthrough, multiple evaluators are
involved (Nielsen recommends three to five) and the users are asked to discover the
answer to given questions by using the system several times (at least twice) according to
the storyboard.
Method 2: Spontaneous feedback gathering during the test sessions with MDT members
and patients separetely (Patients and MDT members)
Method 3: Product reaction cards (Patients and MDT members)
Method 4: The QUIS7 questionnaire on user interaction satisfaction (Patients and MDT
members).
Our usability testing approach is user-centric. The test sessions involved members of the MDT
and patients from the target group spanning three pilot sites in the Basque Country (Spain), Region
Jämtland Härjedalen (Sweden) and South Warwickshire (UK). Health ICT experts were recruited
from the University of Warwick. All MDT members and patients were recruited among people in the
prospective group of software users who speak English. For method 1, the health ICT experts had the
chance to contact technical partners for any questions at any time. For methods 2-4 a language facilitator
from each pilot site moderated each session and was available for any question that was raised from the
participants. All participants received an introduction and a brief overview on the systems to clarify the
test session objectives. The technical project partners developed a walkthrough for both the C3DP and the
PEP. This walkthrough informed test participants about how to access the C3DP and the PEP and listed
test user credentials for all participants. All participants received their own login credentials for the online
High Level Component (HLC) demonstrators of the C3DP and the PEP. This was followed by step-by-
step descriptions activities to be performed by all testers. Technical partners had ensured that all possible
functionalities of the software were covered by the activities that the test participants followed. All test
participants had followed these procedures for a two hour session.
The number of participants attended the test sessions of layer 2 evaluation are depicted in Table
2.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Pilot Sites/ Participant Profiles
Health ICT Experts
Patients
MDT members
University of Warwick
5
-
-
South Warwickshire (SWFT)
-
13
12
Basque Country (BC)
-
2
6
Region Jämtland Härjedalen (RJH)
-
12
2
Total
5
27
20
Table 2: Number of participants for the usability testing
Method 1- heuristic evaluation. Heuristic evaluation (HE) focuses on the interface of the system, in the
C3-Cloud case the PEP and C3DP interfaces for patients and healthcare professionals respectively. It is
performed by individual reviewers isolated from each other, and at the end of the process, results are
collated and fed back to the developers. HE is a process that is part of the iterative development process
of a system. HE can reveal a number of issues about the system. Examples of these include bad design
that may lead the user making a slip or mistake, as well as design that may be seen to not appreciate the
sensitivities of the user (e.g., system dialogues). All these issues are classified in a number of categories,
which are the heuristic categories. Thirteen heuristics were considered in the C3-Cloud HE, including
Visibility of system status, User controls and freedom, consistency and standards, Error recovery.
Five specialists, usability evaluation reviewers from the University of Warwick performed the
heuristic evaluation, consisting of the following steps: (i) Reviewers attended a 30-minute session where
the purpose of the evaluation, process and documentation were explained; (ii) Reviewers reviewed the
PEP and C3DP manuals and walkthrough descriptions; (iii) Reviewers made a first structure-free
evaluation of the interfaces; (iv) A second structured pass was done following the workflows in the
manuals and comments were classified under each heuristic; (v) Based on the comments, reviewers
completed a spreadsheet with common issues for each heuristic category, frequency and severity were
combined to create an overall risk matrix that will prioritise modifications by the technical teams.
The results for C3DP and PEP are shown in Table 3, where table presents the distribution of % of
usability errors, in each heuristic.
Usability issues
Distribution of
issues for C3DP
(total=196)
Distribution of
issues for PEP
(total=67
1. Visibility of System Status
13%
10%
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
2. Match Between the System and the Real World
19%
13%
3. User Control and Freedom
7%
7%
4. Consistency and Standards
20%
10%
5. Help Users Reconnize, Diagnose, and Recover from
Errors
6%
7%
6. Error Prevention
9%
0%
7. Recognition Rather than Recall
3%
10%
8. Flexibility and Efficienct of Use
2%
3%
9. Aesthetic and Minimalist Design
2%
0%
10. Help and Documentation
4%
10%
11. Skills
3%
7%
12. Pleasurable and Respectful Interaction with the User
4%
9%
13. Privacy
4%
0%
14. Accessibility
5%
10%
Table 3: Summary of overall usability issues
Method 2- spontaneous feedback. The objectives of the test sessions were explained to all MDT members
and patient participants and they were given an introduction in the softwares to be tested. Subsequently,
they followed the activities shown in a two hour session. Any spontaneous feedback that was given
during the test sessions in May 2018 was recorded and reported by the session moderator. Feedback was
clustered for general feedback on the HLCs and feedback on specific functionalities (usability; care plan
goals; care plan activities; terminology). The number of clustered comments recorded from the
spontaneous feedback can be seen in Table 4. Duplicated feedback was not reported. Feedback was
supported with screenshots when needed or useful. The software developers studied and prioritised the
feedback using an internal issue tracking system. Prioritization was done for bugs, improvements, features
and cosmetic changes. In total, 101 comments on the C3DP and 44 comments on the PEP platform were
obtained by recording the spontaneous feedback of test participants. Testers were generally very positive
about the C3-Cloud concept and experienced it as very promising, helpful and easy to use. MDT members
have particularly liked the clinical focus of the platforms. The fact that the C3DP suggests goals and
activities was experienced as being very positive and helpful. The software developers translated relevant
issues into a tracking tool and collaborated with the projects’ clinical reference group and the pilot sites to
resolve open issues. Feedback responses were incorporated in respective activities of software
development, software deployment at the pilot sites and training plans.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Category
C3DP Platform
PEP Platform
General feedback
11 comments
9 comments
Usability feedback
58 comments
28 comments
Care plan goals
5 comments
-
Care plan activities
17 comments
-
System terminology
10 comments
7 comments
Table 4: Spontaneous feedback
Figure 12: Set of 118 words for the “product reaction cards” [43]
Method 3- Product reaction cards. The “product reaction cards”
1
is a fast and simple method used for an
overall system evaluation. It allows the user to describe the system from a predefined set of 118 words
(Figure 12). This list includes positive words, together with negatives and neutral words. The main
advantage of this approach is that it does not rely on a questionnaire or rating scales and users do not have
to generate words themselves [44]. For the “product reaction cards” method, 48 people (22 MDTs and 26
Patients) from the three pilot sites: Osakidetza, RJH and SWFT; participated to the evaluation. The
participants received login credentials for the online demonstrators of the C3DP (for MDTs) and the PEP
(for Patients/ICGs) as well as training materials including a walkthrough that guided them through certain
activities on the demonstrators The participants were asked to pick the words that best describe the C3-
Cloud platform or how using the product made them feel. We limit the choice number to 5 words, as
commonly used in such approach. At the end of the study, a scoring was made to identify the most
commonly words by the participants to describe the system.
For the 22 participants of the MDT profile, 30% describe the system as "Collaborative", 16% find
it "Comprehensive”, 17% find it both “Empowering” and “Innovative" and 20% as "Time Consuming”.
For the 26 participants of the patient profile, 25% describe the system as "Useful", 21% find it both
"Accessible and Convenient" and 17% find it Appealing and 16% find Advanced". In general, for all the
48 participants, the C3-Cloud system appears to be "Collaborative" at 23%, "Useful and Empowering" at
1
The “product reaction cards” were developed by Microsoft as part of a “desirability toolkit” created to
understand the illusive, intangible aspect of desirability resulting from a user’s experience with a product.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
17%, "Innovative" at 15% and both "Complex" and "Comprehensive" at 14%. In this study, the “product
reaction cards” method gives a good understanding of the user’s experience. This method is easy and
quick to conduct and permits to get user’s experience with the system. When we collate the responses
from all participants, results we obtained show that users pick either same card or a closely related card.
Overall, this “product reaction cards” method is an early evaluation exercise. It permited to get first users’
feedback and feelings. This helped the technical partners/component owners to further improve the C3-
Cloud design, application and its implementation to better align with what the user’s expectation.
Method 4- QUIS7. The Questionnaire for User Interaction Satisfaction (QUIS, 7th version) is a tool that
measures attitude towards software interface factors: screen factors, terminology and system feedback,
learning factors, system capabilities, technical manuals, on-line tutorials, multimedia, voice recognition,
virtual environments, internet access and software installation. Respondends are asked to rate one or more
questions for these categories on a 0-9 scale. The original QUIS7 questionnaire items were adapted to
cater for the requirements of the software. Specific items were omitted as they were considered not useful
for the evaluation at this stage of the project. In addition to the standard English and Spanish versions, the
questionnaire was translated to Swedish for use at the RJH deployment site. After the test session
participants finished the walkthrough, they were asked to fill in the QUIS7 online questionnaire
anonymously. This early usability testing was performed with both MDT members for the C3DP platform
and patients for the PEP platform. The results of the QUIS7 questionnaire are used for shaping the design
and redesign of the platforms, detecting areas for usability improvement, and the comparative evaluation
of the platform from its current status and later during the technology trial. While the QUIS7 follows a
clear structure and helps identifying areas for improvement, it lacks detail and reasoning when certain
aspects were rated less positive. Thus, method 2 (spontaneous feedback) complements this method for
more specific insight.
Figure 13: Learning (C3DP)
For each QUIS7 category the mean rating per question was derived (see Figure 13). In addition
the mean rating, the standard deviation (STD), the distribution of ratings on a bar chart and a pie chart
were derived per question for further detail (see Figure 14). The bar-chart displayed only valid responses,
while the pie chart shows all responses, including the percentage of non-responders. The bar-charts and
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
pie-charts are colour coded orange for values from 0-4 and green for values from 5-9. The total number of
valid patient responses for the C3DP usability testing is n=20, all items that were rated lower than 6 in the
mean; all items that had a STD larger than 2; all items that had a non-response rate of more than 15%.
The total number of valid patient responses for the PEP usability testing is n=26, all items that were rated
lower than 5.9 in the mean; all items that had a STD larger than 2.3; all items that had a non-response rate
of more than 20%.
Figure 14: Getting started is (difficult – easy) (C3DP)
Results. Usability studies held with MDT members and patients have shown that the proposed method is
able to address the needs of care plan personalization via CDS services implementing clinical guidelines.
Testers were generally very positive about the C3-Cloud concept and experienced it as very promising,
helpful and easy to use. They have particularly liked how the tools were very clinically focused. The fact
that C3DP suggests goals and activities was experienced as being very positive and helpful. Especially the
two front-end facing components, C3DP and PEP, have benefited from feedback for improvements
during the usability and application testing. Unstructured feedback, expressed through the think-aloud
method by the test participants, has been particularly useful. Development teams have responded to the
feedback received and incorporated them in relevant tasks, such as development, deployment and
training.
5. Discussion and Conclusions
The C3-Cloud system has been co-designed and co-produced with the end-users from the very early
stages of the project. In order to facilitate co-production, technical partners have first created user
interface mock-ups (especially for C3DP) during the requirements analysis phase, and then created
producing early prototypes starting from the architectural design phase, followed by several iterations and
reviews by the end-users till the end of integration. This process has helped a lot in achieving easy
adaptation and better acceptance of the end-users to the C3-Cloud solution. Usability studies held with the
clinical experts with simulated patient data in evaluation layers 1 and 2 have shown that the proposed
method is able to address the needs of care plan personalization via CDS services implementing clinical
guidelines. Moreover, the recommendations from these usability studies have helped to improve further
the user-facing components C3DP and PEP.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
As the next step, the system will be operated and validated in real life within the scope of
evaluation layers 3 and 4 to examine usability and acceptance of personalized care plans for chronic
disease management in three pilot sites: Basque Country (Spain), Region of Jämtland Härjedalen
(Sweden) and South Warwickshire NHS Foundation Trust (UK) via a 12 months pilot study to be carried
out with approximately 62 health and social care workers including general practitioners, nurses and
specialists and 526 patients using the C3-Cloud solution. The integrated solution has already been
deployed to staging environments of all three pilot sites. Soon the final training activities with the actual
users will take place and then pilots in operational environments will be started where patients will be
directly involved.
5.1. Limitations, Challenges and Future Work
C3-Cloud is a very complex project with ambitious aims and developing a common solution for varying
settings, restrictions and needs of three different pilot sites has been quite a challenging task. We have
overcome these challenges by working closely with the end-users and their local IT teams, and loosely
coupling the C3-Cloud end-user facing components with the local site EHR/EMR systems via a
standards-based interoperability layer composed of an HL7 FHIR Repository and pilot site specific
adapters transforming the data in local formats into FHIR resources compliant with the C3-Cloud profiles.
It is a fact that, the interoperability layer requires manual activities especially for semantic mapping.
Currently, the semantic interoperability challenges have been effectively dealt with by focusing on the
specific data requirements of the targeted chronic diseases and associated CDS services implementing
clinical guidelines, and validating all the relevant but limited set of terminology code mappings by
clinical experts.
In addition to this, the reconciliation of the recommendations from multiple clinical guidelines
had been a manual work that has been carried out by our clinical expert groups. As a future work we aim
to explore semantic reasoning tools to semi-automatically detect possible clashes between the
recommendations coming from different clinical guidelines.
Finally, although the data flow and transformation from the local EHR/EMR systems into the C3-
Cloud solution is achieved completely, and the personalised care plans as the outcome of the C3-Cloud
solution can be provided back to the local systems in a widely-used international standard (HL7 FHIR), it
could not be possible to integrate the care plan goals and activities back to the local systems in the local
formats due to legal and technical constraints. As a long term future work, this will be enabled for better
exploitation of the system to third parties.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
Acknowledgements
The research leading to these results has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 6891810, C3-Cloud Project.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
References
[1] OECD, Oecd factbook (edition 2009). URL https://www.oecd-ilibrary.org/content/data/data-
00377-en
[2] N. Goodwin, A. Dixon, G. Anderson, W. Wodchis, Providing integrated care for older people with
complex needs: Lessons from seven international case studies, Report, The Kings Fund (2014).
[3] F. Luppi, F. Franco, B. B. L. Fabbri, Treatment of chronic obstructive pulmonary disease and its
comorbidities, in: Proceedings of the American Thoracic Society, Vol. 5, 2008, p. 26.
[4] J. Piette, C. Richardson, M. Valenstein, Addressing the needs of patients with multiple chronic
illnesses: the case of diabetes and depression, Am J Manage Care 10 (2004) 152–62.
[5] Integrated care models: an overview, Working document, Health Services Delivery Programme,
Division of Health Systems and Public Health, WHO Europe (2016).
[6] B. Guthrie, K. Payne, P. Alderson, M. E. T. McMurdo, S. W. Mercer, Adapting clinical guidelines
to take account of multimorbidity, BMJ 345.
arXiv:https://www.bmj.com/content/345/bmj.e6341.full.pdf, doi:10.1136/bmj.e6341. URL
https://www.bmj.com/content/345/bmj.e6341
[7] L. Hughes, M. E T McMurdo, B. Guthrie, Guidelines for people not for diseases: The challenges of
applying uk clinical guidelines to people with multimorbidity, Age and ageing 42.
doi:10.1093/ageing/afs100.
[8] Adherence to long term strategies-evidence from action, Working document, Chronic diseases and
health promotion, WHO (2003).
[9] L. Osterberg, T. Blaschke, Adherence to medication, New England Journal of Medicine 353 (5)
(2005) 487–497, pMID: 16079372. doi:10.1056/NEJMra050100.
[10] K. Vanhaecht, The impact of clinical pathways on the organisation of care processes, Phd thesis,
University Leuven, Faculty of Medicine (2007).
[11] K. . Kawamoto, C. Houlihan, E. Balas, D. Lobach, Improving clinical practice using clinical
decision support systems: a systematic review of trials to identify features critical to success, BMJ
330 (2005) 765.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
[12] M. Jaspers, M. Smeulers, H. Vermeulen, L. Peute, Effects of clinical decision-support systems on
practitioner performance and patient outcomes: A synthesis of high-quality systematic review
findings, Journal of the American Medical Informatics Association : JAMIA 18 (2011) 327–34.
doi:10.1136/amiajnl-2011-000094.
[13] N. M. Souza, R. J. Sebaldt, J. A. Mackay, J. C. Prorok, L. Weise-Kelly, T. Navarro, N. L.
Wilczynski, R. B. Haynes, the CCDSS Systematic Review Team, Computerized clinical decision
support systems for primary preventive care: A decision-maker-researcher partnership systematic
review of effects on process of care and patient outcomes, Implementation Science 6 (1) (2011) 87.
doi:10.1186/1748-5908-6-87. URL https://doi.org/10.1186/1748-5908-6-87
[14] J. Grimshaw, M. P Eccles, J. Lavis, S. Hill, J. E Squires, Knowledge translation of research
findings, Implementation science : IS 7 (2012) 50. doi:10.1186/1748-5908-7-50.
[15] R. Lenz, R. Blaser, M. Beyer, O. Heger, C. Biber, M. Bäumlein, M. Schnabel, It support for
clinical pathways—lessons learned, International Journal of Medical Informatics 76 (2007) S397 –
S402, ubiquity: Technologies for Better Health in Aging Societies - MIE 2006.
doi:https://doi.org/10.1016/j.ijmedinf.2007.04.012.
[16] S. W. Tu, M. A. Musen, R. Shankar, J. Campbell, K. Hrabak, J. McClay, S. M. Huff, R. McClure,
C. Parker, R. Rocha, R. Abarbanel, N. Beard, J. Glasgow, G. Mansfield, P. Ram, Q. Ye4, E. Mays,
T. Weida, C. G. Chute, K. McDonald, D. Mohr, M. A. Nyman, S. Scheitel, H. Solbrig, D. A. Zill,
M. K. Goldstein, Modeling guidelines for integration into clinical workflow, Stud Health Technol
Inform 107 (2004) 174–178.
[17] M. Peleg, Computer-interpretable clinical guidelines: a methodological review., Journal of
biomedical informatics 10. doi:10.1016/j.jbi.2013.06.009.
[18] K. Heiden, B. Böckmann, Generating evidence-based pathway models for different hospital
information systems, in: G. Huang, X. Liu, J. He, F. Klawonn, G. Yao (Eds.), Health Information
Science, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 42–52.
[19] Evidence-Based Medicine Electronic Decision Support Study (EBMeDS) (cited December 2018).
URL https://clinicaltrials.gov/ct2/show/NCT00915304
[20] M. W Beckmann, H. Schlieter, P. Richter, S. Wesselmann, Considerations on the improved
integration of medical guidelines into routine clinical practice - a review and concept proposal,
Geburtshilfe und Frauenheilkunde 76 (2016) 369–376. doi:10.1055/s-0042-102056.
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
[21] W. Ricciardi, A. de Belvis AG, M. Specchia, ’Assessing Chronic Disease Management in
European Health Systems: Country reports, Observatory Studies Series 39, European Observatory
on Health Systems and Policies, Copenhagen (Denmark) (2015). URL
https://www.ncbi.nlm.nih.gov/books/NBK458735/
[22] Medix Care Management Tool, . http://www.medixteam.com/caremanagement/ (2018).
[23] C3-Cloud: A Federated Collaborative Care Cure Cloud Architecture for Addressing the Needs of
Multi-morbidity and Managing Poly-pharmacy, . http://c3-cloud.eu/ (2016).
[24] D. Bender, K. Sartipi, Hl7 FHIR: An agile and restful approach to healthcare information
exchange, in: Proceedings of the 26th IEEE International Symposium on Computer-Based Medical
Systems, 2013, pp. 326–331. doi:10.1109/CBMS.2013.6627810.
[25] J. Grosjean, T. Merabti, B. Dahamna, I. Kergourlay, B. Thirion, L. Soualmia, S. Darmoni, Health
multi-terminology portal: a semantic added-value for patient safety, Studies in health technology
and informatics 166 (2011) 129–38.
[26] onFHIR: HL7 FHIR® Based Secure Data Repository, . https://onfhir.io// (2016).
[27] D. Hardt, The OAuth 2.0 authorization framework, RFC 6749, RFC Editor, Fremont, CA, USA
(Oct. 2012). URL http://www.rfc-editor.org/rfc/rfc6749.txt
[28] NG28 Type 2 diabetes in adults: management, Clinical guideline, NICE (May 2017). URL
https://www.nice.org.uk/guidance/ng28
[29] CG182 Chronic kidney disease in adults: assessment and management, Clinical guideline, NICE
(2015). URL https://www.nice.org.uk/guidance/cg182
[30] CG108 Chronic heart failure in adults: management, Clinical guideline, NICE (2010). URL
https://www.nice.org.uk/guidance/cg108
[31] CG90 Depression in adults: recognition and management, Clinical guideline, NICE (2018). URL
https://www.nice.org.uk/guidance/cg90
[32] D7.1 evidence based clinical guideline definitions and flowcharts for individual chronic conditions,
Deliverable, C3-Cloud Consortium (2016). URL http://c3-cloud.eu/c3/deliverables
[33] D7.2 clinical decision support modules for personalised care plan development and execution,
Deliverable, C3-Cloud Consortium (2017).
[34] CDS Hooks, Technical specifications (2018). URL https://cds-hooks.org/specification/1.0/
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
[35] Guideline Definition Language v2, Technical specifications (2018). URL https://github.com/gdl-
lang/gdl2
[36] British National Formulary (BNF) (cited December 2018). URL https://www.bnf.org/
[37] D5.2 data collection and feedback mechanism, Deliverable, C3-Cloud Consortium (2017).
[38] D5.3 responsive multi-channel patient empowerment platform, Deliverable, C3-Cloud Consortium
(2017).
[39] L. Traore, A. Assele-Kama, S. N. L. C. Keung, L. Karnic, G. O. Klein, M. Liljad, I. Scandurra, D.
Verdoy, M. Yuksel, T. N. Arvanitis, R. Tsopra, M.-C. Jaulent, User-centered design of the c3-
cloud platform for elderly with multiple diseases – functional requirements and application
testing, in: Accepted for publication to MEDINFO 2019, 2019.
[40] S. Weiss, Handheld Usability, John Wiley & Sons, Inc., New York, NY, USA, 2002.
[41] A. Kushniruk, E. M Borycki, J. Kannry, Commercial versus in-situ usability testing of healthcare
information systems: Towards "public" usability testing in healthcare organizations., Studies in
health technology and informatics 183 (2013) 157–61. doi:10.3233/978-1-61499-203-5-157.
[42] C3-cloud research protocol v.3.0, Working document, C3-Cloud Consortium (2018).
[43] J. Benedek, T. Miner, Measuring desirability: New methods for evaluating desirability in a
usability lab setting.
[44] T. Tullis, W. Albert, Measuring the User Experience: Collecting, Analyzing, and Presenting
Usability Metrics: Second Edition, 2008.
ACCEPTED MANUSCRIPT
