ArticlePDF Available

A Collaborative Platform for Management of Chronic Diseases via Guideline-Driven Individualized Care Plans


Abstract and Figures

Older age is associated with an increased accumulation of multiple chronic conditions. The clinical management of patients suffering from multiple chronic conditions is very complex, disconnected and time-consuming with the traditional care settings. Integrated care is a means to address the growing demand for improved patient experience and health outcomes of multimorbid and long-term care patients. Care planning is a prevalent approach of integrated care, where the aim is to deliver more personalized and targeted care creating shared care plans by clearly articulating the role of each provider and patient in the care process. In this paper, we present a method and corresponding implementation of a semi-automatic care plan management tool, integrated with clinical decision support services which can seamlessly access and assess the electronic health records (EHRs) of the patient in comparison with evidence based clinical guidelines to suggest personalized recommendations for goals and interventions to be added to the individualized care plans. We also report the results of usability studies carried out in four pilot sites by patients and clinicians.
Content may be subject to copyright.
Accepted Manuscript
A collaborative platform for Management of Chronic Diseases via
guideline-driven individualized care plans
Gokce B. Laleci Erturkmen, Mustafa Yuksel, Bunyamin Sarigul,
Theodoros N. Arvanitis, Pontus Lindman, Rong Chen, Lei Zhao,
Eric Sadou, Jacques Bouaud, Lamine Traore, Alper Teoman,
Sarah N. Lim Choi Keung, George Despotou, Esteban de Manuel,
Dolores Verdoy, Antonio de Blas, Nicolas Gonzalez, Mikael
Liljaj, Malte von Tottleben, Marie Beach, Christopher Marguerie,
Gunnar O. Klein, Dipak Kalra
PII: S2001-0370(18)30350-7
Reference: CSBJ 335
To appear in: Computational and Structural Biotechnology Journal
Received date: 20 December 2018
Revised date: 18 March 2019
Accepted date: 4 June 2019
Please cite this article as: G.B. Laleci Erturkmen, M. Yuksel, B. Sarigul, et al., A
collaborative platform for Management of Chronic Diseases via guideline-driven
individualized care plans, Computational and Structural Biotechnology Journal,
This is a PDF file of an unedited manuscript that has been accepted for publication. As
a service to our customers we are providing this early version of the manuscript. The
manuscript will undergo copyediting, typesetting, and review of the resulting proof before
it is published in its final form. Please note that during the production process errors may
be discovered which could affect the content, and all legal disclaimers that apply to the
journal pertain.
A Collaborative Platform for Management of Chronic
Diseases via Guideline-Driven Individualized Care
Gokce B. Laleci Erturkmena,*, Mustafa Yuksela,
Bunyamin Sarigula, Theodoros N. Arvanitisb, Pontus Lindmanc, Rong
Chend,e, Lei Zhaob, Eric Sadoug, Jacques Bouaudf,g, Lamine Traoreg,
Alper Teomana, Sarah N. Lim Choi Keungb, George Despotoub, Esteban
de Manuelh, Dolores Verdoyh, Antonio de Blasi, Nicolas Gonzalezi,
Mikael Liljaj, Malte von Tottlebenk, Marie Beachl, Christopher
Margueriel, Gunnar O. Kleinm, Dipak Kalran
aSRDC Software Research Development and Consultancy Corp, Ankara, Turkey
bInstitute of Digital Healthcare, WMG, University of Warwick, Coventry, UK
cMedixine, Finland
dCambio Healthcare Systems, Sweden
eHealth Informatics Center, Karolinska Institutet, Sweden
fAP-HP, Delegation for Clinical Research and Innovation, Paris, Frances
gInserm, Sorbonne Université, Univ Paris 13, Laboratoire d'Informatique Médicale
d'Ingénierie des Connaissances pour la e-Santé, LIMICS, F-75011 Paris, France
hKronikgune, Research Center in Chronicity, Spain
iOsakidetza, Spain
jDepartment of Public Health and Clinical Medicine, Unit of Research, Education,
and Development, Östersund Hospital, Umeå University, Umeå, Sweden
kempirica Gesellschaft fÜr Kommunikations- und Technologieforschung mbH,
Bonn, Germany
lSouth Warwickshire NHS Foundation Trust, UK
mÖrebro University School of Business, Informatics, Örebro, Sweden
nEuropean Institute for Innovation through Health Data, Belgium
*Corresponding author.
Older age is associated with an increased accumulation of multiple chronic conditions. The clinical
management of patients suffering from multiple chronic conditions is very complex, disconnected and
time-consuming with the traditional care settings. Integrated care is a means to address the growing
demand for improved patient experience and health outcomes of multimorbid and long-term care patients.
Care planning is a prevalent approach of integrated care, where the aim is to deliver more personalized
and targeted care creating shared care plans by clearly articulating the role of each provider and patient in
the care process. In this paper, we present a method and corresponding implementation of a semi-
automatic care plan management tool, integrated with clinical decision support services which can
seamlessly access and assess the electronic health records (EHRs) of the patient in comparison with
evidence based clinical guidelines to suggest personalized recommendations for goals and interventions to
be added to the individualized care plans. We also report the results of usability studies carried out in four
pilot sites by patients and clinicians.
integrated care chronic disease management clinical decision support systems evidence based clinical
1. Introduction
A growing share of the population (15% in 2010) in OECD countries is over 65 and expected to reach 22
% by 2030 [1]. Older age is associated with an increased accumulation of multiple chronic conditions
(multi-morbidity), including a growing number of functional and cognitive impairments [2]. More than
half of all older people have at least three chronic conditions, and a significant proportion has five or
more [3]. Multi-morbidity creates diverse, and sometimes, contradictory needs, which challenge patients
and the delivery of health services [4]. The clinical management of patients suffering from multiple
chronic conditions is very complex, disconnected and time-consuming within the traditional care settings
and, hence, currently those with chronic conditions and long-term care needs experience shortcomings
and gaps in care provision.
Integrated care is seen as a means to transform health services to meet these challenges of 21st
century, by addressing the growing demand for improved patient experience and health outcomes of
multimorbid and long-term care patients [5]. Care planning is a central approach of integrated care, where
the aim is to deliver more personalized and targeted care creating shared care plans that map care
processes (care pathways) by clearly articulating the role of each provider and patient in the care process.
In the state of the art practices, the multidisciplinary teams (MDT) meet face-to-face to discuss and revise
the care plans of several patients at once, at regular time intervals; usually monthly. Individualized care
plans are created by manually going over the standard steps of care pathways, i.e. template care plans
which are documentation of the optimal management for typical, defined disease patterns. Although
implementation of integrated care via these manual processes is already an enhancement over traditional
fragmented care practices, we believe significant improvement can be achieved if intelligent collaborative
tools can be developed to support both the care teams and also patients and their care givers. In this paper,
we present the architecture and implementation of a coordinated care and cure delivery platform, as an
integrated care management tool encompassing clinical decision support services for MDTs and patient
empowerment tools for patients and their care givers (Figure 1).
Figure 1: The basic concepts of the C3-Cloud system
In particular, we address the following bottlenecks in traditional care delivery mechanisms,
through the automated tools and services we provide:
Traditionally, the adoption of clinical practice guidelines has been promoted for
managing chronic conditions. However, although CPGs may contain some basic
references, their scope is predominantly focused on single diseases, without sufficient
consideration of co-morbidity and multimorbidity. Following more than one clinical
guideline can result in inefficiencies for the patient and for the health system due to
duplicated and inconveniently scheduled investigations and clinic visits and, more
importantly, treatments that may adversely affect another condition [6, 7]. Care providers
are in need of clinical decision support services to detect and warn them about guideline
conflicts, to select upon most suitable treatment options in the light of evidence based
guidelines and to schedule and prioritize treatment activities. Our platform equips the
MDTs with intelligent services to suggest personalized goals and interventions for the
care plan of the patient based on the most recent context of the patient and evidence
based guidelines. The coordinated care and cure delivery platform enables the MDTs to
coordinate the execution and monitoring of the integrated care plans in close cooperation.
Managing multi-morbidity, through the current treatment methods, results in specialty
silos involving multiple health and social care providers who are not effectively
communicating and sharing information. As the number and complexity of health
conditions increase over time and episodes of acute illness are superimposed, the type
and number of care providers contributing to the care of individuals also increases. It
becomes significantly more difficult to align and coordinate care across care teams and
associated settings. This results in fragmented care, due to poor communication and
information sharing. Without secure information exchange among the actors involved in
health, social and informal care services and a process to reconcile potentially conflicting
treatment plans, it is impossible to avoid redundant and potentially harmful interventions.
Informed decision making also requires information to be shared between the
regional/institutional Electronic Health Records (EHRs), Social Care Records (SCRs) and
homecare services. C3-Cloud encompasses interoperability adapters that allow
heterogeneous data sources to share their EHR data securely, and an online collaboration
that offers MDT members a single coherent view (of that data). The interoperability
architecture also enables the clinical decision support services to seamlessly access and
assess the electronic health records (EHRs). In this way they can provide personalized
recommendations for goals and interventions to be added to the individualized care plans
of the patient.
Patients and their informal care givers including family members often do not have a
Rvoice in the management of their own care. WHO reports that adherence to long-term
therapy for chronic illnesses in developed countries averages only 50 %, and is even
lower in developing countries [8]. Adherence rates drop significantly in complex
treatment and care regimes compared to simpler ones [8]; multi-morbidity together with
the increased probability of poly-pharmacy reinforces non-adherence behaviour further
[9]. Adherence to treatment regimes (lifestyle and drugs) necessitates behaviour change
by the patients, which can be more difficult for the elderly. Patients and their informal
care givers, including family members, need to receive complete information about the
benefits and risks of treatments; and to be offered real opportunities for shared decision
making, expressing preferences and engaging in self-management. Our solution provides
a patient empowerment platform to ensure active participation of patients and their
informal care givers to the management of their multi-morbid chronic conditions. Patient
empowerment platform presents the care plan via quantitative and qualitative outcome
goals and action items to the patients. It continuosly collects feedback from the patient to
record their activities and problems they encounter, monitor risk factors through online
patient-reported outcome measures questionnaires, and establish a two way
communication between MDT members and the patients.
The paper is organized as follows: in Section 2, we briefly present existing research results in this
field and explain how we complement and extend these work. Section 3 provides a detailed description of
our architecture, elaborating the architectural choices and implementation strategies for each
subcomponent. Section 4 reports the results of the usability studies that have been conducted. Finally, in
Section 5 we present the planned future work, and elaborate on the advantages of our platform and
conclude the paper.
2. Related Work
Clinical guidelines are used in the healthcare domain to improve the quality of care [10]. It has been
demonstrated that clinical guidelines provided as real-time decision support systems improve patient care
significantly [11, 12, 13, 14] and decrease undesired practice variability [15]. Yet, the success of clinical
decision-support systems requires that they are seamlessly integrated with clinical workflows [16, 17].
Several methodological approaches exist to implement clinical guidelines into operational practice.
Narrative guidelines can be formalized via computer interpretable guideline representation languages
such as Arden Syntax and PROforma [18]. These can be served as modular clinical decision support
(CDS) services, that can be utilized by hospital information systems during patient treatment to provide
alert and reminders about missing or contraindicating interventions (e.g., EBMeDS [19]). However, this
does not directly support healthcare professionals to follow a standardized plan of care for a specific
condition, as a clinical workflow. Clinical pathways are appropriate for that purpose; however clinical
guidelines and care pathways are often viewed as separate entities, their synergistic potential remaining
only partially exploited [20].
Clinical information systems have been built to automate care pathways to send reminders for
providers to enable periodical assessments, diagnostic tests and treatments; data collection on process and
outcome indicators for performance assessment; continuous monitoring of progress and information
sharing, examples include InformaCare, Medix [21, 22]. Yet, these do not include personalized clinical
decision support in the light of clinical guidelines.
In this paper, we present an approach to effectively integrate clinical guidelines and care
pathways: we show that it is possible to semi-automatically personalize care pathways to create
individualized care plans, by automatically processing knowledge in clinical guidelines and patient’s
EHRs. In this way, this will enable following the recommendations of clinical guidelines as a clinical
workflow executed via integrated care plans for addressing the demanding needs of patients suffering
from long-term chronic conditions.
3. C3-Cloud System Architecture
The C3-Cloud project [23] aims to change the currently fragmented medical care provided for the patients
suffering from multiple chronic conditions. It provides an ICT infrastructure for patients and multi-
disciplinary care teams, to coordinate the integrated care for the patients in a patient centered fashion.
We have implemented a Coordinated Care and Cure Delivery Platform (C3DP) that allows
collaborative creation and execution of personalised care plans for multi-morbid patients by a
multidisciplinary care team (MDT) including GPs, specialists, study nurses, pharmacists,
physiotherapists, geriatricians, nutritionists, social care and homecare workers. C3DP is the Web
application for collaborative and personalized care plan management by the members of MDT. In the C3-
Cloud architecture, C3DP sits at the top of the hierarchy and is directly integrated with all the other C3-
Cloud components and indirectly with the local EHR/EMR systems of the pilot sites as presented in
Figure 2. All the patient data required for care planning are fetched from the C3-Cloud FHIR Repository,
which is continuously fed with existing EHR data of the pilot sites via our interoperability architecture
composed of the Technical and Semantic Interoperability Suites (TIS and SIS). With the help of Clinical
Decision Support Modules (CDSM) automating multiple clinical guidelines, C3DP processes electronic
health records of the individual patients and provides guidance to the multidisciplinary care team
members for i) risk prediction and stratification, ii) personalized selection of treatment goals and
interventions in the light of evidence based guidelines, iii) reconciliation of conflicting treatment options
and iv) management of polypharmacy. Active patient involvement and treatment adherence is achieved
through a Patient Empowerment Platform (PEP), ensuring patient needs are respected in decision making
and taking into account preferences and psychosocial aspects. Finally, the Security and Privacy Suite
(SPS) provides common security features for user authentication, authorization and audit logging to all of
the other components.
Figure 2: High Level System Architecture
In order to ensure wide adoption, we have chosen to build a standards-based architecture, where
widely accepted industry standards are chosen as building blocks of our implementation. In the following
sub-sections, we first briefly present our interoperability infrastructure, then outline the details of the
architecture sub-components which are integrated to implement an intelligent platform to support
integrated care by enabling the personalization of care pathways as care plans dynamically.
3.1. C3-Cloud Interoperability Architecture
Aiming to orchestrate the care across multiple care givers and treatment sites, and automatically process
patients’ EHRs to be able to recommend personalized treatment goals and interventions, inevitably
requires interoperability to exchange and seamlessly process medications, conditions, interventions,
episodic treatment plans, preferences and patient reported data including sensor measurements. We have
chosen to build our technical interoperability layer based on clinical resources and RESTful interfaces of
the HL7 Fast Healthcare Interoperability Resources (FHIR) STU3 standards framework [24]. The C3DP
accesses patient’s most recent EHRs, through FHIR based interfaces implemented on top of the
proprietary APIs provided by local EHR systems in our pilot sites.
Technical Interoperability Suite. The Technical Interoperability Suite (TIS) provides a standard-based
data exchange protocol, in order to enable information exchange between local care systems and C3-
Cloud components, such as C3DP. C3-Cloud is being piloted in three different pilot sites: a) Basque
Health Service - Osakidetza [OSAKI], Spain; b) Region Jamtland Harjedalen [RJH], Sweden and c)
South Warwickshire NHS Foundation Trust [SWFT], UK. The EHR API, data representation, and
operational environment vary amongst local care systems, which hinders the processing of a unified
record while creating the care plan. In order to provide maximum flexibility and extensibility, TIS is
implemented as an extract, transform and load (ETL) software development kit (SDK). TIS utilizes the
ETL model to pull patient data out of a local EHR system through its native API, convert the data into
selected FHIR resources compliant with C3-Cloud profiles with support of the Semantic Interoperability
Suite (SIS), and push the transformed FHIR data into the C3-Cloud FHIR repository. The core of TIS is
an ETL engine, which is able to schedule and execute ETL tasks. TIS also provides an extensible library
of functions, on top of which is easy to assemble an ETL task for integration with an EHR data source.
TIS provides both a web-based user interface for system administrator to execute or schedule an ETL
task, and a RESTful service API for other C3-Cloud components, such as C3DP, to trigger an immediate
ETL action, so as to get the latest patient data.
Figure 3: TIS interfaces to local EHR systems for fetching patient data
A set of pipelines have been developed for addressing the diverse needs of the three different C3-
Cloud pilot environments and the heteregeneous EHR system APIs provided. Figure 3 summarises the
patient data APIs that each pilot site exposes. The first pilot site (Basque Health Service - Osakidetza
[OSAKI], Spain) provides two separate Web services: one for exposing medical summary of patient as
HL7 CDA documents, and a second one to expose the lab results of the patients as a separate CDA
document. The second pilot site (Region Jamtland Harjedalen [RJH], Sweden) provides 6 different
RESTful services that expose patient data (patient demographics, diagnoses, lab results, medications,
notes and encounters) via proprietary JSON documents. Finally, the third pilot site (South Warwickshire
NHS Foundation Trust [SWFT], UK) provides patient data through two daily CSV exports; one from
primary care system, and the second from secondary care and community care encounters. All the
pipelines follow a similar pattern: retrieve patient data by patient identifiers; invoke SIS Structure
Mapping Service to transform the data into FHIR resources; combine all resources into a FHIR
transaction bundle; include an AuditEvent with timestamp in the transaction bundle; and commit the
transaction bundle into the C3-Cloud FHIR Repository. If it is the first data import, i.e. the patient has not
been created in the repository yet, TIS will add a FHIR Patient resource to the bundle, which includes C3-
Cloud study identifier and evaluation group assignment information, and notify C3DP by sending a
PatientCreated event through C3DP Event API. If an error occurs at any step of the pipeline execution,
TIS logs the error in the database and presents it via the control panel.
Semantic Interoperability Suite. The Semantic Interoperability Suite (SIS) handles both structural
mappings among different information models and resolves semantic mismatches due to the use of
different terminology systems and different compositional aggregations, as used to represent the same
clinical concept. Due to local implications of terminologies used, the SIS is developed in close relation
with the pilot sites. Two different types of mappings are performed in the semantic interoperability suite:
structural mappings and semantic mappings. Structural mappings are involved in the transformation
between local pilot sites data in local format and FHIR resources data format used in C3-Cloud. Semantic
mappings perform the transcoding between coding systems used in local sites and the C3-Cloud
Figure 4: Semantic Interoperability Suite Architecture
The architecture of the SIS is provided in Figure 4. SIS is articulated around two main sub-
components: SIS Structural Mapper and SIS Semantic Mapper.
1. SIS Structural Mapper: The structural mapper of SIS is the internal SIS sub-component in
charge of the generation of FHIR resources, which have to be filled with data provided in
pilot site local format by TIS. To achieve its purpose, the structural mapper consists of
pilot site dedicated local format mappers. These mappers provide precise mappings to
create correspondence to every relevant data exported by the pilot site to its correct
interpretation and place in FHIR resource. FHIR resources, mapped from pilot site data,
are defined in the C3-Cloud data dictionary.
2. SIS Semantic Mapper: The semantic mapper of SIS is in charge of transcoding, using the
HeTOP service [25], the vocabulary used to describe data exported by pilot site into
standard codes that are used in the high-level components of C3-Cloud. A clinical
concept mapping sheet is being maintained as the source of truth, which includes all the
clinical concepts that are needed by the CDS services, in reference terminologies like
SNOMED-CT, LOINC and WHO ATC, and all the local codes (e.g., Spanish and
Swedish versions of ICD-10, completely local terminologies for laboratory tests) that are
used by the pilot sites for these concepts. In total, 218 common clinical concepts
including conditions, active ingredients of medications, procedures, lab results, vital
signs, immunizations and family member history have been identified and bound to well-
known terminology systems like SNOMED-CT, LOINC and ATC. It should be noted
that this list includes not only leaf-level but also high-level concepts such as
antihypertensive drugs (ATC:C02) and beta blockers (ATC:C07) or diabetes (SNOMED-
CT:73211009); hence the number of leaf-level concepts in effect is much higher. These
concepts have been mapped to 516 different codes from locally used terminology systems
of three pilot sites. These local systems are composed of localized versions of
international systems like Spanish and Swedish versions of ICD-10 and ATC in the case
of Basque Country and Region Jamtland Harjedalen, and national systems like DBP
codes for Spanish observations and READ codes for UK diagnoses. Mapping benefits
from the implicit hierarchical relationship between high-level and leaf-level concepts in
these local systems as well.
The Structural Mapper generates JSON encoded FHIR resources. The semantic mapping is based
on a pre-filled registry containing, for each concept, the corresponding code(s) for each site’s
terminology, and the code used as reference by C3-Cloud. The registry is continuously updated via a
dedicated service during the time of the project. Multiple codes can be specified for a single concept if the
used terminology has several codes corresponding to the concept (narrower-than relation). Multiple
terminologies are used as reference, in order to match each concept exactly. Both the Structural Mapper
and Semantic Mapper provide a REST API for integration with other C3-Cloud components. An example
mapping response, which is represented as an HL7 FHIR ConceptMap resource, is provided in Figure 5.
Figure 5: Semantic Mapping Example
In this example, the request is to map 44054006 in SNOMED-CT terminology that is used as a
reference for diagnoses in C3-Cloud to the terminology that is used by the Osakidetza pilot site, which is
ICD-10 Spanish (ICD-10-SE) in this case. A JSON-encoded FHIR ConceptMap resource is provided as a
response, describing the URIs of the input and output code systems (SNOMED-CT as the source and
ICD-10-SE as the target), the input code and the corresponding code in the target system (ICD-10-SE
code E11 for Diabetes mellitus type 2 in local language). It also shows the type of relation; equivalent in
this case, meaning that the two concepts are identical.
FHIR Repository. The C3-Cloud FHIR Repository acts as the centralized data repository for existing
clinical data of the patients and newly created care planning related data. It stores the data, which arrive
from EHR systems via TIS and newly created or updated care plan data from other C3-Cloud components
like C3DP and PEP, as HL7 FHIR resources. C3-Cloud FHIR Repository, [26] is fully
compliant to FHIR STU3 specification and implemented on top of MongoDB noSQL database. An
authorized user or system can use native FHIR STU3 API, i.e Restful interfaces to
store/query/update/delete patient data. It is not possible to access any resource in the secured repository
without first acquiring a valid access token. The authorization flow is fully compliant with the Smart App
Authorization specification, which is based on the well-known OAuth 2.0 specification [27] as supported
by C3-Cloud SPS module. Thanks to C3-Cloud FHIR Repository’s automatic auditing functionality, audit
trail records are kept for each access and manipulation of data as FHIR AuditEvent resources to ensure
accountability. These audit resources are available from the same API for authorized users with
administrator roles as any other FHIR resource.
3.2 Clinical Decision Support Services
The Clinical Decision Support (CDS) services are supporting modules of the C3-Cloud Coordinated Care
and Cure Delivery Platform (C3DP). The CDS services enable the reconciliation of clinical guidelines for
individual diseases, risk stratification, poly-pharmacy management and care plan goal setting and
monitoring. As part of the C3DP platform, CDS services access, fuse and analyse patient data. This is
accomplished in order to: perform risk assessment and stratification of candidate elderly people for
inclusion in integrated care programmes; reconcile clinical guidelines for individual diseases to develop
personalised care plans; detect and propose resolutions for guideline clashes; detect duplicate,
unnecessary or contraindicating medications; and monitor and detect deviations from the outcome goals
set in a patient’s care plan. C3-Cloud focuses on elderly patients, who have at least 2 out of the following
4 chronic diseases:
Diabetes Mellitus type 2 (T2D)
Renal Failure (RF) (excluding Glomerular filtration rate (GFR) or estimated (eGFR)<30)
Heart Failure (HF) (including New York Heart Association (NYHA) Functional
Classification I-II; excluding NYHA III-IV)
Depression (Dp) (mild/moderate conditions only)
The clinical expert group of the C3-Cloud project have examined the clinical literature, and have
identified four NICE (National Institute for Health and Care Excellence) guidelines to be followed for the
management of these four conditons as depicted in Table 1. NICE guidelines are already used in UK; the
clinical experts from Spanish and Swedish pilot sites have reviewed them and provided the required local
extensions. For example, while NICE guidelines suggest using ’Atorvastatin 20mg or 80mg’ for lipid
lowering, this recommendation has been modified as ’Simvastatin 20-40 mg’ in Basque localization.
Based on these selected NICE guidelines, CDS services have been implemented to support care planning
for the 4 specific diseases and their combinations, as listed in Table 1.
Table 1: CDS Services implemented for each disease
Type 2 Diabetes
Renal Failure
Heart Failure
NG28: Type 2
diabetes in adults
CG182 Chronic
kidney disease in
adults [29]
CG108: Chronic
heart failure in
adults [30]
CG90: Depression
n adults [31]
CDS Services
DM Blood
CKD referral
CHF vaccination
QRISK2 assesment
and Lipid
CKD eGFR-control
CHF Stability
Mild to moderate
HbA1c targets
prevention and
CHF Diuretics
Blood glucose
CKD blood
pressure treatment
Diabetic foot
Diabetic neuropathy
Diabetic retinopathy
Services for
Life style
Diet management
The clinical expert group examined these guidelines and have designed several flowcharts that
can be used as a guidance to ease the development of care plans addressing the individual needs of
patients [32]. As an example, 19 flowcharts have been designed by clinical experts covering the
recommendations of NICE Type 2 diabetes in adults: management clinical guideline (NG28) [28]. These
flowcharts constituted the basis for the CDS algorithms, which, in collaborations with the project
engineers, have been implemented as a real-time executable CDS services. The ones that can provide
computable suggestions have been identified and the inputs and possible outputs of these CDS services
have been specified as FHIR resources. An example annotated flowchart in Figure 6, depicts possible
CDS recommendations about the required lipid lowering goals and interventions. The specifications of
these CDS services have been validated once again by clinical experts. 7 different CDS services have
been implemented automating the 19 flowcharts extracted from the NG 28 guidelines. These 7 CDS
services implement a total of 80 different clinical rules, checking 108 different patient criteria, to
recommend 119 different personalized goal and intervention suggestions.
Figure 6: A sample flowchart for Lipid Managment CDS
An uncritical combination of clinical guidelines for separate diseases when treating multi-morbid
patients could have contradictions that would increase risk and in some cases even result in unfeasible
treatment. There is a need for reconciliation to support clinicians in decision making in risk assessment,
setting goals, choosing activities or pharmacologic treatment to include in a care plan of a multi-morbid
patient. The reconciliation exercise we followed aimed to analyze the Disease-Disease, Disease-Drug and
Drug-Disease interactions between the recommendations of given by the flowcharts designed to address
the needs of single conditions. The final aim is to reconcile relevant recommendations for different
chronic conditions by identifying the synergies, cautions and contradictions. The clinical expert groups
have carefully analyzed the flowcharts identified for individual conditions, and checked the interactions in
multi-morbid patients (patterns of double, triple or quadruple comorbidity). Potential conflicts were
identified such as repetition, wrong sequence or overlaps of activities, contradictory goals, location
inconsistencies alternative options, constraints, potential treatment synergies (either beneficial or
harmful), outliers, and inconsistencies within pairs of pathways. The next step was to reconcile these
recommendations to mitigate the conflict or inconsistency by modifiying them according to available
knowledge (select, removal, merge, substitution, modify with extra input and output). Reconciled rules
have been designed and integrated into existing flowcharts by modfying them where necessary.
Altogether, 52 reconciled rules were defined: 50 rules for the two-diseases combination , 1 rule for three-
diseases combination and 1 rule for four-diseases combination [33].
These guideline-based flowcharts and reconciliation rules are implemented as FHIR based the
CDS Hooks services [34], with decision logic encoded in the Guideline Definition Language (GDL)
version 2 [35]. GDL is a formal language used to express clinical rules and guidelines in a machine-
readable format by leveraging semantically interoperable EHR standards. GDLv2 supports FHIR and
CDS Hooks. The project uses the GDL2 Editor software to develop both the CDS guideline definitions
and CDS Hooks services (Figure 7).
To be able to intelligently propose individualized goals and activity suggestions for the selected
health concerns, C3DP is integrated with these external CDS services via CDS Hooks API. The electronic
health records of the patient retrieved from local care systems, are passed as input to CDS services after
the semantic mapping of the local codes into international code systems are handled as explained in
Section 3.1. Within CDS logic, processing these patient specific diagnoses, lab results, medication data
enables the selection of individualized goals and interventions for this specific patient. The response
consists of textual recommendations communicated as information cards and computable
recommendations communicated as suggestion cards in conformance to CDS Hooks API. In suggestion
cards, the recommended goals and activities are represented as FHIR resources (such as
MedicationRequest, Goal, Appointment resources) which can readily be included into the care plan
model. The finalized personalized care plan can be shared back with the local EHR Systems by exporting
the care plan as a FHIR CarePlan resource serialized as a JSON instance.
Figure 7: A screenshot of the Diabetic foot problem guideline in GDL2 editor
In addition to the guideline-based services, a RESTful service is developed to manage drug-drug
and drug-disease interactions. The drug interaction service provides warning of potential adverse
interactions between drugs, as well as a list of side-effects, for use by the rest of the C3-Cloud
architecture. The service implements the interactions between drugs, as specified by the National Institute
of Care Excellence’s implementation of the British National Formulary (BNF) [36]. BNF is a
pharmaceutical reference book, used by the UK NHS. The information provided by the service, is
identification of potential adverse interaction between drugs, the effects of the interaction, the severity of
the interaction, as well as the basis on which the interaction has been specified by the NICE-BNF. For
example, Acarbose is a drug active ingredient Alpha-glucosidase inhibitor, commonly used by patients
with type 2 diabetes, which reduces the effects of carbohydrates on blood sugar. Acarbose is listed as
having a pharmacokinetic interaction with the active ingredient Digoxin used in patients with Congestive
Heart Failure to improve quality of life and prevent hospitalisation. The interaction is listed as moderate
in criticality, having an effect as decreasing the concentration of Digoxin. In addition to interactions, the
service provides a list of side-effects for each substance, along with their frequency (i.e., common,
uncommon and rare).
The information collected by the NICE BNF is encoded as a database defining the relationships
between the main concepts e.g., interactions and drugs, and contains instances for each active ingredient.
The current database contains 108,600 instances of interactions and 26,403 instances of side-effects for
1,009 substances. The service is designed to receive a list of active ingredients of the drugs a patient may
be taking. The list is checked against the database and returns a data object containing the interaction
information as well as the list of side-effects. The information can be shown to the C3DP dashboard
notifying the MDT members of potential risks.
3.3 Patient Empowerment Platform
The objective of the Patient Empowerment Platform (PEP) is to provide patients with access to the
published care plan and its information and thus increase patient and informal caregiver participation in
decision making [37, 38]. PEP aims to provide computerised means to improve the interaction between
patients and health professionals and to digitally collect relevant data and information to enable
monitoring of care plan related activity status and progress. It directly interacts with C3DP to receive new
and updated care plans, and to send back patient reported observations. It allows MDT members to assign
questionnaires to patients to collect patient reported outcome measures, which are later filled in by the
patient via PEP interfaces and shared back with C3DP through standard based interfaces based on HL7
FHIR. It, also, directly communicates with the supported set of sensor devices to record patient
measurements. The core user functionalities and features provided to PEP users are:
Make published care plans available to the users.
Send reminders to patients to help them comply and stay on track with the interventions
and activities included in the care plan.
Allow patients to actively collect data related to the care plan activities.
Allow health professionals and patients to communicate with each other using either
messages or video appointments.
Provide patients with access to relevant self-management material.
Allow patients to provide feedback to MDT members about the care plan activities (such
as reporting probable side effects of medications)
Figure 8: Patient Empowerment Platform - Care Plan Details Screen
The PEP user application is a modern web application, which allows PEP Users (patients and
their informal caregivers) to access all the functionality via web browsers (see Figure 8). PEP is built on
top of the Medixine Suite product. The Medixine Suite technology stack follows traditional logic for web-
based services and consists of an OS (Microsoft Windows Server), data storage (Microsoft SQL Server),
Web Server (IIS) and Programming platform (.NET). The database layer contains some supporting
functionalities, but the main business logic is built into the application layer. The application logic is
based on modern resource-based thinking and is accessed through a REST API that is structured around
those resources. The core logic includes role-based access configuration for different operations, full audit
trails of operations performed in the system, an application ecosystem model, dynamic and extensible
data modelling tools, event subscription model for integrations and extensible support for multiple
languages and cultures.
3.4. Security and Privacy Suite
The Security and Privacy Suite (SPS) is responsible for authentication and authorisation of the care team
members, while they are managing personalised care plans of patients and ensuring that all data
exchanged within and across C3-Cloud software components is encrypted and properly auditable. In the
C3-Cloud architecture, the patient’s electronic health records received from the local EHR systems via the
TIS, patient reported observations from the PEP, and the care plan of the patient managed through C3DP,
are all managed in the C3-Cloud FHIR Repository. Hence, each of these client apps, i.e. TIS, PEP and
C3DP needs to be authenticated and authorized to access (read, write, and update) patient data to the C3-
Cloud FHIR Repository, via the functionalities provided by SPS. All such operations need to be logged
for ensuring accountability via SPS. SPS enables authentication of the care team members into the C3-
Cloud applications in two ways: i) via their already existing accounts (e.g., username-password) provided
by the local authorities by integrating with the existing Identity Provider (IdP) systems of the pilot sites;
and ii) by creating C3-Cloud specific user accounts for those users whose IdP’s cannot be integrated with
the SPS, for example, the social care workers. The SPS has three sub-components:
C3-Cloud SPS Server provides services for user registration, privacy policy management
and endpoints defined in the OpenID Connect 1.0 standard to perform authentication and
authorization (Authorization Endpoint, Token Endpoint, etc.). By implementing the
OpenID Connect API, it serves C3-Cloud Identity Provider (IdP), which is the default
IdP when the IdP of some users (e.g., social care workers) of the pilot sites cannot be
integrated within the scope of the project. The SPS Server also manages the C3-Cloud
Access Control Policy Store.
C3-Cloud SPS Manager is a web application for representing the functionalities of C3-
Cloud SPS Server with the following user interfaces: single sign on UIs, policy
management UI, client registration UI, user registration UI and audit viewer UI.
Audit Record Repository is a FHIR repository that maintains audit trail records
implemented as FHIR AuditEvent resource. In C3-Cloud architecture, the C3-Cloud
FHIR Repository is used as the Audit Record Repository. An extra instance of the same
repository is not created for practical reasons.
3.5. Coordinated Care and Cure Delivery Platform (C3DP)
As depicted in Figure 9, the Coordinated Care and Cure Delivery Platform (C3DP) is implemented as a
Web application for collaborative and personalized care plan management by the members of a
multidisciplinary team of care (MDT). All the patient data required for care planning are fetched from the
C3-Cloud FHIR Repository, which is continuously updated with EHR data from the pilot sites via TIS
and SIS. C3DP visualizes these data and helps the health professionals to easily manage the integrated
care coordination process for multi-morbid elderly patients. C3DP implements the HL7 Care Plan DAM,
and enables health professionals to design a care plan for a patient from scratch by selecting health
concerns to be addressed from the EHR of the patient, and setting goals and activities to address the needs
of this health concern. This process is formalized as a FHIR Care Plan resource, which consists of
building blocks like Goal and different types of Activity resources (Figure 10).
Figure 9: Care Plan Summary Screen of C3DP
Figure 10: Building blocks of a care plan
The major functionalities enabled by the C3DP are:
Review of medical summary. C3DP provides a mechanism to review a complete view of patient’s medical
record. All the patient data that are provided by the local EHR/EMR systems, and also by the patient via
Patient Empowerment Platform (PEP), including conditions, medications, allergies, lab results, vital
signs, procedures and social history are presented to the care team members in a single location.
Preparation of an individualised care plan based on evidence based clinical guidelines. In addition to the
addressed health concerns and risk factors, an integrated care plan is mainly composed of goals to manage
the health concerns and the activities (i.e., interventions) to achieve the identified goals and improve the
associated health concerns. In C3-Cloud, education materials for the empowerment of the patients are also
part of the care plans, while they are treated separately from the rest of the activities. There are three ways
of adding a goal / activity / education material to a care plan:
Manual entry from scratch: Care team members can, at any time, create a goal / activity /
education material themselves via the C3DP user interface.
Recommendations from the CDS services: Personalised goal, activity and education
material suggestions, provided by the CDS services according to patient data can be
directly added to the care plan of a patient by the care team members, or after some
modifications. Figure 11 depicts a snapshot where personalized suggestions based on
CDS recommendations are presented to MDT members.
Transfer from the older care plan: When provided by the local systems of the pilot sites,
it is possible to transfer existing goals and activities from a treatment plan of a patient
into an integrated care plan during its initialisation.
Figure 11: A snapshot from C3DP presenting personalized activity suggestions
Cross-check of all patient data that are needed as input by the CDS services. The CDS services process
patient EHR data to recommend personalized goal and interventions. However the missing or incomplete
data can affect the correctness of a CDS recommendation. In order to address this challenge, all clinical
concepts that may affect the CDS recommendation and result in adverse events are presented to the care
team members along with contextual information that will help the user flag potential errors. For
example, the lipid management CDS service checks the existence of three diseases (and some other
parameters such as lab results and specific medications) in its decision tree: type 2 diabetes, chronic
kidney disease and cardiovascular disease. The patient records retrieved from the local EHR system show
that the patient has type 2 diabetes, but there is no information about the existence of chronic kidney
disease and cardiovascular disease. The GP of the patient can declare that this patient has also a
cardiovascular disease, which was somehow missing in the patient’s EHR system records. Such newly
provided patient data, via the C3DP interface, can be provided back to the original EHR/EMR systems.
Execution of a care plan. Integrated care planning is a continuous process. Ideally, an integrated care plan
lives with the patient and is adjusted to the most recent patient context. It is updated during planned and
unplanned encounters of the patient with health professionals and social care workers, and also with
patient provided feedback via the Patient Empowerment Platform. All updates can be shared with the
local EHR/EMR systems as well. Hence, execution of a care plan refers to the continuous follow-up and
update of an integrated care plan. This can happen in a number of ways in C3-Cloud:
Updating the progress of goals and activities: The status of any goal or activity can be
updated (e.g., a goal can be set as achieved or on-target) by a care team member. The
patient can also provide feedback on their progress.
Re-execution of CDS services during planned and unplanned encounters: This is akin to
the CDS service usage for the first time during initialization of a care plan. Relevant
progress in the patient status is reflected in the recommendations of the CDS services.
Display of patient provided data: Patient and his informal care giver are active
participants of the care planning process. Goals and activities are decided with his active
involvement, and for an activity that is assigned to themselves, the patient is able to
provide update via the Patient Empowerment Platform (PEP). Patient provided data
includes questionnaire responses, medical device measurements (e.g., blood glucose,
blood pressure), daily meal photographs and more. All patient provided data are matched
with the corresponding care plan items and shown to the care team members.
Commenting on the care plan items: It is also possible to comment on specific goals and
activities of a care plan, which are visible to the care team members.
Management of the care team. It is possible to invite new care team members to a care plan, during
initialization or at any time. An invitation is subject to the confirmation of the invited care team member,
who is informed via a notification in the system and an email depending on the preference of the pilot
sites. The care team manager, who is always the GP of the patient in all 3 pilot sites of C3-Cloud, can also
remove a professional from a care team, or a member may want to leave a care team. It is also possible
for a health professional or social care worker to request joining an existing care team for a specific
patient. Different roles can have different rights in the care team; for example, a nurse assistant or a social
care worker can see a care plan but not modify it.
Communication among care team members and with the patient / informal care giver. C3DP has its own
messaging module that enables safe messaging among all care team members, and also with the patients
due to the integration between C3DP and PEP. HL7 FHIR Communication resource is used for
Dashboard view. Dashboard view enables a signed in care team member to quickly go over the important
updates in the care plans of all her patients since the previous login, such as new messages received,
awaiting appointments, new system notifications.
Patient provided data screen. This view collates all the patient provided data such as vital sign
measurements, meal photos, feedback on the care plan and messages to the care team members in a single
Activity calendar. It enables view and update of scheduled activities of a care team member on a calendar.
Real-time system notifications. Real-time system notifications are implemented for several events (e.g.,
for care plan update, new patient feedback, new message, invitation to a care team, etc.). When the user is
already logged in to the system, such notifications are displayed in real time. It is also possible to access
care team members via email for offline scenarios. SMS option was dropped by the pilot sites for real-
time clinical notifications.
4. Usability Studies
To ensure an iterative and holistic approach, the evaluation and impact assessment of the C3-Cloud
project has been split into four layers in accordance with the different stakeholder groups it effects and the
different stages of development and deployment:
Evaluation layer 1 targeted C3-Cloud software component and application tests along
defined protocols by making use of 5 health ICT experts from the University of Warwick,
26 patients and 22 MDT members in the pilot sites.
Evaluation layer 2 included a heuristic evaluation, a Nielsen walkthrough [?] and a
questionnaire, in preparation for layer 3 evaluation. Layer 2 involves 5 health ICT experts
from the University of Warwick, 27 patients and 20 MDT members in the 3 pilot sites.
Evaluation layer 3 will employ an exploratory technology trial that uses baseline and
closure patient observations. Approximately 150 intervention patients and 52 MDT
members will be involved in layer 3 evaluations by answering questionnaires and being
involved in interviews.
Evaluation layer 4 will employ a predictive modelling tool to model the C3-Cloud impact
when scaled up, using intervention and control patient data. Approximately 526
intervention patients and 62 members of the multidisciplinary teams will be involved in
layer 4 evaluation by answering questionnaires and giving access to their anonymized
EHR data. In addition, the data of 526 control patients will be used for data analysis.
Among these, we have completed the execution of first two layers, and evaluation layers 3 and 4
will be initiated in early 2019.
We aimed to ensure continuous, open feedback loops to the development team in evaluation
layers 1 and 2 for the software improvement. In the first evaluation layer, we first carried out tests
specified based on functional requirements of the software and performed application testing before
deployment to ensure that all software components work well together. Results of this study is reported in
[39]. In this section we will focus on reporting the usability studies realized in evaluation layer 2.
Usability testing is a process of tracking real users via a carefully designed protocol to test a
system before and during deployment. This is useful to avoid technology-induced errors; identify issues
and validate and improve the performance of a final product [40]. Kushniruk and colleagues argue that
both commercial vendor based testing and in-situ testing are needed to ensure system usability [41].
The research objective of the presented approach is to identify and categorize early usability
issues of the C3-Cloud components that are used by MDT members and patients. This objective served to
answer the following research questions in accordance with the C3-Cloud research protocol [42].
How usable is the C3-Cloud application perceived by experts, patients and MDT
What usability issues can be identified that must be improved before deployment of the
The following four methods were selected for our usability testing approach:
Method 1: A heuristic evaluation with health IT experts following the Nielsen
walkthrough (Health ICT Experts). Heuristic evaluation is a usability engineering method
which was comprehensively discussed by Jakob Nielsen in 1994 in his book, “Usability
inspection methods” [?]. According to Nielsen, it is “a usability engineering method for
finding the usability problems in a user interface design so that they can be attended to as
part of an iterative design process”. In Nielsen walkthrough, multiple evaluators are
involved (Nielsen recommends three to five) and the users are asked to discover the
answer to given questions by using the system several times (at least twice) according to
the storyboard.
Method 2: Spontaneous feedback gathering during the test sessions with MDT members
and patients separetely (Patients and MDT members)
Method 3: Product reaction cards (Patients and MDT members)
Method 4: The QUIS7 questionnaire on user interaction satisfaction (Patients and MDT
Our usability testing approach is user-centric. The test sessions involved members of the MDT
and patients from the target group spanning three pilot sites in the Basque Country (Spain), Region
Jämtland Härjedalen (Sweden) and South Warwickshire (UK). Health ICT experts were recruited
from the University of Warwick. All MDT members and patients were recruited among people in the
prospective group of software users who speak English. For method 1, the health ICT experts had the
chance to contact technical partners for any questions at any time. For methods 2-4 a language facilitator
from each pilot site moderated each session and was available for any question that was raised from the
participants. All participants received an introduction and a brief overview on the systems to clarify the
test session objectives. The technical project partners developed a walkthrough for both the C3DP and the
PEP. This walkthrough informed test participants about how to access the C3DP and the PEP and listed
test user credentials for all participants. All participants received their own login credentials for the online
High Level Component (HLC) demonstrators of the C3DP and the PEP. This was followed by step-by-
step descriptions activities to be performed by all testers. Technical partners had ensured that all possible
functionalities of the software were covered by the activities that the test participants followed. All test
participants had followed these procedures for a two hour session.
The number of participants attended the test sessions of layer 2 evaluation are depicted in Table
Pilot Sites/ Participant Profiles
Health ICT Experts
MDT members
University of Warwick
South Warwickshire (SWFT)
Basque Country (BC)
Region Jämtland Härjedalen (RJH)
Table 2: Number of participants for the usability testing
Method 1- heuristic evaluation. Heuristic evaluation (HE) focuses on the interface of the system, in the
C3-Cloud case the PEP and C3DP interfaces for patients and healthcare professionals respectively. It is
performed by individual reviewers isolated from each other, and at the end of the process, results are
collated and fed back to the developers. HE is a process that is part of the iterative development process
of a system. HE can reveal a number of issues about the system. Examples of these include bad design
that may lead the user making a slip or mistake, as well as design that may be seen to not appreciate the
sensitivities of the user (e.g., system dialogues). All these issues are classified in a number of categories,
which are the heuristic categories. Thirteen heuristics were considered in the C3-Cloud HE, including
Visibility of system status, User controls and freedom, consistency and standards, Error recovery.
Five specialists, usability evaluation reviewers from the University of Warwick performed the
heuristic evaluation, consisting of the following steps: (i) Reviewers attended a 30-minute session where
the purpose of the evaluation, process and documentation were explained; (ii) Reviewers reviewed the
PEP and C3DP manuals and walkthrough descriptions; (iii) Reviewers made a first structure-free
evaluation of the interfaces; (iv) A second structured pass was done following the workflows in the
manuals and comments were classified under each heuristic; (v) Based on the comments, reviewers
completed a spreadsheet with common issues for each heuristic category, frequency and severity were
combined to create an overall risk matrix that will prioritise modifications by the technical teams.
The results for C3DP and PEP are shown in Table 3, where table presents the distribution of % of
usability errors, in each heuristic.
Distribution of
issues for C3DP
Distribution of
issues for PEP
Table 3: Summary of overall usability issues
Method 2- spontaneous feedback. The objectives of the test sessions were explained to all MDT members
and patient participants and they were given an introduction in the softwares to be tested. Subsequently,
they followed the activities shown in a two hour session. Any spontaneous feedback that was given
during the test sessions in May 2018 was recorded and reported by the session moderator. Feedback was
clustered for general feedback on the HLCs and feedback on specific functionalities (usability; care plan
goals; care plan activities; terminology). The number of clustered comments recorded from the
spontaneous feedback can be seen in Table 4. Duplicated feedback was not reported. Feedback was
supported with screenshots when needed or useful. The software developers studied and prioritised the
feedback using an internal issue tracking system. Prioritization was done for bugs, improvements, features
and cosmetic changes. In total, 101 comments on the C3DP and 44 comments on the PEP platform were
obtained by recording the spontaneous feedback of test participants. Testers were generally very positive
about the C3-Cloud concept and experienced it as very promising, helpful and easy to use. MDT members
have particularly liked the clinical focus of the platforms. The fact that the C3DP suggests goals and
activities was experienced as being very positive and helpful. The software developers translated relevant
issues into a tracking tool and collaborated with the projects’ clinical reference group and the pilot sites to
resolve open issues. Feedback responses were incorporated in respective activities of software
development, software deployment at the pilot sites and training plans.
C3DP Platform
PEP Platform
General feedback
Usability feedback
Care plan goals
Care plan activities
System terminology
Table 4: Spontaneous feedback
Figure 12: Set of 118 words for the “product reaction cards” [43]
Method 3- Product reaction cards. The “product reaction cards”
is a fast and simple method used for an
overall system evaluation. It allows the user to describe the system from a predefined set of 118 words
(Figure 12). This list includes positive words, together with negatives and neutral words. The main
advantage of this approach is that it does not rely on a questionnaire or rating scales and users do not have
to generate words themselves [44]. For the “product reaction cards” method, 48 people (22 MDTs and 26
Patients) from the three pilot sites: Osakidetza, RJH and SWFT; participated to the evaluation. The
participants received login credentials for the online demonstrators of the C3DP (for MDTs) and the PEP
(for Patients/ICGs) as well as training materials including a walkthrough that guided them through certain
activities on the demonstrators The participants were asked to pick the words that best describe the C3-
Cloud platform or how using the product made them feel. We limit the choice number to 5 words, as
commonly used in such approach. At the end of the study, a scoring was made to identify the most
commonly words by the participants to describe the system.
For the 22 participants of the MDT profile, 30% describe the system as "Collaborative", 16% find
it "Comprehensive”, 17% find it both “Empowering” and “Innovative" and 20% as "Time Consuming”.
For the 26 participants of the patient profile, 25% describe the system as "Useful", 21% find it both
"Accessible and Convenient" and 17% find it Appealing and 16% find Advanced". In general, for all the
48 participants, the C3-Cloud system appears to be "Collaborative" at 23%, "Useful and Empowering" at
The “product reaction cards” were developed by Microsoft as part of a “desirability toolkit” created to
understand the illusive, intangible aspect of desirability resulting from a user’s experience with a product.
17%, "Innovative" at 15% and both "Complex" and "Comprehensive" at 14%. In this study, the “product
reaction cards” method gives a good understanding of the user’s experience. This method is easy and
quick to conduct and permits to get user’s experience with the system. When we collate the responses
from all participants, results we obtained show that users pick either same card or a closely related card.
Overall, this “product reaction cards” method is an early evaluation exercise. It permited to get first users’
feedback and feelings. This helped the technical partners/component owners to further improve the C3-
Cloud design, application and its implementation to better align with what the user’s expectation.
Method 4- QUIS7. The Questionnaire for User Interaction Satisfaction (QUIS, 7th version) is a tool that
measures attitude towards software interface factors: screen factors, terminology and system feedback,
learning factors, system capabilities, technical manuals, on-line tutorials, multimedia, voice recognition,
virtual environments, internet access and software installation. Respondends are asked to rate one or more
questions for these categories on a 0-9 scale. The original QUIS7 questionnaire items were adapted to
cater for the requirements of the software. Specific items were omitted as they were considered not useful
for the evaluation at this stage of the project. In addition to the standard English and Spanish versions, the
questionnaire was translated to Swedish for use at the RJH deployment site. After the test session
participants finished the walkthrough, they were asked to fill in the QUIS7 online questionnaire
anonymously. This early usability testing was performed with both MDT members for the C3DP platform
and patients for the PEP platform. The results of the QUIS7 questionnaire are used for shaping the design
and redesign of the platforms, detecting areas for usability improvement, and the comparative evaluation
of the platform from its current status and later during the technology trial. While the QUIS7 follows a
clear structure and helps identifying areas for improvement, it lacks detail and reasoning when certain
aspects were rated less positive. Thus, method 2 (spontaneous feedback) complements this method for
more specific insight.
Figure 13: Learning (C3DP)
For each QUIS7 category the mean rating per question was derived (see Figure 13). In addition
the mean rating, the standard deviation (STD), the distribution of ratings on a bar chart and a pie chart
were derived per question for further detail (see Figure 14). The bar-chart displayed only valid responses,
while the pie chart shows all responses, including the percentage of non-responders. The bar-charts and
pie-charts are colour coded orange for values from 0-4 and green for values from 5-9. The total number of
valid patient responses for the C3DP usability testing is n=20, all items that were rated lower than 6 in the
mean; all items that had a STD larger than 2; all items that had a non-response rate of more than 15%.
The total number of valid patient responses for the PEP usability testing is n=26, all items that were rated
lower than 5.9 in the mean; all items that had a STD larger than 2.3; all items that had a non-response rate
of more than 20%.
Figure 14: Getting started is (difficult – easy) (C3DP)
Results. Usability studies held with MDT members and patients have shown that the proposed method is
able to address the needs of care plan personalization via CDS services implementing clinical guidelines.
Testers were generally very positive about the C3-Cloud concept and experienced it as very promising,
helpful and easy to use. They have particularly liked how the tools were very clinically focused. The fact
that C3DP suggests goals and activities was experienced as being very positive and helpful. Especially the
two front-end facing components, C3DP and PEP, have benefited from feedback for improvements
during the usability and application testing. Unstructured feedback, expressed through the think-aloud
method by the test participants, has been particularly useful. Development teams have responded to the
feedback received and incorporated them in relevant tasks, such as development, deployment and
5. Discussion and Conclusions
The C3-Cloud system has been co-designed and co-produced with the end-users from the very early
stages of the project. In order to facilitate co-production, technical partners have first created user
interface mock-ups (especially for C3DP) during the requirements analysis phase, and then created
producing early prototypes starting from the architectural design phase, followed by several iterations and
reviews by the end-users till the end of integration. This process has helped a lot in achieving easy
adaptation and better acceptance of the end-users to the C3-Cloud solution. Usability studies held with the
clinical experts with simulated patient data in evaluation layers 1 and 2 have shown that the proposed
method is able to address the needs of care plan personalization via CDS services implementing clinical
guidelines. Moreover, the recommendations from these usability studies have helped to improve further
the user-facing components C3DP and PEP.
As the next step, the system will be operated and validated in real life within the scope of
evaluation layers 3 and 4 to examine usability and acceptance of personalized care plans for chronic
disease management in three pilot sites: Basque Country (Spain), Region of Jämtland Härjedalen
(Sweden) and South Warwickshire NHS Foundation Trust (UK) via a 12 months pilot study to be carried
out with approximately 62 health and social care workers including general practitioners, nurses and
specialists and 526 patients using the C3-Cloud solution. The integrated solution has already been
deployed to staging environments of all three pilot sites. Soon the final training activities with the actual
users will take place and then pilots in operational environments will be started where patients will be
directly involved.
5.1. Limitations, Challenges and Future Work
C3-Cloud is a very complex project with ambitious aims and developing a common solution for varying
settings, restrictions and needs of three different pilot sites has been quite a challenging task. We have
overcome these challenges by working closely with the end-users and their local IT teams, and loosely
coupling the C3-Cloud end-user facing components with the local site EHR/EMR systems via a
standards-based interoperability layer composed of an HL7 FHIR Repository and pilot site specific
adapters transforming the data in local formats into FHIR resources compliant with the C3-Cloud profiles.
It is a fact that, the interoperability layer requires manual activities especially for semantic mapping.
Currently, the semantic interoperability challenges have been effectively dealt with by focusing on the
specific data requirements of the targeted chronic diseases and associated CDS services implementing
clinical guidelines, and validating all the relevant but limited set of terminology code mappings by
clinical experts.
In addition to this, the reconciliation of the recommendations from multiple clinical guidelines
had been a manual work that has been carried out by our clinical expert groups. As a future work we aim
to explore semantic reasoning tools to semi-automatically detect possible clashes between the
recommendations coming from different clinical guidelines.
Finally, although the data flow and transformation from the local EHR/EMR systems into the C3-
Cloud solution is achieved completely, and the personalised care plans as the outcome of the C3-Cloud
solution can be provided back to the local systems in a widely-used international standard (HL7 FHIR), it
could not be possible to integrate the care plan goals and activities back to the local systems in the local
formats due to legal and technical constraints. As a long term future work, this will be enabled for better
exploitation of the system to third parties.
The research leading to these results has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 6891810, C3-Cloud Project.
[1] OECD, Oecd factbook (edition 2009). URL
[2] N. Goodwin, A. Dixon, G. Anderson, W. Wodchis, Providing integrated care for older people with
complex needs: Lessons from seven international case studies, Report, The Kings Fund (2014).
[3] F. Luppi, F. Franco, B. B. L. Fabbri, Treatment of chronic obstructive pulmonary disease and its
comorbidities, in: Proceedings of the American Thoracic Society, Vol. 5, 2008, p. 26.
[4] J. Piette, C. Richardson, M. Valenstein, Addressing the needs of patients with multiple chronic
illnesses: the case of diabetes and depression, Am J Manage Care 10 (2004) 15262.
[5] Integrated care models: an overview, Working document, Health Services Delivery Programme,
Division of Health Systems and Public Health, WHO Europe (2016).
[6] B. Guthrie, K. Payne, P. Alderson, M. E. T. McMurdo, S. W. Mercer, Adapting clinical guidelines
to take account of multimorbidity, BMJ 345.
arXiv:, doi:10.1136/bmj.e6341. URL
[7] L. Hughes, M. E T McMurdo, B. Guthrie, Guidelines for people not for diseases: The challenges of
applying uk clinical guidelines to people with multimorbidity, Age and ageing 42.
[8] Adherence to long term strategies-evidence from action, Working document, Chronic diseases and
health promotion, WHO (2003).
[9] L. Osterberg, T. Blaschke, Adherence to medication, New England Journal of Medicine 353 (5)
(2005) 487497, pMID: 16079372. doi:10.1056/NEJMra050100.
[10] K. Vanhaecht, The impact of clinical pathways on the organisation of care processes, Phd thesis,
University Leuven, Faculty of Medicine (2007).
[11] K. . Kawamoto, C. Houlihan, E. Balas, D. Lobach, Improving clinical practice using clinical
decision support systems: a systematic review of trials to identify features critical to success, BMJ
330 (2005) 765.
[12] M. Jaspers, M. Smeulers, H. Vermeulen, L. Peute, Effects of clinical decision-support systems on
practitioner performance and patient outcomes: A synthesis of high-quality systematic review
findings, Journal of the American Medical Informatics Association : JAMIA 18 (2011) 32734.
[13] N. M. Souza, R. J. Sebaldt, J. A. Mackay, J. C. Prorok, L. Weise-Kelly, T. Navarro, N. L.
Wilczynski, R. B. Haynes, the CCDSS Systematic Review Team, Computerized clinical decision
support systems for primary preventive care: A decision-maker-researcher partnership systematic
review of effects on process of care and patient outcomes, Implementation Science 6 (1) (2011) 87.
doi:10.1186/1748-5908-6-87. URL
[14] J. Grimshaw, M. P Eccles, J. Lavis, S. Hill, J. E Squires, Knowledge translation of research
findings, Implementation science : IS 7 (2012) 50. doi:10.1186/1748-5908-7-50.
[15] R. Lenz, R. Blaser, M. Beyer, O. Heger, C. Biber, M. Bäumlein, M. Schnabel, It support for
clinical pathwayslessons learned, International Journal of Medical Informatics 76 (2007) S397
S402, ubiquity: Technologies for Better Health in Aging Societies - MIE 2006.
[16] S. W. Tu, M. A. Musen, R. Shankar, J. Campbell, K. Hrabak, J. McClay, S. M. Huff, R. McClure,
C. Parker, R. Rocha, R. Abarbanel, N. Beard, J. Glasgow, G. Mansfield, P. Ram, Q. Ye4, E. Mays,
T. Weida, C. G. Chute, K. McDonald, D. Mohr, M. A. Nyman, S. Scheitel, H. Solbrig, D. A. Zill,
M. K. Goldstein, Modeling guidelines for integration into clinical workflow, Stud Health Technol
Inform 107 (2004) 174178.
[17] M. Peleg, Computer-interpretable clinical guidelines: a methodological review., Journal of
biomedical informatics 10. doi:10.1016/j.jbi.2013.06.009.
[18] K. Heiden, B. Böckmann, Generating evidence-based pathway models for different hospital
information systems, in: G. Huang, X. Liu, J. He, F. Klawonn, G. Yao (Eds.), Health Information
Science, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 4252.
[19] Evidence-Based Medicine Electronic Decision Support Study (EBMeDS) (cited December 2018).
[20] M. W Beckmann, H. Schlieter, P. Richter, S. Wesselmann, Considerations on the improved
integration of medical guidelines into routine clinical practice - a review and concept proposal,
Geburtshilfe und Frauenheilkunde 76 (2016) 369376. doi:10.1055/s-0042-102056.
[21] W. Ricciardi, A. de Belvis AG, M. Specchia, ’Assessing Chronic Disease Management in
European Health Systems: Country reports, Observatory Studies Series 39, European Observatory
on Health Systems and Policies, Copenhagen (Denmark) (2015). URL
[22] Medix Care Management Tool, . (2018).
[23] C3-Cloud: A Federated Collaborative Care Cure Cloud Architecture for Addressing the Needs of
Multi-morbidity and Managing Poly-pharmacy, . (2016).
[24] D. Bender, K. Sartipi, Hl7 FHIR: An agile and restful approach to healthcare information
exchange, in: Proceedings of the 26th IEEE International Symposium on Computer-Based Medical
Systems, 2013, pp. 326331. doi:10.1109/CBMS.2013.6627810.
[25] J. Grosjean, T. Merabti, B. Dahamna, I. Kergourlay, B. Thirion, L. Soualmia, S. Darmoni, Health
multi-terminology portal: a semantic added-value for patient safety, Studies in health technology
and informatics 166 (2011) 12938.
[26] onFHIR: HL7 FHIR® Based Secure Data Repository, . (2016).
[27] D. Hardt, The OAuth 2.0 authorization framework, RFC 6749, RFC Editor, Fremont, CA, USA
(Oct. 2012). URL
[28] NG28 Type 2 diabetes in adults: management, Clinical guideline, NICE (May 2017). URL
[29] CG182 Chronic kidney disease in adults: assessment and management, Clinical guideline, NICE
(2015). URL
[30] CG108 Chronic heart failure in adults: management, Clinical guideline, NICE (2010). URL
[31] CG90 Depression in adults: recognition and management, Clinical guideline, NICE (2018). URL
[32] D7.1 evidence based clinical guideline definitions and flowcharts for individual chronic conditions,
Deliverable, C3-Cloud Consortium (2016). URL
[33] D7.2 clinical decision support modules for personalised care plan development and execution,
Deliverable, C3-Cloud Consortium (2017).
[34] CDS Hooks, Technical specifications (2018). URL
[35] Guideline Definition Language v2, Technical specifications (2018). URL
[36] British National Formulary (BNF) (cited December 2018). URL
[37] D5.2 data collection and feedback mechanism, Deliverable, C3-Cloud Consortium (2017).
[38] D5.3 responsive multi-channel patient empowerment platform, Deliverable, C3-Cloud Consortium
[39] L. Traore, A. Assele-Kama, S. N. L. C. Keung, L. Karnic, G. O. Klein, M. Liljad, I. Scandurra, D.
Verdoy, M. Yuksel, T. N. Arvanitis, R. Tsopra, M.-C. Jaulent, User-centered design of the c3-
cloud platform for elderly with multiple diseases – functional requirements and application
testing, in: Accepted for publication to MEDINFO 2019, 2019.
[40] S. Weiss, Handheld Usability, John Wiley & Sons, Inc., New York, NY, USA, 2002.
[41] A. Kushniruk, E. M Borycki, J. Kannry, Commercial versus in-situ usability testing of healthcare
information systems: Towards "public" usability testing in healthcare organizations., Studies in
health technology and informatics 183 (2013) 15761. doi:10.3233/978-1-61499-203-5-157.
[42] C3-cloud research protocol v.3.0, Working document, C3-Cloud Consortium (2018).
[43] J. Benedek, T. Miner, Measuring desirability: New methods for evaluating desirability in a
usability lab setting.
[44] T. Tullis, W. Albert, Measuring the User Experience: Collecting, Analyzing, and Presenting
Usability Metrics: Second Edition, 2008.
... A collaboratively working care team is more likely to be responsive, efficient, and provide improved care [61]. As multiple behaviours interact and play a role in the provision of structured diabetes education and counselling and collaborative care of type 2 diabetes [62][63][64][65], the co-design panel and the research team targeted changing the behaviours of the health professionals (physicians, nurses, and pharmacists, and dietitians or nutritionists) to improve the care of type 2 diabetes at TASH. ...
... In a systematic review of behaviour change interventions, such as education, training, collaborative care including physicians, nurses, and pharmacists, audit and feedback targeted at health professionals were effective in improving healthcare delivery and patient outcomes [93]. Successful management and the improved outcome of diabetes requires interaction and implementation of multiple behaviours of different health professionals, such as motivation and commitment, diabetes management knowledge and skills, interprofessional or intraprofessional communications, and compassion [62][63][64][65]91,94]. As a result, modifying multiple behaviours of professionals of various disciplines helps to improve the management of diabetes and patient outcomes [62][63][64][65]94]. ...
... Successful management and the improved outcome of diabetes requires interaction and implementation of multiple behaviours of different health professionals, such as motivation and commitment, diabetes management knowledge and skills, interprofessional or intraprofessional communications, and compassion [62][63][64][65]91,94]. As a result, modifying multiple behaviours of professionals of various disciplines helps to improve the management of diabetes and patient outcomes [62][63][64][65]94]. ...
Full-text available
There has been little focus on designing tailored diabetes management strategies in developing countries. The aim of this study is to develop a theory-driven, tailored and context-specific complex intervention for the effective management of type 2 diabetes at a tertiary care setting of a developing country. We conducted interviews and focus groups with patients, health professionals, and policymakers and undertook thematic analysis to identify gaps in diabetes management. The results of our previously completed systematic review informed data collection. We used the United Kingdom Medical Research Council framework to guide the development of the intervention. Results comprised 48 interviews, two focus groups with 11 participants and three co-design panels with 24 participants. We identified a lack of structured type 2 diabetes education, counselling, and collaborative care of type 2 diabetes. Through triangulation of the evidence obtained from data collection, we developed an intervention called VICKY (patient-centred collaborative care and structured diabetes education and counselling) for effective management of type 2 diabetes. VICKY comprised five components: (1) patient-centred collaborative care; (2) referral system for patients across transitions of care between different health professionals of the diabetes care team; (3) tools for the provision of collaborative care and documentation of care; (4) diabetes education and counselling by trained diabetes educators; and (5) contextualised diabetes education curriculum, educational materials, and documentation tools for diabetes education and counselling. Implementation of the intervention may help to promote evidence-based, patient-centred, and contextualised diabetes care for improved patient outcomes in a developing country.
... Application of state-of-the-art informatics technologies are increasingly being used to create IT infrastructures that support healthcare, often providing interventions. Numerous research projects offer new capabilities, such as, implementing integrated care to provide a patient-centered services [1]. Failures in their operation may cause harm to patients, something that as part of the EU Medical Device Regulation, is considered a medical device. ...
Full-text available
It is typical for many digital health research projects to develop IT architectures that will implement integrated care services that may also deliver interventions. As part of compliance with the requirements of the regulation, the components that are considered as a medical device will need to be classified to a medical device category. This is often seen as task that may increase the business risk and a major barrier of the project, particularly during the earlier stages when not all information is available. The paper offers a method assisting with classification of such architectures in the context of the Medical Devices Rregulation, offering a structured way to identifying how the initial deliverables of a project can be used to provide assurance to the justification of the classification.
... Context A: Scenarios of use: The main scenario was the validation of clinical decision support applications, providing recommendations to patients or healthcare professionals. 33 The scope encompassed both primary and secondary care. Primary care datasets focus on a relatively low number of variables (e.g. ...
Full-text available
Digital health applications can improve quality and effectiveness of healthcare, by offering a number of new tools to users, which are often considered a medical device. Assuring their safe operation requires, amongst others, clinical validation, needing large datasets to test them in realistic clinical scenarios. Access to datasets is challenging, due to patient privacy concerns. Development of synthetic datasets is seen as a potential alternative. The objective of the paper is the development of a method for the generation of realistic synthetic datasets, statistically equivalent to real clinical datasets, and demonstrate that the Generative Adversarial Network (GAN) based approach is fit for purpose. A generative adversarial network was implemented and trained, in a series of six experiments, using numerical and categorical variables, including ICD-9 and laboratory codes, from three clinically relevant datasets. A number of contextual steps provided the success criteria for the synthetic dataset. A synthetic dataset that exhibits very similar statistical characteristics with the real dataset was generated. Pairwise association of variables is very similar. A high degree of Jaccard similarity and a successful K-S test further support this. The proof of concept of generating realistic synthetic datasets was successful, with the approach showing promise for further work.
... It remains unclear if the clinicians are undertaking care planning for the patients who benefit most from it. We believe that help from clinical decision support systems would bring care planning to the front line in general practice and lessen the number of patients with T2D without a care plan [24,25]. ...
Full-text available
Objective: To study the association of personalised care plans with monitoring and controlling clinical outcomes, prescription of cardiovascular and antihyperglycaemic medication and utilisation of primary care services in patients with type 2 diabetes (T2D). Patients: Primary care T2D outpatients from the Rovaniemi Health Centre. Setting: The municipal health centre, Rovaniemi, Finland. Design: A cross-sectional, observational, retrospective register-based study. The patients were divided into three groups: 'no care plan entries' (usual care); '1-2 care plan entries'; and '3 or more care plan entries'. Main outcome measures: Monitoring of clinical and biochemical measures, achievement of treatment targets, prescription of cardiovascular and antihyperglycemic medication, and use of primary care services. Results: A total of 5104 patients with T2D (mean age 65.5 years (SD 12.4)), of which 67% had at least one care plan entry. Compared to usual care, the establishment of a care plan (either care plan group) was associated with better monitoring of glycosylated haemoglobin A1c, low-density-lipoprotein cholesterol, systolic blood pressure (sBP), and renal function, and there was more frequent prescription of all cardiovascular and antihyperglycemic medication. Patients in either care plan group were more likely to achieve sBP target (p < 0.05). Patients without a care plan had more unplanned primary care physician contacts compared to patients in care plan groups (p < 0.001). Conclusion: Establishment of a care plan is associated with more intensive and focussed care of patients with T2D. The appropriate use of primary care resources is essential and personalised care plans may contribute to the treatment of patients with T2D.Key PointsCare planning aims to empower patients with type 2 diabetes. This study demonstrates that personalised care planning is associated withmore frequent monitoring for clinical outcomes,more frequent prescription of cardiovascular and antihyperglycemic medication andmore frequent utilisation of planned diabetes consultations when compared to usual care.
... Opportunity Threat mHealth Wide user basis of mobile phone users [49,50] Rapid growth in the number of applications supporting self-management [51][52][53] Applicable to a wide scope of diagnoses [47,53] Increased patient engagement during treatment [47,[52][53][54][55] Ethical and legal aspects [53,[56][57][58] Limited evidence of outcomes and benefits (insufficient randomised controlled trials) [47,52,56,59,60] Low interoperability and integration with existing work procedures [56] Uncertainty concerning data reliability [47,56] Declining patient self-discipline over time [52] Absence of personal contact with physician [55] Non-certified applications, large number of applications [ [36,64] Reduction of cost related to poor documentation [64,65,69] Violation of the interoperability condition [53,63,70,71] Problem with aligning operating standards with the current information exchange protocols for Big Data [72] Regulatory restraints [72][73][74] The risk of possible re-identification [74] Financial sustainability [75] Digital biomarkers Wide user base [76] Wide range of information [76] Better diagnostic and decision-making on interventions thanks to continual data collection [58,59] Developing flexible electronic materials for integrating chip technology [77,78] Bad choice of monitored attributes [59] Problems with technology validation [59] Telemedicine Lower risk of disease transmission [79][80][81] Suitable for "social distancing" [82] Reduction in hospitalization cost [83,84] Comparable or better care than that of in-person consultations [79,83,85] Elimination of the feeling of isolation during hospitalization [79] Alleviation of resource scarcity (staff, geographical location) [84,[86][87][88] Shorter waiting times [60,86] Applicable to numerous diagnoses (e.g., in psychiatry, dermatology, etc.) [60,[89][90][91][92] Limited applicability based on diagnosis [79,85] Unreliable Internet connection [79,85] Lack of training in the use of digital devices [60,79,93] Violation of interoperability between healthcare providers and healthcare systems [94] Discrimination of certain patient groups (e.g., people with particular handicaps) [80] Limited evidence of outcomes and benefits (insufficient randomised controlled trials) [60,80] Artificial intelligence (AI) ...
Full-text available
Citation: Hospodková, P.; Berežná, J.; Barták, M.; Rogalewicz, V.; Severová, L.; Svoboda, R. Change Management and Digital Innovations in Hospitals of Five European Countries. Healthcare 2021, 9, 1508. https:// Abstract: The objective of the paper is to evaluate the quality of systemic change management (CHM) and readiness for change in five Central European countries. The secondary goal is to identify trends and upcoming changes in the field of digital innovations in healthcare. The results show that all compared countries (regardless of their historical context) deal with similar CHM challenges with a rather similar degree of success. A questionnaire distributed to hospitals clearly showed that there is still considerable room for improvement in terms of the use of specific CHM tools. A review focused on digital innovations based on the PRISMA statement showed that there are five main directions, namely, data collection and integration, telemedicine, artificial intelligence, electronic medical records, and M-Health. In the hospital environment, there are considerable reservations in applying change management principles, as well as the absence of a systemic approach. The main factors that must be monitored for a successful and sustainable CHM include a clearly defined and widely communicated vision, early engagement of all stakeholders, precisely set rules, adaptation to the local context and culture, provision of a technical base, and a step-by-step implementation with strong feedback.
... We identified eight studies meeting the inclusion criteria for care coordination or care planning including one RCT (Table 2). [90][91][92][93][94][95][96][97] The RCT evaluated an intervention that assessed needs and goals of patients and created a proactive plan of care. 94 97 The health IT component of the intervention was a computerized note template to improve clinician-patient communication. ...
Full-text available
Objective: To review evidence regarding the use of Health Information Technology (health IT) interventions aimed at improving care for people living with multiple chronic conditions (PLWMCC) in order to identify critical knowledge gaps. Data sources: We searched MEDLINE, CINAHL, PsycINFO, EMBASE, Compendex, and IEEE Xplore databases for studies published in English between 2010-2020. Study design: We identified studies of health IT interventions for PLWMCC across three domains: self-management support, care coordination, and algorithms to support clinical decision-making. Data collection/extraction methods: Structured search queries were created and validated. Abstracts were reviewed iteratively to refine inclusion and exclusion criteria. The search was supplemented by manually searching the bibliographic sections of the included studies. The search included a forward citation search of studies nested within a clinical trial to identify the clinical trial protocol and published clinical trial results. Data was extracted independently by two reviewers. Principal findings: The search yielded 1907 articles; 44 were included. Nine randomized controlled trials (RCTs) and 35 other studies including quasi-experimental, usability, feasibility, qualitative studies, or development/validation studies of analytic models. Five RCTs had positive results and the remaining four RCTs showed that the interventions had no effect. The studies address individual patient engagement and assess patient-centered outcomes such as quality of life. Few RCTs assess outcomes such as disability and none assess mortality. Conclusions: Despite a growing body of literature on health IT interventions or multicomponent interventions including a health IT component for chronic disease management, current evidence for applying health IT solutions to improve care for PLWMCC is limited. The body of literature included in this review provides critical information on the state of the science as well as the many gaps that need to be filled for digital health to fulfill its promise in supporting care delivery that meets the needs of PLWMCC. This article is protected by copyright. All rights reserved.
... C3-Cloud is an e-health ICT system, offering integrated, patient-centered care, considering all aspects of multi-morbidity, creating a collaborative environment for all involved stakeholders [5]. C3-Cloud was deployed in 3 pilot sites in: Basque country, Spain; region Jämtland Härjedalen, Sweden; and Warwickshire, UK. ...
Full-text available
Multimorbid patients are prescribed a number of medications in parallel, which may often interact with each other, resulting in adverse effects. However, clinical guidelines on prescription of medications predominantly focus on individual conditions do not consider the guidance in the context of other guidelines, resulting in conflicts. C3-Cloud is an integrated care architecture managing multimorbidity, which amongst others, provides clinical decision support, based on reconciled guidelines, and active monitoring of drug interactions. To identify the severe interactions that resulted from multimorbidity management, in order to reevaluate guidelines as well as to identify knowledge gaps in prescribing practice. Method: Descriptive statistical analysis of interactions identified by the C3-Cloud clinical decision support, collected from the C3-Cloud FHIR repository. As part of a feasibility study, a number of interactions were identified, along with variable practice in how chemicals are represented in the EHR. 191 known severe interactions were identified. The Atorvastatin/Verapamil interaction was the most frequent. The approach has identified a number of interactions where the severity was not available, highlighting the need for further clinical review.
... In order to suggest personalised recommendations to individualised care plans, Laleci et al. [33] developed a semi-automatic care plan management tool, called C3-Cloud, integrated with clinical decision support services. Jakob Nielsen's walk-through method was used to measure the system's usability, using an iterative and holistic approach, through heuristic evaluation, spontaneous feedback, reaction to the product on cards and questionnaire for user interaction satisfaction (QUIS). ...
Background: Clinical decision support systems (CDSSs) are developed to support healthcare practitioners with decision-making about therapy and diagnosis’ confirmation, among others. Although there are many advantages of using CDSSs, there are still many challenges in their adoption. Therefore, it is essential to ensure the quality of the system, so that it can be used confidently and securely. Objective: This study aims to propose a set of (sub)characteristics which should be considered in evaluating the quality-in-use of CDSSs, based on the ISO/IEC 25010 standard and on existing literature. Methods: We reviewed the existing literature on CDSS assessment and presented a list of quality characteristics evaluated. Results: Ten quality characteristics and 56 sub-characteristics were identified and selected from the literature, in which usability was evaluated the most. An example of a scenario has been presented to illustrate our assessment approach of satisfaction and efficiency as important quality-in-use characteristics to be applied in the evaluation of a CDSS. Conclusion: The proposed approach will contribute in bridging the gap between the quality of CDSSs and their adoption.
In case of comorbidity, i.e., multiple medical conditions, Clinical Decision Support Systems (CDSS) should issue recommendations based on all relevant disease-related Clinical Practice Guidelines (CPG). However, treatments from multiple comorbid CPG often interact adversely (e.g., drug-drug interactions) or introduce operational inefficiencies (e.g., redundant scans). A common solution is the a-priori integration of computerized CPG, which involves integration decisions such as discarding, replacing or delaying clinical tasks (e.g., treatments) to avoid adverse interactions or inefficiencies. We argue this insufficiently deals with execution-time events: as the patient's health profile evolves, acute conditions occur, and real-time delays take place, new CPG integration decisions will often be needed, and prior ones may need to be reverted or undone. Any realistic CPG integration effort needs to further consider temporal aspects of clinical tasks—these are not only restricted by temporal constraints from CPGs (e.g., sequential relations, task durations) but also by CPG integration efforts (e.g., avoid treatment overlap). This poses a complex execution-time challenge and makes it difficult to determine an up-to-date, optimal comorbid care plan. We present a solution for dynamic integration of CPG in response to evolving health profiles and execution-time events. CPG integration policies are formulated by clinical experts for coping with comorbidity at execution-time, with clearly defined integration semantics that build on Description and Transaction Logics. A dynamic planning approach reconciles temporal constraints of CPG tasks at execution-time based on their importance, and continuously updates an optimal task schedule.
Full-text available
The number of patients with multimorbidity has been steadily increasing in the modern aging societies. The European C3-Cloud project provides a multidisciplinary and patient-centered "Collaborative Care and Cure-system" for the management of elderly with multimorbidity, enabling continuous coordination of care activities between multidisciplinary care teams (MDTs), patients and informal caregivers (ICG). In this study various components of the infrastructure were tested to fulfill the functional requirements and the entire system was subjected to an early application testing involving different groups of end-users. MDTs from participating European regions were involved in requirement elicitation and test formulation, resulting in 57 questions, distributed via an internet platform to 48 test participants (22 MDTs, 26 patients) from three pilot sites. The results indicate a high level of satisfaction with all components. Early testing also provided feedback for technical improvement of the entire system, and the paper points out useful evaluation methods.
Full-text available
This publication provides insights into the health system context for chronic care in twelve European countries. It looks at the range of care models that have been implemented to better meet the needs of people with long-term health conditions. There is indeed growing recognition of the need to innovate service delivery in order to better bridge the boundaries between professions, providers and institutions. As such initiatives vary from country to country – and even from region to region – this study systematically examines these diverse experiences, using an explicit comparative approach and a unified framework for assessment. Through detailed accounts of the experiences across European countries in their efforts to enhance care for people with chronic conditions, this book tries to provide a better understanding of the range of contexts in which these new approaches to chronic care are implemented and tries to evaluate the outcomes of these initiatives. The content of these new models, which are frequently applied from different disciplinary and professional perspectives, and associated with different goals, are analysed in more detail, including approaches to self-management support, service delivery design and decision-support strategies, financing, availability and access. Significantly, it also illustrates the challenges faced by individual patients as they pass through the system. As this book complements the earlier published study Assessing Chronic Disease Management in European Health Systems it also builds on the findings of the DISMEVAL project (Developing and validating DISease Management EVALuation methods for European health care systems), led by RAND Europe and funded under the European Union's (EU) Seventh Framework Programme (FP7) (Agreement no. 223277).
Conference Paper
Full-text available
This research examines the potential for new Health Level 7 (HL7) standard Fast Healthcare Interoperability Resources (FHIR, pronounced “fire”) standard to help achieve healthcare systems interoperability. HL7 messaging standards are widely implemented by the healthcare industry and have been deployed internationally for decades. HL7 Version 2 (“v2”) health information exchange standards are a popular choice of local hospital communities for the exchange of healthcare information, including electronic medical record information. In development for 15 years, HL7 Version 3 (“v3”) was designed to be the successor to Version 2, addressing Version 2's shortcomings. HL7 v3 has been heavily criticized by the industry for being internally inconsistent even in it's own documentation, too complex and expensive to implement in real world systems and has been accused of contributing towards many failed and stalled systems implementations. HL7 is now experimenting with a new approach to the development of standards with FHIR. This research provides a chronicle of the evolution of the HL7 messaging standards, an introduction to HL7 FHIR and a comparative analysis between HL7 FHIR and previous HL7 messaging standards.
Full-text available
Clinical practice guidelines (CPGs) aim to improve the quality of care, reduce unjustified practice variations and reduce healthcare costs. In order for them to be effective, clinical guidelines need to be integrated with the care flow and provide patient-specific advice when and where needed. Hence, their formalization as computer-interpretable guidelines (CIGs) makes it possible to develop CIG-based decision-support systems (DSSs), which have a better chance of impacting clinician behavior than narrative guidelines. This paper reviews the literature on CIG-related methodologies since the inception of CIGs, while focusing and drawing themes for classifying CIG research from CIG-related publications in the Journal of Biomedical Informatics (JBI). The themes span the entire life-cycle of CIG development and include: knowledge acquisition and specification for improved CIG design, including (1) CIG modeling languages and (2) CIG acquisition and specification methodologies, (3) integration of CIGs with electronic health records (EHRs) and organizational workflow, (4) CIG validation and verification, (5) CIG execution engines and supportive tools, (6) exception handling in CIGs, (7) CIG maintenance, including analyzing clinician's compliance to CIG recommendations and CIG versioning and evolution, and finally (8) CIG sharing. I examine the temporal trends in CIG-related research and discuss additional themes that were not identified in JBI papers, including existing themes such as overcoming implementation barriers, modeling clinical goals, and temporal expressions, as well as futuristic themes, such as patient-centric CIGs and distributed CIGs.
Full-text available
Care of patients with multimorbidity could be improved if new technology is used to bring together guidelines on individual conditions and tailor advice to each patient’s circumstances, say Bruce Guthrie and colleagues Most people with a chronic condition have multimorbidity, but clinical guidelines almost entirely focus on single conditions. It will never be possible to have good evidence for every possible combination of conditions, but guidelines could be made more useful for people with multimorbidity if they were delivered in a format that brought together relevant recommendations for different chronic conditions and identified synergies, cautions, and outright contradictions. We highlight the problem that multimorbidity poses to clinicians and patients using guidelines for single conditions and propose ways of making them more useful for people with multimorbidity. Guidelines have the potential to improve the care of people with chronic disease1 but seldom explicitly account for people with multiple conditions. This reflects the way in which clinical evidence is created but does not match everyday practice, where multimorbidity is common. The figure⇓ illustrates this using data from UK primary care electronic health records taken from a study of the prevalence of multimorbidity in 1.75 million people.2 Most people with any chronic condition have multiple conditions, and although the degree of multimorbidity increases with age, this applies to younger patients as well, particularly those living in the most socioeconomically deprived areas, where multimorbidity develops 10-15 years earlier than in more affluent areas.2 Comorbidity of 10 common conditions among UK primary care patients2 Clinical decision making is more difficult in people with multimorbidity because clinicians and patients often struggle to balance the benefits and risks of multiple recommended treatments3 and because patient preference rightly influences the application of clinical and economic evidence.4 Robust synthesis of clinical and …
Full-text available
Background: currently one of the major challenges facing clinical guidelines is multimorbidity. Current guidelines are not designed to consider the cumulative impact of treatment recommendations on people with several conditions, nor to allow comparison of relative benefits or risks. This is despite the fact that multimorbidity is a common phenomenon. Objective: to examine the extent to which National Institute of Health and Clinical Excellence (NICE) guidelines address patient comorbidity, patient centred care and patient compliance to treatment recommendations. Methods: five NICE clinical guidelines were selected for review (type-2 diabetes mellitus, secondary prevention for people with myocardial infarction, osteoarthritis, chronic obstructive pulmonary disease and depression) as these conditions are common causes of comorbidity and the guidelines had all been produced since 2007. Two authors extracted information from each full guideline and noted the extent to which the guidelines accounted for patient comorbidity, patient centred care and patient compliance. The cumulative recommended treatment, follow-up and self-care regime for two hypothetical patients were then created to illustrate the potential cumulative impact of applying single disease recommendations to people with multimorbidity. Results: comorbidity and patient adherence were inconsistently accounted for in the guidelines, ranging from extensive discussion to none at all. Patient centred care was discussed in generic terms across the guidelines with limited disease-specific recommendations for clinicians. Explicitly following guideline recommendations for our two hypothetical patients would lead to a considerable treatment burden, even when recommendations were followed for mild to moderate conditions. In addition, the follow-up and self-care regime was complex potentially presenting problems for patient compliance. Conclusion: clinical guidelines have played an important role in improving healthcare for people with long-term conditions. However, in people with multimorbidity current guideline recommendations rapidly cumulate to drive polypharmacy, without providing guidance on how best to prioritise recommendations for individuals in whom treatment burden will sometimes be overwhelming.
Medical guidelines have become established as the standard for the comprehensive synopsis of all available information (scientific trials, expert opinion) on diagnosis and treatment recommendations. The transfer of guidelines to clinical practice and subsequent monitoring has however proven difficult. In particular the potential interaction between guideline developers and guideline users has not been fully utilised. This review article analyses the status quo and existing methodological and technical information solutions supporting the guideline life cycle. It is shown that there are numerous innovative developments that in isolation do not provide comprehensive support. The vision of the "Living Guidelines 2.0" is therefore presented. This outlines the merging of guideline development and implementation on the basis of clinical pathways and guideline-based quality control, and building on this, the generation of information for guideline development and research.
Conference Paper
Healthcare providers are facing an enormous cost pressure and a scarcity of resources, so that they need to realign in the tension between economic efficiency and demand-oriented healthcare. Clinical guidelines and clinical pathways have been established in order to improve the quality of care and to reduce costs at the same time. Clinical guidelines provide evident medical knowledge for diagnostic and therapeutic issues, while clinical pathways are a road map of patient management. The consideration of clinical guidelines during pathway development is highly recommended. But the transfer of evident knowledge (clinical guidelines) to care processes (clinical pathways) is not straightforward due to different information contents and semantical constructs. This article proposes a model-driven approach to support the development of guideline-compliant pathways and focuses the generation of ready-to-use pathway models for different hospital information systems. That way, best practice advices provided by clinical guidelines can be provided at the point of care and therefore improve patient care.
The need for improved usability in healthcare IT has been widely recognized. In addition, methods from usability engineering, including usability testing and usability inspection have received greater attention. Many vendors of healthcare software are now employing usability testing methods in the design and development of their products. However, despite this, the usability of healthcare IT is still considered to be problematic and many healthcare organizations that have purchased systems that have been tested at vendor testing sites are still reporting a range of usability and safety issues. In this paper we explore the distinction between commercial usability testing (conducted at centralized vendor usability laboratories and limited beta test sites) and usability testing that is carried out locally within healthcare organizations that have purchased vendor systems and products (i.e. public "in-situ" usability testing). In this paper it will be argued that both types of testing (i.e. commercial vendor-based testing) and in-situ testing are needed to ensure system usability and safety.