![The knowledge intensive-CBR cycle, taken from [12].](https://www.researchgate.net/profile/Pariya-Kashfi/publication/278018428/figure/fig12/AS:361180251082766@1463123490891/The-knowledge-intensive-CBR-cycle-taken-from-12_Q320.jpg)
Content uploaded by Pariya Kashfi
Author content
All content in this area was uploaded by Pariya Kashfi on May 13, 2016
Content may be subject to copyright.
Thesis for the degree of Licentiate of Engineering
Towards Usable open EHR-aware
Clinical Decision Support:
A User-centered Design Approach
Hajar Kashfi
Division of Interaction Design
Department of Applied Information Technology
CHALMERS UNIVERSITY OF TECHNOLOGY
Gothenburg, Sweden 2011
Towards Usable openEHR-aware Clinical Decision Support:
A User-centered Design Approach
Hajar Kashfi
c
Hajar Kashfi, 2011.
ISSN 1651-4769;7
Department of Applied Information Technology
Chalmers University of Technology
SE–412 96 Gothenburg, Sweden
Phone: +46 (0)31–772 1000
Contact information:
Hajar Kashfi
Division of Interaction Design
Department of Applied Information Technology
Chalmers University of Technology
SE–412 96 Gothenburg, Sweden
Phone: +46 (0)31–772 5407
Fax: +46 (0)31–772 3663
Email: hajar.kashfi@chalmers.se
URL: http://puix.org
Printed in Sweden
Chalmers Reproservice
Gothenburg, Sweden 2011
To my love, Mohsen
and my wonderful parents and brothers
Towards Usable open EHR-aware Clinical
Decision Support:
A User-centered Design Approach
Hajar Kashfi
Department of Applied Information Technology,
Chalmers University of Technology
Abstract
Nowadays, the use of computerized approaches to support health care processes
in order to improve quality of health care is widespread in the clinical domain.
Electronic health records (EHR) and clinical decision support (CDS) are con-
sidered to be two complementary approaches to improve quality of health care.
It is shown that EHRs are not able to improve quality of health care without
being supported by other features such as CDS. On the other hand, one of the
success factors of CDS is its integration into EHR, and since there are various
international EHR standards (such as openEHR) being developed, it is crucial
to take these standards into consideration while developing CDS.
Various clinical decision support systems (CDSS) are developed but unfortu-
nately only a few of them are being used routinely. Two of the reasons for
unacceptability of CDSSs among their users, i.e. clinicians, are shown to be
their separation from EHRs and poor usability of the user interfaces. Besides
integration into underlying information framework, i.e. EHR systems, consider-
ation of human-computer interaction (HCI) in designing and evaluating CDS is
one of the success factors that developers of these systems should keep in mind.
This thesis addresses the question of how usable openEHR-aware clinical deci-
sion support can be designed and developed in order to improve the quality of
health care. To answer this research question, several sub-questions were identi-
fied and investigated. This included analyzing “state of the art” in two different
aspects of design and development and evaluation of CDS and also investigating
application of a customized user-centered design (UCD) process in developing
openEHR-based clinical applications.
Analysis of state of the art in interplay between HCI and CDS and also the
intersection between CDS and EHR revealed that consideration of both HCI
and integration of CDS into EHR is more appreciated in theory than in practice
and there is still a long way to go before reaching an acceptable level in these
two success factors of CDS.
Moreover, the experience in designing an openEHR-based clinical application re-
vealed that apart from benefits offered by openEHR approach, such as specifying
different roles and involvement of domain experts in defining domain concepts,
there are various shortcomings that need to be improved, for instance the limited
support for openEHR application developers. Additionally, this study revealed
that there are characteristics of the domain, tasks and users in the domain that
developers should be informed about while applying UCD methods.
Keywords: Medical informatics, Electronic health record, openEHR, Clinical
decision support system, User-centered design, Human-computer interaction,
Interaction design, Usability.
Acknowledgments
I would like to express my gratitude to all of the individuals whose support and
help has made this research a reality; especially my supervisor and advisor, Olof
Torgersson. My gratitude also extends to G¨oran Falkman, Mats Jontell, Marie
Lindgren, Marita Nilsson, and the members of the Clinic of Oral Medicine at
Sahlgrenska University Hospital who were involved in this project.
I also want to thank Sus Lundgren, Martin Hjulstr¨om, Erik Fagerholt and Soren
Lauesen who provided constructive feedback on the design work presented in
this thesis. I am also indebted to Marie Gustafsson Friberger who provided
valuable comments on the thesis. Moreover, I appreciate the opportunities for
helpful discussions provided by the openEHR community members, especially
Ian McNicoll, Koray Atalag, Pablo Pazo, Rong Chen, Seref Arikan, and others
whose names may have been omitted here.
I would also like to convey my gratitude to my colleagues in the Interaction
Design Division, in particular Staffan Bj¨ork and Fang Chen, and members of
the Human-Technology Design research school of which I too am a member. Of
course, I would be remiss not to mention my kind fellow graduate student, Anna
Gryszkiewicz, with whom I share a pleasant and peaceful office.
My friends know that they all hold a very special place in my heart and I am
grateful to them for all of the joy they have brought into my life over the past
few years. But my wonderful family–my parents and brothers–know that they
are and always will be a part of my heart. I hope that the realization that
their love, patience, and willingness to bear the burden of our separation, has
empowered me to make my dreams come true will allow them to look upon my
achievements here with pride.
And finally, thanks to you, Mohsen, my love, and the meaning of my life. You
are the best and happiest thing that has ever happened to me. You are not only
a marvelous life partner, but also a tremendous friend and a remarkable fellow
graduate student. I appreciate the fruitful discussions we have had together and
the feedback you provided on this thesis. I am grateful to you for being such a
cheerful and supportive soul mate through both sunshine and rain, as well as all
of the other moments that will forever remain between you and I. Thank you for
being my everything and everyone–your love is the axis of my entire being.
Hajar Kashfi
Gothenburg, May 2011
List of Appended Papers
This thesis is a summary of the following four papers. References to the papers
will be made using the Roman numbers associated with the papers.
IHajar Kashfi, “Towards Interaction Design in Clinical Decision Support
Development: A Literature Review,” submitted to International Journal
of Medical Informatics, Elsevier
II Hajar Kashfi, “The Intersection of Clinical Decision Support and Elec-
tronic Health Record: A Literature Review,” submitted to 1st International
Workshop on Interoperable Healthcare Systems (IHS2011) - Challenges,
Technologies, and Trends, Szczecin, Poland, September 19-21, 2011.
III Hajar Kashfi, “Applying a User Centered Design Methodology in a Clini-
cal Context,” in Proceedings of The 13th International Congress on Medical
Informatics (MedInfo2010), Studies in health technology and informatics,
2010 Jan ;160(Pt 2):927-31.
IV Hajar Kashfi, “Applying a User-centered Approach in Designing a Clin-
ical Decision Support System,” submitted to Computer Methods and Pro-
grams in Biomedicine, Elsevier.
VHajar Kashfi, Olof Torgersson, “Supporting openEHR Java Desktop Ap-
plication Developers,” to appear in The XXIII International Conference
of the European Federation for Medical Informatics, Proceedings of Medi-
cal Informatics in a United and Healthy Europe (MIE2011),Oslo, Norway,
28-31 August, 2011.
VI Hajar Kashfi, Jairo Robledo Jr., “Towards a Case-Based Reasoning Method
for openEHR-Based Clinical Decision Support,” to appear in Proceed-
ings of The 3rd International Workshop on Knowledge Representation for
Health Care (KR4HC’11), Bled, Slovenia, 6 July, 2011.
i
List of Other Papers
IHajar Kashfi, “An openEHR-Based Clinical Decision Support System:
A Case Study,” in The XXII International Conference Of The European
Federation For Medical Informatics, Proceedings of Medical Informatics in
a United and Healthy Europe (MIE2009), Studies in health technology and
informatics. 2009. p. 348.
II Hajar Kashfi and Olof Torgersson, “A Migration to an openEHR-Based
Clinical Application,” in The XXII International Conference Of The Eu-
ropean Federation For Medical Informatics, Proceedings of Medical Infor-
matics in a United and Healthy Europe (MIE 2009), Studies in health
technology and informatics. 2009. p. 152-6.
ii
Contents
I Introduction
1 Introduction 3
1.1 Overview ............................... 4
2 Frame of Reference 5
2.1 Medical Informatics . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Electronic Health Record . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 The need for Interoperability . . . . . . . . . . . . . . . . 6
2.3 openEHR ............................... 8
2.3.1 Two-level Modeling . . . . . . . . . . . . . . . . . . . . . 8
2.3.2 Two-level Software Engineering . . . . . . . . . . . . . . . 8
2.3.3 The Reference Model . . . . . . . . . . . . . . . . . . . . . 9
2.3.4 Archetype........................... 9
2.3.5 Template ........................... 10
2.4 Other EHR Standardization Approaches . . . . . . . . . . . . . . 10
2.4.1 The CEN/ISO EN13606 standard . . . . . . . . . . . . . 11
2.4.2 The Governmental Computerized Patient Record Project
(G-CPR) ........................... 11
2.4.3 Health Level 7 (HL7) . . . . . . . . . . . . . . . . . . . . 11
2.5 DecisionSupport........................... 11
2.6 Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . 13
3 The Research Process 15
3.1 Conceptual framework . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Objectives............................... 19
4 Methods and Tools 21
4.1 LiteratureReview .......................... 21
4.2 The Research Methodology . . . . . . . . . . . . . . . . . . . . . 21
4.2.1 Design and Creation Research Strategy . . . . . . . . . . 21
4.3 User-Centered Design Process . . . . . . . . . . . . . . . . . . . . 23
4.4 Assumptions ............................. 25
iii
4.5 Archetype Development Process . . . . . . . . . . . . . . . . . . 26
5 Summary of the Attached Papers 30
5.1 PaperI ................................ 30
5.2 PaperII................................ 31
5.3 PaperIII ............................... 31
5.4 PaperIV ............................... 32
5.5 PaperV................................ 32
5.6 PaperVI ............................... 32
6 Thesis Contributions 34
6.1 TheAnswertoRQ1 ......................... 34
6.2 TheAnswertoRQ2 ......................... 35
6.3 TheAnswertoRQ3 ......................... 36
6.4 TheAnswertoRQ4 ......................... 39
7 Future Work 40
8 Conclusion 41
Bibliography 47
II Publications
Paper I 51
Paper II 75
Paper III 91
Paper IV 105
Paper V 139
Paper VI 147
iv
Part I
Introduction
Chapter 1
Introduction
Errors that occur in a clinical process are mostly due to cognitive limitations of
humans, the potential to forget knowledge in the health care flow, and difficulties
in clinical workflows [1, 2]. Clinicians are prone to making errors, especially
because of the limitations of the working memory [3].
Various avoidable errors or adverse events in health care are documented in
the literature. These errors and events have even lead to patients’ deaths in
some cases. About 12 preventable deaths per million inhabitants in Sweden
are reported [4]. According to the Swedish Parliament, around 100,000 patients
suffer from preventable adverse clinical events each year and 3000 of these adverse
events lead to patient deaths [5]. Preventable medical errors resulted in deaths
of up to 98,000 people in the United States in 1999 [6]. Around 11% of the
patients admitted to two hospitals in London, experienced adverse events; 48%
of which were preventable and 8% of which resulted in patient deaths [7].
There have been investigations about the quality of care in various countries.
The existing quality problems belong to three different categories, namely “un-
deruse” (failure to provide the best expected health care service), “overuse”
(providing a health care service that is more harmful than being beneficial for
the patient), or “misuse” (unsuccessful delivery of the best expected health care
service because of some preventive complications), which occur in both small
and large health organizations [8].
Obviously, there are many debates surrounding the quality of health care, but
one should start with defining the meaning of “quality” in this domain. A widely
accepted and robust definition of quality is the definition developed in 1990 by
the Institute of Medicine (IOM) as “the degree to which health services for indi-
viduals and populations increase the likelihood of desired health outcomes and
are consistent with current professional knowledge” [9]. According to Graham
[10] “quality must be judged as good if care, at the time it was given, conformed
to the practice that could have been expected to achieve the best results.”
It has been demonstrated that information systems have the ability to decrease
avoidable clinical errors by supporting clinicians in health care process or in other
3
words to improve the quality of care [11]. Medical informatics is the research
field that is dealing with this matter. Electronic health records (EHR) have been
the leading research focus in this field so far [12, 13, 14]. The EHR research field
deals with issues such as capturing, storing, retrieving and sharing patient data.
For EHRs to be able to improve quality of care, they should be supported by
clinical decision support (CDS) services [15, 14, 16], those services that aid clin-
icians in the process of decision making. Nonetheless, not all CDS developments
have led to improving the clinical practice [17]. Hunt et al. [18] indicate that
66% of the CDS implementations have led to significant improvement in health
care while the remaining 34% did not. Various efforts have been made in order
to identify barriers to low adoption, acceptability and ineffectiveness of CDS
. Efforts have also been made to identify success factors in developing them
[13, 17, 19, 20, 21]. Some of those success factors are the level of integration
with clinicians’ workflow [13, 20, 17, 22, 23, 24], the degree of patient-specificity
[13, 17, 21], availability at the point of care or timely access [13, 17]. Accord-
ingly, two of the main factors that have a direct relation to success of the CDS
are integration of the CDS to EHR systems and proper design of CDS by taking
human-computer interaction (HCI) into consideration.
While there have been various recommendations regarding consideration of these
two success factors in development of CDSSs, related literature suggests that
these factors are being partially or totally ignored in many of the projects.
1.1 Overview
This thesis investigates the question of how usable openEHR-aware clinical de-
cision support can be designed and developed in order to improve the quality of
health care. In order to answer this question, several sub-problems were identified
to be investigated. The study has been carried out in oral medicine. However,
the outcome of the research should be applicable to medical informatics in gen-
eral. The structure of the thesis is as follows. The frame of reference of this
research is presented in Chapter 2. This chapter includes the definition of differ-
ent concepts basic to this research namely medical informatics, EHR, openEHR,
CDS, HCI, usability and user-centered design (UCD).
The research process and the conceptual framework are discussed in Chapter 3.
The research questions and objectives of this study are presented in this chapter
as well. Methods and tools are introduced in Chapter 4. A summary of the
included papers is given in Chapter 5, along with how they can be put in relation
to each other and the research questions. Some directions for future work are
discussed in Chapter 7. Finally, a conclusion is provided in Chapter 8.
4
Chapter 2
Frame of Reference
2.1 Medical Informatics
Medical informatics is defined as a scientific discipline “concerned with the sys-
tematic processing of data, information and knowledge in medicine and health
care” [25, 26]. This domain covers both “computational” and “informational”
aspects of the processes in the clinical domain [26]. Medical informatics deals
with providing solutions for problems of data, information and knowledge pro-
cessing in medicine and health care [25]. As a discipline, medical informatics
has been around for more than 50 years [12] but still is called young especially
compared to other medical disciplines [27]. Nowadays, new names are suggested
for this discipline such as health informatics and clinical informatics since the
word “medical” does not cover nursing informatics, dental informatics and so on
[12].
There are various research areas in the field of medical informatics namely elec-
tronic health record (EHR) systems, information systems, decision support sys-
tems, and image and signal processing [12]. EHRs have been the leading research
focus in this field so far [12, 13, 14]. The EHR research field deals with issues
such as capturing, storing, retrieving and sharing patient data. This has led to a
number of benefits such as reduced number of transportation errors, higher leg-
ibility of reports, and avoiding redundancy [13]. These benefits indirectly affect
patient safety, health care quality and efficiency [13]. Recently, there has been
more and more interest in adoption of EHRs and developing clinical applications
based on EHRs [13, 15, 14].
The idea of benefiting from computers and information technology (IT) in the
clinical domain has been around since the 1950s [28] (or the 1960s as observed
by [13]) when there were initiatives to automate health care and to simulate
the clinical decision making by computers [13, 24]. One of the turning points
of medical informatics is considered to be around 50 years ago when in 1959,
Ledley and Lusted reported on reasoning foundations of medical diagnosis [29].
Even though the concept of atomization of health care and application of com-
5
puters and IT in this domain is an old trade, it has a slow adoption pace and
low impact level compared to other domains such as engineering, marketing,
etc. In other words, health care has fallen behind other disciplines in applying
information technology to improve the processes and outcomes [13, 14].
As mentioned above, since the introduction of computers in the clinical domain
in the past decades, the main progress in this area has been in coping with
information management, i.e. adopting EHRs [13, 14, 12] rather than adopting
CDS in order to improve the decision making process. One of the aims of the
efforts in the area of EHR has been to improve quality of health care but it is
doubted whether EHRs have the ability to fulfill this goal [15]. More information
about EHRs is presented in the following section.
2.2 Electronic Health Record
The idea of computerized medical records has been around as one of the key
research areas in medical informatics for more than 20 years. Iakovidis [30]
defines an EHR as “digitally stored health care information about an individ-
ual’s lifetime with the purpose of supporting continuity of care, education and
research, and ensuring confidentiality at all times”. EHRs include the whole
range of patient-related data such as demographic information, medical history,
medications, and allergies [31].
The main aim of EHRs is to make distributed and cooperating health information
systems and health networks come true [31]. The first benefit of deploying EHRs
is that patients’ information is no longer on a piece of paper and clinicians have
access to all patients’ information when required [13].
Since the introduction of EHRs, various projects were initiated that led to de-
velopment of various commercial EHR products. Nowadays, there are more and
more EHR systems being developed. The interest is also increasing at the gov-
ernmental level in different countries such as the UK and Sweden. However, the
EHRs adoption rate is still low in community hospitals and office practices, while
higher in academic medical centers [13]. The maximum adoption of EHRs in The
United States is demonstrated to be only 40% [32]. In those countries in which
there exists a national health care plan, this rate is considered to be higher [13].
Several reasons have been identified for the low adoption rate of EHRs in small
hospitals and office practices, viz. high implementation and maintenance costs,
additional time and effort and finally the difficulty in choosing among available
systems in the market due to a lack of standardization [13].
2.2.1 The need for Interoperability
To to be able to fully benefit from EHRs, timely and secure access to all of
the EHR systems should be ensured, EHRs should be up-to-date and accurate
in terms of information they contain, and they should be correctly understood
when being communicated [33]. This means that EHR systems should be inter-
6
operable. EHRs are stored in various formats in different products which yields
interoperability problems in the domain. Therefore, developing national and
international EHR standards is important to support interoperability [34].
Before going into details of the approaches suggested to enhance interoperability
of EHRs, it is proper to present a definition of interoperability. Interoperability
of health systems is defined as “the ability, facilitated by ICT applications and
systems, to exchange, understand and act on citizens/patients and other health-
related information and knowledge among linguistically and culturally disparate
health professionals, patients and other actors and organizations within and
across health system jurisdictions in a collaborative manner” [33]. Four levels of
interoperability are defined by Stroetmann [33]:
1. having no interoperability
2. technical and syntactical interoperability
3. partial semantic interoperability
4. full semantic interoperability
A challenging aim regarding EHRs has been to reach semantic interoperability in
EHR systems. Interest in this issue in particular is increasing in the EU recently
with the aim of reaching semantic interoperability at regional, national and even
the EUR level [33].
So far, several efforts have been made to develop EHR standards in order to
structure and exchange patient information and to enable semantic interoper-
ability among medical information systems. The main approaches are as follows:
•The European Committee for Standardization (CEN) Electronic Health
care Record communication standard (CEN/ISO EN13606)
•The Governmental Computerized Patient Record project
•The Health Level 7 (HL7) Reference Information Model and its clinical
document architecture
•The GEHR approach
•The openEHR approach which is a continuation to GEHR
All the above approaches focus on the technical issues related to standardized
and interoperable EHRs. More information about these approaches is provided
in the following sections. The EHR interoperability standard that is applied
in this study is openEHR. According to openEHR website 1“the Swedish gov-
ernment has decided on the use of ISO 13606 as a base standard for national
health data communication. openEHR will be used to define clinical models,
terminology integration, and to implement 13606 in some contexts.” ISO/CEN
1http://www.openehr.org
7
13606 resembles the openEHR reference model (See Section 2.3.2) in a simplified
manner [35] (ISO/CEN 13606 is explained in Section 2.4.1). This has been a
huge motivation for us to consider openEHR as our EHR approach to carry out
this study.
2.3 open EHR
openEHR has its origins in 1992 in an EU research project named Good Euro-
pean Health Record. This project was later continued under the name of Good
Electronic Health Record (GEHR) [36]. Currently the maintenance of openEHR
is done by a non-profit organization named the openEHR Foundation [35].
In the openEHR approach, clinicians are in charge of defining the specifications
of clinical knowledge to be used in information modeling. The main emphasis of
openEHR is on semantic interoperability of medical records. This approach sug-
gests a two-level architecture for clinical applications to separate knowledge and
information levels in order to overcome the problems caused by the ever-changing
nature of clinical knowledge. Patient data is stored in a generic form which is
retrievable in any system using constraints named archetypes. An archetype,
which is designed by domain experts, defines some constraints on data in terms
of types, values, relation of different items and so on. Archetypes are used for
data validation and sharing [37].
The openEHR framework consists of the reference information model (RM),
the implementation technology specification, the archetype definition language
(ADL), the open-source implementations, and an archetype repository (the repos-
itory is explained more in Section 4.5) [37]. A review of the openEHR architec-
ture is presented by Beale [37]. The key concepts of the openEHR architecture
are explained in the following sections.
2.3.1 Two-level Modeling
openEHR suggests a two-level architecture for EHR systems, and accordingly a
two-level software engineering approach for developing such systems. The key
idea in the two-level architecture is the separation of the domain knowledge level
and the information level.
The first level or the lower level is a stable reference information model. Software
and data are built from this stable object model named the openEHR Reference
Model (RM). The second or upper level provides formal definitions of the clinical
domain concepts. This reduces the dependency of the system and underlying
data to the ever-changing clinical concepts.
2.3.2 Two-level Software Engineering
openEHR suggests a two-level software engineering approach. In this approach,
there exists different view points and a separation of responsibilities in software
8
Figure 2.1: The openEHR two level software engineering taken from [37]
development. The main roles involved in the openEHR process are domain
experts, users and IT developers. Different view points introduced by openEHR
are the domain knowledge environment, the runtime system and the technical
development environment. The openEHR approach consists of the following
steps:
•Domain specialists build reusable archetypes, templates (collections of
archetypes, see Section 2.3.4) for local use and terminologies for general
use.
•IT developers focus on generic aspects of the system such as data manage-
ment.
•The user works with a GUI which is derived from the templates. Data is
generated by users via the EHR system and is validated by archetypes at
runtime.
2.3.3 The Reference Model
The openEHR RM is specific to the clinical domain but still includes very generic
clinical concepts such as composition, observation, evaluation, instruction and
action. Moreover, in this RM, different data types are defined such as coded
text, quantity and multimedia.
2.3.4 Archetype
In the upper level of the openEHR two-level modeling approach, domain level
definitions are defined in form of archetypes and templates. These definitions are
9
Figure 2.2: The openEHR archetype meta-architecture taken from [37]
used in the EHR system at runtime. Archetypes are used to define constraints
on the generic RM. For instance, a blood pressure measurement can be defined
in form of an archetype in contrast to a more general clinical concept such as an
observation which is the focus of the RM.
All the information that is based on the RM is archetypeable, or in other words,
can be controlled by archetypes in terms of its creation, modification and so on.
Each archetype is an instance of the archetype model (AM) and is stored sepa-
rated from data in an archetype repository. The archetype definition language
(ADL) is the language that is used to define archetypes based on the AM. It
is recommended that when possible, archetypes should be reused and/or cus-
tomized instead of being created from scratch. The relation of archetypes to the
RM is depicted in Figure 2.2.
2.3.5 Template
Templates encapsulate a group of archetypes to be applied for a local use. Tem-
plates are trees of one or more archetypes and correspondent to user interface
forms, printed reports or other realizations of clinical data [37]. For instance,
using a template, one can put different clinical concepts like “blood pressure
measurement” and “mouth examination” (both defined as archetypes) together
to create an output report for EHRs [37].
2.4 Other EHR Standardization Approaches
Several studies have compared the main interoperability standards and specifi-
cations in the clinical domain [31, 34]. Based on a survey, Eichelberg provides
10
information on the most relevant EHR standards [38]. This includes the level
of interoperability provided by each standard, as well as content structure, ac-
cess services, multimedia support and security. In the following sections, a brief
introduction to the most relevant EHR standardization approaches is presented.
2.4.1 The CEN/ISO EN13606 standard
The CEN/ISO EN13606 is a European norm from CEN which is also approved
as an international ISO standard [39]. The aim of this standard is to enable
semantic interoperability in the electronic health record communication. The
CEN standard is actually a subset of the openEHR specification [34]. The same
as openEHR, this standard is based on the idea of a two-level architecture (i.e. a
dual model architecture [39]) which consists of a reference model and archetypes.
2.4.2 The Governmental Computerized Patient Record
Project (G-CPR)
G-CPR is a joint project between the US Department of Defense (DoD), the
US Department of Veterans Affairs (DVA) and the Indian Health Services (IHS)
[31]. This solution uses object-oriented specification to enable interoperability
and is rather a service-oriented solution than an architecture-based solution [31].
2.4.3 Health Level 7 (HL7)
HL7 is a well-known EHR communication standard in the clinical domain [31].
According to the HL7 website2: “Health Level Seven International (HL7) is a
not-for-profit, ANSI-accredited standards developing organization dedicated to
providing a comprehensive framework and related standards for the exchange,
integration, sharing, and retrieval of electronic health information that supports
clinical practice and the management, delivery and evaluation of health services”.
HL7 in HL7 version 3 presents a comprehensive Reference Information Model
(RIM) [31]. The HL7 clinical document architecture (CDA) templates are similar
to openEHR archetypes [38]. This standard, provides data-level interoperability
but functional level interoperability is not provided [31].
2.5 Decision Support in the Clinical Domain
Clinical decision support is a sub-domain of a more general research area named
decision-making support. According to Gupta [40] “decision-making support
systems (DMSS) are Information Systems designed to interactively support all
phases of a user’s decision-making process.” There are various definitions for
CDSS and CDS in the literature three of which are presented here:
2http://www.hl7.org/
11
•“computer-based clinical decision support (CDS) can be defined as the use
of the computer to bring relevant knowledge to bear on the health care
and well being of a patient” [13].
•“clinical decision support refers broadly to providing clinicians or patients
with clinical knowledge and patient-related information, intelligently fil-
tered, or presented at appropriate times, to enhance patient care” [14].
•“clincial decision support is any EHR-related process that gives a clinician
patient-related healthcare info with the intent of making the clinician’s
decision making more efficient and better informed” [3].
CDS impacts the process of decision making about individual patients. This
support should be provided at the point of care and while the decisions are
made [24]. These systems provide support for diagnosis of diseases, prevention
of errors in the clinical process, treatment, and future evaluation of the patient.
Services supported by CDS include diagnosis, alerting, reminding, treatment
suggestions, and patient education. CDS interventions are the CDS content and
the method for delivering the content.
In providing CDS, three modes of interaction between human and computer can
be defined [13]:
•User in charge (users can override computer’s suggestions)
•Computer in charge (any decision made by computer is expected to be
carried out by users)
•Collaborative decision making (computer controls the input, based on
users’ entries options are provided, users makes the desired choice)
The idea of having both computers and humans in charge of the health care pro-
cess is more practical in the clinical domain than building intelligent autonomous
systems that are in charge of the decision making process [13, 24]. The latter
may work in other disciplines but is less applicable in the clinical domain. Berner
[24] discusses that CDS is not meant to come up with “the answer” but should
provide information for the user and aid him/her in making decisions.
Not all of the information in a clinician’s mind can be transfered to the computer
(the CDSS) so a clinician usually knows more about the patient. Therefore,
having a collaborative pattern in which a clinician can eliminate some of the
choices made by the computer is better [24].
In this thesis, CDSS and CDS are used interchangeably to refer to a computer
program that aids clinicians in the process of decision making, at the point of
care, and based on health information of an individual patient, by presenting
that information coupled with external knowledge in a way that is more suitable
for making decisions regarding the care process of that specific patient. The
system is not meant to make the decisions, rather it is the clinician who makes
the final decision.
12
Planning Context of use Requirements Design Evaluation
◦Usability
planning and
scoping
◦Usability cost
benefit analysis
◦Identify
stakeholders
◦Context of
use analysis
◦Survey of
existing users
◦Field study
or user
observation
◦Diary
keeping
◦Task
analysis
◦Stakeholder
analysis
◦User
cost-benefit
analysis
◦User
requirements
interview
◦Focus groups
◦Scenarios of
use
◦Personas
◦Existing
system or
competitor
analysis
◦Task or
function
mapping
◦Allocation of
function
◦User, usability
and
organizational
requirements
◦Brainstorming
◦Parallel design
◦Design
guidelines and
standards
◦Storyboarding
◦Affinity
diagram
◦Card sorting
◦Paper
prototyping
◦Software
prototyping
◦Wizard-of-Oz
prototyping
◦Organizational
prototyping
◦Participatory
evaluation
◦Assisted
evaluation
◦Heuristic or
expert
evaluation
◦Controlled
user testing
◦Satisfaction
questionnaires
◦Assessing
cognitive
workload
◦Critical
incidents
◦
Post-experience
interviews
Table 2.1: Methods for user-centered (human-centered) design taken from [44]
2.6 Human-Computer Interaction
Human-computer interaction (HCI) is defined as “a discipline concerned with
the design, evaluation and implementation of interactive computing systems for
human use and with the study of major phenomena surrounding them” [41].
The concept of usability is considered to be the heart of HCI [42]. Usability
is defined as “the extent to which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency and satisfaction in a specified
context of use” [43]. A great deal of effort in the field of HCI is aimed at designing
and developing more usable computer systems [42].
Usability is a very important factor in designing an interactive system. If the
system is not usable enough for the intended users, it is likely that they do not
use the system so often (underuse) or use the system improperly (misuse) and
stick to their current methods for accomplishing the tasks, that both bring costs
to the organization or ruin the reputation of the team/company that developed
the system [44]. There are benefits in designing a usable system both for the
developer team and for the customers: increased productivity, reduced errors,
reduced training and support, improved acceptance and enhanced reputation
[44].
To develop a usable interactive system, both functional requirements and non-
13
functional requirements, including usability requirements, should be considered.
Traditionally, the focus of software design processes have been on functional
requirements, but nowadays there are design frameworks integrating these two
[44]. According to Shneiderman [45], usability requirements are of five types:
learnability, efficiency, memorability, errors, and satisfaction.
By involving users in the design and development process of a system, the system
will be more usable for the intended users [46, 47, 48]. The design approach which
places emphasis on involving users in the design is called user-centered design
(UCD) or human-centered design (HCD) [46, 47]. The main focuses of UCD
are active user involvement in the design process, multidisciplinary design teams
and iterative design [44]. UCD is not a substitute for software development
methods, but a complementary process to them. The UCD process is depicted
in Figure 4.2. The process starts with planning then context of use analysis,
requirements specification, design and evaluation are repeated until the user is
satisfied with the design and usability requirements are achieved.
Various methods are defined to support UCD. A broad range of methods are
specified by Maguire [44]. This collection is considered to be a proper introduc-
tion of various well-known methods and their relation to different UCD steps.
The methods are summarized in Table 2.1, column names correspond to different
steps in the UCD process.
14
Chapter 3
The Research Process
Oates [49] describes a model of the research process which consists of various
categories of methods such as research strategy, data generation methods, and
data analysis.
Oates describes how self-experiences and motivations along with the knowledge
gained from reviewing the literature in the field and being informed about the
existing gaps lead to a “conceptual framework” and “research questions” (see
Section 3.2).
A conceptual framework clarifies a researcher’s way of thinking and structuring a
research topic and the research process undertaken [49]. A conceptual framework
includes the research topic and its comprising factors, the research methodology
(a combination of strategies and methods used), the data analysis approaches,
the development methodology, and the research evaluation approach. Details
about the conceptual framework in this study is presented in Section 3.1.
In order to conduct a research study, a research strategy should be selected
(see Section 4.2.1), also data generation methods should be applied in order
to gather data and finally analyze the data using qualitative or quantitative
methods. In this study, data generation methods are those methods applied
in the system development process (see Section 4.3) along with the literature
published on the topic of interest, i.e. CDS (referred to as “documents” in
the Oates’s model). Both qualitative (used as part of the system development
process) and quantitative data analysis methods are applied in this study in
order to analyze data. Oates’s model is depicted in Figure 3.1. This figure also
highlights how the process of this study fits into this model.
15
Experiences and
motivations
Literature review
Research questions
Conceptual
framework
Survey
Design and
creation
Experiment
Case study
Action
research
Ethnography
Interviews
Observation
Questionan-
naires
Documents
Data generation
methods
Strategies
Quantitative
Qualitative
Data
analysis
usually
1:1 often
1:N
Figure 3.1: Model of research process adapted from [49]. The methods applied
in this study are colored in gray.
16
3.1 Conceptual framework
As mentioned previously, there have been various efforts in defining CDS success
factors and development difficulties ranging from a local, and practical view
[14, 50, 51, 15] to wider national-level views that have tried to provide guidelines
or road-maps for developing more effective CDS [52, 53, 21, 54].
Accordingly, four main categories of challenges and success factors faced in design
and development of CDS can be defined:
•technical issues concerning knowledge representation and reasoning and
also maintenance of knowledge
•integration into EHRs that deals with the integration to the underlying
IT framework or more specifically EHR systems in the clinical organiza-
tions which is a sub class of the “technical issues”, but because of its
importance, is considered as a separate category in this thesis.
•human-computer interaction issues that focus on the user interface
design of these systems and the way users interact with them. User satis-
faction, effectiveness and acceptability of CDS in a practical setting, and
involving clinicians in the design and development of CDS all belong to
this category.
•cultural and organizational issues that deal with the higher level as-
pects of motivations, utilizations, monitoring and acceptance of these sys-
tems at local, national and international levels.
According to the road-map for the United States national action on clinical de-
cision support [21] to reach widespread adoption of effective CDS, it is crucial
that system developers be supported to design “easy to deploy and use” applica-
tions. It is also recommended that best practices in system development should
be disseminated so that other developers can learn from previous successful ex-
periences.
As discussed before, the ultimate goal of efforts in the area of medical informatics
is to improve the quality of care, specially by introducing and applying EHRs
and CDS. To improve the quality of health care, neither of these two concepts
is an optimal solution individually. It is shown that to improve quality of health
care, EHR systems should be supported by other services such as CDS. On the
other hand, in order to develop effective CDS and to broaden its adoption, CDS
should be integrated into the existing EHR platform in the clinical organizations
and HCI should seriously be taken into consideration in designing and developing
them.
So far, this study has been done in relation to two of the categories of challenges
and success factors in developing CDS namely integration of CDS to EHRs and
taking HCI into account in designing and developing CDS. Figure 3.2 depicts
the factors the comprise this study. In the following, the research questions
(Section 3.2) and objectives (Section 3.3) are presented. Chapter 4 includes
17
more information about the research strategy and methods applied to answer
the research questions.
Quality Health Care
Electronic
Health Record
Highly adoptable
Clinical Decision Support
Integration to EHR
Technical Issues
HCI issues
Cultural and
organizational issues
Figure 3.2: In order to improve the quality of health-care focus on two areas
are inevitable: adopting clinical decision support and adopting electronic health
records (EHR). Development of highly acceptable clinical decision supports is de-
pendent on its integration into EHRs, and also consideration of human-computer
interaction (HCI) with the aim of developing usable clinical decision support sys-
tems (CDSS). There are however other issues such as technical considerations
(e.g. knowledge representation and reasoning) and cultural and organizational
aspects of adopting clinical decision support in health organizations. Among
these pillars, technical issues have gained the most attention so far, but interest
in other aspects has also been increasing recently. This thesis is invovles two
aspects (i.e. colored pillars) namely integration of clinical decision support sys-
tem into EHRs, and taking HCI into consideration with the aim of developing
a usable CDSS that is aware of an EHR standard named openEHR.
3.2 Research Questions
The aim of this study has been to answer this research question:
How can usable openEHR-aware clinical decision support be designed
and developed in order to improve the quality of health care?
In order to answer this question, several sub-problems were set to be investigated:
18
RQ1 Are usability of clinical decision support and methods to reach and as-
sure usability taken into consideration by developers of clinical decision
support?
RQ2 Are integration of clinical decision support into electronic health records
and adopting electronic health record standards taken into consideration
by developers of clinical decision support?
RQ3 Is the openEHR suggested approach suitable for designing and developing
openEHR-aware clinical applications, including clinical decision support
systems?
•Does the two-level software engineering approach suggested by open EHR
work in practice?
RQ4 How can current successful design and development processes such as user-
centered design be customized for designing and developing clinical appli-
cations and clinical decision support?
The question involves the following sub-problem:
•How can the design and development process of an openEHR-aware
clinical application, including clinical decision support systems, be
structured with focus on human-computer interaction and involving
clinicians in the process?
RQ5 Does openEHR offer any new opportunities for clinical decision support
in terms of knowledge representation and reasoning?
The question involves the following sub-problems:
•Can openEHR be used to improve the process of knowledge represen-
tation and reasoning in clinical decision support?
•Can clinical decision support benefit from structured, quality vali-
dated openEHR-based electronic health records?
•Is it feasible and practical to integrate clinical decision support inter-
ventions into openEHR-based electronic health records?
To answer the above research questions1, several objectives should be accom-
plished. These objectives are defined in the following section.
3.3 Objectives
To investigate openEHR and also UCD in a clinical context with the aim of
answering the research questions, it was decided to develop a CDSS for an oral
disease. It was also planned to accomplish the following objectives:
1It should be mentioned that the work presented in this thesis is actually related to RQ1-
RQ4. RQ5 is suggested as the future direction of this study.
19
O1 Literature reviews should be conducted in order to analyze state of the art
in interplay between HCI and CDS, also the intersection of EHR and CDS.
O2 The openEHR framework should be studied and understood. The archetype
concept, two-level modeling, and two-level software engineering suggested
by openEHR should be analyzed.
O3 A UCD process should be applied from the beginning of the project. Dif-
ferent UCD methods that are applicable for the project should be selected
for designing and developing both the CDSS and archetypes.
O4 Domain-specific information about the disease should be gathered and struc-
tured using openEHR archetypes. Additionally, as suggested by openEHR,
reusable existing archetypes should be specified and customized if appli-
cable.
O5 The strengths and shortcomings of the openEHR approach and the limita-
tions of the two-level software engineering suggested by openEHR should
be identified in order to be reconsidered in the proposed approach in this
study.
O6 The characteristics of the clinical domain, clinical tasks and clinicians that
may have an effect on the user-centered design process should be identified.
20
Chapter 4
Methods and Tools
In this section, the methods and tools applied in order to answer the research
questions (see Section 3.2) are elaborated.
4.1 Literature Review
Literature review is a research methodology that aims at summarizing the avail-
able literature on a topic and presenting an analysis based on that and providing
a full picture on the topic [55]. In this study, literature review is used to analyze
state of the art in two different but related topics namely “interplay between
HCI and CDS development”, and “Intersection of CDS and EHR”. The search
strategies are discussed more in detail in the papers I and Paper II.
4.2 The Research Methodology
Oates in [49] defines research methodology in information systems as a com-
bination of “research strategy”, “design and development process” and “data
generation methods”. This is depicted in Figure 4.1.
In this study a design and creation research strategy is used as the research
strategy and a user-centered design process is used as the development process.
More information about these methods can be found in the following sections.
4.2.1 Design and Creation Research Strategy
The focus of a design and creation research strategy is on developing new soft-
ware products i.e. artefacts [49]. In this study the “artefact” to be developed
is a clinical decision support system for dry mouth. To understand this artifact
a description of the disease, i.e. dry mouth, and the characteristics of the CDS
are provided in the following.
21
Strategy(ies)
Design and
creation
[+??]
Data methods
Interviews
Observations
Questionnaires
Documents
Research
methodology
Development
methodology
Figure 4.1: Research methodology and development methodology taken from
[49]. Oates defines research methodology in information systems as a com-
bination of “research strategy”, “design and development process” and “data
generation methods”.
A Clinical Decision Support for Dry Mouth
“Dry mouth or xerostomia is the abnormal reduction of saliva and can be a
symptom of certain diseases or be an adverse effect of certain medications”
[56]. Treatment of Xerostomia is related to finding its cause(s). There are five
main categories for xerostomia namely drug-induced, disease-induced, radiation-
induced, chemotherapy-induced, and cGVHD-induced [56]. Finding cause(s) of
dry mouth is a challenge for clinicians and needs to be supported by a clinical
application.
A potential dry mouth patient should answer, or a clinician should find an answer
to various types of questions such as:
•Do you need to moisten your mouth frequently or sip liquids often?
•Have you noticed any swelling of you salivary glands?
•Do you smoke or have been smoking regularly?
•Are you currently taking 3 drugs or more?
•Have you been subjected to therapeutical radiation against your head-and-
neck region?
•Have you had a feeling of dry mouth daily for more than 3 months?
As in other diseases, each of these questions is considered very important in
diagnosis. They sound very straightforward, but the difficulty arises when all of
these questions should be remembered at the point of care [3] while the answer
provided by a patient should also be supported with an examination by clinicians,
e.g. the swollen salivary gland.
22
The dry mouth CDS is meant to be used in the clinic of oral medicine in Sahlgren-
ska University Hospital, Gothenburg, Sweden. Since this system is going to be
used integrated with an existing clinical data entry application, i.e. MedView
[57], data entry forms are not part of the Graphical User Interface (GUI), how-
ever users should be provided with options to edit existing data. Finally, users
need to be able to enter their own comments; including diagnosis or treatments
to the system.
The intended decision support process includes these four main steps:
1. Presenting an overview of patient-specific information and external knowl-
edge in a way that makes decision making easier
2. Providing proper reminders and alarms
3. Helping the user in finding the cause(s) of disease based on the patient’s
medical record
4. Suggesting related materials and treatment options, patient health infor-
mation and external knowledge
This study has been conducted in oral medicine, however, the outcome of the re-
search should be applicable to medical informatics in general, i.e. other diseases,
and other clinical applications.
4.3 User-Centered Design Process
UCD is a process that places emphasis on involving users in the design [46, 47].
As depicted in Figure 4.2, UCD is a circular design process. The UCD process
consists of five steps [47]. These steps are
1. plan the human-centered process
2. understand and specify the context of use
3. specify the users and organizational requirements
4. produce design solutions
5. evaluate design against requirements
This circle will be repeated until the users are satisfied with the design and the
requirements are met in the design solution.
As recommended [58], a customized UCD process has been applied in this study.
The customization was done in order to make the process suitable for the context
and also the nature of the project i.e. having openEHR as the underlying EHR
standard. To accomplish the UCD process, several methods were utilized such as
prototyping, usability tests, and interviews. Different UCD methods that were
applied in this project are summarized in Table 4.1 and discussed more in detail
in Paper V.
23
Plan the user-centered
design process
Understand and
specify the context of
use
Specify the users and
organizational
requirements
Evaluate design
agains requirements
Produce design
solutions
Meets
requirements?
Figure 4.2: The user centered design cycle [47, 44], see Section 4.3
Context of Use Requirements Design Evaluate
◦Identifying
stakeholders
◦Informal context
of use analysis
◦Interviews
◦Multidisciplinary
group sessions
◦Interview
◦Persona
◦Existing system or
competitor analysis
◦User, usability and
organizational
requirements
◦Literature study
◦Domain concept
modeling
◦Brainstorming
◦Design guidelines
and standards
◦Paper prototyping
◦Software
prototyping
◦Informal user
evluation
◦Informal expert
evaluations
◦Usability test
◦Think aloud
protocol
◦Satisfaction
questionnaires
◦Post experience
interviews
Table 4.1: The UCD methods applied in the project
24
The work presented here is the outcome of the first three iterations of this
project. Several users and domain experts were involved in this process. Several
user interface prototypes were developed, evaluated and improved iteratively.
The characteristics of this clinical context that have an effect on applying a
UCD process were detected and analyzed. Moreover, UCD was not only applied
to reach usability in the design, but also to develop domain concept models
to create archetypes. The latter was also accomplished iteratively by involving
clinicians (see Paper III).
The Multidisciplinary Project Team
The project team included the following members: one specialist in dentistry,
who was also one of the stakeholders and initiators of the project (from now
on, we refer to this person as the main clinical partner), three computer sci-
entists, with knowledge of human-computer interaction, usability and software
engineering, and one programmer.
Users Involved
Besides our main clinical partner, who was involved in the project from the
beginning, three more specialists in dentistry and one dental hygienist were
interviewed during this study both for requirements gathering, archetypes de-
velopment, and informal evaluation of the user interface paper prototypes (from
now on, we refer to this group as expert panel). Another group of three special-
ists in dentistry and one dental hygienist were also asked to participate in the
project as test users for user interface evaluations.
4.4 Assumptions
The focus of the project is on the design and development process and clinicians’
involvement in the process. We assume that openEHR as an interoperability
standard is acceptable for our purposes. The aim of this project is “not” to
prove if openEHR has been successful in EHRs interoperability.
The evaluation process that is mentioned in this thesis refers to the evaluation
which is done before releasing the CDS and even before evaluations based on
clinical trials which is required to prove reliability of CDS before deployment.
Only the openEHR archetype concept is applied in this work and no template is
created. The idea of openEHR templates is skipped for several reasons: imma-
turity, no implementation, and finally since it was possible to develop the system
without applying them.
25
4.5 Archetype Development Process
Domain concept modeling (information modeling) was required to understand
and specify the data needed to be gathered and presented in the CDS system.
Moreover, the domain modeling process was the first step in knowledge gathering
for providing CDS. The openEHR approach suggests archetypes creation as a
more structured way of modeling domain concepts.
The archetype development was conducted in close collaboration with clinicians,
i.e. experts in the domain. The development process was iterative, this means
that domain concept models were created and evaluated by experts in various
steps. More information about the process and tools used to develop archetypes
is provided in the following.
Iterative Domain Concept Modeling
The domain concept modeling started with sessions in which our main clinical
partner was asked to think about dry mouth and its related concepts and to
put as much information as possible on paper. Later, he was asked to prepare a
questionnaire based on this question. The reason for this was that the current
clinical system that clinicians in the clinic use in their everyday work is based
on the idea of clinical questionnaires. Questions on the questionnaire were then
categorized based on openEHR concepts; in other words, their logical relation,
e.g. is it related to the patient’s history or examining the patient?
Mind Mapping Diagrams
For better communication of the domain concepts in order to create archetypes,
simple diagrams were created based on the questionnaires and also the outcome
of the brainstorming sessions. For this purpose, a mind-map application was
used to make it possible for our expert panel to simply understand and edit
the created diagrams. The mind mapping software used in this step is called
XMind 1.
Evaluation
Iterative design of the domain concept model includes evaluations of the current
model based on the literature and experts’ opinions, and story-based assessment.
Information modeling diagrams were improved several times based on the ex-
perts’ opinions. Several experts were involved in this process to minimize the
subjectivity of the design and to be as broad as possible in collecting knowledge.
A sample mind map is depicted in Figure 4.3
1http://www.xmind.net
26
Figure 4.3: The domain concept modeling (information modeling) was done
iteratively and together with the experts in the disease. The XMind application
was used in order to easily understand and create diagrams and to communicate
information.
27
Figure 4.4: Ocean Archetype Editor is a freely available tool for authoring
openEHR archetypes. This tool is meant to be used by clinicians to define spec-
ifications of the domain concepts. This tool is developed based on the openEHR
reference model and supports various concept types introduced by it. In this
figure a sample examination archetype is being edited. On the left side of the
tool, there is a tree structure of the nodes in the archetype. The tools provided
several tabs to support editing various aspects of the archetype such as definition
and terminology.
Archetype Editor
For authoring archetypes, a free tool named the Ocean Archetype Editor2was
used. The Ocean Archetype Editor is a visual tool that supports the authoring
of openEHR archetypes. The editor is unicode-enabled, therefore archetypes
in any language, including Swedish, can be created in this tool, however, in
this project the main language for creating archetypes has been English so far.
This editor supports full openEHR data types and saves archetypes as different
formats such as ADL and XML.
Reusing Existing Archetypes
It is recommended that whenever possible existing archetypes be reused and/or
customized instead of being created from scratch for different local developers.
Accordingly, we have also tried to reuse some of the existing archetypes in this
project. The openEHR community along with other efforts has tried to make
2http://www.oceaninformatics.com
28
Figure 4.5: The openEHR clinical knowledge manager (CKM) is a common
repository of archetypes. Users interested in modeling clinical content may
participate in the creation and/or enhancement of this international set of
archetypes via this online repository.
the idea of share-ability and reuse-ability of archetypes possible by creating an
online repository of reviewed international archetypes. This repository is called
The openEHR Clinical Knowledge Manager which is explained below.
The openEHR Clinical Knowledge Manager
The openEHR clinical knowledge manager (CKM)3is an international, online
clinical knowledge resource. CKM is a library of clinical knowledge artifacts
which at the moment is limited to openEHR archetypes and templates. It is
anticipated that a complementary repository for other related artifacts like ter-
minology subsets be provided in the future. This repository provides the founda-
tion for interoperable EHRs. The openEHR archetypes available in the CKM go
under a review and publication process in order to be accessible to others. Users
interested in modeling clinical content may participate in the creation and/or
enhancement of this international set of archetypes.
3http://www.openehr.org/knowledge
29
Chapter 5
Summary of the Attached
Papers
In this section, a summary of the appended papers is given, along with how
they can be put in relation to each other and the research questions. Figure 5.1
depicts different areas covered in the publications.
5.1 Paper I
A literature review on interplay between HCI and CDS development is presented
in Paper I which is related to RQ1:are usability of clinical decision support and
methods to reach and assure usability taken into consideration by developers of
clinical decision support? This paper contributes to objective O1.
The paper starts with a brief review of the studies dealing with the question
of which factors should be considered in design and development of CDS to re-
sult in an acceptable and effective CDS, to motivate the importance of HCI,
usability and UCD in developing CDSSs. In order to conduct the literature re-
view, two databases (ScienceDirect1and PubMed2) were searched using boolean
combinations of some related keywords (usability, human-computer interaction,
user-centered design, clinical decision support, medical decision support). This
resulted in a total of 153 studies of which only 17 were relevant to the review.
Various concepts such as iterative design, involving clinicians in design and eval-
uation, qualitative evaluation methods, usability and UCD were the focus of
this review. More about the findings of this literature review can be found in
Section 6.1.
1http://sciencedirect.com
2http://pubmed.org
30
Applying openEHR as
the underlying standard
for a clinical application
Papers II, III, V
User centered design
process in a clinical
domain
Papers I, III, IV
EHR
HCI
State of the art in developing clinical decision
support with focus on the success factors
Papers I, II
HCI
EHR
Figure 5.1: A more empirical view of the study compared to (Figure 3.2) is
presented in this figure. Instead of categorization in an abstract level (i.e. HCI
issues, integration into EHRs) realization of these aspects in form of applying
specific approaches (i.e. openEHR, user-centered design) are introduced here.
Moreover, it is shown how these aspects are covered in various publications
attached to this thesis.
5.2 Paper II
Paper II includes a literature review conducted in order to answer RQ2:Are
integration of clinical decision support into electronic health records and adopt-
ing electronic health record standards taken into consideration by developers of
clinical decision support? This paper contributes to objectives O1 and O5.
The paper motivates the important of integrating CDS into EHRs based on
findings by other researchers in the field. It is discussed how CDS and EHRs
support each other’s success, and finally improving the quality of care. Based on
searching one database, i.e. ScienceDirect, and using boolean combinations of
some related keywords (electronic health record, medical health record, clinical
decision support, openEHR, HL7) a total of 98 studies were found where only
25 of them were relevant to the review.
In addition, since the focus of the thesis has been on openEHR, a discussion of
the causes of low adoption of the openEHR approach is presented in this paper
as well. More about the findings of this literature review and the reasons for low
adoption of openEHR can be found in Section 6.2.
5.3 Paper III
Paper III is related to RQ3:Is the openEHR suggested approach suitable for
designing and developing openEHR-aware clinical applications, including clinical
decision support systems? and RQ4:How can current successful design and
development processes such as user-centered design be customized for designing
31
and developing clinical applications and clinical decision support? This paper
contributes to objectives O2,O3, and O4.
This paper describes how a UCD approach can be used in a clinical context for
developing an openEHR-based CDSS. The paper includes a proposed customized
UCD approach along with the preliminary results of designing the GUI, domain
concept models and archetypes. Additionally, some challenges faced in adopting
openEHR are discussed in Paper III.
5.4 Paper IV
Paper IV is related to RQ4:How can current successful design and development
processes such as user-centered design be customized for designing and developing
clinical applications and clinical decision support? This paper contributes to
objectives O3 and O6.
This paper reports on employing a UCD process in developing a CDSS. Paper IV
can be seen as a more detailed version of Paper III, in which the focus has been
on the UCD process and the applied methods while details regarding openEHR
are skipped in this Paper. The paper includes results of the three iterations of the
project and includes various prototypes of the system, evaluations and analysis
of the evaluation results. In addition, those characteristics of the clinical context
that have an effect on applying a UCD process are identified and analyzed in
the paper.
5.5 Paper V
Paper V is indirectly related to RQ3. By “indirect” we mean the paper does
not include an answer to this question but is a more practical effort dealing
with one of the weaknesses of openEHR that has been discussed in Paper II and
Paper III. In this paper, we have dealt with the question: how can developers of
openEHR-based clinical applications connect iteratively designed and evaluated
user interfaces to the underlying framework with minimum effort? This paper
contributes to objective O2 and O5.
In this Paper, a framework for binding pre-designed GUIs to openEHR-based
backends is proposed. The proposed framework contributes to the set of options
available for developers. This approach can be useful especially for various small
scale and experimental systems as well as systems in which the quality of the
user interface is of great importance.
5.6 Paper VI
Paper VI is indirectly related to RQ5. This means that the paper does not
cover the answer to the research question but includes discussions about the
32
opportunities openEHR may provide for knowledge representation and reasoning
in CDSSs.
In this paper, a software architecture for the CDSS for dry mouth is proposed.
The architecture benefits from an existing openEHR framework and also a case-
based reasoning (CBR) framework. Case-based reasoning is a knowledge repre-
sentation and reasoning method that has been popular in the clinical domain
and, based on the available domain knowledge and patient data, seems to be a
proper choice for this project as well.
The paper also includes a methodological approach to developing openEHR
archetypes. In addition, motivations for selecting the knowledge representation
and reasoning method are given in the paper.
33
Chapter 6
Thesis Contributions
The major contributions of this study are the result of accomplishing the objec-
tives O1-O6 (see Chapter 5) and answering the research questions RQ1-RQ4.
This is actually documented in the attached papers as results. A brief summary
of the contributions is provided in the following.
6.1 The Answer to RQ1
The aim of efforts in the area of CDS is to develop such systems that result in the
wider adoption of CDS and accordingly improvement in quality of health care.
Various studies have dealt with the question of which factors should be considered
in design and development of CDS to result in an acceptable and effective CDS.
According to these studies, success factors of CDS can be divided in two main
categories of technical and non-technical (i.e. human-related) factors. Most of
these human-related factors, are the issues covered by the HCI discipline and
related to the concept of usability. HCI suggests methods and approaches to
address the human-related (i.e. user-related) factors and to assure usability of
the applications.
Based on a literature review (see Paper I), one can conclude that while various
researchers have so far introduced human-related factors as factors important
in the success of CDSSs, HCI is not still a routine practice in this field. Only
17 studies were relevant to the literature review whereas just in ScienceDirect
more than 100 practical studies on CDSSs are published. In particular, when
it comes to viewing UCD as a life-long process, very few studies can be found
that have covered this aspect in developing a CDSS. It was observed that some
of the recommended UCD methods are not applied or rarely are applied in CDS
developments. Task analysis, usability expert reviews and heuristic evaluation
are some of those rarely applied methods. Finally, there are still cases in which
evaluation of the system (our focus is on qualitative evaluations) is only con-
ducted after system deployment. All in all, there is a need for further adoption
34
of HCI (including usability) in this field.
6.2 The Answer to RQ2
Taking standards into consideration in any clinical application (and generally
any information system) is very important [14]. Since CDS operates by utilizing
both patient/organizational-specific data and clinical knowledge, it is important
to take the standards that support each of these areas into account [14].
Only 25 studies were found in ScienceDirect to have considered integration of
CDS into EHRs (from more than 100 studies that have documented CDS de-
velopments). For more information please refer to Paper II. We did not find
any study that reports on implementation of a CDS by applying openEHR.
The only study which considers the intersection between openEHR and CDS is
[59] in which the idea of integrating guideline rules into openEHR archetypes is
discussed.
The selected articles were reviewed in order to find out whether they consider any
of the standards related to CDS (i.e. EHR standards, guideline representation
standards, and terminology or vocabulary standards). It was observed that
standardization of guidelines and integration of guidelines into EHR has been
discussed in several studies [59, 60, 61].
The idea of applying standards even for EHR systems is still not mature enough,
and it is not surprising to see that researchers rarely consider this in CDS de-
velopment. For instance, from the 25 studies we reviewed only 6 had considered
EHR standardization (all of them applied HL7).
In conclusion, theory supports the benefits offered by integrating CDS into
EHRs, still, a great deal of effort should be put into this in order to reach an
acceptable level of integration in practice, especially considering standardization
aspects of EHR.
Moreover, if we put the the results of the literature review with focus on HCI
(see Section 5.1), it is observable that there are only a small number of studies
that have considered both HCI and EHR integration while developing CDS as
depicted in Figure 6.1.
35
32.0%
HCI consideration (8)
68.0%
No HCI consideration (17)
Figure 6.1: This chart represents how many of the studies have considered inte-
gration of CDS into EHRs, as well as HCI in developing CDS.
6.3 The Answer to RQ3
In this study, investigations were carried out into various aspects of develop-
ing openEHR-based applications with the focus on the design and development
“process” and with the aim of developing “usable” CDS.
As discussed in Section 2.3, openEHR suggests defining various “roles” in devel-
oping clinical applications, and to divide responsibilities among different roles.
The openEHR two-level software engineering, as might be expected, is compat-
ible with the multi-disciplinary team work suggested in UCD. The clinicians’
expertise can be used by involving them in the domain concept modeling as sug-
gested by openEHR and additionally in user interface design as recommended in
the HCI field. The two-level software engineering (see Figure 2.1) suggested by
the openEHR community is not by itself enough for developing user-friendly ap-
plications inasmuch as it does not consider the importance of involving clinicians
in designing and evaluating the GUI. To develop usable clinical applications, a
close collaboration between clinicians and IT developers is needed. Moreover,
automatic user interface generation results in interfaces with poor usability. This
is discussed further in the following.
Regardless of its advantages, the openEHR standard suffers from a rather low
adoption rate. Some possible reasons for the low adoption are the complexity of
the standard, lack of documentation and training for developers, and a limited
set of tools and frameworks available to ease application development (see Paper
II). The openEHR community seems to have mostly focused on representing and
modeling domain concepts and perfecting the specifications. However, to make
openEHR more practical, there is a need for supporting application developers
with APIs, frameworks and tools.
Surely, a number of application development projects exist such as the open
36
Archetype
Archetype
Archetype
Template
Template
Template
translated to
GUI artefacts
GUI
......
...... User
RM
object
RM
object
RM
object
RM
object
RM
object
AOM
object
validated by
AOM instances
openEHR-based
EHR
Data
Developer
Manual
adjustment
expert
create
Archetype
Archetype
Archetype
User
RM
object
RM
object
AOM
object
validated by
AOM instances
openEHR-based
EHR
Developer
create deisgn
is mapped to GUI
......
......
RM
object
RM
object
RM
object
Data
data binding
expert
A B
Figure 6.2: The two development models. The model on the left is supported
by opereffa
source health information platform [62] (OSHIP), the open EHR-Gen frame-
work [63], GastrOS [64], and the openEHR reference framework and application
[65] (opereffa). To the best of our knowledge, current openEHR frameworks and
tools are based on the idea that clinicians design and create archetypes (and tem-
plates) using existing tools. Later on, a GUI, or some GUI artifacts are generated
based on these archetypes/templates. In order to improve the GUI design, there
is a need for manual adjustment of the GUI or its style files (depicted in Fig-
ure 6.2-A). In contrast to this automatic or semi-automatic approach, there is an
alternative approach where there is no generation of GUI based on archetypes.
Instead, the interface is designed by experts based on the the users’ requirements.
Afterwards, there is a need to connect this GUI to the archetypes designed by
domain experts (illustrated in Figure 6.2-B). Unfortunately, the current frame-
works do not provide sufficient support for this approach. Therefore, we have
developed an extension, a Java desktop user interface data binding layer, to one
of the openEHR application development frameworks, i.e. opereffawith the aim
of supporting openEHR Java application developers who develop applications
according to the aforementioned approach.
37
Model the concepts
Understand and specify the
concepts
Evaluate the modelsEvaluate the design
Understand and specify the
requirements, considering
the context
Prototyping
Is the model agreed upon
among the experts involved?
Are the requirements met?
Specify the effect of changes
on the user interface
Domain expert panel
End user panel
Figure 6.3: The customized user-centered design applied in this study.
38
6.4 The Answer to RQ4
In this study, we have applied a design and development process that combines
UCD and openEHR principles. The suggested approach considers active involve-
ment of the clinicians in design and evaluation of the archetypes, and also the
user interface. Moreover, the effect of the archetypes on the user interface has
been taken into consideration. This customized UCD approach is depicted in
Figure 6.3. The proposed UCD process is compatible with the openEHR soft-
ware development approach illustrated in Figure 6.2-B (see the previous section
for details) More about this UCD process can be found in Paper III.
In addition, we have tried to learn from applying UCD in a clinical context.
Characteristics of the context, users and tasks that may have an effect on apply-
ing UCD are also identified in this study. These characteristics should be taken
into consideration in design and development of clinical applications including
CDS (see Paper IV).
39
Chapter 7
Future Work
The main future direction of this study is to address RQ5:
Does openEHR offer any new opportunities for clinical decision sup-
port in terms of knowledge representation and reasoning?
Various sub-problems related to this RQ would be:
1. Can openEHR be used to improve the process of knowledge representation
and reasoning in clinical decision support?
2. Can clinical decision support benefit from structured, quality validated
openEHR-based electronic health records?
3. Is it feasible and practical to integrate clinical decision support interven-
tions into openEHR-based electronic health records?
Moreover, there are other aspects of the study that need more investigations:
•What are the challenges in applying user-centered design in a clinical con-
text and how to tackle these challenges?
•Is the idea of automatic user interface generation acceptable from a human-
computer interaction perspective?
40
Chapter 8
Conclusion
This thesis investigates the question: how can usable openEHR-aware clinical
decision support be designed and developed in order to improve the quality of
health care? In order to answer this question, several sub-problems were identi-
fied to be investigated, and accordingly, several objectives to be accomplished.
Both theoretical and empirical research strategies were used in order to address
the identified research questions (see Chapter 3). Of the five specified research
questions, four are answered in this thesis.
Analysis of state of the art in interplay between HCI and CDS and also the
intersection between CDS and EHRs revealed that consideration of both HCI
and integration of CDS into EHRs is more appreciated in theory than practice
and are yet to be developed (see Paper I and Paper II).
Moreover, the experience in designing an openEHR-based clinical application
revealed that apart from benefits offered by the openEHR approach such as
defining different roles and involvement of users in defining domain concepts,
there are various shortcomings that should be improved, for instance the insuffi-
cient support for openEHR application developers. Additionally, it was observed
that there are characteristics of the domain, tasks and users in the domain that
developers should be informed about while applying UCD methods.
Finally, several future directions of the research were presented with focus on
both the UCD development process, and investigation of openEHR more in
depth (see Chapter 7).
41
Bibliography
[1] D. Bates and A. Gawande, “Improving safety with information technology,”
New England Journal of Medicine, vol. 348, no. 25, pp. 2526–2534, 2003.
[2] A. Kushniruka, M. Triolab, B. Steinc, E. Boryckid, and J. Kannrye, “The
relationship of usability to medical error: an evaluation of errors associated
with usability problems in the use of a handheld application for prescribing
medications,” in Medinfo 2004: Proceedings Of THe 11th World Congress
On Medical Informatics, vol. 107, p. 1073, Ios Pr Inc, Jan. 2004.
[3] J. Walker, E. Bieber, and F. Richards, Implementing an electronic health
record system. Springer Verlag, 2006.
[4] P. Reizenstein, “Safety problems in health care cause 100 avoidable deaths,”
Lakartidningen, vol. 84, pp. 1680–1681, 1987.
[5] H. Hoff, “Motion 2009/10: So278 Patient safety in healthcare - The Parlia-
ment.” Website, 2010. http://www.riksdagen.se/Webbnav/index.aspx?
nid=410\&typ=mot\&rm=2009/10\&bet=So278.
[6] L. Kohn, J. Corrigan, M. Donaldson, and Others, To err is human: building
a safer health system. Washington, D.C.: NATIONAL ACADEMY PRESS,
1999.
[7] C. Vincent, G. Neale, and M. Woloshynowych, “Adverse events in British
hospitals: preliminary retrospective record review,” Bmj, vol. 322, no. 7285,
p. 517, 2001.
[8] M. R. Chassin, “The Urgent Need to Improve Health Care Quality: In-
stitute of Medicine National Roundtable on Health Care Quality,” JAMA:
The Journal of the American Medical Association, vol. 280, pp. 1000–1005,
Sept. 1998.
[9] “Crossing the Quality Chasm: The IOM Health Care Qual-
ity Initiative.” Institute Of Medicine (IOM) Website, 2010.
http://www.iom.edu/Global/NewsQuality-Chasm-The-IOM-Health-Care
-Quality-Initiative.aspx.
42
[10] N. Graham, Quality in health care: Theory, application, and evolution.
Aspen publishers, Inc., 1995.
[11] T. Graham, A. Kushniruk, M. Bullard, B. Holroyd, D. Meurer, and
B. Rowe, “How usability of a web-based clinical decision support system
has the potential to contribute to adverse medical events,” in AMIA An-
nual Symposium Proceedings, vol. 2008, p. 257, American Medical Infor-
matics Association, Jan. 2008.
[12] A. Hasman, “Challenges for medical informatics in the 21 st century,” In-
ternational journal of medical informatics, vol. 44, no. 1, pp. 1–7, 1997.
[13] R. Greenes, Clinical decision support: the road ahead. Academic Press,
2007.
[14] J. Osheroff, E. Pifer, J. Teich, D. Sittig, and R. Jenders, Improving outcomes
with clinical decision support: An implementer’s guide. HIMSS, 2005.
[15] D. F. Sittig, A. Wright, J. a. Osheroff, B. Middleton, J. M. Teich, J. S.
Ash, E. Campbell, and D. W. Bates, “Grand challenges in clinical decision
support.,” Journal of biomedical informatics, vol. 41, pp. 387–92, Apr. 2008.
[16] R. Greenes, M. Sordo, D. Zaccagnini, M. Meyer, and GJ, “Design of a
standards-based external rules engine for decision support in a variety of
application contexts: report of a feasibility study at Partners HealthCare
System,” Medinfo, 2004.
[17] K. Kawamoto, C. A. Houlihan, E. A. Balas, and D. F. Lobach, “Improving
clinical practice using clinical decision support systems: a systematic review
of trials to identify features critical to success.,” BMJ (Clinical research ed.),
vol. 330, no. 7494, p. 765, 2005.
[18] D. Hunt, R. Haynes, S. Hanna, and K. Smith, “Effects of computer-based
clinical decision support systems on physician performance and patient out-
comes: a systematic review,” Jama, vol. 280, p. 1339, Oct. 1998.
[19] J. Bennett and P. Glasziou, “Computerised reminders and feedback in med-
ication management: a systematic review of randomised controlled trials,”
Medical Journal of Australia, vol. 178, no. 5, pp. 217–222, 2003.
[20] M. Trivedi, J. Kern, A. Marcee, B. Grannemann, B. Kleiber, T. Bettinger,
K. Altshuler, and A. McClelland, “Development and Implementation of
Computerized Clinical Guidelines : Barriers and Solutions,” Methods of
information in medicine, vol. 41, no. 5, pp. 435–442, 2002.
[21] J. Osheroff, J. Teich, B. Middleton, E. Steen, A. Wright, and D. Detmer,
“A roadmap for national action on clinical decision support,” Journal of
the American medical informatics association, vol. 14, no. 2, p. 141, 2007.
43
[22] J. Anderson, “Increasing the acceptance of clinical Information,” MD com-
puting: computers in medical practice, vol. 16, no. 1, p. 62, 1999.
[23] T. Wetter, “Lessons learnt from bringing knowledge-based decision support
into routine use.,” Artificial intelligence in medicine, vol. 24, pp. 195–203,
Mar. 2002.
[24] E. Berner, Clinical Decision Support Systems: Theory and Practice (Health
Informatics). New York, NY 10013, USA: Springer, 2007.
[25] A. Hasman, R. Haux, and a. Albert, “A systematic view on medical infor-
matics.,” Computer methods and programs in biomedicine, vol. 51, pp. 131–
9, Nov. 1996.
[26] R. Haux, “Aims and tasks of medical informatics,” International journal of
medical informatics, vol. 44, pp. 9–20; discussion 39–44, 45–52, 61–6, Mar.
1997.
[27] R. Haux, “Medical informatics: Past, present, future.,” International jour-
nal of medical informatics, vol. 79, pp. 599–610, Sept. 2010.
[28] M. Collen, “Origins of medical informatics,” Western Journal of Medicine,
vol. 145, pp. 778–785, 1986.
[29] R. S. Ledley and L. B. Lusted, “Reasoning Foundations of Medical Diag-
nosis: Symbolic logic, probability, and value theory aid our understanding
of how physicians reason,” Science, vol. 130, pp. 9–21, July 1959.
[30] I. Iakovidis, “Towards personal health record: current situation, obstacles
and trends in implementation of electronic healthcare record in Europe.,”
International journal of medical informatics, vol. 52, no. 1-3, pp. 105–15,
1998.
[31] B. Blobel, “Advanced and secure architectural EHR approaches.,” Interna-
tional journal of medical informatics, vol. 75, no. 3-4, pp. 185–90, 2006.
[32] J. Ash and D. Bates, “Factors and forces affecting EHR system adoption:
report of a 2004 ACMI discussion,” Journal of the American Medical Infor-
matics, pp. 8–12, 2005.
[33] V. Stroetmann, D. Kalra, P. Lewalle, J. Rodrigues, and KA, “Semantic
Interoperability for Better Health and Safer Health Care,” Deployment and
Research, no. January, 2009.
[34] P. Schloeffel, T. Beale, G. Hayworth, S. Heard, and H. Leslie, “The relation-
ship between CEN 13606, HL7, and openEHR,” in In Health Informatics
Conference (2006), vol. 7, p. 24, Health Informatics Society of Australia,
2006.
[35] “openEHR.” Website, 2010. http://openEHR.org.
44
[36] L. Bird, A. Goodchild, and Z. Tun, “Experiences with a two-level modelling
approach to electronic health records,” Journal of Research and Practice in
Information Technology, vol. 35, pp. 121–138, Apr. 2003.
[37] T. Beale and S. Heard, “openehr architecture overview.” Web-
site, 2008. http://www.openehr.org/releases/1.0.2/architecture/
overview.pdf.
[38] M. Eichelberg, T. Aden, J. Riesmeier, A. Dogac, and G. B. Laleci, “A survey
and analysis of Electronic Healthcare Record standards,” ACM Computing
Surveys, vol. 37, pp. 277–315, Dec. 2005.
[39] “CEN.” Website, 2011. http://pangea.upv.es/en13606.
[40] J. Gupta, G. Forgionne, and M. Mora, Intelligent Decision-making Support
Systems: Foundations, Applications and Challenges. Springer-Verlag New
York, Inc. Secaucus, NJ, USA, 2006.
[41] T. Hewett, R. Baecker, S. Card, and T. Carey, “ACM SIGCHI Curricula
for Human-Computer Interaction,” 1996.
[42] M. G. Helander, T. K. Landauer, and P. V. Prabhu, Handbook of Human-
Computer Interaction. Elsevier Science Pub Co, Aug. 1998.
[43] ISO 9241-11, Ergonomic requirements for office work with visual display
terminals (VDTs)Part 11: Guidance on usability. Geneva, Swiss: Interna-
tional Organization for Standardization, 1998.
[44] M. Maguire, “Methods to support human-centred design,” International
Journal of Human-Computer Studies, vol. 55, pp. 587–634, Oct. 2001.
[45] H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human-
Computer Interaction. Wiley, 2007.
[46] K. Vredenburg, S. Isensee, and C. Righi, User-Centered Design: An Inte-
grated Approach. Prentice Hall PTR, Upper Saddle River, NJ, 2002.
[47] ISO 13407, Human-Centred Design Process for Interactive Systems. Geneva,
Swiss: International Organization for Standardization, 1999.
[48] “User-Centered Design.” Website, 2010. https://www-01.ibm.com/
software/ucd/ucd.html.
[49] B. Oates, Researching information systems and computing. Sage Publica-
tions Ltd, 2006.
[50] E. ˙
Arsand and G. Demiris, “User-Centered methods for designing patient-
centric self-help tools,” Informatics for Health and Social Care, vol. 33,
no. 3, pp. 158–169, 2008.
45
[51] R. A. K. Horasani, M. I. T. Anasijevic, B. L. M. Iddleton, and M. S. C,
“Ten Commandments for Effective Clinical Decision Support: Making the
Practice of Evidence-based Medicine a Reality,” Journal of the American
Medical Informatics Association, vol. 10, pp. 523–530, 2003.
[52] K. Kawamoto and D. Lobach, “Proposal for fulfilling strategic objectives of
the US roadmap for national action on decision support through a service-
oriented architecture leveraging HL7 services,” Journal of the American
medical, pp. 146–155, 2007.
[53] I. Cho, J. Kim, J. H. Kim, H. Y. Kim, and Y. Kim, “Design and im-
plementation of a standards-based interoperable clinical decision support
architecture in the context of the Korean EHR.,” International journal of
medical informatics, vol. 9, pp. 611–622, July 2010.
[54] Y. Huang, L. Noirot, and K. Heard, “Migrating toward a next-generation
clinical decision support application: the BJC HealthCare experience,” in
AMIA Annual, pp. 344–8, Jan. 2007.
[55] H. Aveyard, Doing a literature review in health and social care: a practical
guide. Open University Press, 2007.
[56] S. Porter, “An update of the etiology and management of xerostomia,” Oral
Surgery, Oral Medicine, Oral Pathology, Oral Radiology & Endodontics,
vol. 97, pp. 28–46, Jan 2004.
[57] M. Jontell, U. Mattsson, and O. Torgersson, “MedView: an instrument
for clinical research and education in oral medicine.,” Oral surgery, oral
medicine, oral pathology, oral radiology, and endodontics, vol. 99, pp. 55–
63, January 2005.
[58] J. Gulliksen, B. G¨oransson, I. Boivie, S. Blomkvist, J. Persson, and A. s.
Cajander, “Key principles for user-centred systems design,” Behaviour &
Information Technology, vol. 22, pp. 397–409, January 2003.
[59] R. Chen, P. Georgii-hemming, and H. ˚
A hlfeldt, “Representing a
Chemotherapy Guideline Using openEHR and Rules,” Medical Informat-
ics, pp. 653–657, 2009.
[60] S. Barretto, J. Warren, A. Goodchild, L. Bird, S. Heard, and M. Stumptner,
“Linking guidelines to Electronic Health Record design for improved chronic
disease management,” in AMIA Annual Symposium Proceedings, pp. 66–70,
American Medical Informatics Association, Jan. 2003.
[61] G. Schadow, D. C. Russler, and C. J. McDonald, “Conceptual alignment
of electronic health record data with guideline and workflow knowledge.,”
International journal of medical informatics, vol. 64, pp. 259–74, Dec. 2001.
[62] “OSHIP.” Website, 2011. http://www.oship.org/.
46
[63] “Open-EHR-Gen.” Website, 2011. http://code.google.com/p/open-ehr
-gen-framework/.
[64] “GastrOS.” Website, 2011. http://sourceforge.net/projects/gastros.
[65] “opereffa.” Website, 2011. http://opereffa.chime.ucl.ac.uk/introduc
tion.jsf.
47
Part II
Publications
Paper I
Towards Interaction Design
in Clinical Decision Support Development:
A Literature Review
Hajar Kashfi
International Journal of Medical Informatics, Elsevier.
(manuscript submitted)
51
Towards Interaction Design
in Clinical Decision Support Development:
A Literature Review
H.Kashfia,1,∗
aDepartment of Applied Information Technology
Chalmers University of Technology
SE–412 96 G¨
oteborg, Sweden
Abstract
Aim: After motivating the importance of human-related factors in developing highly adoptable
clinical decision support systems (CDSS) according to the previous studies, this paper presents
the results of a literature review on clinical CDSS published literature, with a focus on interac-
tion design (ID) activities that naturally deal with human-related factors in designing interactive
systems. Methods: Two related databases were searched without any limitations on the publi-
cation year. The search yielded a final collection of 17 studies. Relevance criteria included (i)
discussing development and or evaluation of a CDSS (ii) taking one or several ID activities into
account. Results: It was observed that the main emphasis of the literature has so far been on
evaluation after design which is more compatible with the traditional view of human-computer
interaction (HCI) rather than ID. It was also observed that evaluation methods that are based on
user participation were used more often than evaluation based on usability expert participation.
The review results indicate a need for disseminating the knowledge gained by experience and
the existing ID knowledge among CDSS developers. Conclusion: To develop highly adoptable
CDSSs and in order to overcome the chronic problem of CDSSs not being used in practice,
human-related factors are considered to play an important role. ID (or more traditionally HCI)
deals with such factors with an aim to develop usable interactive products. Applying the existing
ID knowledge and adopting various methods to support ID in the clinical domain and especially
in developing CDSSs provides highly valuable opportunities in developing more usable CDSSs
and clinical applications in general. Nonetheless, the adoption rate of ID activities among CDSS
developers is low. Educating CDSS developers about such existing approaches and methods,
together with disseminating the knowledge gained by experience in applying ID in developing
CDSSs are two means to improve the current situation.
Keywords: clinical decision support, clinical decision support system, user-centered design,
usability, qualitative evaluation, interactive system design, user interface evaluation,
human-computer interaction, interaction design
∗Corresponding author
Email address: hajar.kashfi@chalmers.se (H.Kashfi)
Preprint submitted to International Journal of Medical Informatics May 11, 2011
1. Introduction
More than 40 years of research have been spent on clinical decision support (CDS) but as
many studies reveal, clinical decision support systems (CDSS) have been more appreciated in
theory than in practice [1, 2]. Many of the developed CDSSs are mostly research prototypes and
designed for a specific context [2, 3] , therefore not suitable for mainstream use. Consequently,
CDS is still suffering from a low adoption rate [4, 5, 1, 2, 6]. Several researchers have been deal-
ing with specifying challenges that developers of such systems face, and defining success factors.
The final aim of these studies is to provide insight in developing more acceptable and adoptable
CDSSs. Many of these challenges and success factors belong to the category of human-related
factors and goes under the definition of usability. This has been the focus of the interaction
design (ID) research field for years. ID suggests various methods to design and evaluate inter-
active systems with focus on users of such systems with the aim of achieving usability and user
satisfaction. The interesting question is whether developers of CDSSs have applied the existing
knowledge in order to develop more acceptable CDSSs or not at the time usability evaluation is
considered to be crucial for developing interactive applications.
In an effort to discover the practical attitude of CDSS developers towards human-related
factors and ID, this paper reviews the related literature to find out more about the practical attitude
of the CDSS developers towards ID and the related concepts. We will start with a review on
difficulties and success factors reported in developing CDSSs. Then, results from the literature
review is presented. We will end with analysis of the results and suggestions for further studies.
2. Background
The issue of applying computers in clinical decision making is a challenging problem [1].
This difficulty is not limited to complex decision support processes. Even in the easiest cases,
the process of design, development, deployment and maintenance of a CDSS requires a huge
amount of effort.
Several studies have been conducted in order to answer the question of which factors should
be considered in design and development of CDS to result in an acceptable and effective CDS?
A broad range of difficulties and success factors in developing CDS have been identified accord-
ingly. A discussion of what has been observed in these studies is presented in the following; in
addition, a summary of the factors reported in these studies is tabulated in Table 1
•Technical Factors
A great deal of focus is on the issues related to knowledge extraction and maintenance.
Some of the technical factors reported in these studies are: compatibility with the legacy
systems [4], knowledge extraction [7, 4, 8, 6, 9], reliability of the knowledge, and tak-
ing knowledge from reliable sources [9, 7], maintaining, improving and monitoring the
knowledge base [9, 8, 8, 4, 7, 6], and creating new types of CDS interventions [5].
One of the other technical factors observed in the literature is integration of CDS into the
systems that are used to record, organize, and retrieve digitally stored health care informa-
tion of individual patients (electronic health record (EHR) systems). Integration of CDS
into EHR systems as one of the factors that are beneficial in wider adoption of CDS, is
advocated in several studies [2, 1, 5, 4, 7, 10]. Several studies discuss that delivery of
decision support through EHR can improve the quality of care [4, 3, 11, 12]. In overall,
EHR is considered as a leverage for CDS [6, 1].
2
•Efficiency
Being time efficient is considered to be one of the factors that affects adopting CDS [9, 8].
Efficient data entry is one other factor discussed in the literature [8, 13]. One suggestion
to improve efficiency has been to remove tedious duplicate data entry required by some
CDSSs [7] and to utilize existing data for instance by integrating the CDS into EHR sys-
tems.
•Users’ interaction with the system
The importance of user interface design is reported in various studies [14, 7, 5]. Consider-
ing the interaction of the user with the system is important in identifying how the system
is expected to be used [7]. There are many CDSSs that are reported to be implemented but
are not used in practice due to the poor user interface design among other factors [14].
In the ranked list of ten grand challenges in CDS development by Sitting et al. [5] the most
important challenge is considered to be improving the user interface. It should be noted
that clinical information systems with low usability not only do not improve patient care
and reduce clinical errors, but also may have the opposite effect [15, 16, 17, 18, 19, 20].
According to the literature, some detailed factors related to the interaction of users with
CDS are as follows: being understandable and controllable [9], anticipating when and what
information is needed and real time delivery of best knowledge available to overcome that
need at the point of care [8, 6, 21], and changing direction rather than stopping clinicians
[8].
•Workflow
Beside proper design of the user interface other human-related factors such as cultural
issues, users’ workflow and the complex context of use have been considered important
factors in developing CDS [22, 7]. Is it observed that the success of CDS is in relation to
its integration to the complex clinical environment [22]. CDS should fit and be integrated
into users’, i.e. clinicians’, workflow [8, 21, 4]. Especially since clinicians are resistant
to changes in their workflow [8]. Integration of the CDS to both culture and the care
workflow is considered to be necessary [7] in order for these systems to be used optimally.
•Users’ satisfaction and acceptance
As mentioned before, it has been demonstrated that CDS can have a potential influence
on health care, but there will be no effects on improvement in health care if the developed
system is not accepted by users and is not used in practice [7]. In order to design an accept-
able CDS it is required to understand clinicians’ characteristics [8]. Adoption planning of
a CDS should be done in relation to the end users [9]. It is recommended that in case of
using commercial products, the system should be customizable for local use [13]. Finally,
users should be trained properly for using the system and they should also be informed
about limitations of providing automatic CDS so that their expectations are adjusted [13].
All in all, user acceptance and satisfaction are considered to be important factors in adopt-
ing CDS [4], therefore, measuring and considering the user reaction to the CDS is crucial
in order to develop a successful application [23].
•Monitoring, getting feedback and evaluating
3
In order to improve the CDS and also to keep the knowledge base updated, maintenance
of CDS and getting feedback from clinicians is needed [4, 8]. Clinicians should somehow
be motivated to apply CDS [7]. Effectiveness of CDS should be evaluated by the health
organizations [7] Moreover, health organizations should monitor the usage of the system
after deployment [8, 21, 13, 23, 1] to assure proper adoption of CDS by clinicians.
Who Year Difficulties and success factors
non-technical technical
Wetter [9] 2002 ◦Planning in relation to the end users
◦Time and cost efficiency
◦Understandability and controllability
◦Reliability of the knowledge base
1
Trivedi et al. [23] 2002 ◦Human related issues
◦Organizational issues
◦Technical issues
Bates et al. [8] 2003 ◦Efficiency
◦Real time delivery of information
◦Presenting correct information at the
correct time
◦Fitting into user’s workflow
◦Changing direction rather than stopping
◦Understanding clinicians’ characteristics
◦Simple interventions
◦Efficient data entry
◦Monitoring and getting feedback
◦Maintaining the knowledge base
Kawamoto et al. [21] 2005 ◦Integration of automatic decision support
to clinician’s workflow
◦Timely access to decision support at point
of care
◦Providing actionable recommendations
◦Monitoring the use of system after
deployment
◦Computer aided decision support1
Garg et al. [4] 2005 ◦User acceptance
◦Integration to the user workflow
◦Being compatible with the legacy
systems
◦Maturity of the developed system
◦System maintenance
Table 1: Clinical Decision Support Challenges (continued on next page).
1In this paper 22 technical and non-technical factors are specified five of which are considered to be highly
correlated to the success of CDS.
4
Who Year Difficulties and success factors
non-technical technical
Osheroff et al. [6] 2007 ◦Social and technical aspects of developing
a CDSS 2
◦Best knowledge be available when needed
◦High adoption and effective use
◦Natural complexity of decision
making
◦Knowledge extraction issues
◦Continuous improvement of
knowledge and CDS interventions
Berner et al. [13] 2007 ◦Data entry process
◦User interface and vocabulary
◦Motivation of use for clinicians
◦Informing users about the limitation
◦User training
◦Evaluation of the system
◦Customizing commercial applications for
the local use
◦Monitoring the usage of the system after
deployment
◦Knowledge-base creation and
maintenance
◦Reliable knowledge base
◦Monitoring and maintaining the
knowledge base
Sittig et al. [5] 2008 ◦Improving the effectiveness of CDS
interventions
◦Creating new interventions
◦Disseminating existing CDS
knowledge and interventions
Cho et al. [10] 2010 ◦Defining a national CDS
architecture 3
◦Integration to the EHR context
◦Having access to a sustainable and
robust CDS
◦A sharable and reusable
knowledge base
Table 1: Clinical Decision Support Challenges (continued from previous page)
2In this paper three pillars in order to increase the widespread adoption of effective CDS are demonstrated.
Several activities are also recommended considering these three pillars.
3Cho et al. in 2010 published their experience in defining and implementing a national CDS architecture
with the aim of broad adoption of CDS services in Korea. The goal of this study has been to“achieve
widespread adoption of EHR by health care organizations to improve the quality, safety, and efficiency of
care”, “to establish a sharable lifetime EHR system to improve the quality of care and reduce health care
costs incurred by care redundancy in Korea”. They demonstrate that to reach these goals it should be assured
that clinical application providers have access to a “sustainable and robust CDS” when it is needed.
Putting technical factors aside, the related literature suggests the importance of (i) under-
standing users, i.e. clinicians, their needs and characteristics and designing a system according
to them (ii) understanding users’ workflow and design a system that fits into the workflow and
requires the minimum changes in the workflow (iii) user interface and the way users interact
with the system (iv) efficiency of the system (v) finally, getting feedback from users and assuring
users’ satisfaction and acceptance.
The interesting point is that the aforementioned issues have not been in focus just in the
CDSS field. Around 30 years ago, the need for investigating such factors in developing inter-
5
active systems in various domains resulted in the creation of a multidisciplinary field concerned
with designing, evaluating and implementing usable interactive systems with focus on users of
such systems, i.e. human [24, 25]. This field is called Human-computer interaction (HCI) and
later extended to Interaction design (ID) [26]. More about interaction design is discussed in the
following section.
2.1. Interaction Design
ID concerns the development of interactive systems that are usable [26]. By usable it is meant
easy to learn, effective to use and providing an enjoyable user experience [26]. User experience
is how a user feels about a specific product and includes various qualities such as satisfaction
and pleasure [26]. ID is concerned with theory, research and practice of design [26] and covers
a wider scope of issues, topics and paradigms compared to what HCI has traditionally been con-
cerned with [26]. Sharp [26] defines ID as “designing interactive products to support the way
people communicate and interact in their everyday and working lives.” ID relies on understand-
ing the people, i.e. users, what they desire to perform, i.e. tasks and goals, their characteristics
and capabilities, and the technology available. Moreover, it involves the knowledge of identify-
ing requirements and evolving them into a proper design solution [26]. The ID process involves
the following activities [26]:
•“Identifying needs and establishing requirements for the user experience”
•“Developing alternative designs that meet those requirements”
•“Building interactive versions of the design”
•“Evaluating what is being built throughout the process and the user experience it offers”
The heart of ID process is evaluation of the design solution and to ensure that the product is
usable. This is addressed via a user-centered design (UCD) approach [26].
2.2. User-centered Design
UCD means that in a development process, the real users of the product, their goals and not
just the technology should be considered as the driving forces. As it is suggested by its name,
UCD aims for involving users throughout the design process. There are various methods and
techniques that help to achieve this. Three principles are considered to be the basis for UCD.
These principles are:
•Early focus on users and tasks: understanding users and involving them in the design
process (see Section 2.7 and Section 2.6).
•Empirical measurement: observing and measuring performance and reactions of users
when they interact with the product (see Section 2.8).
•Iterative design: repeating the cycle of design, evaluate and redesign as often as required
(see Section 2.4).
UCD principles appear to be obvious metrics but they are not easy to put into practice [26]. More
about the concepts related to ID and UCD can be found in the following sections.
6
2.3. The Life-cycle Model
To perform ID, two factors are important (i) understanding of the ID activities (ii) specifying
the relation of activities to one another and the full development process [26]. The latter is called
life-cycle model or process model. A life-cycle model may also include the when, and how to
move from one activity to the next one, also the description of the deliverables of the activities
[26].
2.4. Iterative Design
To design an interactive system, a clear understanding of the requirements is necessary. Nev-
ertheless, it is observed that not all of the system requirements can be found before starting the
design [27]. To overcome this problem, the idea of iterative design was introduced. The core
idea of iterative design is to have a design process that tries to overcome the problem of incom-
plete requirements specification. This is realized by cycling through the process of design and
incrementally improving the product produced in each pass.
Iterative design involves using prototypes. Prototypes are defined by Dix [27] as “artefacts
that simulate or animate some but not all features of the intended system.” Prototypes are useful
in discussing ideas with stakeholders and they can be used as a communication mean among the
team members [26]. Prototyping provides cheaper and faster opportunities to evaluate a design
solution. A prototype complexity can range from a simple paper-based simulation of the system
to a complex piece of software [26].
2.5. Different Types of Requirements
Sharp [26] defines requirement as “a statement about an intended product that specifies what
it should do or how it should perform”. Requirements should be specified as clearly as possible
via the requirements gathering activities. There are different kinds of requirements such as func-
tional requirements, environmental requirements, i.e. the circumstances in which the system will
be used (context of use), usability requirements, and user characteristics [26]. Therefore, it is
important to apply data gathering techniques in a way that this broad range of issues are covered.
2.6. Data gathering methods
Data gathering methods are used to collect sufficient, precise and relevant data with the aim
of producing the requirements [26]. The users’ tasks, their related goals, the context in which the
tasks are performed and the rationale for the current situation are the most important information
that is needed to be collected via data gathering techniques. Data gathering techniques are also
used in evaluation processes to collect users’ reaction and performance with the prototype of
the intended system [26, 28]. Interview, questionnaire, observations, studying documentation,
and researching similar products are some of the data gathering techniques [26]. Selection of
the data gathering techniques should be based on the participants involved and the nature of the
technique. In order to triangulate the results, usually, data gathering techniques are combined
[26].
2.6.1. Interview
Interview is a friendly and flexible data gathering technique where a goal oriented conversa-
tion is conducted with the interviewee [28, 26, 27]. In evaluation, interviewing users about their
interaction with the system provides an opportunity to gather information about the user’s expe-
rience. Various types of interviews are defined based on the amount of control the interviewer
imposes on this conversation: structured, unstructured, semi-structured and group interview [26].
7
2.6.2. Observation
Observation is a popular data gathering technique that gathers information about how users
achieve a task in the context and also the actual interaction of users with the system [26, 27].
One problem with simple observation is that it does not provide insight into the user’s decision
process and attitude while interacting with the system [27]. One approach to overcome this
problem is to ask users to verbalize their thoughts. This technique is called think aloud [26, 27].
2.6.3. Questionnaire and Survey
Questionnaires are a set of pre-designed questions that are used for collecting demographic
data and users’ opinions [27, 26]. Questionnaires are less flexible than interviews but they can
target a larger group of participants and they help in gathering more precise information [27, 28].
They need less administrative time, and the analysis of the gathered information can be done
more rigorously [27].
2.6.4. Usability Engineering
Usability engineering is an approach to UCD where a usability specification is produced as a
part of the requirements specification. This specification includes the exact criteria, i.e. usability
goals, against which the system will be evaluated. In order to set up usability goals, there is a need
to specify relevant measurable usability characteristics of the product, i.e. usability attributes [25]
and to decide how they are going to be measured.
2.7. Task Analysis
In order to design interactive systems, it is crucial to have a clear understanding of the tasks
the users want to perform [27]. Task analysis is focused on studying the user’s actions and and/or
cognitive processes to achieve the tasks [29]. It deals with existing systems and procedures and
investigates the existing situation [26, 27]. It also contributes to the new system requirements
specification [27]. Task analysis involves using various data gathering techniques such as inter-
views and questionnaires [30].
2.8. Design Evaluation
Even in case of applying UCD, there is a need to assess the design solution and to make
sure that the requirements are met [27]. This is the aim of evaluation. It is recommended not
to consider evaluation as a single phase in the design process but to carry it out throughout
the system life-cycle and to consider evaluation results in improving the design solution [27].
Evaluation should involve assessing not only the functional capabilities of the system, but also
the users’ interaction with the system, their experience and the impact of the interaction on
her/him. Therefore, various aspects such as learnability (how easy it is to learn the system),
usability and the user’s satisfaction should be considered in evaluation.
Various categorizations of evaluation methods are documented in the literature [25, 26, 27].
Helander in [25] defines two main classes of methods. The first class includes those methods
that rely on the users’ participation in the evaluation. This class of evaluations is called empirical
evaluation methods [25]. The second class of methods does not rely on direct user involvement
but on designers’ or usability experts’ judgment of the design. These methods are called usabil-
ity inspection or non-empirical evaluation methods [25] (Dix [27] calls this class expert analysis
methods). Another categorization of methods exists in [26] that divides methods into three cate-
gories of usability testing,field study and analytical evaluation. In this paper, the categorization
made by Helander [25] is used for referring to methods.
8
2.8.1. Empirical Evaluation Methods
Usability testing and field studies are two empirical evaluation approaches. Both usability
testing and field studies use basic data gathering techniques (observation and interview) to gather
information on how users interact with a system or how do they perform a task. Their main
difference is in the settings the observation is performed in.
Usability test is a collection of methods used to evaluate the usability of a product. It is
typically focused on how well a user can use the product to accomplish a specific task, and also
errors that occur during this process [27]. Usability test [26] involves observing a user while the
user is interacting with the system. It is performed in a controlled environment for instance in a
laboratory setting. This means that in this type of observation the test users are isolated from the
usual interruptions in their everyday work.
In contrast to usability testing, filed study [26] involves observing users in their natural work
settings. Field studies can be helpful in requirements gathering as well as evaluating a system.
2.8.2. Usability Inspection Methods
There are circumstances in which developers do not easily have access to users, user partic-
ipation is too expensive or it takes too much time to involve them. In those situations, usability
inspection methods can be used to discover design flaws since in contrast to empirical evaluation
methods, in this category of evaluation methods users do not need to be present [26]. In addition,
for empirical evaluation methods, normally a working prototype or implementation of the sys-
tem is required while usability inspection methods can be performed on very early designs and
prototypes. In the following, three main methods that belong to this category of evaluations are
given.
Heuristic Evaluation
Dix defines heuristic [27] as “a guideline or general principle or rule of thumb that can
guide a design decision or be used to critique a decision that has already been made.” Heuristic
evaluation is a methods that applies a set of heuristics to discover the design flaws. Heuristic
evaluation can be performed early in the project even on the design specification. This method
is considered to be one of the cheapest evaluation methods [27]. It is also called a resource
constraint method since running this method does not need a deep knowledge in usability and
ID and it can be done by developers in case the project team does not have access to usability
experts [25].
Walkthroughs
Walkthroughs are alternative approaches to heuristic evaluation. The same as heuristic eval-
uation, walkthroughs rely on predicting users’ problems without carrying out user tests [26].
Walkthroughs involve walking through a specific task and discovering the usability problems.
The focus in walkthroughs is on learnability of the system, i.e. how easy the system is to learn
[25, 26].
Usually, walkthroughs do not involve users, however, in a sub-class of walkthroughs named
pluralistic walkthroughs [25, 26], users, developers and usability experts work together to find
out usability problems of the design solution.
Usability-Expert Reviews
Usability-Expert review is an evaluation method when usability experts review a system de-
sign against standard and best practice design solutions in order to find out general design flaws
9
[25]. This method is very similar to heuristic evaluation. The main difference is that in this
method, experts usually do not use heuristics to run the test. They actually rely on their own
experience with system and users to identify the source of difficulties in the GUI design [25].
2.9. User-centered Design in Developing Clinical Decision Support Systems
Involvement of end users, i.e. clinicians, in the design process and adjustment of the design
based on their feedback is one way to improve the acceptance of a CDSS. This is achieved by
understanding clinicians, their characteristics, their goals and the tasks that they wish to perform,
and consequently by producing a usable CDSS that fits to that specific clinical context.
As evident from the discussion in the previous sections, ID provides the knowledge required
by CDSS developers (and generally developers of all types of products) in order to develop highly
adoptable CDSSs. But the question is whether this knowledge is applied by the developers or
not. This has been the motivation of conducting this systematic review.
3. Search Method
Systematic review is a research methodology that aims at summarizing the available literature
on a topic and presenting an analysis based on that and providing a full picture on the topic [31].
In this study, a systematic review is used to analyze state of the art in CDSS developers’ attitude
towards ID and ID activities.
The publications were selected from two databases namely ScicenDirect1and PubMed2.
These two databases cover the most important journals in medical informatics, ID and also well-
known conferences in medical informatics.
The search was performed using various boolean combinations of these keywords: “medical
decision support”, “clinical decision support”, “user-centered design”, usability, “user involve-
ment”, “interaction design” and “human-computer interaction”. Later, in two steps, the abstracts
and the contents of the returned results were studied and the most relevant papers were selected.
The search strategy is depicted in Figure 1. Criteria for selection of the papers were positive
answers to these questions:
•Is the study discussing development and or evaluation of a CDSS (i.e. practical science)?
•Are any of the ID concepts or activities considered in the study ( i.e. ID consideration)?
The literature was reviewed to discover whether the ID related concepts, methods and approaches
are documented by the authors. A list of related concepts and methods was incrementally created
based on what was found in the literature (see Table 2 for the list).
4. Results
The literature was reviewed to find out which ID/HCI activities and concepts are documented
by authors. An overview of the ID principles and methods presented in these studies is shown
in Figure 4. The results from this review are presented in the following. More analysis of the
results is given in the next section.
1http://www.sciencedirect.com
2http://www.pubmed.gov
10
Primary searches
N=158
104 excluded
no ID consideration
not practical science
Abstract relevant
N=54
43 excluded
no ID consideration
not practical science
duplicate
Content relevant
N=17
Figure 1: The search strategy
4.0.1. Importance of the User Interface
The literature indicates a general consensus that usability of the user interface plays an im-
portant role in the system acceptance and reducing technology-induced errors [32, 33, 34, 35,
36, 15].
23.5%
Process view (Life-cycle/UCD view) (4)
76.5%
Independent (13)
Figure 2: only in around 24% of cases the concept of life-cycle of the system, and/or consideration of user-centered
design as a process is reported.
11
82.4%
Only user (14)
17.6%
Both user and expert (3)
Figure 3: Evaluation with user participation vs. evaluation with expert participation
4.1. Evaluation Before or After Releasing the Application
Evaluation before releasing the application is typically conducted on a prototype of the sys-
tem [33, 37, 23, 34, 38, 35, 39, 40, 36, 15, 41, 42, 43]. On the other hand, there are cases in
which evaluation is done after releasing the system [32, 44, 45, 46]. In some cases authors do
not make it clear whether the evaluation is made on the system after release or in early stages of
the project before releasing the application [39].
4.1.1. User Involvement
Evident from the literature, involving users in evaluation (see Section 4.2) is more common
than involving them in design. Very few of the studies have documented involvement of users in
design (two studies [38, 35]).
4.1.2. Context of Use
Few studies documented efforts to understand clinicians, their characteristics and the context
of use [41, 42].
4.1.3. The life-cycle model
In contrast to most of the literature, there are few studies that have presented a life-cycle view
on UCD [35, 36, 42]. Some researchers have concluded that using UCD as a process throughout
the whole life-cycle of the system is necessary to design a usable CDS [42].
4.2. Empirical Evaluation
As evident from Table 2, empirical evaluations such as observations [32, 47, 35, 41, 15, 42,
46, 43], interviews [32, 33, 38, 39, 42], questionnaires/surveys [48, 23, 38], field study [48, 49,
41, 42], usability tests [48, 33, 37, 34, 36, 15, 41, 42], and think aloud [33, 37, 38, 35, 15, 42] are
the most popular evaluations documented in the literature.
4.3. Non-empirical Evaluation
Literature suggests that documented cases of applying non-empirical evaluation methods are
much fewer than those of empirical methods. For instance, cognitive walkthrough and heuris-
tic evaluation are applied only in two of the studies [37, 35]. Expert review is missing in the
literature.
12
UCD/life-cycle (4)
24%
Evaluation before (13)
76%
Evaluation after (4)
24%
Multi-disciplinary team (5)
29%
Iterative design (11)
65%
Prototyping (13)
76%
Usability test (9)
53%
Interview (15)
88%
Questionnaire (10)
59%
Survey (7)
41%
Field study (6)
35%
Observation (8)
47%
Heuristic Evaluation (2)
12%
Cognitive Walkthrough (2)
12%
Task analysis (1)
6%
Figure 4: Interaction design (ID) in developing clinical decision support systems (CDSS)
4.3.1. Challenges and Benefits
The literature indicates a general consensus that applying various ID methods, and the UCD
process is effective; and results in more usable CDSSs and will eventually increase the usage of
the system [37, 38, 36, 41]. Nevertheless, some of the studies have concluded that applying UCD
and some of its methods is difficult and challenging [37, 50].
4.4. Summary of Findings
•The qualitative evaluation on the system was performed after releasing the system to the
users in around 24% of the studies.
•Prototyping is used in 76% of the studies.
•The concept of iterative design is considered in 65% of the studies.
•Only in 24% of the studies the concept of life-cycle of the system, and/or consideration of
user-centered as a process is reported (see Figure 2)
13
76.5%
Evaluation before release (13)
23.5%
Evaluation after release (4)
Figure 5: Evaluation of the prototype before releasing the final product, evaluation after installing the product
•Evaluations based on “user participation” (i.e. empirical evaluations) are popular among
developers of CDS (see Figure 3). As depicted in Figure 6, “interview” is used in 88%,
“questionnaire” in 59%, “usability test” in 53% and “field study” in 25% of the studies.
•Only around 18% of the studies have applied non-empirical evaluations (heuristic evalua-
tion, and cognitive walkthrough each in 12% of cases).
5. Discussion
As evident from Table 1, success factors of CDSSs can be divided into two main categories
of technical and non-technical factors. The ID community specifies various methods that help
developers assure these human-related aspects of the system. The aim of this literature review
was to investigate whether existing knowledge in the ID field is applied properly by developers
of CDSSs with the aim of developing more acceptable and adoptable CDS. In the following, a
more detailed analysis of the literature is provided along with discussions to provide insight into
the practical attitude towards ID in the CDS field.
5.1. Interaction Design in Practice
The search strategy in this review resulted in discovering only 17 studies which is a small
number compared to the number of CDSSs being developed and evaluated. Only in one of the
databases, ScienceDirect, around 100 studies were discovered that, according to their titles and
abstracts, have documented developing and/or evaluating a CDSS.
Studies are distributed from 1997 to 20103. More than half of the studies are published
after 2006. This indicates that consideration of ID/HCI in developing CDSSs is becoming more
popular in recent years.
5.1.1. Early focus on Users and Tasks
The first UCD principle is to have early focus on users and tasks (see Section 2.2). This
means that users should be understood and involved in the full design process from the begin-
ning. This however does not mean that users should be actively involved in design or act as
3The search was carried out in October 2010. So it may not include all of the 2010 publications.
14
Reference [32] [33] [37] [23] [34] [44] [45] [38] [35] [39] [40] [36] [15] [41] [42] [46] [43]
Year 97 98 02 02 03 04 05 06 07 07 07 08 08 08 09 09 10
UCD 777777777 7 7 7 7 7
Life-cycle 7 7 777777 7 7 7 7 7
Cognitive aspects 7777777777777 7
Evl. Before 17 7 7 7
Evl After 27777 777777777
Multi-disciplinary team 7777777 7 7 77 7
Iterative design 77 7 7 7 7
Usability 7 73
Usability test 7 7 7 7 7 447 7 7
Usability engineering 7 7777777777777
User involvement 7
Think aloud 7 7 7 7 7 *7 7 7
Prototyping 7 7 7 7
Interview 7 7
Questionnaire 7 7 7777 7
Survey 777 7 7777 7 7
Walkthrough 7 7 77777*7 7 7 7 *7 7
Heuristic evaluation 7 7 77777*777777 7
Field study 777777777 7 7
Observation 777777777
Task analysis 777777777777 7 *7 7
Workflow 77 7
Workflow integration 77 *7 7
Training 7 7 *77 7
Users involved 6 14 8 16 516 35 6 14 713 7 7 8 96 20
1Evaluation before release
2Evaluation after release
3It is called problem
4It is called interview
5Not clear in the text, obtained from correspondence with the author.
*It is mentioned in the paper but is not used practically
Table 2: Human Computer Interaction and Clinical Decision Support in practice (note that indicates the concept/method
is applied in the study.).
15
designers. It is enough if users are consulted when making design decisions [26]. To support the
importance of understanding the users, their goals and tasks, Sharp [26] indicates: “if something
is designed to support an activity with little understanding of the real work involved, it is likely to
be incompatible with current practice, and users do not like to deviate from their learned habits
if operating a new device with similar properties.” Various data gathering techniques as well as
task analysis can be used to support this principle.
Nonetheless, the review findings demonstrate that CDSS developers have not been widely
upholding this principle. Data gathering methods applied in various studies have been mostly
used in evaluations rather than gathering various types of requirements (including understanding
users, and users’ characteristics and the context of use). Task analysis is used only in one study
[36]. Finally, almost none of the studies have documented the usability requirements as a type of
requirements used to motivate design and provide basis for evaluation.
All in all, it is evident that benefiting from various data gathering techniques in order to col-
lect different types of requirements is not considered in many of the studies, and the knowledge
on this needs to be improved in the field. It is important that those aspects of the clinical con-
text that make it different from other domains in terms of collaborating with users and applying
various evaluation or design methods be investigated carefully. Moreover, further investigation
is needed to find out why task analysis, and also formal methods for understanding the context
have not been popular in the CDS field.
5.1.2. Empirical measurement
The second UCD principle is to observe and measure performance and reactions of users
when they interact with the system (see Section 2.2). This principle is satisfied in an acceptable
level in the studies documented in the literature. Usability tests, and various data gathering
techniques were used to gather users’ feedback and to observe and analyze users’ interaction
with the CDSS to discover design problems.
5.1.3. Iterative Design
The third UCD principle is to have an iterative design process (see Section 2.2). Iterative
design as mentioned in the results, has been considered in 65% of the studies. In some studies it
is not clearly mentioned what has been the goal of evaluations and how or when the evaluation
results would be used. Surely, getting users’ feedback and evaluations would be of no use if
they do not inform the design. Considering the importance of iterative design in developing
usable interactive systems (making informed design improvements based on evaluations), this is
an indication of a need for improvement in this area.
5.1.4. Interaction Design versus Human-computer Interaction
As mentioned before, ID focuses on the whole design process but the focal point of most
of the studies is on the evaluation. As evident from the reviewed literature, most of the works
documented in the literature match the more traditional pattern of HCI rather than ID. The focus
of traditional HCI has been mostly on evaluation and pointing out design flaws after design while
ID and the UCD approach to ID emphasize not only on the evaluation but also the whole design
process.
Moreover, in database searches, using the phrase “interaction design” did not result in any
hits. None of the studies has actually used this term in the text. On the other hand, the phrase
“human-computer interaction” is used in some of the studies. This is also in line with the afore-
mentioned observation.
16
5.1.5. The Life-cycle Model
As mentioned in Section 2.3, it is important to specify how various ID activities are going to
be carried out in the life-cycle of the system. Also, there is a need for an early planning of the
UCD process in order to make various important decisions for instance how many users will be
involved in the process and how their involvement is managed [29].
The literature contains very few studies that discuss the theoretical background of UCD as a
design and development process that can be used in the life cycle of the project as a complemen-
tary process to existing software development methods [42, 36, 35]. In addition, the literature
does not include discussions on the UCD planning. This is in line with our observation that the
focus lies mostly on evaluation rather than the whole design process.
5.1.6. Design Evaluation
Evaluating design solutions before releasing the application is performed in many of the
studies which shows an acceptable level of knowledge on this concept especially in recent years
(since only one study [46] was found after 2006 in which evaluation was done after releasing the
system). But still, not all of the evaluation methods, especially non-empirical evaluation methods
have been applied so often in the field.
There are costs and benefits associated with each class of evaluations. To carry out empirical
evaluations, developers should have access to users that are willing to participate in the evaluation
process. To accomplish the non-empirical methods, access to usability experts or developers who
are able to conduct evaluation is required. If usability experts are not available in the project,
getting access to them might be costly for the project.
Regarding benefits linked to each type of evaluation, it is observed that on various occasions,
especially in early stages of the project, non-empirical methods (or as it is called in some liter-
ature, usability inspection methods) can be used to identify general design flaws, discover the
compliance to design standards or heuristics or even detailed qualities such as ease of learning
[25]. On the other hand, these methods are not considered to be a replacement for the empirical
methods especially since the general consensus is that some type of flaws in the system will be
discovered only by having users interact with the system [25].
All in all, it is recommended that choice of evaluation methods be made considering the
timing of evaluation, project progress, cost, and most importantly the goal of evaluation. This
however is almost absent in the literature reviewed. Also, the goals of evaluation are not clearly
specified. There is rare discussion of how various methods are chosen in the studies. For instant
just few cases such as [50, 41] discuss benefits and costs associated with various methods. This
would indicate the following:
•Users in this domain are easy to access and their involvement in evaluation is cheap.
•Usability experts have not been accessible in such projects and or their involvement is
costly for the project.
•Even if the above statements are true, very few cases of applying heuristic evaluations
(where even developers with little knowledge on usability can run the test to discover de-
sign flaws) suggests the lack of knowledge in this area among CDSS developers. Also,
since not all of the design flaws can be discovered by users (especially being compati-
ble by general design patterns) absence of non-empirical methods can be a sign of poor
knowledge on ID in this domain.
17
Usability test (9) 53%
Interview, questionnaire, survey (17) 100%
Field study, observation (9) 53%
Figure 6: A comparison between various evaluations based on user participation.
5.1.7. General Discussion
The problem of low adoption rate of CDS has long been of concern to the researchers in this
field. As a response to a need for answering the question of whether a CDSS will be used by users
and how it will be used, researchers have previously done studies on various CDSS evaluation
approaches [51, 52].
According to a literature review by Kaplan [51], very few evaluations of CDSSs can be
found that focus on other aspects such as “why clinicians accept or do not accept a system” or
“why they change their practice behavior after introducing the system”. In her literature review
on the CDSS evaluation literature [51], Kaplan indicated that “in the evaluation literature, the
main emphasis is on how clinical performance changes. Most studies use experimental methods
or randomized controlled trials (RCT) to assess system performance or to focus on changes in
clinical performance that could affect patient care.” She has profiled 34 CDSS evaluation studies
in her review. Of those, only 12 studies had evaluated other aspects than performance using
various qualitative evaluation techniques such as observations, interviews, field studies and so on
[51].
In another related study [52], she puts steps further forward towards the general issue of
evaluating clinical applications in an attempt to call for alternative approaches. She mentions that
“unless evaluation approaches include social, organizational, cultural, cognitive, and contextual
issues, they cannot answer key questions about why clinicians use or do not use an informatics
application.”
While confirming Kaplan’s observations, we indicate that though qualitative evaluation meth-
ods may be able to answer the aforementioned questions, they cannot broaden the adoption rate
of CDS and the problem of CDSSs not being used in practice, unless they are put together with
a well-informed design and development process. This type of evaluation of course provides
an opportunity to find out the reasons for low adoption. But to uphold the success factors of
CDS documented by researchers in this field (see Section 2), there is a need for activities other
than just evaluation. This would include understanding clinicians, their tasks and their goals,
understanding their characteristics and the context of use of CDS as recommended in the ID
filed.
The next step would be to investigate existing methods and approaches and identify how they
fit into the clinical domain. For instance challenges in applying heuristic evaluation is discussed
in [50] and is being related to various factors such as complexity of the context, and limitations
of the method itself (for instance, the author mentions the existence of some collaborative tasks
in the clinical domain in which more than one user is involved in task completion; while heuristic
evaluation considers independent evaluators in the evaluation process).
In addition, considering that this literature review shows lack of knowledge in some areas,
there is a need to disseminate the existing ID knowledge and the knowledge gained by experi-
ences in developing CDSSs among CDSS developers using UCD.
18
6. Conclusion
It is shown that in the clinical domain, taking human-related (ID-related) factors into consid-
eration is highly recommended in theory especially in recent years.
Although compared to previous studies [51], this review suggests that qualitative evaluations
of CDSSs are gaining more interest and attention, the rate of applying various ID methods and the
UCD approach is still low among CDSS developers. Especially, compared to other domains such
as aviation or automobiles in which usability engineering, and user-centered design is routine
practice [47] and even compared to the clinical domain in general. Lastly, further investigations
to discover reasons for this are required, also the knowledge on ID and UCD and success stories
of applying them in various projects should be disseminated among developers of CDSSs to
improve the general attitude towards ID.
7. Future Work
As a continuation to this study, this research question should be answered: What are the chal-
lenges in applying user-centered design in a clinical context and how to tackle these challenges?
In order to find an answer to this question, the following objectives should be accomplished:
1. conducting a literature review in order to identify “the state of the art in the intersection
between ID and clinical application development” in general
2. gathering and investigating difficulties designers and developers of clinical applications
have faced so far
3. gathering and investigating the point of view of a group of developers and clinicians who
have been involved in any user-centered design process of a clinical application
4. providing a user-centered design guideline aimed at designers and developers of clinical
applications
Acknowledgment
The author extends her gratitude to Olof Torgersson, who provided useful and detailed feed-
back on the paper.
References
[1] R. Greenes, Clinical decision support: the road ahead, Academic Press, 2007.
[2] T. Wendt, P. Knaup-Gregori, A, Decision Support in Medicine: A Survey of Problems of User Acceptance, Stud
Health Technol Inform 77 (2000) 852–856.
[3] B. Chaudhry, J. Wang, S. Wu, M. Maglione, Systematic review: impact of health information technology on quality,
efficiency, and costs of medical care, Annals of internal Med 144 (10) (2006) 742–752.
[4] A. Garg, N. Adhikari, H. McDonald, M. Rosas, Effects of computerized clinical decision support systems on
practitioner performance and patient outcomes: a systematic review, JAMA 293 (10) (2005) 1223–1238.
[5] D. F. Sittig, A. Wright, J. a. Osheroff, B. Middleton, J. M. Teich, J. S. Ash, E. Campbell, D. W. Bates,
Grand challenges in clinical decision support., Journal of biomedical informatics 41 (2) (2008) 387–92.
doi:10.1016/j.jbi.2007.09.003.
[6] J. Osheroff, J. Teich, B. Middleton, E. Steen, A. Wright, D. Detmer, A roadmap for national action
on clinical decision support, Journal of the American medical informatics association 14 (2) (2007) 141.
doi:10.1197/jamia.M2334.Introduction.
[7] E. Berner, Clinical Decision Support Systems: Theory and Practice (Health Informatics), Springer, New York, NY
10013, USA, 2007.
19
[8] D. Bates, G. Kuperman, S. Wang, T. Gandhi, A. Kittler, L. Volk, C. Spurr, R. Khorasani, M. Tanasijevic,
B. Middleton, Ten commandments for effective clinical decision support: making the practice of evidence-
based medicine a reality, Journal of the American Medical Informatics Association 10 (6) (2003) 523–530.
doi:10.1197/jamia.M1370.Although.
[9] T. Wetter, Lessons learnt from bringing knowledge-based decision support into routine use., Artificial intelligence
in medicine 24 (3) (2002) 195–203.
[10] I. Cho, J. Kim, J. H. Kim, H. Y. Kim, Y. Kim, Design and implementation of a standards-based interoperable clinical
decision support architecture in the context of the Korean EHR., International journal of medical informatics 9
(2010) 611–622. doi:10.1016/j.ijmedinf.2010.06.002.
[11] D. Hunt, R. Haynes, S. Hanna, K. Smith, Effects of computer-based clinical decision support sys-
tems on physician performance and patient outcomes: a systematic review, Jama 280 (15) (1998) 1339.
doi:10.1001/jama.280.15.1339.
[12] M. Johnston, K. Langton, R. Haynes, Effects of computer-based clinical decision support systems on clinician
performance and patient outcome. A critical appraisal of reserach, Ann Intern Med 120 (2) (1994) 135–142.
[13] E. Berner, T. J. La Lande, Overview of Clinical Decision Support Systems, Springer, 2007, pp. 3–22.
[14] A. Hasman, Challenges for medical informatics in the 21 st century, International journal of medical informatics
44 (1) (1997) 1–7.
[15] T. Graham, A. Kushniruk, M. Bullard, B. Holroyd, D. Meurer, B. Rowe, How usability of a web-based clinical
decision support system has the potential to contribute to adverse medical events, in: AMIA Annual Symposium
Proceedings, Vol. 2008, American Medical Informatics Association, 2008, p. 257.
[16] J. Yy, Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician
Order Entry System., Pediatrics 116 (2005) 1506–1512.
[17] A. Kushniruka, M. Triolab, B. Steinc, E. Boryckid, J. Kannrye, The relationship of usability to medical error:
an evaluation of errors associated with usability problems in the use of a handheld application for prescribing
medications, in: Medinfo 2004: Proceedings Of THe 11th World Congress On Medical Informatics, Vol. 107, Ios
Pr Inc, 2004, p. 1073.
[18] A. Kushniruk, M. Triola, E. Borycki, B. Stein, J. Kannry, Technology induced error and usability: The relationship
between usability problems and prescription errors when using a handheld application, International Journal of
Medical Informatics 74 (7-8) (2005) 519–526. doi:10.1016/j.ijmedinf.2005.01.003.
[19] J. Saleem, E. Patterson, L. Militello, ML, Exploring barriers and facilitators to the use of computerized clinical
reminders, Journal of the American Medical Informatics (2005) 438–447doi:10.1197/jamia.M1777.On.
[20] R. Koppel, J. P. Metlay, A. Cohen, B. Abaluck, a. R. Localio, S. E. Kimmel, B. L. Strom, Role of computerized
physician order entry systems in facilitating medication errors., JAMA : the journal of the American Medical
Association 293 (10) (2005) 1197–203. doi:10.1001/jama.293.10.1197.
[21] K. Kawamoto, C. A. Houlihan, E. A. Balas, D. F. Lobach, Improving clinical practice using clinical decision
support systems: a systematic review of trials to identify features critical to success., BMJ (Clinical research ed.)
330 (7494) (2005) 765. doi:10.1136/bmj.38398.500764.8F.
[22] J. Anderson, Increasing the acceptance of clinical Information, MD computing: computers in medical practice
16 (1) (1999) 62.
[23] M. Trivedi, J. Kern, A. Marcee, B. Grannemann, B. Kleiber, T. Bettinger, K. Altshuler, A. McClelland, Develop-
ment and Implementation of Computerized Clinical Guidelines : Barriers and Solutions, Methods of information
in medicine 41 (5) (2002) 435–442.
[24] T. Hewett, R. Baecker, S. Card, T. Carey, ACM SIGCHI Curricula for Human-Computer Interaction (1996).
[25] M. G. Helander, T. K. Landauer, P. V. Prabhu, Handbook of Human-Computer Interaction, Elsevier Science Pub
Co, 1998.
[26] H. Sharp, Y. Rogers, J. Preece, Interaction Design: Beyond Human-Computer Interaction, Wiley, 2007.
[27] A. Dix, J. Finlay, G. Abowd, R. Beale, Human-computer interaction, Prentice-Hall, Inc., Upper Saddle River, NJ,
USA, 1997.
[28] D. Stone, C. Jarrett, M. Woodroffe, S. Minocha, User interface design and evaluation, Vol. 21, Morgan Kaufmann,
2005. doi:10.1057/palgrave.ivs.9500112.
[29] M. Maguire, Methods to support human-centred design, International Journal of Human-Computer Studies 55 (4)
(2001) 587–634. doi:10.1006/ijhc.2001.0503.
[30] A. Cooper, R. Reimann, D. Cronin, About Face 3: The Essentials of Interaction Design, Wiley, Indianapolis, IN,
USA, 2007.
[31] H. Aveyard, Doing a literature review in health and social care: a practical guide, Open University Press, 2007.
[32] B. Kaplan, R. Morelli, J. Goethe, Preliminary Findings from an Evaluation of the Acceptability of an Expert
System, in: Proceedings of the AMIA, 1997, p. 865.
[33] C. Gadd, P. Baskaran, D. Lobach, Identification of design features to enhance utilization and acceptance of systems
for Internet-based decision support at the point of care., in: Proceedings of the AMIA, 1998, pp. 91–5.
20
[34] C. Ying-Jui, A. T. Chirh-Yun, B. Y. Min-Li, B. Yu-Chuan, Li, Assessing the Impact of User Interfaces to the
Usability of a Clinical decision support system, in: AIMIA 2003 Symposium Proceedings, 2003, p. 808.
[35] K. Thursky, M. Mahemoff, User-centered design techniques for a computerised antibiotic decision sup-
port system in an intensive care unit, International journal of medical informatics 76 (10) (2007) 760–768.
doi:10.1016/j.ijmedinf.2006.07.011.
[36] A. Narasimhadevara, T. Radhakrishnan, B. Leung, R. Jayakumar, On designing a usable interactive system to
support transplant nursing., Journal of biomedical informatics 41 (1) (2008) 137–51. doi:10.1016/j.jbi.2007.03.006.
[37] C. Carroll, P. Marsden, P. Soden, E. Naylor, J. New, T. Dornan, Involving users in the design and usability evaluation
of a clinical decision support system., Computer methods and programs in biomedicine 69 (2) (2002) 123–135.
[38] S. J. Leslie, M. Hartswood, C. Meurig, S. P. McKee, R. Slack, R. Procter, M. a. Denvir, Clinical decision support
software for management of chronic heart failure: development and evaluation., Computers in biology and medicine
36 (5) (2006) 495–506. doi:10.1016/j.compbiomed.2005.02.002.
[39] A. Wilson, A. Duszynski, D. Turnbull, J, Investigating patients’ and general practitioners’ views of computerised
decision support software for the assessment and management of cardiovascular risk, Informatics in Primary Care
15 (2007) 33–44.
[40] D. F. Lobach, K. Kawamoto, K. J. Anstrom, M. L. Russell, P. Woods, D. Smith, Development, deployment and
usability of a point-of-care decision support system for chronic disease management using the recently-approved
HL7 decision support service standard., Studies in health technology and informatics 129 (Pt 2) (2007) 861–5.
[41] T. W. Marcy, B. Kaplan, S. W. Connolly, G. Michel, R. N. Shiffman, B. S. Flynn, Developing a decision support
system for tobacco use counselling using primary care physicians., Informatics in primary care 16 (2) (2008) 101–9.
[42] M. Peleg, A. Shachak, D. Wang, E. Karnieli, Using multi-perspective methodologies to study users’ interactions
with the prototype front end of a guideline-based decision support system for diabetic foot care., International
journal of medical informatics 78 (7) (2009) 482–93. doi:10.1016/j.ijmedinf.2009.02.008.
[43] J. Trafton, S. Martins, M. Michel, E. Lewis, Evaluation of the Acceptability and Usability of a Decision Support
System to Encourage Safe and Effective Use of Opioid Therapy for Chronic , Noncancer Pain by Primary Care
Providers, Pain Medicine 11 (2010) 575–585.
[44] A. S. Young, J. Mintz, A. N. Cohen, M. J. Chinman, A network-based system to improve care for schizophrenia: the
Medical Informatics Network Tool (MINT)., Journal of the American Medical Informatics Association : JAMIA
11 (5) (2004) 358–67. doi:10.1197/jamia.M1492.
[45] K. Zheng, R. Padman, M. P. Johnson, H. S. Diamond, Understanding technology adoption in clinical care: clinician
adoption behavior of a point-of-care reminder system., International journal of medical informatics 74 (7-8) (2005)
535–43. doi:10.1016/j.ijmedinf.2005.03.007.
[46] J. Saleem, L. Militello, N. Arbuckle, M. Flanagan, Provider Perceptions of Colorectal Cancer Screening Clinical
Decision Support at Three Benchmark Institutions, in: AIMIA Symposium Proceedings, 2009, pp. 558–562.
[47] J. Zhang, Human-centered computing in health information systems. Part 1: analysis and design., Journal of
biomedical informatics 38 (1) (2005) 1–3. doi:10.1016/j.jbi.2004.12.002.
[48] A. Kushniruk, V. Patel, J. Cimino, Usability testing in medical informatics: cognitive approaches to evaluation of
information, in: AMIA Annual Fall Symposium, 1997, pp. 218–222.
[49] M. Beuscart-Zephir, J. Brender, R. Beuscart, I. Menager-Depriester, Cognitive evaluation: how to assess the us-
ability of information technology in healthcare, Computer methods and programs in biomedicine 54 (1-2) (1997)
19–28.
[50] P. Edwards, K. Moloney, J. Jacko, F. Sainfort, Evaluating usability of a commercial electronic health record: A case
study, International Journal of Human-Computer Studies 66 (10) (2008) 718–728. doi:10.1016/j.ijhcs.2008.06.002.
[51] B. Kaplan, Evaluating informatics applications–clinical decision support systems literature review. (November
2001).
[52] B. Kaplan, Evaluating informatics applications–some alternative approaches: theory, social interactionism, and call
for methodological pluralism., International journal of medical informatics 64 (1) (2001) 39–56.
21
Paper II
The Intersection of Clinical Decision
Support and Electronic Health Record:
A Literature Review
Hajar Kashfi
1st International Workshop on Interoperable Healthcare Systems (IHS2011) -
Challenges, Technologies, and Trends, Szczecin, Poland, September 19-21, 2011.
(manuscript submitted)
75
The Intersection of Clinical Decision Support
and Electronic Health Record:
A Literature Review
H.Kashfia,1,∗
aDepartment of Applied Information Technology
Chalmers University of Technology
SE–412 96 G¨
oteborg, Sweden
Abstract
Aim: It is observed that clinical decision support (CDS) and electronic health records (EHR)
should be integrated so that their contribution to improving the quality of health care is enhanced.
In this paper, we present results from a review on the related literature. The aim of this review
was to find out to what extent CDS developers have actually considered EHR integration in
developing CDS. We have also investigated how various clinical standards are taken into account
by CDS developers. Methods: The ScienceDirect database was searched for related studies.
The search yielded a final collection of 25 studies. Relevance criteria included (i) discussing
development of CDS or an EHR with CDS services (ii) taking integration of CDS into EHRs
into account. Results: It was observed that there are few CDS development projects where EHR
integration is taken into account. Also, the number of studies where various clinical standards are
taken into consideration in developing CDS is surprisingly low especially for openEHR, the EHR
standard we aimed for. The reasons for low adoption of openEHR are issues such as complex and
huge specifications, shortcomings in educational aspects, low empirical focus and low support
for developers. Conclusion: There is a need for further investigation to discover the reasons why
the rate of integration of EHRs and CDS is not at an optimum level and mostly to discover why
CDS developers are not keen to adopt various clinical standards.
Keywords: Clinical decision support system, Electronic health record, clinical standards
1. Introduction
Even though more than 50 years of research have been put into the clinical decision support
(CDS) field, the adoption rate of these systems is still low [1, 2, 3, 4, 5, 6]. Various researchers
have investigated the factors that should be considered by developers of such systems in order to
result in higher adoption. One of these factors is the integration of CDS into the electronic health
record (EHR) systems. Different benefits are associated with the integration of CDS into EHRs.
For instance, integration facilitates real time access to the knowledge provided by CDS at point
∗Corresponding author
Email address: hajar.kashfi@chalmers.se (H.Kashfi)
Preprint submitted to Computer Methods and Programs in Biomedicine May 11, 2011
of care, it also eliminates tedious duplicate patient data entry since the preexisting digital patient
data in the EHR system can be utilized for the purpose of providing decision support [1, 7, 8].
The aim of this study has been to answer this research question: is integration of clinical
decision support into electronic health record taken into consideration by developers of clinical
decision support? The practical science literature was reviewed not only to explore CDS de-
velopers’ attitude towards integration of EHR and CDS, but also to discover the status of EHR
standards in this field.
The structure of the paper is as follows. We start with the background information including
the motivation for integration of CDS and EHRs in Section 2. In Section 3 the literature review
search strategy is given. The results of the review are presented in Section 4. Section 5 includes
the discussion of the findings along with our reflection on the low adoption rate of the openEHR
EHR standardization approach. Finally, we end with a conclusion and future directions of the
study in Section 6.
2. Background
The idea of computerized medical records has been around as one of the key research areas
in medical informatics for more than 20 years. Iakovidis defines EHR as “digitally stored health
care information about an individual’s lifetime with the purpose of supporting continuity of care,
education and research, and ensuring confidentiality at all times” [9]. EHRs include the whole
range of patient-related data such as demographic information, medical history, medication, and
allergies [10].
The main aim of EHRs is to make distributed and cooperating health information system and
health networks a reality [10]. The first benefit of deploying EHRs is that clinicians have access
to all patient information, as and when required, and provided that they have authorization for
that [1] since patient information is no longer on a piece of paper.
Several reasons have been identified for the low adoption rate of EHRs in small hospitals and
office practices. This includes high implementation and maintenance costs, additional time and
effort and finally the difficulty in choosing among available systems on the market due to a lack
of standardization [1].
Improving the quality of health care is the ultimate goal of the EHR research domain, but
it is in doubt whether EHRs have the ability to fulfill this goal [5]. EHRs need to be supported
by other services in order to improve the quality of care [5, 11, 12, 13]. To reach the goal of
improved health care quality, it is central to have CDS [5, 14, 3, 2, 6, 12, 15].
It has been observed that if there is no decision support service, the clinical knowledge needed
for making a decision is not always available or applied [16]. Therefore, it is recommended that
clinicians be automatically supported by timely access to clinical decision support tools [7, 8].
The emphasize in the current application of EHRs is on timely access to patient data, patient
tracking and providing decision support with the aim of improving quality of care [13]. In spite
of this fact, the usage of decision support among EHR users is still quite low and there are still
many EHR systems that do not include any CDS features [5]. Nonetheless, interest in applying
CDS in various health care organizations to improve quality of health care has recently shown an
increase [17, 18]. The CDS these organizations are looking for should provide support in patient
specific assessments [17, 1].
2
2.1. Low Adoption of Clinical Decision Support
Results from several studies that deal with the question: which factors should be considered
in the design and development of CDS to result in an acceptable and effective CDS? are sum-
marized in[19]. These studies focus on developing such systems that lead to wider adoption of
CDS and consequent improvement in quality of health care. According to these studies and those
reviewed in this section, two of the main challenges in design and development of CDS are:
•human-related factors that are related to the way CDS systems are designed, evaluated and
introduced to the users
•technical factors that are mainly related to knowledge representation and reasoning in CDS
systems
•Integration to the EHR systems available in health organizations
2.2. Integration of Clinical Decision Support into Electronic Health Records
It is recommended that CDS be integrated into other information systems in the clinical
domain and it has been demonstrated that an integrated system has better effects on the care pro-
cess [20]. Different clinical applications such as computerized physician order entry (CPOE),
electronic prescribing, e-prescribing (eRX) and personal health records (PHR) are valuable un-
derlying platforms for CDS [16, 1]. Several studies discuss how delivery of decision support
through EHRs can improve the quality of care [4, 3, 21, 22]. Moreover, integration of CDS into
EHR systems has been advocated in several studies as being helpful to the wider adoption of
CDS [2, 1, 5, 4, 23, 16, 24]. Overall, EHR is considered as leverage for CDS [6, 1].
Several studies have observed that manual data entry into CDS acts as a barrier for broad
adoption of CDS. It is recommended that the CDS be provided at the point of care and without
any additional effort to invoke it or utilize it [1, 17]. Manual data entry which is a time consuming
task and a burden for clinicians can be removed by integrating CDS into EHR systems and
utilizing the data which is already in an electronic, computer-readable format. In this case, there
is no need for duplicate data entry and the system can query related information from the EHR
system [2, 1, 23, 25, 6]. Therefore, implementation of CDS is facilitated by EHRs. If there is
no integration, data must be extracted from EHRs to be applied in the CDS. Moreover, if CDS
is not integrated into EHRs, that part of the domain knowledge which is included in EHR is not
applied properly [1].
2.3. Interoperability of EHR systems
EHR systems are being developed by various vendors, so they might be stored in different
formats. This results in systems that are not interoperable, and makes sharing EHRs among dif-
ferent health organizations difficult. To overcome this problem, and to support secure and timely
access to EHRs, national and international EHR standards are developed [26, 27]. openEHR [28]
and health level 7 (HL7) [29] are two of the well-known interoperability standards. A description
of these two standards is given in the following.
3
2.3.1. openEHR
openEHR is an open standard specification. The openEHR specification describes how EHRs
are managed, stored, retrieved and exchanged [30]. Three main concepts defined in openEHR
are (i) the two-level software architecture (ii) archetypes (iii) templates. The two-level architec-
ture for clinical applications deals with separation of knowledge and information levels in order
to overcome the problems caused by the ever-changing nature of clinical knowledge. This is
realized by using openEHR archetypes. Archetypes and templates are used for data validation
and sharing [28]. Beale et al. in [31] define archetype and template as follows:
•Archetype is“a computable expression of a domain content model in the form of structured
constraint statements, based on a reference (information) model. openEHR archetypes
are based on the openEHR reference model. Archetypes are all expressed in the same
formalism. In general, they are defined for wide re-use, however, they can be specialized
to include local particularities. They can accommodate any number of natural languages
and terminologies.”
•Template is“a directly locally usable definition which composes archetypes into a larger
structures often corresponding to a screen form, document, report or message. A template
may add further local constraints on the archetypes it mentions, including removing or
mandating optional sections, and may define default values.”
2.3.2. Health Level 7
HL7 is an EHR standard that focuses on communicating EHRs [10]. According to HL7 web-
site1: “Health Level Seven International (HL7) is a not-for-profit, ANSI-accredited standards
developing organization dedicated to providing a comprehensive framework and related stan-
dards for the exchange, integration, sharing, and retrieval of electronic health information that
supports clinical practice and the management, delivery and evaluation of health services”. In
HL7 version 3 a comprehensive Reference Information Model (RIM) is introduced [10]. HL7
clinical document architecture (CDA) templates are analogous to openEHR archetypes [32].
2.4. Other Standards in the Clinical Domain
There are different approaches to support the interoperability among heterogeneous clinical
systems. Other than EHR interoperability standards that concentrate on standardizing the clinical
information model, the initiative has been taken to standardize other concepts in the clinical do-
main such as clinical guidelines and clinical terminologies to improve shareability and reusability
of them among health institutions.
2.4.1. Communicating The Clinical Terminology
The language is not uniform in the clinical domain and clinicians may use different terms
to refer to the same concepts. Therefore, there is a need to standardize the clinical terminology
to enable communicating it [33]. SNOMED CT (Systematized Nomenclature of Medicine –
Clinical Terms) is and advanced clinical terminology and coding system [33]. SNOMED CT
concepts are usually referred to by an information model such as openEHR and HL7 [34].
1http://www.hl7.org/
4
ICD (International Classification of Diseases) is a coding system that is designed to “promote
international comparability in the collection, processing, classification, and presentation of mor-
tality statistics” [35]. This classification standard is suitable for statistical reporting rather than
clinical documentation as is supported by SNOMED CT. There is a map between SNOMED CT
terms and the equivalent ICD codes [34].
2.4.2. Sharing Clinical Guidelines
Developing clinical guidelines involves a lot of effort. Therefore, there have been initiatives
to enable reusability and shareability of clinical guidelines among various health organizations.
The first step to support reusability and shareability of clinical guidelines is to define a common
format for representing them [36]. One well-known language for this purpose is the one devel-
oped by the InterMed Collaboratory named GLIF (the GuideLine Interchange Format) [36].
3. Search Methodology
Literature review is a research methodology that aims at summarizing the available literature
on a topic and presenting an analysis based on that and providing a full picture on the topic [37].
In this study, literature review is used to analyze state of the art in the intersection between CDS
and EHRs.
The search was conducted in the Sciencedirect2database that includes the major journals in
medical informatics3. The queries and results were as follows:
•searching the combination of phrases “clinical decision support” and “electronic health
record” returned 48 articles where 37 of them were selected for further studies.
•searching the combination of phrases “clinical decision support” and “medical health
record” (excluding the papers that had the phrase “electronic health record”) returned 50
articles where 37 of them were selected for further studies.
Of these 74 studies, only 21 turned out to be relevant to the review based on their contents. In
addition to these 21 studies, 4 more studies that the author had found were included in the review.
The search strategy is depicted in Figure 1. Criteria for including the papers in the review were
positive answers to these questions:
•Is the study discussing development and/or evaluation of a CDS system (i.e. practical
science)? If yes, have the authors considered integration of the CDS into EHRs or a related
application (i.e. integration)?
•Is the study discussing development and/or evaluation of an EHR system (i.e. practical
science)? If yes, have the authors considered integration of a CDS feature into the system
(i.e. integration)?
Since, we were particularly interested in openEHR, further searches were carried out in Sci-
enceDirect and PubMed4specifically on openEHR to find out if any development of an openEHR-
based CDSS is documented in the literature:
2http://sciencedirect.com
3The search was done in mid 2010.
4http://pubmed.org
5
Primary searches
N=98
24 excluded
no CDS integration
not practical science
Abstract relevant
N=74
53 excluded
no CDS integration
not practical science
duplicated
Content relevant
N=25
4 external studies included
Figure 1: Search process.
•In ScienceDirect searching the combination of phrases “clinical decision support” and
openEHRresulted in 1 article that was reviewed before (the study by Greenes [1]).
•In PubMed searching the combination of phrases “clinical decision support” and openEHR
resulted in 2 articles by the author of this paper (these papers are not included in the
review).
4. Results
This section includes the preliminary findings from the literature review. Analysis of the
findings are given in the next section.
The 25 selected articles were reviewed in order to find out whether they consider any of the
clinical standards (i.e. EHR standards, guideline representation standards, and terminology or
vocabulary standards).
As evident from Table1, there are various studies that have applied EHR standards (not in-
cluding openEHR) in developing EHRs with CDS functionalities. HL7 is used in 7 studies, GLIF
in 1, and SNOMED CT/ICT in 2 studies. There are also studies in which local representations
or terminologies were used for representing clinical guidelines or clinical terms [38, 39]. Most
of the CDS services were documented to be integrated into a CPOE system. The summary of
findings is presented in Figure 2.
6
Who Year Integration Standards
EHR Guideline Terminology
Stair [40] 1998 3 7 7 7
Gadd et al. [41] 1998 3 7 7 7
Panzarasa et al. [39] 2002 3 7 7 3
Younget al. [42] 2004 3 7 7 7
Shiffman et al. [43] 2004 3HL7 3 7
Rosenbloom et al. [44] 2004 3 7 7 7
Galanter et al. [45] 2005 3 7 7 7
Haller et al. [46] 2007 3 7 7 7
Stutman et al. [47] 2007 3 7 7 7
Wilson et al. [24] 2007 3 7 7 7
Lobach et al. [48] 2007 - HL7 7 7
Graham et al. [49] 2008 - HL7 7 7
Marcy et al. [50] 2008 3 7 7 7
Wright et al. [51] 2008 3HL7 7SNOMED
CT,ICD
Gerard et al. [52] 2008 3 7 7 7
Field et al. [53] 2008 3 7 7 7
Schnipper et al. [54] 2008 3 7 7 7
Peleg et al. [55] 2009 3 7 GLIF 7
Saleem et al. [56] 2009 3 7 7 7
Field et al. [57] 2009 3 7 7 7
Chen et al. [58] 2010 3 7 3 7
Galanter et al. [59] 2010 3 7 7 SNOMED
CT,ICD
Noormohammad et al.
[38]
2010 3HL7 7 3
Trafton et al. [60] 2010 3 7 7 7
Were et al. [61] 2010 3HL7 7 7
Table 1: The summary of the findings (Please note that 3is used when one item is considered in the study). - specifies
that the item is not mentioned in the study.
4.1. HL7 versus openEHR
While searching the combination of phrases “clinical decision support” and HL7 resulted
in 41 papers5, we did not find any study that reports on implementation of a CDS applying
openEHR6.
EHR (6) 24%
Terminology (4) 16%
Guideline(3) 12%
Figure 2: Various standards reported in the reviewed studies. All of the EHR standards that were applied in studies were
HL7. openEHR was not adopted in any of the studies.
5. Discussion
Theory supports the benefits offered by integrating CDS into EHR, but this concept is still
appreciated more in theory than in practice. Only 25 related studies were discovered in this
database while around 100 studies are documented in the literature that, based on their titles and
abstracts, are about developing a CDSS. Nonetheless, the publication years of these 25 studies are
an indication that in recent years, there has been an increase in consideration of EHR integration
in development of CDS.
5Not all of these studies are included in the review.
6The search was done in mid 2010. However, in a new search in 2011, we found more new studies related to
openEHR. These studies are discussed more in the discussion section.
7
Moreover, it is observed that taking standards into consideration in any clinical application
(and generally any information system) is very important [11]. In case of CDSSs, since such
systems operate by utilizing both patient/organizational-specific data and clinical knowledge, it
is important to consider the standards that support each of these areas [11]. This however is
observed to still be in need of further improvements. Of these 25 studies, only 6 had considered
EHR standardization, and 3 had considered terminology standards which are both surprisingly
small numbers.
Finally, one can conclude that based on the literature, HL7 has a higher level of adoption than
openEHR and that applying openEHR in development of clinical applications specially CDS is
yet rare. This brings the question that regardless of the advances in theory why is openEHR
suffering from a low adoption rate in practice? This issue is discussed more in the following
section.
5.1. Low Adoption of openEHR
Below is a list of problems that we or others have faced using openEHR7. These issues are
considered as barriers to higher adoption of openEHR8:
5.1.1. Being huge and complex*
openEHR is naturally complex, and this complexity is not unexpected since openEHR is
considered to be a solution for a complicated problem (i.e. interoperable future-proof EHR) in
a complex domain (i.e. the clinical domain). For instance the powerful archetype model allows
expressing complex clinical concepts, therefore, an inexperienced archetype developer should
expect to spend some time on learning the openEHR concepts. Additionally, getting a grip on
the current specifications (more than 1000 pages), UML diagrams and code documentation is
challenging. At the same time, it is notable that this complexity is intensified by some other
aspects such as improper educational support and limited internationalization.
5.1.2. Shortcoming in educational aspects*
Understanding a concept is the first step to be able to adopt it and this is even more valid for
such complex concepts like those in openEHR. Unfortunately, no formal tutorial document exists
for openEHR, formal training sessions are rare and even worse, not so many openEHR trainers
exist around the world. Easy to understand tutorials are needed to help novice developers get a
grip on openEHR.
5.1.3. Low government and industry penetration
Many of those who are interested in openEHR, in spending time on learning it or adopting
it, are from the academic world (the main of which is University College London9). So far, there
are very few companies that are adopting openEHR and to our knowledge these companies are
7In November 2010, there was a discussion on openEHR mailing list with the same topic. This shows that even
people in the openEHR community have noticed that the adoption rate is low and some actions should be taken in order
to improve it. Especially, it is noticeable that the amount of attention to openEHR is much less than HL7 in various
domains i.e. government, academy and industry.
8Some of the issues presented here are the result of investigating the discussions in the openEHR mailing list, even
though some others had been experienced in this study. Those issues are marked with an asterisk.
9http://ucl.ac.uk
8
considered to be a part of the core openEHR community. The main companies are Oceanin-
formatics10, Cambio11 , and Zilics12 . But what about “ordinary audience”? On the other hand,
low support from the governmental agencies lead to low industry penetration. Considering the
complication and the cost imposed by the openEHR approach, and also limited documentations
and guidelines, applying openEHR is not still cost-efficient and yet commercial companies show
a lot of hesitation to accept risks imposed by adopting this immature standard.
5.1.4. Shortcoming in internationalization aspects
In order to reach an international-wide adoption, it is suggested that establishing regional
communities would be helpful; nevertheless, there are other concerns in this regard. openEHR
community should consider issues such as supporting and providing guidelines for regional com-
munities all around the world. It is also beneficial to publish openEHR specifications in various
languages in order to speed up the process of learning for various people. Regional events such
as educational sessions, gatherings and so on are also valuable to influence collaboration. As
an example, in Sweden, there are around 4 groups of people13 doing research on or adopting
openEHR, but collaboration among them is at a very low level.
5.1.5. Low empirical focus*
openEHR should not just be about complex theoretical specifications and reference models,
but also about implementation and practice. Semantic interoperability, two-level modeling and
involving clinicians are interesting concepts, but so far these have been far from the practice.
Currently, there are just a few empirical efforts on openEHR. Most of the focus of openEHR
community has been on representation of domain concepts and theoretical aspects of the ap-
proach. Still, there is a huge need for supporting developers to make openEHR more practical.
5.1.6. Limited tools and implementations*
As mentioned above, developers needed to be supported in order to improve adoption of
openEHR. One way of delivering this support is by providing frameworks and application pro-
gramming interfaces (API). At this time, the openEHR reference model implementation is still
immature and lacks important parts like templates, persistence, and services.
5.2. Recent Advances in The openEHR-based CDS
When it comes to CDS, there are a few studies that deal with how openEHR offers opportu-
nities for CDS. Most of these efforts however, seem to be more focused on integrating clinical
guidelines into openEHR archetypes or utilizing archetypes for representing clinical guidelines
[62, 25, 63] or to integrate reasoning and clinical archetypes (enhance archetypes by including
knowledge representation capabilities to them) [64]. To our knowledge there is almost no study
that has been focused on benefiting from the well-structured openEHR-based patient data for
adopting data intensive reasoning methods in CDSSs or methods that rely on previous cases to
carry out the reasoning process.
10http://www.oceaninformatics.com
11http://cambio.se
12http://www.zilics.com.br
13Chalmers university of technology, Link¨
oping university, Cambio company and The Swedish NHS.
9
5.3. Why Are Clinical Standards Important for CDS?
According to the discussion in Section 2, enough motivation exists to integrate CDS into
EHRs. There is still a question whether integration of CDS into EHRs can be done without taking
EHR related standards into account. If EHR standards are not considered in CDS development,
all clinical data should be translated to a format understandable for the CDS system. This is not
an efficient way for CDS and the EHR system to communicate. Moreover, there is an increasing
interest in the medical informatics community to share clinical knowledge. This can also be
supported if CDS is developed based on EHR standards. For instance, by enriching standard
compatible EHRs with the reasoning knowledge, EHR sharing will also result in sharing and
reusing the embedded knowledge. In cases where general domain knowledge including clinical
guidelines are integrated into EHRs, the reusability and sharing of knowledge can be achieved as
well.
6. Conclusion
Researchers in the area of CDS and also EHR have motivated that by integrating CDS into
EHRs, the improvement in the quality of health care would be higher than when the systems
operate separately. The integration will be more efficient if the standards related to EHRs are
considered in developing CDS. The possibility to share the domain knowledge, especially the
reasoning knowledge, in decision making is another motivation for taking standards into account
in developing CDS.
Nevertheless, a review of the related literature indicates that not all of CDS developers take
integration into account, also there are very few of them who consider standards in developing
CDS. Discovering the reasons for this however needs further investigation and has not been in
the scope of this review.
Acknowledgment
I would like to give thanks to Olof Torgersson who provided helpful suggestions to improve
the paper.
References
[1] R. Greenes, Clinical decision support: the road ahead, Academic Press, 2007.
[2] T. Wendt, P. Knaup-Gregori, A, Decision Support in Medicine: A Survey of Problems of User Acceptance, Stud
Health Technol Inform 77 (2000) 852–856.
[3] B. Chaudhry, J. Wang, S. Wu, M. Maglione, Systematic review: impact of health information technology on quality,
efficiency, and costs of medical care, Annals of internal Med 144 (10) (2006) 742–752.
[4] A. Garg, N. Adhikari, H. McDonald, M. Rosas, Effects of computerized clinical decision support systems on
practitioner performance and patient outcomes: a systematic review, JAMA 293 (10) (2005) 1223–1238.
[5] D. F. Sittig, A. Wright, J. a. Osheroff, B. Middleton, J. M. Teich, J. S. Ash, E. Campbell, D. W. Bates,
Grand challenges in clinical decision support., Journal of biomedical informatics 41 (2) (2008) 387–92.
doi:10.1016/j.jbi.2007.09.003.
[6] J. Osheroff, J. Teich, B. Middleton, E. Steen, A. Wright, D. Detmer, A roadmap for national action
on clinical decision support, Journal of the American medical informatics association 14 (2) (2007) 141.
doi:10.1197/jamia.M2334.Introduction.
[7] Patient Safety: Achieving a New Standard of Care (2003).
[8] Crossing the Quality Chasm: A New Health System for the 21st Century, Website, http://journals.lww.
com/qmhcjournal/Abstract/2002/10040/Crossing\_the\_Quality\_Chasm\_\_A\_New\
_Health\_System.10.aspx (2001).
10
[9] I. Iakovidis, Towards personal health record: current situation, obstacles and trends in implementation of electronic
healthcare record in Europe., International journal of medical informatics 52 (1-3) (1998) 105–15.
[10] B. Blobel, Advanced and secure architectural EHR approaches., International journal of medical informatics 75 (3-
4) (2006) 185–90. doi:10.1016/j.ijmedinf.2005.07.017.
[11] J. Osheroff, E. Pifer, J. Teich, D. Sittig, R. Jenders, Improving outcomes with clinical decision support: An imple-
menter’s guide, HIMSS, 2005.
[12] R. Greenes, M. Sordo, D. Zaccagnini, M. Meyer, GJ, Design of a standards-based external rules engine for decision
support in a variety of application contexts: report of a feasibility study at Partners HealthCare System, Medinfo.
[13] L. Zhou, C. S. Soran, C. a. Jenter, L. a. Volk, E. J. Orav, D. W. Bates, S. R. Simon, The relationship between
electronic health record use and quality of care over time., Journal of the American Medical Informatics Association
: JAMIA 16 (4) (2009) 457–64. doi:10.1197/jamia.M3128.
[14] J. Anderson, Increasing the acceptance of clinical Information, MD computing: computers in medical practice
16 (1) (1999) 62.
[15] K. Kawamoto, D. Lobach, Proposal for fulfilling strategic objectives of the US roadmap for national action on
decision support through a service-oriented architecture leveraging HL7 services, Journal of the American medical
(2007) 146–155doi:10.1197/jamia.M2298.Introduction.
[16] I. Cho, J. Kim, J. H. Kim, H. Y. Kim, Y. Kim, Design and implementation of a standards-based interoperable clinical
decision support architecture in the context of the Korean EHR., International journal of medical informatics 9
(2010) 611–622. doi:10.1016/j.ijmedinf.2010.06.002.
[17] K. Kawamoto, C. A. Houlihan, E. A. Balas, D. F. Lobach, Improving clinical practice using clinical decision
support systems: a systematic review of trials to identify features critical to success., BMJ (Clinical research ed.)
330 (7494) (2005) 765. doi:10.1136/bmj.38398.500764.8F.
[18] M. Trivedi, J. Kern, A. Marcee, B. Grannemann, B. Kleiber, T. Bettinger, K. Altshuler, A. McClelland, Develop-
ment and Implementation of Computerized Clinical Guidelines : Barriers and Solutions, Methods of information
in medicine 41 (5) (2002) 435–442.
[19] H. Kashfi, Towards Interaction Design in Clinical Decision Sopport Development, manuscript submitted to Inter-
national Journal of Medical Informatics, Elsevier (2011).
[20] R. A. K. Horasani, M. I. T. Anasijevic, B. L. M. Iddleton, M. S. C, Ten Commandments for Effective Clinical
Decision Support: Making the Practice of Evidence-based Medicine a Reality, Journal of the American Medical
Informatics Association 10 (2003) 523–530. doi:10.1197/jamia.M1370.Although.
[21] D. Hunt, R. Haynes, S. Hanna, K. Smith, Effects of computer-based clinical decision support sys-
tems on physician performance and patient outcomes: a systematic review, Jama 280 (15) (1998) 1339.
doi:10.1001/jama.280.15.1339.
[22] M. Johnston, K. Langton, R. Haynes, Effects of computer-based clinical decision support systems on clinician
performance and patient outcome. A critical appraisal of reserach, Ann Intern Med 120 (2) (1994) 135–142.
[23] E. Berner, Clinical Decision Support Systems: Theory and Practice (Health Informatics), Springer, New York, NY
10013, USA, 2007.
[24] A. Wilson, A. Duszynski, D. Turnbull, J, Investigating patients’ and general practitioners’ views of computerised
decision support software for the assessment and management of cardiovascular risk, Informatics in Primary Care
15 (2007) 33–44.
[25] R. Chen, P. Georgii-Hemming, H. Ahlfeldt, Representing a chemotherapy guideline using openEHR and rules.,
Studies in health technology and informatics 150 (2009) 653–7. doi:10.3233/978-1-60750-044-5-653.
[26] V. Stroetmann, D. Kalra, P. Lewalle, J. Rodrigues, KA, Semantic Interoperability for Better Health and Safer Health
Care, Deployment and Research (January). doi:10.2759/38514.
[27] P. Schloeffel, T. Beale, G. Hayworth, S. Heard, H. Leslie, The relationship between CEN 13606, HL7, and
openEHR, in: In Health Informatics Conference (2006), Vol. 7, Health Informatics Society of Australia, 2006,
p. 24.
[28] T. Beale, S. Heard, openehr architecture overview, Website, http://www.openehr.org/releases/1.0.
2/architecture/overview.pdf (2008).
[29] HL7, Website, http://hl7.org (2010).
[30] openEHR, Website, http://en.wikipedia.org/wiki/openehr (2010).
[31] T. Beale, S. Heard, Archetype Definitions and Principles, Website, http://www.openehr.org/releases/
1.0.2/architecture/am/archetype_principles.pdf (2007).
[32] M. Eichelberg, T. Aden, J. Riesmeier, A. Dogac, G. B. Laleci, A survey and analysis of Electronic Healthcare
Record standards, ACM Computing Surveys 37 (4) (2005) 277–315. doi:10.1145/1118890.1118891.
[33] K. Donnelly, SNOMED-CT: The advanced terminology and coding system for eHealth., Studies in health technol-
ogy and informatics 121 (2006) 279–90.
[34] SNOMED CT, Website, http://en.wikipedia.org/wiki/SNOMED_CT (2010).
[35] ICD, Website, http://www.cdc.gov/nchs/icd.htm (2010).
11
[36] L. Ohno-Machado, J. H. Gennari, S. N. Murphy, N. L. Jain, S. W. Tu, D. E. Oliver, E. Pattison-Gordon, R. a.
Greenes, E. H. Shortliffe, G. O. Barnett, The guideline interchange format: a model for representing guidelines.,
Journal of the American Medical Informatics Association : JAMIA 5 (4) (1998) 357–72.
[37] H. Aveyard, Doing a literature review in health and social care: a practical guide, Open University Press, 2007.
[38] S. F. Noormohammad, B. W. Mamlin, P. G. Biondich, B. McKown, S. N. Kimaiyo, M. C. Were, Changing course
to make clinical decision support work in an HIV clinic in Kenya., International journal of medical informatics
79 (3) (2010) 204–10. doi:10.1016/j.ijmedinf.2010.01.002.
[39] S. Panzarasa, S. Maddˇ
c, S. Quaglini, Evidence-based careflow management systems: the case of post-stroke reha-
bilitation, Journal of Biomedical 35 (2) (2002) 123–139. doi:10.1016/S1532-0464(02)00505-1.
[40] T. Stair, REDUCTION OF REDUNDANT LABORATORY ORDERS BY ACCESS TO COMPUTERIZED PA-
TIENT RECORDS, The Journal of emergency medicine 16 (6) (1998) 895– 897.
[41] C. Gadd, P. Baskaran, D. Lobach, Identification of design features to enhance utilization and acceptance of systems
for Internet-based decision support at the point of care., in: Proceedings of the AMIA, 1998, pp. 91–5.
[42] A. S. Young, J. Mintz, A. N. Cohen, M. J. Chinman, A network-based system to improve care for schizophrenia: the
Medical Informatics Network Tool (MINT)., Journal of the American Medical Informatics Association : JAMIA
11 (5) (2004) 358–67. doi:10.1197/jamia.M1492.
[43] R. N. Shiffman, G. Michel, A. Essaihi, E. Thornquist, Bridging the guideline implementation gap: a systematic,
document-centered approach to guideline implementation., Journal of the American Medical Informatics Associa-
tion : JAMIA 11 (5) (2004) 418–26. doi:10.1197/jamia.M1444.
[44] S. T. Rosenbloom, D. Talbert, D. Aronsky, Clinicians’ perceptions of clinical decision support integrated
into computerized provider order entry., International journal of medical informatics 73 (5) (2004) 433–41.
doi:10.1016/j.ijmedinf.2004.04.001.
[45] W. L. Galanter, R. J. Didomenico, A. Polikaitis, A trial of automated decision support alerts for contraindicated
medications using computerized physician order entry., Journal of the American Medical Informatics Association
: JAMIA 12 (3) (2005) 269–74. doi:10.1197/jamia.M1727.
[46] G. Haller, P. S. Myles, J. Stoelwinder, M. Langley, H. Anderson, J. McNeil, Integrating incident reporting into an
electronic patient record system., Journal of the American Medical Informatics Association : JAMIA 14 (2) (2007)
175–81. doi:10.1197/jamia.M2196.
[47] H. Stutman, R. Fineman, K. Meyer, Optimizing the acceptance of medication-based alerts by physicians during
CPOE implementation in a community hospital environment, in: AMIA Annual Symposium, 2007, pp. 701–705.
[48] D. F. Lobach, K. Kawamoto, K. J. Anstrom, M. L. Russell, P. Woods, D. Smith, Development, deployment and
usability of a point-of-care decision support system for chronic disease management using the recently-approved
HL7 decision support service standard., Studies in health technology and informatics 129 (Pt 2) (2007) 861–5.
[49] T. Graham, A. Kushniruk, M. Bullard, B. Holroyd, D. Meurer, B. Rowe, How usability of a web-based clinical
decision support system has the potential to contribute to adverse medical events, in: AMIA Annual Symposium
Proceedings, Vol. 2008, American Medical Informatics Association, 2008, p. 257.
[50] T. W. Marcy, B. Kaplan, S. W. Connolly, G. Michel, R. N. Shiffman, B. S. Flynn, Developing a decision support
system for tobacco use counselling using primary care physicians., Informatics in primary care 16 (2) (2008) 101–9.
[51] A. Wright, D. F. Sittig, SANDS: a service-oriented architecture for clinical decision support in a National Health
Information Network., Journal of biomedical informatics 41 (6) (2008) 962–81. doi:10.1016/j.jbi.2008.03.001.
[52] M. N. Gerard, W. E. Trick, K. Das, M. Charles-Damte, G. A. Murphy, I. M. Benson, Use of clinical decision
support to increase influenza vaccination: multi-year evolution of the system., Journal of the American Medical
Informatics Association : JAMIA 15 (6) (2008) 776–9. doi:10.1197/jamia.M2698.
[53] T. S. Field, P. Rochon, M. Lee, L. Gavendo, S. Subramanian, S. Hoover, J. Baril, J. Gurwitz, Costs associated with
developing and implementing a computerized clinical decision support system for medication dosing for patients
with renal insufficiency in the long-term care setting., Journal of the American Medical Informatics Association :
JAMIA 15 (4) (2008) 466–72. doi:10.1197/jamia.M2589.
[54] J. L. Schnipper, J. A. Linder, M. B. Palchuk, J. S. Einbinder, Q. Li, A. Postilnik, B. Middleton, ”Smart Forms” in an
Electronic Medical Record: documentation-based clinical decision support to improve disease management., Jour-
nal of the American Medical Informatics Association : JAMIA 15 (4) (2008) 513–23. doi:10.1197/jamia.M2501.
[55] M. Peleg, A. Shachak, D. Wang, E. Karnieli, Using multi-perspective methodologies to study users’ interactions
with the prototype front end of a guideline-based decision support system for diabetic foot care., International
journal of medical informatics 78 (7) (2009) 482–93. doi:10.1016/j.ijmedinf.2009.02.008.
[56] J. Saleem, L. Militello, N. Arbuckle, M. Flanagan, Provider Perceptions of Colorectal Cancer Screening Clinical
Decision Support at Three Benchmark Institutions, in: AIMIA Symposium Proceedings, 2009, pp. 558–562.
[57] T. S. Field, P. Rochon, M. Lee, L. Gavendo, J. L. Baril, J. H. Gurwitz, Computerized clinical decision support
during medication ordering for long-term care residents with renal insufficiency., Journal of the American Medical
Informatics Association : JAMIA 16 (4) (2009) 480–5. doi:10.1197/jamia.M2981.
[58] C. C. Chen, K. Chen, C.-y. Hsu, Y.-c. J. Li, Developing guideline-based decision support systems using prot´
eg´
e
12
and jess., Computer methods and programs in biomedicine in print (2010) 1–7. doi:10.1016/j.cmpb.2010.05.010.
[59] W. L. Galanter, D. B. Hier, C. Jao, D. Sarne, Computerized physician order entry of medications and clinical
decision support can improve problem list documentation compliance., International journal of medical informatics
79 (5) (2010) 332–8. doi:10.1016/j.ijmedinf.2008.05.005.
[60] J. Trafton, S. Martins, M. Michel, E. Lewis, Evaluation of the Acceptability and Usability of a Decision Support
System to Encourage Safe and Effective Use of Opioid Therapy for Chronic , Noncancer Pain by Primary Care
Providers, Pain Medicine 11 (2010) 575–585.
[61] M. C. Were, C. Shen, M. Bwana, N. Emenyonu, N. Musinguzi, F. Nkuyahaga, A. Kembabazi, W. M. Tierney, Cre-
ation and evaluation of EMR-based paper clinical summaries to support HIV-care in Uganda, Africa., International
journal of medical informatics 79 (2) (2010) 90–6. doi:10.1016/j.ijmedinf.2009.11.006.
[62] M. Marcos, B. n. Mart´
ınez-Salvador, Towards the interoperability of computerized guidelines and electronic health
records: an experiment with openEHR archetypes and a chronic heart failure guideline, in: Proceedings of the ECAI
2010 conference on Knowledge representation for health-care, KR4HC’10, Springer-Verlag, Berlin, Heidelberg,
2011, pp. 101–113.
[63] L. Xiao, G. Cousins, L. Hederman, T. Fahey, B. Dimitrov, The design of an EHR for clinical decision support, no.
Bmei, IEEE, 2010. doi:10.1109/BMEI.2010.5639694.
[64] L. Lezcano, M.-A. Sicilia, C. Rodr´
ıguez-Solano, Integrating reasoning and clinical archetypes using OWL ontolo-
gies and SWRL rules., Journal of biomedical informatics 44 (2) (2011) 343–53. doi:10.1016/j.jbi.2010.11.005.
13
Paper III
Applying a User-centered Design Methodology
in a Clinical Context
Hajar Kashfi
The 13th International Congress on Medical Informatics (MedInfo2010),
Studies in health technology and informatics, 2010 Jan ;160(Pt 2):927-31.
(reprinted with an updated layout)
91
Applying a User Centered Design Methodology in a
Clinical Context
Hajar Kashfi
Division of Interaction Design, Department of Computer Science and Engineering,
Chalmers University of Technology, Gothenburg, Sweden.
Abstract
A clinical decision support system (CDSS) is an interactive application that is used
to facilitate the process of decision-making in a clinical context. Developing a usa-
ble CDSS is a challenging process; mostly because of the complex nature of domain
knowledge and the context of use of those systems. This paper describes how a user
centered design (UCD) approach can be used in a clinical context for developing a
CDSS. In our effort, a design-based research methodology has been used. The out-
comes of this work are as follow; a customized UCD approach is suggested that
combines UCD and openEHR. Moreover, the GUI developed in the design phase
and the result of the GUI evaluation is briefly presented.
Keywords:
Clinical Decision Support System, User Centered Design, Usability, UCD, Proto-
type, Design and Development process, Iterative design, openEHR.
Introduction
Errors that occur in a clinical process are mostly due to cognitive limitations of hu-
mans, the potential to forget knowledge in the health care flow. Information systems
have the ability to decrease such errors by supporting clinicians in this process e.g.
by reminding them of important factors to be considered for the current case or to
alert them of adverse drug-drug interactions [1]. A Decision Support System (DSS)
is an interactive application that is supposed to facilitate the process of decision
making for decision makers. This support is done by mapping or compiling existing
data to useful information that can be used as a clue for making the best decision [2].
Clinical Decision Support Systems are those DSS:s that are used in the clinical do-
2
main. CDSS:s are intended to help clinicians in the process of decision making. Ser-
vices supported by CDSS:s include diagnosis, alerting, reminding, treatment sugges-
tions, and patient education. Based on a thorough literature review done on around
140 papers about CDSS, it is clear that CDSS:s have the potential to improve care
[3].
Usability of CDSS
ISO 9421 [4] defines usability as the “Extent to which a product can be used by
specified users to achieve specified goals with effectiveness, efficiency and satisfac-
tion in a specified context of use.” Poor usability is the one of the reasons for why
CDSS:s have not yet gained a broad acceptance. While there have been many efforts
in developing CDSS:s, very few of those systems have been accepted in real clinical
environments. Studies show that user interfaces have an impact on acceptability of
CDSS:s in a clinical context. The success of a CDSS has a direct relation to the way
its graphic user interface (GUI) has been designed [5].
CDSS:s are meant to reduce clinical errors, nevertheless, because of improper de-
sign of those systems, other kinds of errors may occur by using them [1,6]. Studies
reveal that clinical information systems with low usability not only do not improve
patient care and reduce clinical errors, but also may have the opposite effect [7-8].
Involving Clinicians in the Design Process of CDSS:s
Not just in the clinical domain, but in every other domain experiences show that by
involving users in the design and development process of a system, the system will
be more usable for the intended users [9-12].The design approach which emphasizes
on involving users in the design is called User Centered Design process(UCD) [9-
10]. Accordingly, one can not develop a CDSS which addresses clinicians’ needs in
a clinical context without a design process in which end users, clinicians, are in-
volved actively [13]. To make a CDSS a usable product, we should consider not
only user needs that reveal functional requirements of the system, but also non-
functional or usability requirements as well as characteristics of the clinical envi-
ronment in which the system will finally be applied.
In this paper, we present issues related to user-centered design of a CDSS for Dry
Mouth, an oral disease. The main reason for selecting Dry Mouth is that our end
users expressed a need for a CDSS for this disease.
3
Methods
The research method we applied in our work is a design-based research method [14].
For this purpose, our collaborators in Sahlgrenska Academy1 suggested the design
and development of a CDSS for an oral disease named Dry Mouth. “Dry mouth or
Xerostomia is the abnormal reduction of saliva and can be a symptom of certain
diseases or be an adverse effect of certain medications”[15]. Treatment of Xerosto-
mia is related to finding its cause(s). There are five main categories for Xerostomia:
Drug-induced, Disease-induced, Radiation-induced, Chemotherapy-Induced, and
cGVHD-induced [15].
The reason for suggesting Dry Mouth was that the dentists and dental hygienists are
commonly the first clinicians to face the complaints by patients regarding this dis-
ease; hence, they should be aware of it and its problems to prevent the deleterious
consequences of this disorder. However, according to our expert panel, finding
cause(s) of Dry Mouth is a challenge for dentist and dental hygienists, and needs to
be supported by a clinical application. The decision support process we aim for in-
cludes these two main steps (1) finding the cause(s) of disease based on the patient’s
medical records (2) suggesting related materials and treatment options, based on
results from the first step. Since this system is intended to be used integrated with an
existing Clinical Data Entry application [16], data entry forms are not part of the
Graphical User Interface (GUI), however we have to provide users with options to
edit existing data. Finally, users need to be able to enter their own comments; in-
cluding diagnosis or treatments to the system.
The Design Process
The approach we use in this design process is UCD. UCD focuses on the end users,
their needs and the context in which the system will be used. The main goal in this
method is user satisfaction. UCD has an iterative nature. It means that during the
design and development process, at several points, prototypes are delivered to users
for evaluation and improvement.
As depicted in Figure 1, the idea of UCD is a circular design process including anal-
ysis, design, prototyping and getting user feedback. End users are in the heart of this
design process and should be involved in all steps. Users are asked about what they
expect the application to do for them and what priorities they have in doing their
tasks using the intended application. Users have the chance to specify their needs as
detailed as possible e.g. which colors do they prefer or what are their time limits
running a specific task using the application.
On the other hand, informaticians can communicate with users to extract vital in-
formation about their current situation and their future needs e.g. what users like
1 http://www.sahlgrenska.gu.se/english
4
about the way they are currently doing their tasks or what would they like to be
changed [11]. Finally, task analysis and evaluation [9-10] should be done based on
the gathered information.
The Importance of Involving Clinicians in the design
Domain knowledge plays the main role in complexity of clinical applications. Clini-
cal tasks may not be complex by themselves but what makes the clinical application
development so complicated is that most of the clinical processes are unstructured.
They are done in clinicians’ mind and based on their expertise. Moreover, clinical
knowledge is ever-changing.
Hundreds of data items are involved in a clinical decision making process. After all,
concepts in clinical domain are not easy to understand for informaticians and they
face difficulties communicating with clinicians or studying literature to get enough
domain knowledge to be able to model it and to develop an application.
Extracting domain knowledge in the clinical domain has always been a bottleneck in
the development process of such systems and a challenge for informaticians. There-
fore, in the clinical domain, we need to involve clinicians in designing the Infor-
mation model or more precisely domain concept models to be used for information
modeling. One of the recent approaches that focuses on involving users in domain
concept modeling is openEHR [17].
Figure 1- User Centered Design Process2
The openEHR Approach
openEHR is an open standard specification that emphasizes on the role of clinicians
in organizing domain knowledge in form of different clinical concepts such as ob-
servation, evaluation, instruction and action [17].
2 Copyright 2009, Kevin Bury Design
5
In the openEHR approach, clinicians are in charge of defining the specification of
clinical knowledge to be used in information modeling. This approach suggests a
two level architecture for clinical applications to separate knowledge and infor-
mation levels in order to overcome the problems caused by the ever-changing nature
of clinical knowledge.
While the main emphasis of openEHR is on semantic interoperability of medical
records, we found the approach highly compatible with UCD. Therefore, we applied
openEHR to facilitate involving clinicians in the design and to ease domain concept
modeling and communicating with our end users.
From an UCD point of view, openEHR is very helpful. This approach recommends
the utilization of expert knowledge not only just by consulting clinicians but also by
letting them design concept models based on what they have in mind. By applying
openEHR, we can communicate better with our end users since clinical concepts
recommended by openEHR are understandable for clinicians.
UCD Principles Applied in The Project
There are number of principles that are recommended in UCD [18]: Multidiscipli-
nary design team, Understanding users and context, Active user participation, Early
prototyping, Continuous evaluation, and Holistic design. Besides Holistic design,
we have been concerned about the other principles, as explained below.
Project Team
Our development team is a multidisciplinary team consisting of an interaction de-
sign expert, two computer scientists with different backgrounds (AI, Software Engi-
neering), a domain expert (specialist in dentistry), a programmer, and a nurse. How-
ever, more end users and experts are involved in different steps in various time peri-
ods.
Intended Users, Tasks and Context of Use
One of the main principles in UCD is to define users and the context of use [10].
Users: In this project, direct users are dentists who work in an oral medicine clinic.
We used narrative explanation of some typical end users, personas, [7] to find more
about our end users’ characteristics. Since the output of the CDSS will be a treat-
ment decision, patients are our indirect users. Nonetheless, patients will not use the
system directly.
Tasks: Based on literature review and interviews with end users and domain experts,
we defined the tasks listed below as the main tasks that dentists carry out with re-
gard to Dry Mouth: (1) Information Overview (2) Information manipulation, (3)
6
Requesting related actions like laboratory tests, (4) Diagnosis, (5) Referring to
guidelines and other clinical evidence, (6) Recording the results.
Context of use: Dentists will use the application while they are visiting a patient in
the clinic. They will use it in presence of the patient, at the same time they are com-
municating with the patient and in a setting with a limited amount of time.
Users’ priorities/Usability goals: The goal is to develop a system, which fits to the
dentists’ workflow as much as possible; experiences show that clinicians should not
need to change their clinical workflow while using a CDSS [18]. It is also important
to consider that not all clinicians are experienced in using information systems. On
the other hand, because of their occupations, they do not manage to spend much
time on learning a new application. Based on this information, we set up our usabil-
ity goals such as: Effectiveness, Efficiency, Safety, Learnability, etc.
Iterative GUI Design and Evaluation
During the design, we have been using both low fidelity and high fidelity prototypes.
From those, we can name sketches designed and improved during brain storming
sessions for collecting functional requirements and usability requirements together
with our expert panel. In this step, conceptual design of the application was done.
These sketches were later translated to some power point prototypes. Afterwards,
low fidelity prototyping tools were used to visualize the design solutions. Finally, a
Java based GUI has been developed to make the final usability tests more realistic
and reliable.
7
Figure 2- GUI Prototype
Iterative Domain Concept Model Design
The domain concept modeling started with brain storming sessions in which our
expert panel (experts in Dry Mouth) were asked to think about Dry Mouth and its
related concepts based on this question: What do you want to know about a patient
who visits you because he/she suffers from Dry Mouth?; and to put as much infor-
mation as possible on a paper. Later, our expert panel has been asked to prepare a
questionnaire based on this question: What do you ask from a patient who visits you
because he/she suffers from Dry Mouth?
Questions on the questionnaire were then categorized based on openEHR concepts;
in other words, their logical relation e.g. is the question related to patient history or
is it a lab result? In the next step, simple diagrams were created based on the ques-
tionnaire. For this purpose, a mind-map application3 was used to make it possible for
our expert panel to simply understand and edit the created diagrams.
3 http://www.xmind.net/
8
GUI and Domain Concept Model Evaluation
For the GUI evaluation, we used the evaluation methods applicable in early stages of
the project. Two main methods we have been utilizing so far are Heuristic Evalua-
tion and Usability Tests [9]. Based on the results from the evaluations we improved
the GUI in several stages. One of the resulting GUI screens is showed in Figure 2.
Iterative design of the domain concept model includes evaluations of the current
model based on the literature and experts’ opinions, and story-based assessment.
Information modeling diagrams were improved several times based on the experts’
opinions. Several experts were involved in this process to minimize the subjectivity
of the design and to be as broad as possible in collecting knowledge. A sample mind
map is depicted in Figure 3.
Figure 3- Information modeling
9
Discussion and Results
UCD emphasizes that users’ needs should be reflected in the GUI design and that
the GUI design should influence the design of the rest of the system [12]. On the
other hand, openEHR emphasizes the domain concept modeling as the starting
point. But how much does the domain model reflect the end users’ needs? By using
the openEHR approach without considering other aspects like usability issues, we
may end up developing highly adaptable systems with comprehensive information
models, which are not usable.
In this project, we tried to benefit from the strengths of the two of approaches and to
introduce an adaptation of UCD in a clinical concept keeping an eye on the
openEHR approach.
The Customized UCD Approach for openEHR Based CDSS Development
As references suggest “The actual contents of the UCD process, the methods used,
the order of activities, etc, must be customized and adapted to the particular organi-
zation and project based on their particular needs” [19]. So it was not a surprise to
see that we need to apply a customized version of UCD in this project.
As shown in Figure 4, the main idea of UCD is used in the process but in three dif-
ferent cycles. One is a general cycle to develop the whole application. This main
cycle contains a cycle to develop the Domain Concept Model; and a cycle to devel-
op the GUI. So the process includes two main steps in parallel (1) Iterative devel-
opment of the domain concept model (2) Iterative development of the GUI. For the
first step, several specialists in dentistry (expert panel) and for the second step, both
domain experts and general dentists (end user panel) were involved.
GUI vs. Domain Concept Model
During the iterative design process we noticed that the impact of the domain concept
model on the GUI is inevitable. Decision about the components to be shown on the
GUI is directly related to the output from the domain concept modeling. Any chang-
es in the domain concept model should be checked from the GUI point of view.
Therefore, as depicted in Figure 3, in each iteration, there should be an input from
the left hand side process (domain concept model) to the right hand one (GUI). In
other words, after each domain concept modeling iteration, the necessity of a new
iteration for GUI design should be checked.
Characteristics of the Customized Approach and prblems
The recommended approach has several characteristics:
• This approach considers active involvement of the end users and domain ex-
perts in designing and evaluating the domain concept model and the GUI
10
• In the suggested approach new GUI and Domain Concept modeling iterations
will be performed until the end users are satisfied with the results.
• In this approach, the effect of the domain concept model on the GUI has
been considered as explained before.
• The approach helps overcoming the knowledge extraction bottleneck by ap-
plying clinical concepts suggested by openEHR for communicating with cli-
nicians and providing an opportunity for them to model domain knowledge
based on their expertise.
• This approach inherits the idea of the knowledge and the information level
separation suggested by openEHR in order to make developed applications
highly adaptable.
• Finally, the approach is applicable for developing not only openEHR-based
applications but also all kinds of clinical applications e.g. OWL based ones.
Important issues to be considered while applying the approach are (I) the
parallel iterative UCD of the domain concept model and the GUI, and (II) the
effect of the domain concept models on the GUI.
Figure 4- Customized User Centered Design Process
We also faced some problems during the design phase. In design and implementa-
tion of CDSS:s a big challenge is choosing a knowledge representation and reason-
ing. Our experience showed that, selecting the representation and reasoning method
have to be done in parallel with the information modeling and the GUI design, oth-
11
erwise, the changes forced by this selection causes modifications in the GUI design
which is more cost effective to be known in the early stages of the GUI design. Sec-
ondary, the classical bottleneck of knowledge acquisition in clinical domain still
exists. While applying the suggested methodology decreases difficulties in mutual
understanding of clinicians and designers, it cannot eliminate the bottleneck problem
totally, especially for the cases that reasoning should be done by applying
knowledge intensive methods.
References
[1] Graham Ta, Kushniruk AW, Bullard MJ, Holroyd BR, Meurer DP, Rowe BH.
How usability of a web-based clinical decision support system has the potential
to contribute to adverse medical events. AMIA Symposium, 2008 ;257-61.
[2] Vikram K, Karjodkar FR. Decision support systems in dental decision making:
an introduction. The journal of evidence-based dental practice, 2009 ;9(2):73-6.
[3] Kaplan B. Evaluating informatics applications-clinical decision support systems
literature review. International journal of medical informatics, 2001 ;64(1):15-
37.
[4] Organization IS. ISO 9421-11. Standard on Display Screen (VDU) Regulations,
Use of Ergonomics for Procurement and Design. Geneva, Swiss, 1998.
[5] Ying-Jui C, Chirh-Yun AT, Min-Li BY, Yu-Chuan BA. Assessing the Impact of
User Interfaces to the Usability of a Clinical decision support system. AIMIA
Symposium, 2003;808.
[6] Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Ex-
ploring barriers and facilitators to the use of computerized clinical reminders.
JAMIA, 2005 ;438-47.
[7] Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC,
Bayir H, Orr RA. Unexpected Increased Mortality After Implementation of a
Commercially Sold Computerized Physician Order Entry System. Pediatrics,
2005 ;116(6):1506-12.
[8] Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL.
Role of computerized physician order entry systems in facilitating medication er-
rors. JAMA, 2005 ;293(10):1197-1203.
[9] Vredenburg K, Isensee S, Righi C. User-Centered Design: An Integrated Ap-
proach. Prentice Hall PTR, Upper Saddle River, NJ, 2002.
[10]Organization IS. ISO 13407. Human Centered Design Process for Interactive
Systems. Geneva, Swiss, 1999.
12
[11]User-Centered Design. IBM. Available from: https://www-
01.ibm.com/software/ucd/ucd.html
[12]Norman DA, Draper SW. User Centered System Design: New perspectives in
Human-computer Interaction. L. Erlbaum Associates Inc. Hillsdale, NJ, USA,
1986.
[13]Marcy TW, Kaplan B, Connolly SW, Michel G, Shiffman RN, Flynn BS. De-
veloping a decision support system for tobacco use counselling using primary
care physicians. Informatics in primary care, 2008 ;16(2):101-9.
[14]Wang F, Hannafin MJ. Design-based research and technology-enhanced learn-
ing environments. Educational Technology Research and Development,
2005;53(4): 5-23.
[15]Porter SR, Rcs FD, Rcse FD, Scully C, Rcps FD. Etiology and management of
Xerostomia. AJP, 2004;97(1):28-46.
[16]Jontell M, Mattsson U, Torgersson O. MedView: an instrument for clinical re-
search and education in oral medicine. Oral surgery, oral medicine, oral patholo-
gy, oral radiology, and endodonlogy, 2005 ;99(1):55-63.
[17]Beale T, Heard S. openEHR Architecture Overview. Available from:
http://www.openehr.org
[18]Gulliksen J, Göransson B, Boivie I, Blomkvist S, Persson J, Cajander A. Key
principles for user-centred systems design. Behaviour & Information Technolo-
gy, 2003;22(6):397-409.
[19]Horasani RA, Anasijevic MI, Iddleton BL, C MS. Ten Commandments for Ef-
fective Clinical Decision Support: Making the Practice of Evidence-based Medi-
cine a Reality. JAMIA, 2003 ;10(6):523-30.
Aknowledgements
Scincere thanks go to Ian McNicoll, Soren Lauesen, and Dowen Birkheld for evalu-
ating domain concept models and the GUI. Many thanks also go to Mats Jontell,
Marie Lindgren and Göran Falkman, our team members; and to my love Mohsen
Nosratinia, and to Anna Gryszkiewicz for proofreading this paper. The author would
also like to express her gratitude to Olof Torgersson under whose supervision this
work has been done as a part of the author’s PhD study. The project was funded by
the Swedish Governmental Agency for Innovation Systems (VINNOVA), grant
2006-02792, as a joint project between Chalmers University of Technology and
Sahlgrenska Academy, and is in progress at the time of writing this paper.
Address for correspondence
Department of Computer Science and Engineering, Chalmers University of Technology, SE-
412 96 Gothenburg, Sweden. Email: hajar.kashfi@chalmers.se
Paper IV
Applying a User-centered Approach in
Designing a Clinical Decision Support System
Hajar Kashfi
Computer Methods and Programs in Biomedicine, Elsevier.
(manuscript submitted)
105
Applying a User-centered Approach in Designing a Clinical
Decision Support System
H.Kashfia,1,∗
aDepartment of Applied Information Technology
Chalmers University of Technology
SE–412 96 Gothenburg, Sweden
Abstract
Aim: To design a clinical decision support system (CDSS) by applying a user-centered design
(UCD) process with the aim of learning from applying UCD in a clinical context. Methods:
A UCD process has been applied in this study. To support this UCD process, several methods
were utilized such as prototyping, usability test, usability expert review and interview. Several
usability experts, users and domain experts were involved in this process. The work presented
in this paper is the outcome of the first three iterations of the project. Results: Several user
interface prototypes were developed, evaluated and improved iteratively. Characteristics of the
clinical context that may have an effect on applying a UCD process were detected and analyzed.
Conclusion: Applying a UCD process for designing this CDSS was very helpful and resulted in
various improvements in the design that would otherwise remain undetected until the application
is implemented and released to the users. The experience showed that UCD is considered to be
helpful but challenging in this context. There are characteristics of the users, the tasks and the
context to be aware of while applying a UCD process in a clinical context.
Keywords: clinical decision support system, user-centered design, human-centered design,
usability, design evaluation, interactive system design, user interface evaluation, qualitative
evaluation, clinical application, empirical usability evaluation method, non-empirical usability
evaluation method, usability test, usability evaluation
1. Introduction
The history of applying computers in the clinical domain goes back to the 50s. Since then,
there have been continuous efforts to benefit from computers in the care process specially diag-
nosis and treatment of patients to reduce preventable clinical errors (mostly human errors) and
to improve quality of care in general. A clinical decision support system (CDSS) is a computer
system that is used to facilitate the process of decision making for clinicians. This support is
provided by mapping or compiling existing data to useful information that can be used as a clue
for making the best decision [1]. Studies have demonstrated that CDSSs can be helpful in various
ways such as improving prescribing practice [2, 3, 4, 5], reducing or preventing errors [6, 5, 7, 8],
∗Corresponding author
Email address: hajar.kashfi@chalmers.se (H.Kashfi)
Preprint submitted to Computer Methods and Programs in Biomedicine May 18, 2011
steering up best practices in medicine or encouraging evidence-based medicine [4, 9, 5, 6] and
finally decreasing cost [6, 5].
More than 40 years have been spent on studying CDSS development. Nevertheless, the
theoretical advances have been ahead of practice so far [6, 10] and it is observed that CDSSs
suffer from a low adoption rate in spite of advances in research [6, 10, 11].
Various researchers have carried out studies to discover the reasons for low adoption of
CDSSs and to identify success and failure factors of these system. According to these studies,
success factors of CDSSs can be divided in two main categories of technical and non-technical
factors. Non-technical factors documented in these studies, in their general form, have been the
focus of the human-computer interaction (HCI) research field for years. To address such issues,
HCI recommends applying a process named user-centered design (UCD).
Taking HCI into consideration has become a routine in domains such as aviation and auto-
motive industry [12], nevertheless this is not the case in the clinical domain. Therefore, applying
such methods that are demonstrated to be helpful in other domains, disseminating the knowl-
edge gained by experience, and learning from successes and failures can be of great help in this
domain, specially since it is observed that these types of success factors of CDSSs are more
appreciated in theory than in practice.
1.1. Aim of the Paper
The aim of this paper is to improve the reader’s knowledge on the UCD process, and methods
to support UCD. The hope is to achieve this by discussing our experience in applying UCD in
a clinical context and presenting the process, phases, and methods carried out in this study1. So
far, three iterations of the project are gone through. Each iteration includes different phases and
various methods in each phase to support UCD. Several prototypes were created and evaluated
using a number of qualitative evaluation methods.
The paper starts with some background information on the importance of HCI in developing
CDSSs. Afterwards, details of three iterations of the UCD process and methods applied in each
iteration are discussed. Lastly, findings and limitations of the study are discussed.
2. Background
2.1. Human-computer Interaction
HCI is defined as “a discipline concerned with the design, evaluation and implementation of
interactive computing systems for human use and with the study of major phenomena surround-
ing them” [15]. The heart of HCI is the concept of usability which is a very important factor in
designing an interactive system [16]. Usability is defined as “the extent to which a product can be
used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction
in a specified context of use” [17].
Low usability can result in underuse (the system is not used so often and users stick to their
current methods for accomplishing the tasks) or misuse (the system is used improperly) of the
1Investigation of openEHR [13] as one of the electronic health record (EHR) interoperability standards, was another
aim of this study. UCD was applied to develop not only the user interface of the CDSS but also the domain models.
Domain models were developed iteratively and by involving domain experts as suggested by both UCD and openEHR.
The details about development of an openEHR-based CDSS is out of the focus of this paper, hence is skipped here. More
information about this aspect of the project can be found in [14].
2
system. Underuse and misuse of the system both bring costs to the organization or ruin the
reputation of the team or the company that developed the system [18]. There are a number of
benefits associated with designing a usable system such as increased productivity, reduced errors,
reduced training and support, improved acceptance and enhanced reputation [18].
To develop a usable interactive system, both functional requirements and non-functional re-
quirements, including usability requirements, should be considered by developers. Traditionally,
the focus of software design processes have been on functional requirements. As mentioned by
Seffah [19] “the importance is given to the functionality, and the interface is really a small issue
to be subordinated”. Nevertheless, nowadays there are design frameworks integrating functional
and non-functional requirements [18].
To develop systems with higher usability, the HCI community suggests a user-centered de-
sign (UCD) (or human-centered design (HCD)) process [20] and various methods to support
this process [18]. These methods cover different phases of the system development. The UCD
process is based on the idea that by involving users in the design and development process of a
system, the system will be more usable for the users in the intended context [20, 21, 22].
2.2. Importance of Human-computer Interaction in Developing Clinical Decision Support Sys-
tems
There are various studies that deal with the question of which factors should be considered
in design and development of a CDSS to result in an acceptable and effective system and to
reach a higher adoption rate? According to these studies, success factors of CDSSs can be
divided in two main categories of technical and non-technical factors. Non-technical factors
are mostly human-related issues such as: users’ interaction with the system, users’ workflow,
users’ training, and users’ satisfaction and acceptance. Moreover, as observed by Kaplan [23], to
evaluate a CDSS not only “quantitative” but also “qualitative” evaluation methods are required to
answer the questions of “why clinicians accept or do not accept a system” or “why they change
their practice behavior after introducing the system”.
Existence of such studies shows that there is motivation in applying qualitative evaluation
methods and consideration of HCI and usability in CDSS design. It is also evident from the
literature that developers of CDSSs are being informed about these matters. Nonetheless, the
published practical studies suggest that neither all of the methods, nor all aspects of UCD are put
into practice by developers of CDSSs. For instance, most of the studies report the application of
individual methods rather than UCD as a process to be applied in the whole life-cycle of systems.
Additionally, not much theoretical discussion of methods and costs and benefits associated with
them are documented in the related literature.
This suggests that there is a need to disseminate the benefits of applying UCD and its recom-
mended standard methods that cover not only the user interface evaluation but also various phases
of the project such as planning, requirements gathering, and designing. In addition, spreading the
knowledge of how to apply UCD in the clinical context for developing systems such as CDSSs,
and publishing success and failure stories of applying UCD in this context would be beneficial
to improve the practical attitude of CDSS developers towards HCI, usability and UCD.
2.3. The Project Background
In 2007, a team of computer scientists from two universities in Sweden started a joint project
with the clinical of oral medicine at Sahlgrenska University Hospital with the aim of developing a
CDSS for an oral disease named dry mouth. Dry mouth or xerostomia is “the abnormal reduction
3
Planning Context of use Requirements Design Evaluation
◦Usability planning
and scoping
◦Usability cost benefit
analysis
◦Identify
stakeholders
◦Context of use
analysis
◦Survey of existing
users
◦Field study or user
observation
◦Diary keeping
◦Task analysis
◦Stakeholder analysis
◦User cost-benefit
analysis
◦User requirements
interview
◦Focus groups
◦Scenarios of use
◦Personas
◦Existing system or
competitor analysis
◦Task or function
mapping
◦Allocation of
function
◦User, usability and
organizational
requirements
◦Brainstorming
◦Parallel design
◦Design guidelines
and standards
◦Storyboarding
◦Affinity diagram
◦Card sorting
◦Paper prototyping
◦Software prototyping
◦Wizard-of-Oz
prototyping
◦Organizational
prototyping
◦Participatory
evaluation
◦Assisted evaluation
◦Heuristic or expert
evaluation
◦Controlled user
testing
◦Satisfaction
questionnaires
◦Assessing cognitive
workload
◦Critical incidents
◦Post-experience
interviews
Table 1: Methods of the user-centered design (UCD) process. For more details about each method refer to [18].
of saliva and can be a symptom of certain diseases or be an adverse effect of certain medications”
[24]. There are various different factors dentists or dental hygienists should remember (e.g.
related diseases and drugs) therefore management of the disease is not always easy and some
information may be unwittingly omitted in the process and this may result in a lower quality of
diagnosis and treatment of the patient. The need for a CDSS for this oral disease was brought up
by specialists in Dentistry in the Clinic of Oral Medicine.
3. User-centered Design Iterations, Phases and Methods
The UCD process (depicted in Figure 1) starts with a planning phase and continues with a cy-
cle including four phases, namely understanding and specifying the context of use, requirements,
design, and evaluation. The UCD cycle will be repeated until the requirements are met and users
are satisfied with the design. End users are at the heart of this process and involvement of users
in all UCD phases is recommended. The HCI community have suggestions for the methods that
one may apply in different phases of the UCD cycle (see Table 1) [18].
Each iteration of the project may include all or some of the UCD phases. In each phase, a
collection of recommended methods is selected, based on the characteristics of the project and
the development team. For instance, in this project, iteration one included all of the five phases
while iteration three included only two of them. Three iterations of the project along with phases
and methods carried out in each iteration are discussed in Section 4, Section 6, and Section 8. A
summary of these three iterations, phases and methods is shown in Table 2.
In the following, definitions of the five UCD phases are given along with examples of the
methods suggested for each phase. Not all of these methods are defined in this paper. Neverthe-
less, the definitions of those methods that were applied in this study can be found in Section 4,
Section 6, and Section 8. For definitions of the other methods the reader can refer to the related
literature [18, 25, 26, 27, 28].
4
Plan the user-centered
design process
Understand and
specify the context of
use
Specify the users and
organizational
requirements
Evaluate design
agains requirements
Produce design
solutions
Meets
requirements?
Figure 1: The user-centered design (UCD) process cycle [21]. In this paper, the five steps or phases of the UCD process
are referred to as: the planning phase, the context of use phase, the requirements phase, and the evaluation phase.
3.1. The Planning Phase
In this phase, planning of the process is done. Stakeholders (those who may be affected
by the system), users, project team members and users who will participate in evaluations are
specified. High level information about users, users’ needs, and the environment (context of use)
is also gathered in the planning phase. Usability planning and scoping and usability cost benefit
analysis are two of the activities suggested to be carried out in this phase [18].
3.2. The Context of Use Phase
The focus of this phase is on understanding the technical, physical and organizational aspects
of the context, the user population, and users’ goals and tasks. Various methods can be used to
gather this information such as context of use analysis [18], task analysis [28, 18] and field study
[18, 25, 28, 27]. Information gathered in this phase is used as a basis for system design and
evaluation.
3.3. The Requirements Phase
The Requirements phase is considered to be one of the most important phases in software
development. In this phase, users and organizational requirements are gathered and documented.
In addition to functional requirements of the system, usability requirements are identified
in this phase in order to set objectives and priorities for the design team [18]. Effectiveness,
efficiency, learnability and satisfaction are some of the general usability requirements [18, 25].
The design should be evaluated against some usability objectives, i.e. usability goals, throughout
the iterative design process. In order to set up usability goals, one needs to specify relevant
measurable usability characteristics of the system, i.e. usability attributes [26] and to decide how
these are going to be measured. Interview (see Section 4.3.2) and existing systems analysis are
the methods suggested for this phase [18].
5
Iteration one Iteration two Iteration three
1. Planning 1. Planning
2. Context of use
3. Requirements
◦Literature review
◦Brainstorming session
◦Multidisciplinary group session
2. Requirements
◦Interview
4. Design
◦Sketching
◦Brainstorming session
◦Power point prototyping
3. Design
◦Design guideline
◦Paper prototyping
◦Java prototyping
1. Design
◦Low fidelity
prototyping
5. Evaluation
◦Informal user evaluation
◦Informal expert evaluation
4. Evaluation
◦Usability test
◦Think aloud
◦Interview
◦Usability expert review
2. Evaluation
◦Usability expert
review
Table 2: Summary of the iterations, phases and methods. Each column corresponds to one of the three iterations of the
project. In each column, various phases that have been carried out in that iteration are shown. Under each phase, there is
a list of methods applied in that phase.
3.4. The Design Phase
In this phase, based on the results from previous phases, design solutions are created. In
order to decrease costs, it is recommended to apply prototyping techniques such as sketching
(see Section 4.4.2) and paper prototyping (see Section 6.3.2) in this phase in early iterations of
the project [25].
3.5. The Evaluation Phase
Evaluation of design solutions against requirements gathered in the requirements phase (and
considering information gathered in the context of use phase) is carried out in the evaluation
phase. Two classes of evaluation methods can be applied in this phase. The first class is em-
pirical evaluation methods [26] (evaluations based on user participation) such as interview (see
Section 4.3.2) and usability test (see Section 6.4.1). The second class is non-empirical evalu-
ation methods [26] (evaluations based on developers or usability experts participation) such as
usability expert review (see Section 6.4.4), heuristic evaluation and cognitive walkthrough.
4. Methods Applied in Iteration One
In the first iteration of the project, all of the five UCD phases were gone through. Explanation
of the methods applied in these phases is presented in the following.
6
4.1. Methods Applied in the Planning Phase
Planning for the project was brief and included specifying the team, stakeholders (including
a specialist in dentistry who was going to be involved in all stages of the project. This person
is referred to as our main clinical partner in this paper). Planning was done in a group meeting
held at the Clinic of Oral Medicine at Sahlgrenska University Hospital.
4.2. Methods Applied in the Context of Use Phase
To understand the context of use and to identify stakeholders, meetings were held with the
Project Manager, and our main clinical partner in the Clinic of Oral Medicine.
4.3. Methods Applied in the Requirements Phase
Literature review, interview [25, 18], and multi-disciplinary group sessions were carried out
in the requirements phase in the first iteration. More about these methods is given in the follow-
ing.
4.3.1. Literature Review
This included studying articles in which the characteristics of dry mouth were explained, and
those that discussed success factors of CDSSs. Moreover, some existing CDSSs were studied.
4.3.2. Interview
Interview is a data gathering method in which a goal oriented conversation is performed with
the interviewee [25]. Based on the amount of control the interviewer imposes on this conver-
sation, four types of interviews can be defined: structured, unstructured, semi-structured and
group interview. Interviews in this phase were unstructured interviews with our main clinical
partner that were held at the Clinic of Oral Medicine. The aim of these interviews was to gather
information about the functionalities users expect from the system.
4.3.3. Multi Disciplinary Group Sessions
Every second week, a meeting was held at the Clinic of Oral Medicine to share the project
progress with some of the stakeholders and to get their opinion on different issues such as graph-
ical user interface (GUI) design, and decisions about user representatives and domain experts
(dentists and dentals hygienists) to be interviewed.
4.4. Methods Applied in the Design Phase
Brainstorming [18] and low fidelity prototyping techniques [25] were applied in this phase
to produce design solutions based on the information gathered in the previous phases.
4.4.1. Brainstorming Sessions
To produce design solutions, brainstorming sessions [18] were held at Sahlgrenska together
with our main clinical partner. At the beginning of the first session, our main clinical partner was
introduced to some existing CDSSs in order to get a better idea about functionalities a CDSS
may provide.
7
4.4.2. Low Fidelity Prototyping: Sketches and PowerPoint Prototyping
To realize the produced designs, prototypes of the GUI were developed in this phase. The
prototypes developed in this phase were low fidelity prototypes [25]. Low fidelity prototypes are
not much like the final prototype. The materials used for creating such prototypes (e.g. paper and
pen) are different from the material in the final product. Low fidelity prototypes can be developed
very cheaply, quickly and simply.
In this phase, two low fidelity prototypes of the system were developed with a close col-
laboration with our main clinical partner. These included sketches drawn in the brainstorming
sessions, and a more interactive Powerpoint prototype, which was later used for evaluations.
Sketching relies on simple drawings of the design ideas [25] and is a low fidelity prototyping
technique. The Powerpoint prototype was the realization of the same design solution as in the
sketches but in a more interactive manner.
4.5. Methods Applied in the Evaluation Phase
In this phase, both empirical and non-empirical evaluations were carried out to evaluate the
Powerpoint prototype created in the previous phase.
4.5.1. Informal User Evaluations
In two meetings, the prototype was shown to our main clinical partner. His comments about
the design were recorded on paper and discussed. In these sessions, the test person was supported
with explanations whenever he did not understand something on the GUI. The meetings were
held at the Clinic of Oral Medicine.
4.5.2. Informal Expert Reviews
In several informal meetings with some usability experts and user interface design experts,
the prototype was evaluated and alternative designs were suggested and discussed2. The meetings
were held at the Chalmers University of Technology.
5. Results from Iteration One
Activities in the first iteration of the project resulted in a basic understanding of the system
requirements and the context of use, i.e. the Clinic of Oral Medicine. As suggested by the
UCD process, this iteration also included the development of early prototypes of the system and
prototype evaluations. In the following, the details of the results are given.
5.1. Results from the Planning Phase
A multidisciplinary team was set up to handle the development of the project. The team
included the following members: our main clinical partner, three computer scientists (for sim-
plicity they are called designers), and one programmer. It was decided to monitor the progress
through regular meetings in the Clinic of Oral Medicine. No clear decision about how and when
to involve more domain experts and tests users were made at this stage.
2the term “usability expert review” is not used since one of the experts was not a usability expert but a person with
experience in user interface design.
8
User Characteristics User Environment User Tasks Usability
Attributes
Measuring instru-
ment
Metrics
◦General dentists
◦Specialists in
dentistry
◦Dental hygienist
◦No time for
learning
◦High pressure
◦Technical
support
◦Many
interruptions
◦Many patient
visits during the
day
◦Partial data
entry
◦Patients’
record
overview
◦Get support
for dry mouth
◦Overview
articles
◦Staisfaction
◦Error rates
◦Efficiency
◦Ease of learn
◦Questionnaire
◦Bench mark
tasks
◦Data log
◦Time to
complete a task
◦Number of
errors in a
specified time
period
◦Number of steps
per task
Table 3: The usability attributes.
5.2. Results from the Context of Use Phase
The system is intended to be used in the Clinic of Oral Medicine at Sahlgrenska. Information
technology (IT) has been around in this clinic for at least 12 years. People in this clinic work
with T4 [29] and MedView [30], two clinical applications for data gathering and information
visualization. Specialists in dentistry and dental hygienists at this clinic are users of the system.
Patients are also affected by the system.
5.3. Results from the Requirements Phase
The data entry for dry mouth would be partially done by patients via the touch screen GUIs
available at the clinic, and partially by clinicians via the MedView application [30]. Afterwards,
if the patient is likely to have dry mouth, the clinician decides whether to use the CDSS. There-
fore, the CDSS is not aimed at data entry, but presentation of patient data in a way that makes
diagnosis easier.
5.3.1. Functional Requirements
System requirements gathered in this phase included: (i) partial data entry (the possibility
to edit patient information if required) (ii) providing an overview of the patient information (iii)
providing hints on the information (e.g. using colors to indicate risky items) to make diagnosis
easier (iv) providing diagnostic suggestions (v) providing related clinical evidences
5.3.2. Usability Requirements
As suggested by Helander [26], based on categorization of the users, the context of use, and
the tasks to be performed by users, a set of usability attributes were defined for the application.
Three categories of users were specified: (i) general dentists (ii) specialists in dentistry (iii)
dental hygienists. The characteristics of the environment in which these users would apply the
system were gathered. Four types of usability requirements suggested by Shneiderman [25] were
considered to be relevant to this system. The main tasks to be undertaken by users were specified.
Finally, measurable usability attributes and their measuring methods were set. A summary of the
aforementioned information gathered in this phase is presented in Table 3. Based on the usability
attributes, and by specifying usability goals a usability specification was prepared for the project
(see Table 4).
9
Attribute Measuring instrument Measuring method Planned level
Ease of locating info Difficult 1 2 3 4 5 Easy Average rating >4
Ease of understanding colors Difficult 1 2 3 4 5 Easy Average rating ≥4
Ease of understanding the decision
support part
Difficult 1 2 3 4 5 Easy Average rating >4
General satisfaction Low 1 2 3 4 5 High Average rating >4
Table 4: The usability specification
5.4. Results from the Design Phase
Based on basic understanding of the users, tasks and context acquired in the previous phases,
an initial design of the system was created for further development. The suggested design solu-
tion was based on the idea of clinical questionnaires. In other words, a list of related questions
were provided together with our main clinical partner to be used for data gathering (using ex-
isting applications) and for data representation in the CDSS being developed 3. This design
idea, however refined in various steps, remained as the core design idea throughout these three
iterations.
To realize the design solution, some sketches and a power point prototype were produced in
this phase. One of the prototypes generated in the first iteration is depicted in Figure 2.
5.5. Results from the Evaluation Phase
The user evaluations on the first prototype revealed some minor changes in the design while
the informal expert reviews resulted in major changes.
5.5.1. Results from the User Evaluations
Informal user evaluations in this iteration did not cause major changes. The design solutions
were developed together with our main clinical partner, hence the evaluation with the same user
did not result in major improvements. In the following, some of the comments given on the
design are presented:
◦Navigation:
To improve navigation and to make the design more compatible with the user workflow, a
change in the order of items was suggested. For instance, it was suggested that chemother-
apy related questions be moved under the “Drugs” section.
◦Color coding:
It was suggested that suitable colors to indicate missing information be applied.
◦Visual clarity:
It was suggested that unnecessary items be removed in order to improve visual clarity of
the GUI. For instance, removal of the risk bar was suggested.
3This questionnaire was also used as a basis for developing openEHR domain models named archetypes [14]
10
A
horizontal collapsible section
Navigation among pages
Several questions are included in
each section
Collapsible sections
Progress bars to
indicate the amount
of risk and the
number of
answered questions
Figure 2: The GUI prototype created in the first iteration. This design solution included three collapsible horizontal
sections. Each section included various questions to be filled in or reviewed. On top of the first section there existed
two progress bars that were meant to show how much of the information is missing, and how much dry mouth risk is
calculated based on answers to the questions. To get the diagnostic suggestions and review related clinical documents,
users needed to navigate among screens.
11
5.5.2. Results from the Expert Reviews
Informal expert review sessions held in the first iteration resulted in major changes in the
design. As illustrated in Figure 2, in the first iteration, designers and our main clinical partner,
came to a design solution in which users needed to navigate among various screens. This design
solution was highly objected to by the experts involved in the evaluation. One suggestion was to
reduce the number of screens by applying the tree table design pattern [31].
6. Methods Applied in Iteration Two
Analysis of the evaluation results in the first iteration revealed that the design solution did not
still meet all the requirements. Therefore, as recommended in the UCD process, it was decided
to go through another cycle of UCD, i.e. the second iteration. The second iteration of the project
included four phases namely planning, requirements, design and evaluation. Various methods
were carried out in these phases as presented more in detail in the following.
6.1. Methods Applied in the Planning Phase
At this stage, more time was spent on planning to get more interviewees and test users for
requirements gathering and running usability tests. To motivate involving more users, in a short
presentation, stakeholders (including our main clinical partner) were informed about the benefits
associated with UCD and involving more users in the UCD process.
6.2. Methods Applied in the Requirements Phase
Interview is the method used in this phase to gather more information about the disease, and
the workflow. The interviews held in this phase were all semi-structured interviews in which
users were asked questions with a specific set of answers and also open questions that enabled
them to share their opinions more easily. Four domain experts, three specialists in dentistry
and one dental hygienist, were interviewed in this phase. The interviews were held at different
hospitals and clinics where the interviewees worked.
6.3. Methods Applied in the Design Phase
Based on the evaluation results from the first iteration and the discovered design flaws, a
new design solution was developed in the second iteration. Design guidelines and two different
prototyping techniques were applied in this phase as discussed in the following.
6.3.1. Design Guidelines
In this phase of the project, some design patterns and guidelines were considered in develop-
ing the new design solution. One of them was the Microsoft health common user interface4that
provides some design guidelines for designing GUIs for clinical applications. Various guide-
lines in the area of clinical noting and terminology, consistent navigation, medication, patient
identification and miscellaneous are provided by Microsoft.
One other known design solution that was applied in this project was tree table [31]. Tree
table is a GUI design pattern that is suggested for application when hierarchical information is
to be presented in columns.
4http://www.mscui.net
12
6.3.2. Low Fidelity Prototyping: Paper Prototyping
Paper prototyping is a low fidelity prototyping technique (see Section 4.4.2) by which a
paper-based simulation of the GUI is developed [25]. A paper prototype of the system was
created in this phase. Paper prototyping was selected since it is easy, time efficient and cost
effective [25].
6.3.3. High Fidelity Prototyping: Software Prototyping
High fidelity Prototypes [25] consist of the materials one expects in the final product. These
prototypes are much like the final product. Also, it takes more time, effort and money to create
such prototypes. The high fidelity prototypes are strong in testing technical functionalities and
also selling the design idea to stakeholders.
In addition to the paper prototype, a Java-based prototype of the system was also developed
in this phase. One motivation for creating a Java-based prototype was that by having a more
interactive prototype, and a prototype which is more similar to the final product, the results of
the user tests would be more reliable. However, this turned out to be an improper decision (see
sec:pro). One other motivation for creating a Java-based prototype was to create an opportunity to
investigate the openEHR related functionalities of the system and to investigate the development
of an openEHR-based application in more detail.
6.4. Methods Applied in the Evaluation Phase
Three empirical evaluation methods namely usability test [28], think aloud protocol [28], and
interview, together with a non-empirical evaluation method namely usability expert review [16]
were carried out in this phase. Details about each method are as follows.
6.4.1. Usability Test
Usability test is a collection of methods used to evaluate the usability of a product. It is
typically focused on how well a user can use the product to accomplish a specific task, and also
errors that occur during this process [28].
Usability tests were conducted in the Clinic of Oral Medicine at Sahlgrenska. Before each
test, the test person was verbally informed about the test procedure, goals of the test and how
she/he can be more helpful e.g. by being honest about the design flaws.
Two persons were responsible for running the test. Four usability tests were run on three
specialists in dentistry and one dental hygienist. Each test session took around two hours. Users
were given 10 tasks to accomplish. Tasks were selected to cover all elements in the design. The
list of tasks was printed and one of the test leaders read each task for the test user. Hints were
given just if the user was not able to proceed.
In the usability test sessions, users were asked to verbalize what they thought while they inter-
acted with the system (see Section 6.4.2). Moreover, after each session, users were interviewed
about their background also their experience of using the system. More about the interviews is
given in the following.
6.4.2. Thinking Aloud Protocol
Thinking aloud is a simple observation method in which the user is asked to verbalize what
she/he thinks and performs during the observation [28]. This method is used in usability test to
gather more information about how a user interacts with the GUI.
In the usability tests carried out in this phase, test users were asked to think aloud. Their
comments were recorded on paper during the test session and analyzed afterwards.
13
6.4.3. Interview
After each usability test, the test user was interviewed about their experience of using the
system, also their background. The interviews were semi-structured (see 4.3.2) and consisted of
44 questions, and around 10 background questions such as personal information and information
about other applications the user works with.
In addition, three specialists in dentistry and one dental hygienist (referred to as domain
experts) were interviewed about the disease and the workflow. Also, by presenting the GUI
prototype to the interviewees and letting them reviewing the design, their opinions about the
prototype were gathered in this stage.
6.4.4. Usability Expert Review
One way to evaluate a GUI against standard and best practice design solutions is to ask
usability experts to review the design [16]. In this phase, three usability experts reviewed the
prototype. The reviews were conducted at Chalmers University of Technology. In each review
session, one usability expert was asked to review the prototype, i.e. the Java-based prototype.
One of the designers was also present at the review sessions that provided an opportunity to
discuss design flaws and suggestions to improve them. Comments and suggestions given by the
experts were recorded on paper for further analysis.
7. Results from Iteration Two
7.1. Results from the Planning Phase
Beside our main clinical partner, who was involved in the project from the beginning, three
more specialists in dentistry and one dental hygienist were selected to be interviewed. Another
group of three specialists in dentistry and one dental hygienist were also specified to participate
in the project as test users for user interface evaluations.
7.2. Results from the Design Phase
Based on analysis of the evaluation results in the previous iteration, also studying some de-
sign guidelines, a new design was produced in this phase. Figure 3 depicts the prototype devel-
oped in this phase. This prototype was later translated to a Java-based prototype to be used in the
usability tests. The design was based on the first developed idea of clinical questionnaires (see
Section 5.4) and the tree table design pattern (see 6.3). As illustrated in Figure 3, the new GUI
consisted of three different collapsible sections that are explained in the following.
In the GUI, different colors (green, gray, orange) were used to indicate different conditions
of the patient data. Orange was used to indicate “risk”, green to show “no risk”, gray to show
“missing information” as a background color for unanswered questions.
7.2.1. Symptoms and Signs
This section included three main categories of information: (i) a tree structure of symptoms
reported by the patient or signs observed by the clinician (marked as A in the figure). This was
in form of a set of questions and answers (see Section 5.4) (ii) details about the total number
of questions, the number of answered questions, and potential dry mouth risk based on the an-
swers to the questions (marked as B in the figure) (iii) a user interface input component for each
question e.g. radio buttons for yes/no questions.
14
F
A
B
C
D
E
G
H
I
Figure 3: The GUI prototype, the second iteration
15
The tree structure of questions which represented symptoms and signs for each patient in-
cluded information about general and local symptoms , i.e. oral symptoms. It also included in-
formation about previous treatments (labeled as “Iatrogenic”) and related diseases. Users might
browse the tree structure from more abstract concepts to detailed patient information. For in-
stance under the “Oral symptoms” heading, users could see if the patient had noticed any de-
crease in her/his amount of saliva. The progress bars and the numbers in front of it (marked
as C in the figure) indicated the total number of questions and answered questions. Based on
answers to the questions, the color of the progress bar indicated if that specific category of items
contributed to the patient’s disease. If the bar was orange it indicated contribution.
7.2.2. Decision Support
This section included: (i) an overview of the diagnostic suggestions (contributing factors) by
the system (marked as D in the figure). The pie chart in this section was designed for showing
the proportion of causes (ii) related clinical documents (marked as E in the figure). Abstracts of
the relevant papers were presented in text format in this section.
7.2.3. Event Details
This section consisted of two tabs: Drug and Lab Value. In the drug tab (marked as F in
the figure), users could view: (i) a list of drugs that the patient took (ii) details about each drug
e.g. if the drug had a dry mouth side effect and a link to a website named Fass5. Fass includes
a comprehensive database of the existing drugs in the market, their characteristics and their side
effects (iii) a possibility to prescribe a new drug.
In the lab value tab, there was: (i) results of previous lab tests (marked as G in the figure) (ii)
details about the results for instance if the results indicated any risk of having dry mouth using
icons (marked as H in the figure) (iii) a possibility to order a lab test (marked as I in the figure).
7.3. Results from the Evaluation Phase
According to the UCD process used in this project, the new design solutions were evaluated
by users and usability experts, in order to find out whether all the requirements are satisfied, and
to discover design flaws.
7.3.1. Results from the Expert Evaluations
The three expert evaluations resulted in discovering various general design flaws regarding
navigation, feedback, pliancy and hinting, consistency, color coding and icons. Experts also
provided suggestions for most of the flaws. Some examples of these flaws are summarized in
Table 5.
7.3.2. Results from the Usability Tests and Interviews
Usability tests were carried out on the Java-based prototype of the system. In addition, more
information about the data to be presented on the GUI, the structure of items and also the work-
flow was gathered in interviews with end users and domain experts. Users also had some opinion
about how the GUI can be improved.
More details about the results are presented in the following. Some of these design flaws are
also summarized in Table 6.
5http://fass.se
16
Flaw Example and details Recommendation
Navigation The nested structure of the sections and
trees, including more than 2 levels, is dif-
ficult to navigate and find information
The collapsible sections should be re-
placed with tabs
Functionality The pie chart is not easy to understand for
users, also users needs to navigate in the
small area provided for related articles in
order to find an article. Also it should be
possible for users to sort questions
A list of contributing factors should be
provided, the articles should be listed in
a larger area or in a pop-up window. A
sorting functionality should be provided
Visual clarity Distinguishing questions and their an-
swers is not visually easy
A stripped background for questions is
needed
Consistency On top of the screen, various collapsible
horizontal sections are presented while at
the bottom, there exists a tabular table
area
A consistent design solution should be
used for representing information (all
tabs or all collapsible sections)
Pliancy and
hinting
Some of the buttons are small, and not
easily detectable, also no pliancy re-
sponse is provided so that users find out
that the button is click-able
Pliancy and hinting is required for vari-
ous items on the screen
Icons Some icons are not easy to understand for
instance the icon used for indicating the
risky items in the tree
New icons should be designed
Color coding The green color in icons and in the bars
are not the same however they have the
same meaning, it is also better to use
more than just one color in each bar to
provide more information
The same colors for bars and icons should
be used. Also 3 colors should be in-
cluded in the bars to show unanswered
questions, and risky and safe answers
Table 5: The design flaws detected by the usability experts in the second prototype.
Flaw Example and details Recommendation
Navigation The nested structure of the sections and
trees was difficult to navigate for users
specially since the structure and order of
some items was not based on the user
workflow model
The collapsible sections should be re-
placed with tabs and arranged according
to the user workflow model
Functionality The pie chart was not easy to understand,
also some functionalities such as suggest-
ing products to users or sending referral
was not included in the design
A list of contributing factors should be
provided, also new functionalities: writ-
ing a comment, suggesting products,
sending referral, etc. should be included
Pliancy and
hinting
Users did not detect some of the clickable
areas
Pliancy and hinting for various items on
the screen should be provided
Color coding The users complained about the yellow
color used in the design, they were ex-
pecting a combination of red, yellow,
green
The new colors : red, blue and green
should be used in the design to indicate
risky, unanswered and safe items
Language Some of the terms such as “Iatrogenic”
was confusing for users
Based on users’ feedback new terms
would be used in the design
Table 6: The design flaws detected in usability tests of the second prototype
17
◦Navigation
The order of sections, also the possibility to collapse them totally, resulted in confusion for
all of the test users. Half of the test users, stuck to the first section (symptoms and signs)
which made it impossible or challenging for them to find the information located in other
sections.
◦The user workflow model
All of the test users reacted to the order of headings and some of them to the order of
sections. One of the complaints was about oral symptoms being located before general
symptoms in the tree.
According to the users and domain experts interviewed, diagnosis and management of dry
mouth (the users workflow model) was a bit different to what was gained in the previous
iteration. According to them, management of a dry mouth patient starts by general anam-
nesis. After general anamnesis, information related to local symptoms and examination
is gathered. The next phase would be to do some tests if needed; including saliva mirror
test and biopsy test. This workflow model affected the users’ interaction with the system.
Even though the system is not going to be used for data entry (but for presenting data),
users had a problem locating information when they found the organization of information
on the screen not compatible with their workflow.
◦Color coding
Surprisingly, all of the test users either misunderstood the colors or did not pay attention
to them. Part of that might be because they were new to the system. They admitted that
if they learned the meaning of colors they would pay more attention to them. Three out
of four test users suggested applying red, yellow and green colors, the same as traffic light
colors, that everyone is familiar with.
◦Language
Most of the headings used in the user interface were familiar to the test users. However,
some of them e.g. “Iatrogenic” were confusing since they normally would not use that
word to indicate previous treatments on users. “General Symptoms” is another heading
which should be changed to the term “Anamnisis” which exists in other applications and
users were familiar with. “Oral Symptoms” should also be replaced with the term “Local
Symptoms and Examinations”.
◦Visual clarity
Having duplicate headings made users confused. For instance, in the “Symptoms and
Signs” tree there was a sub-tree including information about current drugs the patient took.
Users could not however edit that information. On the other hand, in the last section named
“Event Details”, information about current drugs with more details and a possibility to
prescribe new drugs was provided (“Drug” tab). When test users were asked to prescribe
a drug, they were confused about choosing the right section. This therefore resulted in a
number of errors for related tasks.
In addition, all of the users misunderstood the meaning of the pie chart (in the “Decision
Support” section) and the information presented there. The usability tests, revealed that
the i.e. users’ perception of the pie chart (a presentation of the risks percentage, or the
percentage of contributing causes) was obviously wrong.
18
Attribute Number of related
questions
Reached level Is the goal satisfied?
Ease of locating info 11 3 No
Ease of understanding
colors
3 1 No
Ease of understanding
the decision support
section
1 1 No
General satisfaction 5 3 No
Table 7: The results of the usability tests in iteration two
◦Functionality
During usability tests, it was noticed that some functionalities provided in the prototype,
would not be used by users. For instance, all of the test users admitted that they would not
prescribe a drug most of the time, and they would never change ordination of a drug which
was prescribed by another physician.
On the contrary, it was discovered that there were some functionalities not considered in
the design, for instance the possibility to send a referral to a colleague (in most cases a
specialist. e.g. a Sj¨
ogren’s syndrome specialist)
8. Methods Applied in Iteration Three
As presented in Table 7, usability tests and interviews with end users revealed that the usabil-
ity goals were not satisfied. Moreover, various general design flaws were detected by usability
experts that needed to be removed. This motivated the decision of going through another cycle
of the UCD process, i.e. the third iteration.
Based on the results from the previous evaluations, the designer realized that the new design
should lead to improvements in navigation, visual clarity, consistency, language, icons and color
coding. Therefore, changes in colors, icons, labels, and the order of sections should be considered
in the new design.
8.1. Methods Applied in the Design Phase
Analyzing the evaluation results and comparing them to the usability goals (see Table 4),
led to designing two parallel design solutions. By producing parallel designs [18], there was a
chance to evaluate two designs at the same stage and to choose the best design solution for further
development. In this phase, a low fidelity prototyping tool was used to develop two prototypes
of the system.
8.2. Methods Applied in the Evaluation Phase
In this phase, the same three usability experts reviewed the two new prototypes. One of the
designs was selected for further development. Various general design flaws were detected in the
selected design and suggestions were made to improve the selected design. The review sessions
were carried out in the same manner as the reviews in the second iteration (see Section 6.3).
19
Flaw Example and details Recommendation
Functionality The users should be able to see the his-
tory of actions (comments, referrals, etc.)
A history tab should be added to the
screen
Icons Some icons are not easy to understand,
for instance the icons used for “sending
referral” or “ordering a lab test”
New icons were suggested to be used
Visual clarity Some of the icons were duplicated (for
instance the icon used for referral was
used in three locations on the screen)
Duplicated icons should be removed
when not required
Feedback Proper feedback does not exists for some
actions. For instance in case of ordering a
lab test users should be informed clearly
that the action is done
Feedback in form of pop-up windows
should be provided for various function-
alities
Control Users may not clearly see what informa-
tion is being saved or transferred. For in-
stance while sending a message to a col-
league the automatically generated part
of the message is not presented to the user
All the related information should clearly
be shown to the user, and suitable feed-
back should be provided
Table 8: The design flaws detected by HCI experts in the fourth prototype
9. Results from Iteration Three
9.1. Results from the Design Phase
Considering the evaluation results from the previous iteration, as suggested in UCD, new
design solutions were created. As mentioned before, two parallel designs were developed in this
phase.
In the third prototype (Figure 4), the horizontal sections were replaced with tabs to improve
navigation. The order of tabs was changed to be more compatible with the user workflow to
improve navigation. Some headings were updated based on the users’ language. The four tabs on
top(marked as A in the figure) covered information to be gathered for performing the diagnosis
(symptoms and signs together with previous lab test results). At the bottom (marked as B in
the figure), there were seven tabs that included the diagnostic suggestions and various actions
clinicians might perform after diagnosis. The order of tabs was based on their priority in the user
workflow model.
The fourth prototype (Figure 4) was quite similar to the third one. The main difference
between these two prototypes was that the items presented at the bottom of the screen (actions)
in the third prototype (marked as B in Figure 4) were located on the right side of the screen
(marked as A in Figure 5) in the fourth one.
9.1.1. Results from the Evaluation Phase
According to UCD, there was a need to evaluate the improved design solutions developed
in the previous phase. The two new prototypes were reviewed by the three usability experts.
Usability experts approved the fourth prototype since the fourth prototype was observed to be
better in visual clarity and navigation.
In addition, suggestions were given by the usability experts to improve the design and re-
move the remaining minor design flaws. Some examples of the flaws detected by experts in this
prototype are summarized in Table 8.
20
A
B
Figure 4: The GUI prototype, the third iteration
21
A
Figure 5: The GUI prototype, the fourth iteration
22
9.1.2. The Final User Interface Design
Finally, based on usability expert reviews and recommendations provided by them, a final
prototype of the system was created. This prototype is depicted in Figure 6.
In this prototype, four tabs on the left side of the screen included a tree structure of different
categories of the patient’s health information (anamnesis, local symptoms, etc.) (marked as A in
the figure). In each tab, and for each sub-category of information a progress bar was presented
(marked as B in the figure). The bars included various colors and numbers to indicate the number
of unanswered, risky and safe items. Three colors (blue, red and green) indicated the unanswered,
risky and safe items (for each category of information/question). Moreover, three different icons
were used next to each item/question to convey the same information (marked as C in the figure).
On the right side of the screen, the tabs were included (marked as D in the figure). The first tab,
“Take an action”, included the information related to the actions users could take (e.g. writing a
comment). On the second tab, “Previous actions”, users could overview the previous actions on
the patient case (e.g. previous comments)
10. Discussion
UCD is a design and development process which is supported with a collection of standard
methods [18]. These methods cover different phases of the application development process.
UCD is more than just involving users in evaluations. It focuses also on understanding the users,
the users’ tasks, the users’ goals, and the context in which the system will be used.
In this study so far, three iterations of the UCD process were carried out. A collection of
recommended methods were selected to be conducted in various phases of the iterations. Some of
the lessons learned in relation to various UCD phases and methods are discussed in the following.
10.1. Regarding the Planning Phase
It is recommended to have the UCD planning early in the project [32, 18]. The planning
includes decisions about the users to be involved, and how and when they participate in the
process. Moreover, in the planning phase, user representatives should be selected carefully to
cover all of the user types [18].
Improper UCD planning is one of the shortcomings of this study. While active user involve-
ment was considered in the project, it was not clearly specified where, when, and how the users
should participate in the process. We had no agreement about the users and domain experts to be
involved in requirements gathering, design and evaluations.
In the first iteration of this project, only one user, i.e. our main clinical partner, was involved
in the UCD process. Our main clinical partner of course cannot be the representative of all of the
user types of the system.
In the second iteration, the stakeholders were persuaded to involve more users and domain
experts in the UCD process. Consequently, several test users were introduced to the development
team. After interviewing all of the test users (in usability tests), it was revealed that all of them
were dental PhD students. Obviously, these users were not representatives of the all user groups
either. This could have had an effect on our evaluation results. Selection of this group of user
representatives by people at the Clinic of Oral Medicine could have been due to the fact that ,as
researchers, they had more flexible schedules. Therefore, without any cost to the clinic, they
could be asked to act as test users in the project. On the other hand, other specialists in dentistry
and dental hygienists were booked for patient visits. This made it more difficult to book them for
usability tests, and surely using their time for usability tests could be more costly for the clinic.
23
A
B
C
D
Figure 6: The GUI prototype, the last iteration
24
By a proper planning early in the process, number and types of the users to be involved
in requirements gathering, design and evaluations can be discussed and finalized. With timely
planning, in case of facing the aforementioned issues, the project team will have enough time to
motivate and support involving a wide category of users. Although it sounds costly to involve
more users in the process at first glance, it has of course a return of investment in the long run
when a usable system for all types of users is developed.
10.2. Regarding the Requirements Phase
10.2.1. The User Workflow Model
It is demonstrated how the clinical workflow plays an important role in developing usable and
acceptable clinical applications [33, 34, 35, 36]. Researchers in the clinical domain recommend
conducting workflow analysis in order to prevent failures of clinical applications and reducing
clinicians’ frustration while applying the system [36]. It is reported how changes in clinical
workflow, or alternation of clinicians’ interaction to each other, among other reasons, can result
in increased patient mortality after installing a clinical application [35].
Users in the clinical context are known to resist changes in their everyday workflow [33],
therefore, to develop a usable clinical application, it is vital to understand the workflow and
suggest design solutions based on this workflow. All in all, UCD helps to understand the users’
requirements, the user workflow and design a system according to this information.
In this project also, it was experienced how minor changes in the workflow can affect the
users’ reaction to the system. Based on the clinical literature and also interviews with domain
experts, it was concluded that “oral symptoms” or “local symptoms” are of more importance in
patient health information related to dry mouth. Therefore, as general design patterns suggest,
this information preceded other categories of information such as “general health information” or
“anamnesis” in the prototype. However, all of the tests users reacted to this structure of informa-
tion and this had negative effects on the GUI navigation. Users commented that in their everyday
workflow, the process starts with “anamnesis” so they expect that category of information be
located first on the GUI as well.
10.2.2. Understanding Users’s Requirements
As it is expected from an iterative design process, functional requirements of the system were
updated as well as the GUI design in these iterations. Some functionalities were removed and
some were added to the requirements specification of the system. For instance, prescribing was
removed since users indicated that they would never use this feature due to the organizational
rules. Sending referrals was an example of functionalities that were added to the system, as a
result of usability tests and interviews.
10.3. Regarding the Design Phase
10.3.1. Low Fidelity versus High fidelity Prototypes
Both low fidelity and high fidelity prototypes were developed in this UCD process. Low
fidelity prototypes are those prototypes that are thrown away after evaluation, from those paper
prototypes are common. On the contrary, high fidelity prototypes are much more like the final
product and use materials to make this possible. For instance, a Java-based user interface is a
high fidelity prototype. Surely, more time and effort is needed to produce high fidelity prototypes.
Such prototypes are more suitable for evaluating technical issues [25]. It should be noted that
25
inertia to bad design is more likely in cases that high fidelity prototypes are used. This is mainly
because of more time and effort that designers and developers put into such prototypes.
For complicated user interfaces such as ours, it is better to run the usability tests on low
fidelity prototypes so that any changes in design can be made more easily and without high cost.
In this project, around five months were spent on developing the Java-based user interface, and
this means that part of this work should be repeated since some changes should be made in the
prototype based on evaluation results (see Section 6.3). Moreover, since the goal of evaluations
was not to measure technical qualities of the system, paper prototyping or other types of low
fidelity prototypes would be sufficient in these three iterations [25]. This is also evident from the
nature of flaws detected in evaluations that could have been detected by using paper prototypes
as well.
One other motivation for creating a high fidelity prototype of the system was to have an
opportunity to investigate the openEHR related functionalities of the system, this could also be
postponed until the low fidelity prototype of the GUI was evaluated and finalized.
10.3.2. Navigation Improvement
According to Cooper [37] navigation is “any action that takes a user to a new part of the
interface or which requires him to locate objects, tools or data”. Poorly designed navigation is
one of the typical flaws in the usability of interactive applications [37]. More importantly, it is
observed how navigation problems in clinical applications can facilitate clinical errors [38].
As a result of applying UCD and various empirical and non-empirical evaluation methods,
the navigation was improved in the prototypes during the three iterations of the project. At first,
usability experts noticed that there were too many screens to be navigated by users. Therefore,
screens in the prototype were converted to different collapsible sections in the new design as no-
ticeable in Figure 3. Later on, the usability tests, revealed that users still had problems navigating
and locating information. Therefore, in the new design different tabs substituted the horizontal
collapsible sections.
10.4. Regarding the Evaluation Phase
Usability evaluation methods are divided into two main classes namely empirical methods
and non-empirical or usability inspection methods [16]. Interviews [25, 27, 28], field studies
[25, 27, 28], questionnaires [25, 27, 28], usability tests and think aloud methods are some of the
methods that belong to the first category. Usability expert reviews, cognitive walkthrough [16, 25,
28] and heuristic evaluation [16, 25, 28] are examples of the methods that belong to the second
class. Each of these evaluation methods has strengths and weaknesses. For instance, expert
reviews are suitable for detecting general design flaws and one of their strengths is that experts
will provide alternative design solutions to remove detected flaws [16]. In contrast, empirical
methods are strong in discovering detailed design flaws that even the most competent usability
experts may not detect, especially in cases when the flaws are domain specific [16].
There is no need to apply all methods in an individual project. It is, however, recommended
that suitable methods be selected by the development team based on the understanding of trade-
offs associated with each method [16]. Surely, by combining methods from these two categories
a project can benefit from the strengths of each method, as is the case in this study.
The studies in the area of CDSS development suggest that empirical evaluations, i.e. evalua-
tions through user participation, are much more popular among the CDSS developers compared
to the non-empirical methods, i.e. evaluations through expert analysis. Non-empirical methods
26
have been applied in very few studies [39, 40, 41]. Rare application of evaluations through expert
analysis can be due to insufficient knowledge of developers on such methods or inaccessibility
to experts who are capable of carrying out such methods. Identifying the reasons for such dis-
tribution of methods in CDSS development projects needs further investigation and is not within
the scope of this paper.
In this project, both empirical and non-empirical evaluations were applied, and resulted in
improvements in the design. Results detected in usability expert reviews were all general design
flaws (see Table 5, 8), on the other hand, the flaws detected in usability tests were much more
domain-specific including the language and workflow related issues (compare Table 5 and Ta-
ble 6). It is evident from the tables, how applying both classes of methods can be beneficial in
discovering different types of design flaws in a project.
10.5. The Context, Users and Tasks: What Clinical UCD Practitioners Should be Aware of
Based on this study and results reported by other researchers [5, 33, 42, 43], there are char-
acteristics of the users, tasks and the context that developers of clinical applications should take
into consideration in order to design a more usable clinical system. These characteristics are
explained more in detail in the following and summarized in Table 9.
10.5.1. Context
A clinical context is a safety-critical context in which errors may lead to patient death. The
context, is a highly interactive context. Interactions such as interactions between clinicians,
clinicians and devices, and finally clinicians and patients exist in this context.
The context is a complex context in which domain-specific knowledge is not easily under-
standable to outsiders such as clinical application developers. Therefore, to understand the con-
text, the concepts in the context and also the clinicians’ workflow, a close collaboration between
developers and domain experts is inevitable in the process of developing a clinical system.
10.5.2. Users’ characteristics
UCD practitioners should be aware of the users’ characteristics in this domain. Clinicians are
non-tech-savvy users of the software systems. These users (especially physicians and specialists)
are very self-confident when it comes to their work knowledge. Also, as mentioned in the pre-
vious section, these users are resistant to changes in their everyday workflow. In addition, they
would like to be in charge of the process of interacting with the computer system in contrast to
the cases in which the system is in charge [5, 33]. For instance, clinicians prefer to make the final
decisions themselves and do not like the system to force an option (e.g. a diagnosis) Clinicians’
experiences have significant effects on how they work, reflect on patient cases, make decisions
and so on. Another characteristic of clinicians is that they are trained and are asked to be highly
respectful to their colleagues [44]. Finally, there is a hierarchy in the clinical domain, and power
relations among clinicians [44].
All of the aforementioned characteristics should be considered in developing a clinical ap-
plication and in applying a UCD process in this context. For instance, it is very important that
clinicians have high self-esteem regarding their work knowledge. But when it comes to UCD
this may be problematic in getting more users to be involved in the process.
Clinicians wonder “why should other clinicians be involved while we know everything about
our job?”. The second aspect, being respectful to colleagues, can result in evaluation sessions in
which clinicians tend not to talk about the GUI problems, since they want to be respectful to other
27
Users Tasks Context
Not tech-savvy Several actors Safety-critical
Self-confident Interruptions Highly interactive
Resistant to change Dependency Workflow
In charge Time constraints Domain specific knowledge
Experience Complex
Booked schedule
In clinical hierarchy
In power relation
Respectful
Table 9: The characteristics of the users, tasks and context in the clinical domain that may have effect on the UCD process
and design.
people involved in the design process specially their colleagues. The clinical hierarchy can also
affect user involvement and evaluations. There are questions regarding UCD to be answered:
how many clinicians should be involved, and who are they? Who should specify them and
what if a clinician at a higher level says “no” to involving more clinicians in the UCD process?
Moreover, there may be cases where since a colleague from a higher level has been involved in
the design, test users hide their real opinion about the GUI design. Finally, clinicians are usually
booked for patient visits which makes their involvement not straightforward and of course costly
for the clinic.
10.5.3. Tasks
When it comes to the tasks, it should be taken into consideration that there are cases in which
more than one actor is involved in a clinical task [42] for instance a nurse and a physician may
collaborate to accomplish one specific task. Also, there are usually various interruptions while
a clinician is accomplishing a task, the interruptions can be as easy as leaving the computer in
order to examine the patient (in case of this project, to examine the patient’s mouth).
There are other aspects of the clinical tasks that are not directly related to our study but
are worth mentioning here. For instance, there are cases in which time is critical for treating
the patient or even saving his/her life. As another example, some clinical tasks may have inter-
dependency. This means that starting one task may trigger establishing a second task or canceling
one may result in abandoning the second task.
Regarding qualitative evaluations recommended to support UCD, the main question is whether
existing evaluation methods are suitable for measuring qualities of a system including such tasks
[42, 43]. For instance, is the result of usability tests reliable when a real clinical setting cannot
be simulated? Are usability tests and other methods able to evaluate a system with time-critical,
or collaborative tasks? Of course, finding a proper answer to these questions needs a broader
investigation of the methods and their application in the clinical context.
11. Conclusion
In theory, human-related factors and issues that are the focus of the HCI research field are
considered to be a main class of success factors in developing CDSSs. Nonetheless, in prac-
28
tice, methods and processes suggested by HCI to develop more usable systems are not a routine
practice in designing and developing CDSSs.
In this project, besides designing a CDSS, the aim was to learn from the experience of ap-
plying a UCD process in a clinical context, and to detect and analyze challenges associate with
it.
As expected from a UCD process, this study also demonstrated that involving users in the
design makes early detection of flaws possible. Even for a team with experiences in develop-
ing clinical applications and usability, evaluations resulted in various changes, and an improved
(more usable) user interface. Without knowing about users and the context of use, and with-
out involving users it was not easy or even possible to find some flaws before implementing the
system, especially those flaws that are related to the domain concepts.
UCD is not a methodology. It does not propose step by step methods, rather it is a process
which is supported by a collection of suggested methods. UCD can be customized and applied
freely to be made more compatible with the project and the team. Not all phases of the UCD
cycle are required to be carried out in each iteration. Moreover, not all of the methods suggested
in the literature should be applied for an individual project. The choice of methods depends on
the nature of the project and decisions made by the team members.
Finally, we observed that there are characteristics of the context, tasks and users that devel-
opers should be aware of when developing a CDSS by applying a UCD approach (Table 9).
12. Future Work
As a continuation to this study, an empirical evaluation should be performed on the last
prototype of the system to make sure that all of the usability goals are satisfied. Afterwards,
more technical focus is required to implement the CDSS including the knowledge representation
and reasoning aspect of the CDSS. After deployment of the system, quantitative methods should
be applied to evaluate the effectiveness of the system and other qualities such as performance.
The findings and discussions presented in this paper are based on only one experience of
applying UCD in a clinical context, and collaborating with one specific group of clinicians.
Surely, the findings can be a starting point for further investigation of the clinical context, its
characteristics and the effects of these characteristics on UCD. Therefore, there is an interest to
peruse a wider investigation of the characteristics of the context, tasks and users, also to identify
their effects on UCD, and finally provide guidelines and suggest improvements to the existing
methods in order to make them more suitable for the clinical context.
Acknowledgment
The author would like to convey her gratitude towards all the following people who partici-
pated in this project either as reviewers, evaluators or test users: Sus Lundgren, Erik Fagerholt,
Martin G E Hjulstr¨
om, Soren Lauesen, Dowen Brikheld, Mike Brennan, Johan Blomgren, Mia
Zellmer, Maria Westin, Jenny Ohman, Gita Gale and Jessica Skoogh.
Sincere thanks go to Mats Jontell, Marie Lindgren, G¨
oran Falkman and Marita Nilsson, the
team members. The author would also like to express her gratitude to Olof Torgersson for his
valuable comments on this paper. This work has been done as a part of the author’s PhD study
under Olof Torgersson’s supervision.
The project was funded by the Swedish Governmental Agency for Innovation Systems (VIN-
NOVA), grant 2006-02792 and is in progress at the time of writing this paper.
29
References
[1] K. Vikram, F. R. Karjodkar, Decision support systems in dental decision making: an introduction., The journal of
evidence-based dental practice 9 (2) (2009) 73–6. doi:10.1016/j.jebdp.2009.03.003.
[2] J. Bennett, P. Glasziou, Computerised reminders and feedback in medication management: a systematic review of
randomised controlled trials, Medical Journal of Australia 178 (5) (2003) 217–222.
[3] R. Walton, S. Dovey, E. Harvey, N. Freemantle, Computer support for determining drug dose: systematic review
and meta-analysis, British Medical Journal 318 (7189) (1999) 984.
[4] D. Hunt, R. Haynes, S. Hanna, K. Smith, Effects of computer-based clinical decision support sys-
tems on physician performance and patient outcomes: a systematic review, Jama 280 (15) (1998) 1339.
doi:10.1001/jama.280.15.1339.
[5] E. Berner, Clinical Decision Support Systems: Theory and Practice (Health Informatics), Springer, New York, NY
10013, USA, 2007.
[6] R. Greenes, Clinical decision support: the road ahead, Academic Press, 2007.
[7] R. Kaushal, K. Shojania, D. Bates, Effects of computerized physician order entry and clinical decision support
systems on medication safety: a systematic review, Archives of Internal Medicine 163 (12) (2003) 1409.
[8] D. Bates, J. Teich, J. Lee, D. Seger, G. Kuperman, N. Ma’Luf, D. Boyle, L. Leape, The impact of computerized
physician order entry on medication error prevention, Journal of the American Medical Informatics Association
6 (4) (1999) 313.
[9] R. N. Shiffman, Y. Liaw, C. a. Brandt, G. J. Corb, Computer-based guideline implementation systems: a systematic
review of functionality and effectiveness., Journal of the American Medical Informatics Association : JAMIA 6 (2)
(1999) 104–14.
[10] T. Wendt, P. Knaup-Gregori, A, Decision Support in Medicine: A Survey of Problems of User Acceptance, Stud
Health Technol Inform 77 (2000) 852–856.
[11] J. Osheroff, J. Teich, B. Middleton, E. Steen, A. Wright, D. Detmer, A roadmap for national action
on clinical decision support, Journal of the American medical informatics association 14 (2) (2007) 141.
doi:10.1197/jamia.M2334.Introduction.
[12] J. Zhang, Human-centered computing in health information systems. Part 1: analysis and design., Journal of
biomedical informatics 38 (1) (2005) 1–3. doi:10.1016/j.jbi.2004.12.002.
[13] T. Beale, S. Heard, openehr architecture overview, http://www.openehr.org/(2009).
[14] H. Kashfi, Applying a user centered design methodology in a clinical context., Studies in health technology and
informatics.
[15] T. Hewett, R. Baecker, S. Card, T. Carey, ACM SIGCHI Curricula for Human-Computer Interaction (1996).
[16] M. G. Helander, T. K. Landauer, P. V. Prabhu, Handbook of Human-Computer Interaction, Elsevier Science Pub
Co, 1998.
[17] ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDTs)Part 11: Guidance on
usability, International Organization for Standardization, Geneva, Swiss, 1998.
[18] M. Maguire, Methods to support human-centred design, International Journal of Human-Computer Studies 55 (4)
(2001) 587–634. doi:10.1006/ijhc.2001.0503.
[19] A. Seffah, E. Metzker, The obstacles and myths of usability and software engineering, Communications of the
ACM 47 (12) (2004) 71–76.
[20] K. Vredenburg, S. Isensee, C. Righi, User-Centered Design: An Integrated Approach, Prentice Hall PTR, Upper
Saddle River, NJ, 2002.
[21] ISO 13407, Human-Centred Design Process for Interactive Systems, International Organization for Standardiza-
tion, Geneva, Swiss, 1999.
[22] User-Centered Design, Website, https://www-01.ibm.com/software/ucd/ucd.html (2010).
[23] B. Kaplan, Evaluating informatics applications–clinical decision support systems literature review. (November
2001).
[24] S. Porter, An update of the etiology and management of xerostomia, Oral Surgery, Oral Medicine, Oral Pathology,
Oral Radiology & Endodontics 97 (1) (2004) 28–46.
[25] H. Sharp, Y. Rogers, J. Preece, Interaction Design: Beyond Human-Computer Interaction, Wiley, 2007.
[26] M. G. Helander, T. K. Landauer, P. V. Prabhu (Eds.), Handbook of Human-Computer Interaction, Elsevier Science
Inc., New York, NY, USA, 1997.
[27] J. Preece, Y. Rogeres, D. Benyon, S. Holland, T. Carey, Human-computer Interaction, Addison-Wesley, 1994.
[28] A. Dix, J. Finlay, G. Abowd, R. Beale, Human-computer interaction, Prentice-Hall, Inc., Upper Saddle River, NJ,
USA, 1997.
[29] T4 practice management software , Website, http://eamer.carestreamdental.com/en-gb/sweden/
practice-management-systems/1kodak-t4.aspx (2010).
[30] M. Jontell, U. Mattsson, O. Torgersson, MedView: an instrument for clinical research and education in oral
30
medicine., Oral surgery, oral medicine, oral pathology, oral radiology, and endodontics 99 (2005) 55–63.
doi:10.1016/j.tripleo.2004.01.009.
[31] J. Tidwell, Designing Interfaces, O’Reilly, Beijing, 2006.
[32] J. Gulliksen, B. G¨
oransson, I. Boivie, S. Blomkvist, J. Persson, A. s. Cajander, Key principles for user-centred
systems design, Behaviour & Information Technology 22 (2003) 397–409. doi:10.1080/01449290310001624329.
[33] R. A. K. Horasani, M. I. T. Anasijevic, B. L. M. Iddleton, M. S. C, Ten Commandments for Effective Clinical
Decision Support: Making the Practice of Evidence-based Medicine a Reality, Journal of the American Medical
Informatics Association 10 (2003) 523–530. doi:10.1197/jamia.M1370.Although.
[34] A. Garg, N. Adhikari, H. McDonald, M. Rosas, Effects of computerized clinical decision support systems on
practitioner performance and patient outcomes: a systematic review, JAMA 293 (10) (2005) 1223–1238.
[35] Y. Y. Han, J. a. Carcillo, S. T. Venkataraman, R. S. B. Clark, R. S. Watson, T. C. Nguyen, H. Bayir, R. a. Orr,
Unexpected increased mortality after implementation of a commercially sold computerized physician order entry
system., Pediatrics 116 (6) (2005) 1506–12. doi:10.1542/peds.2005-1287.
[36] S. T. Rosenbloom, F. E. Harrell, C. U. Lehmann, J. H. Schneider, S. A. Spooner, K. B. Johnson, Perceived increase
in mortality after process and policy changes implemented with computerized physician order entry., Pediatrics
117 (4) (2006) 1452–5; author reply 1455–6. doi:10.1542/peds.2005-3163.
[37] A. Cooper, R. Reimann, D. Cronin, About Face 3: The Essentials of Interaction Design, Wiley, Indianapolis, IN,
USA, 2007.
[38] R. Koppel, J. P. Metlay, A. Cohen, B. Abaluck, a. R. Localio, S. E. Kimmel, B. L. Strom, Role of computerized
physician order entry systems in facilitating medication errors., JAMA : the journal of the American Medical
Association 293 (10) (2005) 1197–203. doi:10.1001/jama.293.10.1197.
[39] C. Gadd, P. Baskaran, D. Lobach, Identification of design features to enhance utilization and acceptance of systems
for Internet-based decision support at the point of care., in: Proceedings of the AMIA, 1998, pp. 91–5.
[40] M. Peleg, A. Shachak, D. Wang, E. Karnieli, Using multi-perspective methodologies to study users’ interactions
with the prototype front end of a guideline-based decision support system for diabetic foot care., International
journal of medical informatics 78 (7) (2009) 482–93. doi:10.1016/j.ijmedinf.2009.02.008.
[41] a. Narasimhadevara, T. Radhakrishnan, B. Leung, R. Jayakumar, On designing a usable interactive system to
support transplant nursing., Journal of biomedical informatics 41 (1) (2008) 137–51. doi:10.1016/j.jbi.2007.03.006.
[42] P. Edwards, K. Moloney, J. Jacko, F. Sainfort, Evaluating usability of a commercial electronic health record: A case
study, International Journal of Human-Computer Studies 66 (10) (2008) 718–728. doi:10.1016/j.ijhcs.2008.06.002.
[43] M. W. M. Jaspers, A comparison of usability methods for testing interactive health technologies: method-
ological aspects and empirical evidence., International journal of medical informatics 78 (5) (2009) 340–53.
doi:10.1016/j.ijmedinf.2008.10.002.
[44] C. Delany, D. Watkin, A study of critical reflection in health professional education: ’learning where oth-
ers are coming from’., Advances in health sciences education : theory and practice 14 (3) (2009) 411–29.
doi:10.1007/s10459-008-9128-0.
31
Paper V
Supporting open EHR Java Desktop Application
Developers
Hajar Kashfi, Olof Torgersson
The XXIII International Conference of the European Federation for Medical
Informatics, Proceedings of Medical Informatics in a United and Healthy
Europe (MIE2011), Oslo, Norway, 28-31 August, 2011.
(in print)
139
Supporting openEHR Java Desktop
Application Developers
Hajar KASHFI1 and Olof TORGERSSON
Department of Applied Information Technology, Chalmers University of Technology
and University of Gothenburg,
SE-412 96, Gothenburg, Sweden
Abstract. The openEHR community suggests that an appropriate approach for
creating a graphical user interface for an openEHR-based application is to generate
forms from the underlying archetypes and templates. However, current generation
techniques are not mature enough to be able to produce high quality interfaces
with good usability. Therefore, developing efficient ways to combine manually
designed and developed interfaces to openEHR backends is an interesting
alternative. In this study, a framework for binding a pre-designed graphical user
interface to an openEHR-based backend is proposed. The proposed framework
contributes to the set of options available for developers. In particular we believe
that the approach of combining user interface components with an openEHR
backend in the proposed way might be useful in situations where the quality of the
user interface is essential and for creating small scale and experimental systems.
Keywords. openEHR, clinical application, opereffa, application development
framework, data binding.
Introduction
With the growing presence of electronic healthcare record (EHR) systems developed
by various vendors, it becomes increasingly important to agree upon standards that can
overcome the resulting interoperability problems [1]. The goal of the openEHR
initiative is to develop an open standard that can serve as the basis for both developing
EHR systems and to guarantee semantic interoperability between systems [2].
openEHR suggests a two-level architecture for managing data in clinical applications.
The top-level is made up of archetypes, which are domain specific models that should
be developed by domain experts. The lower level is the openEHR reference model
(RM), which is a very generic model for managing clinical data [2].
Regardless of the advantages offered by openEHR, the standard suffers from a
rather low adoption rate. While this is not the place to make a proper analysis of the
reasons for this, some possible reasons are the complexity of the standard, lack of
documentation and training for developers, and a limited set of tools and frameworks
available to ease application development. Most of the focus of the community seems
to have gone into representing and modelling domain concepts and perfecting the
specifications. However, to make openEHR more practical, application developers need
to be supported with APIs, frameworks and tools." Of course, a number of application
development projects exist. Some of the more mature are: the open source health
information platform [3] (OSHIP), the open EHR-Gen framework [4], GastrOS [5],
1 Hajar Kashfi, Department of Applied Information Technology, Chalmers University of Technology,
SE–412 96 Gothenburg, Sweden; E-mail: hajar.kashfi@chalmers.se.
2
and the openEHR reference framework and application (opereffa) [6].
Different approaches can be used to develop openEHR applications. The typical
approach is to automatically generate the graphical user interface (GUI) from openEHR
archetypes/templates. To our knowledge, current openEHR frameworks and tools are
compatible with this model (depicted in Figure 1-A). The idea is that clinicians design
and create archetypes (and templates) using existing tools. Based on these, a GUI or
GUI artefacts are generated from the given archetypes/templates. To improve the GUI
design, manual adjustment of the GUI or its style files is often required. The typical
openEHR application is web based."
In contrast, there is an approach in which no generation of GUI based on
archetypes is done but instead the interface is designed by experts based on the
demands of the users of the application. There is then a need to connect this user
interface to the archetypes designed in parallel by domain experts. Unfortunately, the
current frameworks do not provide enough support for this type of application develop-
ment. To fill this gap, we have developed an extension to one of the existing openEHR
frameworks to help developers easily connect a user interface created by a GUI-
designer to an openEHR based backend.
Figure 1."The two development models. The left side model is supported by opereffa.
1. Methods and Tools
As mentioned in the previous section, in contrast to the automatic or semi-
automatic approach to creating GUIs to openEHR-based applications, there is an
alternative illustrated in Figure 1-B. To support this approach, there is a need for a
simple and efficient data-binding framework connecting an application’s frontend to its
backend (or the business logic). Usually, the idea of data binding is that when the data
changes, the changes are reflected automatically by the bound elements on the user
interface. In the same manner, if the “outer representation” of data changes, then the
Archetype
Archetype
Archetype
Template
Template
Template
translated to
GUI artefacts
GUI
......
...... User
RM
object
RM
object
RM
object
RM
object
RM
object
AOM
object
validated by
AOM instances
openEHR-based
EHR
Data
Developer
Manual
adjustment
expert
create
Archetype
Archetype
Archetype
User
RM
object
RM
object
AOM
object
validated by
AOM instances
openEHR-based
EHR
Developer
create deisgn
is mapped to GUI
......
......
RM
object
RM
object
RM
object
Data
data binding
expert
A B
3
corresponding data in backend should be automatically updated as well to reflect the
change [7].
We have developed an extension, a Java desktop user interface data-binding layer,
to one of the openEHR application development frameworks namely opereffa
(compare the left and right sides of the Figure 2). opereffa is an initiative for creating
an open source clinical application and is built on top of a Java-based open source
framework. Services such as GUI generation, and persistence are supported in opereffa.
opereffa generates web-based user interface artefacts using Java Server Faces (JSF) [8]
and also makes use of the data binding capabilities of JSF.
Java open source
RM implementation
Java open source
framework
Persistent (PostgreSQL)
Automatic GUI generation
etc.
Open source
clinical Application
Web based GUI (JSF)
Document
Document
Archetype
openEHR-based
EHR
PostgreSQL
ADL parser
AOM classes
RM classes
'Data validation
etc.
Medical application
Java open source
RM implementation
Java open source
framework
User interface data binding
Figure 2."The proposed architecture.
2. Results
The new data-binding layer on top of opereffa provides a single entry point for
connecting Java desktop GUIs to an archetype-enabled backend. To make the desktop
connection framework as easy to use as possible for application developers, it is
designed using a facade pattern [9]. This means that there is one single class,
ArchetypeDatahandler, which hides all the details of archetype-based data management
and therefore is the only class someone using the framework needs to know anything
about. To further improve ease of use in applications, the ArchetypeDataHandler is a
singleton, meaning that there is only one instance of it in an application and it can
easily be accessed anywhere. To connect a pre-designed GUI to the openEHR backend
the only steps required are: (i) creating an xml-file specifying the connection between
GUI components and archetypes (ii) adding each component in the GUI to the
ArchetypeDataHandler, i.e. calling the method add (JComponent c) on the
ArchetypeDataHandler.
Underlying the implementation is the assumption that each GUI component is
logically in relation to one and only one item in an archetype (an item in an archetype
can be visually presented in several places on a screen though). This means that for
each component there exists an archetype name, and a unique path for the item in that
archetype. This information is stored in an xml file, which is used by the
4
ArchetypeDataHandler to decide to which data item a certain GUI component should
be connected. A sample of entries in the xml file is shown in Figure 3.
Internally, a class named GUIMapper with support from other mapper classes
provides the actual functionality. The role of this class is to keep track of all GUI
components, their archetypeWrappers and the path of the item in the archetype to
which each component is mapped. The synchronization of data shown in the GUI and
what is stored in the GUIMapper is realized using various listeners. If data changes on
the GUI this change is reflected in the GUIMapper in the backend. On the other hand,
if data is loaded from the DB, the GUIMapper updates the GUI as well. All data is kept
in memory to be persisted at the right time using opereffa persistence services.
To evaluate the functionality of the data-binding framework, a demo application
was created using the graphical GUI-editor of NetBeans. In the demo application, four
different archetypes are used and connected to 39 GUI components. The archetypes are
both special archetypes developed for a decision support system for Xerostomia [10]
and adaptations of common archetypes such as openEHR-EHR-OBSERVATION.lab_
test.v1. A snapshot of the demo application is depicted in Figure 4. The screen shows a
field where one can search for a patient, a list where one can select one of the patient’s
recent sessions and the data recorded at that session. The part of the code that enables it
to load and store archetyped data is only a few lines that add components to the
ArchetypeDataHandler. The framework provides the rest.
Figure 3."Sample entries showing the connection between GUI components and archetypes"
3. Discussion and Conclusion
The proposed framework for binding a pre-designed GUI to an openEHR-based
backend contributes to the set of options available for developers. We particularly
believe that the approach of combining GUI components with an openEHR backend in
the proposed way might be useful for various small scale and experimental systems as
well as systems where the quality of the user interface is of great importance.
The generic proposed model for building openEHR applications was illustrated in
Figure 1-A. While generating GUIs from archetypes and templates has advantages
when it comes to maintenance of large-scale systems, creating good, and not just
mediocre, user interfaces from underlying domain models is a non-trivial problem.
During the time mature generation techniques are being developed, hooking up
designed GUIs to openEHR backends is an interesting alternative. The use of designed
GUIs is also interesting from another point of view. GUIs based on a domain model are
by necessity rather close to the implementation model of the system. However, when a
GUI is designed, the goal of the developers should always be to have an on-screen
representation that is as close as possible to the users’ mental model [11]. Therefore,
having a simple way of replacing parts of a system with GUIs designed by hand in
accordance with the users’ mental models is valuable to handle the situations where
generation cannot produce an appropriate solution.
5
The main disadvantage of the approach used in this paper is maintenance. A gen-
erated GUI can change automatically if the underlying archetypes/templates change
whereas a manually created GUI has to be updated manually. A related problem is that
in complex systems the number of GUI components that need to be connected to the
backend will be very large. The current implementation is rather limited since it only
supports a small number of GUI components. A more complete implementation must
of course support all commonly used standard components and perhaps even provide
some special controls tailored towards clinical applications. This however is work for
the future.
Figure"4."The demo application
References
[1] Schloeffel P, Beale T, Hayworth G, Heard S, Leslie H. The relationship between CEN 13606, HL7, and
openEHR. In: HIC (2006). Health Informatics Society of Australia; 2006. p. 24.
[2] Beale T, Heard S. openEHR Architecture Overview [published on the Internet]. openEHR Foundation;
2009 [cited 2011 April 20]. Available from: http://www.openehr.org
[3] Open Source Health Information Platform (OSHIP) [homepage on the Internet]. Multi Level Healthcare
Information Modelling - MLHIM [cited 2011 April 20]. Available from: http://www.oship.org
[4] Open-EHR-Gen [homepage on the Internet]. Available from: http://code.google.com/p/open-ehr-gen-
framework
[5] GastrOS [homepage on the Internet]. The University of Auckland [cited 2011 April 20]. Available from:
http://sourceforge.net/projects/gastros
[6] openEHR REFerence Framework and Application (Opereffa) [homepage on the Internet]. UCL [cited
2011 April 20]. Available from: http://opereffa.chime.ucl.ac.uk/introduction.jsf
[7] Data Binding Overview [webpage on the Internet]. Microsoft Corporation [cited 2011 April 20].
Available from: 2010. http://msdn.microsoft.com/en-us/library/ms752347.aspx
[8] Burns E, Griffin N, Schalk C. JavaServer faces 2.0: the complete reference. McGraw-Hill Professional;
2009.
[9] Stelting S, Maassen O. Applied Java patterns. USA: Sun Microsystems Inc.; 2002.
[10] Kashfi H. Applying a user centered design methodology in a clinical context. Studies in health
technology and informatics. 2010 Jan ;160(Pt 2):927-31.
[11] Cooper A, Reimann R, Cronin D. About face 3: the essentials of interaction design. Wiley-India; 2007.
Paper VI
Towards a Case-Based Reasoning Method for
open EHR-Based Clinical Decision Support
Hajar Kashfi, Jairo Robledo Jr.
Proceedings of The 3rd International Workshop on Knowledge Representation
for Health Care (KR4HC’11), Bled, Slovenia, 6 July, 2011.
(in print)
147
Towards a Case-Based Reasoning Method
for open EHR-Based Clinical Decision Support
Hajar Kashfi and Jairo Robledo Jr.
1Department of Applied Information Technology
Chalmers University of Technology
SE–412 96, Gothenburg, Sweden
hajar.kashfi@chalmers.se
2Department of Oral Medicine and Pathology
Institute of Odontology
Sahlgrenska Academy
University of Gothenburg
SE–405 30, Gothenburg, Sweden
jrobledo@uces.edu.co
Abstract. In 2007, a team of informaticians and specialists in dentistry
in Sweden started a project to develop a CDSS based on openEHR for an
oral disease named dry mouth. Since openEHR is an emerging standard,
designing a CDSS based on it is an unexplored research area. According
to our findings, so far, very rare (almost none) open EHR-based CDSS
has been released. The methodological approach applied in developing an
openEHR-based CDSS is presented in this paper. This includes typical
activities in developing CDSSs in addition to the activities one needs
to carry out in order to develop an openEHR-based system. In the first
phase of this project, the focus has been on openEHR archetype design,
knowledge acquisition, and choosing a suitable KRR method based on
the available legacy patient records, i.e. a knowledge intensive case-based
reasoning method, and the extracted general domain knowledge. We also
propose an architecture for such a system with the aim of benefiting from
the structured openEHR-based patient data in reasoning.
Key words: Clinical decision support, openEHR, archetype, Case-based
reasoning
1 Introduction
Presently, the use of computerized approaches to improve quality of health care is
widespread in the clinical domain. Electronic health records (EHR) and clinical
decision support systems (CDSS) are two complementary approaches to improve
quality of health care. One of the success factors of CDSSs is observed to be
their integration into EHRs [1–5] and since there are various international EHR
standards (such as openEHR) being developed, it is important to take these
standards into consideration while developing CDSSs [6].
2 Towards a CBR Method for an openEHR-Based CDSS
Developing CDSSs involves challenges such as representing clinical knowl-
edge, keeping it updated and performing the reasoning [6, 4, 3, 5]. At the time
of introducing various EHR standards and calls for moving from standalone
CDSSs towards CDSSs that are integrated into EHR system [6, 4, 3, 5], develop-
ers should adopt new approaches in developing such systems. This is of course
an interesting research problem to see how the presence of these standards may
influence developing CDSSs.
The paper is structured as follows. The background information about openEHR
and the oral disease for which the CDSS is going to be developed is given in Sec-
tion 2. Methods and materials applied in this study for designing an openEHR-
based CDSS are presented in Section 3. This includes the activities carried out in
this methodological approach. Results of the activities are given in Section 4.A
discussion is provided in Section 5. Finally, we end with a conclusion and the
future directions of the study in Section 6
2 Background
2.1 The openEHR Approach
openEHR is a an open specification standard for managing electronic health
records (EHR) so that the problem of shareability and computability of infor-
mation in the clinical domain is overcome [7].
The openEHR approach emphasizes the role of clinicians in organizing do-
main knowledge in the form of different clinical concepts such as observation,
evaluation, instruction and action [7]. This approach suggests a two layer ar-
chitecture for clinical applications to separate knowledge and information layers
in order to overcome the ever-changing nature of clinical knowledge. Patient
data is stored in a generic form which is retrievable in heterogeneous clinical
applications based on some constraints named archetype. An archetype which is
designed by domain experts defines some constraints on data in terms of types,
values, relation of different items and so on [7]. Archetypes are used for data
validation and sharing [7].
Very few methodological approaches are documented to guide developing
archetypes or openEHR-based applications. A well-known methodological ap-
proach for developing archetypes is the one presented by Leslie et al. [8] which
includes these steps: (i) Identifying clinical concepts (ii) Identifying existing
archetypes (iii) Creating new archetypes if necessary.
2.2 Dry Mouth
Dry mouth is “the abnormal reduction of saliva and can be a symptom of certain
diseases or be an adverse effect of certain medications” [9]. There are various
causes for dry mouth from, of which certain previous treatments on the patient,
drugs and diseases can be mentioned. Dry mouth is typically managed with saliva
substitutes, yet these days, clinicians can find more systematic treatments for this
Towards a CBR Method for an openEHR-Based CDSS 3
disease [9]. A number of specialists in dentistry from The Sahlgrenska Academy
at Gothenburg University3proposed a need for getting an automatic support for
diagnosis and treatment of this disease. In their opinion, many general dentists
are not aware of systematic therapies available for dry mouth, and it is not so
easy for them to find potential causes for the disease in order to administer the
optimal treatment.
3 Methods and Materials
In this phase of the project, the focus has been on openEHR archetype design,
knowledge acquisition, and choosing a suitable knowledge representation and
reasoning (KRR) method, based on the available legacy patient records and the
available external domain knowledge. Details of the activities and their outputs
are discussed below.
3.1 Preparing the Assessment Questionnaire
Our domain experts were not able to independently develop the archetypes as
suggested by Leslie et al. [8], especially since they were new to openEHR. Hence,
we decided to use a different approach that suited our domain experts. As part
of their everyday work, specialists in dentistry at Sahlgrenska have access to an
online application named mForm which provides them with facilities to develop
their own examination forms (assessment questionnaires). These forms act as
data entry interfaces for the clinical system available for patient data gathering
in Sahlgrenska. Hence, our domain experts were already familiar with the concept
of independently developing their own examination forms. This fact, initiated our
approach for developing openEHR archetypes based on clinical questionnaires.
3.2 Domain Concept Modeling and Designing openEHR archetypes
The questionnaire created in the previous step, was used as a basis to produce
domain concept models and finally archetypes. Domain concept diagrams were
used mainly for communicating the concepts and the relation between them.
These models were created using a mind-mapping tool4in collaboration be-
tween domain experts and informaticians. In our approach, brainstorming ses-
sions were held with domain experts to iteratively prepare the domain concept
diagrams. Finally the domain concept models were used by informaticians to
create archetypes. This was done using the available openEHR tools.
3.3 Domain Knowledge Gathering and Related Patient Data
At first, we held interviews with domain experts and also studied the related
material to find out more about dry mouth, and related concepts. However, we
3http://www.sahlgrenska.gu.se/english
4http://xmind.org
4 Towards a CBR Method for an openEHR-Based CDSS
Fig. 1. A part of the domain concept model diagram
soon found out that, as informaticians, we would not be able to efficiently extract
knowledge from existing evidences. Therefore, in parallel to domain concept
modeling and archetype creation, a domain expert, was asked to gather such
information.
For this purpose, PubMed5was searched for the following terms: (”xerosto-
mia” OR ”dry mouth” OR ”Sj¨ogren’s Syndrome”) AND (”therapy” OR ”treat-
ment”). Also individual terms were combined, e.g. ”xerostomia” AND ”treat-
ment”; ”xerostomia” AND ”therapy”; etc.
Moreover, the main dentistry database at Sahlrgenska was also searched by
domain experts to find dry mouth related patient cases.
3.4 Selecting the KRR Method
In this project, based on the amount of legacy patient records and domain knowl-
edge available, and some other motivations (see Section 4.4), we chose a knowl-
edge intensive case-based reasoning (CBR) as the knowledge representation and
reasoning method for the CDSS.
Case-based Reasoning One of the reasoning methods which has been used
in the clinical domain is case-based reasoning (CBR) [10, 11]. Begum et al. [11]
explain CBR and its application in the clinical domain as follows: “The CBR is
inspired by human reasoning, i.e., solving a new problem by applying previous
experiences adapted to the current situation. A case (an episodic experience)
normally contains a problem, a solution, and its result. The CBR is an appro-
priate method to explore in a medical context where symptoms represent the
5http://www.pubmed.gov
Towards a CBR Method for an openEHR-Based CDSS 5
problem, and diagnosis and treatment represent the solution”. The CBR cycle
that includes retrieve, reuse, revise and retain [12] is depicted in Figure 2.
In this method, the solution to previous problems is adapted to the new
problem. Cases and indexing information are stored in the knowledge-base and
reasoning is carried out by doing indexing, matching and adapting and storing
new cases. Indexing efficiency is a key issue in this reasoning method [3, 13].
Case-based reasoning may be considered to be a data intensive method,
since it starts with a set of cases for training. However, it is very different from
other data intensive approaches. The knowledge that is extracted from experts
in knowledge-intensive methods, can be subjective. On the other hand, if one
only relied on objective knowledge e.g. the knowledge extracted from evidence,
the valuable experience of experts in domain is not used. CBR uses both ob-
jective and subjective knowledge for reasoning. In other words, reasoning from
previous cases (in our project patient data) is done in this method. Knowledge is
recorded in form of cases. In the cased-based knowledge-base, cases and indexing
information are recorded [3] .
CBR is considered to be a suitable method to be used in CDSSs specially
in the clinical domain [10, 11]. The concept of case is a concept that is used
in medicine as well as for training and discussing treatment of an individual
patient, moreover clinical guidelines include practice cases [10].
One class of CBR methods is a hybrid approach called knowledge intensive-
CBR where the reasoning process is enhanced by benefiting from reasoning on
the existing general knowledge. (note the “general knowledge” close to the “pre-
vious cases” in Figure 2).
Fig. 2. The knowledge intensive-CBR cycle, taken from [12].
6 Towards a CBR Method for an openEHR-Based CDSS
3.5 Creating Artificial Cases
In a need for generating some dry mouth patient cases, we used the assessment
questionnaires generated before, to develop data entry forms using the mForm
application. Manual patient entry was done by one domain expert with the aim
of creating some cases to be used in the reasoning process.
3.6 Proposing an Architecture Based on Two Existing Frameworks
A framework is a set of classes and design solutions and one can see it as a
partial design and implementation of an application [14]. Using a framework,
a huge amount of development time can be saved in a project. For developing
an openEHR-base CDSS it is most efficient to use the existing frameworks to
benefit from the services provided for managing the openEHR-based patient
records. This is the same for CBR. Therefore, this CDSS is going to be built on
top of two frameworks namely opereffa (an openEHR application development
framework) and JColibri (a CBR application development framework).
opereffa [15] is one of the existing frameworks for developing openEHR-based
applications. The framework manages tasks needed for loading archetypes, vali-
dating them and storing data, based on the openEHR reference model. JColibri
[14] is a framework for developing CBR applications. It includes basic functions
and algorithms needed in a CBR application.
4 Results
The workflow of the activities in the first phase of the project are depicted in
Figure 4. As shown in this Figure, the outputs of the activities are: the question-
naire, the domain concept models, the archetypes, general domain knowledge,
and artificial dry mouth patient cases. Also, based on the gathered information
(general domain knowledge, and the available legacy patient records) a KRR
method was selected for the CDSS namely knwoledge-intensive CBR.
4.1 The Questionnaire, Concept Models and Archetypes
The questions covered various aspects of patient data including history of other
diseases and drugs, related lab results, diet habits, age and sex. The questionnaire
consisted of 6 main sections and a total of 41 questions and 15 related laboratory
tests. The questionnaire was created by one domain expert and shared with 4
more domain experts to be revised and approved.
Based on the generated questionnaire a domain concept model was cre-
ated. The model was created in close collaboration with clinicians. Various con-
cepts/sections in the questionnaire were mapped to the related openEHR gen-
eral classes such as observation and evaluation. A sample of the domain concept
model is depicted in Figure 1.
The model was later translated to 23 openEHR archetypes by informaticians.
Before creating the archetypes, the existing shared openEHR archetypes were
Towards a CBR Method for an openEHR-Based CDSS 7
Fig. 3. The methodological approach in developing an openEHR-based clinical decision
support system.
searched to find reusable archetypes. Around 10 archetypes were reused and
almost all were modified for this purpose.
In addition, having a close collaboration with clinicians while developing
these models, the interviews we held with some external domain experts provided
an opportunity for us to gather some basic knowledge that a clinician uses in
diagnosis and treatment of dry mouth. Furthermore, some external resources
were also studies to gather more general information regarding dry mouth.
4.2 General Domain Knowledge
More than 6000 references were obtained from the searches; only review articles
were used due to their scientific evidence level. From a list of around 1000 arti-
cles, 71 were selected. Papers describing treatment strategies in xerostomia, dry
mouth or Sj¨ogren’s Syndrome were the main objective of the search.
No guidelines for the treatment of any of these diseases were found after this
search. This suggests that there is no global agreement about treatment strate-
gies in xerostomia, dry mouth, or oral manifestations of Sj¨ogren’s Syndrome.
However, there are several papers with a high level of scientific evidence about
8 Towards a CBR Method for an openEHR-Based CDSS
different therapy methods for patients with these diseases, such as systematic
reviews or meta-analysis, where some statements can be drawn from.
In contrast to dry mouth and Sj¨ogren’s Syndrome, several treatment guide-
lines and global treatment consensuses can be found for many other medical
conditions. Generally speaking, it can be said that even if there are a lot of pa-
pers in the literature about treatment or therapies in xerostomia, dry mouth or
oral manifestations of Sj¨ogren’s Syndrome, this information is far less compared
to other common diseases.
There is a global agreement in the literature that xerostomia is a common
and significant (or maybe the most common) side effect of many commonly
prescribed drugs. However, according to the literature, it is difficult to establish
relative incidence rates for xerostomia for a particular medication. This happens
because reported rates depend on how the professional collects the information
from the patient, how patients react to a specific kind of drug, the type of drugs
being taken, the cause for which the drug is being taken, the possible presence
of contributing factors (such as Sj¨ogren’s Syndrome, radiation therapy, or other
immunological diseases) to the disease, the dose of the medication, etc. So, as
an example, it is not possible to say that xerostomia will be present if a patient
is taking 3 drugs or a specific amount of drugs; neither about the percentage of
patients who will present with xerostomia if taking 3 or more drugs. Nevertheless,
the risk for xerostomia increases with the number of drugs being taken. Some
studies describe a list of medicaments (over 500) that may cause xerostomia,
and those drugs listed have been reported to cause xerostomia in 10% or more
of patients.
4.3 Legacy Patient Records
The main dentistry database at Sahlrenska contains more than 20 000 patient
records and images. Nevertheless, there are only around 100 dry mouth related
cases in the database. According to our domain experts, the information related
to dry mouth is missing for most of the patient cases, especially information
related to diagnosis and treatment of the disease. One might be able to extract
most of the historical information of patients from the available repositories
but still other vital information is still missing. In other words, there are not a
reasonable number of high quality (complete) dry mouth cases in the repository.
As a solution to this problem, a number of artificial cases were created by
the domain experts to improve the process of CBR.
4.4 Knowledge Representation and Reasoning
There are two main classes of choices for selecting a KRR methods, i.e. data
intensive methods and knowledge intensive methods. Answers to the following
questions would help in choosing the suitable method in a specific project.
–Do we have enough data to be used in data intensive methods?
Towards a CBR Method for an openEHR-Based CDSS 9
–Do we have enough domain experts to extract the knowledge or sufficient
structured domain knowledge in order to adopt a knowledge intensive method?
Unfortunately in this project, the answer to the both the questions was “no”.
Yet, as a result of previous activities we had some basic knowledge that, though
not sufficient for adopting knowledge intensive method, could be used for simple
rule-based reasoning. Additionally, the domain experts involved in the project
indicated that, while they are not able to provide us with probability numbers or
exact rules in how to treat dry mouth, they can create some dry mouth patient
cases. This yielded in a decision for adopting a knowledge intensive-CBR for the
dry mouth CDSS.
Applying a hybrid KRR method is recommended in CDSSs, and for this
project at the moment, the only other approach that is possible to be applied
along with the CBR method, is a rule-based approach. As illustrated in Figure
2, general knowledge is used in CBR to improve the reasoning process. This
general knowledge can be represented in many ways, one of which is a set of
rules. With a rather small set of rules (general domain knowledge) generated as
a result of previous activities, we can support the CBR method we selected and
benefit from a hybrid method. The reasons for selecting the knowledge-intensive
CBR method is summarized in table 1.
Two domain experts were responsible for creating the cases. For this purpose
the aforementioned questionnaire was used to create an online data entry form
using the tools available at the clinic. Total of 14 cases were created at this stage.
4.5 Archetypes s versus Domain Knowledge
In the dry mouth domain, a sample of general knowledge would be: People who
use 3 or more drugs at the same time usually get dry mouth. This kind of
information cannot be extracted from archetypes, but can be extracted from
literature, or gained from the clinicians’ mind, that is why as result of domain
knowledge gathering and domain concept modeling activities we generated some
general domain knowledge.
Based on the openEHR approach, clinicians would be responsible for cre-
ating archetypes even though our experience revealed that sometimes it is too
optimistic to think that this task is done only by a group of clinicians. Espe-
cially in cases where we plan to use archetype data for clinical decision support.
This means that one should be aware of the fact, that beside attributes (items
in archetypes) one needs some knowledge to be used in the CBR. As shown
in Figure 4, the general knowledge was extracted from literature and from the
meetings with the domain experts (the meetings were basically held for design-
ing the archetypes). This information was added to the models that were created
for domain knowledge as descriptions for each item.
4.6 The software architecture proposal
A layered architecture is proposed for this application. As depicted in Figure 5,
the top layer is the view layer or in other words the user interface. Below that,
10 Towards a CBR Method for an openEHR-Based CDSS
Fig. 4. A screen shot from the online form for case creation.
there is a mapper layer which is responsible for mapping the GUI components to
the opereffa framework classes, also to connect them to the JColibri framework.
Automatic generation of cases out of archetypes will be done in this layer. JCol-
ibri manages the CBR process and will have access to patient data repository.
This repository will be shared between JColibri and opereffa. Patient data is
stored in the database based on openEHR reference model. Each case/patient
data will be stored using the related archetype that acts as a constraint on data.
All the processes related to openEHR is managed by opereffa. JColibri and oper-
effa layers are not aware of each other, and everything between them is managed
via the mapper layer. So far, part of the mapper layer which is responsible for
mapping the view layer to the openEHR underlying framework is implemented
and tested. The implementation of the whole application is still in progress.
5 Discussion
As mentioned before, there are very few documented methodological approaches
to guide developing archetypes or openEHR-based applications. This includes
those published by Marcos et al. [16] and Leslie et al. [8]. The most well-known
methodological approach for developing archetypes is the one presented by Leslie
Towards a CBR Method for an openEHR-Based CDSS 11
openEHR- based
EHR repository
JColibri CBR
framework
opereffa openEHR
framework
Mapper Layer
View Layer ( GUI)
Fig. 5. The proposed architecture.
et. al. [8]. This approach however emphasizes the role of clinicians and how they
may apply existing openEHR tools to browse existing archetypes or developing
new ones. Our approach is different in its view on adopting more traditional
clinical approaches for gathering information such as clinical questionnaires.
Additionally, when it comes to CDSSs, there are a few studies that deal with
how openEHR offers opportunities for CDS. Most of these efforts however seem
to be more focused on integrating clinical guidelines into openEHR archetypes
or utilizing archetypes for representing clinical guidelines [16–18] or to integrate
reasoning and clinical archetypes (enhance archetypes by including knowledge
representation capabilities to them) [19]. To our knowledge there is almost no
study that has been focused on benefiting from the well-structured openEHR-
based patient data for adopting data intensive reasoning methods in CDSSs or
methods such as CBR that rely on previous cases to carry out the reasoning
process.
As this study shows, because of the openEHR novelty, it is likely that in many
projects, clinicians are more familiar with the traditional clinical concepts such
as clinical questionnaires compared to the new and complex openEHR concepts
such as archetypes. Therefore, other approaches for developing archetypes can be
applied that are more compatible with the capabilities of the clinicians involved
in the project.
Moreover in case of developing a CDSS, the activities in developing archetypes
can be in form of more informed interviews and/or brainstorming sessions with
domain experts where not only domain concept models and eventually archetypes
are generated but also the available general domain can be extracted from clini-
cians’ minds. The well-defined concepts in openEHR help provide opportunities
for informaticians to use these concepts for knowledge extraction in this manner.
5.1 Why Case-based Reasoning?
As mentioned in previous sections, based on our investigation, CBR is the most
suitable KRR method to be used for this CDSS not only because of the advan-
tages and practicability of this method for this project considering the amount of
data and knowledge available, but also for the similarity found between openEHR
archetypes and cases in CBR.
12 Towards a CBR Method for an openEHR-Based CDSS
Case description is analogous to archetypes in openEHR as discussed in Sec-
tion 5.2. This approach did not require explicit knowledge to be represented and
also the reasoning process is not a black box and can be understood by users.
Therefore, the developed system can be used for training dentistry students as
mentioned before.
The Table 1 shows a brief comparison between the selected method and other
existing approaches.
Method
Criteria
Data Knowledge Knowledge
Intensive Intensive intensive-CBR
No need for deep domain knowledge 3 7 3
No need for huge data volume 7 3 3
Easy Knowledge Acquisition 3 7 3
Objective Knowledge 3 7 3
Subjective Knowledge 7 3 3
Easy to maintain 3 7 3
Use of past experiences 3 7 3
Suitable for Education 7 7 3
Table 1. Motivations for selecting the knowledge intensive-CBR method.
5.2 openEHR archetypes and cases in CBR
As depicted in Figure 2, two types of knowledge are applied in a CBR cy-
cle, domain-dependent knowledge or general knowledge, and specific knowledge
which is encapsulated in cases. Archetypes provide us with all the specific knowl-
edge we need for CBR. It is natural to see that no reasoning knowledge is included
in the concept models or archetypes, but they could be used for extracting gen-
eral knowledge of the domain for instance for extracting basic rules that are
applied for diagnosis.
openEHR defines different classes of patient data. These classes are observa-
tion, evaluation, instruction and action. In openEHR observation is a structure
to record any information which is extracted from the world outside the clini-
cian’s mind [7]. This includes patient history of diseases and other treatments
and symptoms and signs of the disease. In contrast, evaluation type is used to
store the decision made by the clinician which is done in her/his mind. Instruc-
tion is a set of tasks that should be done on a patient; for instance prescription
or orders. Action is used to record information about the action taken on the
patient based on the instructions. Figure 6 illustrates how an openEHR-based
patient record can be mapped to a case in CBR.
On the other hand, representation of the cases in CBR includes Description
of the problem and the Solution. Description of the problem is analogous to
the observation part of the patient data, including information about clinical
history, symptoms and signs, and also lab values. Each case in CBR should
include information about the solution to the specified problem (query) in that
case. This solution in CBR is analogous to the openEHR evaluation, instruction
and action in openEHR-based patient data.
Towards a CBR Method for an openEHR-Based CDSS 13
Fig. 6. The sample archetyped data
6 Conclusion and Future Work
To develop an openEHR-based CDSS, one should carry out not only the typical
CDSS development activities but also activities suggested by openEHR commu-
nity for providing a solid underlying layer for storing and retrieving sharable
clinical data. The openEHR activities start with designing archetypes by involv-
ing domain experts. In this study, we found out that the approach suggested
by the openEHR community [8] is not applicable because of the capabilities of
the clinicians involved and we needed to apply our own approach, which was
designing archetypes based on clinical questionnaires.
Moreover, as in all CDSSs, a knowledge representation and reasoning method
should be selected. There are some criteria for selecting a KRR method for a
CDSS which we applied for selecting CBR for this project. CBR is a suitable
reasoning method for clinical domain since it is analogous to the concept of
individual patients, known as cases, which are also used for training medical
students. Clinicians see each patient as a case and even use this term for sharing
patient data among colleagues. Additionally, CBR applications can be used for
education in clinical domain.
Applying a CBR method in an openEHR-based CDSS is an interesting open
research direction, but needs connecting two different frameworks. Cases have
similarities to archetypes, therefore they can be generated automatically from
them and be used for reasoning purposes. openEHR archetypes help in the knowl-
14 Towards a CBR Method for an openEHR-Based CDSS
edge extraction process, but the classical bottleneck of knowledge acquisition in
clinical domain still exists.
The next phase of the project is to implement the rest of the mapper layer
and investigate the reasoning functionality.
References
1. Sittig, D.F., Wright, A., Osheroff, J.a., Middleton, B., Teich, J.M., Ash, J.S., Camp-
bell, E., Bates, D.W.: Grand challenges in clinical decision support. Journal of
biomedical informatics 41(2) (April 2008) 387–92
2. Garg, A., Adhikari, N., McDonald, H., Rosas, M.: Effects of computerized clinical
decision support systems on practitioner performance and patient outcomes: a
systematic review. JAMA 293(10) (2005) 1223–1238
3. Berner, E.: Clinical Decision Support Systems: Theory and Practice (Health In-
formatics). Springer, New York, NY 10013, USA (2007)
4. Osheroff, J., Teich, J., Middleton, B., Steen, E., Wright, A., Detmer, D.: A roadmap
for national action on clinical decision support. Journal of the American medical
informatics association 14(2) (2007) 141
5. Greenes, R.: Clinical decision support: the road ahead. Academic Press (2007)
6. Osheroff, J., Pifer, E., Teich, J., Sittig, D., Jenders, R.: Improving outcomes with
clinical decision support: An implementer’s guide. HIMSS (2005)
7. Beale, T., Heard, S.: The openEHR Architecture Overview. Website (2009) http:
//www.openehr.org.
8. Leslie, H., Heard, S.: Ocean Informatics. Website (2008) http://www.
oceaninformatics.com/ocean-informatics-resources/Tutorials.html.
9. Porter, S.: An update of the etiology and management of xerostomia. Oral Surgery,
Oral Medicine, Oral Pathology, Oral Radiology & Endodontics 97(1) (Jan 2004)
28–46
10. Bichindaritz, I., Marling, C.: Case-based reasoning in the health sciences: Whats
next? Artifitial Intelligence in Medicine 36(2) (Feb 2006) 127–135
11. Begum, S., Ahmed, M.U., Funk, P., Xiong, N., Folke, M.: Case-Based Reasoning
Systems in the Health Sciences: A Survey of Recent Trends and Developments.
IEEE Transactions on Systems, Man, and Cybernetics, Part C 7(1) (2010) 39–59
12. Aamodt, A., Plaza, E.: Case-based reasoning : Foundational issues , methodological
variations , and system approaches. Artificial Intelligence Communications, IOS
Press 7(1) (1994) 39–59
13. Pandey, B., Mishra, R.B.: Knowledge and intelligent computing system in
medicine. Computers in biology and medicine 39(3) (Mar 2009) 215–230
14. Bello-Tomas J, Gonzalez-Calero P, D.A.B.: JColibri: an object-oriented frame-
work for building CBR systems. in Proc. ECCBR 2004, advances in case-based
reasoning, Berlin, Heidelberg (2004) 3246
15. UCL: The openEHR Reference Framework and Application. Website (2011) http:
//opereffa.chime.ucl.ac.uk:80/introduction.jsf.
16. Marcos, M., Mart´ınez-Salvador, B.n.: Towards the interoperability of computerized
guidelines and electronic health records: an experiment with openEHR archetypes
and a chronic heart failure guideline. In: Proceedings of the ECAI 2010 confer-
ence on Knowledge representation for health-care. KR4HC’10, Berlin, Heidelberg,
Springer-Verlag (2011) 101–113
Towards a CBR Method for an openEHR-Based CDSS 15
17. Chen, R., Georgii-Hemming, P., Ahlfeldt, H.: Representing a chemotherapy guide-
line using openEHR and rules. Studies in health technology and informatics 150
(January 2009) 653–7
18. Xiao, L., Cousins, G., Hederman, L., Fahey, T., Dimitrov, B.: The design of an
EHR for clinical decision support. Number Bmei. IEEE (October 2010)
19. Lezcano, L., Sicilia, M.A., Rodr´ıguez-Solano, C.: Integrating reasoning and clin-
ical archetypes using OWL ontologies and SWRL rules. Journal of biomedical
informatics 44(2) (April 2011) 343–53
