ArticlePDF Available

Empowering Clinicians by eHealth Technologies in Decision-Making Tasks


Abstract and Figures

We present three types of eHealth applications that can enhance quality of clinical decision-making. Formalized electronic medical guidelines are bringing medical knowledge close to clinicians. eHealth tools for evaluation knowledge and competency in a given clinical decision-making problem are demonstrated by systems ExaMe and TECOM. The TECOM system supports training of clinical competence in a given decision-making problems using real clinical cases. The TECOM system estimates the decision-maker abilities using a coefficient of prediction or a classical error rate. Transfer of data and knowledge important for clinical decision-making without language barriers is demonstrated on features of the European Journal for Biomedical Informatics.
Content may be subject to copyright.
Empowering Clinicians by eHealth
Technologies in Decision-Making Tasks
Jana ZVÁROVÁ a,b,1, Helena HEROUTOVÁ b, Hana GRÜNFELDOVÁ a,c,
Karel ZVÁRA a,d, David BUCHTELA a,b
aCenter of Biomedical Informatics, Prague, Czech Republic
bDepartment of Medical Informatics,
Institute of Computer Science of the Academy of Sciences, Prague, Czech Republic
cMunicipal Hospital Caslav, Czech Republic
dEuroMISE s.r.o., Prague, Czech Republic
Abstract. We present three types of eHealth applications that can enhance quality
of clinical decision-making. Formalized electronic medical guidelines are bringing
medical knowledge close to clinicians. eHealth tools for evaluation knowledge and
competency in a given clinical decision-making problem are demonstrated by
systems ExaMe and TECOM. The TECOM system supports training of clinical
competence in a given decision-making problems using real clinical cases. The
TECOM system estimates the decision-maker abilities using a coefficient of
prediction or a classical error rate. Transfer of data and knowledge important for
clinical decision-making without language barriers is demonstrated on features of
the European Journal for Biomedical Informatics.
Keywords. clinical decision-making, eHealth, medical guidelines, education,
1. Introduction
Medical decision-making is the “core” of clinicians’ working activities. Clinicians not
only use their medical knowledge to make diagnostic, therapeutic and prognostic
decisions, but they also coordinate patient care over time and among multiple providers
and settings. Attempts to evaluate medical decision-making processes and the effort to
get insight in their nature have recently become very up-to-date due to the penetration
of ehealth technologies in medical practice. Formalization of existing medical
knowledge and new results in medical research make it more and more easier to
assimilate all the information, which might be useful in making medical decisions. The
role of biomedical ontologies in medical decision support and diagnostic processes
were described in many different ways, see [1], as well as new approaches to medical
education and formalization of medical knowledge were discussed in [2, 3]. Moreover,
new methods for evaluating clinical decision abilities that contribute to measuring of
clinical competence were published [4, 5].
Further we will mention three types of eHealth applications and their realizations
in the Czech environment that can enhance quality of clinical decision-making.
1 Corresponding Author: Jana Zvárová, Pod Vodarenskou vezi 2, 182 07 Prague, Czech Republic; E-mail:
Medical Informatics in a United and Healthy Europe
K.-P. Adlassnig et al. (Eds.)
IOS Press, 2009
© 2009 European Federation for Medical Informatics. All rights reserved.
2. Formalized Electronic Medical Guidelines
Medical guidelines are used for clinician decision support. They are intended to
improve the quality of patient care and reduce costs. Unfortunately, finding information
contained in conventional (free text form) guidelines may be difficult. A prerequisite
for developing decision support systems that use guidelines is creating formalized
electronic medical guidelines. A number of groups are actively developing computer
interpretable guideline representation languages for this purpose. The Arden Syntax [6]
is perhaps the best known representing language, but there are many other related
languages successfully introduced for representing medical knowledge, such as Asbru
[7], EON [8], GUIDE [9], PRODIGY [10], PROforma [11], GLIF [12], etc. We
designed knowledge representation model [13] based on the GLIF3.5 specification. It is
the universal tool for the formalization of knowledge stored in a free text form (e.g.,
medical guidelines). The XML representation of the graphic model and the creation of
the data interface in a paramodel form make it possible to use it in different types of
applications and to connect to real data from different sources (e.g., XML or CSV files).
There is a big potential of formalized electronic clinical guidelines to enhance quality
of clinical decision-making, decease the number of medical errors and save human and
financial resources. However the use of formalized electronic clinical guidelines is also
highly dependent on factors of legal, social and ethical environment.
3. Tools for Evaluation Knowledge and Competency in a Given Clinical Decision-
Making Problem
Although clinicians learn to cope with uncertainty, they seldom master it. In this case
eHealth applications that support learning processes for a given decision-making task
can rapidly increase quality of clinical decision-making. We have developed two
applications that can be used in this area: ExaMe system and system TECOM.
3.1. ExaMe System
Since 1998 the ExaMe system for evaluation of a targeted knowledge has been
developing [14]. The idea of the system is based on generalized multiple-choice
questions, with no prior restrictions on the number of given answers. The only
restriction is that at least one answer is correct and at least one wrong. This new idea
has led to new concepts of standardization of test results and also to new research
problems in statistics.
Evaluation by the ExaMe system is performed using fixed or automated test. A
fixed test is appropriate for evaluation of the group of students in computer classroom
connected to Internet. An automated test is appropriate for self-evaluation on remote
places. Students can pass evaluations by automated tests by themselves and the final
results of the tests are displayed immediately. The displayed results also give
explanation to students why some answers were not correct. The ubiquity of the
Internet and its World Wide Web applications made it possible to realize the new
educational goals in an innovative and creative way.
New features of the ExaMe evaluation system and statistical issues of evaluation
were described in [15]. During the last decades the reliability of didactical tests has
often been examined. It is easy to show (see for example [16]) that equivalently the
Jana Zvárováetal. /Empowering Clinicians by eHealth Technologies in Decision-Making Tasks684
reliability can be expressed as the squared value of the correlation between the
observed score X and the true score T, corr2(X,T). It means that the reliability of a
didactical test can be understood as the strength of the relationship between the score
reached by a student and his true knowledge. The ExaMe system covers also
knowledge on decision-support systems, expert systems and medical guidelines and
can be used for evaluation of students’ knowledge in the field.
Higher education programme on Biomedical Informatics for Ph.D. studies was
presented in [17]. Nowadays, topics of biomedical informatics are covered by five
different types of courses in the Czech language. The courses are focusing on health
information systems and electronic health records, telemedicine, bioinformatics and
biostatistics, knowledge discovery and decision support systems, standards,
interoperability, safety and security, evidence-based medicine and other topics. The
total number of participants in these courses in the years 2006–2007 was 132, the
number of successful graduates was 109. Target groups in these courses were
postgraduate doctoral students of medical faculties, clinicians and general practitioners,
health managers and other healthcare workers. Participants in these courses were
coming from more than 60 workplaces. These courses highly used e-learning tools, e.g.,
electronic books, ExaME program, multimedia presentations of lectures and different
software tools.
3.2. TECOM System
An important part of teaching medical decision-making is the method of measuring the
increase in the decision-maker’s ability to make correct decisions. The new TECOM
system (TEsting COMpetency) can evaluate decision-making ability of decision-
makers for any decision-making task. Simultaneously, clinical competence expressed
by the prediction coefficient is calculated. The use of the TECOM program is
demonstrated by an example from cardiology using real cases from the Municipal
hospital in Caslav.
Table 1. Part of the data matrix from the field of cardiology
Diagnosis Sex Age Height Weight BMI Hypertension
Cardiac infarction male 53 181 86 26.3 yes
Cardiac infarction female 59 * 76 * no
Cardiac infarction female 71 155 69 28.7 no
Cardiac infarction male 72 175 94 30.7 yes
Cardiac infarction female 80 * * * yes
Cardiac infarction male 78 * 72 * yes
Cardiac infarction female 47 159 100 39.6 yes
Pulmonary embolism male 42 * 89 * no
Pulmonary embolism female 63 165 68 25 no
Pulmonary embolism male 71 * * * yes
Pulmonary embolism female 81 * * * yes
Pulmonary embolism male 83 * 72 * Yes
The TECOM system was created in C++ Builder. In essence, it is capable to test
competency of decision makers in any area. The system acquires information for a
decision-making task from a data matrix that can contain any type of data. The data
matrix contains information on cases (e.g., patients) in the rows and correct decisions
as headings of rows. Correct decisions in the data matrix (e.g., correct diagnosis,
optimal therapy, validated prediction of a patient health state) should be carefully
validated by experts (e.g., physicians). Table 1 shows an example as a part of the data
Jana Zvárováetal. /Empowering Clinicians by eHealth Technologies in Decision-Making Tasks 685
matrix for a decision-making task from cardiology translated to English. We use the
TECOM system for evaluating competencies of Czech clinicians and medical students,
therefore we developed the Czech version of the TECOM system. However, the
TECOM system can be translated to other languages if necessary.
The data matrix, partly displayed in Table 1, is based on real cases described in
medical reports generated in the Municipal hospital in Caslav. 76 patients (cases) were
included in the data matrix and the data model contains 75 features describing each
patient. The headers in the first row are used by the TECOM system for the questions
to be asked. A generator of random numbers selects one row (i.e., data about an
examination of one patient and the final correct diagnosis, which of course the tested
student cannot see). The student can ask about values of symptoms, signs and
laboratory tests for the selected patient. The student can stop the process of asking
about these values any time. After that a new window appears with a list of possible
decisions (e.g., diagnoses). Then a student can distribute 100 points among possible
decisions. If the student is quite sure about the correctness of his/her final decision,
he/she gives 100 points to this decision. If he/she is not quite sure about the final
decision he/she divides points among more decisions, where one decision has a
maximal number of points. The decision with the highest number of points (highest
subjective probability) is the final decision made by the student. In case that the final
decision of the student is the same as the correct decision in the data matrix we classify
the final decision of the student as the correct one.
Then decision-making results of a student are evaluated by the traditional error rate
technique (percentage of false and correct decisions)and by the prediction coefficient Q,
1),1(log (1)
where n is the number of cases and pi is the probability of the correct outcome at the
i-th case (i.e., pi states how much the student’s final decision coincides with the correct
decision described in the data matrix, e.g., diagnosis given by an experienced
cardiologist). The prediction coefficient was proposed in [4] and it seems to reflect well
the competence of a student (decision-maker) in the given decision-making problem.
The TECOM system can help clinicians reveal more explicitly their decision-
making competencies and enhance their medical knowledge from cases and correct
decisions stored in the data matrix.
4. European Journal for Biomedical Informatics
In the year 2005 we came with a new initiative to publish the European Journal for
Biomedical Informatics (EJBI) on Internet ( The journal gives the
possibility to publish papers in original English versions with translations to other
European languages simultaneously. The multilingual versions of papers help to solve
problems with terminology and support semantic interoperability. Methods of
Information in Medicine (Methods) is a journal stressing the basic methodology and
scientific fundamentals of data, information and knowledge in medicine and health care.
As an official journal of IMIA, it has an international focus and readership. The editors
of Methods and EJBI agreed on future more close collaboration with the aim to better
meet future demands of timely and high-quality, peer-reviewed publications for a broad
readership, including possibilities for multilingual publications. Therefore EJBI is
Jana Zvárováetal. /Empowering Clinicians by eHealth Technologies in Decision-Making Tasks686
bringing new knowledge to clinicians in their own language that avoids
misunderstandings of notions or procedures.
5. Conclusion
We demonstrated several eHealth applications that can help clinicians to cope better
with uncertainty in decision-making tasks and increase their knowledge and skills. We
stressed the role of education and training in the field as well as the knowledge transfer
in the most understandable native language of clinicians.
Acknowledgement. The work was supported by the project 1M06014 of MSMT CR.
[1] Bodenreider, O. (2008) Biomedical ontologies in action: Role in knowledge management, data
integration and decision support. IMIA Yearbook of Medical Informatics 2008, Methods of Information
in Medicine 47(Suppl 1):67–79.
[2] Hovenga, E., Bricknel, L. (2006) Changing academic roles – New approaches to teaching and distance
learning in Australia. Methods of Information in Medicine 45, 288–293.
[3] Van Bemmel, J.H. (1985) Formalization of medical knowledge. In Van Bemmel, J.H., Grémy, F.,
Zvárová, J. (Eds.) Medical Decision Making: Diagnostic Strategies and Expert Systems, North-Holland,
Amsterdam, 1–11.
[4] Shapiro, A.R. (1977) The evaluation of clinical predictions. New England Journal of Medicine
[5] Zvarova, J., Jedlickova, A., Zvara, K. (1991) Computer-based evaluation of antibiotic therapy
predictions. International Journal of Biomedical Computing 29:207–213.
[6] Buchtela, D., Peleška, J., Veselý, A., Zvárová, J., Zvolský, M. (2008) Guideline knowledge
representation model. European Journal for Biomedical Informatics,
[7] Peleg, M., Ogunyemi, O., Tu, S. et al. (2001) Using features of Arden Syntax with object-oriented
medical data models for guideline modeling. AMIA Symposium Proceedings 2001, 523–527.
[8] Shahar, Y., Miksch, S., Johnson, P. (1998) The Asgaard project: A task-specific framework for the
application and critiquing of time-oriented clinical guidelines. Artificial Intelligence in Medicine 14:29
[9] Tu, S.W., Musen, M.A. (1999) A flexible approach to guideline modeling. AMIA Symposium
Proceedings 1999, 420–424.
[10] Quaglini, S., Stefanelli, M., Lanzola, G., Caporusso, V., Panzarasa, S. (2001) Flexible guideline-based
patient careflow systems. Artificial Intelligence in Medicine 22:65–80.
[11] Johnson, P.D., Tu, S.W., Booth, N., Sugden, B., Purves, I.N. (2000) Using scenarios in chronic disease
management guidelines for primary care. AMIA Symposium Proceedings 2000, 389–393.
[12] Fox, J., Patkar, V., Thomson, R. (2006) Decision support for healthcare: The PROforma evidence base.
Informatics in Primary Care 14:49–56.
[13] Ohno-Machado, L., Gennari, J.H., Murphy, S.N. et al. (1998) The GuideLine Interchange Format: A
model for representing guidelines. Journal of the American Medical Informatics Association 5:357–372.
[14] Zvarova, J., Zvara, K. (2000) Evaluation of knowledge using ExaMe program on the internet. In
Iakovidis, I., Maglaveras, S., Trakatellis, A. (Eds.) User Acceptance of Health Telematics Applications,
IOS Press, Amsterdam, 145–151.
[15] Martinkova, P., Zvara Jr, K., Zvarova, J., Zvara, K. (2006) The new features of the ExaMe evaluation
system and reliability of its fixed tests. Methods of Information in Medicine 45(3):310–315.
[16] Sireci, S.G., Wainer, H., Braun, H. (1998) Psychometrics, Overview. In Armitage, P., Colton, T. (Eds.)
Encyclopaedia of Biostatistics, John Wiley&Sons, New York, 3577–3602.
[17] Zvarova, J. (2008) Education in biomedical informatics and eHealth. In Blobel, B., Pharow, P., Nerlich,
M. (Eds.) Combining Health Telematics, Telemedicine, Biomedical Engineering and Bioinformatics to
the Edge, IOS Press, Amsterdam, 323–337.
Jana Zvárováetal. /Empowering Clinicians by eHealth Technologies in Decision-Making Tasks 687
... These indictors of performance will permit to decrease complication rate, morbidity, mortality and cost of care. One of the ways to optimize the quality of care is to use medical decision support systems [2,3,4,5] based on eHealth and mHealth to serve the unserved [6]. ...
Full-text available
Psychometrics includes educational testing, measurement of attitudes and perceptions, personality testing, opinion surveys, health inventories, and other forms of psychological measurement. Psychometric techniques created for measuring the human mind have also proven useful in biometrics and other fields that deal with measuring the intangible. This article is a comprehensive summary of the techniques for psychometric analysis including reliability, generalizability, and validity assessments. The article also presents the evolution of psychometric theory, popular measurement models, and a snapshot of new directions in the field.
Full-text available
To provide typical examples of biomedical ontologies in action, emphasizing the role played by biomedical ontologies in knowledge management, data integration and decision support. Biomedical ontologies selected for their practical impact are examined from a functional perspective. Examples of applications are taken from operational systems and the biomedical literature, with a bias towards recent journal articles. The ontologies under investigation in this survey include SNOMED CT, the Logical Observation Identifiers, Names, and Codes (LOINC), the Foundational Model of Anatomy, the Gene Ontology, RxNorm, the National Cancer Institute Thesaurus, the International Classification of Diseases, the Medical Subject Headings (MeSH) and the Unified Medical Language System (UMLS). The roles played by biomedical ontologies are classified into three major categories: knowledge management (indexing and retrieval of data and information, access to information, mapping among ontologies); data integration, exchange and semantic interoperability; and decision support and reasoning (data selection and aggregation, decision support, natural language processing applications, knowledge discovery). Ontologies play an important role in biomedical research through a variety of applications. While ontologies are used primarily as a source of vocabulary for standardization and integration purposes, many applications also use them as a source of computable knowledge. Barriers to the use of ontologies in biomedical applications are discussed.
Full-text available
Clinical guidelines can be viewed as generic skeletal-plan schemata that represent clinical procedural knowledge and that are instantiated and refined dynamically by care providers over significant time periods. In the Asgaard project, we are investigating a set of tasks that support the application of clinical guidelines by a care provider other than the guideline's designer. We are focusing on the application of the guideline, recognition of care providers' intentions from their actions, and critique of care providers' actions given the guideline and the patient's medical record. We are developing methods that perform these tasks in multiple clinical domains, given an instance of a properly represented clinical guideline and an electronic medical patient record. In this paper, we point out the precise domain-specific knowledge required by each method, such as the explicit intentions of the guideline designer (represented as temporal patterns to be achieved or avoided). We present a machine-readable language, called Asbru, to represent and to annotate guidelines based on the task-specific ontology. We also introduce an automated tool for the acquisition of clinical guidelines based on the same ontology, developed using the PROTEGE-II framework.
Full-text available
We describe a task-oriented approach to guideline modeling that we have been developing in the EON project. We argue that guidelines seek to change behaviors by making statements involving some or all of the following tasks: (1) setting of goals or constraints, (2) making decisions among alternatives, (3) sequencing and synchronization of actions, and (4) interpreting data. Statements about these tasks make assumptions about models of time and of data abstractions, and about degree of uncertainty, points of view, and exception handling. Because of this variability in guideline tasks and assumptions, monolithic models cannot be custom tailored to the requirements of different classes of guidelines. Instead, we have created a core model that defines a set of basic concepts and relations and that uses different submodels to account for differing knowledge requirements. We describe the conceptualization of the guideline domain that underlies our approach, discuss components of the core model and possible submodels, and give three examples of specialized guideline models to illustrate how task-specific guideline models can be specialized and assembled to better match modeling requirements of different guidelines.
Full-text available
The Prodigy system is a guideline-based decision-support system designed to assist general practitioners in England choose the appropriate therapeutic action for their patients. As part of the system, we developed a novel model for encoding clinical guidelines for managing patients with chronic diseases such as asthma and hypertension. The model structures a guideline as a set of choices to be made by the clinician. It models patient scenarios which drive decision making and are used to synchronize the management of a patient with guideline recommendations. The model is robust with respect to available input data and leaves the control of decision-making to the clinician. We have built execution engines to verify the computability of the model. We intend to test the model integrated in up to 200 live systems from at least four system vendors in English General practice.
Clinical predictions are never certain but are inherently probablisitc. The accuracy coefficient, a measure of probabilistic accuracy based on probability assigned to outcomes that occur, was used to assess the skill of clinical rheumatologists in predicting patient outcomes. Physicians' scores correlated well with degree of clinical experience. An approach to evaluation based on the measure provides a sensitive assessment of marginal benefit of technologies such as laboratory tests, diagnostic procedures or computer consultations. Most currently used methods of computer prediction were not as accurate as the best physicians tested. By allowing measurement of ability to individualize predictions to each patient's unique characteristics, the accuracy-coefficient approach has potential use in physician assessment.
Planning antibiotic treatment according to the given data on patients and results of microbiological examinations are important decision-making goals for physicians. One approach using a computer in medical education to improve the teaching process in this respect is described in this paper. On the basis of data about real patients and the optimal antibiotic treatment validated by the antibiotic center of the Faculty Hospital II, we can evaluate the student's abilities to predict correct antibiotic treatment. The general program written for this task and the method of evaluating the student's predictive skills are described.
In the paper we describe the function of the ExaMe program that serves for evaluation of students' knowledge using Internet. Evaluation is based on the knowledge base of a given course. Two types of evaluation tests are described. The fixed test is appropriate for examination of students by the teacher in computer classroom connected to Internet. By this test the evaluation is done in the limited time period and all students are tested with the same evaluation test. The automatic test is appropriate for self-evaluation and self-study by students on remote places. The student can ask the ExaMe program for the test of different difficulty levels and state the number of questions required. Finally thee first experience with the ExaMe program in practice is given.