Conference PaperPDF Available

Developing Competence Assessment Procedure for Spinal Anaesthesia

Conference Paper

Developing Competence Assessment Procedure for Spinal Anaesthesia

Abstract and Figures

Traditional approaches of assessment in the medical domain are insufficient for evaluating trainees' technical skills. Currently, many European medical training bodies are attempting to introduce competence-based training programmes for technical skills as well as other domains (e.g., communication, professional behaviour, clinical cognition). These efforts are limited due to the absence of appropriate assessment tools. Based on Competence-based Knowledge Space Theory (CbKST), a collaborative project MedCAP intends to develop a valid and reliable competence assessment procedure for one important medical skill, spinal anaesthesia. The paper briefly overviews the current states of training and assessment for medical procedural skills, describes the core ideas of CbKST, and introduces the ongoing project that will transfer the innovative approach of CbKST in personalized learning and competence assessment to the medical domain.
Content may be subject to copyright.
1
Developing Competence Assessment Procedure for
Spinal Anaesthesia
Dajie Zhang
University of Graz
dajie.zhang@uni-graz.at
Dietrich Albert
University of Graz
Cord Hockemeyer
University of Graz
Dorothy Breen
Cork University Hospital
Zsuzsanna Kulcsár
Cork University Hospital
George Shorten
Cork University Hospital
Annette Aboulafia
University of Limerick
Erik Lövquist
University of Limerick
Abstract
Traditional approaches of assessment in the
medical domain are insufficient for evaluating
trainees’ technical skills. Currently, many European
medical training bodies are attempting to introduce
competence-based training programmes for technical
skills as well as other domains (e.g., communication,
professional behaviour, clinical cognition). These
efforts are limited due to the absence of appropriate
assessment tools. Based on Competence-based
Knowledge Space Theory (CbKST), a collaborative
project MedCAP intends to develop a valid and
reliable competence assessment procedure for one
important medical skill, spinal anaesthesia. The paper
briefly overviews the current states of training and
assessment for medical procedural skills, describes the
core ideas of CbKST, and introduces the ongoing
project that will transfer the innovative approach of
CbKST in personalized learning and competence
assessment to the medical domain.
1. Introduction
Worldwide, medical training is undergoing
dramatic changes, moving from a process and
structure-based training paradigm towards a
competence-based paradigm [1,2]. The former
determines learning on the basis of exposure to
specified content over a certain period of time, while
the latter does so on the basis of competence
achievement [3].
Competence-based training necessitates valid and
reliable competence assessment procedures (CAPs).
However, for most medical procedural skills, no such
CAP exists. The challenges in developing such CAPs
lie in defining each competence and taking account of
the many factors that influence learning and
performance of medical procedures. Such determinants
include cognitive, motor, communicative, and human
(e.g. fatigue, anxiety, and fear) factors [4].
In recent years, most European countries have tried
to introduce competence-based training into medical
education. These efforts are restricted by the absence of
a universally-accepted and valid means of assessing
competence in medical procedural skills [5-7].
Evaluation of medical procedural skills entails (i)
rating by supervising clinicians during the
apprenticeship or the residency, and (ii) assessment
based on clinical simulations. The former involves
exposing patients to inexperienced trainees, relying on
selective or second hand data, or non-validated
assessment techniques that are subject to bias (e.g. race
or gender) [6]. Although assessments based on clinical
simulation offer benefits in terms of reliability and
validity, evaluation of the influence of human factors
(i.e., anxiety, fatigue, etc.) is compromised by the
artificial settings [6,7].
In other domains, competence-based knowledge
space theory (CbKST) [8,9] has been successfully
applied to facilitate personalised learning and to assess
competence. Recognizing the success of the CbKST
approach as well as the urgent and great need in the
medical domain, a new project MedCAP (“Competence
Assessment for Spinal Anaesthesia”, homepage:
http://www.medcap.eu) commenced in 2007 [10]. It
aims to transfer the innovative CbKST approach to the
medical domain in order to develop a valid, reliable
and practical CAP for one medical procedural skill,
spinal anaesthesia. As performing spinal anaesthesia
requires elements of competence in several domains
common to many other (and more complex) medical
procedures, the principles applied in developing a CAP
for spinal anaesthesia could be applied to others.
Zhang, D., Albert, D., Hockemeyer, C., Breen, D., Kulcsar, Z., Shorten, G., Aboulafia, A., &
Lövquist, E. (2008). Developing competence assessment procedure for spinal anaesthesia.
Proceedings of the 21st IEEE International Symposium on Computer-Based Medical
Systems, 397-402.
2
2. Training of spinal anaesthesia
Spinal anaesthesia is a delicate procedure involving
the injection of local anaesthetic solution into the fluid
surrounding the spinal cord to facilitate lower
abdominal or lower limb surgery (Figure 1). By feeling
the resistive forces of the needle passing through
various tissues, the anaesthetist places the tip of the
needle into the correct space without causing damage
to surrounding tissues and nerves.
As with training of other medical technical skills,
students learning spinal anaesthesia are routinely taught
manual techniques and necessarily practice the novel
skills on hospital patients. Due to the mounting
pressures in the clinical and training environment, such
as emphasis on operating room efficiency (European
Working Time Directive), execution of the Bologna
Accord, emphasis on patient safety, cost factors and
others, the opportunities for an individual trainees to
acquire hands on” experience in procedural skills has
decreased substantially.
Figure 1. Demonstration of a spinal anaesthetic injection
[11]
Computer-based technology (e.g., simulation, web-
based learning and virtual reality) has been introduced
into medical training purportedly to improve the
efficiency, effectiveness and safety of learning and
teaching of procedural and other skills [12,13]. Some
high-fidelity simulators are available for training and
assessment purposes. For example, (i) a commercially
available simulator (Figure 2) for epidural anaesthesia
(which shares certain characteristics with spinal
anaesthesia) developed by MedicVision (partner of
MedCAP project, see: http://www.medicvision.com.au)
has been successfully marketed in European countries.
It has brought expertise in the development of technical
training using simulators. (ii) To provide effective and
safe training without subjecting patients to risk in
spinal anaesthesia, an interactive virtual learning
system has been developed during the DBMT project
Figure 2. Epidural Simulator (with permission)
(Design Based Medical Training, http://www.dbmt.eu).
As the procedure of spinal anaesthesia relies heavily on
tactile cues, learners are required to recognize the
characteristic “sensations” in the procedure. In DBMT,
a haptic device, PHANTOM
®
Desktop
TM
from
Sensable Technologies (see: http://www.sensable.com),
has been adapted to replicate these sensations. Sense of
touch and resistive forces are simulated. The physical
make-up of each individual layer of tissue in a human
back was modelled. The haptic device has a mechanical
arm with five joints, enabling the user to manipulate,
interact and feel objects and sensations. The arm has
been modified by attaching a spinal needle, thus
providing the user with a realistic instrument to hold.
The movement of the needle is within a three
dimensional space, thereby facilitating easier
navigation. Stereoscopic glasses are used to create the
illusion of depth on the screen (Figure 3). To
implement the 3D model and the force feedback
properties of the various tissue layers, the developing
software H3DAPI (see: http://www.h3d.org) is used
with the extension of VHTK (Volume Haptics Toolkit).
Additionally, CT scan images are used to create the 3D
model of a human back.
With this augmented reality system [14], learners
experience the realistic sensations of inserting the
spinal needle on the one hand, and monitor what
happens under the skin of the patient on the other.
Patient variations and levels of difficulty can be built
into the system to offer different training challenges.
More importantly, the system tracks all the movement
by the user during the procedure, thus providing a basis
for assessing the procedural skills which is required by
the MedCAP project.
Epidural
Needle
Epidural
LOR
Syringe
Patients
back
Laptop
Computer
With 3D
visuals
3
Figure 3. Spinal anaesthesia training system developed
by DBMT [14]
3. Competence-based Knowledge Space
Theory (CbKST)
In MedCAP, the competence assessment procedure
will be developed based on CbKST. Traditional
evaluation of knowledge marks individual
achievements with numerical scores. By nature, a test
score offers no cue to what an examinee can do and
what he still needs to learn, hence contributes little to
further learning and development. When two students
score equal on a test, there is no evidence whether they
possess the comparable competences. In addition, in a
traditional linear test, all examinees are presented with
the same set of items in a predefined form. The test
score for an individual is obtained based on responses
to all the items in the current test, although the items
only cover a fraction of the knowledge in the complete
domain. This is an inefficient and inaccurate way of
assessing ability. Given the intrinsic flaws of the
traditional assessments, new approaches are required
for evaluating individual competences.
CbKST developed by Albert and colleagues [8] is
such an approach suitable for adaptively assessing
individual competences without numerical
representations [cf. 15,16]. It is an extension of the
Knowledge Space Theory (KST) [17,18]. The original
KST was behaviouristic, judging an individual’s
knowledge state via his observable performances (i.e.,
being able or not able to solve particular problems in a
test). Later works of different research groups have
extended the KST by analyzing competences entailed
in a given knowledge domain, and assigning them to
the test problems and learning objects [15,16,19-22].
Both KST and CbKST ground on a basic
phenomenon that acquiring some pieces of knowledge
normally precede some other pieces of knowledge. A
certain type of problems p may be solvable by a student
only if another type of problems q has already been
mastered by the student. For example, if a student is
able to solve the addition of two decimals, he should
already be able to solve the addition of two integers. As
such, problem type q is called the prerequisite or the
precedence of problem type p. By correct responses to
type p problems, correct answers to type q problems
can be surmised. Such a surmise relation (or
prerequisite, precedence relation) can be illustrated in a
Hasse diagram showed in Figure 4, which consists of
five hypothetical problem types a, b, c, d and e. The
prerequisite relation between the problem types is
indicated by the descending segments. In Figure 4, for
instance, problem types a, b, and c are the prerequisites
for type e. If a student responds correctly to problems
of type e, it is likely that he can also solve problems of
type a, b and c.
c
e
b
d
a
Figure 4. A hypothetical Hasse-diagram illustrating
surmise relation of five problems
According to CbKST, a knowledge domain can be
represented by two structures: (a). A collection of
competences that are inherent in a domain. (b). A set of
problems that can be solved in the domain given the
competences in (a). Both (a) and (b) can be structured
based on the surmise relations. Importantly, the number
of competences in a domain is finite while the viable
problems can be solved may be infinite.
Approaching a knowledge domain via (b), a
knowledge state refers to a specific subset of problem
types in the domain that some individual is capable of
solving. The Hasse-diagram in Figure 4 completely
defines the feasible knowledge states in this
hypothetical mini-domain. Analyzing Figure 4, exactly
10 knowledge states can be induced, forming the set
K = [23], {c}, {a, c}, {a, b}, {a, b, c}, {a, b,
d}, {a, b, c, e}, {a, b, c, d}, Q},
of which refers to the empty set, and Q refers to the
complete set of {a, b, c, d, e}. The set K is called the
knowledge structure of this hypothetical domain.
The knowledge structure can be illustrated in a set-
inclusion diagram consisting of all the feasible states
(Figure 5). The structure implies different possible
learning paths moving from the naïve knowledge state
to the full mastery of Q. For example, one can start
by first mastering a, and successively the other types
bĺdĺcĺe (Figure 5). Alternatively, one can also start
with c, and proceed to aĺbĺeĺd. Note that Figure 4
and 5 illustrates a domain with merely 5 types of
problems. In reality, even for an elementary knowledge
4
domain, the number of knowledge states and of
learning paths can become very large [24].
{a, b, c, d}
{a, b, c, d, e}
{a, b, c, e}
{c}
{a, c}
{a, b, c}
{a}
{a, b}
{a, b, d}
Figure 5. Knowledge structure consistent with the
knowledge domain illustrated in Figure 4. The dashed
arrows display one of the possible learning path
As suggested by Figure 5, learning can take place
step by step, one problem type at a time. Specifically,
each knowledge state (except Q) has at least one
immediate successor state which contains all the same
problem types, plus exactly one. The knowledge state
{a, b, c} of K, for instance, has the two states {a, b, c,
d} and {a, b, c, e} as immediate successors. Problem
types d and e are the outer fringe of the state {a, b, c}.
It contains exactly the problem types that a particular
learner processing knowledge state {a, b, c} should
proceed to learn. Conversely, each knowledge state
(except ) also has at least one predecessor state that
contains exactly the same problems, except one. The
knowledge state {a, b, c}, for example, has two
predecessor states, i.e., {a, b} and {a, c}. Problem
types b and c together form the inner fringe of state {a,
b, c}, which are the most sophisticated problem types
the learner has mastered by far. If the learner has
difficulty solving the outer fringe problems, reviewing
materials in the inner fringe should normally be
recommended. The two fringes are sufficient to specify
a particular knowledge state, of which the outer fringe
directs progression while the inner fringe monitors the
possible retreats. Both are crucial for generating
personalized learning paths.
To sum up, knowledge in a particular domain can
be represented by different types of problems organized
by prerequisite relations. An individual’s current
knowledge state is identified by the problems he
masters in the domain. The collection of the feasible
knowledge states forms the knowledge structure. For
any given knowledge structure, divergent learning
paths are possible, each leading from the naïve
knowledge state to the complete mastery of the
knowledge domain. Each knowledge state has an outer
and an inner fringe. The former directs the learning
progression while the latter implies possible reviews.
As mentioned before, a knowledge domain can also be
identified by its inherent competences. A competence
(or skill)
1
is defined as a combination of an action and
a concept
2
(e.g., state Theorem of Pythagoras” and
apply Theorem of Pythagoras” are two different skills)
(see Marte et al. [25] for a discussion of the connection
between CbKST and Bloom’s [26] taxonomy of
hierarchical classification of educational goals).
By comprehensive analysis of a knowledge domain,
underlying competences can be identified. Analogous
to constructing the knowledge structure, a competence
structure can be derived, containing the competence
states organized by surmise relations. For example, in
the competence structure of spinal anaesthesia, the
competence state “performs lumbar puncture” surmises
the state “applies knowledge of anatomy to identify
the interspace”.
The competence states and the knowledge states
(i.e., sets of test problems) can be matched mutually.
On the one hand, for each type of problem, a particular
competence state (or several competence states) is/are
sufficient to solve it (Figure 6, left panel). On the other
hand, given a particular competence state (involving
one or more competences), one or more types of
problems can be solved and the corresponding
knowledge state can be inferred (Figure 6, right panel).
{3,5}d
{3}c
{1,2}b
{1,2,4}, {3,4}a
Competence
state(s)
Problem
type
{3,5}d
{3}c
{1,2}b
{1,2,4}, {3,4}a
Competence
state(s)
Problem
type
{a,c}{3,4}
{c,d}{3,5}
{c}{3}
{b}{1,2}
{a,b}{1,2,4}
Knowledge
state
Competence
state
{a,c}{3,4}
{c,d}{3,5}
{c}{3}
{b}{1,2}
{a,b}{1,2,4}
Knowledge
state
Competence
state
Figure 6. Illustration of the relationship between
competence states and knowledge states. Numbers refer
to different competences
Consequently, an assessment based on CbKST will
not only identify what kinds of problems a learner is
able to solve, more importantly, it will reveal an
individual’s current competence state underlying his
visible behaviour.
In a learning situation, skills are pre-assigned to
each learning object (e.g., a learning scenario contains
tutoring and exercises). A learning object is always
defined by its prerequisite skills and the new skill(s) to
be learned. Appropriate objects will be suggested to the
1
The terms “competence” and “skill” have been used
interchangeably in the literature and are accepted as exchangeable in
this paper.
2
cf. [16], where “concept“ and “action are not separated.
5
learner in a virtual environment, which adapts to the
learner’s current competence state.
4. Competence assessment procedure for
spinal anaesthesia
After the competence structure has been identified,
its induced assessment does not have to exhaust all the
problems in a knowledge domain. Instead, based on the
surmise relations, the assessment will be much more
efficient.
At the onset of the assessment, an individual is
given a randomly selected item of a certain problem
type p, for which he would have about a 50%
likelihood of solving it. The likelihood of each problem
could be derived, for example, from the average
success rate of other comparable peers (e.g., those who
study in the same grade) who have been tested with it.
If the student responds correctly, the likelihoods of all
the knowledge states containing p are increased and,
accordingly, the likelihoods of all the states not
containing p are decreased. A false response given by
the student has the opposite effect: the likelihoods of all
the states not containing p are increased, and those of
the remaining states decreased. The following test
problems are then selected by the same mechanism,
based on the updated likelihoods of the states deriving
from the individual’s previous responses. In this way,
the problem types left to be tested reduce rapidly, and
the likelihood of some states gradually increases. The
procedure stops when some peak in the likelihood
function is reached [27]. The system has now revealed
the most likely knowledge state of the individual. The
state will then be interpreted by the underlying
competences, providing detailed information about
what an individual is able to do and what he is ready to
learn.
To apply CbKST to the medical domain, an
essential task is to comprehensively define competence
and knowledge structures of the relevant domain. As
for spinal anaesthesia, the competence assessment
should encompass: medical knowledge; technical
ability; communication; patient management skills and
other dimensions.
Preliminary work carried out at Cork University
Hospital (CUH, Partner of MedCAP project) implied
that such competence structure exists for spinal
anaesthesia [28]. Since November 2007, five partners
in Europe (CUH, University of Graz, University of
Pecs, Interaction Design Centre and MedicVision Ltd)
have jointed to develop a CAP for spinal anaesthesia.
The project will comprise a learning management
system (LMS) and a Web service. The LMS will
provide the interface with the user, accept user input
and offer graphical output (test objects). The Web
service will provide the main functionality and will be
used to implement the assessment algorithms (Figure
7).
User
LMS
Interface
Assessment
Logic
Logging
of
data
Competence
structure
Ontology
of
competences
Back end
web service
User
LMS
Interface
Assessment
Logic
Logging
of
data
Competence
structure
Ontology
of
competences
Back end
web service
Figure 7. The CAP system in MedCAP
5. Discussion
In order to develop a valid and reliable CAP for
spinal anaesthesia, the partnership of MedCAP will
comprehensively describe the competences required in
the domain, generate algorithms necessary to assess
individual performance, implement the CAP in a user-
friendly, web-based format and test it in simulated and
real clinical settings for construct validity and
reliability. Challenges remain in how to (a)
comprehensively define the competence structure of
the domain; (b) generate corresponding types of test
problems in suitable presenting formats; and to (c)
determine criteria to classify the possible responses to
the problems.
The valid and reliable CAP shall be applied in the
European medical training bodies, supporting
personalized learning and competence-based training as
well as improving the safety and efficiency of the
medical environment. The principles employed in
developing the CAP for spinal anaesthesia could be
extrapolated to developing similar assessment tools for
other medical procedural skills.
Acknowledgement
The project MedCAP is funded by European
Commission: LDV/LLP/TOI/2007/IRL-513
.
References
[1] R. Aggarwal and A. Darzi, "Technical-skills training
in the 21st century," N Engl J Med, vol. 355, pp.
2695-6, 2006.
[2] R. K. Reznick and H. MacRae, "Teaching surgical
skills -- changes in the wind," N Engl J Med, vol. 355,
pp. 2664-9, 2006.
[3] C. Carraccio, S. D. Wolfsthal, R. Englander, K.
Ferentz, and C. Martin, "Shifting paradigms: from
flexner to competencies," Acad Med, vol. 77, pp. 361-
7, 2002.
6
[4] A. F. Smith, C. Pope, D. Goodwin, and M. Mort,
"What defines expertise in regional anaesthesia? An
observational analysis of practice," Br J Anaesth, vol.
97, pp. 401-7, 2006.
[5] A. J. Byrne and J. D. Greaves, "Assessment
instruments used during anaesthetic simulation:
review of published studies," Br J Anaesth, vol. 86,
pp. 445-50, 2001.
[6] M. Srinivasan, J. C. Hwang, D. West, and P. M.
Yellowlees, "Assessment of clinical skills using
simulator technologies," Acad Psychiatr, vol. 30, pp.
505-15, 2006.
[7] R. M. Epstein, "Assessment in medical education," N
Engl J Med, vol. 356, pp. 387-96, 2007.
[8] D. Albert and J. Lukas Eds., Knowledge Spaces:
Theories, Empirical Research, and Applications.
Mahwah, NJ: Lawrence Erlbaum, 1999.
[9] J. Heller, C. Steiner, C. Hockemeyer, and D. Albert,
"Competence-based knowledge structures for
personalised learning," Int J e Learn, vol. 5, pp. 75-
88, 2006.
[10] D. Albert, C. Hockemeyer, and G. Shorten,
"Competence assessment for spinal anaesthesia," in
HCI and Usability for Medicine and Health Care -
Proceedings of the Third Symposium of the
Workgroup Human-Computer Interaction and
Usability Engineering
of the Austrian Computer
Society (USAB 2007) Graz, Austria
, A. Holzinger,
Ed., Berlin: Springer, 2007, pp. 165-70.
[11] U. Dreifaldt, Z. Kulcsar, and P. Gallagher,
"Exploring haptics as a tool toimprove training
of medical doctors in the procedure of spinal
anaesthetics," Eurohaptics 2006, Paris, 3rd-6th of
July, 2006.
[12] Y. A. Zausig, Y. Bayer, N. Hacke, B. Sinner, W.
Zink, C. Grube, and B. M. Graf, "Simulation as an
additional tool for investigating the performance of
standard operating procedures in anaesthesia," Br J
Anaesth, vol. 99, pp. 673-8, 2007.
[13] R. S. Haluck and T. M. Krummel, "Computers and
virtual reality for surgical education in the 21st
century," Arch Surg, vol. 135, pp. 786-92, 2000.
[14] E. Lövquist, A. Aboulafia, and Z. Kulcsar, "A medical
simulator for teaching spinal anaesthesia," INMED,
scientific meeting, Dublin, 7-8 Feb., 2008.
[15] I. Düntsch and G. Gediga, "Skills and knowledge
structures," Br J Math Stat Psychol, vol. 48, pp. 9-27,
1995.
[16] K. Korossy, "Modeling knowledge as competence and
performance," in Knowledge Spaces: Theories,
Empirical Research Applications, D. Albert and J.
Lukas, Eds., Mahwah, NJ: Lawrence Erlbaum
Associates, 1999, pp. 103-32.
[17] J.-P. Doignon and J.-C. Falmagne, "Spaces for the
assessment of knowledge," Int J Man Mach Stud, vol.
23, pp. 175-96, 1985.
[18] J.-P. Doignon and J.-C. Falmagne, Knowledge Space.
Berlin: Springer, 1999.
[19] J.-C. Falmagne, M. Koppen, M. Villano, J.-P.
Doignon, and L. Johannesen, "Introduction to
knowledge spaces: How to build, test and search
them," Psychol Rev, vol. 97, pp. 201-24, 1990.
[20] J.-P. Doignon, "Knowledge spaces and skill
assignments," in Contributions to mathematical
psychology, psychometrics and methodology, G.
Fischer and D. Laming, Eds., New York: Springer,
1994, pp. 111-21.
[21] D. Albert and T. Held, "Establishing knowledge
spaces by systematical problem construction," in
Knowledge Structures, D. Albert, Ed., New York:
Springer, 1994, pp. 78-112.
[22] C. Hockemeyer, O. Colan, V. Wade, and D. Albert,
"Applying competence prerequisite structures for
eLearning and skill management," J Univers Comput
Sci, vol. 9, pp. 1428-36, 2003.
[23] M. Lipman, "Teaching students to think reasonably:
Some findings of the Philosophy for Children
program," Clearing House, vol. 71, pp. 277-80, 1998.
[24] J.-C. Falmagne, E. Cosyn, J.-P. Doignon, and N.
Thiéry. (2004) The assessment of knowledge, in
theory and in practice. [Online]. Available:
http://www.aleks.com/about_aleks/Science_Behind_
ALEKS.pdf
[25] B. Marte, C. Steiner, J. Heller, and D. Albert, "Activiy
and Taxonomy-Based Knowledge Representation
Framework (Contribution to the 2nd EleGI
conference, Oct. 2006, Barcelona)," International
Journal of Knowledge and Learning., In Press.
[26] B. S. Bloom, Taxonomy of educational objectives:
The classification of educational goals - Handbook 1:
cognitive domain. New York, NY: Mckay, 1956.
[27] J.-C. Falmagne and J.-P. Doignon, "A class of
stochastic procedures for the assessment of
knowledge," Br J Math Stat Psychol, vol. 41, pp. 1-
23, 1988.
[28] Z. Kulcsar, A. Aboulafia, T. Hall, and G. Shorten,
"Determinants of learning to perform spinal
anaesthesia - a qualitative study," Eur J Anaesthesiol,
in press.
... This has the potential to streamline assessment according to the competence state of the learner. 4 We have used spinal anaesthesia as the prototype here, but CbKST could be applied in a similar way to other practical procedures. The advantage of using CbKST in this way is to decrease the number of competences that need to be tested at the bedside and to tailor the assessment process to the skill of the learner. ...
Article
Background In recent years there has been a move towards a competency-based model for assessing the performance of practical procedures in clinical medicine rather than the traditional assumption that competency is achieved with increasing experience. For such an assessment to be valid, the necessary competencies comprising that skill must be identified. Our aim was to map the individual competencies necessary to perform a given procedural skill using spinal anaesthesia as the example, and to explore the relationship of individual competencies with each other.Methods In the first part of the study an extensive hierarchical task analysis (HTA) was undertaken to determine the competencies necessary for the performance of spinal anaesthesia. Secondly, the concept of competency-based knowledge space theory (CbKST) was applied to the map. CbKST is based on the principle that acquisition of a specific skill is usually preceded by a number of dependent or prerequisite skills.Our aim was to map the individual competencies necessary to perform a given procedural skillResultsThe analysis yielded a comprehensive HTA of the skills necessary to perform spinal anaesthesia, comprising 509 individual competencies. Applying the concept of CbKST yielded 194 key competences with at least one dependent or prerequisite skill.DiscussionWe have defined a comprehensive HTA or competency map for use in the assessment of the performance of spinal anaesthesia. This CbKST approach will provide clinicians who undertake medical procedures to better understand their own performance, and to improve over time.
... Here, s/he is supervised by an experienced doctor who can intervene whenever necessary and who is asked to answer a questionnaire on the trainee's performance and competencies afterwards. All these information are fed into the assessment procedure (Hockemeyer et al., 2009;Zhang et al., 2008). ...
Chapter
A saying attributed to Kurt Lewin (1951) states ‘There is nothing so practical as a good theory.’ Accordingly, the theory and models outlined in Chapter 11 of this volume have many practical consequences and can be applied for instance in personalized competence assessment, individualized eTeaching and eLearning, and expert query. Furthermore, for practical reasons recent developments in Competence-based Knowledge Space Theory (CbKST) have to be taken into account.
... Berlin: Springer. of the art learning management system. Within this aim, we focus on the technical realisation of previously published conceptual ideas [1, 2]. After briefly introducing spinal anaesthesia and the Competence-based Knowledge Space Theory (CbKST) in the remainder of this introduction, we will give some general information on the web and simulation based system for spinal anaesthesia. ...
Conference Paper
Full-text available
The authors present an approach for implementing a system for the assessment of medical competences using a haptic simulation device. Based on Competence based Knowledge Space Theory (CbKST), information on the learners’ competences is gathered from different sources (test questions, data from the simulator, and supervising experts’ assessments). The envisaged architecture consists of three core modules, an LMS (Moodle) containing user model and content objects and realising the interface between system and user, a simulator interface as an own service connecting the LMS to the (external) simulator system, and a CbKST service offering the assessment logic and visualisations of the assessment result for learner and teacher.
Conference Paper
This paper presents the development and preliminary evaluation of a Virtual Reality-based system for training in dental anesthesia. The development focused the simulation of an anesthesia procedure task. The evaluation involved graphic and haptic issues and had the presence of experts in the dentistry area. The assessment aimed at attributes that may influence the human-computer interaction, hindering realism, an important challenge in systems of this type. The attributes selected were: the update rate, the appearance of the virtual models and the number of viewpoints of the virtual environment, as well as the characteristics of the haptic device. Despite constraints were found, in the perception of the experts, the system may provide realism and help with the training of certain tasks.
Chapter
From its beginning knowledge space theory was developed from a purely behavioristic point of view. It focused on the solution behavior exhibited on the instances of a set of items constituting a knowledge domain. This kind of stimulus-response consideration lead to very successful applications. Knowledge space theory has most effectively been applied especially in educational contexts where there is a curriculum prescribing the content to be covered, allowing for a fairly obvious and more or less complete definition of the relevant knowledge domain. There are, however, good reasons not to limit knowledge space theory to the kind of operationalism that identifies the state of knowledge with the subset of items an individual is capable of solving. The framework offered by knowledge space theory is able to integrate psychological theory by bringing into the picture the underlying cognitive abilities (skills, competencies, . . . ) responsible for the observable behavior. This kind of development may be seen somewhat analogous to traditional mental testing (Falmagne and Doignon, 2011). In this context, psychometric models referring to latent variables are preferred to purely operationalistic approaches, like classical test theory (cf. Borsboom, 2006).
Conference Paper
Several fields of the knowledge have been benefited by technologies from Virtual Reality (VR) area. This paper presents an analysis of problems related to the domain of dentistry, specifically for training in anesthesia, and the possibility of resolution using VR. The analysis assumes that the haptic interaction in virtual environments is relevant in the area of training in health and may contribute in the skills acquisition, providing realistic experiences to the trainess. Initially, a detailed requirements survey with the participation of experts was conducted, as well as the development of a prototype for preliminary testing.
Chapter
Knowledge Space Theory was founded by Doignon and Falmagne (1985). This paper initiated an extensive body of work 6 , which is still in progress. In this chapter, we present recent examples regarding developments in the theory and its relationship to other approaches, methods and applications. As in other chapters of this volume (e. g., Ch. 10) it becomes obvious that there is a high potential for further developments of Knowledge Space Theory. Furthermore, for theoretical as well as practical reasons the relationship to other theoretical approaches such as Formal Concept Analysis, Latent Class Analysis, and Item Response Theory has to be taken into account.
Article
To identify the determinants of learning for one medical procedural skill, spinal anaesthesia, by eliciting the opinions of anaesthetists in Ireland and Hungary. This objective is one component of a research project, Medical Competence Assessment Procedure (MedCAP) funded by the EU Leonardo da Vinci Lifelong Learning Programme. An electronic survey was circulated to anaesthetists in Hungary and Ireland. The survey was designed to identify and prioritise determinants of learning. Primary analysis was performed using the proportions of respondents that either agreed or strongly agreed with each question. A secondary analysis was performed comparing responses from Ireland with those from Hungary. A total of 180 of the 810 anaesthetists surveyed responded in Ireland, and 69 out of 225 responded in Hungary. In both countries, more than 90 per cent agreed or strongly agreed that acquisition of baseline knowledge, clinical demonstration, trainee motivation, feedback to the trainee, trainer motivation and communication skills were important determinants of learning. However, a greater proportion of Hungarian compared with Irish anaesthetists indicated that training should follow a problem-based approach [60/63 (95%) versus 54/124 (43%)]. A greater proportion of Irish anaesthetists indicated that trainee self-awareness was an important determinant of learning [89/122 (73%) versus 22/64 (34%)]. Anaesthetists in Ireland and Hungary believe that learning spinal anaesthesia is determined by factors related to the trainee (motivation, knowledge), the trainer (motivation, communication) and the training programme (feedback, demonstration prior to clinical performance). Differences between respondents from the two countries were identified in regard to attitudes towards problem-based learning and self-awareness. These findings can be used to inform the design of training programmes and simulators.
Article
Full-text available
h e n e w e ng l a n d j o u r na l o f m e dic i n e n engl j med 356;4 www.nejm.org january 25, 2007 A s an attending physician working with a student for a week, you receive a form that asks you to evaluate the student's fund of knowledge, procedural skills, professional-ism, interest in learning, and "systems-based practice." You wonder which of these attributes you can reliably assess and how the data you provide will be used to further the student's education. You also wonder whether other tests of knowledge and com-petence that students must undergo before they enter practice are equally problematic. I n one way or another, most practicing physicians are involved in assessing the competence of trainees, peers, and other health professionals. As the example above suggests, however, they may not be as comfortable using educational assessment tools as they are using more clinically focused diagnostic tests. This article provides a conceptual framework for and a brief update on com-monly used and emerging methods of assessment, discusses the strengths and limitations of each method, and identifies several challenges in the assessment of physicians' professional competence and performance. C ompe tence a nd Per for m a nce
Chapter
Procedures which are to test a subject’s knowledge concerning a specific domain obviously require (in addition to other prerequisites) a set of problems.
Article
Realizing medical education is on the brink of a major paradigm shift from structure- and process-based to competency-based education and measurement of outcomes, the authors reviewed the existing medical literature to provide practical insight into how to accomplish full implementation and evaluation of this new paradigm. They searched Medline and the Educational Resource Information Clearinghouse from the 1960s until the present, reviewed the titles and abstracts of the 469 articles the search produced, and chose 68 relevant articles for full review. The authors found that in the 1970s and 1980s much attention was given to the need for and the development of professional competencies for many medical disciplines. Little attention, however, was devoted to defining the benchmarks of specific competencies, how to attain them, or the evaluation of competence. Lack of evaluation strategies was likely one of the forces responsible for the three-decade lag between initiation of the movement and wide-spread adoption. Lessons learned from past experiences include the importance of strategic planning and faculty and learner buy-in for defining competencies. In addition, the benchmarks for defining competency and the thresholds for attaining competence must be clearly delineated. The development of appropriate assessment tools to measure competence remains the challenge of this decade, and educators must be responsible for studying the impact of this paradigm shift to determine whether its ultimate effect is the production of more competent physicians.
Article
This chapter develops an extension of Doignon and Falmagne's knowledge struc-tures theory by integrating it into a competence-performance conception. The aim is to show one possible way in which the purely behavioral and descriptive knowledge structures approach could be structurally enriched in order to account for the need of explanatory features for the empirically observed solution behav-ior. Performance is conceived as the observable solution behavior of a person on a set of domain-specific problems. Competence (ability, skills) is understood as a theoretical construct accounting for the performance. The basic concept is a mathematical structure termed a diagnostic, that creates a correspondence be-tween the competence and the performance level. The concept of a union-stable diagnostic is defined as an elaboration of Doignon and Falmagne's concept of a knowledge space. Conditions for the construction and several properties of union-stable diagnostics are presented. Finally, an empirical application of the competence-performance conception in a small knowledge domain is reported that shall illustrate some advantages of the introduced modeling approach.
Article
Suppose that Q is a set of problems and S is a set of skills. A skill function assigns to each problem q i.e. to each element of Q — those sets of skills which are minimally sufficient to solve q; a problem function assigns to each set X of skills the set of problems which can be solved with these skills (a knowledge state). We explore the natural properties of such functions and show that these concepts are basically the same. Furthermore, we show that for every family K of subsets of Q which includes the empty set and Q, there are a set S of (abstract) skills and a problem function whose range is just K. We also give a bound for the number of skills needed to generate a specific set of knowledge states, and discuss various ways to supply a set of knowledge states with an underlying skill theory. Finally, a procedure is described to determine a skill function using coverings in partial orders which is applied to set A of the Coloured Progressive Matrices test (Raven, 1965).
Article
The objectives of this project are: (a) to employ a design-based approach by analyzing the teaching and learning components of a defined medical procedural skill (spinal anaesthetic technique) (b) to build and test a Simulated Interactive Learning System (SILS) using a haptic immersive workbench. The intention is to provide an effective training system for trainee doctors in spinal anaesthetic techniques without exposing patients to unnecessary risks.
Article
The concept of a knowledge space is at the heart of a descriptive model of knowledge in a given body of information. Another model explains the observed knowledge of individuals by latent skills. We here reconcile these two underlying approaches by showing that each finite knowledge space can be generated from a skill assignment that is minimal and unique up to an isomorphism. Some more technicalities are required in the infinite case. Part of the results reformulate theorems from the theory of Galois lattices of relations.