Conference PaperPDF Available

Empirical Validation of Concept Maps: Preliminary Methodological Considerations.

Authors:

Abstract and Figures

For their usage in the semantic web, valid ontologies are required for a given domain. Here we focus on ontologies represented as concept maps (semantic nets). For one and the same domain several alternative concept maps may exist, originating from different world views or purposes. Some of these concept maps may be valid, however not all of them. Thus, efforts for validating empirically and objectively concept maps in the respective context are necessary. We outline two methodological approaches for empirically validating concept maps, one for giving evidence of content validity of a concept map and one for application validity. One procedure is to validate a given concept map with concept maps generated systematically by others. For this, persons of different knowledge level, are prompted to externalise their understanding of the domain through a concept mapping task. For giving evidence of the content validity of a given concept map, the similarity between the given concept map and the collected criterion maps has to be examined. As a second method of validating a given concept map, we suggest to observe the performance or behaviour in a relevant situational context as a validation criterion. In this scope, a method for predicting persons' problem solving behaviour by using a given concept map is outlined. In this case, evidence of the application validity of a given concept map can be found by comparing the predicted answer patterns with empirically obtained answer patterns. In general, the purpose and ultimate use of a given concept map has to be taken into consideration for choosing a validation procedure and interpreting its results.
Content may be subject to copyright.
1
Empirical Validation of Concept Maps:
Preliminary Methodological Considerations
Dietrich Albert and Christina M. Steiner
Cognitive Science Section - Department of Psychology
University of Graz
Universitätsplatz 2 / III
8010 Graz, Austria
{dietrich.albert, chr.steiner}@uni-graz.at
Abstract
For their usage in the semantic web, valid
ontologies are required for a given domain. Here
we focus on ontologies represented as concept
maps (semantic nets). For one and the same
domain several alternative concept maps may
exist, originating from different world views or
purposes. Some of these concept maps may be
valid, however not all of them. Thus, efforts for
validating empirically and objectively concept
maps in the respective context are necessary. We
outline two methodological approaches for
empirically validating concept maps, one for
giving evidence of content validity of a concept
map and one for application validity. One
procedure is to validate a given concept map
with concept maps generated systematically by
others. For this, persons of different knowledge
level, are prompted to externalise their
understanding of the domain through a concept
mapping task. For giving evidence of the content
validity of a given concept map, the similarity
between the given concept map and the collected
criterion maps has to be examined. As a second
method of validating a given concept map, we
suggest to observe the performance or behaviour
in a relevant situational context as a validation
criterion. In this scope, a method for predicting
persons’ problem solving behaviour by using a
given concept map is outlined. In this case,
evidence of the application validity of a given
concept map can be found by comparing the
predicted answer patterns with empirically
obtained answer patterns. In general, the purpose
and ultimate use of a given concept map has to
be taken into consideration for choosing a
validation procedure and interpreting its results.
1 Introduction
An ontology allows to structure a domain with respect
to its conceptual organisation. It constitutes a
specification of the concepts in a domain and the
relations among them and thus, defines a common
vocabulary of a knowledge domain (for an overview
see e.g. [Chandrasekaran et al., 1999; Fridman Noy et
al 1997; Mizoguchi, 2003, 2004a, 2004b; Noy et al.,
2001]. Currently, there is still a growing interest in
ontologies and their application in information
technology and computer science. Typical ontology
applications arise in the context of the semantic web
and e-Learning. The Semantic Web in which
information is given well-defined meaning, as an
extension of the current Web, is better enabling
computers and people to work in cooperation
[Berners-Lee et al., 2001]. For making the web
computer-understandable it is necessary to write
metadata for world wide web information and web
services. In this regard, ontologies play the key role,
in order to provide a vocabulary for metadata
descriptions and to allow interoperability among
metadata. In e-Learning one demanding issue is that
of reuseability and interoperability of learning
material, thus on the one hand standardisations (such
as LOM) are required. On the other hand, for e-
Learning ontologies have to be generated and applied
in order to specify the vocabulary of the metadata for
learning objects. On the whole, ontologies serve as a
way to express a common vocabulary, as a help to
information access, as a medium for mutual
understanding, as a specification and as a foundation
of knowledge systematisation [Mizoguchi, 2004a].
There are several alternatives to represent an
ontology. There have been several attempts to create
engineering frameworks for constructing ontologies
[Chandrasekaran et al., 1999], i.e. formal languages
exist for writing an ontology, such as e.g. Ontolingua
or Web Ontology Language (OWL). They allow for
using a logical notation for writing and sharing
ontologies. Aside from such formal languages,
concept maps (semantic nets), which will be the focus
of this paper, provide a natural way of expressing and
presenting ontologies [Hayes et al., 2003]. A concept
map is characterised by a set of concepts and a set of
relationships (or relations) between them. Most
commonly, a concept map is represented as a labelled
directed graph with the vertices (nodes) representing
the concepts and the directed and labelled edges
(arcs) between them representing the relationships
that exist between concepts, such as is-a, part-of etc.
(for an overview see e.g. [Novak, 2001]). In Figure 1
2
an example of a concept map is shown, describing
what a concept map is.
Figure 1. Concept map describing what a concept map is
(adapted from [Ruiz-Primo, 2000])
The combination of two concepts and the link relating
them constitutes a proposition. Hence, another way to
represent a concept map is a list of propositions.
Furthermore, the information contained in a concept
map may also be presented in form of a matrix, with
the set of concepts labelling the columns and rows
and the relations depicted in the cells of the matrix.
Of course, concept maps can be expressed also in a
formal way (e.g. by notations of set theory and logic).
On the whole, concept map representations constitute
a natural way of displaying ontologies [Bokma et al.,
2004; Dicheva et al., 2002; Hayes et al., 2003]. .
Hence, in the sequel we refer to concept maps as
representations of ontologies.
In building a concept map, two different
approaches can be distinguished: a normative and a
descriptive one. The normative approach aims in
obtaining the ‘one and only’ correct concept map of a
specific domain, representing knowledge of complete
consensus. Hence, constructing a concept map in a
normative way assumes, that there exists only one
concept map of completely shared understanding for a
particular domain. Conversely, according to the
descriptive approach, it is assumed that there may
exist alternative concept maps for a specific domain,
representing e.g. differing views or opinions. Hence,
for a given domain different, but not arbitrary concept
maps may exist. In general, this approach seems to be
more reasonable and one should abandon demanding
the ‘one’ correct concept map. A concept map or
ontology necessarily entails some sort of world view
with respect to a given domain [Uschold et al., 1996].
Different concept maps representing the same
domain, even when representing most general
concepts, often differ from each other. Diverging
concept maps may result due to different world views
or alternate possibilities of sub-categorizations
[Chandrasekaran et al., 1999]. Moreover, different
concept maps categorising the same domain may
differ by reason of their purpose and ultimate use.
Therefore, there is not ‘one’ correct concept map for a
specific domain.
Summarizing, according to a descriptive
approach, not arbitrary concept maps of a domain
may exist, but there may exist several valid concept
maps representing the same domain. These
alternatives may differ due to different views or
because of their different purpose. Most likely,
alternative concept maps modelling the same domain
will match with respect to parts of the contained
concepts and relationships or even agree in whole
substructures. The interrelation between such concept
maps is up to further research.
One general aim in constructing concept maps is
to obtain one of possibly several viable and valid
concept maps. Thus, a framework for evaluating the
adequacy of a concept map or different proposals for
a concept map should be provided. The field of
evaluating concept maps is only just emerging. Until
now, there is a lack of ideas and guidelines for
evaluation issues.
[Leclère et al., 2002] claim that the best way for
evaluating a concept map is the application for which
the ontology was designed. This aspect of validity
could be denoted as ‘application validity’ of a concept
map. It refers to the practical usability and usefulness
of a concept map. Thus, different kinds of
applications will require different means of validating
a concept map.
A concept map constitutes a model of a part of
the current knowledge about the world for the given
domain in the given context. Before examining the
quality of a concept map by applying it for the
purpose it is generated, i.e. application validity,
evidence needs to be given regarding its validity as a
model of the respective knowledge. This aspect of
validity we call ‘content validity’. Content validity is
an important issue, as the evaluation of the content of
concept maps is critical for using them. It would be
unwise to publish a concept map, to reuse an existing
concept map, to build a new concept map, and to
implement an application that relies on concept maps
written by others (or even yourself), without first
evaluating it [Gómez-Pérez, 2001]. The content
validity of a concept map refers to the correct
building of the content of the concept map, with the
aim of proving compliance of the world model with
the world modelled. According to [Gómez-Pérez,
2001], for the evaluation of a given concept map, the
following criteria should be considered: consistency,
completeness, conciseness, expandability, and
sensitiveness. Of course, the evaluation whether a
concept map adequately reflects the respective
knowledge will also need to take into consideration
its intended purpose and ultimate use.
3
Subjective evidence for the correctness of a
concept map does not suffice for making a statement
regarding the validity of the concept map. Even the
principle of consensus regarding the correctness of a
concept map does not suffice. In fact, objective and
empirical criteria are needed for getting evidence of
the validity of a given concept map.
To summarize, there has already been done some
work regarding the evaluation and validation of
ontologies and concept maps, but nevertheless there is
still a lack of efficient strategies for validating them
[Breitman et al., 2003]. The validity of a concept map
can be understood from two perspectives. On the one
hand, it may be examined whether a concept map
serves the purpose for that it has been designed
(application validity). On the other hand, before
examining the application validity, it should be
evaluated whether a concept map adequately reflects
the knowledge in question (content validity).
We concentrate on both types of validity and
propose two methods of empirically validating a
concept map (in the sequel denoted as ‘given concept
map’). Regarding the content validity of a concept
map, we propose an approach of validating a given
concept map with empirically collected concept maps.
For examining the application validity of a given
concept map, we suggest to validate the concept map
through performance. After discussing the issue of
reliability in the following section, these two methods
will be outlined in more detail.
2 Reliability of Concept Maps
Before considering the validity, the issue of reliability
of a concept map should be regarded. In this scope the
question arises for a concept map, how it has been
developed. Let us assume an expert of a specific
knowledge domain who constructed a concept map
for this domain. By asking the respective person in a
different point of time to construct a concept map for
the same domain again, the reliability of the concept
map can be examined. If both concept maps
correspond to each other, they reliably represent (the
domain or at least) the understanding of the respective
person regarding the domain. Pre-assuming that no
indication of a changing knowledge or understanding
of the person exists; if the person generates a different
concept map each time, though, it is obvious that
none of them will be a reliable model of the
knowledge of this person. Another aspect of
reliability refers to the consistency of scores assigned
to a concept map, when evaluating it according to
particular characteristics such as number or accuracy
of propositions, number of examples etc. When
having two or more judges that independently from
each other score a concept map according to a
particular scoring system (e.g. [Novak et al., 1984;
Ruiz-Primo et al., 1997]), the interrater reliability can
be determined (e.g. [Ruiz-Primo, 2000]), i.e. whether
different persons consistently score a concept map. In
this regard, interrater reliabilities around .90 could be
found which indicates that different raters are able to
reliably score concept maps [Shavelson et al., In
Press]. By asking to generate a concept map in
alternative forms of representation (e.g. as a directed
graph and as a list of propositions), parallel-forms
reliability could be examined. A reliable concept map
should represent the same model of knowledge for a
person, regardless in which representation format it is
generated.
3 Validating a Concept Map with
Concept Maps of Others: Content
Validity
One approach of validating a given concept map as a
model of knowledge is by taking empirically
collected concept maps (in the sequel denoted as
‘criterion maps’) as a criterion for content validity.
[Cimolino et al., 2002] argue that a concept
mapping task is able to capture a person’s ontology of
a domain. Hence, given a concept map representing
the ontology of a specific domain, it may be validated
by comparing it with empirically collected concept
maps representing personal ontologies of different
individuals. For this, persons of different knowledge
level, including experts, are prompted to externalise
their understanding of the domain through a concept
mapping task. The concept map that is to be validated
is then compared with the concept maps of
individuals.
For posing a concept mapping task, a set of
procedures is available, which will be described in
more detail below. As concept maps for the same
domain may differ due to their intended purpose, a
concept mapping task should include some reference
to the purpose and context when collecting criterion
maps for validation.
3.1 Construct-a-map
One common method is to ask individuals to generate
or construct a concept map concerning a specific
knowledge domain from scratch. This is called
‘construct-a-map’ technique [Ruiz-Primo, 2000]. The
method may vary considerably in practical
application. The concepts and/or relations to be used
for drawing the concept map may have to be
generated or may also be provided. Moreover, it may
either be required to construct a hierarchical or a non-
hierarchical map or no information regarding the
structure is given. The task of constructing a concept
map may be completed individually or in groups. The
maps may be drawn by hand, on the computer, or by
arranging note cards. For evaluating constructed
maps, different scoring systems exist [e.g. Novak et
al., 1984; Ruiz-Primo et al., 1997] considering
different map characteristics such as number of
correct propositions, proposition accuracy, levels of
hierarchy, examples etc.
4
One possibility of validating a given concept map
based on collected criterion maps is to have the given
concept map as well as the criterion maps scored by
several raters. The correlation between the given
concept map and the criterion maps can then be used
in order to give evidence of the validity of the given
concept map.
Another way, which is more direct, is to examine
the similarity of the given concept map to the
criterion maps. For this purpose, the conformance of
propositions, i.e. of pairs of related concepts, can be
examined. For examining the similarity between the
given concept map and the criterion concept maps
suitable measures of association [e.g. Tversky, 1977]
may be chosen and utilized. For example 2x2 tables
can be established for a given concept map and a
criterion map, explicating the number of propositions
that are contained either in both or in only one of
them. Based on this, similarity or distance measures
can be calculated. Moreover, a so-called convergence
score1 can be computed in correspondence to [Ruiz-
Primo, 2000], which expresses the proportion of
propositions in the given concept map out of the total
number of propositions in one criterion map. A high
similarity or convergence between the given concept
map and the criterion maps indicates the content
validity of the given concept map. The same
conclusion can be made using well known advanced
statistical measures of association developed by L.A.
Goodman and W.H. Kruskal.
One prerequisite for the application of the
‘construct-a-map’ method is that persons that are to
undergo this concept mapping task, first need to learn
how to construct concept maps, i.e. they need to
receive some training [Ruiz-Primo, 2000]. This might
be a time-consuming process which probably results
in frustration and rejection of the method [Schau et
al., 2001]. One general problem of this method is, that
there is no universally accepted scoring system for the
evaluation of constructed maps. Therefore, when
using concept map scores for validation, applying and
comparing alternative scoring systems might result in
different findings regarding the validity of a concept
map. Thus, the direct method is preferable, i.e.
validating a concept map by examining the similarity
between the given concept map and the criterion map
in terms of measures of similarity, convergence or
association.
3.2 Fill-in-the-map
Another possibility of posing a concept mapping task
is a method called ‘fill-in-the-map’ or ‘map
completion’, respectively [Ruiz-Primo, 2000; Schau
et al., 2001]. For this, a concept map is provided,
where all or some of the concepts and/or relations
1 In the present context, the convergence C of a given
concept map to one criterion map is calculated in the
following way: C = x / y, whereas x denotes the number of
propositions in the given concept map that are also
contained in the criterion map and y denotes the total
number of propositions in the criterion map.
have been left out. The blanks have to be filled in
requiring either the generation of the words to use or
by selecting them from a list which may include
distractors. When collecting criterion maps with this
method, the concept map to be validated then can be
compared with the criterion maps. This would be
done again, by examining the similarity, e.g. by
calculating typical measures of association.
Some researchers claim that this method should
be preferred to the ‘construct-a-map’ technique, as the
least is assumed to impose a too high cognitive
demand and the generated maps may highly depend
on a person’s communication skills [Schau, et al.
2001]. Map completion tasks can easily and quickly
be administered. However, this method poses the
problem that only a part of the knowledge can be
queried, as it is necessary to provide at least a small
part of the concepts and/or relations in the map to be
filled in. As a representation of the domain structure
is provided, a person does not have to create an
individual structure representing his/her personal
ontology of the domain. Therefore, the validity of a
whole given concept map is likely to be
overestimated, when using a map completion task for
collecting concept maps of individuals. On the other
hand, the ‘fill-in-the-map’ technique has the
advantage of allowing to validate specific parts of a
concept map that are of special interest. This would
be useful, for example, if only the relations of a
concept map are to be validated. The map completion
method would also be of special advantage, if two
proposals for a concept map of a domain are to be
evaluated. Such two concept maps will most probably
overlap in parts of their propositions. For deciding
which one of the two maps is the more adequate one,
criterion maps could be collected by posing a map
completion task querying for those parts that are not
in the intersection of the two maps. When the
validation of particular parts of a given concept map
is intended, this needs to be taken into account for
selecting and applying the methods for statistical
analysis. Furthermore, the number of words that have
to be filled in should also be regarded, as a map
completion task implies – in contrast to the
‘construct-a-map’ technique - a constraint with
respect to the number of concepts and relations that
are contained or requested in the concept map.
3.3 Relatedness ratings of pairs of concepts
One further alternative that can be classified as a
concept mapping task is characterized by a two-stage
indirect approach [Schau et al., 1997]. This technique
is based on relatedness ratings between pairs of
concepts, i.e. individuals are asked to rate the degree
of relatedness between pairs of previously defined
concepts on a numerical scale. The resulting
relatedness matrix then can be used to visually
represent the personal ontology of the respective
person in form of a concept map. This is done
through the application of an algorithm, e.g. by using
the Pathfinder software [Schvaneveldt, 1990], which
allows to create a network representation based on
5
relatedness ratings. Notice, that the generated maps
based on relatedness ratings do not include labelled
relations, but only indicate the degree of relatedness
through the distance between two given concepts.
When collecting criterion maps for validating a given
concept map in this way, this fact has to be taken into
account. Hence, only a particular aspect of validity of
the given concept map can be examined, namely the
existence of relationships between concepts. The type
of the relations between the concepts of the given
concept map, however, can not be validated in this
case.
In order to overcome the drawback of this
method, [Shavelson et al., In Press] propose that after
analysing the relatedness ratings, the resulting
network representation can be provided to the
individual. The person then will be asked to label the
relations between the concepts, and if necessary to
add or remove relations. Based on concept maps
collected in this way, statements about the content
validity of a given concept map could also be made
with respect to the type of relations that exist between
concepts.
In both cases, i.e. when collecting the criterion
concept maps with the original method or with the
extended method proposed by [Shavelson et al., In
Press], evidence on the validity of a given concept
map can be given again by examining the similarity
or conformance to the criterion maps, e.g. by utilizing
appropriate measures of association.
3.4 Proposition correct-incorrect discrimination
task
Another technique for posing a concept mapping task
makes use of the representation of a concept map
through a list of propositions. The propositions of a
concept map representing a particular domain may be
presented to individuals as a correct-incorrect
discrimination task. This technique has been
implemented in the concept mapping software ‘Cmap
Pro’ [Bernd et al., 2000] in order to assess previous
knowledge before a learning sequence takes place. In
[Steiner, 2004] an extension of this method was
proposed and applied, which will be described in the
following. In addition to using the propositions from
the concept map, distractor items are included in the
correct-incorrect discrimination task. In order to
reduce the risk of lucky guesses, a confidence rating
is required for each proposition judgement. Moreover,
a phase of answer checking can be realised. For this,
after finishing the correct-incorrect discrimination
task, for each subject an individual concept map is
drawn with respect to his/her answer pattern. Such an
individual concept map represents the personal
ontology of the respective person, containing all those
propositions that have been judged as being ‘correct’.
Notice that, of course limited by the presented
distractor items, such an individual concept map may
also include misconceptions if distractor items are
wrongly judged as ‘correct’. Based on the individual
concept map, each person then has the opportunity to
check his/her answers and make corrections, if
required. Applying this technique seems easy and
comfortable and does not need any foregoing training.
Moreover, it is possible to query all the propositions
contained in a concept map, which is not possible
when using the fill-in-the-map technique. This
method can also be applied for collecting criterion
maps for the validation of a given concept map. This
technique allows to validate a whole concept map as
well as specific parts of a concept map that are of
special interest. When using this method for
collecting criterion maps, the propositions of the
given concept map will be presented as correct-
incorrect discrimination task in combination with
distractor items, which could be actually incorrect
statements but also statements representing an
alternative world view. The validity of the given
concept map then can be examined by considering its
similarity to the collected criterion maps through
measures of association etc.
Summarising, there are various ways of presenting a
concept mapping task which considerably vary in the
extend of constraints imposed and information
provided to persons, respectively. Based on these
techniques it is possible to empirically collect
criterion maps representing the personal ontologies of
individuals with different knowledge level. These
criterion maps then may be utilised for examining the
content validity of a given concept map, by
considering the similarity or conformance to the
criterion maps. A high association between the given
and the criterion maps indicates the validity of the
given concept map.
4 Validating a Concept Map with
Performance: Application Validity
For examining the application validity of a given
concept map, we propose making use of observing
situational behaviour or performance as another
conceivable and valuable criterion of validation. In
the present context, situational performance refers to
behaviour in real-world situations which does not
consist in performing a concept mapping task. The
performance in question is for instance solving
problems, answering questions, or even social
behaviour in given situations. For this validation
approach, we have a concept map that needs to be
validated on the one hand. On the other hand we have
empirically collected situational performance scores
or profiles of people of different knowledge level,
including experts. These performance measures are
related to the purpose and intended application of the
given concept map. In this case the personal
ontologies are not elicited directly through a concept
mapping task. Indirectly, the performance of a person
- e.g. in a problem solving context - is controlled by
his/her individual knowledge and understanding
regarding the ontology of a domain. Situational
performance as indicating indirectly knowledge is
6
therefore useable as a criterion for examining the
validity of a concept map. When using situational
performance as a criterion for validating a concept
map, the purpose of the given concept map has to be
kept in mind. Due to different purposes that are
intended for a given concept map, different kinds of
situational performance will be appropriate for
validation.
Until now there have been conducted a number
of investigations that tried to validate individual
concept maps on the basis of situational performance.
This means, individuals completed a concept mapping
task on the one hand and underwent a context of
situational performance, as e.g. doing a multiple
choice test. Taking the performance scores as a
criterion, it was tried to give evidence of the validity
of the individual concept maps. When using multiple
choice scores as a criterion, the findings of [Schau et
al., 2001] and [Rice et al., 1998] indicated a high
validity (.75) of individual concept maps. [Ruiz-
Primo, 2000] also obtained relatively high validity
scores (on average .57). [West et al. 2000], who used
standardized tests as a validation criterion, reported
indications of low validity of individual concept
maps. Conversely, [Rice et al. 1998] received high
validity scores (.82-.87) for individual concept maps
when using standardized tests as a criterion. [Schultz,
1999, cited from Shavelson et al., In Press] applied
different criterions based on performance (e.g.
multiple choice test, reading test) and reported
moderately high validity scores (ranging from .43 to
.60) for individual concept maps. [Jöckel et al., 1999]
conducted a study in the context of geometry and
spatial ability. Individual concept maps were
collected by asking participants to generate maps with
note cards. As a criterion for validating these
individual concept maps, problem solving
performance referring to the knowledge domain was
used. The individual concept maps did not prove to be
valid in this investigation applying performance as
validity criterion.
[Steiner, 2004] also used problem solving
performance as a criterion for validating individual
concept maps. This validation approach was
conducted in the knowledge domain of geometry of
right triangles. The individual concept maps were
collected by using the concept mapping task outlined
in the ‘proposition correct-incorrect discrimination
task’ section. Furthermore, typical geometry problems
coming from the respective domain were posed to the
participants of the investigation. The data collected
from subjects’ problem solving performance was then
utilised for examining the validity of the individual
concept maps. The results indicated a high validity of
the individual concept maps (.67-.74). Hence, the
individual concept maps validly represented personal
ontologies of individuals.
Summarising, it could be shown that measures of
situational performance are suitable for validating
individual concept maps or at least, that individual
concept maps and performance are somehow
positively interrelated. It seems natural and obvious,
that an individual’s personal understanding of the
ontology of a domain is reflected in his/her
performance in given situations, at least as declarative
knowledge is involved. Hence, we are in the position
that a given concept map can not only be validated
with concept maps of persons of different knowledge
level, but also with their performances. In the sequel
we will describe such an approach in more detail.
As in the suggested validation approach,
Knowledge Space Theory [Albert et al., 1999;
Doignon et al., 1985, 1999; Falmagne et al., 1990] is
implemented, let us first give a very short
introduction into the basic notions of this theoretical
framework. Knowledge Space Theory provides a
formal model for structuring a domain of knowledge
and for representing the knowledge of individuals
based on prerequisite relationships. Within this
theory, a domain of knowledge is characterised by a
set Q of problems. The knowledge of a learner is
represented by the subset of problems that he/she is
capable of solving. Due to mutual dependencies
between the problems of a domain, from the correct
solution of certain problems the mastery of other
problems can be surmised. In order to capture the
relationships between the problems of a domain the
notion of a surmise relation has been introduced. Two
problems a and b are in a surmise relation whenever
from a correct solution to problem b the mastery of
problem a can be surmised. In other words, problem a
is a prerequisite problem for problem b. A surmise
relation can be illustrated by a so-called Hasse
diagram (see Figure 2 for an example), where
descending sequences of line segments indicate a
surmise relationship.
Figure 2: Example of a Hasse diagram illustrating a
surmise relation on a knowledge domain Q = {a, b, c, d, e}
(adapted from [Falmagne et al., 1990])
According to the surmise relation depicted in Figure
2, from a correct solution to problem b the correct
solution to problem a can be surmised, while the
mastery of problem e implies correct answers to
problems a, b, and c. The surmise relation forms a
quasi-order on the set of problems and restricts the
number of possible knowledge states (i.e. subsets of
problems) that are expected to be observable. The
collection of all possible knowledge states, including
the empty state Ø and the whole set Q, constitutes the
so-called knowledge structure. The knowledge
structure corresponding to the surmise relation shown
in Figure 2 is given by
b
d
c
e
a
7
K = {Ø, {a}, {c}, {a, c}, {a, b}, {a, b, c}, {a, b, d},
{a, b, c, e}, {a, b, c, d}, Q}.
Let us now assume a concept map of a particular
knowledge domain that has to be validated. As a
criterion for validation, situational performance, in
particular in a problem solving context, is intended to
be applied. A set of typical and representative
problems of the domain is chosen. To each problem
the subset of propositions of the given concept map is
assigned, that represents those semantic knowledge
elements that are required for solving the respective
problem. Each proposition can be considered as an
atomic competency or skill [e.g. Falmagne et al.,
1990; Doignon, 1994; Düntsch et al., 1995; Korossy,
1997, 1999] required for solving the problems of the
domain. The subsets of propositions assigned to the
problems will most likely overlap. Based on the
representation of the problems by subsets of the given
concept map, dependencies between problems in
terms of a surmise relation can be derived. This could
be done e.g. by set inclusion, i.e. if the representation
of a problem a in the concept map is a subset of that
of problem b, then problem a is a prerequisite for
problem b. From the derived dependencies between
the problems a knowledge structure can be
established, i.e. the possible knowledge states can be
identified. This means, that we are able to predict
specific answer patterns out of all possible subsets of
items (2IQI). Based on this, we can investigate
empirically whether the observed answer patterns
correspond to the identified and predicted knowledge
states. For this, the problems are posed to individuals
of different knowledge level. The answer patterns that
are obtained empirically then are compared to the
predicted knowledge states. As these knowledge
states have been formed in the basis of the problems’
representations on the concept map, the answer
patterns serve as a criterion for validating the given
concept map. Comparing the empirically obtained
answer patterns with the knowledge states can be
done e.g. by using a discrepancy index, describing the
similarity between the knowledge structure and the
set of answer patterns. If the answer patterns
correspond well to the predicted knowledge states, the
given concept map can be considered also to be valid
– under the condition that both, the chosen set of
problems as well as the sample of persons are
adequate and representative.
5 Discussion
In general, it does not suffice to subjectively evaluate
a concept map and judge it as valid by subjective
evidence or uncontrolled consensus. In fact, objective,
empirical measures or criteria are needed to give
evidence of a concept map’s validity. Efforts for
empirically and objectively validating concept maps
have proven reasonable and promising.
In this paper we suggested two approaches for
empirically validating a given concept map, one for
giving evidence of content validity of a given concept
map and one for application validity. The suggested
approach concerning content validity is to validate a
given concept map by taking empirically collected
concept maps of others as a criterion. The approach
proposed regarding application validity applies
situational performances as a criterion. Both
approaches constitute useful and valuable procedures
that can complement each other in order to build up a
coherent picture about the validity of a given concept
map. Of course, there are also other possibilities for
validating concept maps, such as e.g. consulting the
published literature of the given knowledge domain
for examining content validity.
According to the suggested validation
approaches, it is not necessary to only query experts
in a given field for which a concept map is to be
validated. In fact, we propose to rather involve
persons of different knowledge level, who will
probably afterwards work (indirectly) with the
validated concept map.
Regarding the validation of a given concept map
it is very important to take into account the purpose
and ultimate use of the concept map (e.g. predicting
problem solving behaviour). This refers to validation
issues concerning application validity as well as
content validity. The intended purpose of a given
concept map has to be regarded when collecting
concept maps as validation criterion, i.e. the concept
mapping task should refer to the context and purpose
of the concept map. The choice of a measure of
situational performance will likewise depend on the
purpose that a given concept map will serve in the
future.
When applying validation methods for a given
concept map, it has to be specified which aspects are
intended to be validated (e.g. the whole concept map,
the concepts, the relations, substructures of the given
concept map). Another important issue is to choose
appropriate statistical measures, depending on the
particular criterion for validity and its constrains.
Finally, we want to point out, that there is little
attention paid to the reliability of concept maps.
Before considering the validity of a given concept
map, it should be examined whether the generated
model of knowledge is reliable, i.e. whether the
concept map looks the same when it is constructed
again or by a parallel method. Hence, the issue of
reliability or at least the circumstances under which
the concept map originated, should be taken into
account when the validity of a concept map is
addressed.
We outlined, that for a given knowledge domain
there might be several alternative concept maps that
validly represent the respective knowledge of the
domain. Such alternative concept maps may originate
from different world views or opinions, but may also
be due to different purposes of the maps. Based on
this fact two interrelated questions arise:
8
- Should alternative ontologies be realised in the
context of the semantic web?
- If alternative ontologies are not to be
implemented into the semantic web, which one
should be realised?
These two questions are going far beyond the scope
of validating ontologies in form of concept maps and
give raise to future research.
References
[Albert et al., 1999] Albert, D. & Lukas, J. (1999).
Knowledge Spaces: Theories, Empirical
Research, and Applications. Mahwah, NJ:
Lawrence Erlbaum Associates.
[Bernd et al., 2000] Bernd, H., Hippchen, T., Jüngst,
K.-L., & Strittmatter, P. (2000). Durcharbeiten
von Begriffsstrukturdarstellungen in
unterrichtlichen und computergestützten
Lernumgebungen. In H. Mandl & F. Fischer
(Eds.), Wissen sichtbar machen.
Wissensmanagement mit Mapping-Techniken (pp
15-36). Göttingen: Hogrefe.
[Berners-Lee et al., 2001] Berners-Lee, T., Hendler,
J., & Lassila, O. (2001). The Semantic Web.
Scientific American 284, 34-43.
available at < http://www.sciam.com/print_
version.cfm?articleID=00048144-10D2-1C70-
84A9809EC588EF21 > (30 March 2005)
[Bokma et al., 2004] Bokma, A., Solomon, N.,
Harvey, H., & Jones, S. (2004). NOPIK: Creating
Shared Conceptualisations for Information and
Knowledge Sharing in the Virtual Enterprise.
Proceedings of the 10th International Conference
on Concurrent Enterprising, Seville, June 2004.
available at <http://voyager.sunderland.ac.uk/
Downloads/nopik_ICE.pdf> (10 May 2005)
[Breitman et al., 2003] Breitman, K. & Leite, J. C. S.
(2003). Ontology as a Requirement Engineering
Product. In Proceedings of the 11th IEEE
International Requirements Engineering
Conference. 8-12 Sept. 2003, Monterey Bay,
California, USA. (pp. 309-319).
[Chandrasekaran et al., 1999] Chandrasekaran, B.,
Josephson, J.R., & Benjamins, V.R. (1999).
What Are Ontologies and Why Do We Need
Them?. IEEE Intelligent Systems, 14 (1), 20-26.
available at <http://www.infofusion.buffalo.edu/
conferences_and_workshops/ontology_wkshop_
2/ont_ws2_working_materials/ChandrasekaranR
oleofOntology.pdf> (30 March 2005)
[Cimolino et al., 2002] Cimolino, L & J Kay, (2002).
Verified Concept Mapping for Eliciting
Conceptual Understanding. In Aroyo, L &
Dicheva, D, (Eds.), Proceedings ICCE Workshop
on Concepts and Ontologies in Web-based
Educational Systems, ICCE 2002, International
Conference on Computers in Education, CS-
Report 02-15 Technische Universiteit Eindhoven
(pp. 9-14).
Proceedings available at <http://www.win.tue.nl/
~laroyo/ICCE2002_Workshop/proc-Workshop-
ICCE2002.pdf> (30 March 2005)
[Dicheva et al., 2002] Dicheva, D. & Aroyo, L.
(2002). Concept-based Courseware Authoring:
An Engineering Perspective. Proceedings of the
IEEE International Conference on Advanced
Learning Technologies (ICALT 2002) (pp. 46-
49).
available at <http://lttf.ieee.org/icalt2002/
proceedings/t108_icalt099_End.pdf> (30 March
2005)
[Doignon, 1994] Doignon, J. (1994). Knowledge
spaces and skill assignments. In G. H. Fischer &
D. Laming (Eds.), Contributions to Mathematical
Psychology, Psychometrics and Methodology
(pp. 111–121). New York: Springer–Verlag.
[Doignon et al., 1985] Doignon, J.-P. & Falmagne, J.-
C. (1985). Spaces for the assessment of
knowledge. International Journal of Man-
Machine Studies, 23, 175-196.
[Doignon et al., 1999] Doignon, J.-P. & Falmagne, J.-
C. (1999). Knowledge Spaces. Berlin: Springer.
[Düntsch et al., 1995] Düntsch, I., & Gediga, G.
(1995). Skills and Knowledge Structures. British
Journal of Mathematical and Statistical
Psychology, 48, 9–27.
[Falmagne et al., 1990] Falmagne, J.-C., Koppen, M.,
Villano, M., Doignon, J.-P., & Johannesen, L.
(1990). Introduction to Knowledge Spaces: How
to Build, Test, and Search Them. Psychological
Review, 97, 201-224.
[Fridman et al., 1997] Fridman Noy, N. F. & Hafner,
C.D. (1997). The State of the Art in Ontology
Design. A Survey and Comparative Review. AI
Magazine, 18 (3), 53-74.
[Gómez-Pérez, 2001] Gómez-Pérez, A. (2001).
Evaluation of Ontologies. International Journal
of Intelligent Systems, 16, 391-409.
[Hayes et al., 2003] Hayes, P., Saavedra, R.,
Reichherzer, T. (2003). A Collaboration
Development Environment for Ontologies.
Proceedings of the Semantic Integration
Workshop, Sanibel Island, Florida, October, 20,
2003.
available at <http://sunsite.informatik.rwth-
aachen.de/Publications/CEUR-WS//Vol-
82/SI_demo_05.pdf> (10 May 2005)
[Jöckel et al., 1999] Jöckel, S. & Reiss, K. (1999).
Ergebnisse des DFG-Projekts
„Problemlösestrategien bei räumlich
geometrischen Aufgaben mit konkretem oder
9
computersimuliertem Material“. Universität
Oldenburg: Fachbereich Mathematik.
available at <http://www.math.uni-oldenburg.de/
didaktik/strategie/bericht.html> (30 March 2005)
[Korossy, 1997] Korossy, K. (1997). Extending the
theory of knowledge spaces: a competence-
performance approach. Zeitschrift für
Psychologie, 205, 53-82.
[Korossy, 1997] Korossy, K. (1999). Modeling
knowledge as competence and performance. In
D. Albert & J. Lukas (Eds.), Knowledge Spaces:
Theories, Empirical Research, Applications (pp.
103-132). Mahwah, NJ: Lawrence Erlbaum
Associates.
[Leclère et al., 2002] Leclère, M., Trichet, F., &
Furst, F. (2002). Operationalizing domain
ontologies: towards an ontological level for the
SG family. In Supplementary proceedings of 10th
International Conference On Conceptual
Structures (ICCS'2002), Borovets, Bulgaria.
[Mizoguchi, 2003] Mizoguchi, R. (2003). Tutorial on
ontological engineering - Part 1: Introduction to
Ontological Engineering. New Generation
Computing, 21, 365-384.
available at <http://www.ei.sanken.osaka-
u.ac.jp/pub/miz/Part1-pdf2.pdf> (30 March
2005)
[Mizoguchi, 2004a] Mizoguchi, R. (2004a). Tutorial
on ontological engineering - Part 2: Ontology
development, tools and languages. New
Generation Computing, 22, 61-96.
available at <http://www.ei.sanken.osaka-
u.ac.jp/pub/miz/Part2V3.pdf> (30 March 2005)
[Mizoguchi, 2004b] Mizoguchi, R. (2004b). Tutorial
on ontological engineering - Part 3: Advanced
course of ontological engineering. New
Generation Computiing, 22, 196-220.
available at <http://www.ei.sanken.osaka-
u.ac.jp/pub/miz/Part3V3.pdf> (30 March 2005)
[Novak, 2001] Novak, J.D. (2001). The Theory
Underlying Concept Maps and How To Construct
Them.
available at <http://cmap.coginst.uwf.edu/info/
printer.html> (10 May 2005)
[Novak et al., 1994] Novak, J.D. & Gowin, B.D.
(1984). Learning how to learn. New York:
Campridge University Press.
[Noy et al., 2001] Noy, N. F. & McGuinness, D.L.
(2001). Ontology Development 101: A Guide to
Creating Your First Ontology. Technical Report
KSL-01-05. Stanford University.
available at <http://www.ksl.stanford.edu/people/
dlm/papers/ontology-tutorial-noy-
mcguinness.pdf> (30 March 2005)
[Rice et al., 1998] Rice, D.C., Ryan, J.M, & Samson,
S.M. (1998). Using Concept Maps to Assess
Student Learning in the Science Classroom: Must
Different Methods Compete? Journal of
Research in Science Teaching, 35, 1103-1127.
available at <http://www.clab.edc.uoc.gr/hy302/
papers%5Cconcept_map%20assesment.pdf> (30
March 2005)
[Ruiz-Primo, 2000] Ruiz-Primo, M.A. (2000). On the
Use Of Concept Maps As An Assessment Tool in
Science: What We Have Learned so Far. Revista
Electrónica de Investigación Educativa, 2.
available at <http://redie.uabc.mx/vol2no1/
contents-ruizpri.html> (30 March 2005)
[Ruiz-Primo et al., 1997] Ruiz-Primo, M.A., Schultz,
S.E., & Shavelson, R.J. (1997). Concept map-
based assessment in science: two exploratory
studies. CSE Technical Report 436. Los Angeles:
University of California, Center for the Study of
Evaluation (CSE), National Center for Research
on Evaluation, Standards, and Student Testing.
available at <http://www.stanford.edu/dept/
SUSE/SEAL/Reports_Papers/TECH436.pdf> (30
March 2005)
[Schau et al., 1997] Schau, C. & Mattern, N (1997).
Assessing Students’ Connected Understanding of
Statistical Relationships. In I. Gal & J.B. Garfield
(Eds.), The Assessment Challenge in Statistics
Education (pp. 91-104). Amsterdam: IOS Press.
[Schau et al., 2001] Schau, C., Mattern, N., Zeilik,
M., Teague, K. W. & Weber, R. (2001). Select-
and-Fill-In Concept Map Scores as a Measure of
Students’ Connected Understanding of Science.
Educational and Psychological Measurement,
61, 136-158.
[Schvaneveldt, 1990] Schvaneveldt, R.W. (1990).
Pathfinder associative networks: Studies in
knowledge organization. Norwood: Ablex.
[Shavelson et al., In Press] Shavelson, R.J., Ruiz-
Primo, M.A., & Wiley, E. (In Press). Windows
into the Mind. International Journal of Higher
Education.
[Steiner, 2004] Steiner, C.M. (2004). Beziehung
zwischen Concept Maps und
Kompetenzstrukturen. Unpublished diploma
thesis: University of Graz.
[Tversky, 1977] Tversky, A. (1977). Features of
similarity. Psychological Review, 84, 327-352.
[Uschold, 1996] Uschold, M & Gruninger, M (1996).
Ontologies: Principles, Methods, & Application.
Knoweledge Engineering Review, 11 (2), 93-136.
[West et al., 2000] West, D., Pomeroy, J.R., Park, J.,
Gerstenberger, E., & Sandoval, J. (2000) Critical
thinking in graduate medical education. A Role
for Concept Mapping Assessment? Journal of the
American Medical Association, 284, 1105-1110.
... When considering the validity of concept maps, two types can be distinguished: content validity and application validity (Albert & Steiner, 2005a, 2005b. To ensure the wellfoundedness of a concept map, prior to its use both validity aspects should be considered. ...
... It has to be determined whether the concept map constitutes a valid model of a part of the current knowledge about the world (e.g. Albert & Steiner, 2005a, 2005bAlmeida, 2009;Gómez-Pérez, 2001). The notion of content validation is also in line with the idea of "semantic evaluation", i.e. the detection to what degree a created ontology reflects the knowledge of a domain, as outlined by Zouaq and Nkambou (2009). ...
... For examining the content validity of a concept map, empirically collected concept maps representing personal knowledge (in the sequel denoted as "criterion maps") may be utilised as a criterion and basis for comparison (Albert & Steiner, 2005a, 2005bAlbert et al., 2006). These criterion maps are collected from individuals of different knowledge level, in order to gather a comprehensive picture of personal understandings of the domain. ...
Article
Full-text available
Ontologies play an important role as knowledge domain representations in technology-enhanced learning and instruction. Represented in form of concept maps they are commonly used as teaching and learning material and have the potential to enhance positive educational outcomes. To ensure the effective use of an ontology representing a knowledge domain it needs to be validated. In this paper a previously presented validation methodology for concept maps is exemplified. Two different types of concept map validity are distinguished, referring to the correctness of the concept map’s content (content validity) and to the applicability of the concept map for its designated purpose (application validity), like its use in intelligent tutoring. To demonstrate the usefulness of the two validation types and approaches, they are illustrated by an empirical study. The content validity of a concept map on elementary geometry has been investigated by comparing it with empirically collected criterion maps through s...
... In general, the following advantages of "fill-in-the-map" tasks are mentioned by different researchers: a) students learn to complete "fill-in-the-map" tasks quickly and like completing them (Schau, Mattern, Zeilik, Teague, and Weber, 2001), b) a simple accepted scoring system exists (Himangshu and Cassata-Widera, 2010;Schau et al., 2001), so tasks can be easily and quickly administered (Albert and Steiner, 2005), c) students with lower levels of communication skills can complete tasks where concepts should be inserted in a CM (Schau et al., 2001), d) "fill-inthe-map" tasks allow validation of specific parts of a CM, for example, only relations (Albert and Steiner, 2005), e) tasks where students should provide only linking phrases reduce cognitive load and uncertainty (Gurlitt, Renkl, Motes, and Hauser, 2006), f) "fill-in-the-map" tasks most correlate with traditional assessment methods like standardized tests, multiple choice tests, and instructor designed tests (Himangshu and Cassata-Widera, 2010), and g) students with lower levels of expertise benefit more from tasks where they need to provide linking phrases (Gurlitt, Renkl, Faulhaber, and Fischer, 2007). ...
... In general, the following advantages of "fill-in-the-map" tasks are mentioned by different researchers: a) students learn to complete "fill-in-the-map" tasks quickly and like completing them (Schau, Mattern, Zeilik, Teague, and Weber, 2001), b) a simple accepted scoring system exists (Himangshu and Cassata-Widera, 2010;Schau et al., 2001), so tasks can be easily and quickly administered (Albert and Steiner, 2005), c) students with lower levels of communication skills can complete tasks where concepts should be inserted in a CM (Schau et al., 2001), d) "fill-inthe-map" tasks allow validation of specific parts of a CM, for example, only relations (Albert and Steiner, 2005), e) tasks where students should provide only linking phrases reduce cognitive load and uncertainty (Gurlitt, Renkl, Motes, and Hauser, 2006), f) "fill-in-the-map" tasks most correlate with traditional assessment methods like standardized tests, multiple choice tests, and instructor designed tests (Himangshu and Cassata-Widera, 2010), and g) students with lower levels of expertise benefit more from tasks where they need to provide linking phrases (Gurlitt, Renkl, Faulhaber, and Fischer, 2007). ...
... However, the fact that the structure of a CM is already given to students underlies the main drawback of these CMTs. Students do not represent their unique cognitive structure (Albert and Steiner, 2005;Schau et al., 2001). Therefore, only a part of knowledge can be queried because the other part is already given in a task (Albert and Steiner, 2005). ...
Article
Full-text available
There is a broad set of concept mapping tasks which can be used by teachers for assessing students’ knowledge structures but there are no known works of research studying their adaptation to students’ individual differences and preferences in relation to different tasks. The paper reports first results of activities which aim to offer a solution of the abovementioned problem. It is based on extensive review of theoretical and empirical studies and focuses on: a) identification of advantages and drawbacks of different concept mapping tasks, and b) finding factors which may affect students’ concept mapping ability.
... The other way is to observe the behavior in a relevant situational context. This could be summarized as application validity approach [15]. ...
Chapter
Reliability has always been an important factor for any manufacturing companies. An appropriate level of reliability in a manufacturing system could mean less maintenance cost, higher efficiency and steadier production state. Because each machine in a manufacturing system has its individual level of reliability, reliability on the system level would depend largely on how the machines are configured. In traditional efforts on reconfigurable manufacturing system (RMS) reliability assessments, mean time between failure (MTBF) is usually adopted as reference index of reliability. Also, in existing research efforts of applying reliability analysis in manufacturing system, reliability is merely a single and over-simplified index in the framework. But there exist various forms of reliability inside a RMS, and the complexity of this concept would keep increasing with the development of new technology in manufacturing industry. To analyze reliability in RMS in a more comprehensive way, we built conceptual maps as first step for research agenda -- from the perspective of general layer, reliability analysis and RMS.
... Concept maps is defined as a model of current knowledge about the world in a given domain and given context (Albert and Steiner, 2005). Compared to other graphical model, concept map provides a more natural way of expressing and presenting domain ontologies. ...
Article
Full-text available
The purpose of this paper is to compare two types of graphical models that are widely being used nowadays. The models are called conceptual graph and concept map. Although the names of these models are similar, however, the characteristics of each model are different. A number of 18 papers are compared and important elements for these models are discussed in these papers. This paper is divided into eight sub-topics. The output of this paper is the comparison of characteristics and usage of these two graphical models. In the last part of this paper, a conclusion of the models is made to give better view of it.
... These maps could not be used with students without prior validation [19]. Therefore, before starting the experiment with the students, both concept maps were tried out with four lecturers: #1, #4, #6 and #8. ...
Conference Paper
Full-text available
The Input/Output topic is mandatory in the Computer Architecture branch of the computing curricula. However, in our experience it is a rather complex topic for students to understand. This paper presents the process followed to produce some concept maps that represent the theory we, as lecturers, would like students to learn, and shows the differences found between these concept maps and the concept maps produced by several representative students selected among the students attending the course during the school year 2011-2012. There is also a comparison made between the scores obtained by these representative students in the exam and the results of the concept maps. Finally, a discussion is proposed. This type of study is new to the researchers and the objective of the discussion is to gather knowledge on how well the research has been developed so far, and how to continue.
Book
The five-volume set IFIP AICT 630, 631, 632, 633, and 634 constitutes the refereed proceedings of the International IFIP WG 5.7 Conference on Advances in Production Management Systems, APMS 2021, held in Nantes, France, in September 2021.* The 378 papers presented were carefully reviewed and selected from 529 submissions. They discuss artificial intelligence techniques, decision aid and new and renewed paradigms for sustainable and resilient production systems at four-wall factory and value chain levels. The papers are organized in the following topical sections: Part I: artificial intelligence based optimization techniques for demand-driven manufacturing; hybrid approaches for production planning and scheduling; intelligent systems for manufacturing planning and control in the industry 4.0; learning and robust decision support systems for agile manufacturing environments; low-code and model-driven engineering for production system; meta-heuristics and optimization techniques for energy-oriented manufacturing systems; metaheuristics for production systems; modern analytics and new AI-based smart techniques for replenishment and production planning under uncertainty; system identification for manufacturing control applications; and the future of lean thinking and practice Part II: digital transformation of SME manufacturers: the crucial role of standard; digital transformations towards supply chain resiliency; engineering of smart-product-service-systems of the future; lean and Six Sigma in services healthcare; new trends and challenges in reconfigurable, flexible or agile production system; production management in food supply chains; and sustainability in production planning and lot-sizing Part III: autonomous robots in delivery logistics; digital transformation approaches in production management; finance-driven supply chain; gastronomic service system design; modern scheduling and applications in industry 4.0; recent advances in sustainable manufacturing; regular session: green production and circularity concepts; regular session: improvement models and methods for green and innovative systems; regular session: supply chain and routing management; regular session: robotics and human aspects; regular session: classification and data management methods; smart supply chain and production in society 5.0 era; and supply chain risk management under coronavirus Part IV: AI for resilience in global supply chain networks in the context of pandemic disruptions; blockchain in the operations and supply chain management; data-based services as key enablers for smart products, manufacturing and assembly; data-driven methods for supply chain optimization; digital twins based on systems engineering and semantic modeling; digital twins in companies first developments and future challenges; human-centered artificial intelligence in smart manufacturing for the operator 4.0; operations management in engineer-to-order manufacturing; product and asset life cycle management for smart and sustainable manufacturing systems; robotics technologies for control, smart manufacturing and logistics; serious games analytics: improving games and learning support; smart and sustainable production and supply chains; smart methods and techniques for sustainable supply chain management; the new digital lean manufacturing paradigm; and the role of emerging technologies in disaster relief operations: lessons from COVID-19 Part V: data-driven platforms and applications in production and logistics: digital twins and AI for sustainability; regular session: new approaches for routing problem solving; regular session: improvement of design and operation of manufacturing systems; regular session: crossdock and transportation issues; regular session: maintenance improvement and lifecycle management; regular session: additive manufacturing and mass customization; regular session: frameworks and conceptual modelling for systems and services efficiency; regular session: optimization of production and transportation systems; regular session: optimization of supply chain agility and reconfigurability; regular session: advanced modelling approaches; regular session: simulation and optimization of systems performances; regular session: AI-based approaches for quality and performance improvement of production systems; and regular session: risk and performance management of supply chains *The conference was held online.
Article
Concept maps have a long history in educational settings as a tool for teaching, learning, and assessing. As an assessment tool, they are predominantly used to extract the structural configuration of learners’ knowledge. This article presents an investigation of the knowledge structures of a large group of beginning CS students. The investigation is based on a method that collects, aggregates, and automatically analyzes the concept maps of a group of learners as a whole, to identify common structural configurations and differences in the learners’ knowledge. It shows that those students who have attended CS education in their secondary school life have, on average, configured their knowledge about typical core CS/OOP concepts differently. Also, artifacts of their particular CS curriculum are visible in their externalized knowledge. The data structures and analysis methods necessary for working with concept landscapes have been implemented as a GNU R package that is freely available.
Conference Paper
Full-text available
We present the results of an investigation on how well students are able to understand object-oriented programming (OOP) when learning with only very minimal guidance. We analyzed the source code that the students of a preparatory course produced during the course as well as concept maps that they were asked to draw before and after the course. Our findings show, that there are observable differences between what students know about some concepts and what they’re able to do with it. Generally speaking, it seems that several OOP related concepts can be applied successfully without fully understanding the underlying concepts, while others are hard to understand and apply without a significant amount of prior knowledge. This gives rise to the suspicion that it might be possible to apply a concept without having understood it, at least with respect to some algorithmic concepts of CS.
Chapter
Full-text available
Wir stellen die Konzeption und erste Ergebnisse einer neuartigen Informatik- Lehrveranstaltung für Studierende der Geodäsie vor. Das Konzept verbindet drei didaktische Ideen: Kontextorientierung, Peer-Tutoring und Praxisbezug(“Course“). Die Studierenden sollen dabei in zwei Semestern wichtige Grundlagen der Informatik verstehen und anzuwenden lernen. Durch enge Verzahnung der Aufgaben mit einem für Nichtinformatiker relevanten Kontext, sowie einem sehr hohen Anteil von Selbsttätigkeit der Studierenden soll die Motivation für "fachfremde"Themen gesteigert werden. Die Ergebnisse zeigen, dass die Veranstaltung sehr erfolgreich war.
Conference Paper
Full-text available
In order to explore and validate suitable methods for investigating learning processes, we are currently conducting a case study, exploring the mental models of novice students in the field of object oriented modeling and programming. After abstracting and systemizing the information that was presented to the students of our introductory CS 1 course for non-majors we have asked them to draw concept maps at four points in time. Additionally, we conducted a small midterm exam, where the students had to implement some of the most important concepts and a regular final exam. We found that learning progress can be observed in detail by evaluating the concept maps.
Article
This chapter develops an extension of Doignon and Falmagne's knowledge struc-tures theory by integrating it into a competence-performance conception. The aim is to show one possible way in which the purely behavioral and descriptive knowledge structures approach could be structurally enriched in order to account for the need of explanatory features for the empirically observed solution behav-ior. Performance is conceived as the observable solution behavior of a person on a set of domain-specific problems. Competence (ability, skills) is understood as a theoretical construct accounting for the performance. The basic concept is a mathematical structure termed a diagnostic, that creates a correspondence be-tween the competence and the performance level. The concept of a union-stable diagnostic is defined as an elaboration of Doignon and Falmagne's concept of a knowledge space. Conditions for the construction and several properties of union-stable diagnostics are presented. Finally, an empirical application of the competence-performance conception in a small knowledge domain is reported that shall illustrate some advantages of the introduced modeling approach.
Article
Suppose that Q is a set of problems and S is a set of skills. A skill function assigns to each problem q i.e. to each element of Q — those sets of skills which are minimally sufficient to solve q; a problem function assigns to each set X of skills the set of problems which can be solved with these skills (a knowledge state). We explore the natural properties of such functions and show that these concepts are basically the same. Furthermore, we show that for every family K of subsets of Q which includes the empty set and Q, there are a set S of (abstract) skills and a problem function whose range is just K. We also give a bound for the number of skills needed to generate a specific set of knowledge states, and discuss various ways to supply a set of knowledge states with an underlying skill theory. Finally, a procedure is described to determine a skill function using coverings in partial orders which is applied to set A of the Coloured Progressive Matrices test (Raven, 1965).
Article
This article describes two related studies that began to explore the validity of scores from select-and-fill-in (SAFI) concept map assessments as measures of students’ connected understanding of science. Scores from SAFI maps created for this purpose and used with middle school students and undergraduate astronomy students possessed high internal consistency and exhibited large mean increases with increased domain exposure. SAFI scores were strongly related to scores from a standardized multiple-choice (MC) achievement measure for middle school students; work with individual students suggested that they used strategies requiring connected understanding to successfully complete the maps. SAFI scores from undergraduate students exhibited large relationships with scores from direct-instruction MC exams and scores from a relatedness ratings measure, taken together and separately. Results provide initial evidence of the validity of scores from SAFI maps as measures of connected understanding of science in middle school and undergraduate introductory science students.
Article
The concept of a knowledge space is at the heart of a descriptive model of knowledge in a given body of information. Another model explains the observed knowledge of individuals by latent skills. We here reconcile these two underlying approaches by showing that each finite knowledge space can be generated from a skill assignment that is minimal and unique up to an isomorphism. Some more technicalities are required in the infinite case. Part of the results reformulate theorems from the theory of Galois lattices of relations.