Content uploaded by Dimitra Tsovaltzi
Author content
All content in this area was uploaded by Dimitra Tsovaltzi on Feb 06, 2015
Content may be subject to copyright.
Automating Hinting in an Intelligent Tutorial Dialog System for Mathematics
Armin Fiedler
Department of Computer Science
Saarland University
afiedler@cs.uni-sb.de
Dimitra Tsovaltzi
Department of Computational Linguistics
Saarland University
dimitra@coli.uni-sb.de
Abstract
Despite empirical evidence that natural language
dialog capabilities are necessary for the success
of tutorial sessions, only few state-of-the-art tutor-
ing systems use natural-language style interaction.
Moreover, although hinting has been psychologi-
cally substantiated, most intelligent tutoring sys-
tems do not systematically produce hints. In the DI-
ALOG project, we aim at a mathematical tutoring
system that employs an elaborate natural language
dialog component and a hinting tutoring strategy.
To tutor mathematics, we use a formally encoded
mathematical theory including definitions and the-
orems along with their proofs. We enhance this on-
tology by making relations explicit and show how
these relations can be used when planning the next
utterance of the system. Moreover, we define a
scheme for classifying the student’s input in terms
of the knowledge of the domain demonstrated. Fi-
nally, as a theory of tutoring we define a taxonomy
of hints and a hint determining algorithm that im-
plements the socratic tutoring strategy, whose de-
cisive characteristic is the use of hints in order to
achieve self-explanation. This algorithm takes into
account both the mathematical ontology and the
categories of the student’s answers.
1 Introduction
Despite empirical evidence that natural language dialog ca-
pabilities are necessary for the success of tutorial sessions
[19], only few state-of-the-art tutoring systems use natural-
language style interaction that requires menu-based input or
exact wording of the input [20; 2; 13]. Since domain knowl-
edge, tutoring and pedagogical knowledge, and dialog man-
agement are tightly intertwined, the modeling and integration
of proper natural language dialog capabilities in a tutoring
system turns out to be barely manageable.
Although there has been psychological evidence [7; 23]for
the high educational effect of hinting, most intelligent tutor-
ing systems do not take advantage of the findings. Little sys-
tematic research has been done in automating hinting [16; 10;
26].
In the DIALOG project [21], we aim at a mathematical
tutoring system that employs an elaborate natural language
dialog component. To tutor mathematics, we need a formally
encoded mathematical theory including definitions and the-
orems along with their proofs, means of classifying the stu-
dent’s input in terms of the knowledge of the domain, and a
theory of tutoring that should make use of hints.
Since it might be impossible to precompile all possible
proofs for a given theorem, we make use of the state-of-the-
art theorem prover
MEGA [25]with its mathematical knowl-
edge base and start with restricting ourselves to the domain of
naive set theory.
A mathematical tutoring system must be able to tutor
proofs in a way that not only helps the student understand
the current proof, but also allows for a high learning ef-
fect. What is meant by the latter is the ability of the stu-
dents not only to better understand the problem at hand, but
also to generalize and apply the taught methodology with-
out help later on. We propose to establish those tutoring
aims by making use of the socratic tutoring method [20;
23]. The decisive characteristic of the socratic method is the
use of hints to achieve self-explanation, as opposed to an-
swers and explanations being provided constantly by the tu-
tor, which is characteristic of the didactic method [23; 29].
Hinting is a method that aims at encouraging active learn-
ing. It can take the form of eliciting information that the stu-
dent is unable to access without the aid of prompts, or infor-
mation which he can access but whose relevance he is un-
aware of with respect to the problem at hand. Alternatively, a
hint can point to an inference that the student is expected to
make based on knowledge available to him, which helps the
general reasoning needed to deal with a problem [16].
Our approach to modeling hinting involves enhancing the
existing mathematical knowledge base of the theorem prover
MEGA. We do that by making the relations between enti-
ties explicit. These relations can potentially be used in the
automatic production of hints. Note that in this paper we do
not address the issue of natural language realization of hints,
which is also an ongoing research topic of our project. To
the end of automatically producing hints we have been devel-
oping a taxonomy of hints for the naive set theory domain,
which draws on the domain knowledge representation of the
enhanced mathematical ontology. We have also derived a stu-
dent answer categorization scheme in order to classify the stu-
dent’s answers. The taxonomy and the student answer cate-
gories are used by our hinting algorithm which models the
socratic tutoring method and by employing an implicit stu-
dent model.
The implicit student model of the hinting algorithm makes
use of the student answer categorization and of information
on the domain knowledge demonstrated, or already provided
by previous hints produced. Its aim is to use information from
the running tutoring and the current progress of the student
on the particular problem under considerationin order to pro-
duce hints. Those hints, in turn, intend to address the cog-
nitive state of the student and cause him to reflect upon his
reasoning. We are only concerned here with the process of
making the student actively reflect on his answers. However,
DIALOG uses ACTIVEMATH [18]as an external facility for
user modeling.
In this paper, we first provide a comprehensive description
of our domain ontology (Section 2) and its use (Section 3).
In Section 4 we give an overview of the taxonomy of hints.
In Section 5 we look at our student answer categorization
scheme and in Section 6 the hinting algorithm is presented.
Next, in Section 7 an example dialog demonstrates the use of
the research presented here as envisaged for our system. Fi-
nally, we briefly discuss some related work in Section 8 and
conclude the paper.
2 A Mathematical Ontology for Hinting
The proof planner
MEGA [25]makes use of a mathematical
database that is organized as a hierarchy of nested mathemati-
cal theories. Each theory includes definitions of mathematical
concepts, lemmata and theorems about them, and inference
rules, which can be seen as lemmata that the proof planner
can directly apply. Moreover, each theory inherits all def-
initions, lemmata and theorems, which we will collectively
call assertions henceforth, as well as all inference rules from
nested theories. Since assertions and inference rules draw
on mathematical concepts defined in the mathematical the-
ories, the mathematical database implicitly represents many
relations that can potentially be made use of in tutorial dia-
log. Further useful relations can be found when comparing
the definitions of concepts with respect to common patterns.
In this section, we shall first show in Section 2.1 a part of
MEGA’s mathematical database, which we shall use as an
example domain henceforth. Then, we shall define in Sec-
tion 2.2 the relations to be used in the hinting process.
2.1 A Mathematical Database
In
MEGA’s database, assertions are encoded in a simply-
typed
-calculus, where every concept has a type and well-
formedness of formulae is defined by the type restrictions.
In this paper, we will concentrate on the mathematical con-
cepts from naive set theory given in Table 1. These concepts
draw on the types sets and inhabitants of sets. We give the
definitions of the mathematical concepts in intuitive terms as
well as in more formal terms paraphrasing but avoiding the
-calculus encoding of
MEGA’s database.
Let
be sets and let
be an inhabitant.
: element
: not element
: intersection
: union
: subset
: not subset
: strict subset
: not strict subset
: superset
: not superset
: strict superset
: not strict superset
: powerset
Table 1: Mathematical concepts.
Element: The elements of a set are its inhabitants:
if
and only if
is an inhabitant of
.
Intersection: The intersection of two sets is the set of their
common elements:
and
.
Union: The union of two sets is the set of the elements of
both sets:
or
.
Subset: A set is a subset of another set if all elements of the
former are also elements of the latter:
if and only
if for all
follows that
.
Strict Subset: A set is a strict subset of another set if the latter
has at least one element more:
if and only if
and there is an
such that
.
Superset: A set is a superset of another set if all elements of
the latter are also elements of the former:
if and
only if for all
follows that
.
Equality: Two sets are equal if they share the same elements:
if and only if for all
follows that
and for all
follows that
.
Powerset: The powerset of a set is the set of all its subsets:
"!#$%&'
(
.
The definition of the negated concepts from Table 1 is
straightforward.
Furthermore, we give examples of lemmata and theorems
that use some of these concepts. Let
,
and
)
be sets:
Commutativity of Union:
*
.
Equality of Sets: If
and
then
+
.
Union in Powerset: If
)$!
and
)$!
then
)$!
.
Finally, we give some examples for inference rules. Infer-
ence rules have the following general form:
,.-0/1211/',43
5 6
where
7#89;:<:<:;'7>=
,
?@BA
are the premises, from which the
conclusion
C
is derived by rule
D
.
Let
and
be sets. Example inference rules are:
EGFIH HJFE
ELKLH
Set=
E&MNPORQTSUH+MNPORQ&S
EWVHJMNPORQTS
VN
Note that the left inference rule encodes the lemma about
the equality of sets, and the right inference rule encodes the
lemma about the union in powersets. Since every lemma and
2
theorem can be rewritten as an inference rule and every infer-
ence rule can be rewritten as a lemma or theorem, we identify
lemmata and theorems with inference rules henceforth.
2.2 Enhancing the Ontology
The mathematical database implicitly represents many rela-
tions that can be made use of in tutorial dialog. Further useful
relations can be found when comparing the definitions of con-
cepts with respect to common patterns. We consider relations
between mathematical concepts, between mathematical con-
cepts and inference rules, and among mathematical concepts,
formulae and inference rules. By making these relations ex-
plicit we convert the mathematical database into an enhanced
ontology that can be used in hinting.
Mathematical Concepts
Let
4
be mathematical concepts. We define the following
relations between mathematical concepts:
Antithesis:
is in antithesis to
if and only if it is its oppo-
site concept (i.e., its logical negation).
Examples: antithesis
!
,antithesis
!
Duality:
is dual to
if and only if
is defined in terms of
7
C
and
is defined in terms of
C
7
for some
formulae
7 C
.
Examples: dual
!
,dual
!
Junction:
is in a junction to
if and only if
is defined in
terms of
7#8
"7>=
and
is defined in terms of
7 8
7=
, or vice versa, for some formulae
78<:;:<:;'7 =
,
where
?
.
Examples: junction
!
Hypotaxis:
is in hypotaxis to
if and only if
is defined
using
. We say,
is a hypotaxon of
, and
is a
hypertaxon of
.
Examples: hypotaxon
!
,hypotaxon
!
Primitive:
is a primitive if and only if it has no hypotaxon.
Examples: primitive
!
Note that
is a primitive in
MEGA’s database, since
it is defined using inhabitant, which is a type, but not a
defined concept.
Mathematical Concepts and Inference Rules
Let
,
be mathematical concepts and
D
be an inference
rule. We define the following relations:
Relevance:
is relevant to
D
if and only if
D
can only be
applied when
is part of the formula at hand (either in
the conclusion or in the premises).
Examples: relevant
Set=
!
,relevant
Set=
!
,
relevant
Set=
!
Dominance:
is dominant over
for rule
D
if and only if
appears in both the premises and the conclusion, but
does not.
has to appear in one of the premises or the
conclusion.
Examples: dominant
!
,
Mathematical Concepts, Formulae and Inference Rules
Let
be a mathematical concept,
D
be an inference rule and
7#8%;:<:;: '7>= 'C
formulae, where
7#8%<:;:<:<27>=
are the premises
and
C
the conclusion of
D
.
Introduction: Rule
D
introduces
if and only if
occurs
in the conclusion
C
, but not in any of the premises
7#8 <:<:;: '7>=
.
Examples: introduces
Set=
!
,introduces
!
Elimination: Rule
D
eliminates
if and only if
occurs in
at least one of the premises
78;:<:;:<'7 =
, but not in the
conclusion
C
.
Examples: eliminates
Set=
!
,eliminates
Set=
!
Automation
The automatic enhancement of the mathematical database by
explicitly adding the relations defined previously is straight-
forward. The automation of the enhancement allows us to
plug in any mathematical database and to convert it accord-
ing to the same principles into a database that includes the
relations we want to make use of in hinting.
3 Making Use of the Ontology
In this section we further explain the use of the mathematical
ontology by pointing out its exact relevance with regard to the
general working framework. We also give a few examples of
the application of the ontology in automating hint production.
The mathematical ontology is evoked primarily in the re-
alization of hint categories chosen by the socratic algorithm
described in Section 6. Due to the adaptive nature of the al-
gorithm, and our goal to dynamically produce hints that fit
the needs of the student with regard to the particular proof,
we cannot restrict ourselves to the use of a gamed of static
hints. The algorithm takes as input the number and kind of
hints produced so far, the number of wrong answers by the
student and the categories of the current and previous student
answer. It computes the appropriate hint category to be pro-
duced next from the taxonomy of hints. The hint category is
chosen with respect to the implicit student model that the al-
gorithm’s input constitutes. This means that for every student
and for his current performance on the proof being attempted,
the hint category chosen must be realized in a different way.
Each hint category in the hint taxonomy is defined based
on generic descriptions of domain objects or relations. The
role of the ontology is to map the generic descriptions on the
actual objects or relations that are used in the particular con-
text, that is, in the particular proof and the proof step at hand.
Another equally significant use of the domain ontology is
in categorizing the student’s answers. This use is a side-effect
of the involvement of the ontology in automatically choosing
hints. That is, the algorithm takes as input the analyzed stu-
dent answer. In analyzing the latter, we compare it to the
expected answer (see Section 5) and then look for the em-
ployment of necessary entities.
The necessary entities are defined in terms of the ontology.
The algorithm checks for the student’s level of understanding
by trying to track the use of these entities in the student’s an-
swer to be addressed next. The hint to be produced is then
3
picked according to the knowledge demonstrated by the stu-
dent. Note that this knowledge might have already been pro-
vided by the system itself, in a previous turn when dealing
with the same performable step. Since the algorithm only
checks for generic descriptions of those concepts, we suggest
the use of the present ontology to map the descriptions onto
the actual concepts relevant to the particular context.
Let us now give examples of the use of our domain ontol-
ogy. All the relations mentioned here, which are defined in
the ontology, have been explained in Section 2.
For the hint
give-away-relevant-concept
, which points out
the right mathematical concept that the student has to bring
into play in order to carry the proof out, we use the domain
ontology in the following way:
1. If the inference rule to be applied involves the elimination
or the introduction of a concept, then we identify the rel-
evant concept with the concept eliminated or introduced,
respectively.
2. Otherwise, we look for a relevance relation and identify
the relevant concept with that in the relation.
To produce the hint
elaborate-domain-object
we have to
find in the domain the exact way an inference rule needs to be
applied, for instance, whether it involves an elimination or an
introduction. The student will be informed accordingly.
For more on the construction and use of the mathematical
ontology see [28].
4 A Taxonomy of Hints
In this section we explain the philosophy and the structure of
our hint taxonomy. We also look into some hints that are used
in the algorithm. The names of the categories are intended
to be as descriptive of the content as possible, and should
in some cases be self-explanatory. The taxonomy includes
more than the hint categories mentioned in this section. The
full taxonomy is given in Table 2. Some categories are not
real hints (e.g.,
point-to-lesson
), but have been included in the
taxonomy since they are part of the general hinting process.
4.1 Philosophy and Structure
Our hint taxonomy was derivedwith regard to the underlying
function that can be common for different surface realizations
of hints. The underlying function is mainly responsible for
the educational effect of hints. Although the surface structure,
which undoubtedly plays its own significant role in teaching,
is also being examined in the our project, we do not address
this issue in this paper.
We defined the hint categories based on the needs in the
domain. To estimate those needs we made use of the objects
and the relations between them as defined in the mathematical
ontology. An additional guide for deriving hint categories
that are useful for tutoring in our domain was a previous hint
taxonomy, which was derived from the BE&E corpus [26].
The structure of the hint taxonomy reflects the function
of the hints with respect to the information that the hint ad-
dresses or is meant to trigger. To capture different functions
of a hint we define hint categories across two dimensions.
Before we introduce the dimensions, let us clarify some
terminology. In the following, we distinguish performable
steps from meta-reasoning. Performable steps are the steps
that can be found in the formal proof. These include premises,
conclusion and inference methods such as lemmata, theo-
rems, definitions of concepts, or calculus-level rules. Meta-
reasoning steps consist of everything that leads to the per-
formable step, but cannot be found in the formal proof. To
be more specific, meta-reasoning consists of everything that
could potentially be applied to any particular proof. It in-
volves general proving techniques. As soon as a general tech-
nique is instantiated for the particular proof, it belongs to the
performable step level.
The two hint dimensions consist of the following classes:
1. active vs. passive
2. domain-relation vs. domain-object vs. inference-rulevs.
substitution vs. meta-reasoning vs. performable-step
In the second dimension, we ordered the classes with re-
spect to their subordination relation. We say, that a class is
subordinate to another one if it reveals more information.
Each of these classes consists of single hint categories that
elaborate on one of the attributes of the proof step under con-
sideration. The hint categories are grouped in classes accord-
ing to the kind of information they address in relation to the
domain and the proof. By and large, the hints of the pas-
sive function of a class in the second dimension constitute
the hints of the active function of its immediately subordinate
class, in the same dimension. In addition, the class of prag-
matic hints belongs to the second dimension as well, but we
define it such that it is not subordinate to any other class and
no other class is subordinate to it.
In the following section we look at the structure of the
second dimension just described through some examples of
classes and the hints defined in them.
4.2 First Dimension
The first dimension distinguishes between the active and pas-
sive function of hints. The difference lies in the way the infor-
mation to which the tutor wants to refer is approached. The
idea behind this distinction resembles that of backward- vs.
forward-looking function of dialog acts in DAMSL [8]. The
active function of hints looks forward and seeks to help the
student in accessing a further bit of information, by means
of eliciting, that will bring him closer to the solution. The
student has to think of and produce the answer that is hinted
at.The passive function of hints refers to the small piece of in-
formation that is provided each time in order to bring the stu-
dent closer to some answer. The tutor gives away some infor-
mation, which he has normally unsuccessfully tried to elicit
previously. Due to that relation between the active and pas-
sive function of hints, the passive function of one hint class
in the second dimension consists of hint categories that are
included in the active function in its subordinate class.
4.3 Second Dimension
In this section we will give a few examples of classes and hint
categories that capture the structure of the second dimension.
4
active passive
domain-relation elicit-antithesis give-away-antithesis
elicit-duality give-away-duality
elicit-junction give-away-junction
elicit-hypotaxis give-away-hypotaxis
elicit-specialization give-away-specialization
elicit-generalization give-away-generalization
domain-object give-away-antithesis give-away-relevant-concept
give-away-duality give-away-hypotactical-concept
give-away-junction give-away-primitive-concept
give-away-hypotaxis
give-away-specialization
give-away-generalization
inference rule give-away-relevant-concept give-away-inference-rule
give-away-hypotactical-concept
give-away-primitive-concept
elaborate-domain-object
substitution give-away-inference-rule spell-out-substitution
elicit-substitution
meta-reasoning spell-out-substitution explain-meta-reasoning
performable-step explain-meta-reasoning give-away-performable-step
confer-to-lesson
pragmatic ordered-list take-for-granted
unordered-list point-to-lesson
elicit-discrepancy
Table 2: The taxonomy of hints.
Domain-relation hints address the relations between math-
ematical concepts in the domain, as described in Section 2.
The passive function of domain-relation hints is the active
function of domain-object hints, that is, they are used to elicit
domain objects.
Domain-object hints address an object in the domain. The
hint
give-away-relevant-concept
names the most prominent
concept in the proposition or formula under consideration.
This might be, for instance, the concept whose definition the
student needs to use in order to proceed with the proof, or
the concept that will in general lead the student to under-
stand which inference rule he has to apply. Other examples in
the class are
give-away-hypotactical-concept
and
give-away-
primitive-concept
. The terms hypotactical and primitive con-
cept refer to the relation, based on the domain hierarchy, be-
tween the addressed concept and the original relevant con-
cept, which the tutor is trying to elicit. Since this class is sub-
ordinate to domain-relation, the hints in it are more revealing
than domain-relation hints. The passive function of domain-
object hints is used to elicit the applicable inference rule, and,
therefore, is part of the active function of the respective class.
The same structure holds for inference-rule,substitution,
meta-reasoning and performable-step hints.
Finally, the class of pragmatic hints is somewhat differ-
ent from other classes in that it makes use of minimal do-
main knowledge. It rather refers to pragmatic attributes of the
expected answer. The active function hints are
ordered-list
,
which specifically refers to the order in which the parts of the
expected answer appear,
unordered-list
, which only refers to
the number of the parts, and
elicit-discrepancy
, which points
out that there is a discrepancy between the student’s answer
and the expected answer. The latter can be used in place of
all other active hint categories.
Take-for-granted
asks the stu-
dent to just accept something as a fact either when the student
cannot understand the explanation or when the explanation
would require making use of formal logic.
Point-to-lesson
points the student to the lesson in general and asks him to read
it again when it appears that he cannot be helped by tutoring
because he does not remember the study material. There is
no one-to-one correspondence between the active and pas-
sive pragmatic hints. Some pragmatic hints can be used in
combination with hints from other classes.
For more on the taxonomy of hints see [11].
5 Student Answer Categorization Scheme
In this section we present a scheme for categorizing student
answers. We are only concerned here with the parts of the an-
swer that address domain knowledge. We define student cate-
gories based on their completeness and accuracy with respect
to the expected answer. The latter is always approximated for
the student’s own line of reasoning. The output of the classi-
fication constitutes part of the input to the hinting algorithm,
which models the hinting process.
5.1 Proof Step Matching
The student’s answer is evaluated by use of an expected an-
swer. The expected answer is the proof step which is expected
5
next according to the formal proof whichthe system has cho-
sen for the problem at hand. We want to make use of the stu-
dent’s own reasoning in helping him with the task and avoid
super-imposing a particular solution. We model that by try-
ing to match the student’s answer to an proof step in one of a
set of proofs. To this end we use the state-of-the-art theorem
prover
MEGA [25].
5.2 Parts of Answers and Over-Answering
We now define the relevant units for the categorization of the
student answer. A part is a premise, the conclusion or the in-
ference rule of a proof step. The two former are mathematical
formulae and must be explicitly mentioned for the proof to be
complete. The inference rule can either be referred to nomi-
nally, or it can be represented as a formula itself. In the latter
case, we just consider that formula as one of the premises.
It is up to the student to commit to using the rule one way
or the other. A formula is a higher-order predicate logic for-
mula. Every symbol defined in the logic is a function. For-
mulae can constitute of subformulae to an arbitrary degree of
embedding. Constants are 0-ary functions that constitute the
lowest level of entities considered.
We also consider (accurate or inaccurate) over-answering
as several distinct answers. That is, if the student’s answer
has more proof steps than one, we consider the steps as mul-
tiple answers. The categorization is normally applied to them
separately. Nevertheless, there are cases where the order of
the presentation of the multiple answers is crucial. For exam-
ple, we cannot count a correct answer that is inferred from a
previous wrong answer, since the correct answer would have
to follow from a wrong premise.
5.3 Completeness vs. Accuracy
We define the predicates complete and accurate as follows:
Complete: An answer is complete if and only if all parts of
the expected answer are mentioned.
Accurate: A part of an answer is accurate if and only if the
propositional content of the part is the true and expected
one.
Completeness
From our definition of completeness it follows that complete-
ness is dependent on domain objects, but not on our domain
ontology. That is, the expected answer, which is the basis of
the evaluation of the completeness of a student answer, nec-
essarily makes use of objects in the domain. However, the
relations of the objects in the domain are irrelevant to evalu-
ating completeness. Completeness is a binary predicate. The
only thing relevant to completeness is the presence or absence
of objects in the student’s answer.
In addition, we distinguish between getting the expected
domain object right and instantiating it correctly. The latter
does not follow from the former. Completeness is relevant to
the presence of the object but not to its correct instantiation.
In other words, a place holder for an expected object in the
answer is enough for attributing completeness, no matter if
the object itself is the expected one. That issue is dealt with
by accuracy.
Accuracy
Accuracy, contrary to completeness, is dependent on the do-
main ontology. It refers to the appropriateness of the object
in the student answer with respect to the expected object. An
object is accurate, if and only if it is the exact expected one.
Currently we are using accuracy as a binary predicate in the
same way that we do with completeness. However, we intend
to extend our categorization to include different degrees of
accuracy.
5.4 The Categories
In this section we enumerate the categories of students an-
swers based on our definitions of completeness and accuracy
and with regard to the expected answer.
We define the following student answer categories:
Correct: An answer which is both complete and accurate.
Complete-Partially-Accurate: An answer which is com-
plete, but some parts in it are inaccurate.
Complete-Inaccurate: An answer which is complete, but all
parts in it are inaccurate.
Incomplete-Accurate: An answer which is incomplete, but
all parts that are present in it are accurate.
Incomplete-Partially-Accurate: An answer which is in-
complete and some of the parts in it are inaccurate.
Wrong: An answer which is both incomplete and inaccurate.
For the purposes of this paper, we collapse the cate-
gories complete-partially-accurate, complete-inaccurate and
incomplete-partially-accurate to one category, namely, inac-
curate.
More on our student answer categorization scheme can be
found in [27].
5.5 Subdialogs
Subdialogs are crucial to the correct evaluation of the student
answer and supplementary to the student answer categoriza-
tion scheme just presented. The tutor can initiate subdialogs,
for example, in case of ambiguity in order to resolve it. More-
over, students are given the opportunity to correct themselves,
provide additional information on their reasoning and give es-
sential information about the way they proceed with the task.
Student’s can as well initiate subdialogs, usually with clarifi-
cation questions or requests for particular information.
An example of a subdialog initiation by the tutor is the
case of potential “typos”. We want to treat them differently
from conceptual domain mistakes, such as, wrong instanti-
ations. We can prevent that by asking the student what he
really meant to say. Our assumption is that if the student
really just used the wrong symbol and did not make a con-
ceptual domain mistake, he will realize it and correct it. If
he does not correct it, we categorize the answer taking into
account the domain mistake. This kind of subdialog we iden-
tify with alignment [6], and an instance of it is the example
in Figure 11. The tutor could not make sense of the student’s
1The examples presented here are from our recently collected
corpus on mathematics tutorial dialogs in German [4]. Translations
are included, where necessary.
6
utterance unless he substituted
for
and
!
for
!
. The student could not correct
himself, although he realized from the tutor’s question that he
had used the wrong symbol.
Student (5): wenn
F
O
S
, dann
K
, weil
K
O
S
[if
F
O
S
, then
K
, because
K
O
S
]
Tutor (6): meinen Sie wirklich
K
oder etwas anderes?
[Do you really mean
K
or something else?]
Student (7):
F
Figure 1: Example of alignment, Subject 13
Other cases of subdialog initiation include instances when
the student’s answer cannot be classified, or when a student
gives an answer which is in principle correct but not part of
the current proof.
6 A Hinting Algorithm
A tutoring system ideally aims at having the student find the
solution to a problem by himself. Only if the student gets
stuck should the system intervene. There is pedagogical ev-
idence [7; 23]that students learn better if the tutor does not
give away the answer but instead gives hints that prompt the
student for self-explanations. Accordingly, based on [26]we
have derived an algorithm that implements an eliciting strat-
egy that is user-adaptive by choosing hints tailored to the
students. Only if hints appear not to help does the algo-
rithm switch to an explaining strategy, where it gives away
the answer and explains it. We shall follow Person and col-
leagues [20]and Ros´e and colleagues [23]in calling the elic-
iting strategy socratic and the explaining strategy didactic.
6.1 Description of the Hinting Algorithm
We shall now present an algorithm that implements the so-
cratic strategy. In intuitive terms, the algorithm aims at hav-
ing the student find the proof by himself. If the student does
not know how to proceed or makes a mistake, the algorithm
prefers hinting at the right solution in order to elicit the prob-
lem solving instead of giving away the answer. An implicit
student model makes the algorithm sensitive to students of a
different level by providing increasingly informative hints.
The algorithm takes as input the number and kind of hints
produced so far, the number of wrong answers by the student
and the current and previous student answer category. The
particular input to the algorithm is the category that the stu-
dent answer has been assigned, based on our student answer
categorization scheme, and the domain knowledge employed
in the answer. The category of the student answer in combi-
nation with the kinds of hints already produced and the use
of required entities of the mathematical ontology together in-
form the algorithm of the level of the student’s knowledge.
We described the use of the mathematical ontology in that
context in Section 3.
Moreover, the algorithm computes whether to produce a
hint and which category of hint to produce, based on the num-
ber of wrong answers, the number and kind of hints already
produced, as well as the domain knowledge demonstrated by
the student. The algorithm computes the appropriate hint cat-
egory to be produced next from the taxonomy of hints.
If the hinting does not effect correct student answers af-
ter several hints the algorithm switches to a didactic strategy,
and, without hinting, explains the steps that the student cannot
find himself. Nevertheless, the algorithm continues to ask the
student for the subsequent step. If the student gives correct
answers again and, thus, the tutor need not explain anymore,
the algorithm switches back to the socratic strategy. In effect,
hints are provided again to elicit the step under consideration.
The Main Algorithm
The essentials of the algorithm are as follows:
"!#!$%'&)(!+*,!-./0,"21
"3(4 "!-.56!7 68(39#:&;"3
2/0&<&;"!
=5?>(@>$/0A#B3C<&DEF153(G:>$(F&DHI3
J8&DE@152(G2K &+>$(@&HI3 60&:*,! " "2*L
"3(M>7*3*,256<>(@&HN2 6>(@1PO!Q!R&D3S7
2/0&<*3>//T"E(@*,0!(
socratic
!(P80(@>7*3*,E@ ">$
>$(@1P-.0&"&(O.@> DU&6!$%<&DE@13(G2K &+>$(F&DHI3
The Function socratic
The bulk of the work is done by the function socratic,
which we only outline here. The function takes as an argu-
ment the category
V
of the student’s current answer. If the
origin of the student’s mistake is not clear, a clarification di-
alog is initiated, which we do not describe here. Note, how-
ever, that the function stops if the student gives the correct
answer during that clarification dialog, as that means that the
student corrected himself. Otherwise, the function produces
a hint in a user-adaptive manner.
The function socratic calls several other functions,
which we look at subsequently.
Let
W
denote the number of hints produced so far and
VCX
8
the category of the student’s previous answer. The hint is then
produced as follows:
Y
>&
W
JA
Z
V
'&6H) "!(OQ! )0(@>*2*,E U>[":"3(4*2>$/0/
elicit
Z
V
'&60(@*,!7-Q@/3,\]>*2*,E U>["
2(^@ !515E@*38>$(_>*L"`7 ">O-R>["0*0(G
a
that is,
ordered-list
, or
unordered-list
b
Y
>&
W
Z
V
'&6H) "!(O
2(^
V+X
8
0&6H) "!(@OQ!7 6(@*3!-./0,",\]>*3*3E U>[
"3(4*2>$/0/
up-to-inference-rule
V+X
8
0&60(@>7*3*,E@ ">$
"3(4*2>$/0/
elicit-give-away
VX
8
0&)*3! " c*L
"3(4*2>$/0/
elicit
3/'&D8*3>//
elicit
Y
>&
W
d
Z
V
'&6H) "!(O
2(^e'&)'&I" U1^H) !7(O.>$(@&HN2
"3(_ "!515E@*3
explain-meta-reasoning
2/0&Z "3`#0!E@&I0(G+HI>7&)>$(_>*L"`7
&Ef@&DE5"!7(^@(G
2(_ !515EF*,
spell-out-substitution
3/'&De@ 2`G0!EF&6(G+H6>&
spell-out-substitution
"3(_ "!515E@*,
give-away-performable-step
2/0&8*3>$/0/
up-to-inference-rule
3/'&D8*3>//
elicit-give-away
7
Y
>&
W
Z
V
'&6H) "!(O.>$(@1^+0&)>$+/c>&D6" U1PH) !7(OR>$(@&HN2
2(^@ !515E@*3
point-to-lesson
>$(F1P&D!7
a
The student is asked to read the lesson again. Afterwards,
the algorithm starts anew.
b
3/'&D "!515E@*,
explain-meta-reasoning
Y
>&
W
@
O0`8> HI> A >(@&HN2 +>$(@1_&H)ZU*U_! 15'1>*,'*<&D U>[2OA
&DH)"*U f@>7*
>$ 3 M "3 *3!(@&2*3E50`*3! " c*L >(@&DHI3 U&
H)Z"_>// *3!E(G"3 U&N "2&,
a
After four hints, the algorithm starts to guide the student more
than before to avoid frustration. It switches to the didactic strat-
egy in the fashion described a the beginning of the section.
b
After having produced a hint the function socratic analyzes
the student’s answer to that hint. If the student’s answer is still
not right the function socratic is recursively called. How-
ever, if the student answers correctly and at least two hints
have been produced the algorithm re-visits the produced hints
in the reverse order to recapitulate the proof step and to make
sure the student understands the reasoning so far. This is done
by producing a sequence of active meta-reasoning hints, one
for each hint that have addressed the current step of the proof,
in the reverse order. If the active meta-reasoning hints get
the student to say anything but the right answer, the algo-
rithm produces an
explain-meta-reasoning
hint. This is done
to avoid frustrating the student as his performance is poor.
The function socratic calls several functions, which we
present now. The functions are self-explanatory.
Function elicit
%"<&;"E@152(7
#(![H+&N"8 2/2`[>$(G+*,!(F*,3
@3(_Ze@<&;"E@153(G
#(![H+&N@8(52 2(@*,8 "E/0
2(4*3>$/0/
up-to-substitution
3/'&D8*2>$/0/
up-to-inference-rule
3/'&*2>$/0/
elicit-relevant-concept
Function elicit-give-away
e "3`#0!E@&I0(G+HI>7&)>$(_>*,0`815!-R>(5\ !f
;2*L60(G
@3(4*3>//
up-to-inference-rule
3/'&C 2`#!7E@&60(7+H6>&6>Q@>7&&`7815!-R>$0(5\ !f
;c*L60(7
"3(_ "!515E@*3
elaborate-domain-object
2/0&Z? "3`#0!E@&I0(G+HI>7&
elaborate-domain-object
2(M*3>//
up-to-substitution
3/'&D8@ !515E@*3
give-away-performable-step
>(@1
spell-out-substitution
a
this is to explain the substitution
b
Function up-to-inference-rule
e "3`#0!E@&I0(G+HI>7&
give-away-hypotactical-concept
@3(_ "!#1E@*,
give-away-inference-rule
3/'&C 2`#!7E@&60(7+H6>&
give-away-relevant-concept
"3(_ "!515E@*3
elaborate-domain-object
2/0&8Z "3`#0!E@&I0(G+HI>7&
elaborate-domain-object
2(_ !515EF*,
give-away-hypotactical-concept
3/'&D "!515E@*,
give-away-relevant-concept
Function up-to-substitution
e "3`#0!E@&I0(G+HI>7&
elicit-substitution
@3(_ "!#1E@*,
spell-out-substitution
3/'&C 0OG)(3 "3(@*38 E/0
#(![H)(
"3(_ "!515E@*3
elicit-substitution
2/0& "!#1E@*,
give-away-inference-rule
Function elicit-relevant-concept
%"<&;"E@152(7
#(![H+&6>R*3!(@*3356F>[+'&6 2/0>$21 !."
"3/03`[>$(G+*,!7(@*,25
@3(_ "!#1E@*,8>$(_>*,0`815!7-.>(\ !7f
;2*,6@(G
3/'&C@ !515E@*3
give-away-relevant-concept
The algorithm we just presented can easily be adapted to
other domains, because the hint taxonomy is independent
from the algorithm and only called by it. That means that the
only thing that has to be adapted for applying the algorithm
to different domains are the hint categories in the taxonomy,
while the structure of the taxonomy has to remain the same.
Defining adapted hint categories can be done through the do-
main ontology, which is by definition domain specific.
7 An Example Dialog
Let us consider the following dialog between the tutoring sys-
tem and a student on a simple proof. This will help elucidate
how the algorithm for the socratic tutoring strategy proceeds
and how it makes use of the hinting categories, the enhanced
domain ontology, and the implicit student model. The ex-
cerpt comes from our recently collected corpus of tutorial di-
alogs in mathematics in German [4]. The subject is number
23. The tutoring system, denoted as tutor in the following,
is teaching the proof of the proposition “If
!
, then
!
”, where
stands for the complement of a set
which is defined as follows:
If
is the universal set, and
is its subset (i.e.,
), we
call the difference simply the complement of
and denote it
briefly by
"!
; thus
!
.
Note that the corpus was collected via a Wizard of Oz ex-
periment [5]. That means that the system is only partially
implemented. Subjects interacted through an interface with a
human “wizard” simulating the behavior of a system 2.
We give the example dialog in chunks which consist of
one tutor and one student turn. The tutor’s turn includes a
name of hint flag. The student’s turn includes the answer cat-
egory flag. The explanation of the behavior of the tutor in
each chunk precedes it every time. The original German tran-
scriptions are provided and with them translations in English,
where necessary.
At the beginning of the session, the master proof, which
will be taught, has not yet been determined. This will be
done gradually, as the student will be ’committing’ himself to
a particular proof by the choice of steps he makes (See Sec-
tion 5.1). However, we provide here the proof actually taught
in the particular example first in order to help the reader fol-
low it. The proof was given in reality only as a summary
at the very end of the tutoring session and was presented as
follows:
Tutor (8): Ich zeige Ihnen nun den gesamten Beweis:
Zun¨
achst setzen wir die G¨ultigkeit von
F
O
S
vo-
raus, denn dies ist die Voraussetzung. Dann nehmen wir
ein beliebiges Element
M
und zeigen, daß dieses
auch in
O
S
sein muß. Sei also
in
. Dann ist
nicht
in
O
S
und deshalb nach Voraussetzung auch nicht in
. Wenn
aber nicht in
ist, so ist es in
O
S
, was den
Beweis abschließt, da
beliebig war.
2For details on the experiment see [4]
8
[I will show you the complete proof now : First we assume
the validity of
F
O
S
, as this is the assumption. Then
we take an arbitrary element
M
Q
and show, that it has
to be in
O
S
as well. Let
be in
. Then
is not in
O
S
and therefore according to the assumption also not
in
. But if
is not in
, it is in
O
S
, which concludes
the proof as
was arbitrary.]
When the algorithm starts, the counter
W
for the produced
hints is initialized to
A
. Since the proof is not completed, the
tutor prompts for the first step by introducing the task.
Tutor (1): Bitte zeigen Sie : Wenn
F
O
S
, dann
F
O
S
!
[Please show the following: If
F
O
S
, then
F
O
S
!]
Student (1): (wrong)
F
This answer is categorized as wrong, since it does not con-
tain any accurate parts. Thus the algorithm calls the function
socratic. Since
W
A
, the functions elicit is called
and, through that, elicit-relevant-concept. The latter
checks the student’s knowledge by matching something in the
answer to the relevant concept needed. That is done by help
of the domain ontology. The student has not used any of the
concepts that can be matched to the relevant concept of a pos-
sible proof. Thus, the system has to choose the first step to
teach. Eventually a hint of type
give-away-relevant-concept
is produced. The relevant concept is the implication. The
tutor chooses a clearer referring expression for it, namely “if-
then-relation”. The active function of the hint is to elicit the
inference rule, as will become evident later on in the example.
Here is the realization of the hint:
Tutor (2): (give-away-relevant-concept) Das ist nicht
richtig! Sie m ¨ussen als erstes die wenn-dann-Beziehung
betrachten.
[That is not correct! First you have to consider the if-then-
relation.]
Student (2): (wrong)
F
O
O
SS
With the production of this hint, the counter
W
is incre-
mented by
. However, this hint is not sufficient to help the
student. He gives an answer which is correct in principle, but
does not lead anywhere. Therefore, it is categorized as wrong.
The function socratic is called recursively. Since
W
,
the function up-to-inference-rule is called, which pro-
duces a hint of type
elaborate-domain-object
. This is a sec-
ond attempt to elicit the inference rule. Again, the student’s
knowledge of the necessary concept is checked first in the
same function. The tutor addresses the answer and informs
the student of its status.
Tutor (3): (elaborate-domain-object) Das ist zwar
richtig, aber im Augenblick uninteressant. Wissen Sie,
wie sie die wenn-dann-Beziehung behandeln m ¨ussen?
[That is correct, but at the moment not interesting. Do
you know how to deal with the if-then-relation?]
Student (3): (wrong) Nein
[No]
The counter
W
is incremented by
. The new student answer
is also considered as wrong3. Since this is the third wrong
answer and
W
d
, the algorithm now produces an
explain-
meta-reasoning
hint and explains the reasoning behind this
step. The number of hints and wrong answers imply a bad
student level.
Tutor (4): (explain-meta-reasoning) Sie m¨ussen die
Beziehung au߬
osen, indem Sie die G¨ultigkeit der Vo-
raussetzung annehmen und daraus die G ¨ultigkeit der Be-
hauptung folgern.
[You have to break down the relation by assuming the
validity of the assumption and derive the validity of the
proposition from that.]
Student (4): (unknown)
F
O
S
*F
O
S
The counter
W
is incremented again andit is now
W
. The
student answer cannot be categorized at all. Therefore the
system initiates a subdialog to clarify the problematic point
in the student’s answer.
Tutor (5): Was wollen Sie mit dem Zeichen
ausdr¨ucken?
[What do you want to say with the character
?]
Student (5): (wrong)
F
O
S
daraus folgt
+F
O
S
[
F
O
S
from that follows
+F
O
S
]
The
W
counter is not incremented this time, since there has
been no hint produced. The student’s response to the clarifica-
tion subdialog is categorised as wrong. This is now the fourth
wrong answer and
W
. The system produces a
give-away-
performable-step
hint, which really gives the current answer
away4.
Tutor (6): (give-away-performable-step) Das ist
genau das, was Sie beweisen sollen. Wir setzen nun
voraus, daß
F
O
S
gilt. Daraus wollen wir dann
schließen, daß die Behauptung
F
O
S
gilt. Verste-
hen Sie, warum wir annehmen m¨ussen, daß
F
O
S
gilt?
[That is exactly what you are supposed to prove. We
assume now that
F
O
S
holds. From that we want
to conclude, that the proposition
F
O
S
holds. Do
you understand why we need to assume that
F
O
S
applies?]
Student (6): (wrong) Nein
[No]
The counter
W
is again incremented by
. Since
W
, the algorithm switches to the didactic method in order to
avoid frustrating the student. The system explains the meta-
reasoning for the whole performable step, which has already
been given in the previous tutor turn.
Tutor (7): Wir m¨ussen dies annehmen, weil wir die wenn-
dann-Beziehung au߬
osen wollen. Bei einer solchen
3We are currently investigating the need to differentiate between
“I don’t know” answers, as this one and wrong attempts. However,
these were treated uniformly in this dialog.
4Note that normally the student would have been asked to read
the lesson material again through a
point-to-lesson
and in effect the
tutoring session would start anew after that. However, for the pus-
poses of the experiment and the collection of our corpus, we altered
the algorithm in the way described here.
9
Beziehung muß man stets zeigen, daß aus einem bes-
timmten Ausdruck ein anderer Ausdruck folgt. Deshalb
nehmen wir zuerst die G ¨ultigkeit des ersten Ausdrucks
an, weil wir sonst keine Basis h ¨
atten, von der wir ausge-
hen k¨
onnen. Verstehen Sie, warum wir zuerst die wenn-
dann-Beziehung au߬
osen wollen?
[We have to assume this, because we want to break
down the if-then-relation. With such a relation one al-
ways has to show that from a certain term another term
follows. Therefore we first assume the validity of the first
term, as otherwise we would have no base to start from.
Do you understand, why we first want to break down the
if-then-relation?]
Student (7): (wrong) Nein
[No]
After having completed the explanation of this proof step,
the system re-visits all hints produced by the socratic strategy
while explaining the step, and makes the student aware of
why they were produced. However, we will not go into such
detail here, since it is not in the focus of this paper. This
constitutes the proof summary that we gave at the beginning
of this example.
So far we looked at the behavior of the algorithm when
student constantly gives wrong ansers. To give the reader a
clearer view, let us now examine what happens if the student
gives a partial answer. Let us assume that, in the first turn, the
student gave an incomplete-accurate answer instead of giv-
ing a wrong answer. This example is also from our corpus,
albeit from a different teaching session with another subject
(Subject 22) on the same proof.
Student (1’):
&K
The student here gives a correct answer. However, the con-
tent of the answer cannot easily be inferred from the premise.
That means that it has to be proved before it can be used in
the proof. Since the student has not proved it, the answer
is considered incomplete. That is there are missing parts,
namely the proof of what is stated. Nonetheless, all parts
that are there are accurate. That makes the student’s answer
incomplete-accurate.
Subject 22 took part in the experiment in a different con-
dition, which did not involvehinting. However,wechose this
example for its clarity. In the example that we have been
investigating, the algorithm would call an active pragmatic
hint for this sudent answer category, and more specifically an
elicit-discrepancy
hint. In the example we have been looking
at a possible realization of such a hint would be:
Tutor (2’): That is correct, but why?
Now, if the student follows the hint he could give a correct
answer, which would look like this:
Student (2’): If there was an element
M
/
then it would
be
M
and
M
^
O
S
, which is a contradiction to the
assumption.
This is the correct answer, which completes the first proof
step as it is shown in the formal proof. Have there been more
than one hints so far, and since the student gave a correct
answer, the system would do a recapitulation by revisiting
the hints produced and asking the student to explain why the
particular course of proceeding is followed. If the student
could not answer any of the questions, thenthe system would
give the answer away itself before asking the next question,
addressing the next hint. Since only one hint was produced in
our imaginary dialog, the recapitulation is omitted. Thus the
algorithm proceeds with merely prompting for the next step.
8 Related Work and Discussion
Several other tutoring systems tackle, in one form or the other,
hinting strategies, domain ontologies and student answer cat-
egorization. We will only mention here the ones that we judge
to be most related to our work.
Hinting
Ms. Lindquist [14], a tutoring system for high-school algebra,
has some domain specific types of questions which are used
for tutoring. Although there is some mention of hints, and
the notion of gradually revealing information by rephrasing
the question is prominent, there is no taxonomy of hints or
any suggestions for dynamically producing them.
An analysis of hints can also be found in the CIRCSIM-
Tutor [15; 16], an intelligent tutoring system for blood circu-
lation. Our work has been largely inspired by the CIRCSIM
project both for the general planning of the hinting process
and for the taxonomy of hints. CIRCSIM-Tutor uses domain
specific hint tactics that are applied locally, but does not in-
clude a global hinting strategy that models the cognitive rea-
soning behind the choice of hints. We, instead, make use
of the hinting history in a more structured manner. Our algo-
rithm takes into account the kind of hints produced previously
as well as the necessary pedagogical knowledge,and follows
a smooth transition from less to more informative hints. Fur-
thermore, we have defined a structured hint taxonomy with
refined definition of classes and categories based on the pas-
sive vs. active distinction, which is similar to active-passive
continuum in CIRCSIM. We have distinguished these from
functions, which resemble CIRCSIM tactics, but are again
more detailed and more clearly defined. All this facilitates
the automation of the hint production.
AutoTutor [20]uses curriculum scripts on which the tutor-
ing of computer literacy is based. There is mention of hints
that are used by every script. Although it is not clear exactly
what those hints are, they seem to be static. More emphasis
seems to be put on the pedagogically oriented choice of di-
alog moves, prosodic features and facial expression features,
but not on hints. In contrast, we have presented in this pa-
per a hint taxonomy (cf. Section 4) and a tutoring algorithm
(cf. Section 6) that models the dynamic generation of hints
according to the needs of the student.
AutoTutor also uses a cycle of prompting-hinting-
elaborating. This structure relies on the different role of the
dialog moves involved to capture the fact that the tutor pro-
vides more and more information if the student cannot fol-
low the tutoring well. However, it does not provide hints that
themselves reveal more information as the tutoring process
progresses, which we have modeled in our algorithm for the
Socratic teaching method (cf. Section 6). Thus, the student
is not merely made to articulate the expected answers, as is
the case in AutoTutor, but he is also encouraged to actively
10
produce the content of the answer itself. Furthermore, the
separation of the study material and the tutoring session fa-
cilitates the production of the answers by the student, since
the tutor does not have to present the material and re-elicit it
in one and the same session. The student is guided through
making use of the study material that he has already read in
order to solve the problem.
Ontologies
CIRCSIM-Tutor [17; 9]uses an ontology at three levels: The
knowledge of domain concepts, the computer context of tu-
toring and the meta-language on attacking the problem. The
ontology we present in this paper is not concerned with the
second level. The first level corresponds to our existing
knowledge base. The third level can be viewed as a sim-
plified attempt to model tutoring, which we do via hinting.
They do, however, use their domain ontology in categorizing
the student answer and fixing mistakes.
Within the framework of STEVE [22], Diligent [3]is a tool
for learning domain procedural knowledge. Knowledge is ac-
quired by observing an expert’s performance of a task, as a
first step, subsequently conducting self-experimentation, and
finally by human corrections on what Diligent has taught it-
self. The most relevant part of the knowledge representation
is the representation of procedures in terms of steps in a task,
ordering constraints, causal links and end goals. Although
this is an elaborate learning tool it is not equally elaborate in
its use for interacting with students. It is currently limited to
making use of the procedure representation to learningtext in
order to provide explanations.
Student Answer Categorization
AutoTutor [20]uses a sophisticated latent semantic analy-
sis (LSA) approach. It computes the truth of an input based
on the maximum match between it and the training material
results. The system computes relevance by comparing it to
expected answers. The latter are derived from the curriculum
scripts. AutoTutor further uses the notions of completeness
and compatibility. Both these notions are defined as percent-
ages of the relevant aspect in the expectedanswer. Apart from
the problem of insensitivity of statistical methods to recog-
nizing linguistic phenomena crucial to the evaluation (e.g.,
negation) and the fact that LSA has proven a poor method for
evaluating the kind of small answers representative of our do-
main, we also want a definition of completeness and accuracy
that gives us insight to the part of the student answer that is
problematic, and hence needs to be addressed by appropriate
hinting. Therefore, defining them in terms of percentages is
not enough. We need to represent the relevant domain entities
that can serve as the basis for the hint production.
CIRCSIM-Tutor and PACT-Geometry tutor both use rea-
soning based on domain knowledge which is similar to the
approach presented in this paper. They compare and evalu-
ate the conceptual content of the student’s response by use
of domain ontologies. CIRCSIM-Tutor [30; 12]uses a fine-
grained classification. Unfortunately, the means of classify-
ing the student’s response, that is the rationale of the classifi-
cation, is not documented. That makes comparison difficult.
However, a list of ways of responding after a particular clas-
sification is available. Still no flexibility is allowed in terms
of a student model.
PACT-Geometry tutor [1]uses a semantic representation
of the student’s input that makes use of a domain ontology in
order to resolve concepts used and their relations. The cor-
rectness of the reasoning used is based on a hierarchy of ex-
planation categories that includes legitimate reasoning steps
in the domain. Although we do not use production rules, our
work is similar to this approach in the use of organized do-
main knowledge to evaluate the student’s answer. It differs
from it to the extent that the Geometry tutor appears not to
have a classification, which is essential to us as input to the
hinting algorithm.
9 Conclusion and Future Work
We have presented a domain ontology for the naive set the-
ory in mathematics. We propose to use this ontology in the
general framework of investigating tutorial dialogs for the do-
main. We have defined relations among mathematical con-
cepts, formulae and inference rules and applied this approach
to the domain of naive set theory.
We have further built a hint taxonomy, which was derived
with the aim of automating the hint categories defined in it. It
is based on the needs of the domain, as those were revealed
through the ontology, and previous work done on hints.
Moreover, a student answer categorization which makes
use of the notions of expected answer, completeness and ac-
curacy was derived. The taxonomy of hints in combination
with a student answer categorization is used in a socratic al-
gorithm which models hint production in tutorial dialogs. Our
effort was to create an algorithm which is easily adaptable to
other tutoring domains as well.
We propose to make use of our ontology in mapping de-
scriptions of objects used in the hint categories onto to actual
objects in the domain, which are each time different accord-
ing to the proof under consideration. This facilitates the auto-
matic production of hints, which the socratic algorithm mod-
els, tailored to the needs of a particular student for a particular
attempt with a proof.
The algorithm presented in this paper does not deal with
the realization of the hints. We are currently investigating the
surface structure of hints and their generation. Moreover, the
produced hints do not necessarily complete the tutor’s dialog
turn. Further examination is needed to suggest a model of
dialog moves and dialog specifications in our domain.
In the future we plan to make further developments in our
domain ontology to improve the evaluation of the student’s
answer, for instance, by capturing different degrees of accu-
racy of the parts in the answer (cf. Section 5). We will also
refine our student answer categories and model more subdi-
algues to better treat the refined categories. We are also re-
searching ways of making full use of our hint taxonomy. All
these improvements will, in turn, help us augment the hinting
algorithm.
To meet these aims, we will use the empirical data from our
recently collected corpus on mathematics tutorial dialogs [4]
The corpus was collected through Wizard-of-Oz experiments
with an expert mathematics tutor. Moreover the same data is
precious for our research in the actual sentence level realiza-
11
tion of hints. We are currently working towards incorporatin
gthe study of these empirical data into the research of auto-
matically generating the hints that we can produce.
References
[1]Vicent Aleven, Ocav Popescu, and Kenneth R.
Koedinger. A tutorial dialogue system with knowledge-
based understanding and classification of student expla-
nations. In Working Notes of 2nd IJCAI Workshop on
Knowledge and Reasoning in Practical Dialogue Sys-
tems., Seattle, USA, 2001.
[2]Vincent Aleven and Kenneth Koedinger. The need for
tutorial dialog to support self-explanation. In Ros´e and
Freedman [24], pages 65–73.
[3]Richard Angros, Jr., W. Lewis Johnson, Jeff Rickel, and
Schorel Andrew. Learning domain knowledge for teach-
ing procedural skills. In Proceedings of AAMAS’02,
Bologna, Italy, 2002.
[4]Chris Benzm¨uller, Armin Fiedler, Malte Gabsdil,
Helmut Horacek, Ivana Kruijff-Korbayov´a, Man-
fred Pinkal, J¨org Siekmann, Dimitra Tsovaltzi, Bao
Quoc Vo, and Magdalena Wolska. A Wizard-of-Oz
experiment for tutorial dialogues in mathematics. In
Proceedings of the AIED Workshop on Advanced Tech-
nologies for Mathematics Education, Sidney, Australia,
2003. Submitted.
[5]N.O. Bernsen, H. Dybkjær, and L. Dybkjær. Designing
Interactive Speech Systems — From First Ideas to User
Testing. Springer, 1998.
[6]Jean Carletta, Amy Isard, Stephen Isard, Jacqueline C.
Kowtko, Gwyneth Doherty-Sneddon, and Anne H. An-
derson. The reliability of a dialogue structure coding
scheme. Computational Linguistics, 23(1):13–32, 1997.
[7]Michelene T. H. Chi, Nicholas de Leeuw, Mei-
Hung Chiu, and Christian Lavancher. Eliciting self-
explanation improves understanding. Cognitive Sci-
ence, 18:439–477, 1994.
[8]Mark G. Core and James F. Allen. Coding dialogues
with DAMSL annotation scheme. In AAAI Fall Sym-
posium on Communicative Action in Humans and Ma-
chines, pages 28–35, Boston, MA, 1993.
[9]Martha W. Evens, Stefan Brandle, Ru-Charn Chang,
Reva Freedman, Michael Glass, Yoon Hee Lee,
Leem Seop Shim, Chong Woo Woo, Yuemei Zhang,
Yujian Zhou, Joel A. Michael, and Allen A. Rovick.
CIRCSIM-Tutor: An intelligent tutoring system using
natural language dialogue. In Proceedings of 12th Mid-
west AI and Cognitive Science Conference, MAICS-
2001, pages 16–23, Oxford OH, 2001.
[10]Armin Fiedler and Helmut Horacek. Towards under-
standing the role of hints in tutorial dialogues. In
BI-DIALOG: 5th Workshop on Formal Semantics and
Pragmatics in Dialogue, pages 40–44, Bielefeld, Ger-
many, 2001.
[11]Armin Fiedler and Dimitra Tsovaltzi. Automating hint-
ing in mathematical tutorial dialogue. In Proceedings of
the EACL-03 Workshop on Dialogue Systems: interac-
tion, adaptation and styles of management, pages 45–
52, Budapest, 2003.
[12]Michael Glass. Processing language input in the
circsim-Tutor intelligent tutoring system. In Johanna
Moore, Carol Luckhardt Redfield, and W. Lewis John-
son, editors, Artificial Intelligence in Education, pages
210–212. IOS Press, 2001.
[13]Neil Heffernan and Kenneth Koedinger. Intelligent tu-
toring systems are missing the tutor: Building a more
strategic dialog-based tutor. In Ros ´e and Freedman [24],
pages 14–19.
[14]Neil T. Heffernan and Kenneth R. Koedinger. Building
a 3rd generation ITS for symbolization: Adding a tuto-
rial model with multiple tutorial strategies. In Proceed-
ings of the ITS 2000 Workshop on Algebra Learning,
Montr´eal, Canada, 2000.
[15]Gregory Hume, Joel Michael, Allen Rovick, and Martha
Evens. Student responses and follow up tutorial tactics
in an ITS. In Proceedings of the 9th Florida Artificial
Intelligence Research Symposium, pages 168–172, Key
West, FL, 1996.
[16]Gregory D. Hume, Joel A. Michael, Rovick A. Allen,
and Martha W. Evens. Hinting as a tactic in one-on-one
tutoring. Journal of the Learning Sciences, 5(1):23–47,
1996.
[17]Chung Hee Lee, Jai Hyun Seu, and Martha W. Evens.
Building an ontology for CIRCSIM-Tutor. In Proceed-
ings of the 13th Midwest AI and Cognitive Science Con-
ference, MAICS-2002, pages 161–168, Chicago, 2002.
[18]E. Melis, E. Andres, A. Franke, G. Goguadse,
M. Kohlhase, P. Libbrecht, M. Pollet, and C. Ullrich. A
generic and adaptive web-based learning environment.
In Artificial Intelligence and Education, pages 385–407,
2001.
[19]Johanna Moore. What makes human explanations ef-
fective? In Proceedings of the Fifteenth Annual Confer-
ence of the Cognitive Science Society, pages 131–136.
Hillsdale, NJ. Earlbaum, 2000.
[20]Natalie K. Person, Arthur C. Graesser, Derek Harter,
Eric Mathews, and the Tutoring Research Group. Di-
alog move generation and conversation management in
AutoTutor. In Ros´e and Freedman [24], pages 45–51.
[21]Manfred Pinkal, J¨org Siekmann, and Christoph
Benzm¨uller. Projektantrag Teilprojekt MI3 — DIA-
LOG: Tutorieller Dialog mit einem mathematischen
Assistenzsystem. In Fortsetzungsantrag SFB 378 —
Ressourcenadaptive kognitve Prozesse, Universit¨at des
Saarlandes, Saarbr¨ucken, Germany, 2001.
[22]Jeff Rickel, Rajaram Ganeshan, Charles Rich, Can-
dance L. Sidner, and Neal Lesh. Task-oriented tuto-
rial dialogue: Issues and agents. In Ros´e and Freedman
[24], pages 52–57.
12
[23]Carolyn P. Ros´e, Johanna D. Moore, Kurt VanLehn, and
David Allbritton. A comparative evaluation of socratic
versus didactic tutoring. In Johanna Moore and Keith
Stenning, editors, Proceedings 23rd Annual Conference
of the Cognitive Science Society, University of Edin-
burgh, Scotland, UK, 2001.
[24]Carolyn Penstein Ros´e and Reva Freedman, editors.
Building Dialog Systems for Tutorial Applications—
Papers from the AAAI Fall Symposium, North Falmouth,
MA, 2000. AAAI press.
[25]J¨org Siekmann, Christoph Benzm ¨uller, Vladimir Brezh-
nev, Lassaad Cheikhrouhou, Armin Fiedler, Andreas
Franke, Helmut Horacek, Michael Kohlhase, Andreas
Meier, Erica Melis, Markus Moschner, Immanuel Nor-
mann, Martin Pollet, Volker Sorge, Carsten Ullrich,
Claus-Peter Wirth, and J¨urgen Zimmer. Proof develop-
ment with
MEGA. In Andrei Voronkov, editor, Auto-
mated Deduction — CADE-18, number 2392 in LNAI,
pages 144–149. Springer Verlag, 2002.
[26]Dimitra Tsovaltzi. Formalising hinting in tutorial dia-
logues. Master’s thesis, The University of Edinburgh,
Scotland, UK, 2001.
[27]Dimitra Tsovaltzi and Armin Fiedler. An approach to fa-
cilitating reflection in a mathematics tutoring system. In
Proceedings of AIED Workshop on Learner Modelling
for Reflection, Sydney, Australia, 2003. Submitted.
[28]Dimitra Tsovaltzi and Armin Fiedler. Enhancement and
use of a mathematical ontology in a tutorial dialogue
system. In IJCAI Workshop on Knowledge and Reason-
ing in Practical Dialogue Systems, Acapulco, Mexico,
2003. In press.
[29]Dimitra Tsovaltzi and Colin Matheson. Formalising
hinting in tutorial dialogues. In EDILOG: 6th workshop
on the semantics and pragmatics of dialogue, pages
185–192, Edinburgh, Scotland, UK, 2002.
[30]Zhou Yujian, Reva Freedman, Michael Glass, Joel A.
Michael, Allen A. Rovick, and Martha W. Evens. What
should the tutor do when the student cannot answer
a question? In AAAI Press, editor, Proceedings of
the Twelfth International Florida AI Research Society
Conference (FLAIRS-99), pages 187–191, Orlando, FL,
1999.
13
