Content uploaded by Dimitra Tsovaltzi

Author content

All content in this area was uploaded by Dimitra Tsovaltzi on Feb 06, 2015

Content may be subject to copyright.

Automating Hinting in an Intelligent Tutorial Dialog System for Mathematics

Armin Fiedler

Department of Computer Science

Saarland University

afiedler@cs.uni-sb.de

Dimitra Tsovaltzi

Department of Computational Linguistics

Saarland University

dimitra@coli.uni-sb.de

Abstract

Despite empirical evidence that natural language

dialog capabilities are necessary for the success

of tutorial sessions, only few state-of-the-art tutor-

ing systems use natural-language style interaction.

Moreover, although hinting has been psychologi-

cally substantiated, most intelligent tutoring sys-

tems do not systematically produce hints. In the DI-

ALOG project, we aim at a mathematical tutoring

system that employs an elaborate natural language

dialog component and a hinting tutoring strategy.

To tutor mathematics, we use a formally encoded

mathematical theory including deﬁnitions and the-

orems along with their proofs. We enhance this on-

tology by making relations explicit and show how

these relations can be used when planning the next

utterance of the system. Moreover, we deﬁne a

scheme for classifying the student’s input in terms

of the knowledge of the domain demonstrated. Fi-

nally, as a theory of tutoring we deﬁne a taxonomy

of hints and a hint determining algorithm that im-

plements the socratic tutoring strategy, whose de-

cisive characteristic is the use of hints in order to

achieve self-explanation. This algorithm takes into

account both the mathematical ontology and the

categories of the student’s answers.

1 Introduction

Despite empirical evidence that natural language dialog ca-

pabilities are necessary for the success of tutorial sessions

[19], only few state-of-the-art tutoring systems use natural-

language style interaction that requires menu-based input or

exact wording of the input [20; 2; 13]. Since domain knowl-

edge, tutoring and pedagogical knowledge, and dialog man-

agement are tightly intertwined, the modeling and integration

of proper natural language dialog capabilities in a tutoring

system turns out to be barely manageable.

Although there has been psychological evidence [7; 23]for

the high educational effect of hinting, most intelligent tutor-

ing systems do not take advantage of the ﬁndings. Little sys-

tematic research has been done in automating hinting [16; 10;

26].

In the DIALOG project [21], we aim at a mathematical

tutoring system that employs an elaborate natural language

dialog component. To tutor mathematics, we need a formally

encoded mathematical theory including deﬁnitions and the-

orems along with their proofs, means of classifying the stu-

dent’s input in terms of the knowledge of the domain, and a

theory of tutoring that should make use of hints.

Since it might be impossible to precompile all possible

proofs for a given theorem, we make use of the state-of-the-

art theorem prover

MEGA [25]with its mathematical knowl-

edge base and start with restricting ourselves to the domain of

naive set theory.

A mathematical tutoring system must be able to tutor

proofs in a way that not only helps the student understand

the current proof, but also allows for a high learning ef-

fect. What is meant by the latter is the ability of the stu-

dents not only to better understand the problem at hand, but

also to generalize and apply the taught methodology with-

out help later on. We propose to establish those tutoring

aims by making use of the socratic tutoring method [20;

23]. The decisive characteristic of the socratic method is the

use of hints to achieve self-explanation, as opposed to an-

swers and explanations being provided constantly by the tu-

tor, which is characteristic of the didactic method [23; 29].

Hinting is a method that aims at encouraging active learn-

ing. It can take the form of eliciting information that the stu-

dent is unable to access without the aid of prompts, or infor-

mation which he can access but whose relevance he is un-

aware of with respect to the problem at hand. Alternatively, a

hint can point to an inference that the student is expected to

make based on knowledge available to him, which helps the

general reasoning needed to deal with a problem [16].

Our approach to modeling hinting involves enhancing the

existing mathematical knowledge base of the theorem prover

MEGA. We do that by making the relations between enti-

ties explicit. These relations can potentially be used in the

automatic production of hints. Note that in this paper we do

not address the issue of natural language realization of hints,

which is also an ongoing research topic of our project. To

the end of automatically producing hints we have been devel-

oping a taxonomy of hints for the naive set theory domain,

which draws on the domain knowledge representation of the

enhanced mathematical ontology. We have also derived a stu-

dent answer categorization scheme in order to classify the stu-

dent’s answers. The taxonomy and the student answer cate-

gories are used by our hinting algorithm which models the

socratic tutoring method and by employing an implicit stu-

dent model.

The implicit student model of the hinting algorithm makes

use of the student answer categorization and of information

on the domain knowledge demonstrated, or already provided

by previous hints produced. Its aim is to use information from

the running tutoring and the current progress of the student

on the particular problem under considerationin order to pro-

duce hints. Those hints, in turn, intend to address the cog-

nitive state of the student and cause him to reﬂect upon his

reasoning. We are only concerned here with the process of

making the student actively reﬂect on his answers. However,

DIALOG uses ACTIVEMATH [18]as an external facility for

user modeling.

In this paper, we ﬁrst provide a comprehensive description

of our domain ontology (Section 2) and its use (Section 3).

In Section 4 we give an overview of the taxonomy of hints.

In Section 5 we look at our student answer categorization

scheme and in Section 6 the hinting algorithm is presented.

Next, in Section 7 an example dialog demonstrates the use of

the research presented here as envisaged for our system. Fi-

nally, we brieﬂy discuss some related work in Section 8 and

conclude the paper.

2 A Mathematical Ontology for Hinting

The proof planner

MEGA [25]makes use of a mathematical

database that is organized as a hierarchy of nested mathemati-

cal theories. Each theory includes deﬁnitions of mathematical

concepts, lemmata and theorems about them, and inference

rules, which can be seen as lemmata that the proof planner

can directly apply. Moreover, each theory inherits all def-

initions, lemmata and theorems, which we will collectively

call assertions henceforth, as well as all inference rules from

nested theories. Since assertions and inference rules draw

on mathematical concepts deﬁned in the mathematical the-

ories, the mathematical database implicitly represents many

relations that can potentially be made use of in tutorial dia-

log. Further useful relations can be found when comparing

the deﬁnitions of concepts with respect to common patterns.

In this section, we shall ﬁrst show in Section 2.1 a part of

MEGA’s mathematical database, which we shall use as an

example domain henceforth. Then, we shall deﬁne in Sec-

tion 2.2 the relations to be used in the hinting process.

2.1 A Mathematical Database

In

MEGA’s database, assertions are encoded in a simply-

typed

-calculus, where every concept has a type and well-

formedness of formulae is deﬁned by the type restrictions.

In this paper, we will concentrate on the mathematical con-

cepts from naive set theory given in Table 1. These concepts

draw on the types sets and inhabitants of sets. We give the

deﬁnitions of the mathematical concepts in intuitive terms as

well as in more formal terms paraphrasing but avoiding the

-calculus encoding of

MEGA’s database.

Let

be sets and let

be an inhabitant.

: element

: not element

: intersection

: union

: subset

: not subset

: strict subset

: not strict subset

: superset

: not superset

: strict superset

: not strict superset

: powerset

Table 1: Mathematical concepts.

Element: The elements of a set are its inhabitants:

if

and only if

is an inhabitant of

.

Intersection: The intersection of two sets is the set of their

common elements:

and

.

Union: The union of two sets is the set of the elements of

both sets:

or

.

Subset: A set is a subset of another set if all elements of the

former are also elements of the latter:

if and only

if for all

follows that

.

Strict Subset: A set is a strict subset of another set if the latter

has at least one element more:

if and only if

and there is an

such that

.

Superset: A set is a superset of another set if all elements of

the latter are also elements of the former:

if and

only if for all

follows that

.

Equality: Two sets are equal if they share the same elements:

if and only if for all

follows that

and for all

follows that

.

Powerset: The powerset of a set is the set of all its subsets:

"!#$%&'

(

.

The deﬁnition of the negated concepts from Table 1 is

straightforward.

Furthermore, we give examples of lemmata and theorems

that use some of these concepts. Let

,

and

)

be sets:

Commutativity of Union:

*

.

Equality of Sets: If

and

then

+

.

Union in Powerset: If

)$!

and

)$!

then

)$!

.

Finally, we give some examples for inference rules. Infer-

ence rules have the following general form:

,.-0/1211/',43

5 6

where

7#89;:<:<:;'7>=

,

?@BA

are the premises, from which the

conclusion

C

is derived by rule

D

.

Let

and

be sets. Example inference rules are:

EGFIH HJFE

ELKLH

Set=

E&MNPORQTSUH+MNPORQ&S

EWVHJMNPORQTS

VN

Note that the left inference rule encodes the lemma about

the equality of sets, and the right inference rule encodes the

lemma about the union in powersets. Since every lemma and

2

theorem can be rewritten as an inference rule and every infer-

ence rule can be rewritten as a lemma or theorem, we identify

lemmata and theorems with inference rules henceforth.

2.2 Enhancing the Ontology

The mathematical database implicitly represents many rela-

tions that can be made use of in tutorial dialog. Further useful

relations can be found when comparing the deﬁnitions of con-

cepts with respect to common patterns. We consider relations

between mathematical concepts, between mathematical con-

cepts and inference rules, and among mathematical concepts,

formulae and inference rules. By making these relations ex-

plicit we convert the mathematical database into an enhanced

ontology that can be used in hinting.

Mathematical Concepts

Let

4

be mathematical concepts. We deﬁne the following

relations between mathematical concepts:

Antithesis:

is in antithesis to

if and only if it is its oppo-

site concept (i.e., its logical negation).

Examples: antithesis

!

,antithesis

!

Duality:

is dual to

if and only if

is deﬁned in terms of

7

C

and

is deﬁned in terms of

C

7

for some

formulae

7 C

.

Examples: dual

!

,dual

!

Junction:

is in a junction to

if and only if

is deﬁned in

terms of

7#8

"7>=

and

is deﬁned in terms of

7 8

7=

, or vice versa, for some formulae

78<:;:<:;'7 =

,

where

?

.

Examples: junction

!

Hypotaxis:

is in hypotaxis to

if and only if

is deﬁned

using

. We say,

is a hypotaxon of

, and

is a

hypertaxon of

.

Examples: hypotaxon

!

,hypotaxon

!

Primitive:

is a primitive if and only if it has no hypotaxon.

Examples: primitive

!

Note that

is a primitive in

MEGA’s database, since

it is deﬁned using inhabitant, which is a type, but not a

deﬁned concept.

Mathematical Concepts and Inference Rules

Let

,

be mathematical concepts and

D

be an inference

rule. We deﬁne the following relations:

Relevance:

is relevant to

D

if and only if

D

can only be

applied when

is part of the formula at hand (either in

the conclusion or in the premises).

Examples: relevant

Set=

!

,relevant

Set=

!

,

relevant

Set=

!

Dominance:

is dominant over

for rule

D

if and only if

appears in both the premises and the conclusion, but

does not.

has to appear in one of the premises or the

conclusion.

Examples: dominant

!

,

Mathematical Concepts, Formulae and Inference Rules

Let

be a mathematical concept,

D

be an inference rule and

7#8%;:<:;: '7>= 'C

formulae, where

7#8%<:;:<:<27>=

are the premises

and

C

the conclusion of

D

.

Introduction: Rule

D

introduces

if and only if

occurs

in the conclusion

C

, but not in any of the premises

7#8 <:<:;: '7>=

.

Examples: introduces

Set=

!

,introduces

!

Elimination: Rule

D

eliminates

if and only if

occurs in

at least one of the premises

78;:<:;:<'7 =

, but not in the

conclusion

C

.

Examples: eliminates

Set=

!

,eliminates

Set=

!

Automation

The automatic enhancement of the mathematical database by

explicitly adding the relations deﬁned previously is straight-

forward. The automation of the enhancement allows us to

plug in any mathematical database and to convert it accord-

ing to the same principles into a database that includes the

relations we want to make use of in hinting.

3 Making Use of the Ontology

In this section we further explain the use of the mathematical

ontology by pointing out its exact relevance with regard to the

general working framework. We also give a few examples of

the application of the ontology in automating hint production.

The mathematical ontology is evoked primarily in the re-

alization of hint categories chosen by the socratic algorithm

described in Section 6. Due to the adaptive nature of the al-

gorithm, and our goal to dynamically produce hints that ﬁt

the needs of the student with regard to the particular proof,

we cannot restrict ourselves to the use of a gamed of static

hints. The algorithm takes as input the number and kind of

hints produced so far, the number of wrong answers by the

student and the categories of the current and previous student

answer. It computes the appropriate hint category to be pro-

duced next from the taxonomy of hints. The hint category is

chosen with respect to the implicit student model that the al-

gorithm’s input constitutes. This means that for every student

and for his current performance on the proof being attempted,

the hint category chosen must be realized in a different way.

Each hint category in the hint taxonomy is deﬁned based

on generic descriptions of domain objects or relations. The

role of the ontology is to map the generic descriptions on the

actual objects or relations that are used in the particular con-

text, that is, in the particular proof and the proof step at hand.

Another equally signiﬁcant use of the domain ontology is

in categorizing the student’s answers. This use is a side-effect

of the involvement of the ontology in automatically choosing

hints. That is, the algorithm takes as input the analyzed stu-

dent answer. In analyzing the latter, we compare it to the

expected answer (see Section 5) and then look for the em-

ployment of necessary entities.

The necessary entities are deﬁned in terms of the ontology.

The algorithm checks for the student’s level of understanding

by trying to track the use of these entities in the student’s an-

swer to be addressed next. The hint to be produced is then

3

picked according to the knowledge demonstrated by the stu-

dent. Note that this knowledge might have already been pro-

vided by the system itself, in a previous turn when dealing

with the same performable step. Since the algorithm only

checks for generic descriptions of those concepts, we suggest

the use of the present ontology to map the descriptions onto

the actual concepts relevant to the particular context.

Let us now give examples of the use of our domain ontol-

ogy. All the relations mentioned here, which are deﬁned in

the ontology, have been explained in Section 2.

For the hint

give-away-relevant-concept

, which points out

the right mathematical concept that the student has to bring

into play in order to carry the proof out, we use the domain

ontology in the following way:

1. If the inference rule to be applied involves the elimination

or the introduction of a concept, then we identify the rel-

evant concept with the concept eliminated or introduced,

respectively.

2. Otherwise, we look for a relevance relation and identify

the relevant concept with that in the relation.

To produce the hint

elaborate-domain-object

we have to

ﬁnd in the domain the exact way an inference rule needs to be

applied, for instance, whether it involves an elimination or an

introduction. The student will be informed accordingly.

For more on the construction and use of the mathematical

ontology see [28].

4 A Taxonomy of Hints

In this section we explain the philosophy and the structure of

our hint taxonomy. We also look into some hints that are used

in the algorithm. The names of the categories are intended

to be as descriptive of the content as possible, and should

in some cases be self-explanatory. The taxonomy includes

more than the hint categories mentioned in this section. The

full taxonomy is given in Table 2. Some categories are not

real hints (e.g.,

point-to-lesson

), but have been included in the

taxonomy since they are part of the general hinting process.

4.1 Philosophy and Structure

Our hint taxonomy was derivedwith regard to the underlying

function that can be common for different surface realizations

of hints. The underlying function is mainly responsible for

the educational effect of hints. Although the surface structure,

which undoubtedly plays its own signiﬁcant role in teaching,

is also being examined in the our project, we do not address

this issue in this paper.

We deﬁned the hint categories based on the needs in the

domain. To estimate those needs we made use of the objects

and the relations between them as deﬁned in the mathematical

ontology. An additional guide for deriving hint categories

that are useful for tutoring in our domain was a previous hint

taxonomy, which was derived from the BE&E corpus [26].

The structure of the hint taxonomy reﬂects the function

of the hints with respect to the information that the hint ad-

dresses or is meant to trigger. To capture different functions

of a hint we deﬁne hint categories across two dimensions.

Before we introduce the dimensions, let us clarify some

terminology. In the following, we distinguish performable

steps from meta-reasoning. Performable steps are the steps

that can be found in the formal proof. These include premises,

conclusion and inference methods such as lemmata, theo-

rems, deﬁnitions of concepts, or calculus-level rules. Meta-

reasoning steps consist of everything that leads to the per-

formable step, but cannot be found in the formal proof. To

be more speciﬁc, meta-reasoning consists of everything that

could potentially be applied to any particular proof. It in-

volves general proving techniques. As soon as a general tech-

nique is instantiated for the particular proof, it belongs to the

performable step level.

The two hint dimensions consist of the following classes:

1. active vs. passive

2. domain-relation vs. domain-object vs. inference-rulevs.

substitution vs. meta-reasoning vs. performable-step

In the second dimension, we ordered the classes with re-

spect to their subordination relation. We say, that a class is

subordinate to another one if it reveals more information.

Each of these classes consists of single hint categories that

elaborate on one of the attributes of the proof step under con-

sideration. The hint categories are grouped in classes accord-

ing to the kind of information they address in relation to the

domain and the proof. By and large, the hints of the pas-

sive function of a class in the second dimension constitute

the hints of the active function of its immediately subordinate

class, in the same dimension. In addition, the class of prag-

matic hints belongs to the second dimension as well, but we

deﬁne it such that it is not subordinate to any other class and

no other class is subordinate to it.

In the following section we look at the structure of the

second dimension just described through some examples of

classes and the hints deﬁned in them.

4.2 First Dimension

The ﬁrst dimension distinguishes between the active and pas-

sive function of hints. The difference lies in the way the infor-

mation to which the tutor wants to refer is approached. The

idea behind this distinction resembles that of backward- vs.

forward-looking function of dialog acts in DAMSL [8]. The

active function of hints looks forward and seeks to help the

student in accessing a further bit of information, by means

of eliciting, that will bring him closer to the solution. The

student has to think of and produce the answer that is hinted

at.The passive function of hints refers to the small piece of in-

formation that is provided each time in order to bring the stu-

dent closer to some answer. The tutor gives away some infor-

mation, which he has normally unsuccessfully tried to elicit

previously. Due to that relation between the active and pas-

sive function of hints, the passive function of one hint class

in the second dimension consists of hint categories that are

included in the active function in its subordinate class.

4.3 Second Dimension

In this section we will give a few examples of classes and hint

categories that capture the structure of the second dimension.

4

active passive

domain-relation elicit-antithesis give-away-antithesis

elicit-duality give-away-duality

elicit-junction give-away-junction

elicit-hypotaxis give-away-hypotaxis

elicit-specialization give-away-specialization

elicit-generalization give-away-generalization

domain-object give-away-antithesis give-away-relevant-concept

give-away-duality give-away-hypotactical-concept

give-away-junction give-away-primitive-concept

give-away-hypotaxis

give-away-specialization

give-away-generalization

inference rule give-away-relevant-concept give-away-inference-rule

give-away-hypotactical-concept

give-away-primitive-concept

elaborate-domain-object

substitution give-away-inference-rule spell-out-substitution

elicit-substitution

meta-reasoning spell-out-substitution explain-meta-reasoning

performable-step explain-meta-reasoning give-away-performable-step

confer-to-lesson

pragmatic ordered-list take-for-granted

unordered-list point-to-lesson

elicit-discrepancy

Table 2: The taxonomy of hints.

Domain-relation hints address the relations between math-

ematical concepts in the domain, as described in Section 2.

The passive function of domain-relation hints is the active

function of domain-object hints, that is, they are used to elicit

domain objects.

Domain-object hints address an object in the domain. The

hint

give-away-relevant-concept

names the most prominent

concept in the proposition or formula under consideration.

This might be, for instance, the concept whose deﬁnition the

student needs to use in order to proceed with the proof, or

the concept that will in general lead the student to under-

stand which inference rule he has to apply. Other examples in

the class are

give-away-hypotactical-concept

and

give-away-

primitive-concept

. The terms hypotactical and primitive con-

cept refer to the relation, based on the domain hierarchy, be-

tween the addressed concept and the original relevant con-

cept, which the tutor is trying to elicit. Since this class is sub-

ordinate to domain-relation, the hints in it are more revealing

than domain-relation hints. The passive function of domain-

object hints is used to elicit the applicable inference rule, and,

therefore, is part of the active function of the respective class.

The same structure holds for inference-rule,substitution,

meta-reasoning and performable-step hints.

Finally, the class of pragmatic hints is somewhat differ-

ent from other classes in that it makes use of minimal do-

main knowledge. It rather refers to pragmatic attributes of the

expected answer. The active function hints are

ordered-list

,

which speciﬁcally refers to the order in which the parts of the

expected answer appear,

unordered-list

, which only refers to

the number of the parts, and

elicit-discrepancy

, which points

out that there is a discrepancy between the student’s answer

and the expected answer. The latter can be used in place of

all other active hint categories.

Take-for-granted

asks the stu-

dent to just accept something as a fact either when the student

cannot understand the explanation or when the explanation

would require making use of formal logic.

Point-to-lesson

points the student to the lesson in general and asks him to read

it again when it appears that he cannot be helped by tutoring

because he does not remember the study material. There is

no one-to-one correspondence between the active and pas-

sive pragmatic hints. Some pragmatic hints can be used in

combination with hints from other classes.

For more on the taxonomy of hints see [11].

5 Student Answer Categorization Scheme

In this section we present a scheme for categorizing student

answers. We are only concerned here with the parts of the an-

swer that address domain knowledge. We deﬁne student cate-

gories based on their completeness and accuracy with respect

to the expected answer. The latter is always approximated for

the student’s own line of reasoning. The output of the classi-

ﬁcation constitutes part of the input to the hinting algorithm,

which models the hinting process.

5.1 Proof Step Matching

The student’s answer is evaluated by use of an expected an-

swer. The expected answer is the proof step which is expected

5

next according to the formal proof whichthe system has cho-

sen for the problem at hand. We want to make use of the stu-

dent’s own reasoning in helping him with the task and avoid

super-imposing a particular solution. We model that by try-

ing to match the student’s answer to an proof step in one of a

set of proofs. To this end we use the state-of-the-art theorem

prover

MEGA [25].

5.2 Parts of Answers and Over-Answering

We now deﬁne the relevant units for the categorization of the

student answer. A part is a premise, the conclusion or the in-

ference rule of a proof step. The two former are mathematical

formulae and must be explicitly mentioned for the proof to be

complete. The inference rule can either be referred to nomi-

nally, or it can be represented as a formula itself. In the latter

case, we just consider that formula as one of the premises.

It is up to the student to commit to using the rule one way

or the other. A formula is a higher-order predicate logic for-

mula. Every symbol deﬁned in the logic is a function. For-

mulae can constitute of subformulae to an arbitrary degree of

embedding. Constants are 0-ary functions that constitute the

lowest level of entities considered.

We also consider (accurate or inaccurate) over-answering

as several distinct answers. That is, if the student’s answer

has more proof steps than one, we consider the steps as mul-

tiple answers. The categorization is normally applied to them

separately. Nevertheless, there are cases where the order of

the presentation of the multiple answers is crucial. For exam-

ple, we cannot count a correct answer that is inferred from a

previous wrong answer, since the correct answer would have

to follow from a wrong premise.

5.3 Completeness vs. Accuracy

We deﬁne the predicates complete and accurate as follows:

Complete: An answer is complete if and only if all parts of

the expected answer are mentioned.

Accurate: A part of an answer is accurate if and only if the

propositional content of the part is the true and expected

one.

Completeness

From our deﬁnition of completeness it follows that complete-

ness is dependent on domain objects, but not on our domain

ontology. That is, the expected answer, which is the basis of

the evaluation of the completeness of a student answer, nec-

essarily makes use of objects in the domain. However, the

relations of the objects in the domain are irrelevant to evalu-

ating completeness. Completeness is a binary predicate. The

only thing relevant to completeness is the presence or absence

of objects in the student’s answer.

In addition, we distinguish between getting the expected

domain object right and instantiating it correctly. The latter

does not follow from the former. Completeness is relevant to

the presence of the object but not to its correct instantiation.

In other words, a place holder for an expected object in the

answer is enough for attributing completeness, no matter if

the object itself is the expected one. That issue is dealt with

by accuracy.

Accuracy

Accuracy, contrary to completeness, is dependent on the do-

main ontology. It refers to the appropriateness of the object

in the student answer with respect to the expected object. An

object is accurate, if and only if it is the exact expected one.

Currently we are using accuracy as a binary predicate in the

same way that we do with completeness. However, we intend

to extend our categorization to include different degrees of

accuracy.

5.4 The Categories

In this section we enumerate the categories of students an-

swers based on our deﬁnitions of completeness and accuracy

and with regard to the expected answer.

We deﬁne the following student answer categories:

Correct: An answer which is both complete and accurate.

Complete-Partially-Accurate: An answer which is com-

plete, but some parts in it are inaccurate.

Complete-Inaccurate: An answer which is complete, but all

parts in it are inaccurate.

Incomplete-Accurate: An answer which is incomplete, but

all parts that are present in it are accurate.

Incomplete-Partially-Accurate: An answer which is in-

complete and some of the parts in it are inaccurate.

Wrong: An answer which is both incomplete and inaccurate.

For the purposes of this paper, we collapse the cate-

gories complete-partially-accurate, complete-inaccurate and

incomplete-partially-accurate to one category, namely, inac-

curate.

More on our student answer categorization scheme can be

found in [27].

5.5 Subdialogs

Subdialogs are crucial to the correct evaluation of the student

answer and supplementary to the student answer categoriza-

tion scheme just presented. The tutor can initiate subdialogs,

for example, in case of ambiguity in order to resolve it. More-

over, students are given the opportunity to correct themselves,

provide additional information on their reasoning and give es-

sential information about the way they proceed with the task.

Student’s can as well initiate subdialogs, usually with clariﬁ-

cation questions or requests for particular information.

An example of a subdialog initiation by the tutor is the

case of potential “typos”. We want to treat them differently

from conceptual domain mistakes, such as, wrong instanti-

ations. We can prevent that by asking the student what he

really meant to say. Our assumption is that if the student

really just used the wrong symbol and did not make a con-

ceptual domain mistake, he will realize it and correct it. If

he does not correct it, we categorize the answer taking into

account the domain mistake. This kind of subdialog we iden-

tify with alignment [6], and an instance of it is the example

in Figure 11. The tutor could not make sense of the student’s

1The examples presented here are from our recently collected

corpus on mathematics tutorial dialogs in German [4]. Translations

are included, where necessary.

6

utterance unless he substituted

for

and

!

for

!

. The student could not correct

himself, although he realized from the tutor’s question that he

had used the wrong symbol.

Student (5): wenn

F

O

S

, dann

K

, weil

K

O

S

[if

F

O

S

, then

K

, because

K

O

S

]

Tutor (6): meinen Sie wirklich

K

oder etwas anderes?

[Do you really mean

K

or something else?]

Student (7):

F

Figure 1: Example of alignment, Subject 13

Other cases of subdialog initiation include instances when

the student’s answer cannot be classiﬁed, or when a student

gives an answer which is in principle correct but not part of

the current proof.

6 A Hinting Algorithm

A tutoring system ideally aims at having the student ﬁnd the

solution to a problem by himself. Only if the student gets

stuck should the system intervene. There is pedagogical ev-

idence [7; 23]that students learn better if the tutor does not

give away the answer but instead gives hints that prompt the

student for self-explanations. Accordingly, based on [26]we

have derived an algorithm that implements an eliciting strat-

egy that is user-adaptive by choosing hints tailored to the

students. Only if hints appear not to help does the algo-

rithm switch to an explaining strategy, where it gives away

the answer and explains it. We shall follow Person and col-

leagues [20]and Ros´e and colleagues [23]in calling the elic-

iting strategy socratic and the explaining strategy didactic.

6.1 Description of the Hinting Algorithm

We shall now present an algorithm that implements the so-

cratic strategy. In intuitive terms, the algorithm aims at hav-

ing the student ﬁnd the proof by himself. If the student does

not know how to proceed or makes a mistake, the algorithm

prefers hinting at the right solution in order to elicit the prob-

lem solving instead of giving away the answer. An implicit

student model makes the algorithm sensitive to students of a

different level by providing increasingly informative hints.

The algorithm takes as input the number and kind of hints

produced so far, the number of wrong answers by the student

and the current and previous student answer category. The

particular input to the algorithm is the category that the stu-

dent answer has been assigned, based on our student answer

categorization scheme, and the domain knowledge employed

in the answer. The category of the student answer in combi-

nation with the kinds of hints already produced and the use

of required entities of the mathematical ontology together in-

form the algorithm of the level of the student’s knowledge.

We described the use of the mathematical ontology in that

context in Section 3.

Moreover, the algorithm computes whether to produce a

hint and which category of hint to produce, based on the num-

ber of wrong answers, the number and kind of hints already

produced, as well as the domain knowledge demonstrated by

the student. The algorithm computes the appropriate hint cat-

egory to be produced next from the taxonomy of hints.

If the hinting does not effect correct student answers af-

ter several hints the algorithm switches to a didactic strategy,

and, without hinting, explains the steps that the student cannot

ﬁnd himself. Nevertheless, the algorithm continues to ask the

student for the subsequent step. If the student gives correct

answers again and, thus, the tutor need not explain anymore,

the algorithm switches back to the socratic strategy. In effect,

hints are provided again to elicit the step under consideration.

The Main Algorithm

The essentials of the algorithm are as follows:

"!#!$%'&)(!+*,!-./0,"21

"3(4 "!-.56!7 68(39#:&;"3

2/0&<&;"!

=5?>(@>$/0A#B3C<&DEF153(G:>$(F&DHI3

J8&DE@152(G2K &+>$(@&HI3 60&:*,! " "2*L

"3(M>7*3*,256<>(@&HN2 6>(@1PO!Q!R&D3S7

2/0&<*3>//T"E(@*,0!(

socratic

!(P80(@>7*3*,E@ ">$

>$(@1P-.0&"&(O.@> DU&6!$%<&DE@13(G2K &+>$(F&DHI3

The Function socratic

The bulk of the work is done by the function socratic,

which we only outline here. The function takes as an argu-

ment the category

V

of the student’s current answer. If the

origin of the student’s mistake is not clear, a clariﬁcation di-

alog is initiated, which we do not describe here. Note, how-

ever, that the function stops if the student gives the correct

answer during that clariﬁcation dialog, as that means that the

student corrected himself. Otherwise, the function produces

a hint in a user-adaptive manner.

The function socratic calls several other functions,

which we look at subsequently.

Let

W

denote the number of hints produced so far and

VCX

8

the category of the student’s previous answer. The hint is then

produced as follows:

Y

>&

W

JA

Z

V

'&6H) "!(OQ! )0(@>*2*,E U>[":"3(4*2>$/0/

elicit

Z

V

'&60(@*,!7-Q@/3,\]>*2*,E U>["

2(^@ !515E@*38>$(_>*L"`7 ">O-R>["0*0(G

a

that is,

ordered-list

, or

unordered-list

b

Y

>&

W

Z

V

'&6H) "!(O

2(^

V+X

8

0&6H) "!(@OQ!7 6(@*3!-./0,",\]>*3*3E U>[

"3(4*2>$/0/

up-to-inference-rule

V+X

8

0&60(@>7*3*,E@ ">$

"3(4*2>$/0/

elicit-give-away

VX

8

0&)*3! " c*L

"3(4*2>$/0/

elicit

3/'&D8*3>//

elicit

Y

>&

W

d

Z

V

'&6H) "!(O

2(^e'&)'&I" U1^H) !7(O.>$(@&HN2

"3(_ "!515E@*3

explain-meta-reasoning

2/0&Z "3`#0!E@&I0(G+HI>7&)>$(_>*L"`7

&Ef@&DE5"!7(^@(G

2(_ !515EF*,

spell-out-substitution

3/'&De@ 2`G0!EF&6(G+H6>&

spell-out-substitution

"3(_ "!515E@*,

give-away-performable-step

2/0&8*3>$/0/

up-to-inference-rule

3/'&D8*3>//

elicit-give-away

7

Y

>&

W

Z

V

'&6H) "!(O.>$(@1^+0&)>$+/c>&D6" U1PH) !7(OR>$(@&HN2

2(^@ !515E@*3

point-to-lesson

>$(F1P&D!7

a

The student is asked to read the lesson again. Afterwards,

the algorithm starts anew.

b

3/'&D "!515E@*,

explain-meta-reasoning

Y

>&

W

@

O0`8> HI> A >(@&HN2 +>$(@1_&H)ZU*U_! 15'1>*,'*<&D U>[2OA

&DH)"*U f@>7*

>$ 3 M "3 *3!(@&2*3E50`*3! " c*L >(@&DHI3 U&

H)Z"_>// *3!E(G"3 U&N "2&,

a

After four hints, the algorithm starts to guide the student more

than before to avoid frustration. It switches to the didactic strat-

egy in the fashion described a the beginning of the section.

b

After having produced a hint the function socratic analyzes

the student’s answer to that hint. If the student’s answer is still

not right the function socratic is recursively called. How-

ever, if the student answers correctly and at least two hints

have been produced the algorithm re-visits the produced hints

in the reverse order to recapitulate the proof step and to make

sure the student understands the reasoning so far. This is done

by producing a sequence of active meta-reasoning hints, one

for each hint that have addressed the current step of the proof,

in the reverse order. If the active meta-reasoning hints get

the student to say anything but the right answer, the algo-

rithm produces an

explain-meta-reasoning

hint. This is done

to avoid frustrating the student as his performance is poor.

The function socratic calls several functions, which we

present now. The functions are self-explanatory.

Function elicit

%"<&;"E@152(7

#(![H+&N"8 2/2`[>$(G+*,!(F*,3

@3(_Ze@<&;"E@153(G

#(![H+&N@8(52 2(@*,8 "E/0

2(4*3>$/0/

up-to-substitution

3/'&D8*2>$/0/

up-to-inference-rule

3/'&*2>$/0/

elicit-relevant-concept

Function elicit-give-away

e "3`#0!E@&I0(G+HI>7&)>$(_>*,0`815!-R>(5\ !f

;2*L60(G

@3(4*3>//

up-to-inference-rule

3/'&C 2`#!7E@&60(7+H6>&6>Q@>7&&`7815!-R>$0(5\ !f

;c*L60(7

"3(_ "!515E@*3

elaborate-domain-object

2/0&Z? "3`#0!E@&I0(G+HI>7&

elaborate-domain-object

2(M*3>//

up-to-substitution

3/'&D8@ !515E@*3

give-away-performable-step

>(@1

spell-out-substitution

a

this is to explain the substitution

b

Function up-to-inference-rule

e "3`#0!E@&I0(G+HI>7&

give-away-hypotactical-concept

@3(_ "!#1E@*,

give-away-inference-rule

3/'&C 2`#!7E@&60(7+H6>&

give-away-relevant-concept

"3(_ "!515E@*3

elaborate-domain-object

2/0&8Z "3`#0!E@&I0(G+HI>7&

elaborate-domain-object

2(_ !515EF*,

give-away-hypotactical-concept

3/'&D "!515E@*,

give-away-relevant-concept

Function up-to-substitution

e "3`#0!E@&I0(G+HI>7&

elicit-substitution

@3(_ "!#1E@*,

spell-out-substitution

3/'&C 0OG)(3 "3(@*38 E/0

#(![H)(

"3(_ "!515E@*3

elicit-substitution

2/0& "!#1E@*,

give-away-inference-rule

Function elicit-relevant-concept

%"<&;"E@152(7

#(![H+&6>R*3!(@*3356F>[+'&6 2/0>$21 !."

"3/03`[>$(G+*,!7(@*,25

@3(_ "!#1E@*,8>$(_>*,0`815!7-.>(\ !7f

;2*,6@(G

3/'&C@ !515E@*3

give-away-relevant-concept

The algorithm we just presented can easily be adapted to

other domains, because the hint taxonomy is independent

from the algorithm and only called by it. That means that the

only thing that has to be adapted for applying the algorithm

to different domains are the hint categories in the taxonomy,

while the structure of the taxonomy has to remain the same.

Deﬁning adapted hint categories can be done through the do-

main ontology, which is by deﬁnition domain speciﬁc.

7 An Example Dialog

Let us consider the following dialog between the tutoring sys-

tem and a student on a simple proof. This will help elucidate

how the algorithm for the socratic tutoring strategy proceeds

and how it makes use of the hinting categories, the enhanced

domain ontology, and the implicit student model. The ex-

cerpt comes from our recently collected corpus of tutorial di-

alogs in mathematics in German [4]. The subject is number

23. The tutoring system, denoted as tutor in the following,

is teaching the proof of the proposition “If

!

, then

!

”, where

stands for the complement of a set

which is deﬁned as follows:

If

is the universal set, and

is its subset (i.e.,

), we

call the difference simply the complement of

and denote it

brieﬂy by

"!

; thus

!

.

Note that the corpus was collected via a Wizard of Oz ex-

periment [5]. That means that the system is only partially

implemented. Subjects interacted through an interface with a

human “wizard” simulating the behavior of a system 2.

We give the example dialog in chunks which consist of

one tutor and one student turn. The tutor’s turn includes a

name of hint ﬂag. The student’s turn includes the answer cat-

egory ﬂag. The explanation of the behavior of the tutor in

each chunk precedes it every time. The original German tran-

scriptions are provided and with them translations in English,

where necessary.

At the beginning of the session, the master proof, which

will be taught, has not yet been determined. This will be

done gradually, as the student will be ’committing’ himself to

a particular proof by the choice of steps he makes (See Sec-

tion 5.1). However, we provide here the proof actually taught

in the particular example ﬁrst in order to help the reader fol-

low it. The proof was given in reality only as a summary

at the very end of the tutoring session and was presented as

follows:

Tutor (8): Ich zeige Ihnen nun den gesamten Beweis:

Zun¨

achst setzen wir die G¨ultigkeit von

F

O

S

vo-

raus, denn dies ist die Voraussetzung. Dann nehmen wir

ein beliebiges Element

M

und zeigen, daß dieses

auch in

O

S

sein muß. Sei also

in

. Dann ist

nicht

in

O

S

und deshalb nach Voraussetzung auch nicht in

. Wenn

aber nicht in

ist, so ist es in

O

S

, was den

Beweis abschließt, da

beliebig war.

2For details on the experiment see [4]

8

[I will show you the complete proof now : First we assume

the validity of

F

O

S

, as this is the assumption. Then

we take an arbitrary element

M

Q

and show, that it has

to be in

O

S

as well. Let

be in

. Then

is not in

O

S

and therefore according to the assumption also not

in

. But if

is not in

, it is in

O

S

, which concludes

the proof as

was arbitrary.]

When the algorithm starts, the counter

W

for the produced

hints is initialized to

A

. Since the proof is not completed, the

tutor prompts for the ﬁrst step by introducing the task.

Tutor (1): Bitte zeigen Sie : Wenn

F

O

S

, dann

F

O

S

!

[Please show the following: If

F

O

S

, then

F

O

S

!]

Student (1): (wrong)

F

This answer is categorized as wrong, since it does not con-

tain any accurate parts. Thus the algorithm calls the function

socratic. Since

W

A

, the functions elicit is called

and, through that, elicit-relevant-concept. The latter

checks the student’s knowledge by matching something in the

answer to the relevant concept needed. That is done by help

of the domain ontology. The student has not used any of the

concepts that can be matched to the relevant concept of a pos-

sible proof. Thus, the system has to choose the ﬁrst step to

teach. Eventually a hint of type

give-away-relevant-concept

is produced. The relevant concept is the implication. The

tutor chooses a clearer referring expression for it, namely “if-

then-relation”. The active function of the hint is to elicit the

inference rule, as will become evident later on in the example.

Here is the realization of the hint:

Tutor (2): (give-away-relevant-concept) Das ist nicht

richtig! Sie m ¨ussen als erstes die wenn-dann-Beziehung

betrachten.

[That is not correct! First you have to consider the if-then-

relation.]

Student (2): (wrong)

F

O

O

SS

With the production of this hint, the counter

W

is incre-

mented by

. However, this hint is not sufﬁcient to help the

student. He gives an answer which is correct in principle, but

does not lead anywhere. Therefore, it is categorized as wrong.

The function socratic is called recursively. Since

W

,

the function up-to-inference-rule is called, which pro-

duces a hint of type

elaborate-domain-object

. This is a sec-

ond attempt to elicit the inference rule. Again, the student’s

knowledge of the necessary concept is checked ﬁrst in the

same function. The tutor addresses the answer and informs

the student of its status.

Tutor (3): (elaborate-domain-object) Das ist zwar

richtig, aber im Augenblick uninteressant. Wissen Sie,

wie sie die wenn-dann-Beziehung behandeln m ¨ussen?

[That is correct, but at the moment not interesting. Do

you know how to deal with the if-then-relation?]

Student (3): (wrong) Nein

[No]

The counter

W

is incremented by

. The new student answer

is also considered as wrong3. Since this is the third wrong

answer and

W

d

, the algorithm now produces an

explain-

meta-reasoning

hint and explains the reasoning behind this

step. The number of hints and wrong answers imply a bad

student level.

Tutor (4): (explain-meta-reasoning) Sie m¨ussen die

Beziehung auﬂ¨

osen, indem Sie die G¨ultigkeit der Vo-

raussetzung annehmen und daraus die G ¨ultigkeit der Be-

hauptung folgern.

[You have to break down the relation by assuming the

validity of the assumption and derive the validity of the

proposition from that.]

Student (4): (unknown)

F

O

S

*F

O

S

The counter

W

is incremented again andit is now

W

. The

student answer cannot be categorized at all. Therefore the

system initiates a subdialog to clarify the problematic point

in the student’s answer.

Tutor (5): Was wollen Sie mit dem Zeichen

ausdr¨ucken?

[What do you want to say with the character

?]

Student (5): (wrong)

F

O

S

daraus folgt

+F

O

S

[

F

O

S

from that follows

+F

O

S

]

The

W

counter is not incremented this time, since there has

been no hint produced. The student’s response to the clariﬁca-

tion subdialog is categorised as wrong. This is now the fourth

wrong answer and

W

. The system produces a

give-away-

performable-step

hint, which really gives the current answer

away4.

Tutor (6): (give-away-performable-step) Das ist

genau das, was Sie beweisen sollen. Wir setzen nun

voraus, daß

F

O

S

gilt. Daraus wollen wir dann

schließen, daß die Behauptung

F

O

S

gilt. Verste-

hen Sie, warum wir annehmen m¨ussen, daß

F

O

S

gilt?

[That is exactly what you are supposed to prove. We

assume now that

F

O

S

holds. From that we want

to conclude, that the proposition

F

O

S

holds. Do

you understand why we need to assume that

F

O

S

applies?]

Student (6): (wrong) Nein

[No]

The counter

W

is again incremented by

. Since

W

, the algorithm switches to the didactic method in order to

avoid frustrating the student. The system explains the meta-

reasoning for the whole performable step, which has already

been given in the previous tutor turn.

Tutor (7): Wir m¨ussen dies annehmen, weil wir die wenn-

dann-Beziehung auﬂ¨

osen wollen. Bei einer solchen

3We are currently investigating the need to differentiate between

“I don’t know” answers, as this one and wrong attempts. However,

these were treated uniformly in this dialog.

4Note that normally the student would have been asked to read

the lesson material again through a

point-to-lesson

and in effect the

tutoring session would start anew after that. However, for the pus-

poses of the experiment and the collection of our corpus, we altered

the algorithm in the way described here.

9

Beziehung muß man stets zeigen, daß aus einem bes-

timmten Ausdruck ein anderer Ausdruck folgt. Deshalb

nehmen wir zuerst die G ¨ultigkeit des ersten Ausdrucks

an, weil wir sonst keine Basis h ¨

atten, von der wir ausge-

hen k¨

onnen. Verstehen Sie, warum wir zuerst die wenn-

dann-Beziehung auﬂ¨

osen wollen?

[We have to assume this, because we want to break

down the if-then-relation. With such a relation one al-

ways has to show that from a certain term another term

follows. Therefore we ﬁrst assume the validity of the ﬁrst

term, as otherwise we would have no base to start from.

Do you understand, why we ﬁrst want to break down the

if-then-relation?]

Student (7): (wrong) Nein

[No]

After having completed the explanation of this proof step,

the system re-visits all hints produced by the socratic strategy

while explaining the step, and makes the student aware of

why they were produced. However, we will not go into such

detail here, since it is not in the focus of this paper. This

constitutes the proof summary that we gave at the beginning

of this example.

So far we looked at the behavior of the algorithm when

student constantly gives wrong ansers. To give the reader a

clearer view, let us now examine what happens if the student

gives a partial answer. Let us assume that, in the ﬁrst turn, the

student gave an incomplete-accurate answer instead of giv-

ing a wrong answer. This example is also from our corpus,

albeit from a different teaching session with another subject

(Subject 22) on the same proof.

Student (1’):

&K

The student here gives a correct answer. However, the con-

tent of the answer cannot easily be inferred from the premise.

That means that it has to be proved before it can be used in

the proof. Since the student has not proved it, the answer

is considered incomplete. That is there are missing parts,

namely the proof of what is stated. Nonetheless, all parts

that are there are accurate. That makes the student’s answer

incomplete-accurate.

Subject 22 took part in the experiment in a different con-

dition, which did not involvehinting. However,wechose this

example for its clarity. In the example that we have been

investigating, the algorithm would call an active pragmatic

hint for this sudent answer category, and more speciﬁcally an

elicit-discrepancy

hint. In the example we have been looking

at a possible realization of such a hint would be:

Tutor (2’): That is correct, but why?

Now, if the student follows the hint he could give a correct

answer, which would look like this:

Student (2’): If there was an element

M

/

then it would

be

M

and

M

^

O

S

, which is a contradiction to the

assumption.

This is the correct answer, which completes the ﬁrst proof

step as it is shown in the formal proof. Have there been more

than one hints so far, and since the student gave a correct

answer, the system would do a recapitulation by revisiting

the hints produced and asking the student to explain why the

particular course of proceeding is followed. If the student

could not answer any of the questions, thenthe system would

give the answer away itself before asking the next question,

addressing the next hint. Since only one hint was produced in

our imaginary dialog, the recapitulation is omitted. Thus the

algorithm proceeds with merely prompting for the next step.

8 Related Work and Discussion

Several other tutoring systems tackle, in one form or the other,

hinting strategies, domain ontologies and student answer cat-

egorization. We will only mention here the ones that we judge

to be most related to our work.

Hinting

Ms. Lindquist [14], a tutoring system for high-school algebra,

has some domain speciﬁc types of questions which are used

for tutoring. Although there is some mention of hints, and

the notion of gradually revealing information by rephrasing

the question is prominent, there is no taxonomy of hints or

any suggestions for dynamically producing them.

An analysis of hints can also be found in the CIRCSIM-

Tutor [15; 16], an intelligent tutoring system for blood circu-

lation. Our work has been largely inspired by the CIRCSIM

project both for the general planning of the hinting process

and for the taxonomy of hints. CIRCSIM-Tutor uses domain

speciﬁc hint tactics that are applied locally, but does not in-

clude a global hinting strategy that models the cognitive rea-

soning behind the choice of hints. We, instead, make use

of the hinting history in a more structured manner. Our algo-

rithm takes into account the kind of hints produced previously

as well as the necessary pedagogical knowledge,and follows

a smooth transition from less to more informative hints. Fur-

thermore, we have deﬁned a structured hint taxonomy with

reﬁned deﬁnition of classes and categories based on the pas-

sive vs. active distinction, which is similar to active-passive

continuum in CIRCSIM. We have distinguished these from

functions, which resemble CIRCSIM tactics, but are again

more detailed and more clearly deﬁned. All this facilitates

the automation of the hint production.

AutoTutor [20]uses curriculum scripts on which the tutor-

ing of computer literacy is based. There is mention of hints

that are used by every script. Although it is not clear exactly

what those hints are, they seem to be static. More emphasis

seems to be put on the pedagogically oriented choice of di-

alog moves, prosodic features and facial expression features,

but not on hints. In contrast, we have presented in this pa-

per a hint taxonomy (cf. Section 4) and a tutoring algorithm

(cf. Section 6) that models the dynamic generation of hints

according to the needs of the student.

AutoTutor also uses a cycle of prompting-hinting-

elaborating. This structure relies on the different role of the

dialog moves involved to capture the fact that the tutor pro-

vides more and more information if the student cannot fol-

low the tutoring well. However, it does not provide hints that

themselves reveal more information as the tutoring process

progresses, which we have modeled in our algorithm for the

Socratic teaching method (cf. Section 6). Thus, the student

is not merely made to articulate the expected answers, as is

the case in AutoTutor, but he is also encouraged to actively

10

produce the content of the answer itself. Furthermore, the

separation of the study material and the tutoring session fa-

cilitates the production of the answers by the student, since

the tutor does not have to present the material and re-elicit it

in one and the same session. The student is guided through

making use of the study material that he has already read in

order to solve the problem.

Ontologies

CIRCSIM-Tutor [17; 9]uses an ontology at three levels: The

knowledge of domain concepts, the computer context of tu-

toring and the meta-language on attacking the problem. The

ontology we present in this paper is not concerned with the

second level. The ﬁrst level corresponds to our existing

knowledge base. The third level can be viewed as a sim-

pliﬁed attempt to model tutoring, which we do via hinting.

They do, however, use their domain ontology in categorizing

the student answer and ﬁxing mistakes.

Within the framework of STEVE [22], Diligent [3]is a tool

for learning domain procedural knowledge. Knowledge is ac-

quired by observing an expert’s performance of a task, as a

ﬁrst step, subsequently conducting self-experimentation, and

ﬁnally by human corrections on what Diligent has taught it-

self. The most relevant part of the knowledge representation

is the representation of procedures in terms of steps in a task,

ordering constraints, causal links and end goals. Although

this is an elaborate learning tool it is not equally elaborate in

its use for interacting with students. It is currently limited to

making use of the procedure representation to learningtext in

order to provide explanations.

Student Answer Categorization

AutoTutor [20]uses a sophisticated latent semantic analy-

sis (LSA) approach. It computes the truth of an input based

on the maximum match between it and the training material

results. The system computes relevance by comparing it to

expected answers. The latter are derived from the curriculum

scripts. AutoTutor further uses the notions of completeness

and compatibility. Both these notions are deﬁned as percent-

ages of the relevant aspect in the expectedanswer. Apart from

the problem of insensitivity of statistical methods to recog-

nizing linguistic phenomena crucial to the evaluation (e.g.,

negation) and the fact that LSA has proven a poor method for

evaluating the kind of small answers representative of our do-

main, we also want a deﬁnition of completeness and accuracy

that gives us insight to the part of the student answer that is

problematic, and hence needs to be addressed by appropriate

hinting. Therefore, deﬁning them in terms of percentages is

not enough. We need to represent the relevant domain entities

that can serve as the basis for the hint production.

CIRCSIM-Tutor and PACT-Geometry tutor both use rea-

soning based on domain knowledge which is similar to the

approach presented in this paper. They compare and evalu-

ate the conceptual content of the student’s response by use

of domain ontologies. CIRCSIM-Tutor [30; 12]uses a ﬁne-

grained classiﬁcation. Unfortunately, the means of classify-

ing the student’s response, that is the rationale of the classiﬁ-

cation, is not documented. That makes comparison difﬁcult.

However, a list of ways of responding after a particular clas-

siﬁcation is available. Still no ﬂexibility is allowed in terms

of a student model.

PACT-Geometry tutor [1]uses a semantic representation

of the student’s input that makes use of a domain ontology in

order to resolve concepts used and their relations. The cor-

rectness of the reasoning used is based on a hierarchy of ex-

planation categories that includes legitimate reasoning steps

in the domain. Although we do not use production rules, our

work is similar to this approach in the use of organized do-

main knowledge to evaluate the student’s answer. It differs

from it to the extent that the Geometry tutor appears not to

have a classiﬁcation, which is essential to us as input to the

hinting algorithm.

9 Conclusion and Future Work

We have presented a domain ontology for the naive set the-

ory in mathematics. We propose to use this ontology in the

general framework of investigating tutorial dialogs for the do-

main. We have deﬁned relations among mathematical con-

cepts, formulae and inference rules and applied this approach

to the domain of naive set theory.

We have further built a hint taxonomy, which was derived

with the aim of automating the hint categories deﬁned in it. It

is based on the needs of the domain, as those were revealed

through the ontology, and previous work done on hints.

Moreover, a student answer categorization which makes

use of the notions of expected answer, completeness and ac-

curacy was derived. The taxonomy of hints in combination

with a student answer categorization is used in a socratic al-

gorithm which models hint production in tutorial dialogs. Our

effort was to create an algorithm which is easily adaptable to

other tutoring domains as well.

We propose to make use of our ontology in mapping de-

scriptions of objects used in the hint categories onto to actual

objects in the domain, which are each time different accord-

ing to the proof under consideration. This facilitates the auto-

matic production of hints, which the socratic algorithm mod-

els, tailored to the needs of a particular student for a particular

attempt with a proof.

The algorithm presented in this paper does not deal with

the realization of the hints. We are currently investigating the

surface structure of hints and their generation. Moreover, the

produced hints do not necessarily complete the tutor’s dialog

turn. Further examination is needed to suggest a model of

dialog moves and dialog speciﬁcations in our domain.

In the future we plan to make further developments in our

domain ontology to improve the evaluation of the student’s

answer, for instance, by capturing different degrees of accu-

racy of the parts in the answer (cf. Section 5). We will also

reﬁne our student answer categories and model more subdi-

algues to better treat the reﬁned categories. We are also re-

searching ways of making full use of our hint taxonomy. All

these improvements will, in turn, help us augment the hinting

algorithm.

To meet these aims, we will use the empirical data from our

recently collected corpus on mathematics tutorial dialogs [4]

The corpus was collected through Wizard-of-Oz experiments

with an expert mathematics tutor. Moreover the same data is

precious for our research in the actual sentence level realiza-

11

tion of hints. We are currently working towards incorporatin

gthe study of these empirical data into the research of auto-

matically generating the hints that we can produce.

References

[1]Vicent Aleven, Ocav Popescu, and Kenneth R.

Koedinger. A tutorial dialogue system with knowledge-

based understanding and classiﬁcation of student expla-

nations. In Working Notes of 2nd IJCAI Workshop on

Knowledge and Reasoning in Practical Dialogue Sys-

tems., Seattle, USA, 2001.

[2]Vincent Aleven and Kenneth Koedinger. The need for

tutorial dialog to support self-explanation. In Ros´e and

Freedman [24], pages 65–73.

[3]Richard Angros, Jr., W. Lewis Johnson, Jeff Rickel, and

Schorel Andrew. Learning domain knowledge for teach-

ing procedural skills. In Proceedings of AAMAS’02,

Bologna, Italy, 2002.

[4]Chris Benzm¨uller, Armin Fiedler, Malte Gabsdil,

Helmut Horacek, Ivana Kruijff-Korbayov´a, Man-

fred Pinkal, J¨org Siekmann, Dimitra Tsovaltzi, Bao

Quoc Vo, and Magdalena Wolska. A Wizard-of-Oz

experiment for tutorial dialogues in mathematics. In

Proceedings of the AIED Workshop on Advanced Tech-

nologies for Mathematics Education, Sidney, Australia,

2003. Submitted.

[5]N.O. Bernsen, H. Dybkjær, and L. Dybkjær. Designing

Interactive Speech Systems — From First Ideas to User

Testing. Springer, 1998.

[6]Jean Carletta, Amy Isard, Stephen Isard, Jacqueline C.

Kowtko, Gwyneth Doherty-Sneddon, and Anne H. An-

derson. The reliability of a dialogue structure coding

scheme. Computational Linguistics, 23(1):13–32, 1997.

[7]Michelene T. H. Chi, Nicholas de Leeuw, Mei-

Hung Chiu, and Christian Lavancher. Eliciting self-

explanation improves understanding. Cognitive Sci-

ence, 18:439–477, 1994.

[8]Mark G. Core and James F. Allen. Coding dialogues

with DAMSL annotation scheme. In AAAI Fall Sym-

posium on Communicative Action in Humans and Ma-

chines, pages 28–35, Boston, MA, 1993.

[9]Martha W. Evens, Stefan Brandle, Ru-Charn Chang,

Reva Freedman, Michael Glass, Yoon Hee Lee,

Leem Seop Shim, Chong Woo Woo, Yuemei Zhang,

Yujian Zhou, Joel A. Michael, and Allen A. Rovick.

CIRCSIM-Tutor: An intelligent tutoring system using

natural language dialogue. In Proceedings of 12th Mid-

west AI and Cognitive Science Conference, MAICS-

2001, pages 16–23, Oxford OH, 2001.

[10]Armin Fiedler and Helmut Horacek. Towards under-

standing the role of hints in tutorial dialogues. In

BI-DIALOG: 5th Workshop on Formal Semantics and

Pragmatics in Dialogue, pages 40–44, Bielefeld, Ger-

many, 2001.

[11]Armin Fiedler and Dimitra Tsovaltzi. Automating hint-

ing in mathematical tutorial dialogue. In Proceedings of

the EACL-03 Workshop on Dialogue Systems: interac-

tion, adaptation and styles of management, pages 45–

52, Budapest, 2003.

[12]Michael Glass. Processing language input in the

circsim-Tutor intelligent tutoring system. In Johanna

Moore, Carol Luckhardt Redﬁeld, and W. Lewis John-

son, editors, Artiﬁcial Intelligence in Education, pages

210–212. IOS Press, 2001.

[13]Neil Heffernan and Kenneth Koedinger. Intelligent tu-

toring systems are missing the tutor: Building a more

strategic dialog-based tutor. In Ros ´e and Freedman [24],

pages 14–19.

[14]Neil T. Heffernan and Kenneth R. Koedinger. Building

a 3rd generation ITS for symbolization: Adding a tuto-

rial model with multiple tutorial strategies. In Proceed-

ings of the ITS 2000 Workshop on Algebra Learning,

Montr´eal, Canada, 2000.

[15]Gregory Hume, Joel Michael, Allen Rovick, and Martha

Evens. Student responses and follow up tutorial tactics

in an ITS. In Proceedings of the 9th Florida Artiﬁcial

Intelligence Research Symposium, pages 168–172, Key

West, FL, 1996.

[16]Gregory D. Hume, Joel A. Michael, Rovick A. Allen,

and Martha W. Evens. Hinting as a tactic in one-on-one

tutoring. Journal of the Learning Sciences, 5(1):23–47,

1996.

[17]Chung Hee Lee, Jai Hyun Seu, and Martha W. Evens.

Building an ontology for CIRCSIM-Tutor. In Proceed-

ings of the 13th Midwest AI and Cognitive Science Con-

ference, MAICS-2002, pages 161–168, Chicago, 2002.

[18]E. Melis, E. Andres, A. Franke, G. Goguadse,

M. Kohlhase, P. Libbrecht, M. Pollet, and C. Ullrich. A

generic and adaptive web-based learning environment.

In Artiﬁcial Intelligence and Education, pages 385–407,

2001.

[19]Johanna Moore. What makes human explanations ef-

fective? In Proceedings of the Fifteenth Annual Confer-

ence of the Cognitive Science Society, pages 131–136.

Hillsdale, NJ. Earlbaum, 2000.

[20]Natalie K. Person, Arthur C. Graesser, Derek Harter,

Eric Mathews, and the Tutoring Research Group. Di-

alog move generation and conversation management in

AutoTutor. In Ros´e and Freedman [24], pages 45–51.

[21]Manfred Pinkal, J¨org Siekmann, and Christoph

Benzm¨uller. Projektantrag Teilprojekt MI3 — DIA-

LOG: Tutorieller Dialog mit einem mathematischen

Assistenzsystem. In Fortsetzungsantrag SFB 378 —

Ressourcenadaptive kognitve Prozesse, Universit¨at des

Saarlandes, Saarbr¨ucken, Germany, 2001.

[22]Jeff Rickel, Rajaram Ganeshan, Charles Rich, Can-

dance L. Sidner, and Neal Lesh. Task-oriented tuto-

rial dialogue: Issues and agents. In Ros´e and Freedman

[24], pages 52–57.

12

[23]Carolyn P. Ros´e, Johanna D. Moore, Kurt VanLehn, and

David Allbritton. A comparative evaluation of socratic

versus didactic tutoring. In Johanna Moore and Keith

Stenning, editors, Proceedings 23rd Annual Conference

of the Cognitive Science Society, University of Edin-

burgh, Scotland, UK, 2001.

[24]Carolyn Penstein Ros´e and Reva Freedman, editors.

Building Dialog Systems for Tutorial Applications—

Papers from the AAAI Fall Symposium, North Falmouth,

MA, 2000. AAAI press.

[25]J¨org Siekmann, Christoph Benzm ¨uller, Vladimir Brezh-

nev, Lassaad Cheikhrouhou, Armin Fiedler, Andreas

Franke, Helmut Horacek, Michael Kohlhase, Andreas

Meier, Erica Melis, Markus Moschner, Immanuel Nor-

mann, Martin Pollet, Volker Sorge, Carsten Ullrich,

Claus-Peter Wirth, and J¨urgen Zimmer. Proof develop-

ment with

MEGA. In Andrei Voronkov, editor, Auto-

mated Deduction — CADE-18, number 2392 in LNAI,

pages 144–149. Springer Verlag, 2002.

[26]Dimitra Tsovaltzi. Formalising hinting in tutorial dia-

logues. Master’s thesis, The University of Edinburgh,

Scotland, UK, 2001.

[27]Dimitra Tsovaltzi and Armin Fiedler. An approach to fa-

cilitating reﬂection in a mathematics tutoring system. In

Proceedings of AIED Workshop on Learner Modelling

for Reﬂection, Sydney, Australia, 2003. Submitted.

[28]Dimitra Tsovaltzi and Armin Fiedler. Enhancement and

use of a mathematical ontology in a tutorial dialogue

system. In IJCAI Workshop on Knowledge and Reason-

ing in Practical Dialogue Systems, Acapulco, Mexico,

2003. In press.

[29]Dimitra Tsovaltzi and Colin Matheson. Formalising

hinting in tutorial dialogues. In EDILOG: 6th workshop

on the semantics and pragmatics of dialogue, pages

185–192, Edinburgh, Scotland, UK, 2002.

[30]Zhou Yujian, Reva Freedman, Michael Glass, Joel A.

Michael, Allen A. Rovick, and Martha W. Evens. What

should the tutor do when the student cannot answer

a question? In AAAI Press, editor, Proceedings of

the Twelfth International Florida AI Research Society

Conference (FLAIRS-99), pages 187–191, Orlando, FL,

1999.

13