Content uploaded by Dimitra Tsovaltzi

Author content

All content in this area was uploaded by Dimitra Tsovaltzi on Feb 04, 2016

Content may be subject to copyright.

How to Support Meta-Cognitive Skills for Finding and Correcting Errors?

Erica Melis

German Research Center for Artiﬁcial Intelligence (DFKI)

Stuhlsatzenhausweg 3 (Building D3 2)

D-66123 Saarbr¨

ucken

melis@dfki.de

Andreas Sander

Universit¨

at des Saarlandes

D-66123 Saarbr¨

ucken

andreas.sander@dfki.de

Dimitra Tsovaltzi

German Research Center for Artiﬁcial Intelligence (DFKI)

Stuhlsatzenhausweg 3 (Building D3 2)

D-66123 Saarbr¨

ucken

Dimitra.Tsovaltzi@dfki.de

Abstract

Meta-cognitive skills to be developed in learning for the 21st

century is the detection and correction of errors in solu-

tions. These meta-cognitive skills can help to detect errors

the learner has made her/himself as well as errors others have

made.

Our investigations in learning from errors have the ultimate

goal to adapt the selection and presentation to the learner so

that he/she can better learn from erroneous examples others

have made. In our experiments we found that (1) erroneous

examples with help provision can promote students skill of

ﬁnd errors, (2) the beneﬁt from erroneous examples depends

on the relation between the student’s level and the example’s

difﬁculty, i.e. if the student is prepared for the problem, (3)

for many students it is very difﬁcult to correct errors.

Introduction

Some of the meta-cognitive skills to be developed in learn-

ing for the 21st century are the detection and correction of

errors in problem solutions. These meta-cognitive skills can

help to detect errors the learner has made her/himself as well

as errors others have made, e.g., errors hidden in solutions

on the web or in other instructions.

However, confronting students with erroneous solutions

of mathematical problems and asking them to ﬁnd the error

is rather unusual in most schools (except Japanese schools)

and many teachers are ambivalent about – or even hostile to

– discussing errors in the classroom (Tsamir & Tirosh 2003).

Our ultimate goal is to use adaptive technology to help

students to learn by dealing with errors. That is, help stu-

dents learn mathematics by presenting erroneous examples

together with exercises in an adaptive fashion within the

TEL system ACTIVEMATH, a web-based learning environ-

ment for mathematics. We hope to improve students’ cog-

nitive competencies in math learning, as well as their meta-

cognitive competencies in error discovery, error correction

and self-monitoring.

Erroneous examples are worked solutions that include one

or more errors that the student is asked to detect and/or cor-

rect. Technology opens new possibilities of instruction with

erroneous examples. For example, erroneous examples can

Copyright c

2010, Association for the Advancement of Artiﬁcial

Intelligence (www.aaai.org). All rights reserved.

be presented in a variety of ways, with different kinds of

feedback, diverse tutorial strategies and diverse sequencing

of learning material. Adapting such features to the needs of

individual students could contribute to better learning.

We conjecture that erroneous examples will give students

the opportunity to ﬁnd and reﬂect upon errors in a way that

will lead to deeper learning, while at the same time not

causing students to feel ashamed or demotivated, as is more

likely when their own errors are exposed.

During the last years we conducted a number of experi-

ments, some of them strictly controlled lab and classroom

studies, other more preliminary less controlled classroom

experiments, and a Wizard-of-Oz experiment.

Finding Errors in Erroneous Examples

Our experiments (Melis & Kriesell 2009; Tsovaltzi et al.

2009; 2010) tested whether erroneous examples in the do-

main of fractions can help students learn in the context

of Technology-Enhanced Learning (TEL). This could spare

students the embarrassment and demotivation of confronting

their own errors. We conducted lab studies and classroom

studies with students of different curriculum levels to mea-

sure the effects of learning through erroneous examples. We

summarize the results of these studies that compare learning

gains in three experimental conditions: a control condition, a

condition in which students were presented with erroneous

examples without help, and a condition in which students

were provided with help for error detection and correction.

Empirical Results

The erroneous examples’ beneﬁt depends on the relation be-

tween the student’s level and the example’s difﬁculty, i.e.

the student’s readiness statement. and needs, (similar to the

results of investigations dealing with more vs. less interac-

tivity in (vanLehn et al. 2007)).

We hypothesize that the effect of erroneous examples de-

pends on when and how they are introduced to the students

and whether students are supported in ﬁnding and correcting

the error.

Design

Erroneous examples in ACTIVEMATH include instances of

typical errors students make, which address standard prob-

lems students face with rule-application, or errors which tar-

get common misconceptions and address more fundamental

conceptual understanding infractions. Erroneous examples

consist of two phases: error detection and error correction.

Based on pilot studies, we designed feedback for help-

ing students understand and correct the errors. There are

four types of unsolicited feedback: minimal feedback (ﬂag

feedback), error-awareness and detection (EAD) feedback,

self-explanation feedback (realized as MCQs) and error-

correction scaffolds.

We conducted lab studies with 6th, 7th and 8th-graders

and school studies with 9th and 10th-graders. The partici-

pants came from both, urban and suburban, German schools.

All studies used three experimental conditions. The con-

dition No-Erroneous-Examples (NOEE) was the control and

included standard fraction exercises with minimal feedback

and the correct solution, but no erroneous examples in the in-

tervention. The condition Erroneous-Examples-With-Help

(EEWH) included standard exercises, and erroneous exam-

ples with provision of help (EAD, error detection/correction

MCQs, and error-speciﬁc help). The condition Erroneous-

Examples-Without-Help (EEWOH) included standard ex-

ercises, and erroneous examples but without additional

help. The design included a pre-questionnaire, a pretest,

a familiarization, an intervention, a posttest and a post-

questionnaire. The pre- and posttests were counterbalanced.

Results

In the studies with different levels of students we had results

that showed differences in how erroneous examples with

error detection and -correction help inﬂuence mathematics

learning. We found that, at least, more advanced students

(9th- and 10th-grade) beneﬁt from erroneous examples with

help in terms of cognitive skills in general, as opposed to

erroneous examples without help, or no use of erroneous ex-

amples at all. We also found that erroneous examples as a

whole and error detection and -correction help can promote

conceptual understanding for the same students.

Erroneous examples can also inﬂuence the metacogni-

tive skill of error detection for less advanced (6th-grade) but

highly competent students.

We did not ﬁnd evidence for the use of erroneous ex-

amples for medium-advanced students (7th- and 8th-grade).

However, this can be a result of the combined high level and

high competence which our participants had in comparison

to the students of the 9th and 10th-grade.

In those experiments we found that most students

do not have the meta-cognitive skill of correcting

errors. Why is this the case? Therefore, we

wanted to employ the Wizard-of-Oz (WOz) technique

(cf. http://en.wikipedia.org/wiki/Wizard of Oz experiment)

in order to ﬁnd out more about reasons of correction weak-

nesses.

Correcting Errors in Erroneous Examples

In a WOz experiment we addressed the question, why many

students – even those who can ﬁnd the errors or can solve

a (fraction) problem on their own – have great difﬁculties

when asked to correct the errors in erroneous examples. We

wanted to ﬁnd reasons for these difﬁculties.

Experimental Set-Up

The experiment took place at a secondary school in

Heusweiler, Germany. 35 7th grade students (8 male and 27

female students) participated in this experiment and for each

students the experiment took 45 minutes. Before the main

experiment started, each subject received a short introduc-

tion about the user interface. During all experiments a total

of 102 instances of erroneous examples were processed and

for 65 instances, students detected the error at ﬁrst request.

We used a WOz environment developed in our group in

which the student and the Wizard can communicate. The

Wizard (the second author) selected one to four predeﬁned

erroneous examples. The selection criteria included: (1) not

too difﬁcult to ﬁnd the error, (2) errors are close to errors or

misconceptions the student made in the pretest. Always, the

ﬁrst question to the subject was whether the computations in

the exercise are correct.

The Wizard’s plan was to ﬁrst, ask the student why the

result cannot be correct, then the subject should show where

the error actually is and tell what the error is and why it is

an error. Then the student was supposed to correct the error

and simultaneously write down what was in her/his mind. In

cases where subjects had problems with this method during

problem solving, a retrospective protocol after the problem

solving process was used. Both the thinking aloud method

and retrospective protocol did not yet provide much input.

Results

At ﬁrst, many students claimed to see a correct solution and

did not ﬁnd the error at ﬁrst try. This was the case for 47

out of the 102 instances. In 27 out of 65 cases the partici-

pants could not correct the error properly at all. That is, in

more than 40% of the cases, in which the students spotted

the errors, they could not correct them .

In the following, we present two typical interaction ex-

cerpts which were based on the erroneous example contain-

ing a wrong computation. The translation of the task in En-

glish is

The rain barrel: Ulli collects rain water in a rain barrel in

his garden and waters the plants in his garden. After several

days of rain the 320 liter barrel was ﬁlled. The day after the

rains Ulli asks: yesterday I had 320 liter in the barrel. Then I

used 5 1

2liter for my plants. How much is still in the barrel?

and he computes...

320 −51

2

= 320 −

11

2

=320 −11

2

=309

2

... did he compute correctly? Please answer this question

with ”YES” or ”NO”

Log-Excerpt 1

Original Log-Excerpt of a participant (behind the

German you ﬁnd the English translation)

student:Ja En: yes

WOz:Nein, es wurde nicht richtig gerech-

net/argumentiert. En: no, the calcula-

tion/argumentation is incorrect

WOz:Warum kann das errechnete Ergebnis nicht

stimmen? En: why can the result not be correct?

student:Emm weil er die (320-11)/2 nicht soo rechnen

darf. En: emm, because you cannot compute like this

WOz:Genau, warum darf er das nicht? En: correct,

why must he not?

student:Weil er nicht die 320 auf die 11/2 mit drauf

setzten darf. En: because he must not put the 320 on

top of 11/2

WOz:Was muss er machen, damit er das auf einen

Bruchstrich schreiben darf? En: what should he do

before writing this in the nominator?

student:Er muss die Zahl 320 als Bruch umwandeln

entweder 320/1 oder soo un dann den Hauptnenner

rechnen. En: he needs to extend the number 320 to

the fraction 320/1 or so and then compute the common

denominator

WOz:Richtig! Du hast den Fehler gefunden! En:

Correct, you’ve found the error

WOz:Korrigiere nun (in deinem Eingabefenster) den

ersten fehlerhaften Schritt und f¨

uhre die Berechnungen

zu Ende. [...] En: now correct the ﬁrst incorrect step

and complete the computation

student:keine ahnung En: no idea

WOz:Wie w¨

urdest du vorgehen, um den fehlerhaften

Schritt zu korrigieren? Beschreibe in Worten. En: what

would you do to correct the wrong step? describe in

your own words.

student:Also ich w¨

urde die 320/1 mit der 11/2

zusammen auf den Hauptnenner bringen dann das

minus rechnen. En: well, I’d compute the common

denominator of 320/1 and 11/2 and then substract

WOz:Richtig, versuche dies nun rechnerisch umzuset-

zen! En: correct, now try to compute this

WOz:Korrigiere nun (in deinem Eingabefenster) den

ersten fehlerhaften Schritt und f¨

uhre die Berechnungen

zu Ende. [...] Please correct (in the input window) only

the ﬁrst incorrect step

student: wie bringt man es wieder auf den Hauptnen-

ner? En: how is the common denominator computed?

In this interaction the participant ﬁrst answered, that all

computations in the exercise are correct, but then (after a

short hint) the student found the error. Nevertheless, the stu-

dent could not correct the mistake (compute the common de-

nominator and adapt the two fractions), because the student

did not know how the procedure works practically.

This is a good example for the dissociation between ex-

ecution (action) and monitoring (evaluation) or, the dis-

tinction between knowing that and knowing how (Ohlsson

1996), which we encountered in our earlier studies (Tso-

valtzi et al. 2010). The student knew that the fractions have

to have the same denominator before subtraction of two frac-

tions is possible. However, the student did not know how

to calculate a common denominator for the fractions. This

makes a big difference.

If a student knows what, he/she often only remembers the

(name of the) algorithm that needs to be applied. This does

not mean he/she knows exactly what to do/the single steps.

That is, the student may not even remember the deﬁnition

(rather than a vague notion) of a common denominator.

Log-Excerpt 2

Original Log-Excerpt of another participant

student:Ja En: yes

WOz: Nein, es wurde nicht richtig gerech-

net/argumentiert. En: no the solution is incorrect

student: lies dir die Aufgabe noch mal genau durch.

En: please read the problem again

Woz:Nein En: no

WOz:Warum kann das errechnete Ergebnis nicht

stimmen? En: why can the result not be correct?

WOz:Hast du Fragen? En: do you have questions?

student:nein En: no

WOz:Warum kann das errechnete Ergebnis nicht

stimmen? En: why can the result not be correct?

student:weil ulli die 320 liter auch in einen bruch

umrechnen muss En: because ulli needs to transform

the 320 liter into a fraction

WOz:Wo ist der Fehler? [...] En: where is the error?

student:Schritt 3;weil ulli die 320 liter auch in einen

bruch umrechnen muss En: step 3; because ulli needs

to transform the 320 liter into a fraction

WOz:Richtig! Du hast den Fehler gefunden! En:

correct! you found the mistake

WOz:Was genau hat er falsch gemacht? En: what

exactly did he make wrong?

student:Schritt 3;er hat die 320 liter nicht in einen

bruch umgerechnet. En: step3; he didn’t transform 320

liter to a fraction

WOz:Wie w¨

urdest du vorgehen, um den fehlerhaften

Schritt zu korrigieren? En: How would you proceed in

order to correct the incorrect step?

student:Schritt 3;ich w¨

urde aus den 320 liter 160/2

machen und dann die 11/2 davon abziehen En: step3;

I’d transform 320 liter to 160/2 and then substract 11/2

WOz: Wie w¨

urdest du vorgehen. Erkl¨

are mit deinen

eigenen Worten. Ohne genaue Rechnungen. En: how

would you proceed? explain in your own words.

without exact computations.

WOz:Hast du Fragen? En: do you have questions?

student:Schritt 3; ich w¨

urde die 320 liter nehmen und

in 160/2 liter umwandeln, dann die 5 1/2 liter in 11/2

liter um wandeln und dann die 11/2 liter von den 160/2

liter abziehen so w¨

are das ergebnis: 149/2 liter En:

step 3; I’d take the 320 liter and transform it to 160/2

liter, then transform the 5 1/2 liter into 11/2 liter, and

then substract 11/2 liter from 160/2 liter with the result

149/2 liter.

In this interaction the student found the mistake (subtrac-

tion of two fractions with different denominators without

ﬁrst computing the common denominator) and had an idea

how to correct the error (expansion of 320 to a fraction with

the denominator two), which means he/she was a step ahead

but still could not complete the computation of the common

denominator, i.e., did not fully understand the principle of

expanding fractions to equal denominators..

Would understanding the principle conceptually allow

students to perform the necessary actions automatically?

What exactly it is that prevents them from correcting the er-

ror once they recognize it?

Related Work

Erroneous examples in mathematics have rarely been in-

vestigated or used as a learning intervention, either within

a technology-enhanced learning (TEL) system or within

the classroom. Overall, the related previous studies did

not target adaptation by the technology. The empirical re-

sults (sometimes inconclusive) evidence that studying errors

can promote student learning (Borasi 1994; Siegler 2002;

Curry 2004; Grosse & Renkl 2004; Monthienvichienchai &

Melis 2005; Grosse & Renkl 2007; Siegler & Chen 2008).

The skill of correcting errors was not addressed before.

A controlled comparison of the study of correct and incor-

rect examples was done by Siegler and colleagues (Siegler

2002; Siegler & Chen 2008). They investigated whether

self-explaining both correct and incorrect examples is more

beneﬁcial than self-explaining correct examples only. They

found that when students studied and self-explained both

correct and incorrect examples they learned better. They

hypothesized that self-explanation of correct and erroneous

examples strengthened correct strategies and weakened in-

correct problem solving strategies, respectively. Grosse and

Renkl studied whether explaining both correct and incorrect

examples makes a difference to learning and whether high-

lighting errors helps students learn from those errors. Their

empirical studies in which no help was provided showed

some learning beneﬁt of erroneous examples, but unlike the

less ambiguous Siegler et al results, the beneﬁt they uncov-

ered was only for learners with strong prior knowledge and

for far transfer learning. Research in other domains, such as

medical education, has shown beneﬁts of erroneous exam-

ples in combination with elaborate feedback in the acquisi-

tion of problem-solving schemata.

More relevant to the work presented in this paper is Ohls-

son’s theory (Ohlsson 1996) argues that errors occur due to

overgeneralization (applying rules to situations where they

do not apply). Error-detection requires declarative knowl-

edge (e.g.knowledge of rules) in the domain. The error

causes a conﬂict between this knowledge which the learner

believes to be true and what the learner perceives as the

current situation. The way to avoid errors is by special-

izing knowledge structures to trigger only in the situations

where they apply. Ohlsson distinguished between declar-

ative knowledge (knowing that) and practical knowledge

(knowing how), which he deﬁnes as the knowledge required

to choose the appropriate action in a perceive-decide-act cy-

cle.

The contribution of our work are empirical studies that

explicitly tackle this dissociation, in particular, in the do-

main of fractions. This in turn can inform the design of

erroneous examples and intruction not only in the context

of technology-enhanced learning, which we focus on, but in

general.

Conclusion

Our goal is a cognitive tool that supports learning from er-

ror, in particular from erroneous examples and that increases

monitoring alert. As the results show, the tool needs to

adapt to the student’s mastery level for sequencing and meta-

cognitive help provision.

This article describes results on our way to support stu-

dents in developing the meta-cognitive skills of error detec-

tion and error correction in ACTIVEMATH.

Acknowledgement This work was supported by Deutsche

Forschungsgemeinschaft in the project ME 1136/7-1.

References

Borasi, R. 1994. Capitalizing on errors as ”springboards for

inquiry”: A teaching experiment. Journal for Research in

Mathematics Education 25(2):166–208.

Curry, L. 2004. The effects of self-explanations of cor-

rect and incorrect solutions on algebra problem-solving per-

formance. In D. Gentner, K. F., and Regier, T., eds., 26th

Annual Conference of the Cognitive Science Society, 1548.

Mahwah, NJ: Erlbaum.

Grosse, C., and Renkl, A. 2004. Learning from worked ex-

amples: What happens if errors are included? In Gerjets, P.;

Elen, J.; Joiner, R.; and Kirschner, P., eds., Instructional de-

sign for effective and enjoyable computer-supported learn-

ing. Tuebingen: Knowledge Media Research Center. 356–

364.

Grosse, C., and Renkl, A. 2007. Finding and ﬁxing er-

rors in worked examples: Can this foster learning outcomes?

Learning and Instruction 17:612–634.

Melis, E., and Kriesell, D. 2009. Effects of erroneous exam-

ples, results of a preliminary school experiment. In V. Dim-

itrova, R. Mizoguchi, B. d. B., and Graesser, A., eds., AIED

2009, volume 200 of Frontiers in Artiﬁcial Intelligence and

Applications, 578–580. IOS press.

Monthienvichienchai, R., and Melis, E. 2005. Implementing

courseware to support learning through real-world erroneous

examples: Student’s perceptions of tertiary courseware and

obstacles to implementing effective delivery through vle. In

Remenyi, D., ed., Proceedings of European Conference on

eLearning (ECEL-2005), 281–289. Academic Conferences

International.

Ohlsson, S. 1996. Learning from performance errors. Psy-

chological Review 103:241–262.

Siegler, R., and Chen, Z. 2008. Differentiation and inte-

gration: Guiding principles for analyzing cognitive change.

Developmental Science 11:433–448.

Siegler, R. 2002. Microgenetic studies of self-explanation.

In Granott, N., and Parziale, J., eds., Microdevelopment,

Transition Processes in Development and Learning. Cam-

bridge University Press. 31–58.

Tsamir, P., and Tirosh, D. 2003. In-service mathematics

teachers’ views or errors in the classroom. In International

Symposium : Elementary Mathematics Teaching.

Tsovaltzi, D.; Melis, E.; McLaren, B.; Dietrich, M.;

Goguadze, G.; and Meyer, A.-K. 2009. Erroneous exam-

ples: A preliminary investigation into learning beneﬁts. In

Ulrike Cress, Vania Dimitrova, M. S., ed., European Con-

ference for Technology-Enhanced Learning, EC-TEL 2009,

LNCS 5794, 688–693. Springer-Verlag.

Tsovaltzi, D.; Melis, E.; McLaren, B.; Meyer, K.; Dietrich,

M.; and Goguadze, G. 2010. Learning from erroneous ex-

amples: When and how do students beneﬁt from them? In ,

ed., EC-TEL 2010, volume - of LNCS, –. Springer-Verlag.

vanLehn, K.; Graesser, A.; Jackson, G.; Jordan, P.; and

C.P.Rose. 2007. When are tutorial dialogues more effec-

tive than reading? Cognitive Science 31:3–62.