How to support meta-cognitive skills for finding and correcting errors?

Abstract

Meta-cognitive skills to be developed in learning for the 21st century is the detection and correction of errors in solu-tions. These meta-cognitive skills can help to detect errors the learner has made her/himself as well as errors others have made. Our investigations in learning from errors have the ultimate goal to adapt the selection and presentation to the learner so that he/she can better learn from erroneous examples others have made. In our experiments we found that (1) erroneous examples with help provision can promote students skill of find errors, (2) the benefit from erroneous examples depends on the relation between the student's level and the example's difficulty, i.e. if the student is prepared for the problem, (3) for many students it is very difficult to correct errors. Copyright ©2010, Association for the Advancement of Artificial Intelligence. All rights reserved.

Full text

How to Support Meta-Cognitive Skills for Finding and Correcting Errors?
Erica Melis
German Research Center for Artificial Intelligence (DFKI)
Stuhlsatzenhausweg 3 (Building D3 2)
D-66123 Saarbr¨
ucken
melis@dfki.de
Andreas Sander
Universit¨
at des Saarlandes
D-66123 Saarbr¨
ucken
andreas.sander@dfki.de
Dimitra Tsovaltzi
German Research Center for Artificial Intelligence (DFKI)
Stuhlsatzenhausweg 3 (Building D3 2)
D-66123 Saarbr¨
ucken
Dimitra.Tsovaltzi@dfki.de
Abstract
Meta-cognitive skills to be developed in learning for the 21st
century is the detection and correction of errors in solu-
tions. These meta-cognitive skills can help to detect errors
the learner has made her/himself as well as errors others have
made.
Our investigations in learning from errors have the ultimate
goal to adapt the selection and presentation to the learner so
that he/she can better learn from erroneous examples others
have made. In our experiments we found that (1) erroneous
examples with help provision can promote students skill of
find errors, (2) the benefit from erroneous examples depends
on the relation between the student’s level and the example’s
difficulty, i.e. if the student is prepared for the problem, (3)
for many students it is very difficult to correct errors.
Introduction
Some of the meta-cognitive skills to be developed in learn-
ing for the 21st century are the detection and correction of
errors in problem solutions. These meta-cognitive skills can
help to detect errors the learner has made her/himself as well
as errors others have made, e.g., errors hidden in solutions
on the web or in other instructions.
However, confronting students with erroneous solutions
of mathematical problems and asking them to find the error
is rather unusual in most schools (except Japanese schools)
and many teachers are ambivalent about – or even hostile to
– discussing errors in the classroom (Tsamir & Tirosh 2003).
Our ultimate goal is to use adaptive technology to help
students to learn by dealing with errors. That is, help stu-
dents learn mathematics by presenting erroneous examples
together with exercises in an adaptive fashion within the
TEL system ACTIVEMATH, a web-based learning environ-
ment for mathematics. We hope to improve students’ cog-
nitive competencies in math learning, as well as their meta-
cognitive competencies in error discovery, error correction
and self-monitoring.
Erroneous examples are worked solutions that include one
or more errors that the student is asked to detect and/or cor-
rect. Technology opens new possibilities of instruction with
erroneous examples. For example, erroneous examples can
Copyright c
2010, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
be presented in a variety of ways, with different kinds of
feedback, diverse tutorial strategies and diverse sequencing
of learning material. Adapting such features to the needs of
individual students could contribute to better learning.
We conjecture that erroneous examples will give students
the opportunity to find and reflect upon errors in a way that
will lead to deeper learning, while at the same time not
causing students to feel ashamed or demotivated, as is more
likely when their own errors are exposed.
During the last years we conducted a number of experi-
ments, some of them strictly controlled lab and classroom
studies, other more preliminary less controlled classroom
experiments, and a Wizard-of-Oz experiment.
Finding Errors in Erroneous Examples
Our experiments (Melis & Kriesell 2009; Tsovaltzi et al.
2009; 2010) tested whether erroneous examples in the do-
main of fractions can help students learn in the context
of Technology-Enhanced Learning (TEL). This could spare
students the embarrassment and demotivation of confronting
their own errors. We conducted lab studies and classroom
studies with students of different curriculum levels to mea-
sure the effects of learning through erroneous examples. We
summarize the results of these studies that compare learning
gains in three experimental conditions: a control condition, a
condition in which students were presented with erroneous
examples without help, and a condition in which students
were provided with help for error detection and correction.
Empirical Results
The erroneous examples’ benefit depends on the relation be-
tween the student’s level and the example’s difficulty, i.e.
the student’s readiness statement. and needs, (similar to the
results of investigations dealing with more vs. less interac-
tivity in (vanLehn et al. 2007)).
We hypothesize that the effect of erroneous examples de-
pends on when and how they are introduced to the students
and whether students are supported in finding and correcting
the error.
Design
Erroneous examples in ACTIVEMATH include instances of
typical errors students make, which address standard prob-

lems students face with rule-application, or errors which tar-
get common misconceptions and address more fundamental
conceptual understanding infractions. Erroneous examples
consist of two phases: error detection and error correction.
Based on pilot studies, we designed feedback for help-
ing students understand and correct the errors. There are
four types of unsolicited feedback: minimal feedback (flag
feedback), error-awareness and detection (EAD) feedback,
self-explanation feedback (realized as MCQs) and error-
correction scaffolds.
We conducted lab studies with 6th, 7th and 8th-graders
and school studies with 9th and 10th-graders. The partici-
pants came from both, urban and suburban, German schools.
All studies used three experimental conditions. The con-
dition No-Erroneous-Examples (NOEE) was the control and
included standard fraction exercises with minimal feedback
and the correct solution, but no erroneous examples in the in-
tervention. The condition Erroneous-Examples-With-Help
(EEWH) included standard exercises, and erroneous exam-
ples with provision of help (EAD, error detection/correction
MCQs, and error-specific help). The condition Erroneous-
Examples-Without-Help (EEWOH) included standard ex-
ercises, and erroneous examples but without additional
help. The design included a pre-questionnaire, a pretest,
a familiarization, an intervention, a posttest and a post-
questionnaire. The pre- and posttests were counterbalanced.
Results
In the studies with different levels of students we had results
that showed differences in how erroneous examples with
error detection and -correction help influence mathematics
learning. We found that, at least, more advanced students
(9th- and 10th-grade) benefit from erroneous examples with
help in terms of cognitive skills in general, as opposed to
erroneous examples without help, or no use of erroneous ex-
amples at all. We also found that erroneous examples as a
whole and error detection and -correction help can promote
conceptual understanding for the same students.
Erroneous examples can also influence the metacogni-
tive skill of error detection for less advanced (6th-grade) but
highly competent students.
We did not find evidence for the use of erroneous ex-
amples for medium-advanced students (7th- and 8th-grade).
However, this can be a result of the combined high level and
high competence which our participants had in comparison
to the students of the 9th and 10th-grade.
In those experiments we found that most students
do not have the meta-cognitive skill of correcting
errors. Why is this the case? Therefore, we
wanted to employ the Wizard-of-Oz (WOz) technique
(cf. http://en.wikipedia.org/wiki/Wizard of Oz experiment)
in order to find out more about reasons of correction weak-
nesses.
Correcting Errors in Erroneous Examples
In a WOz experiment we addressed the question, why many
students – even those who can find the errors or can solve
a (fraction) problem on their own – have great difficulties
when asked to correct the errors in erroneous examples. We
wanted to find reasons for these difficulties.
Experimental Set-Up
The experiment took place at a secondary school in
Heusweiler, Germany. 35 7th grade students (8 male and 27
female students) participated in this experiment and for each
students the experiment took 45 minutes. Before the main
experiment started, each subject received a short introduc-
tion about the user interface. During all experiments a total
of 102 instances of erroneous examples were processed and
for 65 instances, students detected the error at first request.
We used a WOz environment developed in our group in
which the student and the Wizard can communicate. The
Wizard (the second author) selected one to four predefined
erroneous examples. The selection criteria included: (1) not
too difficult to find the error, (2) errors are close to errors or
misconceptions the student made in the pretest. Always, the
first question to the subject was whether the computations in
the exercise are correct.
The Wizard’s plan was to first, ask the student why the
result cannot be correct, then the subject should show where
the error actually is and tell what the error is and why it is
an error. Then the student was supposed to correct the error
and simultaneously write down what was in her/his mind. In
cases where subjects had problems with this method during
problem solving, a retrospective protocol after the problem
solving process was used. Both the thinking aloud method
and retrospective protocol did not yet provide much input.
Results
At first, many students claimed to see a correct solution and
did not find the error at first try. This was the case for 47
out of the 102 instances. In 27 out of 65 cases the partici-
pants could not correct the error properly at all. That is, in
more than 40% of the cases, in which the students spotted
the errors, they could not correct them .
In the following, we present two typical interaction ex-
cerpts which were based on the erroneous example contain-
ing a wrong computation. The translation of the task in En-
glish is
The rain barrel: Ulli collects rain water in a rain barrel in
his garden and waters the plants in his garden. After several
days of rain the 320 liter barrel was filled. The day after the
rains Ulli asks: yesterday I had 320 liter in the barrel. Then I
used 5 1
2liter for my plants. How much is still in the barrel?
and he computes...
320 −51
2
= 320 −
11
2
=320 −11
2
=309
2
... did he compute correctly? Please answer this question
with ”YES” or ”NO”

Log-Excerpt 1
Original Log-Excerpt of a participant (behind the
German you find the English translation)
student:Ja En: yes
WOz:Nein, es wurde nicht richtig gerech-
net/argumentiert. En: no, the calcula-
tion/argumentation is incorrect
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
student:Emm weil er die (320-11)/2 nicht soo rechnen
darf. En: emm, because you cannot compute like this
WOz:Genau, warum darf er das nicht? En: correct,
why must he not?
student:Weil er nicht die 320 auf die 11/2 mit drauf
setzten darf. En: because he must not put the 320 on
top of 11/2
WOz:Was muss er machen, damit er das auf einen
Bruchstrich schreiben darf? En: what should he do
before writing this in the nominator?
student:Er muss die Zahl 320 als Bruch umwandeln
entweder 320/1 oder soo un dann den Hauptnenner
rechnen. En: he needs to extend the number 320 to
the fraction 320/1 or so and then compute the common
denominator
WOz:Richtig! Du hast den Fehler gefunden! En:
Correct, you’ve found the error
WOz:Korrigiere nun (in deinem Eingabefenster) den
ersten fehlerhaften Schritt und f¨
uhre die Berechnungen
zu Ende. [...] En: now correct the first incorrect step
and complete the computation
student:keine ahnung En: no idea
WOz:Wie w¨
urdest du vorgehen, um den fehlerhaften
Schritt zu korrigieren? Beschreibe in Worten. En: what
would you do to correct the wrong step? describe in
your own words.
student:Also ich w¨
urde die 320/1 mit der 11/2
zusammen auf den Hauptnenner bringen dann das
minus rechnen. En: well, I’d compute the common
denominator of 320/1 and 11/2 and then substract
WOz:Richtig, versuche dies nun rechnerisch umzuset-
zen! En: correct, now try to compute this
WOz:Korrigiere nun (in deinem Eingabefenster) den
ersten fehlerhaften Schritt und f¨
uhre die Berechnungen
zu Ende. [...] Please correct (in the input window) only
the first incorrect step
student: wie bringt man es wieder auf den Hauptnen-
ner? En: how is the common denominator computed?
In this interaction the participant first answered, that all
computations in the exercise are correct, but then (after a
short hint) the student found the error. Nevertheless, the stu-
dent could not correct the mistake (compute the common de-
nominator and adapt the two fractions), because the student
did not know how the procedure works practically.
This is a good example for the dissociation between ex-
ecution (action) and monitoring (evaluation) or, the dis-
tinction between knowing that and knowing how (Ohlsson
1996), which we encountered in our earlier studies (Tso-
valtzi et al. 2010). The student knew that the fractions have
to have the same denominator before subtraction of two frac-
tions is possible. However, the student did not know how
to calculate a common denominator for the fractions. This
makes a big difference.
If a student knows what, he/she often only remembers the
(name of the) algorithm that needs to be applied. This does
not mean he/she knows exactly what to do/the single steps.
That is, the student may not even remember the definition
(rather than a vague notion) of a common denominator.
Log-Excerpt 2
Original Log-Excerpt of another participant
student:Ja En: yes
WOz: Nein, es wurde nicht richtig gerech-
net/argumentiert. En: no the solution is incorrect
student: lies dir die Aufgabe noch mal genau durch.
En: please read the problem again
Woz:Nein En: no
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
WOz:Hast du Fragen? En: do you have questions?
student:nein En: no
WOz:Warum kann das errechnete Ergebnis nicht
stimmen? En: why can the result not be correct?
student:weil ulli die 320 liter auch in einen bruch
umrechnen muss En: because ulli needs to transform
the 320 liter into a fraction
WOz:Wo ist der Fehler? [...] En: where is the error?
student:Schritt 3;weil ulli die 320 liter auch in einen
bruch umrechnen muss En: step 3; because ulli needs
to transform the 320 liter into a fraction
WOz:Richtig! Du hast den Fehler gefunden! En:
correct! you found the mistake
WOz:Was genau hat er falsch gemacht? En: what
exactly did he make wrong?
student:Schritt 3;er hat die 320 liter nicht in einen
bruch umgerechnet. En: step3; he didn’t transform 320
liter to a fraction

WOz:Wie w¨
urdest du vorgehen, um den fehlerhaften
Schritt zu korrigieren? En: How would you proceed in
order to correct the incorrect step?
student:Schritt 3;ich w¨
urde aus den 320 liter 160/2
machen und dann die 11/2 davon abziehen En: step3;
I’d transform 320 liter to 160/2 and then substract 11/2
WOz: Wie w¨
urdest du vorgehen. Erkl¨
are mit deinen
eigenen Worten. Ohne genaue Rechnungen. En: how
would you proceed? explain in your own words.
without exact computations.
WOz:Hast du Fragen? En: do you have questions?
student:Schritt 3; ich w¨
urde die 320 liter nehmen und
in 160/2 liter umwandeln, dann die 5 1/2 liter in 11/2
liter um wandeln und dann die 11/2 liter von den 160/2
liter abziehen so w¨
are das ergebnis: 149/2 liter En:
step 3; I’d take the 320 liter and transform it to 160/2
liter, then transform the 5 1/2 liter into 11/2 liter, and
then substract 11/2 liter from 160/2 liter with the result
149/2 liter.
In this interaction the student found the mistake (subtrac-
tion of two fractions with different denominators without
first computing the common denominator) and had an idea
how to correct the error (expansion of 320 to a fraction with
the denominator two), which means he/she was a step ahead
but still could not complete the computation of the common
denominator, i.e., did not fully understand the principle of
expanding fractions to equal denominators..
Would understanding the principle conceptually allow
students to perform the necessary actions automatically?
What exactly it is that prevents them from correcting the er-
ror once they recognize it?
Related Work
Erroneous examples in mathematics have rarely been in-
vestigated or used as a learning intervention, either within
a technology-enhanced learning (TEL) system or within
the classroom. Overall, the related previous studies did
not target adaptation by the technology. The empirical re-
sults (sometimes inconclusive) evidence that studying errors
can promote student learning (Borasi 1994; Siegler 2002;
Curry 2004; Grosse & Renkl 2004; Monthienvichienchai &
Melis 2005; Grosse & Renkl 2007; Siegler & Chen 2008).
The skill of correcting errors was not addressed before.
A controlled comparison of the study of correct and incor-
rect examples was done by Siegler and colleagues (Siegler
2002; Siegler & Chen 2008). They investigated whether
self-explaining both correct and incorrect examples is more
beneficial than self-explaining correct examples only. They
found that when students studied and self-explained both
correct and incorrect examples they learned better. They
hypothesized that self-explanation of correct and erroneous
examples strengthened correct strategies and weakened in-
correct problem solving strategies, respectively. Grosse and
Renkl studied whether explaining both correct and incorrect
examples makes a difference to learning and whether high-
lighting errors helps students learn from those errors. Their
empirical studies in which no help was provided showed
some learning benefit of erroneous examples, but unlike the
less ambiguous Siegler et al results, the benefit they uncov-
ered was only for learners with strong prior knowledge and
for far transfer learning. Research in other domains, such as
medical education, has shown benefits of erroneous exam-
ples in combination with elaborate feedback in the acquisi-
tion of problem-solving schemata.
More relevant to the work presented in this paper is Ohls-
son’s theory (Ohlsson 1996) argues that errors occur due to
overgeneralization (applying rules to situations where they
do not apply). Error-detection requires declarative knowl-
edge (e.g.knowledge of rules) in the domain. The error
causes a conflict between this knowledge which the learner
believes to be true and what the learner perceives as the
current situation. The way to avoid errors is by special-
izing knowledge structures to trigger only in the situations
where they apply. Ohlsson distinguished between declar-
ative knowledge (knowing that) and practical knowledge
(knowing how), which he defines as the knowledge required
to choose the appropriate action in a perceive-decide-act cy-
cle.
The contribution of our work are empirical studies that
explicitly tackle this dissociation, in particular, in the do-
main of fractions. This in turn can inform the design of
erroneous examples and intruction not only in the context
of technology-enhanced learning, which we focus on, but in
general.
Conclusion
Our goal is a cognitive tool that supports learning from er-
ror, in particular from erroneous examples and that increases
monitoring alert. As the results show, the tool needs to
adapt to the student’s mastery level for sequencing and meta-
cognitive help provision.
This article describes results on our way to support stu-
dents in developing the meta-cognitive skills of error detec-
tion and error correction in ACTIVEMATH.
Acknowledgement This work was supported by Deutsche
Forschungsgemeinschaft in the project ME 1136/7-1.
References
Borasi, R. 1994. Capitalizing on errors as ”springboards for
inquiry”: A teaching experiment. Journal for Research in
Mathematics Education 25(2):166–208.
Curry, L. 2004. The effects of self-explanations of cor-
rect and incorrect solutions on algebra problem-solving per-
formance. In D. Gentner, K. F., and Regier, T., eds., 26th
Annual Conference of the Cognitive Science Society, 1548.
Mahwah, NJ: Erlbaum.
Grosse, C., and Renkl, A. 2004. Learning from worked ex-
amples: What happens if errors are included? In Gerjets, P.;
Elen, J.; Joiner, R.; and Kirschner, P., eds., Instructional de-
sign for effective and enjoyable computer-supported learn-

ing. Tuebingen: Knowledge Media Research Center. 356–
364.
Grosse, C., and Renkl, A. 2007. Finding and fixing er-
rors in worked examples: Can this foster learning outcomes?
Learning and Instruction 17:612–634.
Melis, E., and Kriesell, D. 2009. Effects of erroneous exam-
ples, results of a preliminary school experiment. In V. Dim-
itrova, R. Mizoguchi, B. d. B., and Graesser, A., eds., AIED
2009, volume 200 of Frontiers in Artificial Intelligence and
Applications, 578–580. IOS press.
Monthienvichienchai, R., and Melis, E. 2005. Implementing
courseware to support learning through real-world erroneous
examples: Student’s perceptions of tertiary courseware and
obstacles to implementing effective delivery through vle. In
Remenyi, D., ed., Proceedings of European Conference on
eLearning (ECEL-2005), 281–289. Academic Conferences
International.
Ohlsson, S. 1996. Learning from performance errors. Psy-
chological Review 103:241–262.
Siegler, R., and Chen, Z. 2008. Differentiation and inte-
gration: Guiding principles for analyzing cognitive change.
Developmental Science 11:433–448.
Siegler, R. 2002. Microgenetic studies of self-explanation.
In Granott, N., and Parziale, J., eds., Microdevelopment,
Transition Processes in Development and Learning. Cam-
bridge University Press. 31–58.
Tsamir, P., and Tirosh, D. 2003. In-service mathematics
teachers’ views or errors in the classroom. In International
Symposium : Elementary Mathematics Teaching.
Tsovaltzi, D.; Melis, E.; McLaren, B.; Dietrich, M.;
Goguadze, G.; and Meyer, A.-K. 2009. Erroneous exam-
ples: A preliminary investigation into learning benefits. In
Ulrike Cress, Vania Dimitrova, M. S., ed., European Con-
ference for Technology-Enhanced Learning, EC-TEL 2009,
LNCS 5794, 688–693. Springer-Verlag.
Tsovaltzi, D.; Melis, E.; McLaren, B.; Meyer, K.; Dietrich,
M.; and Goguadze, G. 2010. Learning from erroneous ex-
amples: When and how do students benefit from them? In ,
ed., EC-TEL 2010, volume - of LNCS, –. Springer-Verlag.
vanLehn, K.; Graesser, A.; Jackson, G.; Jordan, P.; and
C.P.Rose. 2007. When are tutorial dialogues more effec-
tive than reading? Cognitive Science 31:3–62.