A Metacognitive ACT-R Model of Students' Learning Strategies in Intelligent Tutoring Systems.

Full text

Page 1

J.C. Lester et al. (Eds.): ITS 2004, LNCS 3220, pp. 854–856, 2004.
© Springer-Verlag Berlin Heidelberg 2004
A Metacognitive ACT-R Model of Students’ Learning
Strategies in Intelligent Tutoring Systems
Ido Roll, Ryan Shaun Baker, Vincent Aleven, and Kenneth R. Koedinger
Human Computer Interaction Institute, Carnegie Mellon University,
5000 Forbes Ave, Pittsburgh, PA 15218
{iroll, rsbaker, aleven}@cs.cmu.edu, koedinger@cmu.edu
Abstract. Research has shown that students’ problem-solving actions vary in
type and duration. Among other causes, this behavior is a result of strategies
that are driven by different goals. We describe a first version of a computa-
tional cognitive model that explains the origin of these strategies and identifies
the tendencies of students towards different learning goals. Our model takes
into account (i) interpersonal differences, (ii) an estimation of the student’s
knowledge level, and (iii) current feedback from the tutor, in order to predict
the next action of the student – a solution, a guess or a help request. Our long-
term goal is to use identification of the students’ strategies and their efficiency
in order to better understand the learning process and to improve the metacog-
nitive learning skills of the students.
1 Introduction
Studies have found some evidence to the connection between students’ metacognitive
decisions while working with ITS and their learning gains (Aleven et al. in press,
Baker et al. 2004, Wood and Wood 1999). We describe here a computational model
that explains such relations, by identifying various learning goals and strategies, as-
signing them to students, and relate them to learning outcomes.
We based our model on log-files of students working with the Geometry Cognitive
Tutor, an ITS based on ACT-R theory (Anderson et al, 1995), which is now in exten-
sive use in American public high schools.
2 The Model
The model identifies various goals and associates each goal with a different local-
strategy that attempts to accomplish it. It assumes that students’ actions, which are
determined by the strategies, are driven by (i) their estimated ability to solve the step,
(ii) their earlier actions and the system’s feedback (e.g., error messages), and (iii)
their tendency towards the different goals. The model assumes that every student has
some tendency towards all goals. The exact combination of tendencies uniquely iden-
tifies the pattern of the individual student.

Page 2

A Metacognitive ACT-R Model of Students’ Learning Strategies 855
Currently, the model includes the following goals and strategies:
Table 1. The goals and strategies in the model
Goal
Learning Oriented
(Learn as much as possible)
Performance Oriented (Solve quickly,
pretend to be working hard...)
Least Effort Oriented (Make progress
with minimal thinking effort)
Help Avoider
(I want to do it myself)
Strategy
After thinking about the question, I solve it if I
can or ask for a hint.
I repeatedly ask for hints until the answer is
revealed to me.
I guess repeatedly until I get it right.
After thinking about the question, I solve it if I
can. Otherwise I guess.
As seen in figure 1, the model has the following stages:
•
The student evaluates her ability to solve the question correct immediately (1).
If she thinks she can, she does so (2).
•
If the student decides that she needs to spend more time thinking 3), she
chooses a local strategy (4) and acts upon it (5).
Fig. 1. Student’s local goals determine their strategies and actions.
The model is implemented in ACT-R, a theory of mind and a framework for cog-
nitive modeling (Anderson et al., 1998)
2.1 Fitting Data
We used data from Aleven et al. (in press), to identify the students’ tendencies ac-
cording to the model. We included only “new questions” data at this point (and not
“after a hint” or “after an error”), for tractability. In addition, only questions to which
the Cognitive Tutor evaluates the skill-level of the student as intermediate were in-
cluded since these actions had the most between-student variance. 1400 actions, per-
formed by 11 students, were analyzed in total.
23
4
5
1

Page 3

856 I. Roll et al.
The correlation between the data to the model’s prediction is 1.00 for all students, and
the average SD across all students is 0.09 (SD = 0.02). The high correlation is proba-
bly an over-fit as a result of too many parameters.
We see a high tendency towards Learning-Orientated and Help-Avoider (0.29 and
0.28 respectively), whereas tendencies towards I-know-it, Performance-Oriented and
Least-Effort are 0.15, 0.15 and 0.12 respectively. These values make sense, given that
students take their time and rarely use hints on their first actions on a new step.
We calculated the correlation between these tendencies and an independent meas-
ure of learning outcomes (as measured by the progress students made from pre- to
post-test, divided by their maximum possible improvement). The only significant
result is that Help-Avoider is highly correlated with learning gain, F(1,9)=5.14,
p<0.05, r=0.58, suggesting that students with higher tendency to avoid help on their
first actions did better in the overall learning experience.
3 Conclusions and Future Work
We observe high correlation with the actions of students, but poorer than expected
correlation to learning gains. We hypothesize that due to too many parameters, the
students’ behavior can be explained in more than one manner, affecting the single
representation of each student and the correlation to learning outcomes. We currently
reduce the number of parameters and update the characteristics of the strategies.
The model should be fitted to all collected data, across all skill levels and including
actions taken after errors and hints. In addition, we plan to run the model on data
from other tutors and correlate the findings to other means of analysis.
We would like to thank John R. Anderson for his suggestions and helpful advice.
References
1. Aleven, V., McLaren, B., Roll, I., Koedinger, K. Toward Tutoring Help Seeking: Applying
Cognitive Modeling to Meta-Cognitive Skills. To appear at Intelligent Tutoring Systems
Conference (2004)
2. Anderson, J. R., A. T. Corbett, K. R. Koedinger, and R. Pelletier, (1995). Cognitive tutors:
Lessons learned. The Journal of the Learning Sciences, 4, 167-207.
3. Baker, R. S., Corbett, A. T., Wagner, A. Z. & Koedinger, K. R., Off-Task Behavior in the
Cognitive Tutor Classroom: When Students “Game the System”, Proceedings of the
SIGCHI conference on human factors in computing systems (2004), p. 383-390, Vol. 6 no.
1.
4. McNeil, N.M. & Alibali, M.W. (2000), Learning Mathematics from Procedural Instruction:
Externally Imposed Goals Influence What Is Learned, Journal of Educational Psychology,
92 #4, 734-744.
5. Wood, H., & Wood, D. (1999). Help seeking, learning and contingent tutoring. Computers
and Education, 33, 153-169.