Page 1

J.C. Lester et al. (Eds.): ITS 2004, LNCS 3220, pp. 854–856, 2004.

© Springer-Verlag Berlin Heidelberg 2004

A Metacognitive ACT-R Model of Students’ Learning

Strategies in Intelligent Tutoring Systems

Ido Roll, Ryan Shaun Baker, Vincent Aleven, and Kenneth R. Koedinger

Human Computer Interaction Institute, Carnegie Mellon University,

5000 Forbes Ave, Pittsburgh, PA 15218

{iroll, rsbaker, aleven}@cs.cmu.edu, koedinger@cmu.edu

Abstract. Research has shown that students’ problem-solving actions vary in

type and duration. Among other causes, this behavior is a result of strategies

that are driven by different goals. We describe a first version of a computa-

tional cognitive model that explains the origin of these strategies and identifies

the tendencies of students towards different learning goals. Our model takes

into account (i) interpersonal differences, (ii) an estimation of the student’s

knowledge level, and (iii) current feedback from the tutor, in order to predict

the next action of the student – a solution, a guess or a help request. Our long-

term goal is to use identification of the students’ strategies and their efficiency

in order to better understand the learning process and to improve the metacog-

nitive learning skills of the students.

1 Introduction

Studies have found some evidence to the connection between students’ metacognitive

decisions while working with ITS and their learning gains (Aleven et al. in press,

Baker et al. 2004, Wood and Wood 1999). We describe here a computational model

that explains such relations, by identifying various learning goals and strategies, as-

signing them to students, and relate them to learning outcomes.

We based our model on log-files of students working with the Geometry Cognitive

Tutor, an ITS based on ACT-R theory (Anderson et al, 1995), which is now in exten-

sive use in American public high schools.

2 The Model

The model identifies various goals and associates each goal with a different local-

strategy that attempts to accomplish it. It assumes that students’ actions, which are

determined by the strategies, are driven by (i) their estimated ability to solve the step,

(ii) their earlier actions and the system’s feedback (e.g., error messages), and (iii)

their tendency towards the different goals. The model assumes that every student has

some tendency towards all goals. The exact combination of tendencies uniquely iden-

tifies the pattern of the individual student.

Page 2

A Metacognitive ACT-R Model of Students’ Learning Strategies 855

Currently, the model includes the following goals and strategies:

Table 1. The goals and strategies in the model

Goal

Learning Oriented

(Learn as much as possible)

Performance Oriented (Solve quickly,

pretend to be working hard...)

Least Effort Oriented (Make progress

with minimal thinking effort)

Help Avoider

(I want to do it myself)

Strategy

After thinking about the question, I solve it if I

can or ask for a hint.

I repeatedly ask for hints until the answer is

revealed to me.

I guess repeatedly until I get it right.

After thinking about the question, I solve it if I

can. Otherwise I guess.

As seen in figure 1, the model has the following stages:

•

The student evaluates her ability to solve the question correct immediately (1).

If she thinks she can, she does so (2).

•

If the student decides that she needs to spend more time thinking 3), she

chooses a local strategy (4) and acts upon it (5).

Fig. 1. Student’s local goals determine their strategies and actions.

The model is implemented in ACT-R, a theory of mind and a framework for cog-

nitive modeling (Anderson et al., 1998)

2.1 Fitting Data

We used data from Aleven et al. (in press), to identify the students’ tendencies ac-

cording to the model. We included only “new questions” data at this point (and not

“after a hint” or “after an error”), for tractability. In addition, only questions to which

the Cognitive Tutor evaluates the skill-level of the student as intermediate were in-

cluded since these actions had the most between-student variance. 1400 actions, per-

formed by 11 students, were analyzed in total.

23

4

5

1

Page 3

856 I. Roll et al.

The correlation between the data to the model’s prediction is 1.00 for all students, and

the average SD across all students is 0.09 (SD = 0.02). The high correlation is proba-

bly an over-fit as a result of too many parameters.

We see a high tendency towards Learning-Orientated and Help-Avoider (0.29 and

0.28 respectively), whereas tendencies towards I-know-it, Performance-Oriented and

Least-Effort are 0.15, 0.15 and 0.12 respectively. These values make sense, given that

students take their time and rarely use hints on their first actions on a new step.

We calculated the correlation between these tendencies and an independent meas-

ure of learning outcomes (as measured by the progress students made from pre- to

post-test, divided by their maximum possible improvement). The only significant

result is that Help-Avoider is highly correlated with learning gain, F(1,9)=5.14,

p<0.05, r=0.58, suggesting that students with higher tendency to avoid help on their

first actions did better in the overall learning experience.

3 Conclusions and Future Work

We observe high correlation with the actions of students, but poorer than expected

correlation to learning gains. We hypothesize that due to too many parameters, the

students’ behavior can be explained in more than one manner, affecting the single

representation of each student and the correlation to learning outcomes. We currently

reduce the number of parameters and update the characteristics of the strategies.

The model should be fitted to all collected data, across all skill levels and including

actions taken after errors and hints. In addition, we plan to run the model on data

from other tutors and correlate the findings to other means of analysis.

We would like to thank John R. Anderson for his suggestions and helpful advice.

References

1. Aleven, V., McLaren, B., Roll, I., Koedinger, K. Toward Tutoring Help Seeking: Applying

Cognitive Modeling to Meta-Cognitive Skills. To appear at Intelligent Tutoring Systems

Conference (2004)

2. Anderson, J. R., A. T. Corbett, K. R. Koedinger, and R. Pelletier, (1995). Cognitive tutors:

Lessons learned. The Journal of the Learning Sciences, 4, 167-207.

3. Baker, R. S., Corbett, A. T., Wagner, A. Z. & Koedinger, K. R., Off-Task Behavior in the

Cognitive Tutor Classroom: When Students “Game the System”, Proceedings of the

SIGCHI conference on human factors in computing systems (2004), p. 383-390, Vol. 6 no.

1.

4. McNeil, N.M. & Alibali, M.W. (2000), Learning Mathematics from Procedural Instruction:

Externally Imposed Goals Influence What Is Learned, Journal of Educational Psychology,

92 #4, 734-744.

5. Wood, H., & Wood, D. (1999). Help seeking, learning and contingent tutoring. Computers

and Education, 33, 153-169.