Content uploaded by Arnon Hershkovitz
Author content
All content in this area was uploaded by Arnon Hershkovitz on Mar 11, 2015
Content may be subject to copyright.
Goal Orientation and Changes of Carelessness
over Consecutive Trials in Science Inquiry
A. HERSHKOVITZ
R.S.J.D. BAKER
J. GOBERT
AND
M. WIXON
Worcester Polytechnic Institute, USA
________________________________________________________________________
In this paper, we studied the relationship between goal orientation within a science inquiry learning
environment for middle school students, and the manifestation of carelessness over consecutive trials.
Carelessness is defined as not demonstrating a skill despite knowing it; we measured carelessness using a
machine learned model. Findings suggest that students with performance goals demonstrate an increase in
carelessness sooner in the set of trials than do students with learning goals, and that students with lack of goals
are consistent with their carelessness over trials.
Key Words and Phrases: carelessness, goal orientation, science inquiry, cluster analysis, automated detectors
________________________________________________________________________
1. INTRODUCTION
In recent years, there is increasing evidence that the goals students have during learning
play a key role in their learning outcomes. These goals might impact learning by creating
different forms of disengagement. One disengaged behavior is carelessness, i.e., when a
student fails to answer an item correctly despite possessing the required skills (Clements,
1982). Previous research has indicated that students possessing mastery or performance
goal orientation have (on average) double the probability of carelessness as students
characterized by low scores for these goal orientations (Hershkovitz et al., 2011). In this
research, carelessness was automatically detected using a log-based machine-learned
model. This measurement can be quickly applied to data from new students and
situations. Within this study, we examine how carelessness is manifested over
consecutive trials by students with different goal orientations.
2. METHODOLOGY
The learning environment. We used the Science Assistments learning environment
(www.scienceassistments.org) and a microworld for phase change to study carelessness
in demonstrating three science inquiry skills – namely, control for variable strategy
(CVS), testing articulated hypotheses, and planning using the table tool. The microworld
engages students in authentic science while supporting inquiry via widgets. The students
completed three learning activities involving these three skills. The learning environment
detects whether students demonstrate inquiry skills using validated machine-learned
models of these behaviors.
Participants, Data. 130 public middle school 8th graders (Central Massachusetts), 12-14
years old. Students’ fine-grained actions were logged, then analyzed at the “clip” level (a
consecutive set of a student’s actions during experimentation). Goal orientation was
assessed with the PALS (Patterns of Adaptive Learning Scales) survey (Midgley et al.,
__________________________________________________________________________________________
Authors’ addresses: Department of Social Science and Policy Studies, Worcester Polytechnic Institute,
Worcester, MA 01609, USA. E-mails: {arnonh, rsbaker, jgobert, mwixon}@wpi.edu.
1997).
Carelessness Detector. We developed the carelessness detector in RapidMiner 5.0 using
REPTree, a regression tree classifier. Carelessness, first predicted at the clip-level, was
averaged at the student-level. The resulting regression tree (a 6-fold cross-validation
correlation of r=0.63) includes 13 variables, has a size of 35, and a total depth of 13.
Cluster Analysis. Exploratory cluster analysis was conducted to group students by their
PALS scores in order to examine whether certain sub-groups of students with specific
characteristic patterns on the PALS survey also differ on carelessness manifestation over
trials. We used Two-step Cluster Analysis (in SPSS 17.0) with the PALS scores (Z-
standardized) and a log-likelihood distance measure. We chose k=3 as it led to more
interesting separations between aspects of the PALS. The clusters corresponded to: 1)
mastery goal orientation (N=35), 2) performance goal orientation (N=66), and 3) lack of
goal orientation (N=20).
3. RESULTS
The mean value of carelessness for the population was 0.08 (SD=0.12; N=130). Overall,
carelessness increased over all trials (as students have more practice opportunities for the
same skill), and its means for activities 1-3 were 0.04 (SD=0.08; N=126), 0.07 (SD=0.13;
N=127), and 0.11 (SD=0.19; N=112), respectively. According to a paired-sample t-test,
the increase in carelessness from activity 1 to 2 is significant, t(122)=2.7, p<0.01, as was
the increase from activity 2 to 3, t(109)=2.2, p<0.05. Next, we analyzed how carelessness
manifests over trials in the different clusters. These results are summarized in Figure 1.
For the students in the Learning Goals cluster, mean carelessness does not significantly
increase from activity 1 to 2, t(32)=1.3, p=0.2, nor from activity 2 to 3, t(27)=1.5, p=0.15;
however, it is significantly higher in activity 3 compared to activity 1, t(28)=3.1, p<0.01.
In the Performance Goals cluster, the mean carelessness significantly increases between
activities 1 to 2, t(61)=2.6, p<0.05, but does not significantly increase between activities
2 to 3, t(56)=1.5, p=0.14.
These results suggest that
the increase in
carelessness for students
with learning goals is
slower but more dramatic
than for students with
performance goals. As for
Lack of Goals cluster, no
significant differences
were found in mean
carelessness for either pair
of activities (all t-tests had
p>0.63).
REFERENCES
CLEMENTS, M.A. 1982. Careless errors made by sixth-grade children on written mathematical tasks. Journal for
Research in Mathematics Education, 13, 136-144.
HERSHKOVITZ, A., BAKER, R.S.J.d., GOBERT, J., WIXON, M. AND SAO PEDRO, M. 2011. Carelessness and goal
orientation in science microworld. Poster presented at AIED’2011, Auckland, New Zealand.
MIDGLEY, C., MAEHR, M., HICKS, L., ROESER, R., URDAN, T., ANDERMAN, E., & KAPLAN, A., ARUN-KUMAR,
R., MIDDLETON, M. (1997). Patterns of adaptive learning survey (PALS). Ann Arbor, MI: University of
Michigan.
Fig. 1. Mean Carelessness over consecutive trials by PALS-based clusters
