Further Evidence for Regularity in Student Learning Rates Across Demographic, Academic Proficiency, and Motivational Groups

Abstract

To replicate and expand on previous results showing that student learning rates are regular under favorable learning conditions (Koedinger et al., 2023), we used a dataset of 426 students who engaged with a cognitive tutoring system throughout an academic year. We used the individual additive factors model (iAFM) to estimate student parameters: intercept (initial knowledge) and slope (learning rate). Our findings replicate regularity in learning rates, including across student subgroups defined by sex assigned at birth, socioeconomic status, academic proficiency, and self-reported measures of motivation. Moreover, initial knowledge within subgroups was positively correlated with academic performance and self-reported goal orientation at the onset of the school year. There were no significant correlations found between learning rate and demographic or motivational measures. One important implication of these findings is that interventions should target prior knowledge and availability of additional practice opportunities for struggling students, along with motivational support so that students seek those external additional opportunities.

Full text

Companion Proceedings 14th International Conference on Learning Analytics & Knowledge (LAK24)
Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0)
Further Evidence for Regularity in Student Learning Rates Across
Demographic, Academic Proficiency, and Motivational Groups
Gillian Gold Conrad Borchers Paulo Carvalho
Yale University Carnegie Mellon University Carnegie Mellon University
gillian.gold@yale.edu cborcher@cs.cmu.edu pcarvalh@andrew.cmu.edu
ABSTRACT: To replicate and expand on previous results showing that student learning rates
are regular under favorable learning conditions (Koedinger et al., 2023), we used a dataset of
426 students who engaged with a cognitive tutoring system throughout an academic year. We
used the individual additive factors model (iAFM) to estimate student parameters: intercept
(initial knowledge) and slope (learning rate). Our findings replicate regularity in learning rates,
including across student subgroups defined by sex assigned at birth, socioeconomic status,
academic proficiency, and self-reported measures of motivation. Moreover, initial knowledge
within subgroups was positively correlated with academic performance and self-reported goal
orientation at the onset of the school year. There were no significant correlations found
between learning rate and demographic or motivational measures. One important implication
of these findings is that interventions should target prior knowledge and availability of
additional practice opportunities for struggling students, along with motivational support so
that students seek those external additional opportunities.
Keywords: tutoring systems, student motivation, learning rate, cognitive modeling, K-12
1 INTRODUCTION
Intelligent tutoring systems (ITS), such as cognitive tutors, create favorable learning conditions by
providing deliberate practice, feedback, and step-by-step instruction (Koedinger et al., 2023). By
recording fine-grained student interactions with learning systems, ITS data also allows for modeling
student learning and knowledge growth (Koedinger et al., 2023). Koedinger et al. (2023) found a lack
of variation in learning rates, measured by increases in correctness after each practice opportunity of
given Knowledge Components (KCs). However, there may be learning variation within specific student
subgroups. Past research has demonstrated that diligent students tend to engage more readily with
learning tasks, while less diligent students often fail to fully utilize available learning opportunities
(Bernacki et al., 2013). Taken together, we hypothesized that, when exploring different subgroups,
academic proficiency and goal orientation would be related to increased variation only in students’
initial knowledge. If the cognitive tutor creates optimal learning conditions (Koedinger et al., 2023), it
will help all students learn at the same rate, but differences in initial knowledge will contribute to
differences in academic proficiency and motivational approaches.
2 METHODOLOGY
2.1 Data Collection
We retrieved the dataset (ds613) used in this analysis from DataShop (Koedinger et al., 2010). The
data was collected over a year-long study in a suburban middle school in the mid-Atlantic region of
the United States. Students utilized the Carnegie Learning Cognitive Tutor software (CogTutor). The
dataset comprises math practice learning transactions of 426 6-12th grade students who used the
tutor for approximately two class periods per week for 8 months. It also includes demographic

Companion Proceedings 14th International Conference on Learning Analytics & Knowledge (LAK24)
Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0)
information, academic proficiency measures, and motivational survey responses (Bernacki et al.,
2013). 155 students had no accompanying demographic or external achievement information, so 271
students were included in the analysis.
2.2 Analysis Methods
We used the individual additive factors model (iAFM) to calculate individual student parameters:
intercept represents initial knowledge and slope represents learning rate (Koedinger et al., 2023).
Student subpopulations were based on demographics (sex assigned at birth, socioeconomic status),
academic proficiency (previous final grade, final math grade, state math exam), and self-reported
motivational measures (mastery approach, performance approach, performance avoidance)
(Bernacki et al., 2013). We followed the same approach as in Koedinger et al. (2023) to calculate the
number of opportunities to reach mastery. To test whether variability in learning rate and intercept
within subgroups, we divided students into high and low groups (using median split) for each
measure of interest, and then compared opportunities to reach 90% mastery across students with
low/high initial knowledge and those with low/high learning rate. A larger difference suggests higher
variability (see Koedinger et al., 2023). To test for relationships between each measure of interest
and initial knowledge or learning rate, we used Pearson’s correlation. We hypothesized that learning
rate would not vary substantially even within sub-groups and that proficiency and motivational
measures would correlate more with initial knowledge than learning rate.
3 RESULTS
We found that, across subgroups in the academic and motivational measures, initial knowledge varied
considerably. For example, students who rated high on mastery approach and had high initial
knowledge took on average 4.41 opportunities to reach mastery, whereas those who had lower initial
knowledge took on average 12.20 opportunities: a difference of 7.79 opportunities. However, for
learning rate, we found small variation in reaching mastery (0.37 opportunities). Table 1 presents the
difference in learning opportunities to reach mastery between high and low initial knowledge/learning
rate students for selected subgroups of students.
Table 1: Difference in number of opportunities to reach mastery between high and low initial
knowledge/learning rate for different sub-groups of students.
State Math Exam
Mastery Approach
Low
High
Low
High
(High-Low) Initial Knowledge
6.30
5.40
8.33
7.79
(High-Low) Learning Rate
0.33
0.42
0.39
0.37
There were no statistically significant differences in initial knowledge or learning rate for students
based on sex assigned at birth (r(267)=.08, p=.170) or eligibility for free/reduced lunch (r(268)= .04,
p=.490). Higher initial knowledge was significantly associated with higher prior final grade (r(238)=.40,
p<.001), final math grade (r(254)=.44, p<.001), and state math exam (r(101)=.54, p<.001) (Figure 1).
Moreover, higher initial knowledge was significantly associated with higher reports of mastery
approach (r(266)=.17, p=.004) (Figure 1), performance approach (r(266)=.13, p=.038), and

Companion Proceedings 14th International Conference on Learning Analytics & Knowledge (LAK24)
Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0)
performance avoidance (r(266)=.20, p<.001). There were no significant differences in learning rate in
subgroups based on academic proficiency or achievement goals (all r’s<.12, p’s>.05).
Figure 1: Left: Positive correlation between initial knowledge and final state math exam scores
(95% CI [0.39, 0.66]). Right: Positive correlation between initial knowledge and self-reported
mastery approach at the beginning of the school year (95% CI [0.06, 0.29]).
4 DISCUSSION
This study supports the hypothesis that learning rates are generally regular across students (Koedinger
et al., 2023). Consistent with Koedinger et al.’s findings and theoretical interpretation, prior
knowledge varied substantially: a student with low prior knowledge requires about 8 more
opportunities to reach mastery than a student with high prior knowledge, regardless of their mastery
approach. Comparatively, learning rate varied very little. However, and importantly, we found positive
correlations between initial knowledge and both academic proficiency and motivational measures.
These findings suggest that prior knowledge may contribute to higher variability in learning outcomes.
If a student is highly motivated, they might have sought more opportunities in the past, begin with a
higher level of knowledge, and perform better at the start of the year. However, higher motivation
does not seem to affect the speed at which students learn. Provided that the learning conditions are
favorable, as in the cognitive tutor used in this study, all students will learn at approximately the same
rate. Thus, potential interventions should target initial knowledge by providing additional practice
opportunities, which may also require motivational support for struggling students.
REFERENCES
Bernacki, M. L., Nokes-Malach, T. J. & Aleven, V. (2013). Fine-grained assessment of motivation over
long periods of learning with an intelligent tutoring system: methodology, advantages, and
preliminary results. In R. Azevedo, & V. Aleven (Eds.) International Handbook of Metacognition
and Learning Technologies.
Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J. (2010) A Data
Repository for the EDM community: The PSLC DataShop. In Romero, C., Ventura, S.,
Pechenizkiy, M., Baker, R.S.J.d. (Eds.) Handbook of Educational Data Mining. Boca Raton, FL:
CRC Press.
Koedinger, K. R., Carvalho, P. F., Liu, R., & McLaughlin, E. A. (2023). An astonishing regularity in student
learning rate. Proceedings of the National Academy of Sciences of the United States of
America, 120(13), e2221311120. https://doi.org/10.1073/pnas.2221311120