Conference PaperPDF Available

Using Anticipatory Diagrammatic Self-explanation to Support Learning and Performance in Early Algebra

Authors:

Abstract and Figures

Prior research shows that self-explanation promotes understanding by helping learners connect new knowledge with prior knowledge. However, despite ample evidence supporting the effectiveness of self-explanation, an instructional design challenge emerges in how best to scaffold self-explanation. In particular, it is an open challenge to design self-explanation support that simultaneously facilitates performance and learning outcomes. Towards this goal, we designed anticipatory diagrammatic self-explanation, a novel form of self-explanation embedded in an Intelligent Tutoring System (ITS). In our ITS, anticipatory diagrammatic self-explanation scaffolds learners by providing visual representations to help learners predict an upcoming strategic step in algebra problem solving. A classroom experiment with 108 middle-school students found that anticipatory diagrammatic self-explanation helped students learn formal algebraic strategies and significantly improve their problem-solving performance. This study contributes to understanding of how self-explanation can be scaffolded to support learning and performance.
Content may be subject to copyright.
Using Anticipatory Diagrammatic Self-explanation to Support
Learning and Performance in Early Algebra
Tomohiro Nagashima, Carnegie Mellon University, tnagashi@cs.cmu.edu
Anna N. Bartel, University of Wisconsin, Madison, anbartel@wisc.edu
Gautam Yadav, Carnegie Mellon University, gyadav@andrew.cmu.edu
Stephanie Tseng, Carnegie Mellon University, stseng2@andrew.cmu.edu
Nicholas A. Vest, University of Wisconsin, Madison, navest@wisc.edu
Elena M. Silla, University of Wisconsin, Madison, esilla@wisc.edu
Martha W. Alibali, University of Wisconsin, Madison, martha.alibali@wisc.edu
Vincent Aleven, Carnegie Mellon University, aleven@cs.cmu.edu
Abstract: Prior research shows that self-explanation promotes understanding by helping
learners connect new knowledge with prior knowledge. However, despite ample evidence
supporting the effectiveness of self-explanation, an instructional design challenge emerges in
how best to scaffold self-explanation. In particular, it is an open challenge to design self-
explanation support that simultaneously facilitates performance and learning outcomes.
Towards this goal, we designed anticipatory diagrammatic self-explanation, a novel form of
self-explanation embedded in an Intelligent Tutoring System (ITS). In our ITS, anticipatory
diagrammatic self-explanation scaffolds learners by providing visual representations to help
learners predict an upcoming strategic step in algebra problem solving. A classroom experiment
with 108 middle-school students found that anticipatory diagrammatic self-explanation helped
students learn formal algebraic strategies and significantly improve their problem-solving
performance. This study contributes to understanding of how self-explanation can be scaffolded
to support learning and performance.
Introduction
Self-explanation
Self-explanation is a learning strategy in which learners attempt to make sense of what they learn by generating
explanations to themselves (Chi et al.,1989; Rittle-Johnson et al., 2017). A number of studies have provided
evidence for the effectiveness of self-explanation across domains (Ainsworth & Loizou, 2003; Bisra et al., 2018).
From a cognitive perspective, self-explanation helps learners integrate to-be-learned information with their prior
knowledge, leading to deeper understanding of the content (Bisra et al., 2018). For example, in the context of
problem solving in mathematics, learners may be asked to provide reasoning for their solved steps in order to
deepen their conceptual understanding of the procedures. Although self-explanation activities may take different
forms (e.g., explaining worked examples, explaining while solving problems, and explaining text passages), they
share the core principle of supporting deeper understanding through connecting new content with existing
knowledge.
Scaffolding self-explanation as a challenging design problem
The demonstrated effectiveness of self-explanation does not guarantee that effective self-explanation activities
are easily designed. Self-explanation can be a demanding task for learners. It has been reported that scaffolding
self-explanation activities facilitates learning (Rittle-Johnson et al., 2017). Prior studies have designed and tested
various types of scaffolded self-explanation, such as presenting menu-based, multiple-choice explanations
(Aleven & Koedinger, 2002; Berthold et al., 2011; Rau et al., 2015), providing training on self-explanation (Hodds
et al., 2014), using visual representations (Ainsworth & Loizou, 2003; Nagashima, Bartel et al., 2020), using
contrasting cases (Sidney et al., 2015), and providing feedback on self-explanation (Aleven & Koedinger, 2002).
All of these types of self-explanation support have been shown to be effective. Yet, there are still
challenges in how best to design optimal scaffolding support for self-explanation. A first challenge lies in how to
design scaffolded self-explanation to promote both conceptual and procedural knowledge. Acquiring both
conceptual and procedural knowledge is fundamental to learning (Rittle-Johnson & Alibali, 1999); however,
studies on scaffolded self-explanation have typically shown it to be effective for enhancing either conceptual
knowledge or procedural knowledge, but not both (Berthold et al., 2011; Nagashima, Bartel et al., 2020; Rau et
al., 2015, but see Aleven & Koedinger, 2002). Rittle-Johnson et al. (2017) explain that this disassociation may be
due to the unique characteristics of specific forms of scaffolding. Self-explanation scaffolding designed to focus
on one aspect of content may hinder learners’ focus on other important aspects. For example, asking students to
select a correct conceptual explanation from among a list of similar explanations in a multiple-choice format
would encourage learners to focus on conceptual understanding of the content, but it would not give an opportunity
for learners to develop their procedural skills (e.g., problem-solving skills).
A second challenge is how to design scaffolded self-explanation that enhances problem-solving
performance when combined with, or embedded in, problem-solving activities. Self-explanation can be time-
consuming, and because self-explanation requires learners to engage in additional cognitive activities, learners
who receive self-explanation support may solve fewer problems in a limited amount of time compared to solving
problems without self-explanation support. If scaffolded appropriately during self-explanation, learners’
performance on the target task would improve. This would result in efficient learning (i.e., learners with self-
explanation achieve similar learning gains with fewer problems or less time spent compared to those without self-
explanation). Most prior studies of self-explanation do not report measures of the problem-solving performance
and efficiency of learning with self-explanation, such as time spent on the task (Bisra et al., 2018; but see Aleven
& Koedinger, 2002). In sum, there are persistent design challenges in how to design effective and efficient self-
explanation that supports both learning and performance.
Designing evidence-based self-explanation scaffolding
To approach these challenges, we designed self-explanation support for a web-based educational software called
an Intelligent Tutoring System (ITS) for algebra problem solving (Long & Aleven, 2014). In our design, self-
explanation is interleaved with problem solving; learners are asked to explain the next strategic problem-solving
step in the form of a diagram before doing the same step in symbols (Figures 1-3). They receive feedback from
the ITS both on their explanation and their step using mathematical symbols. We designed the self-explanation
support following several evidence-based principles from cognitive psychology, educational psychology,
instructional design, and the learning sciences, which we describe below.
Figure 1. The ITS starts by asking a learner to select a correct diagram for the given equation. The ITS gives
correctness feedback on the learner’s choice of diagram.
Figure 2. Next, the ITS asks the learner to explain (by selecting a diagram) what would be a correct and
strategic step to take next. The ITS gives feedback on the choice of diagram.
Figure 3. After selecting a correct and strategic step, the learner enters the step in symbols.
Visual representations designed to support students’ conceptual understanding
Research has shown that visual representations can support conceptual understanding (Rau, 2017). Visual
representations can depict information that is difficult to express through verbal means and can make important
information salient. We chose a visual representation called tape diagrams, which are commonly used in algebra
classrooms in countries such as Japan, Singapore, and the United States (Booth & Koedinger, 2012; Chu et al.,
2017; Murata, 2008). Prior studies using tape diagrams in algebra problem solving show that tape diagrams help
students gain conceptual understanding and avoid conceptual errors (Chu et al., 2017; Nagashima, Bartel et al.,
2020). In particular, our own prior experiment found that diagrammatic self-explanation (in which students, after
each equation-solving step, are asked to select, from three options, a diagram that corresponds to the step) helped
learners gain conceptual knowledge in algebra (Nagashima, Bartel et al., 2020). In the present study, students are
similarly asked to choose tape diagrams as a way to explain their steps, following the principle of anticipatory
self-explanation (Bisra et al., 2018; Renkl, 1997), as explained next.
Anticipatory self-explanation to support understanding of problem-solving strategies
Anticipatory self-explanation is a type of self-explanation in which learners generate inferences about future steps.
Previously, Renkl (1997) found that, when prompted to talk aloud while studying worked examples that provided
solutions step-by-step, many successful self-explainers predicted solutions in advance. In algebra problem
solving, such anticipatory self-explanation, rather than post-hoc self-explanation, can potentially support
inference generation about strategic problem-solving steps (e.g., “what would be a good next step for the equation,
3x + 2 = 8?”). If students consider the mathematical symbols as the target representation to learn, engaging in
step-level anticipatory self-explanation could help students understand strategic next steps, which would improve
both understanding of strategic solution steps and problem-solving performance. On the other hand, post-hoc self-
explanation might not be particularly effective for helping students take strategic problem-solving steps.
Contrasting cases that differ on conceptual features and problem-solving strategies
The use of contrasting cases is an established instructional strategy in which learners are presented with
contrasting examples that differ in meaningful conceptual aspects (Schwartz et al., 2011). Contrasting cases help
learners notice meaningful differences. This instructional strategy is typically used with prompts for self-
explanation, to encourage learners to cognitively and constructively engage with the cases (Sidney et al., 2015).
In the self-explanation support used in the current study, three options of tape diagrams are displayed,
which differ in one conceptual aspect and one strategy-related aspect. For example, in Figure 2 the tutor displays
three diagrams that represent a correct and strategic next step (diagram on the left), an incorrect option (diagram
on the right, in which the subtraction is done on only one side of the equation) and an option that is correct but
not strategic (diagram in the middle, in which 2x was added to both sides, which does not get the learner closer to
the solution). This set of options allows learners to distinguish, not only between correct and incorrect steps, but
also between correct and strategic steps and correct but not strategic steps. In problem states in which two correct
and strategic steps are available (e.g., subtracting 2x from 8x = 2x + 6 or dividing both sides by 2), the ITS shows
those two options and one incorrect option. Engaging with contrasting cases prior to practicing the target problem-
solving skill with symbols might be particularly meaningful, because students would be able to follow the selected
diagram option when entering the solution step with symbols and thereby learn to use correct and strategic steps.
Present investigation and hypotheses
In the present study, we investigate the effectiveness of scaffolded self-explanation support on learning and
performance. We hypothesize that (H1) the anticipatory diagrammatic self-explanation will promote students’
conceptual understanding, enhance procedural skills, and help students learn formal algebraic strategies. We also
hypothesize that (H2) the anticipatory diagrammatic self-explanation will enhance performance during problem
solving in the ITS; students with the support will perform better on learning process measures (e.g., fewer hint
requests and fewer incorrect attempts per step) while solving symbolic problem-solving steps, and they will solve
a similar number of problems as students who do not receive the scaffolded self-explanation support.
Method
Participants
We conducted an in vivo experiment (i.e., a randomized controlled experiment in a real classroom context) at two
private schools in the United States. Participants included 55 6th graders and 54 7th graders across nine class
sections taught by four teachers. The experiment was conducted in October 2020, when both schools adopted a
hybrid teaching mode in which the majority of students (n = 102) attended study sessions in-person and the rest
of the students attended remotely (n = 7). Teachers reported that they had never focused their instruction on tape
diagrams, although they indicated that some students might have seen tape diagrams in their learning materials.
Materials
Intelligent Tutoring System for equation solving
In addition to the anticipatory diagrammatic self-explanation ITS described above, we used a version that did not
include tape diagrams (Figure 4) (Long & Aleven, 2014). In this No-Diagram ITS, students learn to solve
equations step-by-step, but without diagrammatic self-explanation steps. All other features (e.g., step-level
feedback messages and hints) are the same as in the version with tape diagrams. Both ITS versions had four
different types of equations, which were chosen in consultation with the teachers (Table 1). We only used
equations with positive numbers since tape diagrams were not found useful for representing negative numbers
(Nagashima, Yang et al., 2020). Most of the participants in this study, per teachers’ report, had seen or practiced
Levels 1 and 2 problems, but had not learned Levels 3 and 4 problems.
Figure 4. A version of ITS with no diagrammatic self-explanation.
Table 1: Types of equations the tutor contained and the number of problems in the tutor
Equation type
Example
Number of problems in the ITS
Level 1
x + a = b
x + 3 = 5
4
Level 2
ax + b = c
2x + 3 = 7
5
Level 3
ax + b = cx
5x = 3x + 2
6
Level 4
ax + b = cx + d
5x +2 = 3x + 8
13
Test instruments
We developed web-based pretest and posttest assessments to assess students’ conceptual and procedural
knowledge of basic algebra. The tests contained several items drawn from our previous work (Nagashima, Bartel
et al., 2020) as well as new items. The conceptual knowledge items consisted of eight multiple-choice questions
and one open-ended question, which assessed a wide range of conceptual knowledge constructs, including like
terms, inverse operations, isolating variables, and the concept of keeping both sides of an equation equal. We also
included four problem-solving items (e.g., “solve for x: 3x + 2 = 8”), including two items that were similar to
those included in the ITS and two transfer items involving negative numbers. We developed two isomorphic
versions of the test that varied only with respect to the specific numbers used in the items; participants received
one form as pretest and the other as posttest (with versions counterbalanced across subjects).
Procedure
The study took place during two regular mathematics classes. The classes were virtually connected to the
experimenters and remote learners through a video conferencing system. Students were randomly assigned to
either the Diagram condition or the No-Diagram condition. In the Diagram condition, students used the ITS with
anticipatory diagrammatic self-explanation. In the No-Diagram condition, students used the ITS with no self-
explanation support. The only difference between the Diagram and No-Diagram conditions was whether students
self-explained their solution steps in the form of tape diagrams or not.
On the first day, all students first worked on the web-based pretest for 15 minutes. Then a teacher or the
experimenter showed a 5-minute video describing how to use the ITS and what tape diagrams represent to all
students. Next, students practiced equation solving using their randomly-assigned ITS version for approximately
15 minutes. On the second day, students started the class by solving equation problems in the assigned ITS for
approximately 15 minutes. After working with the ITS, students took the web-based posttest for 15 minutes.
Students were given access to both ITS versions a week after the study.
Results
Pre-post test results
One 6th grader was absent for the second day and excluded from the analysis; therefore, we analyzed data from
the remaining 108 students, namely, 54 6th-graders (28 Diagram, 26 No-Diagram) and 54 7th-graders (27 Diagram,
27 No-Diagram). Open-ended items were coded for whether student answers were correct or incorrect by two
researchers (Cohen’s kappa = .91). Table 2 presents raw pretest and posttest performance on conceptual
knowledge (CK) and procedural knowledge (PK) items. The maximum scores were 9 and 4, respectively.
Table 2: Means and standard deviations (in parentheses) for CK and PK on the pretest and posttest
CK (maximum score: 9)
PK (maximum score: 4)
Pretest
Posttest
Pretest
Posttest
3.53 (1.56)
3.80 (1.94)
1.51 (1.17)
1.73 (1.39)
3.42 (2.02)
4.01 (2.27)
1.55 (.92)
1.83 (1.57)
We first tested hypothesis H1 (benefits of anticipatory diagrammatic self-explanation with respect to
learning outcomes). We analyzed the data using hierarchical linear modeling (HLM) because the study was
conducted in nine classes taught by four teachers at two schools. According to both AIC and BIC, a two-level
model showed the best fit, in which students (level 1) were nested in classes (level 2). The inclusion of teachers
(level 3) and schools (level 4) did not improve the model fit. We ran two HLMs with posttest scores on CK and
PK as dependent variables, type of ITS assigned as the independent variable, and pretest scores (either CK or PK
given the dependent variable) as a covariate. For both CK and PK, there was no significant effect of the
Diagram/No-Diagram condition (CK: t(99.3) = -1.030, p = .31, PK: t(99.4) = -0.292, p = .77). We also ran two
additional HLMs, regressing pretest-posttest gains for CK and PK (dependent variables) on type of ITS. There
was a significant gain from pretest to posttest for CK (t(108) = 2.778, p < .01) but not for PK (t(106) = 1.153, p
= .26), and no significant effect of ITS type. This suggests that students in both ITS conditions improved in
conceptual knowledge but not in procedural knowledge.
We then analyzed the strategies that students used to solve the problem-solving items on the pretest and
posttest. We adopted a coding scheme by Koedinger et al. (2008), which identified both formal (algebraic) and
informal (non-algebraic) ways of solving equations (Cohen’s kappa = .73; Table 3). We were primarily interested
in the Algebra strategy because the goal of the ITS was to help students learn the formal algebraic strategy. We
performed the strategy coding independent of the correctness coding used to calculate students’ test scores. On
the pretest, 11 students in the Diagram condition and 17 students in the No-Diagram condition used the Algebraic
strategy on one or more problem-solving items. More students did so on the posttest; 26 students in the Diagram
condition and 23 students in the No-Diagram condition used the Algebraic strategy. We used McNemar’s test to
compare the frequency of use of the Algebra strategy at pretest and posttest for each condition. The increase in
frequency was significant (p < .01) for students in the Diagram condition but was not significant (p = .11) for
students in the No-Diagram condition. This pattern also held when we limited the analysis to problems involving
negative numbers (transfer problems); there was a pretest-posttest increase of only 1 student in the No-Diagram
condition, but 12 students in the Diagram condition (p < .01). These findings suggest that, although students who
learned with anticipatory diagrammatic self-explanation did not have greater gains on tests of conceptual and
procedural knowledge, they were more likely to learn the formal algebraic strategy and to apply it to problems
with no diagram support, even for problem types that they did not practice in the ITS (H1 partially supported).
Table 3: Strategies used to solve equations, adapted from Koedinger et al. (2008)
Strategy name
Description
Example answer for 3x + 2 = 8
Algebra
Student uses algebraic manipulations to find a solution
3x = 6
x = 6/3 = 2
Unwind
Student works backward using inverse operations to
find a solution
8 – 2 = 6
6/3 = 2
Guess and Check
Student tests potential solutions by substituting
different values
3*2 + 2 = 8
6 + 2 = 8
Other
Student uses other non-algebraic strategies
3 + 2 = 5
8/5 = 1.6
Answer Only
Student provides an answer without showing any
written work
x = 2
No Attempt
Student leaves problem blank or explicitly indicates
that she/he does not know how to solve the problem
“I don’t know”
Log data analysis on students’ learning processes
Next, we tested hypothesis H2 (benefits of anticipatory diagrammatic self-explanation with respect to learning
processes), using log data from the ITS. Specifically, we looked at “learning curves”, which plot students’
performance within the ITS over time (Rivers et al., 2016). Figure 5 depicts learning curves for the two conditions.
The y-axis shows the error rate on steps in tutor problems, averaged across students and skills, and the x-axis
shows the sequence of opportunities for practicing each skill. Learning curve analysis assumes that learning
occurs when a curve starts with a relatively high initial error rate and gradually goes down as students practice
the target skills. The curves are fit to student performance data using the Additive Factors Model (AFM), a
specialized form of logistic regression (Rivers et al., 2016). In our study, students practiced a variety of equation-
solving skills (e.g., subtracting variable terms). We expected that students who learned with diagrammatic self-
explanation support would perform better in the ITS than their peers who did not receive the support (H2). On the
symbolic problem-solving steps in the ITS (i.e., excluding the performance on the self-explanation steps, which
only occurred in the Diagram condition), students in the Diagram condition had a lower error rate than students
in the No-Diagram condition. Figure 5 shows learning curves averaged across all symbolic equation-solving skills
students in both conditions practiced. Students in the Diagram condition made fewer errors than those in the No-
Diagram condition, especially on the earlier opportunities. Both groups improved as they solved more problems
(i.e., both curves show a gradual decline). After much practice, the No Diagram condition eventually lowered
their error rate to the same level as the Diagram condition.
In parallel to the trend observed in the learning curves, we found that, when restricting the analysis to
symbolic steps only (i.e., excluding diagrammatic self-explanation steps), students who received the self-
explanation support trended toward using fewer hints (t(89.52) = -1.812, p = .07) and spent significantly less time
on each symbolic problem-solving step (t(99.51) = -2.238, p = .03) than students who did not receive the self-
explanation support (Table 4). The average number of problems solved in the ITS during the (fixed amount of)
available time did not differ significantly across conditions, (t(99.30) = -0.528, p = .60) (Table 4).
In summary, the students in both conditions practiced a similar number of problems in the ITS in a similar
amount of time overall, and the anticipatory diagrammatic self-explanation helped students spend less time and
ask for fewer hints on symbolic steps (H2 partially supported). In addition, the learning curves indicate that
students in both conditions learned equation-solving skills eventually, but the students in the Diagram condition
learned them faster and had a smoother experience, with fewer errors.
Table 4: Average number of problems solved, number of incorrect attempts, number of hint requests, and average
time spent on symbolic steps in the ITS (standard deviation).
Average number of
problems solved
Average number of hints
requested per step
Average time spent per
step
Diagram
15.40 (9.02)
0.68 (0.96)
15.99 (9.91)
No-Diagram
16.17 (11.16)
1.02 (1.38)
20.27 (13.95)
Figure 5. Learning curves for the Diagram condition (red) and the No-Diagram condition (green) averaged
across the skills students practiced during the symbolic problem-solving steps. Dark and light blue lines show
predicted curves based on the AFM (dark blue: Diagram condition, light blue: No-Diagram condition).
Discussion and Conclusion
Self-explanation has been shown to support student learning in various domains, but it is not easy to design
appropriately-scaffolded self-explanation activities. Our study investigated the effectiveness of anticipatory
diagrammatic self-explanation as a proposed approach to enhancing both learning and performance. We found
that anticipatory diagrammatic self-explanation embedded in an Intelligent Tutoring System (ITS) helped students
learn to apply a formal, algebraic problem-solving strategy to problems outside the ITS and to transfer problems
involving negative numbers (H1). Anticipatory diagrammatic self-explanation also supported student
performance within the ITS, measured by lower learning curves, less frequent use of hints, and less time spent on
each symbolic equation-solving step (H2). Anticipatory self-explanation did not lead to differences in posttest
scores, contrary to H1, but it helped students learn more efficiently; students learned the formal algebraic strategy
while solving a similar number of problems with less time and fewer errors and hint requests, and they achieved
similar gains on conceptual knowledge (H2).
We attribute these findings to the design and learning principles used in supporting anticipatory
diagrammatic self-explanation. Specifically, we reason that the process of selecting the next correct-and-strategic
problem-solving step, depicted diagrammatically, helped students perform better and faster on the corresponding
step with symbols. On steps with symbols, students had a diagrammatic representation of the step available to
them on the screen. They could refer to this representation as they sought to express the step using mathematical
symbols. Engaging in this cognitive process may have helped students understand step-level formal strategies in
a visual form (e.g., visually seeing that constant terms are taken out from both sides of an equation). Comparing
and contrasting the different tape diagrams may have supported students in selecting steps that were both correct
and strategic, and it may have helped them avoid using informal strategies, such as guessing. It may be, as well,
that the better performance resulting from the anticipatory diagrams gave students a bit more confidence to take
on the challenge of moving towards formal algebra.
An intriguing question is why the ITS with anticipatory diagrammatic self-explanation did not lead to
greater gains in conceptual and procedural knowledge than the ITS with no diagram support. Regarding
procedural knowledge, students did not make gains from pretest to posttest in either condition. Further, there was
no difference in solving equations correctly between the conditions at post-test, even though students with
diagrams exhibited greater use of formal problem-solving strategies. It is possible that students in the Diagram
condition might need more practice in correctly applying the formal strategy they acquired in the ITS without the
help of diagrams. In other words, it seems that students in the Diagram condition developed further towards formal
use of algebra than their counterparts in the No-Diagram condition, but not yet to the extent that the use of the
more challenging formal strategies paid off in terms of improved correctness. Regarding conceptual knowledge,
it might be that the anticipatory use of diagrams in the ITS focused students primarily on strategic issues, as the
diagrams were used in planning problem-solving steps. It may be that students need to engage in “principle-based
explanation” (Renkl, 1997) to facilitate acquisition of conceptual knowledge (e.g., verbally explaining why the
selected diagram is correct and strategic). It might also be that students with varying levels of prior knowledge
benefit from diagrammatic self-explanation differently (Booth & Koedinger, 2012). Future studies should
examine the effects of anticipatory diagrammatic self-explanation with students having varying degrees of prior
knowledge and experience in algebra.
Our study has several limitations. First, the study was conducted with one specific type of diagrams, tape
diagrams, and it focused on one specific task domain, equation solving in algebra. To understand how the results
could generalize across domains and types of visual representations, more research is needed to examine the
effects of anticipatory diagrammatic self-explanations. Also, it is possible that students were not very motivated
to work on the posttest, especially given that the study was conducted remotely during the COVID-19 pandemic.
This may have contributed to the absence of pretest-posttest gains in procedural knowledge, even though the
learning curves suggest that learning occurred.
In summary, we designed a novel self-explanation scaffolding support for students in middle-school
algebra, namely, anticipatory diagrammatic self-explanation. We investigated the effectiveness of this support
embedded in an Intelligent Tutoring System, in a classroom study. We found that anticipatory diagrammatic self-
explanation helped students learn formal algebraic strategies and perform better on problem solving, while making
similar conceptual gains as students who did not receive the support. Our study contributes to the theoretical and
practical understanding of how visual representations, contrasting cases, and anticipatory self-explanation can be
integrated into scaffolding support that helps students learn and perform effectively and efficiently.
References
Ainsworth, S., & Loizou, A. (2003). The effects of self-explaining when learning with text or diagrams. Cognitive
Science, 27(4), 669-681.
Aleven, V. A., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining
with a computer-based cognitive tutor. Cognitive Science, 26(2), 147-179.
Berthold, K., Röder, H., Knörzer, D., Kessler, W., & Renkl, A. (2011). The double-edged effects of explanation
prompts. Computers in Human Behavior, 27(1), 69-75.
Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F., & Winne, P. H. (2018). Inducing self-explanation: A meta-analysis.
Educational Psychology Review, 30, 703-725.
Booth, J. L., & Koedinger, K. R. (2012). Are diagrams always helpful tools? Developmental and individual
differences in the effect of presentation format on student problem solving. British Journal of
Educational Psychology, 82(3), 492-511.
Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study
and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182.
Chu, J., Rittle-Johnson, B., & Fyfe, E. R. (2017). Diagrams benefit symbolic problem-solving. British Journal of
Educational Psychology, 87(2), 273-287.
Hodds, M., Alcock, L., & Inglis, M. (2014). Self-explanation training improves proof comprehension. Journal
for Research in Mathematics Education, 45(1), 62-101.
Koedinger, K. R., Alibali, M. W., & Nathan, M. J. (2008). Trade-offs between grounded and abstract
representations: Evidence from algebra problem solving. Cognitive Science, 32(2), 366-397.
Long, Y., Aleven, V. (2014). Gamification of joint student/system control over problem selection in a linear
equation tutor. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the
12th International Conference on Intelligent Tutoring Systems (pp. 378-387). New York: Springer.
Murata, A. (2008). Mathematics teaching and learning as a mediating process: The case of tape diagrams.
Mathematical Thinking and Learning, 10(4), 374-406.
Nagashima, T., Bartel, A. N., Silla, E. M., Vest, N. A., Alibali, M. W., & Aleven, V. (2020). Enhancing conceptual
knowledge in early algebra through scaffolding diagrammatic self-explanation. In M. Gresalfi & I. S.
Horn (Eds.), Proceedings of International Conference of the Learning Sciences (pp. 35-43). Nashville,
TN: International Society of the Learning Sciences.
Nagashima, T., Yang, K., Bartel, A. N., Silla, E. M., Vest, N. A., Alibali, M. W., & Aleven, V. (2020). Pedagogical
Affordance Analysis: Leveraging teachers’ pedagogical knowledge to elicit pedagogical affordances and
constraints of instructional tools. In M. Gresalfi & I. S. Horn (Eds.), Proceedings of International
Conference of the Learning Sciences (pp. 1561-1564). Nashville, TN: International Society of the
Learning Sciences.
Rau, M. A. (2017). Conditions for the effectiveness of multiple visual representations in enhancing STEM
learning. Educational Psychology Review, 29(4), 717-761.
Rau, M. A., Aleven, V., & Rummel, N. (2015). Successful learning with multiple graphical representations and
self-explanation prompts. Journal of Educational Psychology, 107(1), 30.
Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive Science,
21(1), 1-29.
Rittle-Johnson, B., & Alibali, M. W. (1999). Conceptual and procedural knowledge of mathematics: Does one
lead to the other?. Journal of Educational Psychology, 91(1), 175.
Rittle-Johnson, B., Loehr, A. M., & Durkin, K. (2017). Promoting self-explanation to improve mathematics
learning: A meta-analysis and instructional design principles. ZDM, 49(4), 599-611.
Rivers, K., Harpstead, E., & Koedinger, K. R. (2016). Learning curve analysis for programming: Which concepts
do students struggle with? In Proceedings of the 2016 ACM Conference on International Computing
Education Research (pp. 143-151).
Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus inventing with contrasting
cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103(4),
759.
Sidney, P. G., Hattikudur, S., & Alibali, M. W. (2015). How do contrasting cases and self-explanation promote
learning? Evidence from fraction division. Learning and Instruction, 40, 29-38.
Acknowledgments
This research was supported by NSF Award #1760922 and by the Institute of Education Sciences, U.S.
Department of Education, through Award #R305B150003 to the University of Wisconsin–Madison. The opinions
expressed are those of the authors and do not represent views of NSF or the U.S. Department of Education. We
thank Max Benson, Susan Brunner, Octav Popescu, Jonathan Sewall, and all the participating students and
teachers. Copyright 2021 International Society of the Learning Sciences. This is an author’s copy.
... Both participating teachers noted that students' prior exposure to tape diagrams was minimal; some students had seen tape diagrams in learning materials, but the instruction had never focused specifically on tape diagrams. 1 Data from the 6 th and 7 th graders were also included in Nagashima et al. (2021), which reports a few overlapping results but with different research questions and types of analyses. ...
Preprint
Full-text available
Although visual representations are generally beneficial for learners, past research also suggests that often only a subset of learners benefits from visual representations. In this work, we designed and evaluated anticipatory diagrammatic self- explanation, a novel form of instructional scaffolding in which visual representations are used to guide learners’ inference generation as they solve algebra problems in an Intelligent Tutoring System. We conducted a classroom experiment with 84 students in grades 5-8 in the US to investigate the effectiveness of anticipatory diagrammatic self-explanation on algebra performance and learning. The results show that anticipatory diagrammatic self-explanation benefits learners on problem-solving performance and the acquisition of formal problem-solving strategies. These effects mostly did not depend on students’ prior knowledge. We analyze and discuss how performance with the visual representation may have influenced the enhanced problem-solving performance. Presented and published at the Annual Meeting of the Cognitive Science Conference 2021. https://escholarship.org/uc/item/17q6g4db#main
Chapter
Despite the drastic change to school environments due to the COVID-19 pandemic, it is still important that educational technology researchers conduct school-based research to understand the impact of technology in an authentic learning context, even remotely. However, the transition to remote research has made it challenging for researchers to collect classroom data, observe teacher-student-technology interactions, and facilitate study sessions. To explore how researchers can effectively plan and conduct technology-based educational studies in the new, evolving classroom research environment, we interviewed seven US teachers, investigating their perceptions of participating in remote classroom studies. Based on the findings and the authors’ experience of running classroom studies, we propose a framework that educational technology researchers can refer to when planning and conducting research in the evolving classroom research environment. Specifically, the framework informs researchers of several types of questions they can explore with teachers, students, and researchers themselves to be better prepared to address potential confusion, unexpected issues, and practical benefits in remote classroom research. Our work contributes by providing a practical guide for running technology-based research remotely, which may remain as a means of classroom research in the future. Some of the findings and the framework would also be applied to in-person classroom research setting.
Conference Paper
Full-text available
When designing an instructional tool and using it in pedagogical activities, it is essential that designers and users understand what pedagogical affordances and constraints the tool provides to support its successful integration into targeted pedagogical activities. Toward this end, we developed Pedagogical Affordance Analysis (PAA). PAA involves analyzing teachers' Pedagogical Content Knowledge and/or Technological Pedagogical Content Knowledge to elicit pedagogical affordances and constraints that are specific to a given instructional goal. Information obtained through PAA can help in designing, refining, and/or evaluating instructional tools. We present a case study in which we used PAA to successfully design a visual representation for middle-school algebra. To the best of our knowledge, PAA is the only available systematic method that leverages teachers' pedagogical knowledge in identifying pedagogical affordances and constraints. PAA can be used across a wide range of existing tools and prototypes of to-be-designed tools.
Conference Paper
Full-text available
Many studies have shown that visual representations can enhance student understanding of STEM concepts. However, prior research suggests that visual representations alone are not necessarily effective across a broad range of students. To address this problem, we created a novel, scaffolded form of diagrammatic self-explanation in which students explain their problem-solving steps in the form of diagrams. We used contrasting cases to support students' sense-making between algebraic equations and diagrams in the self-explanation activity. We conducted a classroom experiment with 41 students in grades 5 and 6 to test the effectiveness of this strategy when embedded in an Intelligent Tutoring System for algebra. We found that scaffolded diagrammatic self-explanation enhanced conceptual knowledge for students who did not have prior knowledge of formal equation-solving strategies. The study is the first experimental study showing that visual representations can enhance conceptual knowledge in early algebra.
Article
Full-text available
Self-explanation is a process by which learners generate inferences about causal connections or conceptual relationships. A meta-analysis was conducted on research that investigated learning outcomes for participants who received self-explanation prompts while studying or solving problems. Our systematic search of relevant bibliographic databases identified 69 effect sizes (from 64 research reports) which met certain inclusion criteria. The overall weighted mean effect size using a random effects model was g = .55. We coded and analyzed 20 moderator variables including type of learning task (e.g., solving problems, studying worked problems, and studying text), subject area, level of education, type of inducement, and treatment duration. We found that self-explanation prompts are a potentially powerful intervention across a range of instructional conditions. Due to the limitations of relying on instructor-scripted prompts, we recommend that future research explore computer-generation of self-explanation prompts.
Article
Full-text available
Background: The format of a mathematics problem often influences students' problem-solving performance. For example, providing diagrams in conjunction with story problems can benefit students' understanding, choice of strategy, and accuracy on story problems. However, it remains unclear whether providing diagrams in conjunction with symbolic equations can benefit problem-solving performance as well. Aims: We tested the impact of diagram presence on students' performance on algebra equation problems to determine whether diagrams increase problem-solving success. We also examined the influence of item- and student-level factors to test the robustness of the diagram effect. Sample: We worked with 61 seventh-grade students who had received 2 months of pre-algebra instruction. Method: Students participated in an experimenter-led classroom session. Using a within-subjects design, students solved algebra problems in two matched formats (equation and equation-with-diagram). Results: The presence of diagrams increased equation-solving accuracy and the use of informal strategies. This diagram benefit was independent of student ability and item complexity. Conclusions: The benefits of diagrams found previously for story problems generalized to symbolic problems. The findings are consistent with cognitive models of problem-solving and suggest that diagrams may be a useful additional representation of symbolic problems.
Article
Full-text available
Promoting self-explanation (i.e., generating explanations for oneself in an attempt to make sense of new information) is a recommended study strategy and instructional practice. A meta-analysis of the literature on prompting self-explanation to improve mathematics learning confirmed that prompted self-explanation leads to a small to moderate improvement in procedural knowledge, conceptual knowledge and procedural transfer when assessed immediately after the intervention. However, evidence that self-explanation reliably promotes learning within a classroom context or retention of knowledge over a delay is much more limited. Moderator analyses indicated that the effect on immediate outcomes was stronger if scaffolding of high-quality explanation was provided but did not vary based on whether time on task was controlled across conditions. Based on the research literature, we propose instructional recommendations for mathematics educators: (a) scaffold high-quality explanations via training on self-explanation or structuring the self-explanation responses, (b) design explanation prompts so they do not sacrifice attention to other important content, (c) prompt learners to explain correct information, and (d) prompt learners to explain why common misconceptions are incorrect. We conclude with issues for future research, such as the need for additional research on effective use of self-explanation in classroom contexts and the merits of self-explanation relative to alternative instructional techniques.
Article
Full-text available
Visual representations play a critical role in enhancing science, technology, engineering, and mathematics (STEM) learning. Educational psychology research shows that adding visual representations to text can enhance students’ learning of content knowledge, compared to text-only. But should students learn with a single type of visual representation or with multiple different types of visual representations? This article addresses this question from the perspective of the representation dilemma, namely that students often learn content they do not yet understand from representations they do not yet understand. To benefit from visual representations, students therefore need representational competencies, that is, knowledge about how visual representations depict information about the content. This article reviews literature on representational competencies involved in students’ learning of content knowledge. Building on this review, this article analyzes how the number of visual representations affects the role these representational competencies play during students’ learning of content knowledge. To this end, the article compares two common scenarios: text plus a single type of visual representations (T+SV) and text plus multiple types of visual representations (T+MV). The comparison yields seven hypotheses that describe under which conditions T+MV scenarios are more effective than T+SV scenarios. Finally, the article reviews empirical evidence for each hypothesis and discusses open questions about the representation dilemma.
Conference Paper
The recent surge in interest in using educational data mining on student written programs has led to discoveries about which compiler errors students encounter while they are learning how to program. However, less attention has been paid to the actual code that students produce. In this paper, we investigate programming data by using learning curve analysis to determine which programming elements students struggle with the most when learning in Python. Our analysis extends the traditional use of learning curve analysis to include less structured data, and also reveals new possibilities for when to teach students new programming concepts. One particular discovery is that while we find evidence of student learning in some cases (for example, in function definitions and comparisons), there are other programming elements which do not demonstrate typical learning. In those cases, we discuss how further changes to the model could affect both demonstrated learning and our understanding of the different concepts that students learn.
Article
This study examined relations between children's conceptual understanding of mathematical equivalence and their procedures for solving equivalence problems (e.g., 3 + 4 + 5 = 3 + -). Students in 4th and 5th grades completed assessments of their conceptual and procedural knowledge of equivalence, both before and after a brief lesson. The instruction focused either on the concept of equivalence or on a correct procedure for solving equivalence problems. Conceptual instruction led to increased conceptual understanding and to generation and transfer of a correct procedure. Procedural instruction led to increased conceptual understanding and to adoption, but only limited transfer, of the instructed procedure. These findings highlight the causal relations between conceptual and procedural knowledge and suggest that conceptual knowledge may have a greater influence on procedural knowledge than the reverse.
Article
Past research has shown that both contrasting cases instruction and prompts to self-explain promote students' learning in mathematics. However, it is not clear whether these instructional approaches enhance learning through similar mechanisms or whether each supports learning in distinct ways. The purpose of this study was to investigate the unique and combined effects of comparison, defined as noticing similarities and differences, and explanation, defined as making sense of problems, on student learning, and to assess whether these processes are more effective when combined than when implemented separately. We also investigated potential mechanisms involved in comparison and explanation. We addressed these issues in the domain of fraction division. Prompts to self-explain promoted conceptual learning, but inviting comparison, without prompts to self-explain, did not. The quality of students' self-explanations was affected by both prompts to self-explain and encouragement to compare. The findings suggest that contrasting cases instruction is effective, at least in part, because it typically involves self-explanation.
Article
Research shows that multiple external representations can significantly enhance students' learning. Most of this research has focused on learning with text and 1 additional graphical representation. However, real instructional materials often employ multiple graphical representations (MGRs) in addition to text. An important open question is whether the use of MGRs leads to better learning than a single graphical representation (SGR) when the MGRs are presented separately, 1-by-1 across consecutive problems, accompanied by text and numbers. A further question is whether providing support for students to relate the different representations to the key concepts that they depict can enhance their benefit from MGRs. We investigated these questions in 2 classroom experiments that involved problem solving practice with an intelligent tutoring system for fractions. Based on 112 sixth-grade students, Experiment 1 investigated whether MGRs lead to better learning outcomes than 1 commonly used SGR, and whether this effect can be enhanced by prompting students to self-explain key concepts depicted by the graphical representations. Based on 152 fourth-and fifth-grade students, Experiment 2 investigated whether the advantage of MGRs depends on the specific representation chosen for the SGR condition because prior research suggests that some SGRs might promote learning more than others. Both experiments demonstrate that MGRs lead to better conceptual learning than an SGR, provided that students are supported in relating graphical representations to key concepts. We extend research on multiple external representations by demonstrating that MGRs (presented in addition to text and 1-by-1 across consecutive problems) can enhance learning.