Content uploaded by Esther Ziegler

Author content

All content in this area was uploaded by Esther Ziegler on Oct 13, 2017

Content may be subject to copyright.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 1

Delayed Benefits of Learning Elementary Algebraic Transformations

Through Contrasted Comparisons

Esther Zieglera and Elsbeth Sterna

aCenter for Research on Learning and Instruction, ETH Zurich, Switzerland

Last submitted and accepted version:

in Learning and Instruction, DOI: 10.1016/j.learninstruc.2014.04.006

Correspondence concerning this article should be addressed to Esther Ziegler, Research

on Learning and Instruction, Institute for Behavioral Sciences, ETH Zurich, Universitätsstrasse

41, 8092 Zurich, Switzerland. Tel: +41 446 325 820. E-mail: esther.ziegler@ifv.gess.ethz.ch.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 2

Abstract

Students studying algebra often make mistakes because of superficial similarities between

addition and multiplication problems. In two experiments, we investigated whether these errors

can be prevented by presenting addition and multiplication problems in such a way that students

are encouraged to compare the problems at a deeper level. In Experiment 1, 72 sixth graders

were assigned to two self-learning programs. In the contrast program, addition and multiplication

were mixed and juxtaposed. In the sequential program, students first received only addition

problems followed by multiplication problems. The results revealed that during the training,

students performed worse under the contrast condition. However, in the follow-up tests (1-day,

1-week, 3-months), these findings were reversed: the contrast group clearly outperformed the

sequential group. The findings were replicated under improved methodological conditions in

Experiment 2 with 154 sixth graders. These experiments show that contrasted comparison of

superficially similar but conceptually different material results in improved long-term learning.

KEYWORDS: comparison, contrasting, mathematical knowledge, concept knowledge,

algebra learning

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 3

1. Introduction

Research has provided strong empirical support for comparison as a fundamental learning

mechanism that has a positive effect on meaningful conceptual learning in different areas (for an

overview, see Gentner, 2010; Holyoak, 2005; Rittle-Johnson & Star, 2011). When two units

(e.g., objects, problems, pictures) are juxtaposed, intentional comparison promotes a deep

processing of the materials because their similarities and differences become highlighted. This

phenomenon helps learners to abstract principles that may be used to solve novel problems

(Catrambone & Holyoak, 1989; Gentner, 1983; Gick & Holyoak, 1983). In this way,

comparisons were used to learn complex concepts by being presented two examples of the same

concept that differed in their surfaces. Learners who studied two problems simultaneously

outperformed learners who studied the two problems separately. This outperformance has for

instance been demonstrated for negotiation principles using two different cover stories for the

same negotiation strategy (Gentner, Loewenstein, & Thompson, 2003) or for the concept of heat

flow using two different scenarios depicting heat flow (Kurtz, Miao, & Gentner, 2001). The

direct comparison of superficially different but structurally equal (i.e., isomorphic) examples

appears to help learners overcome contextual limitations, a crucial step in understanding

complex concepts. The instruction to compare material offers learners a way to actively construct

meaningful knowledge.

Positive effects of comparisons were also demonstrated for learning mathematical

procedures in real-life school settings, e.g., when learning how to solve equations (Rittle-Johnson

& Star, 2007) and for computational estimation (Star & Rittle-Johnson, 2009). Comparing two

solution strategies led to greater learning gains compared to the sequential processing of these

strategies. Chase, Shemwell, and Schwartz (2010) showed that learners who had to compare

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 4

three cases of electromagnetic flux outperformed learners who studied the cases separately.

Other research has shown beneficial effects of comparing examples of the same concept.

According to Hattikudur and Alibali (2010), comparing equal signs with other relational symbols

is more effective than simple instruction about the equal sign. Inventing the physical formula for

density using contrasted cases was superior to being told the formula, followed by practicing

with the cases (Schwartz, Chase, Oppezzo, & Chin, 2011). Moreover, it has been shown that

comparing incorrect examples can facilitate learning (Durkin & Rittle-Johnson, 2012).

Although comparing two or three examples of the same concept has been shown to be

beneficial for the meaningful learning of various school topics, this method is rarely used in

classrooms, as for instance, in mathematics, where a great number of concepts and procedures

must be learned and distinguished from each other. Therefore, we wish to examine whether the

application of comparisons may be extended beyond learning single complex concepts to

learning a broad range of concepts, using algebra as an example.

1.1. The challenge to acquire algebraic language competence

Mathematical competencies grow through acquiring knowledge of concepts and

procedures which, over the years and under favorable conditions, build on each other and form a

network of knowledge that is broadly applicable and transferable (Schneider & Stern, 2009;

Stern, 1997). One challenge in learning mathematics is that the concepts are often very similar

and highly related, e.g., in algebra, where students commonly have substantial difficulties

learning the rule system (Blume & Heckman, 2000; Kamii & Dominick, 1997; Kieran, 1992).

Learning algebra requires not only making use of the formal language in rich contextual

settings but also learning the language itself with its rules and conventions (Kieran, 2004;

Kirshner & Awtry, 2004). In fact, a thorough knowledge of algebraic language provides a solid

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 5

foundation and thus may be used flexibly in later applications, e.g., to successfully solve

equations. Learning the language is at the core of elementary algebra. Algebraic language is best

learned through frequent transformations of algebraic expressions, but this key practice is often

neglected (Ottmar, Landy, & Goldstone, 2012). A common problem in managing algebraic

expressions flexibly is confusion between the two dominant operations of addition and

multiplication, e.g., incorrectly solving the problem “a2 + a2” as “a4” instead of “2a2”.

Many authors have documented confusion errors occurring in basic transformations as

well as in handling fractions and solving equations (Booth, 1988; Hirsch & Goodman, 2006;

Kirshner & Awtry, 2004). In algebraic addition, like terms are summarized, e.g., “x + x = 2x”.

However, x and xy are different types and thus cannot be simplified by transformation; instead,

they must be written “x + xy”. In algebraic multiplication, like factors are summarized to

exponents, e.g., “x ∙ x ∙ x = x3”, and unlike factors are joined to form a product, e.g., “x ∙ y ∙ z =

xyz”. Therefore, by transforming expressions, the continuity of terms in addition (xy + x + xy +

x = x + x + xy + xy = 2x + 2xy) must be clearly distinguished from the splitting of factors in

multiplication (xy ∙ x ∙ xy ∙ x = x ∙ y ∙ x ∙ x ∙ y ∙ x = x4y2). As core operations in algebraic

transformations, addition and multiplication must be understood thoroughly. Instruction should

clearly distinguish between these two structurally different and perceptually similar principles

from the very beginning to prevent confusion and to offer a strong start in learning algebra.

Two reasons for frequent confusion between addition and multiplication in solving

algebra problems are the tendency to focus on perceptual features of the problem and the

tendency to hastily automate mathematical procedures. We expect systematic comparison to

offer a means of overcoming both these misleading tracks.

1.2. Comparisons as a means of overcoming the tendency to focus on perceptual features

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 6

It is well documented that, when initially faced with new information, people tend to pay

more attention to the salient perceptual features instead of noticing the more important structural

characteristics (Chi, Feltovich, & Glaser, 1981; Gentner & Namy, 1999; Loewenstein &

Gentner, 2001). In algebra, problems often appear very similar because they are composed of

only letters, numbers, and signs. This similarity may induce misperceptions of the forms of the

correct rules and mislead the learner’s understanding of algebra rules (Kirshner & Awtry, 2004;

Ottmar et al., 2012). Goldstone and colleagues (2010) describe this confusion as a conflict

between rule-based and perceptual processes. Without redirecting the learner’s attention from the

surface features to the crucial structural elements, confusion between addition and multiplication

is preprogrammed. Hence, when teaching algebra, it is necessary to train students explicitly to

focus on the syntactic structure, i.e., to make them recognize rules and procedures (Kirshner &

Awtry, 2004; Ottmar et al., 2012).

Comparisons help to overcome this misleading tendency to focus on surface patterns (Chi

et al., 1981; Gentner & Namy, 1999; Holyoak, 2005). Providing learners with the explicit

instruction to compare objects may shift the focus from the surface to the deeper structural level,

which results in a significant effort to detect and learn the underlying principles (Catrambone &

Holyoak, 1989; Mason, 2004; Schwartz & Bransford, 1998); therefore, choosing the appropriate

comparison material is a crucial factor. Depending on the presented material, certain features of

the juxtaposed examples are accentuated. Winston (1975) introduced “near miss” for concepts

that only differ in a small number of features. Such near-miss contrasts enhance the principle

extracting and appear to be more resistant to interference effects, likely because critical

convergence features are highlighted (Gick & Paterson, 1992).

Two examples may be more or less similar on the surface or structural level. During the

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 7

comparison process, even the surface similarities may help to identify structural differences more

easily because they make the differences more salient, which again promotes principle learning

and transfer (Loewenstein & Gentner, 2001; Markman & Gentner, 1993). However, if surfaces

are too dissimilar, children and even adults may miss the underlying concepts that the examples

are supposed to demonstrate. This may be a problem if the learners do not have enough prior

knowledge to align the examples (Gentner, 2010). The best alignment is enabled when the

examples under comparison are similar both in their surfaces and in their relational structure

(e.g., Richland, Morrison, & Holyoak, 2006). Thus, Gentner (2010, p. 769) suggests that

"sequences of close, highly alignable exemplars should be the ideal learning situation". Inspired

by this idea of repeatedly offering comparisons of similar materials, we wanted to examine

whether the application of comparisons may be extended beyond learning single complex

concepts to learning a broader range of principles. This might be especially promising in

mathematics, where a great number of concepts and procedures must be learned and

distinguished from each other.

1.3. Comparisons as a means to prevent learners from hastily automate mathematical

procedures

Learning in many academic domains (particularly in mathematics) is often limited to

purely superficial procedure memorization (Kamii & Dominick, 1997; NCTM, 2000). When

students are repeatedly presented with similar types of problems, they may automate the solution

procedure. When trying to solve new problems, a student may automatically retrieve an incorrect

procedure because the problem resembles a different type of problem. An example of blind

adherence to an automated procedure is illustrated when algebraic multiplication problems are

incorrectly solved by adding the problem components instead of multiplying them. Students who

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 8

repeatedly practiced addition rules such as “xy + xy + xy = 3xy” are prone to retrieve and apply

this rule when faced with the similar multiplication problem “xy ∙ xy ∙ xy =” and may answer with

“3xy” instead of “x3y3”.

By analyzing Swiss and German mathematics textbooks, one predominantly finds a

sequenced, blocked concept introduction, followed by extensive practice before proceeding to

the next type of problem. Often, the introduction follows a progression of operational signs: first

addition, then subtraction, multiplication, and division, usually followed by mixed sessions.

People have the strong illusion that blocking or massing is more effective, and textbooks rely on

this practice (Kornell & Bjork, 2008; Rohrer & Pashler, 2010), which may stem from learning

experiences that memorizing one algorithm at time was easier than learning multiple algorithms

simultaneously. If concepts are introduced in this sequential manner, they will eventually result

in interferences (Anderson, 1983), which McCloskey and Cohen (1989) have described as “the

sequential learning problem”. In mathematics, concepts frequently build on precedents and thus

often differ only in a few respects. Therefore, more similar concepts are more prone to

interferences as well as to the false extrapolation or an overgeneralization of an automated rule

(Kirshner & Awtry, 2004; Matz, 1982). An example of interferences or false extrapolation is

when addition and multiplication procedures are confused in the same problem such as when one

incorrectly splits the factors in additions, then correctly summarizes the letters: “xy + xy + xy = 3

∙ xy = 3xy”, and “xy ∙ xy ∙ xy = x ∙ y ∙ x ∙ y ∙ x ∙ y = x3y3”, but the answer for “xy + xy + xy” is “x +

y + x + y + x + y = 3x + 3y”.

Because it is more demanding to memorize two algorithms simultaneously, comparison

tends to suppress simple memorizing and instead forces students to actively engage in

understanding the underlying structures of the presented examples (Kang & Pashler, 2012;

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 9

Mitchell, Nash, & Hall, 2008; Sweller & Chandler, 1994). Comparison is often desirable for

students to encounter difficulties during learning as a result of instructional impediments because

it pushes learners to overcome difficulties they otherwise would not experience. (R. Bjork, 1994;

Kapur & Bielaczyc, 2012). Although this approach may lead to more errors at the beginning and

slow the learning process, it tends to enhance long-term retention and transfer of the material

(Rohrer & Pashler, 2010; Rohrer & Taylor, 2007). A desirable level of difficulty might consist of

offering challenging material that makes it necessary for students to connect pieces of

information to fully understand the problem. This type of connection is what Sweller and

Chandler (1994) called the “intrinsic cognitive load”, a term denoting the inherent complexity of

materials. Because the complexity of materials may be described by the number of interacting

elements, a higher cognitive load is required if several related elements must be processed

simultaneously (Sweller, 1994). This requirement implies that complex material cannot be

processed well in single elements. If this material were processed sequentially, the cognitive load

would be lower, but the overall comprehension would be limited. Therefore, even though a

comparison-based instructional design would increase the difficulty and the mental effort

needed, contrasting may offer exactly this opportunity to interconnect different concepts by

directing the students’ attention to the relevant aspects of the material.

1.4. The current project

Based on the promising effects of comparison for learning a complex concept or

procedure, our aim was to examine the effects of learning material that repeatedly fosters the

comparison of principles that are perceptually similar and therefore easy to confuse. In our

design, a series of contrasted examples were presented over several days in a real school setting;

this material was tested against a traditional sequential presentation of the same material. We

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 10

chose the two core principles of addition and multiplication in algebraic transformations, which

are prone to confusion and error. We hypothesized that juxtaposing superficially similar addition

and multiplication problems would increase the difficulty and the mental effort needed but would

also make their differences more salient and thereby help to gain better mastery in the long term.

Prior research has shown that only a thoughtful selection of the units to be compared,

along with explicit instruction telling the learners to pay attention, makes comparison beneficial

(Catrambone & Holyoak, 1989; Schwartz & Bransford, 1998). Deep processing through

comparison is ensured with the help of prompts that help focus the learners’ attention, as in the

instruction to self-explain examples (Chi, 2000; Renkl, 1997; Rittle-Johnson, 2006). Self-

explaining has proven to encourage the active elaboration of learning materials and works

especially well in combination with worked examples to create a basis for activities that promote

cognitive activation. Worked examples provide a complete solution procedure: the problem, the

solution steps, and the final solution. Because the result is already given, the goal for the students

is to discover the underlying rules by themselves (e.g., Renkl, 2002). Moreover, worked

examples may support an understanding of the problem and the underlying principle if learners

are encouraged to deeply engage the problem. This engagement is what the instruction to self-

explain may stimulate; therefore, we constructed our materials as a self-learning program with

worked examples and prompts for self-explanation.

The learning material in our study comprised algebraic expressions that had to be

simplified by strategies of algebraic transformations. The students were required to provide

explicit explanations of the principles underlying the strategy of transformation. The learning

gains were measured with two separate tests. The “algebraic transformation test” comprised

algebraic expressions to be simplified, and assessed students’ ability to correctly apply

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 11

transformations. The second test, “algebraic transformation explanations”, required students to

write down the two algebra principles and demonstrate how to apply them; it assessed students’

ability to explain how transformations are solved. We expected to find learning benefits for the

contrasted introduction compared to the sequential introduction of addition and multiplication

problems.

To explore the effect of contrasts on learning algebra, two experiments were conducted.

In the first experiment, we tested the impact of a contrasted concept introduction on the

acquisition of algebra knowledge compared to a sequential, consecutive introduction. In the

second experiment, we replicated the findings of the first experiment with extended testing and

an improved design.

2. Method

2.1. Participants

Sixth graders were chosen as participants because we wanted students with no prior

algebra knowledge. In the Swiss mathematics curriculum, algebra is not introduced until

secondary school, which starts at Grade 7; consequently, sixth graders are unlikely to have

received any formal instruction in algebra, but, based on their arithmetic competencies they can

be expected to be ready to learn elementary algebra principles. Participants were recruited from

four urban and suburban public schools of the canton Zurich. Teachers were asked not to choose

students (a) with insufficient German language comprehension, (b) with special needs, or (c)

who were unable to fulfill the minimum standard of school performance. All the students were

volunteers, and the parents had to give their written consent. Every class was rewarded with 200

Swiss francs (approximately150 euros), and each student received a small gift.

2.2. Design and Procedure

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 12

In a 2 (group: contrast, sequential) x 3 (time: one day later, one week later, three months

later) mixed-factorial design, we investigated effects on algebraic transformations and on explicit

transformation explanations. Each student participated in four training sessions and in three

follow-up sessions as listed in Table 1. Both groups participated in 90 minutes training sessions

on four consecutive days, during which the students were asked to work through a self-study

program. In the contrast group, students were presented worked examples in a contrasted mixed

order; i.e., addition and multiplication problems were given simultaneously. In the sequential

group, students practiced the problems in a sequential blocked order; i.e., all addition problems

in a sequential manner for two days, followed by multiplication problems for two days. The two

programs contained the same worked examples but differed in the order of the presentation of

the examples and tasks (see Figure 1 and Table 2).

The training occurred in groups of 10 – 15 students in rooms of the school. The students

worked individually on their learning programs and sat at sufficient distances from their

classmates so they could not look at each other’s work sheets. They were instructed to work on

their own and to ask the instructors directly if they had questions or problems with the material.

All groups were trained by the first author, who was also educated as a primary school teacher,

and who was present at all times to guide the training and the testing together with a research

assistant.

2.3. Materials

2.3.1. Pretest on prior algebra knowledge: presented in training session 1

Although Swiss sixth graders have not yet received any formal algebra instruction, it is

possible that some of them could spontaneously solve algebraic transformation problems by

referring to their arithmetic knowledge. The test presented at the very beginning of Session 1 was

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 13

composed of the following eight algebra problems requiring transformations: “a + a + a + a =”,

“5 + a + a + 5 + a =”, “c · c · c =”, “2 · 2 · z · 2 · z =”, “7b + 7b =”, “7b · 7b =”, “ab · 4ab =”,

and “xy + xy + xy + xy”.

2.3.2. Self-study material: presented in training sessions 1 – 4

The instruction material was a paper-pencil version and consisted of several worksheets,

each with worked algebra examples and a self-study part. In the first session, a slide presentation

was given to explain how to read and write terms with letters and how to use the mathematical

expression “raise to the power of”. No other direct instruction was given; instead, students were

asked to derive all the rules on their own from the worked examples with the help of the self-

study instructions. Therefore, it was necessary to present the worked examples in increasing

difficulty. The challenge of designing the learning material was to determine the appropriate

order and content of the worked examples for each algebra concept that contained all the rules in

a suitable sequence. The addition and multiplication problems chosen used the same numbers

and letters and thus differentiated only in the use of addition and multiplication signs, which

naturally impacted the solution steps and the results. This characteristic implied that the worked

examples could be presented side by side in the contrast version of the teaching (see Figure 2A).

This side-by-side presentation highlighted the superficial similarity of the worked examples,

making the underlying differences between addition and multiplication more salient to the

students in the contrast group.

Several pilot studies were conducted, first with children on a one-on-one basis and then

with small groups. The worked examples and tasks were changed several times until the

definitive version of the learning steps was developed. Finally, there were two series of worked

examples, which illustrated nine addition and nine multiplication learning steps (for an excerpt

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 14

see Figure 1). Every learning step exemplified one or two rules. The steps were balanced in the

contrast group such that the corresponding steps in both addition and multiplication could be

introduced simultaneously in the contrasted format. In the sequential group, all the addition steps

were introduced first, followed by the multiplication steps. Finally, the addition and

multiplication learning steps were assigned to the work sheets (see Table 2). For the contrast

group, there were nine work sheets. The sequential group was assigned ten work sheets. In the

sequential version, two blocks were provided on the same work sheet to keep the processing

amount equal; however, the fifth and tenth work sheet contained only one learning step to keep

from mixing addition with multiplication.

The work sheets were comprised of three sections: the worked examples, a self-

explanation section, and trial tasks. At the top of each work sheet, students were presented with

two blocks of two or three worked examples, either one block of addition and one block of

multiplication (contrast group) or one or two consecutive blocks of the same operation

(sequential group). An example for each group is presented in Figure 2.

In the self-explanation section, students were instructed to write down how such

problems are solved and to ascertain the underlying principle on their own. To facilitate this step,

students were given questions prompting them to explain the worked examples. These questions

were used to guide them in looking more carefully at the examples. In the contrast group, a block

of addition and multiplication examples were always juxtaposed, and the questions prompted the

students to compare the worked examples: e.g., "Compare the addition and multiplication

examples", "Describe the different solutions", "Can you explain why students confuse addition

with multiplication?", or "Give a tip to a classmate as to what he/she should pay attention to in

order not to confuse addition with multiplication". In the sequential group, one or two

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 15

consecutive blocks of the same operation were always processed. Because two consecutive steps

disclose the same principle, comparison of the steps does not make sense; therefore, these

questions prompted students to describe the worked examples in detail: e.g., "What have you

noticed in the examples", "Describe exactly how to solve such problems", "What do you have to

pay attention to?", or "Explain it to a classmate in detail". Students’ self-explanations were

always assessed by the research leader (first author). If the explanations were incorrect, students

were required to correct them. If the explanations were too short, students were asked to write

them out in greater detail before continuing to the next section. If students had problems

understanding the examples, they were helped by directing their attention to the corresponding

worked examples and by instructing them to describe each step in the solution process. At the

end of every work sheet were more self-explanation prompts asking students to write out a short

reflection about what they had learned from filling out the work sheet. Contrast students were

asked to consider possible confusing errors that could occur and how to prevent them. Sequential

students were prompted to describe what they had to pay attention to. At the end of each day’s

training, students were told to summarize the rules they learned about the algebra concepts in a

written explanation.

There were two parts in the trial tasks section. In the first part, students were instructed to

generate two to three of their own examples for each block of worked examples following the

model of the presented examples. They were asked to write down the intermediate steps to the

solution and to invent varied and interesting examples using other numbers and letters, e.g.,

“Invent two varied examples for each block! Write down all intermediate steps to the solution as

in the worked examples!” The generated examples were assessed by the research leader and, if

necessary, corrected by the students before continuing to the next part. In the second part,

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 16

students were given trial problems to apply and deepen the learned algebra principles. Both

groups were given 4 to 6 trial tasks per sheet with instructions to solve them by writing down the

intermediate steps in addition to the solution. Before the research leader corrected the trials,

students were told to review these problems by themselves using the examples on the front of the

page.

The sequential learners were given an additional training of 20 mixed addition and

multiplication problems at the end of the fourth training day with the instruction, “You learned

addition and multiplication in algebra, now you will receive mixed problems to solve.” The

problems were then checked and returned to the students for corrections. This extra training for

the sequential group was designed such that the sequential learners would not be at a

disadvantage and to mimic the teaching given in an ordinary school setting. After the sequential

introduction of two operations, mathematics teachers usually give mixed tasks to make sure both

concepts are understood correctly. While the same learning material was used for both groups,

the presentation order of the material was unique to each group and thus presented some slight

variation. However, all together, the tasks were balanced so that at the end of the training

students in both groups processed exactly the same nine addition and nine multiplication worked

examples and exactly the same number of problems (trial tasks, repetition tests, immediate

learning tests).

When students had finished their work sheets they received some optional non-

mathematical tasks to prevent the students from disturbing others, e.g., finding the differences

between two pictures, or solving riddles or sudoku puzzles.

2.3.3. Immediate learning tests: presented in training sessions 1 – 4

To compare the immediate learning gains of both groups, two different types of algebra

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 17

tests were given to all participants in sessions 1 to 4. At the beginning of each session, students

were given a repetition test, on which they solved 6 – 8 algebra items similar to the ones they

had worked on the day before to assess what they remembered. Immediately after every work

sheet, students were given a learning test, on which they solved 3-8 algebra items per test sheet

without being allowed to use the instruction material. These problems were similar to the ones

presented on the work sheet the students had just finished. When students did not remember the

examples, they were instructed to guess the solution and told that the next learning step would

repeat the solution steps for such problems. This step allowed us to assess what they had learned

during the session.

2.3.4. Control measures: presented in follow-up session 1

To control for individual characteristics that might affect algebra learning, four measures

were assessed. Logical Reasoning was assessed with a figural and a numerical subtest of a

German intelligence test (subtests 3 and 4 of the LPS by Horn (1983)). The test is based on

Thurstone’s primary mental abilities with a maximum score of 40 for each subtest. Arithmetic

Knowledge was assessed with a speed test consisting of five sheets, each with 28 - 52 items.

Students were required to solve each sheet within 90 seconds and were allowed to write only the

results. The first three sheets contained simple arithmetic problems (additions with one single-

digit addend, subtractions with a single-digit subtrahend, and single-digit multiplications). In

addition, there were two sheets with more complex arithmetic problems (two-digit additions and

multiplications). The arithmetic knowledge score was determined by the solution rates of correct

answers. The students’ mathematical school achievement was measured in the form of their

school grades on mathematics and school grades on German. The grades were reported by

students’ teachers. In Switzerland, school grades range from 6 (best) to 1 (worst).

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 18

2.3.5. Algebraic transformation test: presented in follow-up sessions 1 – 3

The algebraic transformations test contained 58 items, which assessed the students’

ability to correctly apply transformations practiced in sessions 1 - 4 (e.g., “a2 · a · ay · 4a =”,

“5ab + b + 3b + 2ab + 2b =”, “y · y3 · y2 · y =“, “2 + 5x + 4 + 2x + 3 =”). Examples from every

work sheet were represented, with 2 – 4 problems representing each learning step. The items

were matched to the problems of the work sheets with other variables and numbers, i.e., every

item was an algebraic term that needed to be transformed into the shortest version by applying

the correct rules. The items were ordered in increasing difficulty but always overlapped by 2 or 3

learning steps, within which the items were randomly mixed. The same algebra test was applied

in all three follow-up sessions, because students were not supposed to remember the items.

Scoring. Transformation knowledge was determined by the number of correct answers.

In addition to the total score of the correctly solved problems, there was also a score of careless

errors. Careless errors were such mistakes as miscounting the number of letters, e.g., ”n · n · n · n ·

n = n4”, or an arithmetic error, e.g., “b · a · 4 · a · 4 · a = 62a3b”.

In addition, all problems the students worked on, regardless of whether they were solved

correctly, were analyzed according to the alphabetical-order convention and the number-one

convention. Alphabetical-ordering is a convention used to sort letters alphabetically, which

provides a better overview when there are many variables, e.g., “u2 · ax · u2 · u · ax = a2u5x2“, or “n + b +

n + x + b + n = 2b + 3n + x”. The number-one convention is an agreement that it is not necessary to

write the number "1" if there is a single letter, e.g., “z + n + n = 2n + z” and not “2n + 1z”, or “b · a · 4 · a ·

4 · a = 64a3b”, and not “64a3b1”. Therefore, for convention errors, we assessed the number of

answers with incorrect alphabetical-ordering as well as the number of answers with a

superfluously written number 1. These conventions were not explicitly taught during the training,

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 19

and mistakes involving them were not marked as errors in the algebra tests, because these

conventions are not central to understanding and distinguishing algebraic additions and

multiplications. Nevertheless, these conventions help to provide an overview when handling

complex algebraic expressions and thus are practical to be learned. Students could learn the

conventions incidentally while processing the work sheets, and the purpose of this analysis was

to determine the extent to which learners directed their attention to superficial characteristics

when focusing on structures. If the students did not pay attention to these two conventions, their

attention was sometimes directed to these points, however not in a systematic way. This

assistance was eliminated in Experiment 2 to make the convention-processing clearly incidental

(see section 4.1).

2.3.6. Algebraic transformation explanations: presented in follow-up sessions 1 – 3

In this test, students’ ability to elucidate how to apply algebraic additions and

multiplications was assessed. Students were asked to write down two separate descriptions

explaining how to solve each type of problem. For each explanation, they were prompted with

four hints, which were designed to help activate their knowledge: (a) "Describe in detailed steps

how problems with letters are solved", (b) "Mention what one has to pay attention to", (c) "You

can explain it by means of examples", and (d) "Imagine you would like to explain the rules to

classmates".

Scoring. The two written transformation explanations of algebraic addition and

multiplication were scored for accuracy and completeness of the answer. To do so, a coding

scheme was developed, which is depicted in the Appendix. Two measures were assessed: (a)

Explicit transformation knowledge was judged by the amount of correctly reported algebra

concept features. For both addition and multiplication, the features were divided into defining

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 20

features and secondary features, and the scores for each were added to the overall score of

explicit transformation knowledge. (b) Misconceptions were judged by the number of errors in

the transformation explanations including errors in the examples used in the explanations. The

errors of both the addition and multiplication explanations were added for a total score of

misconceptions.

Two trained raters independently coded the algebraic transformation explanations. The

raters discussed and unified diverging judgments together. Inter-rater reliability for coding the

answers was 93.8% for the first point of measurement, and 94.3% for the second point of

measurement (exact agreement). Because of the high reliability there was only one of the two

raters who coded the third point of measurement.

3. Experiment 1

The primary purpose was to compare two different methods of teaching students

introductory algebra by using a contrasted self-study program for one group and a sequential

self-study program for the second group. In the contrasted condition, addition and multiplication

problems were mixed and juxtaposed, whereas in the sequential condition, addition problems

were presented first, and multiplication problems were presented second. Students in the contrast

group were expected to outperform students in the sequential group on both algebraic

transformations (Hypothesis 1) and algebraic transformation explanations (Hypothesis 2). We

also expected the gains in learning to persist in the long run one week and three months later

(Hypothesis 3). Although we expected that the contrasted comparison program would especially

help learners construct meaningful and applicable algebra knowledge, we also assumed that it

would not affect their routines; therefore, we did not expect any differences between the numbers

of careless errors made by each group. To find differences in learning trajectories between the

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 21

two groups, we registered the solution rates achieved during the training sessions.

3.1. Participants

A total of 74 students (37 females) participated from three schools that had two parallel

classes in the sixth grade so that one class of each school could be randomly assigned to the

contrast and the other to the sequential group. Two students were excluded because they did not

finish the training. Each group comprised 36 students (contrast group: M = 12.9 years, SD = 0.5;

sequential group: M = 12.6 years, SD = 0.5). At the third point of measurement, three months

later and after the summer holidays, we were only able to retest 65 students (32 in the contrast

group and 33 in the sequential group) because the students had moved to secondary schooling

and were dispersed in different classes and school levels.

3.2. Results

The results are presented in three sections: the students' preconditions, the effect of group

on the follow-up tests, and the effect of group on immediate learning during the training. No

significant gender differences were observed on any of the immediate or follow-up measures.

3.2.1. Students’ preconditions: algebra pretest and control variables

Prior algebra knowledge. As expected, given the age of students chosen for the study,

there was no difference between groups on prior algebra knowledge, p = .96. A floor effect was

found, indicating that the students appeared to have almost no prior algebra knowledge before

the training (M = 1.9 out of 8, SD = 1.4). The few correctly solved tasks showed that the students

were solving almost only the algebraic addition problems with equal letters, similar to length

measures they knew from school (“a + a + a + a =”; “7b + 7b =”, and “xy + xy + xy + xy =”;

however, they appeared to have no intuition about solving algebra problems that went beyond

these simple additions.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 22

Control variables. The control variables included logical reasoning, arithmetical

knowledge, and grades in mathematics and German. A multivariate analysis of variance

(MANOVA) revealed no significant difference between groups, F(4, 67) = .44, p = .78, ns. None

of the separate univariate ANOVA tests showed a significant effect, all p > .53, indicating that

both the contrast group and sequential group did not differ in cognitive preconditions nor school

achievement.

3.2.2. Group differences on performance in the follow-up tests

Students in the contrast group performed better in most of the three main follow-up test

measures: (a) transformation knowledge, (b) explicit transformation knowledge, and (c)

misconceptions. Figure 3 illustrates means and standard errors across the three points of

measurement for both groups. For each of the follow-up test measures, separate mixed-factorial

ANOVAs were conducted with group as a between-subject factor (contrast versus sequential)

and time as a within-subject factor (T1: one day, T2: one week, T3: three months). When there

was a main effect of group, separate post-hoc comparisons were made for the three points of

measurements. Students in the contrast group were expected to perform better in all follow-up

test measures. After a main effect of time, post-hoc tests were conducted to determine how stable

the effects were over time.

Transformation knowledge. As expected, there was a main effect of group on the main

score of algebraic transformations, F(1, 63) = 3.35, p = .036, η2 = .05 in favor of the contrast

group. Post-hoc tests revealed a tendency of group differences at T1, t(70) = 1.61, p = .057, d =

.33, and a significant group difference at T2, t(70) = 2.26, p = .014, d = .53, whereas at T3, the

difference between the groups was no longer significant at p = .127. These results indicate the

advantage of contrast learning, although the effect was only weak-to-moderate and disappeared

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 23

three months later. There was also a main effect of time, F(2, 126) = 60.44, p < .001, η2 = .49,

and no interaction. Simple contrasts revealed a significant change from T1 to T3, F(1, 63) =

69.40, p < .001, η2 = .52, yet not from T1 to T2, p = .053, showing that the effects remained

stable for one week and then declined. This decrease in performance after several weeks during

which material is not repeated is normal and expected in learning environments. Again, as

expected for careless errors, there was neither a group (p = .66), a time (p = .45), nor an

interaction effect (p = .33) concerning the amount of careless errors, indicating that careless

errors were independent of group and intervention.

Transformation knowledge: additional analysis of conventions. As previously

described, there were two types of conventions that were not taught explicitly but were assessed

in the learning analysis: the alphabetical-order convention and the number-one convention. We

did not formulate hypotheses concerning these two conventions because they were not explicitly

taught, and we only decided to analyze them later. For both conventions, there was no main

effect of group, p = .64 and p = .28 (see Table 3 for the means and standard deviations). For the

number-one convention, there was also no time effect or interaction; however, there was a main

effect of time for the alphabetical-order convention, F(1.79, 112.73) = 24.79, p < .001, η2 = .28,

that was opposite of what we expected, but with no interaction. Simple contrasts revealed a

significant increase at T2, F(1, 63) = 6.96, p = .011, η2 = .10, but a reduction at T3, F(1, 63) =

18.67, p < .001, η2 = .23, indicating an improvement for both groups at T3. These deviant time

effects indicate that over the three months between the first and last follow-up test, the

conventions were remembered and followed better over time.

Explicit transformation knowledge. Unexpectedly, there was no significant main effect

of group in the main score of the algebraic transformation explanations, p = .39 (see Figure 3);

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 24

however, there was a main effect of time, F(2, 124) = 3.47, p = .034, η2 = .05. Simple contrasts

showed a decrease from T1 to T2, F(1, 62) = 4.33, p = .041, η2 = .07, and from T1 to T3, F(1,

62) = 5.42, p = .023, η2 = .08, indicating that the algebra concept knowledge initially acquired

was not sustained. There was no interaction, showing that both groups’ scores decreased equally.

Misconceptions. As expected, there was a main effect of group in the misconceptions,

F(1, 62) = 8.80, p = .002, η2 = .12 (see Figure 3). Post-hoc tests revealed tendencies of group

differences in favor of the contrast group at T1, t(70) = 1.51, p = .065, d = .25 and at T2, t(70) =

3.05, p = .002, d = .50, and a clear difference three months after the training at T3, t(70) = 4.02, p

< .001, d = .66. There was a main effect of time, F(1.79, 110.80) = 11.81, p < .001, η2 = .16, with

the simple contrasts revealing a significant increase of errors from T1 to T3, F(1,62) = 16.68, p <

.001, η2 = .21, and a significant interaction, F(1.79, 110.80) = 3.38, p = .043, η2 = .05, showing

that the increase in the number of misconceptions during this time period was more pronounced

in the sequential group.

In summary, students in the contrast group showed better results in algebraic

transformations than the sequential group, with clear gains on the follow-up tests one day and

one week later, but there were no longer differences in sustained knowledge three months later.

The students in the contrast group also did not perform better in the section of written

explanations. Conversely, the contrast group’s lower score on misconceptions in the long-term

indicates better knowledge about both algebraic addition and multiplication.

3.2.3. Group differences on performance during the training

The previous section revealed a clear superiority of the contrast group on most of the

follow-up test measures. The question now arises: was this result predictable from the students’

achievement during the training, or does it only appear delayed? By analyzing group differences

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 25

in the two immediate learning tests presented during the training, we find a reversed effect with a

better performance of the sequential group.

Immediate learning gains. A multivariate analysis of variance (MANOVA) revealed a

significant effect of group, F(2, 69) = 9.27, p < .001, η2 = .21. Surprisingly, students in the

sequential group outperformed students in the contrast group on both immediate learning

measures (see Figure 4 for the means and standard errors). The separate univariate ANOVA tests

on the outcome variables revealed significant treatment effects on repetition tests, F(1, 70) =

5.15, p = .026, η2 = .07, and on learning tests, F(1, 70) = 18.80, p < .001, η2 = .21. Therefore,

with regard to short-term learning, the results show a clear advantage of the sequential group.

3.2.4. Does general reasoning ability moderate learning gains?

We wanted to test whether higher or lower-achieving students benefited more from the

contrasted introduction. The results were examined for the influence of intelligence (logical

reasoning ability) by contrasting the results of the upper half with the lower half of the

participants. We conducted a 2 (group: contrast versus sequential) x 2 (ability: high versus low) x

3 (time: one day, one week, three months) ANOVA on the follow-up test measures.

No significant interaction was found for any of the five follow-up measures:

transformation knowledge (p = .76), explicit transformation knowledge (p = .41), misconceptions

(p = .22), the alphabetical-order convention (p = .60), and the number-one convention (p = .96).

With respect to the two immediate learning measures, no interaction was found for the repetition

tests was non-significant (p = .30). However for the learning test the interaction was significant

(p = .008), with the graphs revealing that the low-achieving contrast learners performed

especially poor. Checking the means of the learning test revealed high solution rates for the

sequential learners (96.7% for the high-achievers, 93.2% for the low-achievers), implicating

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 26

limited variance caused by ceiling effects. More importantly, however, the low-achieving

contrast learners entirely compensated their disadvantage in the follow-up tests. Altogether, there

is no evidence from our data set that the extent to which students gain from comparison

instruction depends on the individual characteristics of logical reasoning.

3.3. Discussion of Experiment 1

The results from Experiment 1 showed that the contrasted training led to greater gains in

algebraic transformation knowledge compared to the sequential training, with the strongest effect

found one week after the training (Hypothesis 1). Three months after training, however, this

effect disappeared, although the means remained higher in the contrast group (Hypothesis 3).

The test conditions at this third point of measurement (after summer holidays and dispersed in

different classes) were not as controlled as they were for the first two follow-up tests. The fact

that the contrast group remained slightly better at this third point of measurement was a hint that

there might be a long-term advantage for the contrasted learning.

Students in the contrast group also showed fewer misconceptions in the written

explanations than students in the sequential group; however, there was no group difference in the

total score of explicit transformation knowledge (Hypothesis 2). During the testing, it became

obvious that many students did not like writing down the written explanations and thus did not

write them in very much detail. This result may show that students have more explicit knowledge

than they are willing or able to express. Time restrictions and insufficient instruction might have

been responsible for the fact that students’ written explanations were not as informative and

complete as they could have been. Moreover, writing explanations may be hard for students, as

knowledge is often intuitive and only implicitly represented, which makes it difficult to

formalize (Bou-Llusar & Segarra-Ciprés, 2006; von Aufschnaiter & Rogge, 2010). Therefore, it

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 27

is very important to direct students to express their knowledge and to prompt them appropriately.

For our next experiment, we plan to reject minimally written explanations and to request more

detailed descriptions to examine this question. We also decided to give students more time

during the follow-up tests so they could write their explanations more carefully and have time to

review the test after finishing it.

A remarkable finding of Experiment 1 was that the contrast group, which outperformed

the sequential group in transformation knowledge and misconceptions, showed clearly worse

results in immediate learning during the four days of training. This pattern of delayed benefits

corresponds to retention effects found for intermixed concept learning (e.g., Rohrer & Pashler,

2010). Good performance during training often leads teachers to mistakenly conclude that

sequential learning is favorable; however, with regard to learning and instruction, knowledge

acquisition must be considered on a long-term basis. The fewer errors made by sequential

learners revealed that the training was less challenging for them and that they did not need to

invest as much effort to learn the material as the contrast learners did. We also sensed through

casual observation that more sequential learners finished their work sheets earlier, although we

did not assess this phenomenon systematically. Therefore, the facility of learning in the training

phase appears to be a poor indicator of long-term learning gains. The fact that the contrast

students had higher error rates may indicate that some errors must be made for meaningful

learning to occur.

In the trial tasks and the repetition tests, all the students received a “right” or “wrong”

feedback on the problems they completed, and they had to correct the problems that had been

marked “wrong”. Corrections had to be made more often by the contrast learners because they

made more errors. This feedback and the resulting corrections may have helped them deepen

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 28

their processing, and allowed them to consciously distinguish between the two algebra

principles. Feedback is seen as an important means of instruction (e.g., Hattie & Timperley,

2007). Although our feedback was given as a simple mark of “right” or “wrong”, it entailed

revising the incorrect problems and we cannot exclude that it contributed to the effect of

contrasted learning.

A limitation in the sampling procedure that may have contributed to the moderate effects

of Experiment 1 was that the classes were matched as a whole to the same group. Although there

were no group differences in the control variables, some classes were more disciplined and

motivated to achieve, whereas others needed more guidance and control. This difference may

have been due to different styles of the teachers, but we did not systematically assess this

phenomenon. Despite the limitations discussed above, the findings of the first experiment

supported the contrasted training as a promising introductory method for teaching two similar

principles in algebra with long-term benefits. Hence, we decided to replicate the first experiment

under improved conditions and a controlled, within-class matching to the groups to see if the

results would be the same and highlight the superiority of a contrasted and mixed training.

4. Experiment 2

The two-group experimental design, the hypotheses, the training materials, and the tests

were the same as in Experiment 1 (see sections 2.2, 2.3 and 3), whereas the way of assigning the

participants to the groups and some details of the procedure were changed.

4.1. Procedure

In Experiment 2, there were slight differences in the procedure and in the instruction.

Assignment of the participants to the groups. In Experiment 1, assignment to the two

conditions was performed using entire classes. To avoid selection effects, a within-classroom

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 29

assignment was used to assign groups in Experiment 2. For each class, half the students were

assigned to the contrast group, and the other half were assigned to the sequential group. As in

Experiment 1, the students were matched according to their grades in mathematics and German.

Time allocated to the follow-up tests. Students were given an extra 15 minutes on the

follow-up tests, i.e., 60 minutes instead of 45 minutes, so they could work through the test

without time pressure, review it and make corrections.

Time between the second and third presentation of the follow-up test. The third follow-

up test was conducted 10 weeks after the training instead of 3 months, so that the test would not

interfere with the summer holiday.

Follow-up test instructions. To ensure the thorough processing of the materials and to

improve the reliability of the tests, more detailed and precise instructions were provided. First,

students were instructed to write down the intermediate steps to the solutions of the algebraic

transformation problems, which only some of the students did in Experiment 1. Second, after

finishing the transformation test, students were instructed to review all their answers for omitted

problems and careless errors because some students skipped a problem or an entire sheet in

Experiment 1. Third, if the algebraic transformation explanations did not have enough detail,

students were asked to expand their answers.

Elimination of assistance. In Experiment 1, students were sometimes given help when

they had difficulties. For instance, with respect to self-explanations, students were helped when

the experimenter directed their attention to the corresponding worked examples and tell them to

describe what to do first, second, and third. In addition, concerning the trial tasks, some students

were given hints to pay attention to the alphabetical-ordering convention and the number-one

convention. Again, this help was only given to some students and not provided in a controlled

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 30

way. The assistance was never given during the assessment in the follow-up testing. In

Experiment 2, we decided to eliminate this assistance: students were not given any hints to notice

conventions or to describe the steps needed to reach a solution.

For self-explanations, the feedback was standardized to the following: (a) “This is

incorrect, look at this worked example” or (b) “This is not enough detail, can you supplement

it?” For the trial tasks, the feedback was standardized to the following: (a) “This is incorrect,

check it again” and (b) “This is incorrect, look at this worked example”.

4.2. Participants

A total of 157 sixth graders (84 females) from six schools participated. The participation

conditions were the same as in Experiment 1. Three students did not finish the training and were

excluded. For each class, students were first assigned randomly to the two groups. If there was

an imbalance in the students’ grades in German and Mathematics, the assignment of the students

was aligned. The contrast group comprised 79 students (M = 12.4 years, SD = 0.5), and the

sequential group comprised 75 students (M = 12.3 years, SD = 0.5).

4.3. Results

As in Experiment 1, the results are presented in three sections. Again, no significant

gender differences were observed on any of the immediate or follow-up measures.

4.3.1. Students’ preconditions: algebra pretest and control variables

Prior algebra knowledge. There was no difference between groups concerning prior

algebra knowledge, p = .97. Compared to Experiment 1, students' mean solution rates on the

pretest were even lower (M = 0.8 out of 8, SD = 1.0). Only 23% of the students solved more than

one task correctly, and none of the students solved more than three tasks. This finding strongly

confirmed that the students had negligible direct prior algebra knowledge and no intuition about

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 31

how to solve algebra problems.

Control variables. There was no significant difference between the two groups in their

logical reasoning, arithmetical knowledge, and grades in mathematics and German, F(4, 149) =

.44, p = .78, ns. None of the separate univariate ANOVA tests showed a significant effect, all p

> .48.

4.3.2. Group differences on performance in the follow-up tests

Students in the contrast group performed better in most of the three main follow-up test

measures: (a) transformation knowledge, (b) explicit transformation knowledge, and (c)

misconceptions. Figure 5 illustrates means and standard errors across the three points of

measurement for both groups. As expected, the results of Experiment 1 were replicated with

improved significance levels. The same statistical analyses used in Experiment 1 were used in

Experiment 2.

Transformation knowledge. As expected, there was a main effect group, F(1, 151) =

15.59, p < .001, η2 = .09, in favor of the contrast group. Post-hoc tests revealed significant group

differences at all points of measurements, at T1, t(152) = 2.79, p = .003, d = .46, at T2, t(152) =

3.08, p = .002, d = .50, and at T3, t(151) = 4.65, p < .001, d = .76, indicating an advantage of

contrast learning with moderate-to-strong effects. There was also a main effect of time, F(1.38,

207.89) = 78.48, p < .001, η2 = .34, and a significant interaction, F(1.38, 207.89) = 7.24, p =

.003, η2 = .05. Simple contrasts also showed a decline of the results over time from T1 to T2,

F(1, 151) = 4.49, p = .036, η2 = .03, and from T1 to T3, F(1, 151) = 91.84, p < .001, η2 = .38,

although the significant interaction effect shows that the decrease over time is more pronounced

for the sequential group than for the contrast group. As expected, there was no difference

between groups in the number of careless errors, p = .52.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 32

Transformation knowledge: additional analysis of conventions. Based on the results

from Experiment 1, we did not expect any differences in the convention errors. Surprisingly,

however, the contrast group performed worse on these measures than the sequential group (see

Table 4 for the means and standard deviations). There was a main effect of group for

alphabetical ordering, F(1, 151) = 19.56, p < .001, η2 = .12, with significant post-hoc tests at all

points of measurements, at T1, t(152) = 2.64, p = .009, d = .43, at T2, t(152) = 3.50, p = .001, d =

.57, and at T3, t(151) = 5.25, p < .001, d = .86. There was a time effect, F(1.64, 248.29) = 33.82,

p < .001, η2 = .18, with the simple contrasts showing a reduction in errors at T3 for both groups,

F(1,151) = 34.88, p < .001, η2 = .19. There was also an interaction of Group x Time, F(1.64,

248.29) = 6.35, p = .004, η2 = .04, that indicated a less pronounced decline for the contrast group

at T3. For the number-one convention, the contrast group was slightly worse on all points of

measurement; however, the effect was not significant, p = .13. Only the time effect was found to

be significant, F(1.76, 265.77) = 12.77, p < .001, η2 = .08, with the simple contrasts showing a

decline of errors at T3 for both groups, F(1,151) = 14.58, p < .001, η2 = .09. The results suggest

that the sequential group performed better in maintaining the two conventions, which had not

been explicitly taught.

Explicit transformation knowledge. Different from Experiment 1, there was a significant

main effect of group, F(1, 151) = 12.62, p = .001, η2 = .08 (see Figure 5). Significant post-hoc

tests at all points of measurement showed that students from the contrast group expressed more

explicit knowledge in their written explanations than did students from the sequential group, at

T1, t(152) = 2.77, p = .003, d = .45, at T2, t(152) = 3.01, p = .002, d = .49, and at T3, t(151) =

3.20, p = .001, d = .52. Unlike Experiment 1, there was no time effect and no interaction,

indicating that the explicit transformation knowledge was sustained over time in both groups.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 33

Misconceptions. Similar to Experiment 1, the contrast group had fewer misconceptions

than the sequential group F(1, 151) = 17.11, p < .001, η2 = .10 (see Figure 5). Post-hoc tests

revealed significant group differences at T2, t(70) = 3.05, p = .002, d = .50, and at T3, t(70) =

4.02, p < .001, d = .66. At T1, there was only a tendency, t(70) = 1.51, p = .067. There was a

time effect, F(1.83, 276.16) = 17.68, p < .001, η2 = .11, with the simple contrasts showing an

increase from T1 to T3, F(1,151) = 27.14, p < .001, η2 = .15, and an interaction of Group x Time,

F(1.83, 276.16) = 5.46, p = .006, η2 = .04. The graph revealed that the increase of

misconceptions was more pronounced in the sequential group.

4.3.3. Group differences during the training

Experiment 1 revealed superior performance of the sequential group in the immediate

tests presented during the training. We were interested in determining whether this finding would

be replicated in Experiment 2.

Immediate learning gains. In accordance with Experiment 1 the MANOVA revealed a

significant effect of group on immediate learning gains, F(2, 151) = 27.75, p < .001, η2 = .27, in

favor of the sequential group (see Figure 6). Separate univariate ANOVA tests on the outcome

variables revealed significant group effects on both variables: on repetition tests, F(1, 152) =

31.84, p < .001, η2 = .17, and on learning tests, F(1, 152) = 52.26, p < .001, η2 = .26, indicating a

clear advantage for the sequential group on the short-term learning outcome.

4.3.4. Does general reasoning ability moderate learning gains?

Similar to Experiment 1, the examination for the influence of logical reasoning ability by

median split showed no significant treatment effects for all the follow-up measures:

transformation knowledge, (p = .28), explicit transformation knowledge (p = .81) ,

misconceptions (p = .21), the alphabetical-order convention (p = .35), and the number-one

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 34

convention (p = .85). Also the pattern of the immediate learning measures was similar to

Experiment 1: no significant interaction was found for the repetition tests (p = .83), while the

interaction for the learning test (p = .014) was significant. The graphs again revealed

disadvantages for the low-achieving contrast learners which, however, were compensated in the

follow-up. The solutions rates of the learning tests were again high for the sequential learners

(94.5% for the high-achievers, 92.6% for the low-achievers) which underline potential ceiling

effects. Altogether, also the results of Experiment 2 suggest that students of all abilities benefited

more from the contrasted than from the sequential exposure to material.

4.4. Summary of Experiment 2

Students in the contrast group again performed better on transformation knowledge with

a strong effect on the ten-week follow-up test (Hypothesis 1). After the more consequent

instruction, the two not-explicitly-taught conventions were better perceived by sequential

exposure to the material. Concerning explicit transformation knowledge, under the improved

conditions for more detailed transformation explanations in Experiment 2, students in the

contrast group clearly performed better with significant differences at all measurement points

(Hypotheses 2 and 3). Most remarkably, students in the contrast group again scored clearly lower

in the two immediate learning measures. Therefore, we may conclude that, as expected, the

results of Experiment 2 were essentially the same as Experiment 1, but with clearer results.

5. General Discussion

The aim of this study was to investigate the impact of repeated contrasted comparisons on

algebra learning in a classroom setting. In two experimental training studies, we showed that

juxtaposing addition and multiplication algebra problems in a self-learning program is a

promising method for supporting early secondary school students in learning elementary

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 35

principles of algebraic transformations. Both experiments revealed that the students who

processed the contrasted, mixed program performed better in differentiating superficially similar

algebra principles than the students who received the more conventional sequential teaching

materials (Hypotheses 1 and 2). There was a significant decline in performance after several

weeks, but this finding was expected, as learning was not continued. Nevertheless, the

persistence of the group difference, despite the decline in performance with its strong effect over

ten weeks, is important (Hypothesis 3). These results strengthen the robustness of previous

findings regarding the comparison effect on gaining problem solving knowledge (Rittle-Johnson

& Star, 2007); however, both experiments revealed that the superiority of the contrasted learning

program could not be predicted from the students’ performance during the training. Rather, in

both experiments, the contrast group made more errors on the immediate tests compared to

sequential group.

Some of the limitations of Experiment 1 were overcome in Experiment 2 by randomizing

the participants on an individual level rather than on a classroom level and by optimizing the

instructions and the timing of the follow-up tests. As a consequence, although the findings were

similar in both experiments, they were more pronounced in Experiment 2. This finding

especially applies to explicit transformation knowledge, with clear gains found for contrast

learners, who concurrently perform better on all three follow-up measures. This finding may

confirm that students in Experiment 1 did not write all they knew in the written explanations;

therefore, in Experiment 2, when given more time and prompted to write in greater detail,

students appear to express more of their knowledge – an important point to consider when

designing tests.

Our findings of delayed benefits for contrasted mixed learning represent a confirmation

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 36

of previous results of mixed practice, which has been demonstrated with categorization tasks or

mathematical rule and formula learning (e.g., Kang & Pashler, 2012). We have contributed with

an experimentally controlled intervention study, which was run under realistic classroom

conditions over several days and included a ten-week follow-up test. Our results have

highlighted misleading discrepancies between short-term and long-term results; i.e., the short-

term benefit and the long-term disadvantage of the conventional sequential introduction of

algebraic addition and multiplication. This finding emphasizes the importance of follow-up

studies investigating long-term learning gains instead of relying on immediate learning gains.

Moreover, it reveals the importance of organizing and assessing learning from a long-term

perspective.

Another benefit of contrasted learning may be splitting the learner’s attention between

addition and multiplication from the very beginning. The switching between concepts and

procedures helps the learner become aware of the existence of the two concepts that must be

distinguished. This effect may lead to the learner’s making more conscious choices when using

memorized procedures and may reduce the random, overly automatic retrieval of memorized

procedures when problem solving. The higher number of errors observed in the contrast group

during the training period may be a concern for teachers who do not like to overburden students,

especially lower-achieving students; however, our results revealed that even though contrasting

is a more demanding instruction method, the added challenge also appears to provide an

advantage.

In our data, there was no indication that either lower- or higher-ability students benefited

more from the contrast materials. These results confirm the lack of evidence for aptitude-

treatment interactions, which are scarcely found despite the increasing interest in individual

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 37

differences (Scott, 2013). Studies have established the interactions of learning gains in a field

with prior knowledge in this field (Day & Goldstone, 2010; Hiebert & Wearne, 1996; Rittle-

Johnson, Durkin, & Star, 2009). For example, Rittle-Johnson and colleagues found that

comparing is only beneficial if students have had some prior knowledge on a topic. In contrast,

our participants performed very poorly on prior algebra knowledge; they appeared to have

negligible prior knowledge and no intuitive algebra comprehension. The algebra material in our

experiments was new for all students, and contrasting was their first contact with the new topic;

therefore, we can recommend the practice of comparing and contrasting from the beginning of a

student’s training.

The additional analyses of students’ transformation errors suggested that the two training

programs shaped learners’ attention in different ways. Particularly in Experiment 2, students

from the sequential group adopted the not-explicitly-taught number-one and alphabetical-order

conventions better. Learners of both groups could incidentally deduce these conventions from

the worked examples, but the sequential learners made more use of this information. This finding

suggests that sequential students had unused capacities and were better able to process these

secondary superficial characteristics of the material, although these capacities may have

distracted students from completely learning the more crucial core features for distinguishing the

two concepts over an extended period of time.

Together with the aforementioned better performance in immediate learning, it appears

that sequential learning is less demanding. This finding also corresponds with the results of

reduced intrinsic cognitive load in sequential learning (Sweller & Chandler, 1994; Sweller, Van

Merrienboer, & Paas, 1998) and with the demand of desirable difficulties (E. Bjork & Bjork,

2011; Kester, Paas, & Van Merriënboer, 2010). Desirable difficulties are those that increase the

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 38

intrinsic cognitive load or complexity of materials in such a way that it helps learners to focus

their attention on critical aspects of the material. The principles of algebraic addition and

multiplication are perceptually very similar, comparable to near-miss contrasts (Gick & Paterson,

1992; Winston, 1975). Desirable difficulties in our algebra material are the difficulties that help

students to distinguish between the two operations and the corresponding rules.

The superiority of the contrasted practice compared to the sequential practice was

surprisingly clear. We ascribe the clear learning gains to the explicit comparisons the contrast

learners were prompted for. The juxtaposition may have highlighted the underlying core

principles (Gick & Paterson, 1992; Loewenstein & Gentner, 2001; Mitchell et al., 2008),

whereas the explicit comparisons may have forced students to connect the two similar principles

and to become aware of the features and rules that help to distinguish them (Sweller & Chandler,

1994). Kang and Pashler (2012) investigated the difference between the interleaved and

simultaneous presentation of different concepts. Their simultaneous group showed slightly,

though not significantly, better results than their interleaved group. In our instruction, not only

were algebra problems processed simultaneously, but students were explicitly prompted to

compare and describe addition and multiplication differences, which may have been the main

reason for the strong contrast effect. Our design does not allow the full disentanglement of the

impact of explicit comparison from the impact of desirable difficulties caused by mixed

presentation. Therefore, we cannot exclude that the effect may be due entirely to desirable

difficulty. By including a condition with mixed problem presentations, which does not explicitly

request comparisons, we could have tested whether solely simultaneous processing of addition

and multiplication problems would have caused learning gains. However, it was not the goal of

this study to demonstrate the efficacy of comparison, as this has already been done extensively in

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 39

many other studies. In our study, the prompts to compare for contrast learners were implemented

to ensure the deep processing of the materials. Similarly, the sequential learners were prompted

for explicit and thorough explanations of their sequential materials. The instruction “to compare”

and “to explain in detail” had the same intention: to guarantee deep processing.

5.1. Limitations and Future Directions

Contrast learners showed clear gains in the ability to apply transformations and in the

explicit availability of transformation knowledge. Because of the already loaded design, we had

no additional measure to assess conceptual understanding, e.g., as questions about the principles,

or ratings of definitions (Rittle-Johnson & Star, 2007). Nevertheless, we assert that the ability to

distinguish and correctly apply the two algebra principles of addition and multiplication is a clear

sign of understanding the syntactic structure of algebra. This seeing of the structure is crucial for

algebra learners and forms the basis for a later flexible use in applications (Kirshner & Awtry,

2004; Ottmar et al., 2012). Further research should differentiate whether and how the thorough

distinction of the two principles from the beginning will influence a broader understanding of

algebra.

Further controlled studies are needed to replicate the present results with other topics

before deciding whether text books should be changed so they introduce similar concepts in a

contrasted format. We intentionally restricted the investigation to algebraic additions and

multiplications and did not include subtractions and divisions so as not to overload the

experimental setting, but we would expect subtraction and division to be analogous to addition

and multiplication, as they are each operations of the same level (Kirshner & Awtry, 2004). This

phenomenon should be tested in a classroom setting. We also suggest the further examination of

contrasted comparisons with many algebraic and mathematical topics, an area where we would

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 40

expect broad application possibilities. It is especially important to investigate all characteristics

of compared examples in greater depth.

In our experiments, contrasting was applied in a self-learning program; however, a very

common instruction technique is direct instruction. Therefore, it would be interesting to

investigate a learning environment in which students receive direct instruction with a

simultaneous and contrasted introduction of two concepts, i.e., a contrasted direct instruction

compared to a sequential direct instruction. Only further research will show whether our results

are an algebra-specific or a method-specific effect and whether contrasting may be established as

general method of instruction for introducing similar and frequently confused concepts.

Acknowledgements

We are grateful to Sara Ziegler for her assistance in the implementation of the

Experiments in the school classes and the evaluation of the test materials. We thank Theresa

Treasure for helpful comments on earlier versions of this paper.

Parts of this article report findings from the author’s unpublished doctoral dissertation

accepted at the ETH of Zurich, Switzerland.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 41

Appendix

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 42

References

Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University

Press.

Bjork, E., & Bjork, R. (2011). Making things hard on yourself, but in a good way: Creating

desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M.

Hough & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating

fundamental contributions to society (pp. 56-64). New York: Worth Publishers.

Bjork, R. (1994). Institutional impediments to effective training. In D. Druckman & R. A. Bjork

(Eds.), Learning, remembering, believing: Enhancing human performance (pp. 295-306).

Washington, DC: National Academy Press.

Blume, G. W., & Heckman, D. S. (2000). Algebra and functions. In E. A. Silver & P. A. Kenney

(Eds.), Results from the seventh mathematics assessment (pp. 225-277). Reston, VA:

National Council of Teachers of Mathematics.

Booth, L. R. (1988). Children's difficulties in beginning algebra. In A. F. Coxford & A. P. Shulte

(Eds.), The ideas of algebra, K-12 (pp. 20-32). Reston, Virginia: 1988 NCTM Yearbook,

National Council of Teachers of Mathematics.

Bou-Llusar, J. C., & Segarra-Ciprés, M. (2006). Strategic knowledge transfer and its

implications for competitive advantage: An integrative conceptual framework. Journal of

knowledge management, 10(4), 100-112. doi: 10.1108/13673270610679390

Catrambone, R., & Holyoak, K. J. (1989). Overcoming contextual limitations on problem-

solving transfer. Journal of Experimental Psychology-Learning Memory and Cognition,

15(6), 1147-1156. doi: 10.1037/0278-7393.15.6.1147

Chase, C. C., Shemwell, J. T., & Schwartz, D. L. (2010). Explaining across contrasting cases for

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 43

deep understanding in science: An example using interactive simulations. Paper

presented at the Proceedings of the 9th International Conference of the Learning

Sciences.

Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating

inferences and repairing mental models. In R. Glaser (Ed.), Advances in instructional

psychology: Educational design and cognitive science (Vol. 5, pp. 161-238). Mahwah,

NJ: Lawrence Erlbaum Associates, Inc.

Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics

problems by experts and novices. Cognitive Science, 5(2), 121-152.

Day, S., & Goldstone, R. L. (2010). The effects of similarity and individual differences on

comparison and transfer. Paper presented at the Thirty-Second Annual Conference of the

Cognitive Science Society, Portland, Oregon: Cognitive Science Society.

Durkin, K., & Rittle-Johnson, B. (2012). The effectiveness of using incorrect examples to

support learning about decimal magnitude. Learning and Instruction, 22(3), 206-214. doi:

10.1016/j.learninstruc.2011.11.001

Gentner, D. (1983). Structure-Mapping - A theoretical framework for analogy. Cognitive

Science, 7(2), 155-170. doi: 10.1016/S0364-0213(83)80009-3

Gentner, D. (2010). Bootstrapping the mind: Analogical processes and symbol systems.

Cognitive Science, 34(5), 752-775. doi: 10.1111/j.1551-6709.2010.01114.x

Gentner, D., Loewenstein, J., & Thompson, L. (2003). Learning and transfer: A general role for

analogical encoding. Journal of Educational Psychology, 95(2), 393-408. doi:

10.1037/0022-0663.95.2.393

Gentner, D., & Namy, L. L. (1999). Comparison in the development of categories. Cognitive

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 44

Development, 14(4), 487-513. doi: 10.1016/S0885-2014(99)00016-7

Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive

Psychology, 15(1), 1-38. doi: 10.1016/0010-0285(83)90002-6

Gick, M. L., & Paterson, K. (1992). Do contrasting examples facilitate schema acquisition and

analogical transfer? Canadian Journal of Psychology, 46(4), 539–550. doi:

10.1037/h0084333

Goldstone, R. L., Landy, D. H., & Son, J. Y. (2010). The education of perception. Topics in

Cognitive Science, 2(2), 265-284. doi: 10.1111/j.1756-8765.2009.01055.x

Hattie, J. A., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research,

77(1), 81-112. doi: 10.3102/003465430298487

Hattikudur, S., & Alibali, M. W. (2010). Learning about the equal sign: Does comparing with

inequality symbols help? Journal of Experimental Child Psychology, 107(1), 15-30. doi:

10.1016/j.jecp.2010.03.004

Hiebert, J., & Wearne, D. (1996). Instruction, understanding, and skill in multidigit addition and

subtraction. Cognition and Instruction, 14(3), 251-283. doi: 10.1207/s1532690xci1403_1

Hirsch, L. R., & Goodman, A. (2006). Understanding elementary algebra with geometry: A

course for college students (6 ed.). Pacific Grove, CA: Brooks Cole, Thompson Learning.

Holyoak, K. J. (2005). Analogy. In K. J. Holyoak & R. G. Morrison (Eds.), The cambridge

handbook of thinking and reasoning (pp. 117-142). Cambridge, UK: Cambridge

University Press.

Horn, W. (1983). L-P-S Leistungsprüfsystem. 2. Auflage. Göttingen: Hogrefe.

Kamii, C., & Dominick, A. (1997). To teach or not to teach algorithms. Journal of Mathematical

Behavior, 16(1), 51-61. doi: 10.1016/S0732-3123(97)90007-9

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 45

Kang, S. H. K., & Pashler, H. (2012). Learning painting styles: Spacing is advantageous when it

promotes discriminative contrast. Applied Cognitive Psychology, 26(1), 97-103. doi:

10.1002/acp.1801

Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the

Learning Sciences, 21(1), 45-83. doi: 10.1080/10508406.2011.591717

Kester, L., Paas, F., & Van Merriënboer, J. J. G. (2010). Instructional control of cognitive load in

the design of complex learning environments. In J. L. Plass, R. Moreno & R. Brunken

(Eds.), Cognitive load theory (pp. 109-130). Cambridge: Cambridge University Press.

Kieran, C. (1992). The learning and teaching of school algebra. In D. A. Grouws (Ed.),

Handbook of research on mathematics teaching and learning (pp. 390-419). New York:

Macmillan Publishing Company.

Kieran, C. (2004). The core of algebra: Reflections on its main activities. In K. Stacey, H. Chick

& M. Kendal (Eds.), The future of the teaching and learning of algebra: The 12th ICMI

study (pp. 21-34). Dordrecht, The Netherlands: Kluwer.

Kirshner, D., & Awtry, T. (2004). Visual salience of algebraic transformations. Journal for

Research in Mathematics Education, 35(4), 224-257. doi: 10.2307/30034809

Kornell, N., & Bjork, R. (2008). Learning concepts and categories: Is spacing the “enemy of

induction”? Psychological Science, 19(6), 585-592. doi: 10.1111/j.1467-

9280.2008.02127.x

Kurtz, K. J., Miao, C. H., & Gentner, D. (2001). Learning by analogical bootstrapping. Journal

of the Learning Sciences, 10(4), 417-446. doi: 10.1207/S15327809JLS1004new_2

Loewenstein, J., & Gentner, D. (2001). Spatial mapping in preschoolers: Close comparisons

facilitate far mappings. Journal of Cognition and Development, 2(2), 189-219. doi:

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 46

10.1207/S15327647JCD0202_4

Markman, A. B., & Gentner, D. (1993). Splitting the differences - A structural alignment view of

similarity. Journal of Memory and Language, 32(4), 517-535. doi:

10.1006/jmla.1993.1027

Mason, L. (2004). Fostering understanding by structural alignment as a route to analogical

learning. Instructional Science, 32(4), 293-318. doi:

10.1023/B:TRUC.0000026512.88700.32

Matz, M. (1982). Towards a process model for high school algebra errors. In D. Sleeman & J. S.

Brown (Eds.), Intelligent tutoring systems (pp. 25-50). London: Academic Press.

McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The

sequential learning problem. The Psychology of Learning and Motivation, 24, 109-165.

Mitchell, C., Nash, S., & Hall, G. (2008). The intermixed-blocked effect in human perceptual

learning is not the consequence of trial spacing. Journal of Experimental Psychology:

Learning, Memory, and Cognition, 34(1), 237-242. doi: 10.1037/0278-7393.34.1.237

NCTM. (2000). Principles and standards for school mathematics: Reston, VA: National Council

of Teachers of Mathematics.

Ottmar, E., Landy, D., & Goldstone, R. L. (2012). Teaching the perceptual structure of algebraic

expressions: Preliminary findings from the pushing symbols intervention. Paper presented

at the Thirty- Fourth Annual Conference of the Cognitive Science Society, Sapporo,

Japan.

Renkl, A. (1997). Learning from worked-out examples: A study on individual differences.

Cognitive Science, 21(1), 1-29. doi: 10.1016/S0364-0213(99)80017-2

Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 47

explanations. Learning and Instruction, 12(5), 529-556. doi: 10.1016/S0959-

4752(01)00030-5

Richland, L. E., Morrison, R. G., & Holyoak, K. J. (2006). Children's development of analogical

reasoning: Insights from scene analogy problems. Journal of Experimental Child

Psychology, 94(3), 249-273. doi: 10.1016/j.jecp.2006.02.002

Rittle-Johnson, B. (2006). Promoting transfer: Effects of self-explanation and direct instruction.

Child Development, 77(1), 1-15. doi: 10.1111/j.1467-8624.2006.00852.x

Rittle-Johnson, B., Durkin, K., & Star, J. R. (2009). The importance of prior knowledge when

comparing examples: Influences on conceptual and procedural knowledge of equation

solving. Journal of Educational Psychology, 101(4), 836-852. doi: 10.1037/a0016026

Rittle-Johnson, B., & Star, J. R. (2007). Does comparing solution methods facilitate conceptual

and procedural, knowledge? An experimental study on learning to solve equations.

Journal of Educational Psychology, 99(3), 561-574. doi: 10.1037/0022-0663.99.3.561

Rittle-Johnson, B., & Star, J. R. (2011). The power of comparison in learning and instruction:

Learning outcomes supported by different types of comparisons. Psychology of Learning

and Motivation: Cognition in Education, 55, 199-225. doi: 10.1016/B978-0-12-387691-

1.00007-7

Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges conventional

instructional strategies. Educational Researcher, 39(5), 406-412. doi:

10.3102/0013189X10374770

Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems boosts

learning. Instructional Science, 35(6), 481-498. doi: 10.1007/s11251-007-9015-8

Schneider, M., & Stern, E. (2009). The inverse relation of addition and subtraction: A knowledge

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 48

integration perspective. Mathematical Thinking and Learning, 11(1-2), 92-101. doi:

10.1080/10986060802584012

Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4),

475-522. doi: 10.1207/s1532690xci1604_4

Schwartz, D. L., Chase, C. C., Oppezzo, M. A., & Chin, D. B. (2011). Practicing versus

inventing with contrasting cases: The effects of telling first on learning and transfer.

Journal of Educational Psychology, 103(4), 759-775. doi: 10.1037/a0025140

Scott, C. (2013). The search for the key for individualised instruction. In J. A. Hattie & E. M.

Anderman (Eds.), International guide to student achievement (pp. 385-388). New York

Routledge.

Star, J. R., & Rittle-Johnson, B. (2009). It pays to compare: An experimental study on

computational estimation. Journal of Experimental Child Psychology, 102(4), 408-426.

doi: 10.1016/j.jecp.2008.11.004

Stern, E. (1997). Early training: Who, what, when, why, and how? In M. Beishuizen, K. P.

Gravemeijer & E. C. van Lieshout (Eds.), The role of contexts and models in the

development of mathematical strategies and procedures (pp. 239-253). Utrecht: CD ß

Press.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning

and Instruction, 4(4), 295-312. doi: 10.1016/0959-4752(94)90003-5

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and

Instruction, 12(3), 185-233. doi: 10.1207/s1532690xci1203_1

Sweller, J., Van Merrienboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional

design. Educational Psychology Review, 10(3), 251-296. doi: 10.1023/A:1022193728205

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 49

von Aufschnaiter, C., & Rogge, C. (2010). Misconceptions or missing conceptions. Eurasia

Journal of Mathematics, Science and Technology Education, 6(1), 3-18.

Winston, P. H. (1975). Learning structural desciptions from examples. In P. H. Winston (Ed.),

The psychology of computer vision (pp. 157-209). New York: Mc Graw-Hill.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 50

Table 1

Overview of the Activities of Each Session

Session

Duration

Activities

1st day (Mon.)

2 lessonsa

Pretest - prior algebra knowledge (5 min)

Introduction - short slide presentation: (5 min)

Training session 1: work sheets and learning tests 1 - 3

2nd day (Tue.)

2 lessonsa

Repetition test (5 min)

Training session 2: work sheets and learning tests 4 + 5

3rd day (Wed.)

2 lessonsa

Repetition test (5 min)

Training session 3:

contrast group: work sheets and learning tests 6 + 7

sequential group: work sheets and learning tests 6 - 8

4th day (Thu.)

2 lessonsa

Repetition test (5 min)

Training session 4:

contrast group: work sheets and learning tests 8 + 9

sequential group: work sheets and learning tests 9 + 10

1 day later

(Fri.)

2 lessons

Follow-up test 1: “algebraic transformation test” and

“algebraic transformation explanations” (45 minb)

Survey part: personal data, logical reasoning test, and

arithmetic test (45 min)

1 week later

1 lesson

Follow-up test 2: “algebraic transformation test” and

“algebraic transformation explanations” (45 minb)

3 monthsc later

1 lesson

Follow-up test 3: “algebraic transformation test” and

“algebraic transformation explanations” (45 minb)

Note. ain total 90 min. b60 min in Experiment 2. c10 weeks later in Experiment 2.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 51

Table 2

Work Sheets: Presentation Order of Learning Steps

Group

Work sheet

Contrast

Sequential

1

A1 + M1

A1 + A2

2

A2 + M2

A3 + A4

3

A3 + M3

A5

4

A4 + M4

A6 +A7

5

A5 + M5

A8 + A9

6

A6 + M6

M1 + M2

7

A7 + M7

M3 + M4

8

A8 + M8

M5

9

A9 + M9

M6 + M7

10

--

M8 + M9

Note. A1 – A9 = addition learning steps, and M1 – M9 = multiplication learning steps.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 52

Table 3

Experiment 1: Means and Standard Deviations (in Parentheses) of Students' Performance by

group on the Convention Errors of the Transformation Tests

T1

T2

T3

Contrast

Sequential

Contrast

Sequential

Contrast

Sequential

Alphabetical-ordering

10.2 (4.0)

8.9 (4.3)

10.8 (4.6)

9.9 (4.8)

6.6 (4.1)

7.5 (4.9)

Number-one

1.3 (1.6)

1.6 (3.1)

1.2 (2.0)

2.3 (3.1)

1.9 (3.1)

1.8 (2.7)

Note. T1 = one day later, T2 = 1 week later, T3 = 3 months later.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 53

Table 4

Experiment 2: Means and Standard Deviations (in Parentheses) of Students' Performance by

group on the Convention Errors of the Transformation Tests

T1

T2

T3

Contrast

Sequential

Contrast

Sequential

Contrast

Sequential

Alphabetical-ordering

11.8 (4.2)

9.9 (4.8)

12.5 (4.2)

9.8 (5.2)

11.0 (5.1)

6.9 (4.6)

Number-one

4.7 (4.1)

3.9 (3.4)

4.7 (4.1)

4.1 (3.3)

3.9 (3.5)

2.9 (2.9)

Note. T1 = one day later, T2 = 1 week later, T3 = 10 weeks later.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 54

Figure 1. An excerpt of the worked examples: addition learning steps (A3 - A6), and

multiplication learning steps (M3 - M6). Grey-cursive = the intermediate steps to the solution

that were marked red in the original version; black-bold = the problems and the results.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 55

Figure 2. Worked examples of the contrast group, example of work sheet 3 (A). Worked

examples of the sequential group, example of work sheet 2 (B). Grey-italic = the intermediate

steps to the solution that were marked red in the original version; black-bold = the problems and

the results.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 56

Figure 3. Experiment 1: Performance on the follow-up tests. T1 = 1 day later, T2 = 1 week later,

T3 = 3 months later. + p < .10, * p < .05, ** p < .01. Error bars represent standard errors.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 57

Figure 4. Experiment 1: Performance on the immediate learning testes. * p < .05, *** p < .001.

Error bars represent standard errors.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 58

Figure 5. Experiment 2: Performance on the follow-up tests. T1 = 1 day later, T2 = 1 week later,

T3 = 10 weeks later. + p < .10, ** p < .01, *** p < .001. Error bars represent standard errors.

Running head: DELAYED BENEFITS OF CONTRASTED LEARNING 59

Figure 6. Experiment 2: Performance on the immediate learning tests. *** p < .001. Error bars

represent standard errors.