Content uploaded by Derek Lomas

Author content

All content in this area was uploaded by Derek Lomas on Sep 28, 2015

Content may be subject to copyright.

S.A. Cerri and B. Clancey (Eds.): ITS 2012, LNCS 7315, pp. 591–593, 2012.

© Springer-Verlag Berlin Heidelberg 2012

The Effects of Adaptive Sequencing Algorithms on Player

Engagement within an Online Game

Derek Lomas1, John Stamper1, Ryan Muller1, Kishan Patel2, and Kenneth Koedinger1

1 Carnegie Mellon University

{dereklomas,jstamper,rmuller,koedinger}@cs.cmu.edu

2 Dhirubhai Ambani Institute of Information and Communication, Gujarat, India

kishan_patel@daiict.ac.in

Abstract. Using the online educational game Battleship Numberline, we have

collected over 8 million number line estimates from hundreds of thousands of

players. Using random assignment, we evaluate the effects of various adaptive

sequencing algorithms on player engagement and learning.

Keywords: games, number sense, engagement, adaptive sequencing.

1 Introduction

Number line estimation accuracy is highly correlated with math achievement scores in

grades K-8 (Siegler, Thompson, Schneider, 2011). To promote practice with number

line estimation, we have developed Battleship Numberline, a game involving estimat-

ing the location of ships on a number line. Using this game, we have collected over 8

million number line estimates from several hundred thousand online players. The

order of instructional items in the game is typically presented at random, but we hypo-

thesize that an adaptive sequence will result in an improved learning experience.

Adaptive instructional sequences are best known for increasing the efficiency of

learning [2]. However, Pavlik et al. [3] reported that students tended to chose an adap-

tive sequence of foreign language instructional items over a random sequence of

items. We further explore this phenomenon by investigating whether adaptive se-

quences can increase motivation to engage in a learning activity.

2 Adaptive Sequences

Conati et al. [1] describe using Bayesian Knowledge Tracing (BKT) to promote learn-

ing in an educational game. However, many games use far simpler algorithms to pro-

mote learning and player interest; for instance, they may require a player to perform

flawlessly on a level before progressing to the next. Could simpler adaptive

algorithms achieve comparable performance to Bayesian Knowledge Tracing? Specif-

ically, could they produce comparable learning (pre-post test gain) and player

engagement (duration of intrinsically-motivated play)?

In our implementation of BKT, we developed a knowledge component model with

five knowledge components (KC). The parameters for the model were developed

592 D. Lomas et al.

based on data collected from a prior classroom study involving 150 students in 4th-6th

grade. These parameters included the probability of existing knowledge (L0), learning

rates (T), and the probability for slipping (S) and guessing (G). The sequencing algo-

rithm worked by randomly choosing an item belonging to the KC with the highest

probability of being known, so long as it was below the threshold of .9 probability of

being known. When a KC exceeded .9, it was removed from the sequence. Once all

KCs in the level exceeded .9, the level was over.

The Difficulty Ladder (dLadder) is an adaptive sequencing algorithm that requires

mastery of easier items before allowing progress to more difficult items. Based on the

same dataset from which the BKT parameters were derived, the items in the instruc-

tional sequence were divided into 5 bins of difficulty, each with 4 items. Players be-

gan in the easiest bin; if they were correct twice in a row, they advanced to the next

more difficult bin. If they were incorrect twice in a row, they went back to the pre-

vious, less difficult bin. When the player completed the hardest bin, the level was

over. A high performing player could complete the ladder in only 10 trials.

Naïve ITS is based on the idea that a successful response tends to generate more

learning than an unsuccessful response. To promote success, if a player gets an item

incorrect, they are given another opportunity to attempt the item after a delay of one

other item. The delay of one trial facilitates working memory retrieval without mak-

ing the task trivially easy (as it might be if there was no delay). Once the player gets

every item correct at least once, the level is over.

The random sequence randomly presents (without replacement) one of 20 different

fractions. Unlike the adaptive sequences, the random sequence is not affected by the

player’s prior performance.

3 Experiment 1: Structure, Participants and Metrics

The adaptive sequencing experiment involved randomly assigning 1087 players to

one of sixteen different level sequences representing four different experimental con-

ditions (BKT, Difficulty Ladder, Naïve ITS, & Random) with four different

pre/posttest form combinations (A-B, B-C, C-D, D-A). Each level sequence consisted

of a pretest level, a level with one of four sequencing algorithms, a post-test level, and

then additional levels of the same sequencing algorithm (so that patterns of extended

play could be compared over the different algorithms). The pre/post tests involve four

fraction estimation problems, presented fully within the context of the game.

Our participants are anonymous online players who freely access our game through

the educational portal Brainpop.com. Despite this anonymity, we can infer from the

demographics of Brainpop.com that our users are likely to be third to eighth grade

students, probably playing in a classroom setting. Brainpop.com offers a number of

different educational games. We assume that students are free to stop playing Battle-

ship Numberline at any time; indeed, over 50% of students play less than 10 trials.

In this study, we define engagement as the number of trials that a player chooses to

play, as this is believed to reflect the players intrinsic motivation to participate in the

gameplay sequence. We measure learning as the gain from pretest to posttest.

The Effects of Adaptive Sequencing Algorithms on Player Engagement 593

Table 1. Initial conditions of experiment

Completed

Pretest

Pretest Av. Av. # of

Trials

Median

# of

Trials

% playing

> 40 Tri-

als

BKT 265 23% (.42) 25(30) 14 20%

DLadder

NaiveITS

Random

267

279

276

23% (.42)

26% (.44)

23% (.42)

29(35)

30(37)

24(22)

16

16

15

25%

24%

21%

Table 2. Here, data is presented only for the players that completed the posttest. Gain is

significant from pre to post test over all conditions (p<.02, p<.01, p<.001) using a paired t-test.

Completed

Posttest

Pretest Av. Posttest Av. Median #

of Trials

BKT 0 n/a n/a n/a

DLadder

NaiveITS

Random

22

55

103

46% (.50)

31% (.46)

25% (.43)

65%(.48)

47%(.50)

37%(.48)

30.5

49

28

4 Discussion

The data presented here suggests a modest effect from the sequencing algorithms.

Unfortunately, learning gains are impossible to compare directly, without statistically

correcting for the substantial rates of attrition. Our BKT algorithm apparently set the

bar too high—no players in this sample actually completed the level, despite some

players completing more than 100 trials. Future work will involve tuning the parame-

ters of the BKT algorithm, developing more comparable measures of learning, and

validating our online engagement construct in a classroom setting.

References

1. Conati, C., Zhao, X.: Building and evaluating an intelligent pedagogical agent to improve

the effectiveness of an educational game. In: Proceedings of the 9th International Confe-

rence on Intelligent User Interfaces, pp. 6–13. ACM (2004)

2. Corbett, A.T., Anderson, J.R.: Knowledge tracing: Modeling the acquisition of procedural

knowledge. User Modelling and User-Adapted Interaction 4(4), 253–278 (1995)

3. Pavlik Jr., P., Bolster, T., Wu, S.-M., Koedinger, K., MacWhinney, B.: Using Optimally Se-

lected Drill Practice to Train Basic Facts. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie,

S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 593–602. Springer, Heidelberg (2008)

4. Siegler, R.S., Thompson, C.A., Schneider, M.: An integrated theory of whole number and

fractions development. Cognitive Psychology 62(4), 273–296 (2011)