Vivek Fitkariwala’s scientific contributions

Showing progress towards a goal is a well-established motivational design tactic. This paper describes how university students designed a new "progress meter" for an online learning game and then evaluated the effects of the design using a controlled online experiment, or A/B test. Using the UpGrade A/B testing platform, we randomly assigned 3,200 online players to the original game or to an updated version of the game with progress meters. We hypothesized that progress meters would significantly increase student engagement, measured as the voluntary time on task (duration of play) and number of items completed. We were surprised to find that the new design significantly reduced player engagement by~15%. Does this mean that progress meters are a sham? No. We conclude that the appropriate response to this surprising finding is to keep testing new iterations of the game mechanic. Therefore, this paper points towards a future where instructional designers and learning engineers can continuously improve online education through rapid cycles of design and A/B testing.

UpGrade is an open-source tool for A/B testing in educational software. We this study, we used UpGrade to run large scale online experiments in an educational game. Our experiments were aimed at increasing the student engagement. We experimented with various features of the game such as question difficulty, game narrative, feedback style, etc. to find out which conditions produced optimal outcomes. One after another, we conducted 3 different experiments. All of these experiments were created and monitored through UpGrade. We faced several issues during the implementation of these experiments within UpGrade. We discovered that buggy programming logic in the educational software can produce invalid experiment enrollments in UpGrade. We also found out that without tracking version of the educational software, it is possible to get noise in the experimental data. We present several recommendations to avoid these pitfalls. Results of our experiments are not discussed in this paper.

This paper describes a new, open source tool for A/B testing in educational software called UpGrade. We motivate UpGrade's approach, describe development goals and UpGrade's software architecture, and provide a brief overview of working within UpGrade to define and monitor experiments. We conclude with some avenues for future research and development.