Content uploaded by Daniel T. Seaton
Author content
All content in this area was uploaded by Daniel T. Seaton on May 11, 2016
Content may be subject to copyright.
Transforming Advanced Placement High School Classrooms
Through Teacher-Led MOOC Models
Daniel Seaton, John Hansen
Harvard University
Cambridge, MA 02139
daniel_seaton@harvard.edu
Julie Goff, Patrick Sellers
Davidson College,
Davidson, NC
jugoff@davidson.edu
Aaron Houck
Queens University,
Charlotte, NC 28274
houcka@queens.edu
Abstract: The Massive Open Online Course (MOOC) movement--which has typically focused on
independent, online learners--has additional opportunities to reach students and teachers in
traditional classrooms. Through collaboration between Davidson College, College Board, and edX,
we report early findings of a teacher-led model utilizing open online materials in Advanced
Placement (AP) classrooms. From the summer of 2014 to present, Davidson Next modules covering
AP Calculus, AP Macroeconomics, and AP Physics have been delivered to high school teachers and
students in three phases: 1) Pilot: a program prototyping content with teachers and students in
North Carolina high schools, 2) edX: open online releases of content through edX.org, and 3) CCX:
a new custom-course tool (CCX) offering private instances of MOOC material to teachers for use
with students. We report enrollment and backgrounds of learners in each phase, and offering
indications our content benefits student performance on externally validated AP exams.
1. Introduction
The Massive Open Online Course (MOOC) movement has been proclaimed as both higher
education’s savior and an ineffective waste of limited educational resources. The technological
potential to teach students across the world has led organizations like edX and its partners to produce
and distribute nearly one thousand MOOCs since 2012. Despite the millions of students enrolled in
these courses, the traditional broadcast nature of these online materials has produced mixed
educational results and widespread criticism. As with many innovations, MOOC producers are
adapting in response to criticism, using core components -- technology and content -- to explore
more effective ways to deliver content to students. While the transformative technology of MOOCs
can enable the widespread distribution of educational content, this paper illuminates more effective
ways to deliver content to specific groups. In our case, high school teachers and their students.
Course surveys have shown that the MOOC movement has unintentionally engaged teachers around
the world [1] and some teachers have responded positively to the idea of using MOOC material for
in their own instruction [2]. Teachers offer a unique funnel of participation which could reorient
criticisms that MOOCs only reach the well educated [3] and potentially allow organizations to more
directly interact with the most disadvantaged via teachers. However, technology aimed at delivering
MOOC content directly to teachers has generally lagged behind development of purely open online
instruction. Institutions that are energetically developing MOOC content could benefit from
distribution models that facilitate both open online and teacher-led instructional models.
In this paper, we describe a unique program aimed at disseminating Advanced Placement content in
both open online and teacher-led models of instruction. Called Davidson Next, this project has
delivered Advanced Placement high school material to teachers and students around the world
through three distinct program phases: 1) a Pilot with high school teachers and students, 2) open
online via edX.org, and 3) using a new Custom Course (CCX) tool that allows teachers to manage a
MOOC directly in their classrooms. We report enrollments in each phase and highlight analyses
showing that usage of our content is correlated with student performance on AP Calculus AP.
2. Davidson Next Overview and Courses
Davidson College, in partnership with the College Board and edX, has created a suite of free,
instructional modules addressing the 14 most challenging concepts in three Advanced Placement
(AP) subjects. The project -- deemed Davidson Next -- covers challenging concepts from AP
Calculus AB & BC, AP Macroeconomics, and AP Physics 1 & 2, where challenging concepts were
selected based upon 2011-2013 AP exam data provided by the College Board. All modules are
designed to help independent students master difficult concepts in the AP curriculum, while also
providing AP instructors content which can be used to supplement (blend or flip) their classrooms.
2.1 Davidson Next
The Davidson Next team spent the summer of 2014 to spring of 2015 developing modules in each
subject, overseen by four Davidson College faculty members with more than 80 years of combined
experience writing AP curricula and exams, as well as overseeing the national grading of AP exams.
Each module addressing a challenging concept was designed and taught by a high school AP
instructor or college faculty member who also serves as a College Board consultant (e.g writing
curricula, grading exams, etc.). Furthermore, a unique pilot program was administered in the
Charlotte-Mecklenburg School district to have students and teachers interact with our content in real
classroom settings. The overall design and testing of Davidson Next content represents a departure
from many Massive Open Online Courses (MOOCs), in three ways:
Design with Teachers in mind. The original focus of the Davidson Next project was to support AP
teachers through content designed to supplement teacher-led instruction in real classrooms (blended
learning). Such design remains true, but content has been adapted such that any student with Internet
access has the opportunity to access material and thereby gain mastery of difficult topics in the AP
curriculum through edx.org. Delivering MOOC content for use in individual classrooms has been a
particular challenge over the last year, but new tools from MITx and edX are making this possible.
AP high school teachers developed content. Each Davidson Next course features 14 modules, and
each module is designed and delivered by a high school AP instructor, all of whom have worked
with the College Board to write AP curricula and exams and/or grade AP exams. Between the
summer of 2014 and the spring of 2015, thirty-eight AP instructors from across the U.S. traveled to
Davidson College to film instructional videos and develop content. All instructional material was
developed in conjunction with Davidson faculty content-editors. Workflows focused on individual
AP instructors first prototyping content in shared Google docs. These templates allowed content
editors and module authors to make collaborative changes before and after visiting Davidson
College for filming of instructional videos. The Davidson Next staff then loaded content from these
templates into edX. In addition, interactive content and simulations were created for modules:
vector-drawing problems, data-entry and plotting tools, and mathematical simulations. This content
extended the collaborative process to include outside software developers that either created new
tools, or adapted existing tools for the edX platform. Such interactive technology has been shown to
effectively support student mastery of conceptual and even physical topics in mathematics and
science-based courses[4] and was a major focus of this project.
Pilot program in Charlotte-Mecklenburg Schools. The Davidson Next team also initiated a unique
“pilot program” with teachers and students in the Charlotte-Mecklenburg School district -- as well as
a handful of other school in North and South Carolina. The Davidson Next Pilot Program was
designed to both collect feedback from these users and to generate data from which the effectiveness
of Davidson Next in achieving its goals (particularly the goal of mastery) would be assessed. In
total, the Davidson Next materials were used on a trial basis by a group of 31 teachers and over
1,100 students over the 2014–2015 academic year. By design, the Davidson Next Pilot program
collected a substantial amount of qualitative and quantitative data in addition to platform and survey
data described above: responses to a series of student and teacher surveys (a pre-semester, mid-
semester, and post-AP-exam survey), in-class observations, usage data from edX servers, teacher
experience journals, teacher comments on the content, and teacher final reflection reports. The
school districts from which the Pilot teachers were drawn have also provided data on student scores,
backgrounds, and AP exam scores for all students from 2013-2014 to 2014-2015 school years.
An important aspect of the pilot is that the Davidson Next team was “building the plane as they were
flying it.” The team created the module content over the 2014-2015 academic year, with most
Table 1: Curriculum coverage, number of AP instructors designing and leading modules, and number of
resources in each Davidson Next course.
Calculus
Macroeconomics
Physics
Curriculum Coverage
AP Calculus AB / BC
AP Macroeconomics
AP Physics 1 & 2
AP Content Creators
8
14
15
Number of Modules
14
14
14
HTML Pages
349
326
397
Videos
65
62
111
Problems
457
509
593
Interactive Problems and Simulations
50
66
85
classes only starting to use the material in January or February of 2015. The delay in implementation
made it harder for teachers to use the online materials extensively, as well as reduced the total time
that students could use the materials before taking the AP exam.
2.2 Module Structure and Courses
Davidson Next modules are organized in terms of the 14 most challenging concepts in each subject,
as determined by College Board provided exam data from 2011 through 2013. Each Davidson Next
course covers topics in the AP Calculus AB/BC, AP Macroeconomics, and AP Physics 1 & 2
curriculums. A challenging concept is one module, and all modules are designed to facilitate both
teacher-led and individual-online instruction. The same general outline applies in each module:
● “Let’s See What You Already Know”—a series of pre-assessment questions to help students
and teachers gauge their preparation.
● “Learning Cycles”— between 2 and 4 sequences containing an instructor-led video;
technology-enhanced activities and questions meant to mimic AP-style exam questions.
● “Let’s See What You Have Learned”—a set of post-assessment questions allowing teachers
to gauge student mastery and self-paced students to determine if they need further review.
Table 1 provides counts of the number of HTML pages, videos, problems, and interactive
problems/simulations in each course. Efforts were made to insure a substantial number of AP
aligned problems in each learning cycle within a module, with interactive problems, simulations, and
activities introduced where applicable [5]. Overall content counts are on par with semester long
MOOCs from MIT [6]. In addition, we use “course” to describe our content in Calculus,
Macroeconomics, and Physics. This description can be problematic since our content is a collection
of modules covering challenging concepts intended for dual use in open online and teacher-led
settings. However, “course” represents the best term across each phase of our project.
3. Davidson Next Program Phases
Davidson Next has three distinct phases:
1. Pilot: The aforementioned Pilot with teachers and students served as both a means of
collecting feedback on our content and a means for measuring effectiveness by collecting
and analyzing student performance data.
2. edX MOOCs: On July 22nd, 2015, Davidson Next released all three MOOCs on edx.org -
Challenging Concepts in AP Calculus, AP Macroeconomics, and AP Physics. The self-paced
courses are free and open to enrollment by anyone.
3. CCX (custom courses on edx): On August 20th, 2015, Davidson Next began offering
private instances of its MOOC content for teachers to use with their students. We announced
an open call for teachers to beta test the tool in the 2015 – 2016 school year.
Table 2: Key features of the three Davidson Next program phases.
Pilot
edX
CCX
Dates
Jan. 2015 - Jun. 2015
Jul. 2015 - present
Aug. 2015 - present
Audience
Charlotte-Mecklenburg Students
& Instructors
Open Online
AP Instructors & their Students
around the world
Recruitment
District agreements / Instructor
recruitment
Advertised by edx.org
Communication from College
Board and Davidson College
Users (1 click) as
of 03/01/2016
1193
22,201
4661
Data Collected
Clickstream via edX
Surveys
Teacher focus groups
Content markup from teachers
Clickstream via edX
Surveys
Clickstream via edX
Surveys
Google Group
3.1 Data Collected in Each Phase
Across each phase we rely greatly on clickstream data collected via the edX platform and survey
data collected via the Qualtrics survey platform. Clickstream data from edX are often the largest and
most complex data sets, but offer the ability to observe many aspects of student behavior such as
click-frequency, time-on-task, and pathways through content [7]. We have a survey embedded in the
initial module of each course in the edX and CCX phases intended to gather a wide variety of
background and program evaluation information, with particular questions focused on identifying
teachers and students. The Davidson Next Pilot program collected a myriad of additional data sets
from district data to focus groups. We summarize these data where appropriate.
3.2 Technology used to deliver content in each program phase
The technology of how we worked with teachers and students largely defines the three program
phases, and hence, the data collected at each phase. Our most straightforward phase involved
releasing our edX MOOCs through the edx.org site. Our courses were released asynchronously with
no restrictions on enrollment. We are actively reviewing survey and clickstream data and finding
trends comparable to previous reporting on MOOCs [2,7].
The most challenging phase was the Pilot phase, which required the creation of workflows and
tutorials for providing teachers content, helping teachers enroll their students in the platform, and
periodically updating content for all teachers. Tools to manage small private online courses -
sometimes referred to as SPOCs - were essentially non-existent for our model. When providing a
SPOC to a teacher, a clone of the original Davidson Next content was created and teachers could
only point their students to content. That meant that any time content needed to be updated, all
SPOCs had to be updated individually. Since some of our 34 teachers requested multiple SPOCs to
facilitate multiple course sections, we had to maintain and update 64 individual instances of our
content. Another difficulty of the SPOC process was a lack of content control by the individual Pilot
teachers. Without discussing nuances, the SPOC model on edX at the time did not allow our teachers
Figure 1: Enrollment by day for each phase of Davidson Next. Two curves are shown for each phase: solid
lines indicate all enrollment to date, while dashed lines are users with greater than 100 clicks. The click filter
is intended as a rough approximation of engaged users.
to regulate course parameters such as content assignment, due dates, or grading criteria. This meant
that Pilot teachers were locked into global content parameters set by the Davidson Next team.
Serendipitously, in late Summer 2015 MITx began piloting a new tool allowing any edX MOOC to
contain sub instances of that MOOC led by users separate from the course staff. The so called
custom courses (CCX) is lead by a coach - or teacher in our case - who has access to course
parameters like due dates and which content appears, while also allowing them to enroll learners and
track their progress. In addition, the CCX resides inside the main edX MOOC, meaning any updates
to the main course content are automatically inherited by the CCX instances. Hence, the CCX tool
eliminated two of largest pain points from our Pilot phase, and immediately provided us an ability to
scale our program to classrooms around the world. Table 2 offers a summary of key features for
each Davidson Next phase. Throughout the remainder of this article, we will address data from each
phase as “Pilot”, “edX”, and “CCX”.
A major concern for the Davidson Next is understanding how we can scale our program. To that
end, Figure 1 shows daily overall enrollment (solid) and enrollees with greater than 100 clicks
(dashed) in each phase of Davidson Next - the region between curves in a single phase is shaded for
clarity and we note that 100 clicks is approximately the number of interactions required for an expert
to complete the first module in each course. The edX phase by far enrolls the largest number of users
and is still growing, while the Pilot and CCX phases saturate to a steady state of users. The
difference between users that only enroll and those that have some minimal level of activity is much
more drastic for edX, but can also be considered a natural phenomenon for the freely available, open
online format. Saturation of the Pilot and CCX phase can be attributed to marketing; both programs
had initial recruitment of teachers and students aiming to reach a minimum number of teachers and
students. In contrast, edX marketing continues to recruit users in the open online setting.
Table 3: Cross-tabulation of high school teachers and students participating in each phase and course of
Davidson Next. The “**” indicates Qualtrics surveys collected student / teacher and high school / non-high
school distinctions. All data are taken as of 02/28/2016.
Pilot
edX
CCX
Instances
64 instances
3 MOOCs
261 instances
N High School Teachers
34
1187**
261*
N High School Students
1129
2105**
4400
Total Users
1193
22201
4661
N Users by course
Calculus
681
6929
2425
Macroeconomics
193
7252
479
Physics
319
8020
1757
We are particularly interested in comparing scale between the edX phase - reaching anyone around
the world - and CCX phase - reaching teachers and their students in high schools. The right plot in
Figure 1 zooms in on enrollment and participation for edX and CCX, and one can see similar
structure in how the number of participants grows. We note that marketing for our CCX phase took
place over a short window in August 2015, and although we do not draw inferences about
participation growth, we postulate that the similar scaling behavior between the two phases from
July to October suggests that the CCX tool can also scale instruction effectively.
4. Results
4.1 Are we reaching high school students and teachers?
The short answer is yes. Table 3 summarizes the total number of high school teachers and students
in each phase of Davidson Next. For the pilot program, teachers and students are identified through
direct interaction. All 34 teachers were recruited for the pilot, and teachers were responsible for
enrolling the 1129 students in each of their instances. On edX.org, survey responses identify 1187
teachers and 2105 students. With only a ~31% response rate, there are potentially 3 times the
number of teachers and students assuming a random sample. For the CCX phase, 294 teachers have
worked with Davidson Next staff to set up custom courses, and like in the pilot, these teachers are
responsible for enrolling the 4400 students. Also in Table 3, we highlight the enrollments by course.
For the Pilot and CCX phases, enrollments require direct interaction between teachers and Davidson
Next staff, and generally, we regard an instance as a single teacher-led classroom utilizing Davidson
Next content. Teachers must request instances, and it is a natural assumption that enrollees are
students. There is an off-chance that a teacher would enroll a colleague or some other hypothetical,
but we assume “non-student” enrollment to be a negligible factor in our analyses.
4.2 Partnering with Existing Educational Institutions
Partnerships have been a key component of the Davidson Next project, and have helped us more
effectively reach high school students and teachers. On August 20th, 2015, the College Board sent
targeted emails to AP teachers in these 3 disciplines, letting them know about the Davidson Next
courses and how to register for a private instance (CCX). These emails were the culmination of a
year and a half of collaboration, and the communication greatly enhanced our ability to reach users.
From August 20th to present, there have been 608 requests for CCX versions of Davidson Next’s
Calculus, Macroeconomics, and Physics. A map showing geolocation of U.S. requests can be found
in our recent blog post [8]. The College Board’s communication was our only marketing effort.
4.3 Evaluating Effectiveness of Pilot Phase
As a part of our pilot program, we collected student and teacher level data from the Charlotte-
Mecklenburg school district in an attempt to measure how student usage data from the edX platform
predicts mastery and AP exam performance. The district data include student backgrounds,
academic performance, and AP exam scores, while edX data includes various metrics detailing
interactions with the platform [2,7]. At this stage our analysis focuses on AP Calculus AB/BC for
the 2013-2014 to 2014-2015 school years. The AP Physics data were incomplete, and the AP
Macroeconomics data did not contain enough students for reliable analyses.
To study the relationship between usage of the Davidson Next materials and success on the AP
Calculus (BC) exam, we fit a difference in differences model [9] to predict the expected AP exam
score for each student using only data available from the Charlotte-Mecklenburg School (CMS)
system. Our independent variables include each student’s gender, race, year in high school, and
GPA, along with whether their teacher participated in the Pilot program. This approach compares
Figure 2: Difference in
Differences model
comparing student predicted
AP exam performance
versus usage of Davidson
Next for AP Calculus BC
students. Exam performance
is predicted using district
data collected from CMS
school system. Exam score
residuals are then correlated
with student usage relative
to class median indicating
0.08 points per hour spent
(p < .05).
Davidson Next pilot students to students with similar characteristics taking AP Calculus from the
same teacher in the year before the pilot. Second, we subtracted students' predicted performance in
our first step from their actual performance on the AP exam. This residual variability in performance
cannot be attributed to observable differences between students, such as their high school GPA or
demographic characteristics. Third, we plotted each student's residual against their actual usage of
the Davidson materials relative to their classmates (specifically, we compared individual student
usage to the class median) - see Figure 2. Our reliance on within-class variation in usage helps us
answer the question: Compared to similar students in the same class, did greater usage of the
Davidson Next materials explain why some students performed better than expected? We found that
it did. We estimate that each additional hour of usage predicted an additional 0.08 points on the AP
Calculus BC exam. The findings are statistically significant (p < .05) and similar across a variety of
usage metrics. While we cannot conclude that this relationship is causal, and measures of usage time
in online settings are only estimates, the existence of a relationship between usage and performance
beyond what one would predict based on student characteristics such as GPA is reassuring evidence
of the efficacy of the Davidson Next materials.
5. Discussion
Most open online courses already enroll a significant percentage of teachers. An initial study of
spring 2014 MITx courses revealed 28% of survey respondents identified as teachers[1]; a broader
survey of both HarvardX and MITx courses indicated 39% of survey respondents were teachers[2].
Surprisingly, Davidson Next has slightly lower rate of teacher enrollment of 23% on edX, with 17%
from high school. The edX phase currently has 1187 high school teachers registered.
Table 3 provided the number of high school students in each phase. The average pilot class roughly
20 students - on par with average class size across the United States. For CCX, the number of
instances does not quite meet the projected value of 5220 students. The structure of the pilot
program - permission slips, regular interaction between Davidson Next and pilot teachers - may have
led to the vast majority of students being enrolled. The CCX phase provides far less structured
interaction, which may lead to fewer students enrolled. But, the CCX enrollment (with limited
marketing) approximates the edX enrollment of 6882 high school students.
Finally, the analysis of our Pilot program indicated that greater usage of our content had a positive
relationship with AP exam performance, but we also point out some important caveats: 1) content
was being built during the pilot, hence, this was not a full treatment with all Davidson Next content,
2) student usage varied greatly, and 3) teacher motivation and style of usage also varied. To cut
through this variability, our analysis examined how usage relative to classmates predicts AP exam
performance. Our hope is to continue this analysis by reviewing more of the pilot qualitative data, as
well as quantitative evidence from the edX and CCX phase, which will require AP exam scores.
6. Conclusions
In conclusion, Davidson Next innovates in numerous and significant ways, from the data-driven
development of content by leading high school AP teachers, to the engaging interactive online
content matching established College Board curricula. This matching enables a standardized,
authoritative evaluation of content’s effectiveness (via the AP exams), which in turn translates into
established course credit at U.S. institutions of higher education. These strengths have led thousands
of students and teachers across the world to use the Davidson Next materials during its first year.
This widespread adoption is only part of the rapidly expanding wave of digital learning. The most
effective of these innovations combine engagement through online interaction and modular
personalization to meet each user’s needs. These qualities are central to Davidson Next. In a world
of increasingly demanding, distracted, and segmented audiences, these qualities will increasingly
become necessary conditions for any product aiming to help students access and master challenging
material like AP Chemistry, Macroeconomics, and Physics.
7. Acknowledgments
Davidson Next thanks our partners for their time, talent, and financial support: The Laura and John
Arnold Foundation, Davidson College, College Board, edX, Charlotte-Mecklenburg Schools,
2Revolutions, OpenCraft, Jean-Michel Claus, Christopher Chudzicki, Analia Barrantes, Elina Hu,
Peter Pinch, the Office of Digital Learning at MIT, MITx Engineering, and Priceless Misc. Our
sincere thanks to Davidson faculty members who served as lead editors for Davidson Next: Dr.
Larry Cain, Dr. Clark Ross, Dr. Stephen Davis, and Dr. Ben Klein. Finally, our sincere thanks to 40
AP teachers and college faculty who traveled to Davidson College to help film and build materials,
as well as the more than 30 high school AP teachers who piloted our modules in 2014-2015.
8. References
[1] Seaton, Daniel Thomas, et al. "Enrollment in MITx MOOCs: Are We Educating Educators." Educause
Review (February 2015) (2015).
[2] Ho, Andrew Dean, et al. "Harvardx and mitx: Two years of open online courses fall 2012-summer 2014."
Available at SSRN 2586847 (2015).
[3] Emanuel, Ezekiel J. "Online education: MOOCs taken by educated few." Nature 503.7476 (2013).
[4] Finkelstein, N. D., et al. "When learning about the real world is better done virtually: A study of
substituting computer simulations for laboratory equipment." PRST-Physics Education Research 1.1 (2005).
[5] http://blog.edx.org/davidson-next-interactive-content-for-students-and-teachers
[6] Comparison with MIT is made out of convenience. Course reports released in 2014 highlight course
structure. For example: Seaton, Daniel Thomas, et al. "8.02 x Electricity and Magnetism MITx on edX
Course Report-2013 Spring." (2014). http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2382328
[7] Seaton, Daniel T., et al. "Who does what in a massive open online course?." Communications of the ACM
57.4 (2014).
[8] http://blog.edx.org/empowering-teachers-and-students-around-the-world?track=blog
[9] Angrist, J. D., & Pischke, J. S. (2008). Mostly harmless econometrics: An empiricist's companion.
Princeton university press.
