ChapterPDF Available

The Impact of Adaptive Activities in Acrobatiq Courseware - Investigating the Efficacy of Formative Adaptive Activities on Learning Estimates and Summative Assessment Scores

Authors:
  • Acrobatiq by VitalSource
  • VitalSource Technologies

Abstract and Figures

The purpose of this paper is to explain the learning methodologies behind the adaptive activities within Acrobatiq’s courseware, and to investigate the impact of these adaptive activities on learning estimates and summative assessment scores using real course data. The adaptive activities used for this analysis were part of a Probability and Statistics course, which was delivered to college students at a public four-year institution as part of an educational grant. The data were analyzed to identify if the adaptive activities had an impact on learning estimates as well as on summative assessment scores. Results showed that the adaptive activities had a net positive effect on learning estimates. Results also showed that not only did learning estimate states correlate to mean summative assessment scores, but improving learning estimates after completing the adaptive activity practice yielded higher mean summative assessment scores. The implications of this analysis and future research are discussed.
Content may be subject to copyright.
The Impact of Adaptive Activities in Acrobatiq
Courseware - Investigating the Efcacy
of Formative Adaptive Activities on Learning
Estimates and Summative Assessment Scores
Rachel Van Campenhout, Bill Jerome, and Benny G. Johnson
(&)
Acrobatiq by VitalSource, Pittsburgh, PA, USA
benny@acrobatiq.com
Abstract. The purpose of this paper is to explain the learning methodologies
behind the adaptive activities within Acrobatiqs courseware, and to investigate
the impact of these adaptive activities on learning estimates and summative
assessment scores using real course data. The adaptive activities used for this
analysis were part of a Probability and Statistics course, which was delivered to
college students at a public four-year institution as part of an educational grant.
The data were analyzed to identify if the adaptive activities had an impact on
learning estimates as well as on summative assessment scores. Results showed
that the adaptive activities had a net positive effect on learning estimates. Results
also showed that not only did learning estimate states correlate to mean sum-
mative assessment scores, but improving learning estimates after completing the
adaptive activity practice yielded higher mean summative assessment scores.
The implications of this analysis and future research are discussed.
Keywords: Adaptive activities Doer Effect Formative practice Learn
by doing Learning modeling Learning objectives Learning outcomes
Scaffolded practice
1 Introduction
One goal of this paper is to illuminate the learning theory used to develop the adaptive
activities in Acrobatiqs courseware. The adaptive activities require many inputs from a
complex learning environment in order to adapt appropriately for each student. Within
the courseware, learning objectives are used to organize both content and formative
practice within lessons. As students answer the formative questions, their data are sent
to Acrobatiqs predictive model and a learning estimate is generated for each student
against each learning objective. The adaptive activities use these learning estimates to
select the appropriate scaffolded questions against each learning objective for the
students. Each student will receive a set of questions with the appropriate scaffolding
specically selected for their needs.
A second goal of this paper is to analyze data in order to identify the impact these
activities have for students. Impact could mean different things in this environment,
©Springer Nature Switzerland AG 2020
R. A. Sottilare and J. Schwarz (Eds.): HCII 2020, LNCS 12214, pp. 543554, 2020.
https://doi.org/10.1007/978-3-030-50788-6_40
which leads to two different research questions. The rst of those questions is: do the
adaptive activities increase learning estimates for students? Students have learning
estimates for each learning objective included in the module for which the activity is
written. The questions within the activity are also formative in nature, and therefore can
contribute to that students learning estimate. This research question could use within-
student data as well as between-student data. We will look to see if students increase
their learning estimates after completing the adaptive activities. We will also look to
see if completion of the adaptive activities changes learning estimates differently
between groups of students who have different learning estimates. Students with low
learning estimates will have more scaffolded practice and may be more likely to
increase their learning estimates than those students who have already been successful
and have high learning estimates.
In addition to learning estimates, another measure to investigate is the impact of the
adaptive activities on summative assessment scores. These activities are placed
immediately before module quizzes as a last scaffolded practice before students take the
scored assessment. The second research question is: do the adaptive activities increase
student scores on summative assessments? This research question investigates the
summative scores of students in different learning estimate categories to identify the
differences between groups. This analysis will also look at summative scores for stu-
dents who increased their learning estimates (after working through the adaptive
activity), and students who did not.
The ultimate purpose of the adaptive activities is to assist the learner in their
progression through the courseware and in mastery of the content. The signicance of
the data analysis against both research questions would verify that the purpose of these
activities is being met. An increase in learning estimates after completing the activities
would indicate that the learning methodology behind the activities is working. While
the learning estimate is a metric both created and used by the Acrobatiq platform,
improvement in it could indicate the courseware and design of the adaptive activities
are functioning as designed. Changes in summative assessment scores after completing
the adaptive activities would be a signicant nding for these activities. Summative
assessments produce scores which are part of the students gradebook. To assist in
improving student scores, and therefore possibly their course grades, would be
incredibly valuable to students. The analysis and results of this paper indicate how
future research could be conducted to verify results, as well as provide new ideas on
how to continue to help students master content through adaptive experiences.
2 Learning Methodology
The purpose of this section is to outline the learning methodology behind the design of
the Acrobatiq courseware used in this analysis, as the course features and reasoning for
them are key to the investigation. The adaptive activities are integrated into the
courseware content and require specic instructional design practices in order for them
to properly adapt for each learner.
Formative Practice. The rst requirement is the inclusion of formative practice
questions for each learning objective in a module of content. The formative practice
544 R. Van Campenhout et al.
questions can have a variety of formats (such as multiple choice, text input, equation,
drag and drop, etc.), provide immediate targeted feedback for each answer option, and
allow students to continue answering until they get the correct answer. Formative
practice is a well-established technique shown to increase learning gains for students of
all ages, and across subjects [1]. Moreover, studies have shown that formative
assessment can raise achievement for low performing students most of all, while
improving learning for all [1]. The formative practice questions distributed throughout
the text and the adaptive activity also act as no- or low-stakes practice testing. Practice
testing increases learning gains and retention, and including feedback has been shown
to outperform practice testing without feedback [2].
Formative practice is integrated with the content of the Acrobatiq Probability and
Statistics course. The lessons in the courseware begin with the learning objective and
are followed by the relevant content and formative practice opportunities, which are
interspersed throughout the lesson. This chunking method of following short sections
of content with formative practice is key for the learn by doing approach, which
produces the Doer Effect [35]. Research utilizing interactive courseware from Car-
negie Mellons Open Learning Initiative shows that students who did more interactive
activities had a learning benet approximately six times that of reading text and three
times that of watching video [3]. Follow-up analysis showed this relationship between
doing and learning to be causal [4], and this nding has been replicated in our previous
work [5].
Learning Estimates. While the formative questions do not produce a grade, the
students responses impact his or her learning estimate for that learning objective. The
learning estimate is a predictive measure generated by Acrobatiqs analytics engine for
each student on each objective to estimate how well a student will perform on the
learning objectives summative assessment. This learning estimate is required for the
adaptive activity to adapt for the learner (see below).
The machine learning model underlying the learning estimate uses item response
theory (IRT) [6,7] to construct an estimate of the students ability for the objective,
from which a prediction can be made of performance on the objectives summative
assessment. An advantage of an IRT-based approach is it can take the psychometric
properties of the questions into account when constructing the ability estimate. A two-
parameter logistic model is used, which models difculty and discrimination for each
question. A Bayesian approach [8] is used to estimate the posterior distributions of the
IRT question parameters from data, as well as the student ability posterior distribution
from the formative and adaptive questions answered. From the ability posterior a
numerical learning estimate value between 0 and 1 is derived (higher values indicating
better expected performance). When the model has sufcient condence in its pre-
diction based on the data available to it, a category of low, medium or high is also
assigned; otherwise the category is labeled as unknown.
Adaptive Activities. The adaptive activities in the Probability and Statistics course-
ware were designed by instructional designers and subject matter experts to include
scaffolded questions against each learning objective in the module. The goal of the
adaptive activity is to provide students with the appropriate level of scaffolding for their
needs. In this activity, questions are written against the learning objectives from the
The Impact of Adaptive Activities in Acrobatiq Courseware 545
module at three increasing levels of difculty (low, medium, and high). By organizing
the questionsand contentin this way the adaptive activities align with Vygotskys
zone of proximal development, which structures content and interactions in such a way
as to meet the learner at their level of understanding and build upon it [9]. Providing
struggling students with foundational questions as scaffolds to more challenging
questions helps to reduce cognitive load in a similar way as worked examples [10].
At the start of the adaptive activity, the platform identies the students learning
estimate for each learning objective used in the activity. The learning estimate deter-
mines the level of scaffolding to deliver. A student with a low learning estimate on a
learning objective did poorly on the formative practice, and therefore is given the
additional scaffolded practice for the learning objective. A student with a high learning
estimate did well on the formative practice for the learning objective, and therefore are
only delivered the highest difcultyor corequestions. It is important to note that
while the previous formative practice informed the learning estimates which deter-
mined how the activity adapted to the student, the adaptive activity itself also con-
tributes to the learning estimate, asthe adaptive activities are also formative in nature.
3 Methodology
The Courseware. The course used for this analysis was a Probability and Statistics
courseware developed as an introduction to the subject at the university level. This
Acrobatiq courseware was originally developed at the Open Learning Initiative at
Carnegie Mellon University and has been substantially revised over the years as an
Acrobatiq course. Included in those revisions were the addition of the adaptive ele-
ments, new summative assessments, and a new learning model, which are all critical
elements to this analysis. The Probability and Statistics courseware included 5 units of
content with a combined 10 total content modules within them. The lessons within each
module begin with a learning objective and typically have interleaved content and
formative questions. Learning objectives appear at the top of the lesson, are student
centered and measurable, and all content and formative questions align to and are
tagged with them. Formative questions are intended as learning opportunities and
therefore do not create a grade. Students receive immediate feedback for formative
questions and have the opportunity to continue to answer until they choose the correct
response. Each module ends with an adaptive activity and a quiz which contain
questions against the learning objectives covered in the module. The adaptive activity
is a formative activity as well by providing immediate feedback and multiple attempts,
however it does produce a completion score for the gradebook. The quiz is a sum-
mative assessment; students do not receive immediate feedback and it produces a score
for the gradebook.
The courseware elements necessary for this analysis include each studentsfor-
mative question attempts and accuracy, learning estimate states, adaptive activity
attempts and accuracy, and summative assessment attempts and accuracy. These data
points provide a window into each students journey through the courseware.
546 R. Van Campenhout et al.
The Population. This course was delivered to students at a large public 4-year
institution in the fall 2018 semester. There were no experimental manipulations used on
this student population, and the platform does not collect demographic information on
students, so no analysis according to demographic information would be possible. The
data set includes numeric identiers for individual students, for which all the students
interactions are recorded against. The data collected will be used only for analysis of
the relationship of each students interactions with the adaptive activities in relation to
the formative and summative activities.
The Data Set. The data set includes 306 students and 47 learning objectives. The unit
of analysis was student-learning objective pairs, i.e. the data records corresponded to a
single students work on a single learning objective. Out of the 306 47 = 14,382
possible student-learning objective combinations, there were 12,612 combinations with
data. Each data record contained: number of formative questions answered; learning
estimate value and category after formative questions; number of adaptive questions
answered; learning estimate value and category after adaptive questions; number of
summative questions answered; and mean summative question scores. Not all records
contained formative, adaptive, and summative data; for example, in some cases stu-
dents answered adaptive and summative but not formative questions.
This original data set was reduced to include only learning objectives with forma-
tive, adaptive, and summative attempts. For some learning objectives, the number of
formative questions contained in the course was not consistent with best practices of
course design and/or the requirements of the learning model. As such, learning
objectives were ltered to those having a minimum of 5 formative questions and a
maximum of 45 formative questions. The data set was also cleaned to remove errors in
data collection or storage. The nal data set included 21 learning objectives, 300
students, and 5,971 total records.
The data analysis was performed on both the full original data set as well as this
reduced data set. The reduced data set was chosen for presentation to attempt to obtain
the clearest picture of the relationships between student practice and summative
assessment performance. However, the qualitative conclusions of the analysis were
consistent between both data sets.
4 Results
4.1 Research Question 1
The rst analysis done was to address the rst research question: do the adaptive
activities increase learning estimates for students? The adaptive activities were
designed to provide scaffolded questions personalized for each students learning
estimate, with the goal of assisting students who needed additional help before they
took the high-stakes summative assessment. Therefore, we hypothesized the adaptive
activities would have a positive impact on learning estimates for some portion of
students.
The Impact of Adaptive Activities in Acrobatiq Courseware 547
Overall Learning Estimate Changes. To answer this question, the data set was
analyzed with regard to whether or not student learning estimates before the adaptive
activity changed after students completed the adaptive activity questions. There were
3,972 cases in which a learning estimate was available immediately before and after
adaptive practice, with a mean learning estimate change of 0.062 (0.169). In 2,550
instances (64.2%), the learning estimate was increased by adaptive practice, with a
mean increase of 0.132 (0.167). In the remaining 1,422 instances, the learning estimate
decreased by a mean of 0.064 (0.069). A Shapiro-Wilk test showed that the learning
estimate differences were not normally distributed, so a one-sample Wilcoxon signed
rank test was used to check if the median learning estimate change was of the popu-
lation was statistically signicantly different than 0, which it was (p0.001).
These results support the hypothesis that the adaptive activity had a net positive
impact on learning estimates for the majority of students. The smaller portion of
students whose learning estimates decreased could be explained by considering those
students who may have been near the threshold of a learning estimate category. Getting
questions wrong in the adaptive activity could have shifted their learning estimate
down to the lower category (Table 1).
Learning Estimate Changes within Categories. In addition to understanding how
the student learning estimates increased or decreased as a whole, we also investigated
how learning estimates changed for students within different learning estimate cate-
gories. The learning estimates for students determine which questions are delivered in
the activitywhether or not students need the scaffolded questions, and at which
difculty level. This analysis will reveal if these different groups of students changed
learning estimates differently after completing the adaptive activity. There was a total
of 3,954 records for which student summative scores were also available for the
learning objectives included in this analysis. A Kruskal-Wallis Htest was performed on
Table 1. The descriptive statistics for the increase or decrease of learning estimates after the
completion of adaptive questions.
Statistic Learning estimate increase Learning estimate decrease
Count 2550.000000 1422.000000
Mean 0.131999 0.064386
Std 0.167449 0.068799
Min 0.000013 0.511564
25% 0.023794 0.086208
50% 0.058838 0.039807
75% 0.164347 0.017508
Max 0.806394 0.000008
548 R. Van Campenhout et al.
the summative scores in the different groups, which showed the groups do not all have
the same median (p0.001) (Table 2).
The rst data analyzed were those students who achieved high learning estimates for
learning objectives after they completed the formative questions. There was a total of
1,648 (41.7%) instances of students achieving high learning estimates after completing
the formative questions. After completing the adaptive activities, there were three
possible outcomes based on how they performed: retain high learning estimate, change
to medium learning estimate, or change to low learning estimate. Of the total high
learning estimate instances, 1,511 (91.69%) remained in the high category after
completing the adaptive activity questions. There were 112 (6.8%) instances of stu-
dents whose learning estimate changed to medium, and 25 (1.52%) instances of stu-
dents whose learning estimate changed to low. These results were consistent with
expectations for this group of students. Students who did well enough on the formative
practice to earn high learning estimates similarly did well enough on the adaptive
activity questions to retain that learning estimate category. For the less than 10% of
instances where learning estimates dropped to medium or low, this means students did
poorly enough on adaptive questions to lower their learning estimates. Likely these
students were just over the high learning estimate threshold and answering incorrectly
moved them to the lower category.
The next category of data analyzed was students who achieved medium learning
estimates for learning objectives after they completed the formative activities. There
was a total of 918 (23.22%) instances in this category. After completing the adaptive
activities, there were three possible outcomes based on how they performed: retain
medium learning estimate, change to high learning estimate, or change to low learning
estimate. Of the total medium learning estimate instances, 441 (48.04%) remained in
the medium category after completing the adaptive activity questions. There were 348
(37.91%) instances of students whose learning estimate changed to high, and 129
Table 2. The number of instances of learning estimate changes after completing adaptive
questions, grouped by learning estimate category.
Learning estimate
category
High (after
adaptive)
Medium (after
adaptive)
Low (after
adaptive)
Unknown (after
adaptive)
High (before
adaptive)
1511 112 25 NA
Medium (before
adaptive)
348 441 129 NA
Low (before
adaptive)
47 141 555 NA
Unknown (before
adaptive)
339 136 146 24
The Impact of Adaptive Activities in Acrobatiq Courseware 549
(14.05%) instances of students whose learning estimate changed to low. This medium
learning estimate category shows more change in learning estimate state than the high
category. These results are in line with expectations, as the medium category has
thresholds next to both the high and low category, and therefore students near those
thresholds could shift their states if they do well or poorly on the adaptive questions.
The next category of data analyzed were those students who achieved low learning
estimates for learning objectives after they completed the formative activities. There
was a total of 743 (18.79%) instances in this category. After completing the adaptive
activities, there were three possible outcomes based on how they performed: retain low
learning estimate, change to medium learning estimate, or change to high learning
estimate. Of the total low learning estimate instances, 555 (74.7%) remained in the low
category after completing the adaptive activity questions. There were 141 (18.98%)
instances of students whose learning estimate changed to medium, and 47 (6.32%)
instances of students whose learning estimate changed to high. This low learning
estimate category had fewer changes to other categories than the medium category
previously. However, while nearly 75% of students who struggled on learning
objectives continued to do so in the adaptive activity, just over 25% of instances show
students who were able to increase their learning estimates to medium or high, which is
a positive nding.
The nal category of data to be analyzed was students who did not complete enough
formative practice against learning objectives to generate a learning estimate. This
unknown learning estimate category is not an indicator of ability, but rather a state of
the predictive analytics not having enough information to determine a category. Of the
645 (16.31%) instances of an unknown learning estimate, there were four categories of
learning estimates after the adaptive activity questions were completed: high, medium,
low, or unknown. The change in learning estimate states for these instances after the
adaptive activity questions were as follows: 339 (52.56%) changed to high, 136
(21.09%) changed to medium, 146 (22.64%) changed to low, and 24 (3.72%) remained
unknown. We cannot determine if the adaptive activity questions helped to shift the
nal learning estimates, but the activity at least moved all but 3.72% of instances into
known categories.
4.2 Research Question 2
The second research question to investigate is: do the adaptive activities increase
student scores on summative assessments?While the changes in learning estimates is
one indicator that the adaptive activities have an impact on student learning, another
measure is the summative assessment scores which correspond with the same learning
objectives. In this Probability and Statistics course, the adaptive activity is always
placed before a summative quiz, with the goal of trying to help students prepare for this
scored assessment. The change for each learning estimate category is compared by
looking at the mean summative assessment score for each group (Table 3).
550 R. Van Campenhout et al.
The ndings of this analysis showed both expected and unexpected results. Stu-
dents with higher learning estimates generally had higher mean summative scores.
Within original learning estimate categories, students who improved their learning
estimates after the adaptive questions did better on summative questions than students
who maintained their learning estimate category, and students who lowered their
learning estimate category after the adaptive questions did worse than their counterparts
who maintained their category. Mean summative scores also decreased slightly within
the post-adaptive learning estimate category depending on the pre-adaptive learning
estimate category. Mann-Whitney Utestscomparing median summative scores of the
low and medium category before the adaptive activity to their respective category
changes after the activityfound that all changes were signicant. This indicates that
not only do learning estimates correlate to summative score performance, but students
who increase their learning estimates perform better on summative assessment
questions.
A surprising nding was the nal mean summative assessment scores for the
unknown learning estimate category. Recalling the earlier changes in learning estimates
for this category: 339 (52.56%) changed to high, 136 (21.09%) changed to medium,
146 (22.64%) changed to low, and 24 (3.72%) remained unknown. Given the ndings
above, the mean summative scores for the unknown to high, medium, and low learning
estimate categories do not align with the other mean summative scores for those
categories. The scores for unknown to high, medium, and low are all above 0.7. The
only other categories with mean summative scores above 0.7 were high to high, and
medium to high. So, despite which learning estimate category the unknown state
changed to, all performed as well as those with high learning estimates. The only
unknown category who did not perform as well were those who remained unknown, for
which they scored the second lowest of all categories.
4.3 Learn by Doing
During the data analysis, there were additional ndings which were notable with regard
to the learning theory underlying the coursewares design. Learn by doing is a key
principle which supported the addition of frequent formative practice. The adaptive
Table 3. Mean summative scores by learning estimate category before and after the adaptive
activity questions.
Learning Estimate
Category
High (after
adaptive)
Medium(after
adaptive)
Low (after
adaptive)
Unknown (after
adaptive)
High (before
adaptive)
0.775 0.689 0.617 NA
Medium (before
adaptive)
0.716 0.668 0.612 NA
Low (before
adaptive)
0.676 0.617 0.543 NA
Unknown (before
adaptive)
0.767 0.711 0.709 0.569
The Impact of Adaptive Activities in Acrobatiq Courseware 551
activity was similarly expected to help students learn as it was another formative
practice activity and gave students one more personalized chance to prepare for a quiz.
The analysis of the ndings for summative assessment scores showed the correlation
between increased learning estimates and increased mean summative assessment
scores. This led us to review the mean summative assessment scores for students
grouped by whether they participated in formative and adaptive practice.
Table 4shows the breakdown of students who did or did not participate in formative
practice and the adaptive activity practice. We hypothesize that because the adaptive
activities produced a completion score, doing the adaptive practice was the largest
category for both doing and not doing formative practice. The instances of students who
did or did not do the formative practice are also broken down by whether they did or did
not do the adaptive activity practice. What we are able to see are the combinations of
practice from least to most, and the mean summative scores correlate with the amount of
practice completed. Not doing formative or adaptive practice produced the lowest mean
summative scores (0.566), while doing both the formative and adaptive practice pro-
duced the highest mean summative scores (0.694). A Kruskal-Wallis Htest was per-
formed on the groups and there was a statistically signicant difference among them (p
0.001). These ndings are in line with those from previous research [3], which
identied correlational relationships between doing formative practice and summative
assessment scores. While testing for a causal relationship between practice and out-
comes requires additional data not available in this study, the ndings of causality from
related research lead us to anticipate the same would be true in this instance [4,5].
5 Discussion
This study was critical in helping to understand the efcacy of the adaptive activities in
Acrobatiqs courseware. The learning science and instructional design principles uti-
lized to create the course content were intended to help students better learn the
material and prepare them for high-stakes summative assessments. The ndings which
show that not only do the adaptive activities help increase learning estimates for many
students, but that learning estimates correlate to mean summative scores validates the
primary function of the adaptive activities.
While not every learning estimate increased after completing the adaptive practice,
nearly 38% of students with medium learning estimate instances increased to high
Table 4. Mean summative scores for students grouped by participation status for formative and
adaptive practice.
Formative Adaptive Count Mean summative score
False False 146 0.5663
False True 1142 0.5744
True False 168 0.6098
True True 4411 0.6935
552 R. Van Campenhout et al.
learning estimates and over 25% of students with low learning estimate instances
increased to medium or high learning estimates. With increased learning estimates
correlating to increased mean summative assessment scores, assisting this percentage of
students through adaptive practice to increase learning estimates and summative scores
is a very positive nding. While nal course gradingpolicies are up to instructors,
summative assessments produce a grade in the courseware gradebook which is often
part of the studentsnal course grade. If the adaptive practice can improve summative
assessment scores for students in the low and medium learning estimate category, that
could make a difference in nal course grades for those students.
This analysis also supported for the research team the principle that more doingis
better for students. The more formative practice students completed, the higher the
mean summative assessment scores. Students who did not do formative practice but did
do the adaptive practice scored slightly higher than those who did no practice at all
(note that when no formative practice is done, the adaptive engine has no data to adapt
the activity). Yet the adaptive practice had a larger effect when students also completed
the formative practice, thereby giving the adaptive practice a basison which to adapt.
Instructors could capitalize on the benets of both the formative and adaptive practice
through their expectations and requirements of students when using the courseware.
Finding ways to actively enforce the completion of formative and adaptive practice
through classroom policies could increase the benets of these activities for students.
The results of this analysis also show that these ndings could be replicated in
many other courses. The Probability and Statistics courseware used did not have
perfect associationsor taggingbetween the formative practice, adaptive practice,
and summative assessments. The data set was reduced to only include the data where
learning objectives had data for all three. Yet despite the imperfections in the
courseware itself, results were still very positive. This is encouraging, as developing
perfect courseware is a difcult task. Future research could aim to replicate ndings
using different courseware for different subject domains, as well as courseware with
different instructional design practices.
The data used for the analysis in this paper was also gathered from a course where
no experimental controls were used. It is encouraging that this non-experimental, real-
world course showed such positive results. Additional validation of results could be
found by partnering with instructors in the future to compare variations in classroom
controls and practices. Initial plans for data analysis included comparing this data to a
course from a different semester which did not include the adaptive activities, but no
version could be found where the only substantive change was the adaptive practice.
Future research could also review individual adaptive activities to try to identify the
ideal number of scaffolded questions to deliver to students. The analysis of this data
revealed support for the Doer Effect, but does not indicate how much practice is ideal to
help students increase learning estimates and summative scores. Such research could
also analyze the difculty levels of questions across formative, adaptive, and sum-
mative assessments to determine the most benecial combination and frequency of
difculty levels. The primary function of the adaptive activities is to support student
learning. Additional research into ideal number and types of questions to deliver to
students could potentially improve learning.
The Impact of Adaptive Activities in Acrobatiq Courseware 553
References
1. Black, P., William, D.: Inside the black box: raising standards through classroom
assessment. Phi Delta Kappan 92(1), 8190 (2010). https://doi.org/10.1177/003172171009
200119
2. Dunlosky, J., Rawson, K., Marsh, E., Nathan, M., Willingham, D.: Improving students
learning with effective learning techniques: promising directions from cognitive and
educational psychology. Psychol. Sci. Public Interest 14(1), 458 (2013). https://doi.org/10.
1177/1529100612453266
3. Koedinger, K., Kim, J., Jia, J., McLaughlin, E., Bier, N.: Learning is not a spectator sport:
doing is better than watching for learning from a MOOC. In: Learning at Scale, pp. 111120.
Vancouver, Canada (2015). http://dx.doi.org/10.1145/2724660.2724681
4. Koedinger, K., McLaughlin, E., Jia, J., Bier, N.: Is the doer effect a causal relationship? How
can we tell and why its important. learning analytics and knowledge. Edinburgh, United
Kingdom (2016). http://dx.doi.org/10.1145/2883851.2883957
5. Olsen, J., Johnson, B.G.: Deeper collaborations: a nding that may have gone unnoticed.
Paper Presented at the IMS Global Learning Impact Leadership Institute, San Diego, CA
(2019)
6. Baker, F.: The basics of item response theory. In: ERIC Clearinghouse on Assessment and
Evaluation. College Park, MD (2001). http://echo.edres.org:8080/irt/baker/
7. Embretson, S., Reise, S.: Item Response Theory for Psychologists. Erlbaum, Mahwah (2000)
8. Fox, J.: Bayesian Item Response Modeling: Theory and Applications. Springer, Heidelberg
(2010). https://doi.org/10.1007/978-1-4419-0742-4
9. Sanders, D., Welk, D.: Strategies to scaffold student learning: applying Vygotskys zone of
proximal development. Nurse Educ. 30(5), 203204 (2005)
10. Sweller, J.: The worked example effect and human cognition. Learn. Instr. 16(2), 165169
(2006). https://doi.org/10.1016/j.learninstruc.2006.02.005
554 R. Van Campenhout et al.
... Carnegie Mellon fostered and spread learning engineering [14], through such institutes as the Open Learning Initiative (OLI) where early courseware was developed and researched (described later in the chapter). Acrobatiq was founded from OLI, and the team who formed Acrobatiq intentionally organized the company and its development process around the learning engineering process, led by learning engineers [15]. The Acrobatiq team engaged in many different LEPs over the years. ...
... Courseware, as developed at Acrobatiq, is a comprehensive learning environment that combines learning content with frequent formative practice in short, topical lessons aligned to learning objectives that are grouped into modules and units and followed by adaptive activities and summative assessments [15]. Students receive all the learning content, practice, and assessments needed for a semester-long university course in one unified learning environment. ...
... As the student begins the adaptive activity, the learning estimate determines the level of scaffolding the student receives for each learning objective in the activity. Research has shown that students who increase their learning estimate in the adaptive activities also increase their scores on the summative assessments [15]. ...
Chapter
As educational data science (EDS) evolves and its related fields continue to advance, it is imperative to employ EDS to solve real-world educational challenges. One such challenge is to research how students learn and study effectively in digital learning environments and apply those findings to better their learning resources. The volume of educational data collected by digital platforms is growing tremendously, so it is a pivotal moment for EDS to be applied with an ethical approach in which the best interests of the learner are kept at the forefront. Learning engineering provides a practice and process to engage in EDS with a student-centered approach. In this work, we exemplify how the learning engineering process (LEP) guided large-scale data analyses to advance learning science (i.e., the doer effect), developed new artificial intelligence (AI)–based learning tools, and scaled both effective learning methods in natural educational contexts and automated data analysis methods—all in the service of students. The examples of analyses in this chapter serve to showcase how EDS—applied as a part of learning engineering—can validate learning science theory and advance the state of the art in learning technology.KeywordsEducational data scienceLearning engineeringCoursewareLearn by doingDoer effectArtificial intelligenceAutomatic question generation
... Modules are made up of lesson pages, and each lesson contains readings, images, and formative practice questions all tied to a central learning objective. Learning objectives are student-centered and measurable, and the practice questions are tagged with the learning objective to feed data to the platform's learning analytics engine [21], as well as to inform post hoc analysis. The formative practice questions are interleaved with small chunks of content to provide practice to students at the point of learning that content. ...
... In addition to formative practice questions integrated within the content, there are adaptive activities and summative assessments in the courseware [21]. The adaptive activities are placed at the end of the module and cover all learning objectives included in that module. ...
Article
Full-text available
The doer effect is a learning science principle that proves students who engage with formative practice at the point of learning have higher learning gains than those who only read expository text or watch video. This principle has been demonstrated through both correlational and causal analysis. It is imperative that learning science approaches capable of increasing student learning gains be rigorously tested and replicated to confirm their validity before wide-scale use. Previously we replicated causal doer effect results using student data from courseware used at a major online university. In this paper, we will replicate both the correlational doer effect analysis as well as the causal analysis using both unit tests from the courseware and the course final exam. These multiple analyses of the doer effect on the same course data provide a unique comparison of this method and the impact of the doer effect on near and intermediate learning assessments. Findings of the correlational doer effect analyses confirmed doing was more significant to outcomes than reading, and further analysis determined these results could not be attributed to student characteristics. Results of the causal analysis verified doing was causal to learning on both the unit tests and final exam. The implications of these doer effect replication results and future research will be discussed.
... Previous research found that similarly designed courseware environments helped students learn more efficiently than traditional methods (Lovett et al., 2008). Additional features such as the adaptive activities have also been found to help struggling students increase their learning outcomes (Van Campenhout et al., 2020). ...
... The courseware used in this study was initially generated by an artificial intelligence-based process called SmartStart (Dittel et al., 2019). This process uses an e-textbook as the corpus and applies natural language processing and machine learning techniques to identify learning objectives, divide the content into lessons aligned to learning objectives, and apply an automatic question generation process to create formative practice and feedback for each lesson (Jerome et al., 2020). For this psychology course, a Psychology of Sex and Gender (Bosson et al., 2019) textbook was used for the SmartStart process. ...
... Advances in educational technology are increasingly beneficial to learning, yet increasingly complex in nature. Courseware is one such digital tool, which is designed to provide a comprehensive learning environment for students and real-time data insights to instructors [19]. The creation of tools such as courseware, however, is a daunting task to undertake. ...
... Each unit contains an introduction, up to three modules of subtopic content, and a summary. Each module contains an adaptive activity and a quiz on the content from that module, and each unit summary contains a unit test cumulative to all modules in that unit [19]. ...
Conference Paper
Full-text available
There is a dire need for replication research in the learning sciences, as methods put forth for increasing student learning should be unequivocally grounded in reproducible, reliable research. Learning science research is not only a critical input in the learning engineering process during the development of educational technology tools, such as courseware, but also as an output after student data have been analyzed to determine if the learning methods used were effective for students in their natural learning context. Furthermore, research that can provide causal evidence that a method of learning is effective for students should be reproduced-and the generality for its use expanded-so that methods that cause learning gains can be widely applied. One such method is the doer effect: the principle that students who engage with more practice have higher learning gains than those who only read expository text or watch video. This effect has been shown to be causal in prior research through statistical modeling using data mined from natural learning contexts. The goal of this paper is to replicate this research using a large-scale data set from courseware used at a major online university. The learning-by-doing data recorded by the courseware platform were combined with final exam data to replicate the statistical model of the causal doer effect study. Results from this analysis similarly point to a causal relationship between doing practice and learning outcomes. The implications of these doer effect results and future learning science research using large-scale data analytics will be discussed.
... Data were collected by the platform as students engaged with the courseware, and these data were then analyzed during the investigation phase. The data analysis of this investigation stage provided examples of effective learning methods such as adaptivity [16] and replication of findings of previous research on the learn by doing method [17]. The investigation phase also uncovered areas in need of improvement, which created additional iterations of the learning engineering process. ...
Conference Paper
Full-text available
Learning engineering provides both a practice and process for solving educational challenges. While the circumstances of each challenge require a unique application of learning engineering, the learning engineering process was designed in such a way to provide guidance across a broad range of contexts. In this paper, the learning engineering process is articulated from the perspective of the developers of online courseware used in higher education. Within this use-case, we exemplify how an initial learning engineering process for the creation of the courseware provided a starting point for iteration, and in this instance, the beginning of an entirely new process on instructor enactment of that courseware. Whereas the initial challenge was to develop the courseware environment, this emergent challenge now focuses on understanding and addressing contextual factors that affect the successful instructor application of the courseware learning environment at scale.
... Adaptive activities were written for the most challenging chapters of content in both courses. Designed to scaffold based on each student's predictive learning estimate, the adaptive activities have been shown to improve outcomes (Van Campenhout et al., 2020). The faculty and instructional designers wrote the adaptive activities to assist students with the most challenging content where they would most need scaffolded support. ...
Article
Full contents of the issue
... Adaptive activities were written for the most challenging chapters of content in both courses. Designed to scaffold based on each student's predictive learning estimate, the adaptive activities have been shown to improve outcomes (Van Campenhout et al., 2020). The faculty and instructional designers wrote the adaptive activities to assist students with the most challenging content where they would most need scaffolded support. ...
Article
Full-text available
... Adaptive activities were written for the most challenging chapters of content in both courses. Designed to scaffold based on each student's predictive learning estimate, the adaptive activities have been shown to improve outcomes (Van Campenhout et al., 2020). The faculty and instructional designers wrote the adaptive activities to assist students with the most challenging content where they would most need scaffolded support. ...
Article
Full-text available
While research in the learning sciences has spurred advancements in educational technology, the implementation of those learning resources in natural learning contexts advances teaching and learning. In this paper, two faculty members at the University of Central Florida used courseware generated with artificial intelligence as the primary learning resource for their students. The selection and enhancement of this courseware is contextualized for each course. Instructor implementation practices over multiple semesters are described and related to resulting student engagement and exam scores. Finally, benefits of the adaptive courseware are discussed not only for student outcomes, but the qualitative changes faculty identified and the impact that iterative changes in teaching practice had on instructors as well as students.
Chapter
Innovation and advancements have led to the ability of higher education administrators and innovators to use machine learning (ML) to identify student academic risk behavior patterns at increasingly early points within a semester. These models bring with them the promise to help prioritize allocation of finite resources and inform scalable interventions to promote learner success. However, it may be more difficult to prioritize student needs when the measures for which a university is held accountable and use ML to predict are not specific to learning. How do we best navigate the ethical waters to emphasize and support student growth while simultaneously addressing business reporting needs? To begin this transformation, it’s critical that we gather better, more meaningful direct measures to build the models we use to predict outcomes, even if it means sacrificing some level of predictive validity, and then use our intervention strategies to improve these specific behavioral inputs feeding the models.KeywordsLearning AnalyticsNudgingMachine LearningEthics
Chapter
Full-text available
Advances in artificial intelligence and automatic question generation have made it possible to create millions of questions to apply an evidence-based learn by doing method to thousands of e-textbooks, an unprecedented scale. Yet the scaling of this learning method presents a new challenge: how to monitor the quality of these automatically generated questions and take action as needed when human review is not feasible. To address this issue, an adaptive system called the Content Improvement Service was developed to become an automated part of the platform architecture. Rather than adapting content or a learning path based on student mastery, this adaptive system uses student data to evaluate question quality to optimize the learning environment in real time. In this paper, we will address the theoretical context for a platform-level adaptive system, describe the methods by which the Content Improvement Service functions, and provide examples of questions identified and removed through these methods. Future research applications are also discussed.KeywordsContent Improvement ServiceAdaptive instructional systemsIterative improvementGrey-box systemsArtificial intelligenceAutomatic question generation
Article
Full-text available
The printing press long ago and the computer today have made widespread access to information possible. Learning theorists have suggested, however, that mere information is a poor way to learn. Instead, more effective learning comes through doing. While the most popularized element of today's MOOCs are the video lectures, many MOOCs also include interactive activities that can afford learning by doing. This paper explores the learning benefits of the use of informational assets (e.g., videos and text) in MOOCs, versus the learning by doing opportunities that interactive activities provide. We find that students doing more activities learn more than students watching more videos or reading more pages. We estimate the learning benefit from extra doing (1 SD increase) to be more than six times that of extra watching or reading. Our data, from a psychology MOOC, is correlational in character, however we employ causal inference mechanisms to lend support for the claim that the associations we find are causal.
Book
This graduate-level textbook is a tutorial for item response theory that covers both the basics of item response theory and the use of R for preparing graphical presentation in writings about the theory. Item response theory has become one of the most powerful tools used in test construction, yet one of the barriers to learning and applying it is the considerable amount of sophisticated computational effort required to illustrate even the simplest concepts. This text provides the reader access to the basic concepts of item response theory freed of the tedious underlying calculations. It is intended for those who possess limited knowledge of educational measurement and psychometrics. Rather than presenting the full scope of item response theory, this textbook is concise and practical and presents basic concepts without becoming enmeshed in underlying mathematical and computational complexities. Clearly written text and succinct R code allow anyone familiar with statistical concepts to explore and apply item response theory in a practical way. In addition to students of educational measurement, this text will be valuable to measurement specialists working in testing programs at any level and who need an understanding of item response theory in order to evaluate its potential in their settings. • Combines clearly written text and succinct R code • Utilizes a building-block approach from simple to complex, enabling readers to develop a clinical feel for item response theory and how its concepts are interrelated • Includes downloadable R functions that implement various facets of item response theory Frank B. Baker, Ph.D., is Professor Emeritus of the Department of Educational Psychology at the University of Wisconsin-Madison. He is author of numerous publications dealing with item response theory and statistical methodology. He received his B.S., M.S., and Ph.D. degrees from the University of Minnesota, Minneapolis. Seock-Ho Kim, Ph.D., is Professor in the Department of Educational Psychology at the University of Georgia. He is author of numerous publications in psychometrics and applied statistics and is a member of the American Educational Research Association, the American Statistical Association, the National Council on Measurement in Education, and the Psychometric Society, among other organizations. He received his B.A. from Korea University and his M.S. and Ph.D. degrees from the University of Wisconsin-Madison.
Conference Paper
The "doer effect" is an association between the number of online interactive practice activities students' do and their learning outcomes that is not only statistically reliable but has much higher positive effects than other learning resources, such as watching videos or reading text. Such an association suggests a causal interpretation--more doing yields better learning--which requires randomized experimentation to most rigorously confirm. But such experiments are expensive, and any single experiment in a particular course context does not provide rigorous evidence that the causal link will generalize to other course content. We suggest that analytics of increasingly available online learning data sets can complement experimental efforts by facilitating more widespread evaluation of the generalizability of claims about what learning methods produce better student learning outcomes. We illustrate with analytics that narrow in on a causal interpretation of the doer effect by showing that doing within a course unit predicts learning of that unit content more than doing in units before or after. We also provide generalizability evidence across four different courses involving over 12,500 students that the learning effect of doing is about six times greater than that of reading.
Article
A perennial problem for language testers is the need to construct and select test items with 'good' properties. The difficulty lies in the need to assess the properties of items by trying them out on a sample of subjects whose abilities, in turn, it ought to be possible to measure by observing their response to the items. This paper discusses the more important concepts of item response theory (IRT) - a technique, or set of tech niques, developed over the last 25 years, mainly by psychometricians. (An application of IRT was discussed in a recent issue of this journal (Henning, (1984).) Basic concepts are introduced and their implications considered by concentrating on the simplest IRT tool, the Rasch (1960) Model.
Book
A revision will be coming out in the next few months.
Article
Teaching strategies like modeling, feedback, questioning, instructing, and cognitive structuring are applications of Vygotsky's Zone of Proximal Development. These strategies "scaffold" student learning from assistance by others to self-learning toward the goal of internalization. This higher-order learning stems from interactions with those who have more knowledge than the learner. Practical applications of Vygotsky's theory to any basic nursing education program are presented.
Improving students' learning with effective learning techniques: promising directions from cognitive and educational psychology
  • J Dunlosky
  • K Rawson
  • E Marsh
  • M Nathan
  • D Willingham
Dunlosky, J., Rawson, K., Marsh, E., Nathan, M., Willingham, D.: Improving students' learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychol. Sci. Public Interest 14(1), 4-58 (2013). https://doi.org/10. 1177/1529100612453266