Content uploaded by Benny G. Johnson
Author content
All content in this area was uploaded by Benny G. Johnson on May 16, 2021
Content may be subject to copyright.
The Impact of Adaptive Activities in Acrobatiq
Courseware - Investigating the Efficacy
of Formative Adaptive Activities on Learning
Estimates and Summative Assessment Scores
Rachel Van Campenhout, Bill Jerome, and Benny G. Johnson
(&)
Acrobatiq by VitalSource, Pittsburgh, PA, USA
benny@acrobatiq.com
Abstract. The purpose of this paper is to explain the learning methodologies
behind the adaptive activities within Acrobatiq’s courseware, and to investigate
the impact of these adaptive activities on learning estimates and summative
assessment scores using real course data. The adaptive activities used for this
analysis were part of a Probability and Statistics course, which was delivered to
college students at a public four-year institution as part of an educational grant.
The data were analyzed to identify if the adaptive activities had an impact on
learning estimates as well as on summative assessment scores. Results showed
that the adaptive activities had a net positive effect on learning estimates. Results
also showed that not only did learning estimate states correlate to mean sum-
mative assessment scores, but improving learning estimates after completing the
adaptive activity practice yielded higher mean summative assessment scores.
The implications of this analysis and future research are discussed.
Keywords: Adaptive activities Doer Effect Formative practice Learn
by doing Learning modeling Learning objectives Learning outcomes
Scaffolded practice
1 Introduction
One goal of this paper is to illuminate the learning theory used to develop the adaptive
activities in Acrobatiq’s courseware. The adaptive activities require many inputs from a
complex learning environment in order to adapt appropriately for each student. Within
the courseware, learning objectives are used to organize both content and formative
practice within lessons. As students answer the formative questions, their data are sent
to Acrobatiq’s predictive model and a learning estimate is generated for each student
against each learning objective. The adaptive activities use these learning estimates to
select the appropriate scaffolded questions against each learning objective for the
students. Each student will receive a set of questions with the appropriate scaffolding
specifically selected for their needs.
A second goal of this paper is to analyze data in order to identify the impact these
activities have for students. Impact could mean different things in this environment,
©Springer Nature Switzerland AG 2020
R. A. Sottilare and J. Schwarz (Eds.): HCII 2020, LNCS 12214, pp. 543–554, 2020.
https://doi.org/10.1007/978-3-030-50788-6_40
which leads to two different research questions. The first of those questions is: do the
adaptive activities increase learning estimates for students? Students have learning
estimates for each learning objective included in the module for which the activity is
written. The questions within the activity are also formative in nature, and therefore can
contribute to that student’s learning estimate. This research question could use within-
student data as well as between-student data. We will look to see if students increase
their learning estimates after completing the adaptive activities. We will also look to
see if completion of the adaptive activities changes learning estimates differently
between groups of students who have different learning estimates. Students with low
learning estimates will have more scaffolded practice and may be more likely to
increase their learning estimates than those students who have already been successful
and have high learning estimates.
In addition to learning estimates, another measure to investigate is the impact of the
adaptive activities on summative assessment scores. These activities are placed
immediately before module quizzes as a last scaffolded practice before students take the
scored assessment. The second research question is: do the adaptive activities increase
student scores on summative assessments? This research question investigates the
summative scores of students in different learning estimate categories to identify the
differences between groups. This analysis will also look at summative scores for stu-
dents who increased their learning estimates (after working through the adaptive
activity), and students who did not.
The ultimate purpose of the adaptive activities is to assist the learner in their
progression through the courseware and in mastery of the content. The significance of
the data analysis against both research questions would verify that the purpose of these
activities is being met. An increase in learning estimates after completing the activities
would indicate that the learning methodology behind the activities is working. While
the learning estimate is a metric both created and used by the Acrobatiq platform,
improvement in it could indicate the courseware and design of the adaptive activities
are functioning as designed. Changes in summative assessment scores after completing
the adaptive activities would be a significant finding for these activities. Summative
assessments produce scores which are part of the student’s gradebook. To assist in
improving student scores, and therefore possibly their course grades, would be
incredibly valuable to students. The analysis and results of this paper indicate how
future research could be conducted to verify results, as well as provide new ideas on
how to continue to help students master content through adaptive experiences.
2 Learning Methodology
The purpose of this section is to outline the learning methodology behind the design of
the Acrobatiq courseware used in this analysis, as the course features and reasoning for
them are key to the investigation. The adaptive activities are integrated into the
courseware content and require specific instructional design practices in order for them
to properly adapt for each learner.
Formative Practice. The first requirement is the inclusion of formative practice
questions for each learning objective in a module of content. The formative practice
544 R. Van Campenhout et al.
questions can have a variety of formats (such as multiple choice, text input, equation,
drag and drop, etc.), provide immediate targeted feedback for each answer option, and
allow students to continue answering until they get the correct answer. Formative
practice is a well-established technique shown to increase learning gains for students of
all ages, and across subjects [1]. Moreover, studies have shown that formative
assessment can raise achievement for low performing students most of all, while
improving learning for all [1]. The formative practice questions distributed throughout
the text and the adaptive activity also act as no- or low-stakes practice testing. Practice
testing increases learning gains and retention, and including feedback has been shown
to outperform practice testing without feedback [2].
Formative practice is integrated with the content of the Acrobatiq Probability and
Statistics course. The lessons in the courseware begin with the learning objective and
are followed by the relevant content and formative practice opportunities, which are
interspersed throughout the lesson. This chunking method of following short sections
of content with formative practice is key for the learn by doing approach, which
produces the Doer Effect [3–5]. Research utilizing interactive courseware from Car-
negie Mellon’s Open Learning Initiative shows that students who did more interactive
activities had a learning benefit approximately six times that of reading text and three
times that of watching video [3]. Follow-up analysis showed this relationship between
doing and learning to be causal [4], and this finding has been replicated in our previous
work [5].
Learning Estimates. While the formative questions do not produce a grade, the
student’s responses impact his or her learning estimate for that learning objective. The
learning estimate is a predictive measure generated by Acrobatiq’s analytics engine for
each student on each objective to estimate how well a student will perform on the
learning objective’s summative assessment. This learning estimate is required for the
adaptive activity to adapt for the learner (see below).
The machine learning model underlying the learning estimate uses item response
theory (IRT) [6,7] to construct an estimate of the student’s ability for the objective,
from which a prediction can be made of performance on the objective’s summative
assessment. An advantage of an IRT-based approach is it can take the psychometric
properties of the questions into account when constructing the ability estimate. A two-
parameter logistic model is used, which models difficulty and discrimination for each
question. A Bayesian approach [8] is used to estimate the posterior distributions of the
IRT question parameters from data, as well as the student ability posterior distribution
from the formative and adaptive questions answered. From the ability posterior a
numerical learning estimate value between 0 and 1 is derived (higher values indicating
better expected performance). When the model has sufficient confidence in its pre-
diction based on the data available to it, a category of low, medium or high is also
assigned; otherwise the category is labeled as unknown.
Adaptive Activities. The adaptive activities in the Probability and Statistics course-
ware were designed by instructional designers and subject matter experts to include
scaffolded questions against each learning objective in the module. The goal of the
adaptive activity is to provide students with the appropriate level of scaffolding for their
needs. In this activity, questions are written against the learning objectives from the
The Impact of Adaptive Activities in Acrobatiq Courseware 545
module at three increasing levels of difficulty (low, medium, and high). By organizing
the questions—and content—in this way the adaptive activities align with Vygotsky’s
zone of proximal development, which structures content and interactions in such a way
as to meet the learner at their level of understanding and build upon it [9]. Providing
struggling students with foundational questions as scaffolds to more challenging
questions helps to reduce cognitive load in a similar way as worked examples [10].
At the start of the adaptive activity, the platform identifies the student’s learning
estimate for each learning objective used in the activity. The learning estimate deter-
mines the level of scaffolding to deliver. A student with a low learning estimate on a
learning objective did poorly on the formative practice, and therefore is given the
additional scaffolded practice for the learning objective. A student with a high learning
estimate did well on the formative practice for the learning objective, and therefore are
only delivered the highest difficulty—or core—questions. It is important to note that
while the previous formative practice informed the learning estimates which deter-
mined how the activity adapted to the student, the adaptive activity itself also con-
tributes to the learning estimate, asthe adaptive activities are also formative in nature.
3 Methodology
The Courseware. The course used for this analysis was a Probability and Statistics
courseware developed as an introduction to the subject at the university level. This
Acrobatiq courseware was originally developed at the Open Learning Initiative at
Carnegie Mellon University and has been substantially revised over the years as an
Acrobatiq course. Included in those revisions were the addition of the adaptive ele-
ments, new summative assessments, and a new learning model, which are all critical
elements to this analysis. The Probability and Statistics courseware included 5 units of
content with a combined 10 total content modules within them. The lessons within each
module begin with a learning objective and typically have interleaved content and
formative questions. Learning objectives appear at the top of the lesson, are student
centered and measurable, and all content and formative questions align to and are
tagged with them. Formative questions are intended as learning opportunities and
therefore do not create a grade. Students receive immediate feedback for formative
questions and have the opportunity to continue to answer until they choose the correct
response. Each module ends with an adaptive activity and a quiz which contain
questions against the learning objectives covered in the module. The adaptive activity
is a formative activity as well by providing immediate feedback and multiple attempts,
however it does produce a completion score for the gradebook. The quiz is a sum-
mative assessment; students do not receive immediate feedback and it produces a score
for the gradebook.
The courseware elements necessary for this analysis include each students’for-
mative question attempts and accuracy, learning estimate states, adaptive activity
attempts and accuracy, and summative assessment attempts and accuracy. These data
points provide a window into each student’s journey through the courseware.
546 R. Van Campenhout et al.
The Population. This course was delivered to students at a large public 4-year
institution in the fall 2018 semester. There were no experimental manipulations used on
this student population, and the platform does not collect demographic information on
students, so no analysis according to demographic information would be possible. The
data set includes numeric identifiers for individual students, for which all the students’
interactions are recorded against. The data collected will be used only for analysis of
the relationship of each student’s interactions with the adaptive activities in relation to
the formative and summative activities.
The Data Set. The data set includes 306 students and 47 learning objectives. The unit
of analysis was student-learning objective pairs, i.e. the data records corresponded to a
single student’s work on a single learning objective. Out of the 306 47 = 14,382
possible student-learning objective combinations, there were 12,612 combinations with
data. Each data record contained: number of formative questions answered; learning
estimate value and category after formative questions; number of adaptive questions
answered; learning estimate value and category after adaptive questions; number of
summative questions answered; and mean summative question scores. Not all records
contained formative, adaptive, and summative data; for example, in some cases stu-
dents answered adaptive and summative but not formative questions.
This original data set was reduced to include only learning objectives with forma-
tive, adaptive, and summative attempts. For some learning objectives, the number of
formative questions contained in the course was not consistent with best practices of
course design and/or the requirements of the learning model. As such, learning
objectives were filtered to those having a minimum of 5 formative questions and a
maximum of 45 formative questions. The data set was also cleaned to remove errors in
data collection or storage. The final data set included 21 learning objectives, 300
students, and 5,971 total records.
The data analysis was performed on both the full original data set as well as this
reduced data set. The reduced data set was chosen for presentation to attempt to obtain
the clearest picture of the relationships between student practice and summative
assessment performance. However, the qualitative conclusions of the analysis were
consistent between both data sets.
4 Results
4.1 Research Question 1
The first analysis done was to address the first research question: do the adaptive
activities increase learning estimates for students? The adaptive activities were
designed to provide scaffolded questions personalized for each student’s learning
estimate, with the goal of assisting students who needed additional help before they
took the high-stakes summative assessment. Therefore, we hypothesized the adaptive
activities would have a positive impact on learning estimates for some portion of
students.
The Impact of Adaptive Activities in Acrobatiq Courseware 547
Overall Learning Estimate Changes. To answer this question, the data set was
analyzed with regard to whether or not student learning estimates before the adaptive
activity changed after students completed the adaptive activity questions. There were
3,972 cases in which a learning estimate was available immediately before and after
adaptive practice, with a mean learning estimate change of 0.062 (0.169). In 2,550
instances (64.2%), the learning estimate was increased by adaptive practice, with a
mean increase of 0.132 (0.167). In the remaining 1,422 instances, the learning estimate
decreased by a mean of 0.064 (0.069). A Shapiro-Wilk test showed that the learning
estimate differences were not normally distributed, so a one-sample Wilcoxon signed
rank test was used to check if the median learning estimate change was of the popu-
lation was statistically significantly different than 0, which it was (p⋘0.001).
These results support the hypothesis that the adaptive activity had a net positive
impact on learning estimates for the majority of students. The smaller portion of
students whose learning estimates decreased could be explained by considering those
students who may have been near the threshold of a learning estimate category. Getting
questions wrong in the adaptive activity could have shifted their learning estimate
down to the lower category (Table 1).
Learning Estimate Changes within Categories. In addition to understanding how
the student learning estimates increased or decreased as a whole, we also investigated
how learning estimates changed for students within different learning estimate cate-
gories. The learning estimates for students determine which questions are delivered in
the activity—whether or not students need the scaffolded questions, and at which
difficulty level. This analysis will reveal if these different groups of students changed
learning estimates differently after completing the adaptive activity. There was a total
of 3,954 records for which student summative scores were also available for the
learning objectives included in this analysis. A Kruskal-Wallis Htest was performed on
Table 1. The descriptive statistics for the increase or decrease of learning estimates after the
completion of adaptive questions.
Statistic Learning estimate increase Learning estimate decrease
Count 2550.000000 1422.000000
Mean 0.131999 −0.064386
Std 0.167449 0.068799
Min 0.000013 −0.511564
25% 0.023794 −0.086208
50% 0.058838 −0.039807
75% 0.164347 −0.017508
Max 0.806394 −0.000008
548 R. Van Campenhout et al.
the summative scores in the different groups, which showed the groups do not all have
the same median (p⋘0.001) (Table 2).
The first data analyzed were those students who achieved high learning estimates for
learning objectives after they completed the formative questions. There was a total of
1,648 (41.7%) instances of students achieving high learning estimates after completing
the formative questions. After completing the adaptive activities, there were three
possible outcomes based on how they performed: retain high learning estimate, change
to medium learning estimate, or change to low learning estimate. Of the total high
learning estimate instances, 1,511 (91.69%) remained in the high category after
completing the adaptive activity questions. There were 112 (6.8%) instances of stu-
dents whose learning estimate changed to medium, and 25 (1.52%) instances of stu-
dents whose learning estimate changed to low. These results were consistent with
expectations for this group of students. Students who did well enough on the formative
practice to earn high learning estimates similarly did well enough on the adaptive
activity questions to retain that learning estimate category. For the less than 10% of
instances where learning estimates dropped to medium or low, this means students did
poorly enough on adaptive questions to lower their learning estimates. Likely these
students were just over the high learning estimate threshold and answering incorrectly
moved them to the lower category.
The next category of data analyzed was students who achieved medium learning
estimates for learning objectives after they completed the formative activities. There
was a total of 918 (23.22%) instances in this category. After completing the adaptive
activities, there were three possible outcomes based on how they performed: retain
medium learning estimate, change to high learning estimate, or change to low learning
estimate. Of the total medium learning estimate instances, 441 (48.04%) remained in
the medium category after completing the adaptive activity questions. There were 348
(37.91%) instances of students whose learning estimate changed to high, and 129
Table 2. The number of instances of learning estimate changes after completing adaptive
questions, grouped by learning estimate category.
Learning estimate
category
High (after
adaptive)
Medium (after
adaptive)
Low (after
adaptive)
Unknown (after
adaptive)
High (before
adaptive)
1511 112 25 NA
Medium (before
adaptive)
348 441 129 NA
Low (before
adaptive)
47 141 555 NA
Unknown (before
adaptive)
339 136 146 24
The Impact of Adaptive Activities in Acrobatiq Courseware 549
(14.05%) instances of students whose learning estimate changed to low. This medium
learning estimate category shows more change in learning estimate state than the high
category. These results are in line with expectations, as the medium category has
thresholds next to both the high and low category, and therefore students near those
thresholds could shift their states if they do well or poorly on the adaptive questions.
The next category of data analyzed were those students who achieved low learning
estimates for learning objectives after they completed the formative activities. There
was a total of 743 (18.79%) instances in this category. After completing the adaptive
activities, there were three possible outcomes based on how they performed: retain low
learning estimate, change to medium learning estimate, or change to high learning
estimate. Of the total low learning estimate instances, 555 (74.7%) remained in the low
category after completing the adaptive activity questions. There were 141 (18.98%)
instances of students whose learning estimate changed to medium, and 47 (6.32%)
instances of students whose learning estimate changed to high. This low learning
estimate category had fewer changes to other categories than the medium category
previously. However, while nearly 75% of students who struggled on learning
objectives continued to do so in the adaptive activity, just over 25% of instances show
students who were able to increase their learning estimates to medium or high, which is
a positive finding.
The final category of data to be analyzed was students who did not complete enough
formative practice against learning objectives to generate a learning estimate. This
unknown learning estimate category is not an indicator of ability, but rather a state of
the predictive analytics not having enough information to determine a category. Of the
645 (16.31%) instances of an unknown learning estimate, there were four categories of
learning estimates after the adaptive activity questions were completed: high, medium,
low, or unknown. The change in learning estimate states for these instances after the
adaptive activity questions were as follows: 339 (52.56%) changed to high, 136
(21.09%) changed to medium, 146 (22.64%) changed to low, and 24 (3.72%) remained
unknown. We cannot determine if the adaptive activity questions helped to shift the
final learning estimates, but the activity at least moved all but 3.72% of instances into
known categories.
4.2 Research Question 2
The second research question to investigate is: do the adaptive activities increase
student scores on summative assessments?While the changes in learning estimates is
one indicator that the adaptive activities have an impact on student learning, another
measure is the summative assessment scores which correspond with the same learning
objectives. In this Probability and Statistics course, the adaptive activity is always
placed before a summative quiz, with the goal of trying to help students prepare for this
scored assessment. The change for each learning estimate category is compared by
looking at the mean summative assessment score for each group (Table 3).
550 R. Van Campenhout et al.
The findings of this analysis showed both expected and unexpected results. Stu-
dents with higher learning estimates generally had higher mean summative scores.
Within original learning estimate categories, students who improved their learning
estimates after the adaptive questions did better on summative questions than students
who maintained their learning estimate category, and students who lowered their
learning estimate category after the adaptive questions did worse than their counterparts
who maintained their category. Mean summative scores also decreased slightly within
the post-adaptive learning estimate category depending on the pre-adaptive learning
estimate category. Mann-Whitney Utests—comparing median summative scores of the
low and medium category before the adaptive activity to their respective category
changes after the activity—found that all changes were significant. This indicates that
not only do learning estimates correlate to summative score performance, but students
who increase their learning estimates perform better on summative assessment
questions.
A surprising finding was the final mean summative assessment scores for the
unknown learning estimate category. Recalling the earlier changes in learning estimates
for this category: 339 (52.56%) changed to high, 136 (21.09%) changed to medium,
146 (22.64%) changed to low, and 24 (3.72%) remained unknown. Given the findings
above, the mean summative scores for the unknown to high, medium, and low learning
estimate categories do not align with the other mean summative scores for those
categories. The scores for unknown to high, medium, and low are all above 0.7. The
only other categories with mean summative scores above 0.7 were high to high, and
medium to high. So, despite which learning estimate category the unknown state
changed to, all performed as well as those with high learning estimates. The only
unknown category who did not perform as well were those who remained unknown, for
which they scored the second lowest of all categories.
4.3 Learn by Doing
During the data analysis, there were additional findings which were notable with regard
to the learning theory underlying the courseware’s design. Learn by doing is a key
principle which supported the addition of frequent formative practice. The adaptive
Table 3. Mean summative scores by learning estimate category before and after the adaptive
activity questions.
Learning Estimate
Category
High (after
adaptive)
Medium(after
adaptive)
Low (after
adaptive)
Unknown (after
adaptive)
High (before
adaptive)
0.775 0.689 0.617 NA
Medium (before
adaptive)
0.716 0.668 0.612 NA
Low (before
adaptive)
0.676 0.617 0.543 NA
Unknown (before
adaptive)
0.767 0.711 0.709 0.569
The Impact of Adaptive Activities in Acrobatiq Courseware 551
activity was similarly expected to help students learn as it was another formative
practice activity and gave students one more personalized chance to prepare for a quiz.
The analysis of the findings for summative assessment scores showed the correlation
between increased learning estimates and increased mean summative assessment
scores. This led us to review the mean summative assessment scores for students
grouped by whether they participated in formative and adaptive practice.
Table 4shows the breakdown of students who did or did not participate in formative
practice and the adaptive activity practice. We hypothesize that because the adaptive
activities produced a completion score, doing the adaptive practice was the largest
category for both doing and not doing formative practice. The instances of students who
did or did not do the formative practice are also broken down by whether they did or did
not do the adaptive activity practice. What we are able to see are the combinations of
practice from least to most, and the mean summative scores correlate with the amount of
practice completed. Not doing formative or adaptive practice produced the lowest mean
summative scores (0.566), while doing both the formative and adaptive practice pro-
duced the highest mean summative scores (0.694). A Kruskal-Wallis Htest was per-
formed on the groups and there was a statistically significant difference among them (p
⋘0.001). These findings are in line with those from previous research [3], which
identified correlational relationships between doing formative practice and summative
assessment scores. While testing for a causal relationship between practice and out-
comes requires additional data not available in this study, the findings of causality from
related research lead us to anticipate the same would be true in this instance [4,5].
5 Discussion
This study was critical in helping to understand the efficacy of the adaptive activities in
Acrobatiq’s courseware. The learning science and instructional design principles uti-
lized to create the course content were intended to help students better learn the
material and prepare them for high-stakes summative assessments. The findings which
show that not only do the adaptive activities help increase learning estimates for many
students, but that learning estimates correlate to mean summative scores validates the
primary function of the adaptive activities.
While not every learning estimate increased after completing the adaptive practice,
nearly 38% of students with medium learning estimate instances increased to high
Table 4. Mean summative scores for students grouped by participation status for formative and
adaptive practice.
Formative Adaptive Count Mean summative score
False False 146 0.5663
False True 1142 0.5744
True False 168 0.6098
True True 4411 0.6935
552 R. Van Campenhout et al.
learning estimates and over 25% of students with low learning estimate instances
increased to medium or high learning estimates. With increased learning estimates
correlating to increased mean summative assessment scores, assisting this percentage of
students through adaptive practice to increase learning estimates and summative scores
is a very positive finding. While final course gradingpolicies are up to instructors,
summative assessments produce a grade in the courseware gradebook which is often
part of the student’sfinal course grade. If the adaptive practice can improve summative
assessment scores for students in the low and medium learning estimate category, that
could make a difference in final course grades for those students.
This analysis also supported for the research team the principle that more “doing”is
better for students. The more formative practice students completed, the higher the
mean summative assessment scores. Students who did not do formative practice but did
do the adaptive practice scored slightly higher than those who did no practice at all
(note that when no formative practice is done, the adaptive engine has no data to adapt
the activity). Yet the adaptive practice had a larger effect when students also completed
the formative practice, thereby giving the adaptive practice a basison which to adapt.
Instructors could capitalize on the benefits of both the formative and adaptive practice
through their expectations and requirements of students when using the courseware.
Finding ways to actively enforce the completion of formative and adaptive practice
through classroom policies could increase the benefits of these activities for students.
The results of this analysis also show that these findings could be replicated in
many other courses. The Probability and Statistics courseware used did not have
perfect associations—or tagging—between the formative practice, adaptive practice,
and summative assessments. The data set was reduced to only include the data where
learning objectives had data for all three. Yet despite the imperfections in the
courseware itself, results were still very positive. This is encouraging, as developing
perfect courseware is a difficult task. Future research could aim to replicate findings
using different courseware for different subject domains, as well as courseware with
different instructional design practices.
The data used for the analysis in this paper was also gathered from a course where
no experimental controls were used. It is encouraging that this non-experimental, real-
world course showed such positive results. Additional validation of results could be
found by partnering with instructors in the future to compare variations in classroom
controls and practices. Initial plans for data analysis included comparing this data to a
course from a different semester which did not include the adaptive activities, but no
version could be found where the only substantive change was the adaptive practice.
Future research could also review individual adaptive activities to try to identify the
ideal number of scaffolded questions to deliver to students. The analysis of this data
revealed support for the Doer Effect, but does not indicate how much practice is ideal to
help students increase learning estimates and summative scores. Such research could
also analyze the difficulty levels of questions across formative, adaptive, and sum-
mative assessments to determine the most beneficial combination and frequency of
difficulty levels. The primary function of the adaptive activities is to support student
learning. Additional research into ideal number and types of questions to deliver to
students could potentially improve learning.
The Impact of Adaptive Activities in Acrobatiq Courseware 553
References
1. Black, P., William, D.: Inside the black box: raising standards through classroom
assessment. Phi Delta Kappan 92(1), 81–90 (2010). https://doi.org/10.1177/003172171009
200119
2. Dunlosky, J., Rawson, K., Marsh, E., Nathan, M., Willingham, D.: Improving students’
learning with effective learning techniques: promising directions from cognitive and
educational psychology. Psychol. Sci. Public Interest 14(1), 4–58 (2013). https://doi.org/10.
1177/1529100612453266
3. Koedinger, K., Kim, J., Jia, J., McLaughlin, E., Bier, N.: Learning is not a spectator sport:
doing is better than watching for learning from a MOOC. In: Learning at Scale, pp. 111–120.
Vancouver, Canada (2015). http://dx.doi.org/10.1145/2724660.2724681
4. Koedinger, K., McLaughlin, E., Jia, J., Bier, N.: Is the doer effect a causal relationship? How
can we tell and why it’s important. learning analytics and knowledge. Edinburgh, United
Kingdom (2016). http://dx.doi.org/10.1145/2883851.2883957
5. Olsen, J., Johnson, B.G.: Deeper collaborations: a finding that may have gone unnoticed.
Paper Presented at the IMS Global Learning Impact Leadership Institute, San Diego, CA
(2019)
6. Baker, F.: The basics of item response theory. In: ERIC Clearinghouse on Assessment and
Evaluation. College Park, MD (2001). http://echo.edres.org:8080/irt/baker/
7. Embretson, S., Reise, S.: Item Response Theory for Psychologists. Erlbaum, Mahwah (2000)
8. Fox, J.: Bayesian Item Response Modeling: Theory and Applications. Springer, Heidelberg
(2010). https://doi.org/10.1007/978-1-4419-0742-4
9. Sanders, D., Welk, D.: Strategies to scaffold student learning: applying Vygotsky’s zone of
proximal development. Nurse Educ. 30(5), 203–204 (2005)
10. Sweller, J.: The worked example effect and human cognition. Learn. Instr. 16(2), 165–169
(2006). https://doi.org/10.1016/j.learninstruc.2006.02.005
554 R. Van Campenhout et al.