What learning analytics based prediction models tell us about feedback preferences of students

Abstract

Learning analytics (LA) seeks to enhance learning processes through systematic measurements of learning-related data and to provide informative feedback to learners and educators. This study examines the use of preferred feedback modes in students by using a dispositional learning analytics framework, combining learning disposition data with data extracted from digital systems. We analyze the use of feedback of 1062 students taking an introductory course mathematics and statistics, enhanced with the digital tools. Our findings show that compared with hints, fully worked-out solutions demonstrate a stronger effect on academic performance and act as a better mediator between learning dispositions and academic performance.

Full text

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
What learning analytics based prediction models tell us about feedback preferences of
students
Quan Nguyen, Open University UK, Institute of Educational Technology
Dirk T. Tempelaar, Maastricht University School of Business and Economics
Bart Rienties, Open University UK, Institute of Educational Technology
Bas Giesbers, Rotterdam School of Management, Information Management and Consulting
Abstract
Learning analytics (LA) seeks to enhance learning processes through systematic
measurements of learning related data and to provide informative feedback to learners and
educators (Siemens & Long, 2011). This study examined the use of preferred feedback
modes in students by using a dispositional learning analytics framework, combining
learning disposition data with data extracted from digital systems. We analyzed the use of
feedback of 1062 students taking an introductory mathematics and statistics course,
enhanced with digital tools. Our findings indicated that compared with hints, fully worked-
out solutions demonstrated a stronger effect on academic performance and acted as a
better mediator between learning dispositions and academic performance. This study
demonstrated how e-learners and their data can be effectively re-deployed to provide
meaningful insights to both educators and learners.
Keywords: blended learning; dispositional learning analytics; e-tutorials; learning
feedback; learning dispositions; higher education; problem solving; STEM

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Introduction
In educational settings, an enormous volume of potentially valuable information is
generated by both students and educators. Such information may include academic performance,
tracking data from online learning environments, emails and social network data. In recent years,
the term “learning analytics” has emerged as educational institutions and corporate learning
started to harness this wealth of information to provide real time feedback to students while
offering valuable insights for educators to improve teaching quality (Siemens, Dawson, &
Lynch, 2013). In the corporate world, LA can help learning and development of professionals by
identifying successful learning activities and patterns, with clear indications of the learning
progress of its employees. In a higher education context, students and teachers may benefit from
personalized and adaptive learning experiences (Knewton, 2016). To better catalyze the
processes of learning for individuals and collectives, Buckingham Shum and Crick (2012) have
proposed a dispositional learning analytics infrastructure that combines learning activity
generated data with learning dispositions, values and attitudes measured through self-report
surveys which are fed back to students and teachers through visual analytics. Tempelaar,
Rienties, and Giesbers (2015) have investigated the predictive power of learning dispositions,
outcomes of continuous formative assessments, and other system generated data on modeling
student performance and their potential to generate informative feedback. The study found that
computer-assisted formative assessments can best detect underperforming student and academic
performance.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
In learning theory, monitoring and evaluation play a crucial role as they provide feedback
on how activities coordinate across several stages of studies (task definition, goal setting and
planning, and enacting study tactics and strategies) (Winne & Hadwin, 1998). Feedback assesses
the level of understanding of learners and can provide cues for reinforcement. In a meta-study by
Hattie (2013), feedback is considered one of the most powerful tools in enhancing the learning
experience. In the past, traditional formal feedback is limited in the form of a grade, which is
available only after finishing all learning activities. However, the involvement of educational
technology allows us to gather feedback on learning-in-progress activities, which provides a real-
time assessment to both students and teachers. For instance, a study by Duffy and Azevedo
(2015) revealed that students in the “prompt and feedback” condition deployed more self-
regulated learning strategies and spent more time viewing relevant science material compared to
students in the control condition in which learners did not receive any support. Additionally,
McLaren, van Gog, Ganoe, Karabinos, and Yaron (2016) categorized different feedback modes
into worked examples, erroneous examples, tutored problems, and problem solving. Their study
showed clear efficiency benefits of the use of worked examples in a web-based learning
environment: equal levels of test performance were achieved, with significantly less investment
of time and effort during learning. Given the importance of feedback and the advancement in
assessment technology, the investigation of the effects of feedback use by students on their
academic performance suggests being a promising research trajectory in learning analytics.
This study examines how learning dispositions and feedback preferences affect academic
performance. The article is organized as follows. The next section (Section 2) introduces the

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
context of the study and its instruments. This is followed by Section 3, which presents the
results, and is followed by the discussion in section 4. Finally, section 5 concludes the study and
discusses the implications of big data in education and learning analytics (LA) for online
learners/instructors, and how this study bridges the gap between existing LA literature and
pedagogy.
Research design
Data source
The educational system in which students learn mathematics and statistics is best
described as a ‘blended’ or ‘hybrid’ system. The main component is a face-to-face instruction
that employs problem-based learning (PBL), in small groups (14 students), as an instructional
strategy. As part of the PBL approach. Learners are coached by a content expert tutor (Schmidt,
Van der Molen, Te Winkel, & Wijnen, 2009). Participation in these tutorial groups is required, as
is the case for all courses based on the Maastricht PBL system. Within the online component of
the blended learning, students can optionally make use of the two e-tutorials Sowiso
(mathematics) and MyStatLab (statistics) (Tempelaar, Heck, Cuypers, van der Kooij, & van de
Vrie, 2013; Tempelaar et al., 2015). This choice is based on the philosophy of student-centered
education, placing the responsibility for making educational choices primarily on the student.
However, the use of e-tutorials and achieving good scores in the practicing modes of the MyLab
environments is stimulated by making bonus points available for good performance in the
quizzes. Quizzes are taken every two weeks and consist of items that are drawn from the same

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
item pools applied in the practicing mode. We chose this particular constellation as it stimulates
students with limited prior knowledge to make intensive use of the MyLab platforms. The bonus
is maximized to 20% of what one can score on the exam.
The student-centered characteristic of the PBL-based instructional model requires, first
and foremost, adequate, informative feedback to students so that they are able to monitor their
study progress and their topic mastery in absolute and relative sense. The provision of relevant
feedback starts on the first day of the course when students take a diagnostic entry test for
mathematics. Feedback from this entry test provides a first signal of the importance of using the
digital learning platforms made available to the students. Next, the Sowiso and MyStatLab
environments take over the monitoring function: at any time, students can see their progress in
preparing the next quiz, get feedback on the performance in completed quizzes, and on their
performance in the practice sessions.
Participants in this study are 1069 students in a blended introductory quantitative course at a
public university in the Netherlands during 2015-2016. A large diversity in the student population
is present: only 24% were educated in the Dutch high school system. The largest proportion,
46% of the students, was educated according to the German Abitur system. High school systems
in Europe differ strongly, most particularly in the teaching of mathematics and statistics.
Therefore, it is crucial that the first module offered to these students is flexible and allows for
individual learning paths (Tempelaar et al., 2013; Tempelaar et al., 2015). In the investigated
course, students work an average 10 hours in Sowiso, and 25 hours in MyStatLab, which
represents 12.5% to 31% of the available time of 80 hours for learning on both topics.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Instruments and procedure
In this empirical study, we investigate the relationships between course performance
measures, learning management system (LMS) trace variables, student information system (SIS)
based variables, and learning disposition variables measured in six self-report surveys. Most
learning dispositions incorporated in this study are assumed to be relative context-independent.
Examples of such are attitudes and learning styles. These are relative stable constructs, not
impacted by the specific learning activity the student is in: trait-like type of variables. For that
reason, these self-report surveys were all administered at the start of the course, to make their
data available as early as possible. On the other hand, learning emotions are context-dependent:
they relate to emotions of students in specific learning activities. These state-like variables
cannot be measured at the start of the course, since students need to have sufficient experience
with the learning context in order to be able to assess their contextual learning emotions. To
differentiate between test emotions and learning emotions, the measurement should also not take
place too late in the course, and therefore, we opted to do so exactly half way the course. Thus, it
gives students sufficient experience with the topics and the learning activities, without being in
danger that the approaching exam would strongly impact learning emotions. In the subsections
that follow, several instruments are described to provide the groundwork for our analysis.
Course Performance Measures
The ultimate aim of the predictive modeling endeavor is to understand how student
dispositions and learning activity relate to four relevant course performance measures:

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
performance on the exam, both for mathematics (MathExam) and statistics (StatsExam), and the
aggregated bonus for both topics, which was based on performance in the three quizzes:
MathBonus and StatsBonus, combined with the mastery level achieved in the e-tutorials for each
topic: MathMastery and StatsMastery.
LMS Trace Data
Three different digital systems have been used to organize the learning of students, and to
facilitate the creation of individual learning paths: BlackBoard (learning management system),
and the two e-tutorials Sowiso and MyStatLab (MSL). Students worked in the two e-tutorials for
all seven weeks, practicing homework exercises selected by the module coordinator. The e-
tutorial systems track the mastery score achieved in each task, which is measured as the number
of successful attempts, (MathMastery and StatsMastery), time on task (MathHours and
StatsHours), the total number of attempts required to get to the mastery level achieved
(MathAttempts and StatsAttempts), the number of fully worked-out solutions called for
(MathSolutions and StatsSolutions), and the number of hints asked for (MathHints and
StatsHints). In this study, feedback preferences imply the use of fully worked-out solutions and
the use of hints. Overall, students who see more fully worked-out solutions, and who ask for
more hints, perform better. These data were aggregated over the on average 25 weekly tasks for
mathematics, and about 20 tasks for statistics, to produce ten predictor variables, five for each
topic, for each of the seven weeks, and next, aggregated over all seven education weeks. Less
aggregated data sets have been investigated, but due to high collinearity in data of individual
tasks, these data sets produced less stable prediction models.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
The preliminary results from this study suggest that the five types of track data for both
topics appear to be collinear: in general, active students spend more time in the e-tutorials,
making more attempts, achieving higher mastery, and in doing so, they use more hints and
examples. Due to this collinearity, the added value of time on task and number of attempts in
predicting course performance appeared to be minimal, with mostly non-significant betas.
Therefore, in the final version of prediction models, only mastery level, the use of hints and the
number of examples were included. In this article, we are particularly interested in which factors
influence the way students use feedback (fully worked-out solutions versus hints), and how
different feedback modes can help to explain students’ academic performance.
SIS System Data
The Maastricht University SIS provided four further variables which are used as controls.
Standard demographic variables are Gender (an indicator variable for female students),
Studytrack (Economics and Business Economics, Fiscal Economics, and International Business)
and MathMajor (indicator for the advanced mathematics track in high school). Distinguishing
between national and international students is key, given the strong focus on statistics in the
Dutch high school system (with a large variation in other countries, but never as extreme as in
the Dutch case). The MathMajor indicator is constructed on the basis of distinguishing prior
education preparing for sciences, or for social sciences. Students in the sample are from 45
different national high school systems, all being very different, but in all cases differentiating
between advanced and intermediate level mathematics track (students of basic mathematics track
are not admitted into the program).

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Upon entering the course, students were required to do a mathematics diagnostic entry
test (MathEntryTestScore), of which the scores were added to the SIS data.
Dispositional Attitude Data
Attitudes towards the learning of mathematics were assessed with the SATS instrument
(Tempelaar, Gijselaers, van der Loeff, & Nijhuis, 2007), based on the expectancy-value theory
(Wigfield & Eccles, 2000). The instrument contains six mathematics related attitudes, and two
general attitudes. However, in this study, we only focus on the two general learning related self-
perceptions referred to as RiskTaking, how strong risk seeking and how less risk avoidant
students are, and Procrastination, the tendency to avoid doing learning activities.
Dispositional Academic Motivation Scale
Vallerand et al. (1992) propose three main categories of motivations in learning: Intrinsic,
Extrinsic, and Amotivation. First, Intrinsic motivations (IM) refer to the pleasure and satisfaction
derived from doing the task itself. IM consists of (1) Intrinsic motivation to know (IMknow), that
refers to the satisfaction while learning or trying to understand something new, (2) Intrinsic
motivation toward accomplishments (IMacc), in which individuals get pleasure from
accomplishing or creating something, and (3) Intrinsic motivation to experience stimulation
(IMstim), referring to the fulfillment from engaging in the activity. Second, Extrinsic
motivations (EM) pertains to a wide variety of behaviors which are engaged in as a means to an
end and not for their own sake. EM can be differentiated between (1) EM external regulation
(EMext), which refers to rewards or constraints, (2) EM introjection (EMint), in which

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
individuals begin to internalize the reason for his or her actions, and (3) EM identification
(EMiden), in which the behavior is perceived as valuable and important for oneself. Third,
individuals are Amotivated when they are neither intrinsically or extrinsically motivated. They
perceive their behaviors are caused by forces that are out of their control.
In this study, we combine IMknow, IMacc, IMstim, and EMiden into a new construct
called Autonomous, which is the total average of the mentioned motivations. In addition, Control
is also created by taking the mean of EMintro, and EMext.
Dispositional Help-Seeking Behavior Data
Help seeking can be conceptualized as a general problem-solving strategy that allows
learners to cope with academic difficulties in gaining the assistance of others. Gall (1985) draws
a distinction between "executive" or dependency-oriented help seeking and "instrumental" or
mastery-oriented help seeking. The former refers to those instances in which the student's
intention is to have someone else solve a problem or attain a goal on his or her behalf, whereas
the latter is limited to the amount and type of assistance needed for the student to solve the
problem independently. Avoidance of help-seeking is a situation in which help is needed, but the
student refuses to seek help. Perceived benefits of help seeking are students’ beliefs about the
outcomes of help-seeking activities, such as interest or learning. In addition, the source of help
can also be distinguished between Formal source and Informal source. The former refers to
institutional resources such as instructors, or tutors, while the latter refers to non-institutional
resources such as classmates, friends, and family members (Knapp & Karabenick, 1988).

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Dispositions on Self-Regulated Learning
The learning processing and regulation strategies that shape self-regulated learning are
based on the Inventory of Learning Styles (ILS) instrument (Vermunt, 1996). Our study focuses
on two of the four domains or components of learning of Vermunt’s model: cognitive processing
strategies, and metacognitive regulation strategies. Each of these components are composed of
three scales. Different processing strategies include Deep processing strategy, in which students
relate, structure and critically process the new knowledge they learn, Stepwise processing
strategy (also called surface processing) based on memorizing, rehearsing and analyzing, and
Concrete processing strategy, focusing on making new knowledge concrete, and applying it
(Vermunt, 1996). Likewise, three metacognitive regulation strategies are Self-Regulation of
learning processes and learning content, External Regulation of learning processes and learning
results, and lastly, Lack of Regulation: the absence of regulation be it by the student or out of the
environment.
Dispositional Epistemic Emotions Data
Epistemic emotions distinguish from activity emotions in that they are related to
cognitive aspects of the task itself (Pekrun, 2011). Prototypical epistemic emotions are curiosity
and confusion. In this study, epistemic emotions were measured with the Epistemic Emotion
Scales (EES) (Pekrun & Meier, 2011), including Surprise, Curiosity, Confusion, Anxiety,
Frustration, Enjoyment, and Boredom.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Research questions
In order to examine how learning dispositions and feedback preferences affect academic
performance, the following research questions were posed:
Q1: How do feedback preferences influence academic performance?
Q2: How do learning dispositions influence feedback preferences?
Q3: To what extent do feedback preferences mediate the relationship between learning
dispositions and academic performance?
Data analysis
The data analysis steps of this study are all based on linear, multivariate models, making
use of Sobel-Goodman mediation analysis (Figure 1). In the first step, we investigate the direct
effects of the four performance measures, the feedback preferences data derived from LMS, and
several types of disposition data, with SIS data as controls. For space limitations, we restrict
ourselves to static models that are estimated on all available, aggregated track data, rather than
dynamic models estimated on weekly data. In the second step, we focus on the indirect effects of
dispositional data on academic performance through feedback preferences track data: the
mediation effect is calculated as the product of the coefficients of dispositional data and feedback
preferences, and feedback preferences and academic performance. In the final step, the total
effect is computed as the sum of direct effect and indirect effect.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Figure 1: Research design
Results
Feedback preferences (LMS Track Data)
Figure 2 summarizes the relationship between feedback preferences, as revealed by
student behavior which represented through their actions within the tools, and academic
performance. MasteryLevel
1
in both tools, that is the average number of exercises successfully
finished, is strongly positively related to all performance measures. Most strongly for
performance in quizzes, with MathBonus and StatsBonus with betas of .78 and .92, respectively,
and somewhat less strong for performances in the exams, with MathExam and StatsExam with
betas of .40 and .53, respectively (p < .01). This difference in explained variation is easily
interpreted using the strong tie between quizzes and practicing in the tools. Second, the average
number of fully worked-out solutions asked for per exercise, MathSolutions and StatsSolutions,
are associated with a significant decrease in Mathexam (B = -.16, p < .01) and Statsexam (B = -
1
See section 2.2.2 LMS Trace Data
Learning dispositions
Academic performance
Feedback preferences

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
.31, p < .01), respectively. This may seem counter-intuitive, but is to be interpreted in a
multivariate context: given the same MasteryLevel, students who requested less fully worked-out
solutions are the better performers, and therefore, scored higher on the final exam. The effect of
StatsSolutions is salient on StatsBonus whereas MathSolutions has an insignificant impact on
MathBonus. Third, while the average number of hints asked for per exercise (Mathhints) has no
significant effect on Mathexam and MathBonus, the Statshints variable is negatively correlated
with Statsexam (B = -.14, p < .01) and Statsbonus (B = -.08, p < .01). Its interpretation follows
the multivariate context: given the same level of mastery in Statistics, students who asked for
fewer hints are the ones who perform better on the quizzes and the final exam.
Figure 2: The effects of mastery and feedback preferences on academic performance
(standardized beta coefficients, p<.01)
SIS System Data
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Mathexam Statsexam Mathbonus Statsbonus
Mastery
Solutions
Hints

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
In terms of academic performance, there are no significant differences amongst study
tracks and revealed feedback preferences, except for Economics students, for whom the
performance in Mathexam is significantly higher than International Business students (B = .05, p
< .1).
The indicator for prior mathematics schooling, MathMajor, impacts both academic
performance and feedback preferences (Table 1). The beta weights of advanced level prior
education are .12 for MathExam, .07 for StatsExam, and .09 for MathBonus. Evidently, the
benefits of having more prior knowledge in mathematics are greater on Mathematics related
performance than Statistics related performance. Regarding feedback preferences, students with
MathMajor level asked for less fully worked-out solutions than non-MathMajor students in both
mathematics and statistics, with the stronger effect on the former. Similarly, MathEntryTestScore
also demonstrated similar patterns with stronger effects on MathExam and MathBonus than on
StatsExam and StatsBonus.
While the difference in academic performance across gender is not significant, there are
some interesting patterns in feedback preferences between females and males. On average,
female students use more fully worked-out solutions and have higher mastery score than male
students in Mathematics. However, in the multivariate model, the beta of the indicator Female is
negative. This is to be understood by the gender difference in MathMajor, the main predictor of
the Solutions variable. Female students are underrepresented in the MathMajor category, but
within both categories, female students use fewer Solutions than male students.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Table 1: The effects of SIS system data on academic performance and feedback
preferences
Math
Stats
Exam
Bonus
Mastery
Solutions
Hints
Exam
Bonus
Mastery
Solutions
Hints
Economicsa
.05*
.02
-.04
-.03
.00
.00
.02
.00
-.04
.02
FiscalEconomicsa
.03
.01
-.06*
-.03
-.01
-.02
-.03*
.02
.00
-.05
MathMajor
.12***
.09***
.05
-.14***
-.01
.07**
.01
.02
-.06*
-.03
MathEntryTestScore
.17***
.09***
.18***
.01
.03
.05*
.01
.13***
.04
.01
Female
.04
.02
.08**
-.06*
.02
.04
.01
.03
.03
-.06*
Note: standardized coefficients; Baseline groups are InternationalBusiness, MathMinor, and Male; * p < .1; ** p < .05; ** p <
.01
Mediation tests
After carrying out the analysis of the direct effects of revealed feedback preferences on
academic performance, and how SIS system data impact feedback preferences, we are interested
in investigating how learning dispositions influence feedback preferences, and to what extent
feedback preferences mediate the relationship between learning dispositions and academic
performance. In order to do so, we once more apply Sobel-Goodman mediation tests to measure
the indirect effect of any learning disposition on academic performance, multiplying the
coefficient of the learning disposition on feedback preference, as well as the coefficient of
feedback preference on academic performance.
Dispositional Attitude Data
Direct effects of dispositional attitudes on performance measures are limited, with only
one significant relation: students with higher levels of Procrastination perform on average worse
on Mathexam (B = -.06, p < .1). In contrast, indirect effects through feedback preferences are

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
resilient (Table 2). Procrastination hinders all student activity, and above all, has deteriorating
effects on mastery in the tools. Due to the strong tie between mastery in the tools and bonus (the
score on the quizzes), major negative indirect effects are those from Procrastination through
Mastery to Bonus score. There is a weak positive indirect effect, composed of two negative
paths, from Procrastination through both types of Solutions to Exam and Bonus scores.
Long-term orientation has no direct effects on performance and a very weak indirect
effect through Hints for both statistics performance types.
Table 2: Mediation analyses from dispositional learning attitudes to activity measures to
academic performance
Direct path
Indirect path through the following mediators
Math
Stats
Math
Stats
Mastery
Solutions
Hints
Mastery
Solutions
Hints
Procrast
-.18***
-.14***
-.09**
-.22***
-.09**
-.02
Longterm
.03
.05
.08**
.05
-.04
.08**
Procrast  Exam
-.06*
-.04
-.07
.02***
.00
-.18***
.03***
.00
Procrast  Bonus
.02
.02
-.14***
.00
.00
-.20***
.02**
.00
Longterm  Exam
-.01
.02
.01
-.01
.00
.03
.01
-.01***
Longterm  Bonus
-.02
.01
.02
.00
.00
.04
.01
-.01**
Note: standardized coefficients; * p <.1 ; ** p <.05 ; *** p <.01
Dispositional Academic Motivation Scale
In reporting the role of autonomous, controlled, and lack of motivation, indirect effects
are of a very limited size, and absent for both Mastery and Hints variables, as shown in Table 3.
The only significant indirect effect is through the Solutions variable. Autonomously motivated
students more often follow ‘their own learning plan’ by calling fully worked-out examples,

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
rather than solving the problems themselves, both in mathematics and statistics. Amotived
students do too, for mathematics. This negatively impacts performance scores, mostly for the
exam scores. These negative indirect effects add to direct effects, also negative, of all motivation
types: Autonomous and Amotivation, as well as Controlled motivation, producing all negative
total effects. It is especially the exam component of performance, Mathexam and Statsexam that
is most strongly affected.
Table 3: Mediation analyses from dispositional academic motivation to activity measures to
academic performance
Direct path
Indirect path through the following mediators
Math
Stats
Math
Stats
Mastery
Solutions
Hints
Mastery
Solutions
Hints
Autonomous
.03
.10**
.01
.05
.12***
-.01
Control
.04
.02
-.01
.02
.01
-.02
Amotivation
-.01
.09**
.03
-.01
.03
-.04
AutoExam
-.08**
-.06
.01
-.02**
.00
.03
-.04***
.00
AutoBonus
-.06**
-.06**
.02
.00
.00
.05
-.02***
.00
ControlExam
-.06**
-.07**
.02
.00
.00
.01
.00
.00
ControlBonus
-.03
.03
.03
.00
.00
.02
.00
.00
AmotivExam
.01
-.08**
.00
-.01**
.00
-.01
-.01
.01
AmotivBonus
-.02
-.03
-.01
.00
.00
-.01
-.01
.00
Note: Standardized coefficients; * p < .1 ; ** p <.05 ; *** p <.01
Dispositional Help-Seeking Data
In help-seeking dispositions, we find more instances of opposite directions of direct and
indirect effects. The direct effect of the preference to solve problems independently
(Instrumental) is positive for Mathbonus, Statsexam, and Statsbonus (Table 4). The indirect
effect for performance in statistics, is, however, negative and about the same size, making the

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
total effect indeterminate. The same mechanism is at work for the Executive help-seeking
disposition, the preference to have someone else solve the problem on one’s behalf. For
performance in statistics, its direct effect is positive and its indirect effect negative. It is only in
students who are in need of help but refuse to seek it (Avoidance), that negative direct effects add
to negative indirect effects.
Indirect effects are mainly through lower levels of mastery in both tools. Concerning
feedback preferences, students with an Executive help-seeking disposition ask for more worked-
out solutions in Statistics (B = .10, p < .05), which lead to lower performance (StatsBonus &
StatsExam). In contrast, students whose help-seeking is Perceived to support Learning, search
for help is that is beneficial for their learning, ask for less worked-out solutions (B = - .07, p <
.05).
Table 4: Mediation analyses from help-seeking behaviors to learning activities to academic
performance
Direct path
Indirect path through the following mediators
Math
Stats
Math
Stats
Mastery
Solutions
Hints
Mastery
Solutions
Hints
Instrumental
-.03
-.05
-.06
-.07**
-.02
.00
Avoidance
-.10***
-.02
-.01
-.09**
-.06
-.05
Executive
.02
.06
.06
.04
.10**
.05
Perceived interest
.00
-.03
.00
-.01
-.06
.02
Perceived learning
-.02
.06
.06
-.01
-.07**
-.04
Formal
.00
.00
.01
-.03
-.07
.01
InstrumentalExam
.00
.06*
-.01
.01
.00
-.04**
.01
.00
InstrumentalBonus
.03*
.07***
-.02
.00
.00
-.06**
.00
.00
AvoidanceExam
-.05
-.06*
-.04**
.00
.00
-.05**
.02
.01
AvoidanceBonus
-.04**
-.09***
-.08***
.00
.00
-.08**
.01
.00
ExecutiveExam
.02
.04
.01
-.01
.00
.02
-.03**
-.01
ExecutiveBonus
.03
.06***
.01
.00
.00
.04
-.02**
.00

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Perceived InterestExam
-.02
-.03
.00
.01
.00
-.01
.02
.00
Perceived InterestBonus
.01
.00
.00
.00
.00
-.01
.01
.00
Perceived LearningExam
.02
.00
-.01
-.01
.00
-.01
-.02
.01
Perceived LearningBonus
.00
-.01
-.02
.00
.00
-.01
-.01
.00
FormalExam
.01
.06**
.00
.00
.00
-.02
.02**
.00
FormalBonus
.02
.01
.00
.00
.00
-.03
.01**
.00
Note: standardized coefficients; * p < .1 ; ** p < .05 ; *** p < .01
Dispositions on Self-Regulated Learning
The effects of self-regulated learning strategies on academic performance are
summarized in Table 5. First, students with performing a Deep processing style, who tend to
relate elements of the subject matter to each other and to prior knowledge, structure these
elements into a whole, and form a critical view on the materials, performing focused better in
Mathexam, Statsexam, and Statsbonus. In this case, direct and indirect effects are reinforcing:
both are positive. The Step-wise processing style, focused more on memorizing and analyzing
the subject matter, and the Concrete processing style, where students have the tendency to apply
the subject matter in practice, are unrelated to performance measures, lacking both direct and
indirect effects.
This pattern of reinforcing direct and indirect effects repeats itself with metacognitive
learning regulation styles. Students with a Self-regulated learning style, who prefer to regulate
their learning process themselves, do less well, both as a direct effect, and an indirect effect
through mastery. In contrast, students with an External-regulated learning style, who prefer to
orient on tutors and peers in the regulation of learning, do slightly better due to positive indirect

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
effects through mastery. Neither Solutions nor Hints play any role in the indirect effects of
learning styles.
Table 5: Mediation analyses from self-regulated learning styles to activity measures to academic
performance
Direct path
Indirect path through the following mediators
Math
Stats
Math
Stats
Mastery
Solutions
Hints
Mastery
Solutions
Hints
Deep
.03
-.01
-.02
.12***
.01
-.01
Step
-.06
.00
.00
-.03
-.01
.01
Concrete
-.05
-.10**
.04
-.06
.00
.00
Self
-.04
.00
-.05
-.10**
-.01
.00
Extern
.07*
.00
.05
.08**
.06
.04
Lackr
-.03
.02
-.04
-.01
.03
-.03
DeepExam
.09**
.12***
.01
.00
.00
.06***
.00
.00
DeepBonus
.03
.05**
.03
.00
.00
.11***
.00
.00
StepExam
-.06*
-.01
-.02
.00
.00
-.02
.00
.00
StepBonus
-.03
-.03
-.05
.00
.00
-.03
.00
.00
ConcreteExam
-.02
-.07
-.02
.02
.00
-.03
.00
.00
ConcreteBonus
-.01
-.01
-.04
.00
.00
-.06
.00
.00
SelfExam
-.08**
-.10**
-.02
.00
.00
-.05**
.00
.00
SelfBonus
-.03
-.03
-.03
.00
.00
-.09**
.00
.00
ExternExam
-.03
-.03
.03
.00
.00
.04**
-.02
-.01
ExternBonus
.03
.02
.05*
.00
.00
.07**
-.01
.00
LackrExam
.03
-.02
-.01
.00
.00
-.01
-.01
.00
LackrBonus
.02
.01
-.02
.00
.00
-.01
-.01
.00
Note: standardized coefficients; * p < .1 ; ** p < .05 ; *** p < .01
Dispositional Epistemic Emotions Data
The direct effects of epistemic emotions for mathematics, reported in Table 6, are mostly
in line with expectations: positively valenced epistemic emotions have positive effects, such as
Curiosity, and negatively valenced epistemic emotions have negative effects, such as Anxiety and
Frustration. The emotion without a straightforward valence, Confusion, carries a small positive

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
effect. Surprisingly, enJoyment comes with a negative effect on performance in statistics (but not
in mathematics). Its reason may be in the specific constellation of high school mathematics
education in Europe: students who enjoy mathematics will typically opt for advanced
mathematics tracks, but such tracks do not include statistics, whereas students not enjoying
mathematics may opt for social science oriented tracks that do contain statistics.
In the indirect effects, the Solutions variables act as mediator variables. In the statistics
domain students high in Frustration call for more worked-out Solutions, lowering on average
their performance scores in statistics. In mathematics, students with high levels of Confusion call
for fewer worked-out Solutions, increasing on average their Exam score.
Table 6: Mediation analyses from epistemic emotions to activity measures to academic
performance
Direct path
Indirect path through the following mediators
Math
Stats
Math
Stats
Mastery
Solutions
Hints
Mastery
Solutions
Hints
Surprise
-.04
.00
.02
-.02
.01
.03
Curiosity
.05
.03
-.04
-.01
-.05
.04
Confusion
-.04
-.13**
.07
-.03
-.06
.02
Anxiety
-.09*
.03
-.08
-.06
.00
.06
Frustration
.04
.03
.04
.09*
.12**
-.03
Joy
.04
-.05
.01
.03
.04
-.01
Boredom
-.03
-.05
-.05
-.05
-.05
-.01
SurpriseExam
.05
-.02
-.02
.00
.00
-.01
.00
.00
SurpriseBonus
-.01
.02
-.03
.00
.00
-.02
.00
.00
CuriosityExam
.04
.07*
.02
-.01
.00
-.01
.02
-.01
CuriosityBonus
.05**
.05**
.04
.00
.00
-.01
.01
.00
ConfusionExam
-.03
-.03
-.02
.02**
.00
-.02
.02
.00
ConfusionBonus
.00
.05*
-.03
.00
.00
-.03
.01
.00
AnxietyExam
-.19
-.04
-.04*
-.01
.00
-.03
.00
-.01
AnxietyBonus
-.06**
-.01
-.07*
.00
.00
-.06
.00
-.01
FrustExam
-.04
-.01
.02
.00
.00
.05*
-.04**
.00

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
FrustBonus
-.04
-.07**
.03
.00
.00
.08*
-.02**
.00
JoyExam
-.05
-.07*
.02
.01
.00
.02
-.01
.00
JoyBonus
.04
-.09***
.03
.00
.00
.03
-.01
.00
BoredomExam
.00
-.01
-.01
.01
.00
-.03
.02
.00
BoredomBonus
.03
-.01
-.02
.00
.00
-.05
.01
.00
Note: standardized coefficients; * p < .1 ; ** p < .05 ; *** p < .01
Discussion and conclusion
Q1: How do feedback preferences influence academic performance?
Trace data from the two e-tutorials, Sowiso and MSL, are incorporated in all models with
a consistent pattern: mastery levels in the tools are by far the strongest predictor of all
performance types, whilst number of fully worked-out solutions called for (and in some cases the
number of hints called for), negatively impact performance. These findings are in line with
previous studies (Tempelaar et al., 2015). These negative betas may surprise, since all bivariate
relationships between the number of hints, and the number of solutions, demonstrate positive
correlations with each of the four performance types. Overall, students who see more fully
worked-out solutions, and who ask for more hints, perform better. However, in the context of
multivariate prediction equations, the favorable effect of intensive practicing is already contained
in the mastery variables, reducing the impact of the hints and fully worked-out solutions
variables on conditional relationships. The negative betas tell us that for students with a given
mastery level, requiring more hints to reach that mastery level, or requiring more worked-out
solutions, lowers the expected performance for each of the performance categories.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
The findings also indicate the stronger effect of fully worked-out solutions compared with
the use of hints as a feedback channel. Furthermore, the use of hints has little to insignificant
impact on mathematics performance while the effect is more salient in statistics performance.
Our results confirm the advantage of fully worked-out solutions in multi-media learning
environments as indicated in previous literature (Hoogerheide, Loyens, & Van Gog, 2014;
McLaren et al., 2016; Renkl, 2005). Especially, it addresses a common limitation of the
methodology of the aforementioned studies which is the generalizability from lab/controlled
settings to authentic settings. In real life, the effects of feedback preferences are inter-linked
rather than being isolated and individually examined. Thus, LA help resolves this issue by using
trace data which reflect actual user behaviors.
Q2: How do learning dispositions influence feedback preferences?
Out of 25 dispositions, only nine have a significant impact on feedback preferences (Figure
3). Overall, learning dispositions have stronger and more significant impact on fully worked-out
solutions compared with hints. Students who are inclined toward Autonomous and Amotivation
types of academic motivation, the Executive help-seeking disposition and the Frustration
emotion use more fully worked-out solutions. In contrast, the Concrete learning strategy,
Procrastination attitude, Formal help-seeking disposition, and the Confusion emotion are
associated with the lower use of fully worked-out examples. Procrastination and Longterm are
the only two measurements which have a salient impact on the use of hints, in which the use of
hints is lower in the former and higher in the latter.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Figure 3: The effects of learning dispositions on feedback preferences: standardized beta
coefficients, p<.05)
Our findings contribute to the development and implications of educational policies
concerning learner/instructor data by bridging the existing gap between LA and pedagogy
(Gašević, Dawson, & Siemens, 2015). Most studies at the early stage of LA have built upon data
extracted from both institutional SIS and the log data retrieved from digital platforms that
organize and facilitate learning, such as LMSs and e-tutorials (Arnold & Pistilli, 2012;
Macfadyen & Dawson, 2010). While these studies provide important markers on the potential of
LA in education, most are still unable to go beyond the descriptive function of LA, which is mostly
based on demographic, grades, and trace data. Hence, effective instructional and intervention
practices are hindered by the lack of pedagogical-based findings. Using dispositional characteristics
of students, this study addressed some of the limitations of conventional LA by providing educators
with ‘actionable feedback’, which not only describes how students prefer certain feedback types but
also explains why students follow certain behavioral patterns based on their learning dispositions.
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
MathSolutions
MathHints
StatsSolutions
StatsHints

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Q3: To what extent do feedback preferences mediate the relationship between learning
dispositions and academic performance?
In general, fully worked-out solutions appear to be a better mediator than hints. The
mediating effect is stronger for performance in statistics, the topic about which most students had
little prior knowledge, than for mathematics. Next, we find several cases of feedback preferences
where direct and indirect effects of dispositions have opposite directions. In general, the use of
more Hints has little to no impact on performance measures beyond the effect already included
in the mastery level, whereas the use of more worked-out Solutions tends to have a negative
effect on performance levels. In all situations where the learning disposition is positively related
to the mediator variable Solutions, the indirect effect, being the product of a positive and a
negative beta, becomes negative. An example is the Executive feedback disposition: the tendency
to use others to solve your own (academic) problems. Students who score high on Executive
feedback tend to call for more worked-out Solutions, which, for example, in turn lowers their
expected performance scores. However, this small indirect effect is completely offset by the
positive direct effect of Executive feedback on performance in the statistics quizzes. Apparently,
in the end, it pays to have the disposition to let others work for you.
A crucial conclusion relates to the role of systematic comparison of direct and indirect
model effects, and the diverging outcomes, to which such a comparison may lead. Learning
analytics (LA) models are typically of input-output kind, directly relating performance
components, (the outputs) to measured input variables. Restricting to direct effects only,
surpassing the process effects visible in an input-process-output type of model, would leave all

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
indirect effects unobserved. As the example above indicates, this could lead to incorrect
conclusions, and incorrect interventions, derived from an input-output prediction model.
A second important finding from this research is similar in nature: it relates to the
importance of a systematic comparison of bivariate and multivariate relationships. Simple
correlations, often applied in LA applications, do not provide proper insight. In our context
correlational analyses would have led us to the conclusion that feedback modes, the use of hints,
and the use of fully worked-out solutions, all contribute positively to all performance types. This
would suggest stimulating students to use more hints, and to use more worked-out solutions, in
their learning. However, these bivariate relationships are confounded by overall student activity
in the e-tutorials. When correcting for this confound, by looking at multivariate relationships, we
find opposite conclusions: the use of hints is completely neutral, and the use of worked-out
solutions is, in fact, detrimental to learning outcomes. Another striking example of the
divergence between bivariate and multivariate modeling outcomes relates to gender differences
in revealed feedback behavior of students. Within the Dutch context, empirical research into
mathematics education suggests that female learners may profit more from example-based
education (Tempelaar, Rienties, & Nguyen, 2016). Based on this finding, one would expect
female students to more often make use of worked-out solutions than male students. And indeed,
in a bivariate context, we can confirm that hypothesis. However, in a multivariate context, the
confounding factor “prior mathematics track” pops up: female students more often take the
social-science track in high school, male students more often the science track, and social-
science track students use worked-out solutions more often. Correcting for this confound, the

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
gender effect completely disappears, and is even reversed in direction (but not statistically
significant). Therefore, consider how inadequate an intervention could have been, derived from a
simple, correlation-based LA prediction model.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
References
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to
increase student success. Paper presented at the Proceedings of the 2nd international
conference on learning analytics and knowledge.
Buckingham Shum, S., & Crick, R. D. (2012). Learning dispositions and transferable
competencies: pedagogy, modelling and learning analytics. Proceedings of the 2nd
International Conference on Learning Analytics and Knowledge, 92-101.
Duffy, M. C., & Azevedo, R. (2015). Motivation matters: Interactions between achievement
goals and agent scaffolding for self-regulated learning within an intelligent tutoring
system. Computers in Human Behavior, 52, 338-348.
Gall, S. N.-L. (1985). Help-seeking behavior in learning. Review of research in education, 55-90.
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about
learning. TechTrends, 59(1), 64-71.
Hattie, J. (2013). Visible learning: A synthesis of over 800 meta-analyses relating to
achievement: Routledge.
Hoogerheide, V., Loyens, S. M., & Van Gog, T. (2014). Comparing the effects of worked
examples and modeling examples on learning. Computers in Human Behavior, 41, 80-91.
Knapp, J. R., & Karabenick, S. A. (1988). Incidence of formal and informal academic help-
seeking in higher education. Journal of College Student Development.
Knewton. (2016). Knewton. Retrieved 28 March, 2016, from https://www.knewton.com/
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system”
for educators: A proof of concept. Computers & education, 54(2), 588-599.
McLaren, B. M., van Gog, T., Ganoe, C., Karabinos, M., & Yaron, D. (2016). The efficiency of
worked examples compared to erroneous examples, tutored problem solving, and
problem solving in computer-based learning environments. Computers in Human
Behavior, 55, 87-99.
Pekrun, R. (2011). Emotions as drivers of learning and cognitive development New perspectives
on affect and learning technologies (pp. 23-39): Springer.
Pekrun, R., & Meier, E. (2011). Epistemic emotion scales (EES): Munich, Germany: Department
of Psychology, University of Munich. Unpublished manuscript.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Renkl, A. (2005). The worked-out-example principle in multimedia learning. The Cambridge
handbook of multimedia learning, 229-245.
Schmidt, H. G., Van der Molen, H. T., Te Winkel, W. W., & Wijnen, W. H. (2009).
Constructivist, problem-based learning does work: A meta-analysis of curricular
comparisons involving a single medical school. Educational psychologist, 44(4), 227-
249.
Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality and productivity of the
higher education sector: Policy and strategy for systems-level deployment of learning
analytics.
Siemens, G., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education.
EDUCAUSE review, 46(5), 30.
Tempelaar, D., Gijselaers, W. H., van der Loeff, S. S., & Nijhuis, J. F. (2007). A structural
equation model analyzing the relationship of student achievement motivations and
personality factors in a range of academic subject-matter areas. Contemporary
Educational Psychology, 32(1), 105-131.
Tempelaar, D., Heck, A., Cuypers, H., van der Kooij, H., & van de Vrie, E. (2013). Formative
assessment and learning analytics. Paper presented at the Proceedings of the Third
International Conference on Learning Analytics and Knowledge.
Tempelaar, D., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for
feedback generation: Learning Analytics in a data-rich context. Computers in Human
Behavior, 47, 157-167.
Tempelaar, D., Rienties, B., & Nguyen, Q. (2016). Learning dispostions to build pedagogical
antecedents for pedagogy-based LA (submitted). ZFHE(Learning Analytics: Implications
for Higher Education).
Vallerand, R. J., Pelletier, L. G., Blais, M. R., Briere, N. M., Senecal, C., & Vallieres, E. F.
(1992). The Academic Motivation Scale: A measure of intrinsic, extrinsic, and
amotivation in education. Educational and psychological measurement, 52(4), 1003-
1017.
Vermunt, J. D. (1996). Metacognitive, cognitive and affective aspects of learning styles and
strategies: A phenomenographic analysis. Higher education, 31(1), 25-50.
Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation.
Contemporary Educational Psychology, 25(1), 68-81.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition in
educational theory and practice, 93, 27-30.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Bios:
Quan Nguyen is a PhD student in learning analytics at the Open University, UK, funded
by the Leverhulme Trust. After finishing a master in Information and Network Economics at
Maastricht University, he has become passionate about the application of data analytics in
education. He concentrates on analyzing big data set in educational context in order to provide
meaningful feedback to educators and students.
Dirk Tempelaar is an associate professor in the Department of Quantitative Economics
of the Maastricht University School of Business and Economics. His main teaching is in the
areas of statistics and research methods, and the design of problem-based learning modules in
these topics. His research interests are understanding student learning in self-regulated learning
contexts, investigating students’ learning patterns in blended learning environments, and the use
of a broad range of learner and learning data, both consisting of survey data and computer track
data, to create feedback cycles that improve learning: dispositional learning analytics.
Bart Rienties is Reader in Learning Analytics at the Institute of Educational Technology
at the Open University UK. He is programme director Learning Analytics within IET and Chair
of Student Experience Project Intervention and Evaluation group. As educational psychologist,
he conducts multi-disciplinary research on work-based and collaborative learning environments
and focuses on the role of social interaction in learning, which is published in leading academic
journals and books. His primary research interests are focused on Learning Analytics, Computer-
Supported Collaborative Learning, and the role of motivation in learning. He successfully led a
range of institutional/national/European projects.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Bas Giesbers is learning innovation consultant and researcher at Rotterdam School of
Management, Information Management and Consulting. After studying educational psychology,
he gained experience in educational technology, teaching, and research on (remedial) distance
education, distance supervision (e.g. of internships and thesis writing) and teacher
professionalisation in the field of distance education. His PhD research focused on the use of
synchronous communication in distance education, and the role of motivation and learner
interaction.

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Appendix A
Table 8: Descriptive statistics for demographics
Variable
N
Percent
Sex
Male
616
56.1
Female
482
43.9
Total
1098
100
Study
International Business
752
70.81
Economics
269
25.33
Fiscal Economics
41
3.86
Total
1062
100
MathMajor
0
722
66
1
372
34
Total
1094
100
Table 9: Descriptive statistics for academic performance and learning activities
Variable
Obs
Mean
Std. Dev.
Min
Max
Performance
FinalGrade
1062
6.60
2.40
1.00
10.00
MathExam
1062
12.05
3.72
2.00
21.00
StatsExam
1062
13.20
3.50
3.00
20.00
Activity
MathMastery
1061
0.51
0.30
0.00
0.99
MathSolutions
1061
0.38
0.39
0.00
4.32
MathHints
1061
0.13
0.18
0.00
1.36
StatsMastery
1056
0.70
0.32
0.00
1.00
StatsSolutions
1058
0.30
0.33
0.00
1.50
StatsHints
1058
0.07
0.11
0.00
0.80

Running Head: LEARNING ANALYTICS AND STUDENT FEEDBACK PREFERENCES
Table 10: The effects of learning dispositions on academic performance and learning activities
Math
Stats
Exam
Bonus
Mastery
Solutions
Hints
Exam
Bonus
Mastery
Solutions
Hints
MathMastery
.4***
.78***
MathSolutions
-.16***
.00
MathHints
.00
.00
StatsMastery
.53***
.92***
StatsSolutions
-.31***
-.17***
StatsHints
-.14***
-.08***
Autonomous
-.08**
-.06**
.03
.10**
.01
-.06
-.06**
.05
.12***
-.01
Control
-.06**
-.03
.04
.02
-.01
-.07**
.03
.02
.01
-.02
Amotivation
.01
-.02
-.01
.09**
.03
-.08**
-.03
-.01
.03
-.04
Deep
.09**
.03
.03
-.01
-.02
.12***
.05**
.12***
.01
-.01
Step
-.06*
-.03
-.06
.00
.00
-.01
-.03
-.03
-.01
.01
Concrete
-.02
-.01
-.05
-.10**
.04
-.07
-.01
-.06
.00
.00
Self
-.08**
.03
-.04
.00
-.05
-.1**
-.03
-.10**
-.01
.00
Extern
-.03
.03
.07*
.00
.05
-.03
.02
.08**
.06
.04
Lackr
.03
.02
-.03
.02
-.04
-.02
.01
-.01
.03
-.03
Acadbuoy
-.08***
-.04**
-.07*
-.02
-.01
-.05
-.03
-.12***
-.04
-.06
Procrast
-.06*
.02
-.18***
-.14***
-.09**
-.04
.02
-.22***
-.09**
-.02
Longterm
-.01
-.02
.03
.05
.08**
.02
.01
.05
-.03
.08**
Instrumental
.00
.03*
-.03
-.05
-.06
.06*
.07***
-.07**
-.02
.00
Avoidance
-.05
-.04**
-.10***
-.02
-.01
-.06*
-.09***
-.09**
-.06
-.05
Executive
.02
.03
.02
.06
.06
.04
.06***
.04
.1**
.05
Perceived interest
-.02
.01
.00
-.03
.00
-.03
.00
-.01
-.06
.02
Perceived learning
.02
.00
-.02
.06
.06
.00
-.01
-.01
.07
-.04
Formal
.01
.02
.00
.00
.01
.06**
.01
-.03
-.07**
.01
Surprise
.05
-.01
-.04
.00
.02
-.02
.02
-.02
.01
.03
Curiosity
.04
.05
.05
.03
-.04
.07*
.05**
-.01
-.05
.04
Confusion
.03
.00
-.04
-.13**
.07
-.03
.05*
-.03
-.06
.02
Anxiety
-.19***
-.06**
-.09*
.03
-.08
-.04
-.01
-.06
.00
.06
Frustration
-.04
-.04
.04
.03
.04
-.01
-.07**
.09*
.12**
-.03
Joy
-.05
.04
.04
-.05
.01
-.07*
-.09***
.03
.04
-.01
Boredom
.00
.03
-.03
-.05
-.05
-.01
-.01
-.05
-.05
-.01
Economicsa
.05*
.02
-.04
-.03
.00
.00
.02
.00
-.04
.02
FiscalEconomicsa
.03
.01
-.06*
-.03
-.01
-.02
-.03*
.02
.00
-.05
MathMajor
.12***
.09***
.05
-.14***
-.01
.07**
.01
.02
-.06*
-.03
MathEntryTestScore
.17***
.09***
.18***
.01
.03
.05*
.01
.13***
.04
.01
Female
.04
.02
.08**
-.06*
.02
.04
.01
.03
.03
-.06*
Constant
16.41***
1.42***
.79***
.73***
.17
14.85***
2.05***
1.09***
.13
.08
Observations
959
958
959
959
959
960
960
960
960
960
R-squared
.400
.764
.214
.116
.051
.365
.743
.189
.094
.050
Note: All coefficients are standardized * p < .1 ; ** p < .05 ; *** p < .01