Content uploaded by Julia Moeller
Author content
All content in this area was uploaded by Julia Moeller on May 14, 2024
Content may be subject to copyright.
Book chapter ESM in research on emotion and motivation
1
Pre-print. Please contact the first author for the most recent version before citing.
The Experience Sampling Method
in the research on achievement-related emotions and motivation
Julia Moeller1, Julia Dietrich2, & Jessica Baars1
1 = Leipzig University; 2= University of Jena
Book chapter for:
R.C. Lazarides, Hagenauer, H. Järvenoja (Eds.), Motivation and Emotion in Learning and
Teaching across Educational Contexts: Theoretical and Methodological Perspectives and
Empirical Insights. Routledge.
This work was funded by a Jacobs Foundation Early Career Research Fellowship and a grant
by the German Research Foundation (DFG; #451682742) to Julia Moeller and Julia Dietrich.
Book chapter ESM in research on emotion and motivation
1
Abstract
Motivation and emotions often change during learning processes and show much
fluctuation across learning situations and contexts. That makes it necessary to assess and
analyze their processes of change and the situation-specific, context-specific, and other
sources of variation. A method to assess the situation- and context-specificity and fluctuation
of emotions and motivation is the Experience Sampling Method (ESM). The ESM produces
intensive longitudinal data with many measurement time points per person (see Figure 1).
Typically, participants, such as students, are surveyed repeatedly during their day about their
current emotions or motivational states with self-report surveys on portable devices, such as
smartphones.
This chapter serves as a resource for both beginners and advanced ESM researchers. It
starts with an introduction to the Experience Sampling Method, referencing useful resources
to researchers interested in applying the ESM in their own studies of emotion and motivation
in education. Then, the contribution of the ESM to the recent and expected future theoretical
shifts towards situated models of motivation and emotions are discussed. The unique insights
that the ESM can provide to the research on situated and contextualized learning-related
motivation and emotions are illustrated by discussing empirical examples.
The chapter gives an overview of cutting-edge innovations in the research with the
ESM and addresses current challenges in this field, including limitations to the replicability
and generalizability of ESM studies across contexts. First solutions to these challenges are
proposed, along with a description of further directions for future research.
Keywords: Experience Sampling Method, Emotion, Motivation
Book chapter ESM in research on emotion and motivation
2
What is the Experience Sampling Method and how does it contribute to the research on
Motivation and Emotion in Learning and Instruction?
Since learning is by definition a process of change, studying it requires methods that
can capture such change. Crucial learning-related processes, such as sensing academic
emotions, subjectively weighing motivating and demotivating aspects of a learning task,
paying attention, processing new information, or giving up on a task all happen in specific
learning situations. Understanding these processes therefore requires us to study them in the
moments in which they happen.
The Experience Sampling Method (ESM, Hektner et al., 2007), as well as the related
methods of ambulatory assessment (Fahrenberg, 1996), and ecological momentary assessment
(Shiffman et al., 2008), are well suited to capture situated, fluctuating aspects of motivation
and motivation, along with information about their changes over time, and their time-,
context-, and person-specific determinants. With the ESM, researchers ask participants in
real-time via mobile devices about their current psychological states. The individual surveys
typically last only a few minutes and are repeated multiple (typically at least > 10) times.
ESM surveys aim to capture fluctuating psychological experiences (motivation and emotions)
as well as their context information (the current task, current company, current location, etc.).
Such data enable researchers to disentangle situational, contextual, and personal determinants,
because the same students or the same teachers are assessed many times in different situations
(see Figure 1). The resulting data type is called intensive longitudinal data.
Book chapter ESM in research on emotion and motivation
3
Figure 1. The complex multilayered nature of ESM data illustrated with the example of time
points nested in individual students nested in classrooms nested in schools, influenced by
characteristics of time, context, and persons.
1
1
This figure is just an illustration of the example of student data. There can always be cross-classifications, in
terms of one unit (one student) belonging to multiple units of a higher level (one student attending different
classes throughout one day), depending on the research design and research question. There can also always be
additional sources of variation between or beyond the levels specified in this figure (multiple teachers cross-
classified teaching multiple classes in each of which they have multiple measurement time points); that could,
and sometimes should, be studied as additional levels, or, alternatively, as predictors (dummy variables) in a
regression model or as group variables in other models.
Context characteristics
Person
characteristics
Remainin g stable across the period
of ESM data collection
e.g.,
personality traits
sociodemographic characteristics
Meso- & Macro-level
Context characteristics
originating in the meso-& macro-system in terms of Bronfenbrenner’s ecological systems theory,
Situation-level
Context characteristics
of specific situations that change
between time points
e.g., curr ent location,
curren t company,
curren t environmental stressors;
curren t demands & resources
Situation- level
Time characteristics
a) Objective characteristics;
e.g., morni ng-specific, weekday, &
season-specific effects,
nonsystematic nonstationarity
b) Subjective & indivi dual
perceptions o f these
characteris tics;
e.g., one p erson being tired in
mor ni ng s ; a noth er p er son f eeli ng
threatened by an upcoming deadline
School 1
School 2
School 3
Level 1: Variance between t ime points Level 2: Variance between i ndividuals Level 3: Variance between classrooms Level 4: Variance between schools
Dif fe ren ce s
between
schools;
school types;
school districts
Dif fe ren ce s
between
teachers;
classrooms
Dif fe ren ce s
between
regions,
countries
Clas s A
Clas s B
Clas s C
Clas s D
Clas s E
Clas s F
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T1
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T2
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
I
I
I
I
I
I
I
I
I
I
I
I
12
6
3
9
T3 T4 T5
Book chapter ESM in research on emotion and motivation
4
Statistical methods to collect and analyze intensive longitudinal data are rapidly
evolving and more and research employing these methods is currently booming (see Gates,
Chow, & Molenaar, 2023; Hamaker & Wichers, 2017; Myin-Germeys & Kuppens, 2022). For
readers interested in getting started with ESM data collections and analyses, Fritz et al. (in
prep.) and Piccirillo et al. (in prep.) have compiled a list of useful resources of readings and
instruments and Hall et al. (2021) provide a review of ESM research designs for the study of
emotions.
Theoretical and methodological innovations: How the ESM contributes to paradigmatic
shifts in the research on motivation and emotions
ESM data offer new perspectives to researchers studying short-term affective
processes and in-the-moment experiences in learning and achievement contexts (Dietrich et
al., 2022; Eccles, 2022; Pekrun & Marsh, 2022; Larson & Csikszentmihalyi, 2014; Murayama
et al., 2017). Such new perspectives are discussed below first in regard to research on learning
and instruction. Following that, we summarize more general paradigmatic insights that the
ESM offers to psychological research.
In recent years, there have been theoretical innovations and novel empirical insights
into learning-related emotions and motivation, in part facilitated by ESM studies becoming
easier and cheaper to collect (for an overview of software, see Arslan et al., 2018; for
instructions, see Fritz et al., in prep.; Myin-Germeys & Kuppens, 2022) and by novel
analytical methods becoming widely available (see the recent textbook by Gates, Chow, &
Molenaar, 2023).
One example of a theoretical shift and new empirical research program is the recently
emphasized focus on situated determinants of achievement motivation in the research on the -
now Situated- Expectancy-Value Theory of achievement motivation (Eccles & Wigfield,
Book chapter ESM in research on emotion and motivation
5
2020). While this theory had been widely established and extensively studied for many years,
the byword “Situated” was added recently in a publication calling for a stronger focus on
context-specific determinants of achievement motivation (Eccles & Wigfield, 2020). This
novel theoretical focus on situated, meaning contextual determinants followed the publication
of several ESM studies proposing and using ESM measures of task values, success
expectancies, and costs (Dietrich et al., 2017). The new ESM measures were validated and
used to study in-the-moment profiles of situational expectancies, values, and costs (Dietrich et
al., 2019). The new measures enabled researchers (and potentially teachers) to monitor
moment-to-moment changes and individual trajectories of situational expectancies and values
in large classrooms (Moeller et al., 2020). The novel focus on the situated nature of
expectancy-value experiences offered new insights into relations between situated expectancy
and value experiences and learning-relevant correlates and outcomes (Edwards &
Taasoobshirazi, 2022; Parrisius, 2021). The ESM provided the expectancy-value research
with the methods needed to study what the theory had often addressed but what had
previously been rarely studied empirically on the situational level: The motivation causing a
person to choose one learning task over another in specific learning situations. This has
enabled both theory development and empirical studies to enhance our understanding of the
influence of context, time, and person-specific factors on achievement motivation. It also has
led to research on intra-individual trajectories in achievement motivation from one learning
situation to the next, and on heterogeneity between situations and between individuals
(Moeller et al., 2022a).
While the focus on the empirical study of such situated determinants of motivation is
rather new in the literature on the (now Situated) Expectancy-Value Theory, other theories
have relied on such situational assessments for much longer: The research on flow states, for
instance, has piloted, used, and further developed the ESM for decades (Csikszentmihalyi,
1975; Engeser & Rheinberg, 2008). Mihalyi Csikszentmihaly, author of the flow theory, has
Book chapter ESM in research on emotion and motivation
6
been an early adopter of the ESM and has plowed a way for the ESM research on
achievement-related motivation and emotions (for overviews, see Larson & Csikszentmihalyi,
2014; Csikszentmihalyi & Rathunde, 1993, Nakamura & Csikszentmihalyi, 2009). The flow
theory describes that intrinsic motivation occurs when there is a fit between the demand of the
task and the person's ability level (Nakamura & Csikszentmihalyi, 2002). Following this
tradition, some authors define situational engagement as states of high levels of challenge,
skill, and interest (Schneider et al., 2016; Salmela-Aro et al., 2016).
Another line of research that has used the ESM for a long time and serves as a
reference for innovative ESM research designs and analyses is the research on academic
emotions (Pekrun et al., 2002; Goetz et al., 2013a; 2013b; Goetz et al., 2006). The research on
academic emotions uses ESM for instance to study situation- and context-specific
determinants of learning-relevant emotions (Schneider et al., 2016). The unique features of
intensive longitudinal data enable researchers to examine situation-specific profiles of
academic emotions, such as different types of boredom, which can differ between situations
within and across individuals (Goetz et al., 2013b). Examining the inter-individual
distributions of such intra-individual patterns revealed substantial heterogeneity between
individuals in regard to within-person relations between academic emotions (see the inter-
individual variation in within-person correlations between situational anxiety and motivation
in Pekrun et al., 2002 and Moeller et al., 2015). Systematic comparisons of ESM measures of
academic emotions (math anxiety) with retrospective measures revealed that the former
appeared to be less biased by gender stereotypes than the latter (Goetz et al., 2013a).
The current boom of the ESM can be seen not only in the research on learning- and
achievement related motivation and emotions, but in many other areas of Psychology
including Personality Psychology (Beck & Jackson, 2020; Fleeson, 2007), Clinical
Psychology (Fried et al., 2020), Developmental Psychology (Sonnenberg et al., 2012), and
Work and Organizational Psychology (Kühnel, 2017). Among the reasons for this victory
Book chapter ESM in research on emotion and motivation
7
march of the ESM across these disciplines are its ability to capture malleable, rapidly
fluctuating changes in people’s experiences, the capability to capture situational and
contextual determinants of these fluctuating subjective experiences, and the ability to describe
within-person structures and processes and their potential differences between individuals.
Due to these features, using the ESM enables researchers to ask research questions
that previously dominating research methods were unable to examine. Table 1 gives an
overview of such paradigmatic changes in the psychological research on motivation and
emotion enabled, accelerated and in part inspired by intensive longitudinal data, some of
which are discussed more in detail below.
Book chapter ESM in research on emotion and motivation
8
Table 1: Paradigmatic changes in the psychological research on motivation and emotion
enabled, accelerated and in part inspired by intensive longitudinal data
Tradition
New perspectives
Theory-guided, hypothesis-guided-deductive:
Deriving hypotheses from theories, falsifying
hypotheses by testing models on data
Open to exploratory, inductive approaches because
many theories do not yet make statements about intra-
individual variation, heterogeneous findings, or
moment-to-moment development; new methods and
unexpected findings lead to theory development
Nomothetic:
In search of general laws of experience and
behavior, one-size-fits-all
From idiographic, person-specific models to
nomothetic models, clarifying which findings can be
generalized under which boundary conditions
Inter-individual:
In search of differences between individuals,
focus on the analysis of variance between
individuals and groups
Intra-individual: Analysis of variance within and
between individuals
Assumption of ergodicity:
Findings obtained with inter-individual
procedures are interpreted as describing structures
and processes within individuals
Openness to lack of ergodicity:
Exploration of possibly different structures and
processes at different levels of analysis
Controlled:
Efforts to control the influence of boundary
conditions (laboratory or control variables)
Research interest in complex interactions in complex
systems (Holism);
attempt to model all relevant influences rather than
control them
Sequential understanding of causality: Search for
causal sequences: Clear distinction between
independent & dependent variables (predictors &
outcomes)
Interest in iterative feedback processes in which one
variable can be both an outcome and a predictor of
another variable
Focus on stationary Development effects (1 effect
instead of multiple effects per person or per
sample)
Openness to non-stationarity: Effects can change over
time
Focus on linear developmental trajectories &
correlations: Most of our methods only examine
linear correlations
Openness to non-linearity: Not all correlations are
linear. Higher exploration and testing of the form of
correlations
Insight 1: Intra-individual analyses reveal differences between situations.
Intensive longitudinal data with many measurement time points per person enable us to study
the within-person variation and co-variation of psychological constructs across measurement
time points (see Murayama et al., 2017), whereas much of the previous research has relied on
Book chapter ESM in research on emotion and motivation
9
between-person analyses. It has been argued that such within-person methods are often more
suited than the previously dominating between-person methods to answer psychological
research questions (Molenaar, 2004; Moeller, 2021; Richters, 2021): For instance, the
frequently studied measurement models and structural models (the factor structure and
relations among factors) can look vastly different if analyzed within individuals versus
between individuals (Dietrich et al., 2017; Ketonen et al., 2018; Vansteenlandt et al., 2005).
Likewise, developmental trajectories across measurement time points can differ tremendously
if examined within individuals versus between individuals (see Figure 2).
Figure 2. The between-person trajectory (red line) can look very different from the person-
specific (within-person) trajectories (blue and orange).
The ESM can reveal distinct types of situations characterized by different profiles of
motivation or emotional experiences (Bergman, Nurmi, & von Eye, 2012), which is used in
the research on learning to identify states of flow or situational engagement (Inkinen et al.,
2020), to describe distinct types of boredom in accompanied by different emotions (Goetz et
al., 2013b), or to describe distinct situational patterns of task values, success expectancies,
and costs (Dietrich et al., 2019).
Book chapter ESM in research on emotion and motivation
10
Insight 2: Predicting one moment by the previous one. The ESM enables us to
study moment-to-moment changes and stabilities in motivation and emotions, for instance by
asking whether Students’ or teachers’ experiences in a certain moment are predicted by
experiences in a preceding moment (for an overview see Reitzle & Dietrich, 2019).
Typical are autoregressive models where researchers are interested in carry-over
effects of, say, feelings of competence from one situation to the next (Malmberg & Martin,
2019). Other studies investigate temporally structured psychological processes (goal failure
during a self-study day leading to higher anger and lower joy in the evening and to
downgrading learning goals on the next day; Theobald et al., 2021). Defining a relevant
previous moment then becomes an important ingredient of empirical studies: It could pertain
to the past couple of minutes or half an hour ago (Moeller et al., 2022a); or it could pertain to
the previous lesson on the same day or even the previous lesson of the same subject on
another day (Malmberg & Martin, 2019).
Insight 3: Heterogeneity: How people, time points and contexts differ from one
another. The study of within-person structures and processes gives rise to another novel
perspective: The insight that factor structure, relations among constructs and development of
psychological constructs over time can differ heterogeneously between different individuals.
An example for this is the finding reported by Pekrun et al. (2002), who found that the intra-
individual correlation between situated motivation and anxiety differed vastly between
students, with an average within-person correlation of about 0 and a number of students
showing positive correlations and another set of students showing negative correlations
between situational motivation and anxiety. Other links between learning-related emotions
and their antecedents and outcomes might, however, be more universal across individuals
(Berweger et al., 2022). It is thus a milestone that heterogeneity vs. universality in moment-
to-moment processes can explicitly be tested rather than be just assumed (see also Neubauer
et al., 2019).
Book chapter ESM in research on emotion and motivation
11
Similar to inter-individual differences reflecting heterogeneity between people, the
characteristics of different situations or contexts (see Rauthmann et al., 2018; 2020) might
affect momentary emotions and motivation.
Insight 4: Developmental dynamics. The above-mentioned features of ESM studies
together make it possible to study concepts of development proposed by dynamic systems
theories, which we expect to be a major driver of innovation in the research on learning-
related motivation and emotions (see Dietrich et al., 2022; Moeller et al., 2022a). The
conceptual underpinnings of the (meta-theoretical) dynamic systems approach are detailed
elsewhere (Granic & Patterson, 2006; Reitzle & Dietrich, 2019).
For instance, concepts borrowed from the dynamical systems literature can help us
study whether repeated situational experiences (of situational success or interest) ultimately
lead to the emergence of stable person characteristics (of stable ability self-concept, self-
efficacy, or personal interest). In an initial test of this idea in the area of motivation, we found
that university students’ frequency of experiencing task values and success expectancies
during a university lecture predicted their stable motivational dispositions and long-term
changes thereof (Dietrich et al., 2017; 2019).
Technological innovations in the ESM research
The ESM is used and combined with other methods in innovative ways which are
rapidly developing and which seem likely to heavily influence the research on motivation and
emotions in learning- and achievement contexts in the upcoming years.
In-the-moment personalized interventions. Due to the person- and situation-specific
information captured in ESM data, ESM can be used to tailor psychological interventions to
an individual’s needs in exactly the moments in which individuals need the interventions
(Schmitz & Wiese, 2006). ESM surveys can administer personalized instruction that is
tailored to previously assessed characteristics of the person, time point or context. This
Book chapter ESM in research on emotion and motivation
12
feature can be used to send individuals personalized experimental instructions (Schmiedek &
Neubauer, 2020), personalized (mental or other) health treatments (Rodebaugh, Frumkin, &
Piccirillo, 2020), or personalized instruction within personalized learning programs (Azevedo
et al., 2022; Harley et al., 2016).
Sensor-augmented ESM. To link ESM data to the characteristics of persons, time
points, and contexts, researchers increasingly connect ESM data with other sources of
information, such as sensor data automatically recorded by the smartphones used to
administer ESM surveys (location data, surrounding noise levels, close-by devices, app usage
and others) or psychophysiological measurement devices (heart rate variability, electrodermal
resistance, eye tracking data, recorded facial expressions, etc.). Such sensor-augmented ESM
can be used for instance to validate ESM self-reports by triangulating them with objective
behavioral data. Sensor data can also be used to trigger ESM surveys in situations in which
the sensors indicate a certain location, or noise level, or close-by companion, or a particular
physiological state (racing heart rate). Such multi-channel data are used in ESM basic
research, in clinical psychology for the context-specific triggering of personalized
interventions and in educational psychology to administer personalized instructions (Azevedo
et al., 2022). A few apps already allow for the integrated assessment of ESM and sensor data
(see the AWARE Framework; Ferreira, Kostakos, & Dey, 2015 or the software movisens;
Krkovic et al., 2018).
Artificial intelligence. Both in personalized interventions and in sensor-augmented
ESM, we can expect machine learning and artificial intelligence to play a larger role in the
future. We define machine learning here as a machine’s ability to find patterns in data
(relations, clusters) and artificial intelligence as a computer’s ability to make decisions and
simulate human reasoning and productivity based on patterns previously identified in data.
Much of the previous ESM data analyses already rely on widely used machine learning
techniques for nomothetic prediction (regressions) and, to a lesser extent, pattern finding
Book chapter ESM in research on emotion and motivation
13
(cluster analyses; Bergman et al., 2012; Dietrich et al., 2019). While recent innovations go
beyond these traditions and employ for instance network analyses to first estimate person-
specific relations among variables across time points, and to then find out which relations
estimated in the person-specific networks generalize across individuals (Beltz et al., 2016; for
further integrations of idiographic and nomothetic approaches, see Wright & Zimmermann,
2019). Others use sophisticated machine learning to identify the best suited personalized
treatment plan based on an individual’s person characteristics and previous responses to a
treatment (Luedtke & van der Laan, 2016; Montoya et al., 2021). Based on this work,
artificial intelligence could be trained in the future to identify the patterns in ESM and sensor
data that are most predictive of relevant behavior for an individual and then make data-driven
decisions about the best time point to administer the best treatment matching this person’s
previous behavior and previous treatment responses. Examples for such applications can
already be found in intelligent tutoring systems, which use current information about a
student, including sensor or self-report data about current emotions) along with information
about learning requirements to provide students with learning materials and support in
roughly similar ways as a human would.
In sensor-augmented ESM, future studies could use machine learning to integrate
ESM and sensor data to predict relevant behavior (drug use relapse) or relevant psychological
states (high stress leading to clinical symptoms), building upon the previous work using
machine learning to make sense of psychophysiological data (Föhr et al., 2015).
Gamification. Being interrupted by ESM surveys in your everyday life up to ten times
a day can feel annoying and therefore reduce a participants’ compliance and response rate.
Gamification has been suggested as a way to increase the participation and retention in ESM
studies by making responding to ESM surveys more enjoyable and motivating for the
Book chapter ESM in research on emotion and motivation
14
participants (van Berkel et al., 2017). Gamification can serve as an instrument to increase the
joy and positive emotions associated by the participants with ESM studies. The hope is that
this will increase compliance and response rates. In particular, the use of leaderboards and
score as gamification elements were effective in increasing the motivation of participants.
More research is needed to find out which further gamification elements can increase
compliance and response rates and to study whether gamification in ESM studies may change
the assessed emotions or motivation, which could limit the validity of gamified ESM surveys.
Current challenges in the ESM research and directions for future research
Many of the current challenges in the research with the ESM can best be understood in
the context of the discussions about the broader trustworthiness crisis in the social sciences.
These discussions have pointed out that the research practices in Social Sciences often do not
live up to the established standards of good research as we teach them to students. Table 2
gives an overview of the meaning of each of these discrepancies between established
standards and wide-spread practices, along with examples of how each challenge is relevant
in the ESM research on learning-related emotions and motivation.
Table 2. Research practices limiting the trustworthiness of psychological findings and their
relevance in research with the ESM
Challenge
Relevance in the ESM research on learning-related emotions & motivation
Theory crisis
Eronen & Bring mann,
2021
theory-meth od gaps in man y ES M studies; many ES M stu dies app ear more data -driven than theory-driven; few
theories specify effects on the levels on which E SM data are analyzed (few theories specifying situation- &
context-specificity and intra-individual structures and processes
Construct validity
crisis
Schimmack, 2021
Since ESM measu res need to be brief, they t ypically capture on ly few aspects of multifaceted motivation al
constructs. Jingle and jangle fallacies plaguing the research on motivation lead to confusion over definitions,
overlaps and differenc es be tween constru cts an d the measures supposed to represent them.
Measurement crisis
Flake & Fried, 2020
Information about psychometric properties are lacking for most employed ESM measures. Many items are
created ad hoc for a new study without being validated. The measurement models used to measure emotions an d
motivations may be theoretically implausible and this debate is only starting (Lange et al., 2020)
Normativity crisis
Lundh, 2019
Due to the norms of how to write and publish, many ESM studies use exploratory approaches but pretend their
approaches were hypothetico-deductive. Unexpected findings are either not reported or declared to be the result
of deductive hypothesis-testing. At the same time, researchers dedicated to a deductive research logic may sh y
away from a much-needed systematic exploration of the boundary conditions that may limit the generalizability
of ESM findings.
Inference crisis
Starns et al., 20 19;
Syed, 2021; closely
related to the
Generalizability crisis
Yarkoni, 2022
ESM studies are often limited with relatively few individuals from only a handful of schools or organizations in
few countries, rarely including enough data from different cultural or ethnic groups. At the same time, their
results are often interpreted as if they were universally valid. ESM data incl ude so many so complex,
multilayered, and n ot yet well un ders tood sourc es of var iation that un known bou nda ry con ditions (hidden
moderators) seem often likely and therefore generalizability is unknown and hard to establish (Moeller et al.,
2022b)
Book chapter ESM in research on emotion and motivation
15
Replicability crisis
Ioannidis, 2005; 2012;
Nosek et al., 2022
Due to rapid innovations in this quickly developing field, many ESM studies are the first of their kind and
consequently warrant replications.
In some regards, ESM studies pose challenges that go beyond those being discussed in
the debates following the replicability crisis. These challenges have been discussed in two
articles by the ManyMoments Consortium (Moeller et al., 2022b; Moeller et al., in prep.). In
the following, we discuss some of the challenges mentioned in Table 2 more in detail, along
with corresponding directions for future research.
Theory. In ESM research, new methods and unexpected findings often precede
theoretical explanations, which conflicts with the widely accepted epistemological foundation
of deductive hypothesis testing. Few theories are yet sophisticated enough to address the
heterogeneity between situations, individuals, and contexts in as much detail as ESM findings
often reveal. Many theories seem to have limited themselves to the statements that previously
available methods were able to examine, while the rapid development of ESM methods and
technology opens new horizons that previous theory builders did not seem to have even
dreamed about. Hence, there is a need to update theories on academic motivation and
emotions addressing variability and generalizability across situations, persons, and contexts
(Dietrich et al., 2022; Moeller et al., 2022b; 2022c).
Perspectives enabled by recent methodological advances that we expect will and
should shape future theory development are, for instance, (1) iterative feedback loops,
referring to the possibility that a construct may be at the same time a predictor and an
outcome of another construct in self-reinforcing processes (vicious circles or virtuous circles
of motivational development, e.g. Pekrun, 2021); (2) the relation between momentary
experiences and stable dispositions, including the question how traits determine the levels and
frequencies of situational emotions and motivational states, or whether the frequency and
intensity of situational experiences leads to the emergence of stabilizing emotional or
motivational dispositions; and (3) heterogeneity and lacking ergodicity (discrepancies in
Book chapter ESM in research on emotion and motivation
16
findings between different levels of analyses, within-person versus between-person) requiring
theoretical explanations. Integrating these perspectives into the research on motivation and
emotions can benefit from theoretical concepts and mathematical instruments that can be
found in the meta-theoretical ideas of the dynamic systems thinking (Moeller et al., 2022a).
Construct validity and measurement. ESM data are affected by the typical
limitations of self-report data, including response styles and subjective perceptions. The hope
is that asking participants about their experiences in the moments in which they occur will
reduce memory errors (Takarangi et al., 2006) as well as stereotypical response biases related
to identity, such as gender (Goetz et al., 2013a). The limitations of self-report data can be
ameliorated by triangulating ESM data with objective observations of behavior, for instance
in sensor-augmented ESM (Ferreira et al., 2015) or through behavioral data in online learning
environments (Azevedo et al., 2022). An alternative is to survey groups of individuals at the
same time in the same context and then use statistical methods to disentangle subjective
perceptions of a situation (a person’s deviation from the group trend) from objective situation
characteristics (indicated by the group trend; see Moeller et al., 2020).
While there is a hope that integrating ESM data with psychophysiological assessments
of emotions will help to validate ESM measures, it has also been argued that the different
channels of information (self-report, psychophysiological measures) assess different, not
entirely exchangeable aspects of emotions (Barrett, 2017), suggesting that the subjectivity of a
self-report can be considered to be a strength as much as a weakness. More research is needed
to find out which exact components of emotions and motivation and which exact outcomes
are best indicated by which combination of ESM self-reports psychophysiological measures
and other sensor data.
There is a lack of knowledge about the psychometric properties of ESM measures,
because many item sets are developed ad hoc by researchers and because methods to ascertain
the reliability of the typical single-item measures are still being developed (Gogol et al.,
Book chapter ESM in research on emotion and motivation
17
2014). More systematic validation studies in combination with repositories of validated ESM
measures (see Kirtley et al., 2018) are needed to obtain trustworthy information about their
reliability, validity, objectivity, measurement invariance across time and across contexts,
translations, and the generalizability of these information.
Analytical methods. The complex structure of ESM data with their multilayered
sources of variation (see Figure 1) poses challenges to the analysis of ESM data. Ever new
analytical methods become available at rapid speed, enabling researchers to address new
types of research questions (such as feedback loops, heterogeneity, see above). Still, there are
statistical challenges, for example in disentangling the variation between time points,
individuals, and contexts even in complex analyses. For example, consider how the
nestedness of students in classrooms is ignored in current DSEM analyses, (Neubauer et al.,
2022).
Many novel analytical approaches are relatively complicated and require a level of
expertise that is rare due to the novelty of the methods. Keeping up with the methodological
developments can be a challenge even for experts in the field. There is a need for accessible
tutorials to build the required methodological expertise, which is addressed in various recent
publications (Fritz et al., in prep; Piccirillo et al., in prep; Gates, Chow, & Molenaar, 2023;
Myin-Germeys & Kuppens, 2022 )
Missing data are a particular problem in ESM studies, since missingness is frequent
due to the intrusiveness of ESM measures and the need for participants to respond multiple
times. Since many ESM-measured constructs are situation- and context-specific with much
variation from one moment to the next, it is questionable whether the responses of a missed
ESM survey should be imputed or otherwise estimated based on existing data, thus limiting
the usual kludges of estimate missing data based on available observations. In the research on
learning-relevant motivation and emotions, it often seems plausible to assume a missingness
not at random, if for instance characteristics of the context, time, or person determine how
Book chapter ESM in research on emotion and motivation
18
likely a person is to respond (think of nonresponse due to being totally absorbed in an exciting
activity, or due to being too depressed, frustrated or stressed to answer, see Wilson et al.,
1992). There is a need for more research on how to avoid missingness via better instructions,
fewer or better timed surveys, or measures ensuring optimal participant compliance. There
also is a need to reflect the limitations of given missing data in the interpretation of results.
Inference and generalizability. ESM studies pose known and less known, unique
challenges to the trustworthiness of research findings (for overviews, see Moeller et al.,
2022b; Moeller et al., in prep.). A challenge to ascertaining the replicability and
generalizability of ESM findings is that ESM is used to study phenomena that are expected to
greatly vary between contexts, time points, and individuals, while very little theoretical
knowledge and hypotheses about the exact sources of variation and their mechanisms of
influencing motivation and emotion are available. That ESM studies are typically conducted
in natural every-day life settings makes it harder to control and monitor sources of variation
than in typical lab experimental studies. At the same time we expect ESM data to be
influenced by many unknown and uncontrolled sources of variation which are multilayered
and nested, implying potentially limited generalizability across time points, contexts, and
persons. There are so far few systematic replication ESM studies, while many studies are the
first of their kind, warranting replications. Multilab data collections and multi-lab data
analyses have been introduced as possible solutions to these challenges (Bastiaansen et al.,
2020; Moeller et al., in prep.).
With lacking specific hypotheses and missing empirical knowledge about sources of
variation, there is a high likelihood that unknown boundary conditions (also called hidden
moderators) may limit the generalizability of many ESM findings. Even if a large number of
previous replication studies had found an effect to be replicable and generalizable across the
previously considered contexts, we can therefore never rule out that a previously
unconsidered person or context characteristic may limit the generalizability of an ESM
Book chapter ESM in research on emotion and motivation
19
finding. This implies that an ESM finding may look generalizable in regard to known factors
when it is non-generalizable in regard to not yet considered factors, or that an ESM finding
looks like it may be non-replicable when in fact it would have been replicable but differed
between two studies due to unknown boundary conditions (which is a lack of generalizability,
not replicability). This requires us to consider new definitions and define and determine
replicability and generalizability in the face of multilayered complex heterogeneity (see
Moeller et al., 2022b).
Until systematic knowledge about the generalizability of an ESM finding is available,
researchers need to hypothesize about possible boundary conditions of their findings in
limitation sections: Might the findings be limited to particular teachers, a particular school,
particular classrooms, subjects, or a neighborhood? Are there hypotheses about possible
boundary conditions that future studies need to address more systematically?
A template for pre-registering ESM studies was proposed to increase the deductive
research process and to increase replicability by preventing problematic practices such as
HARKing or p-hacking (Kirtley et al., 2020). Another challenge to ascertaining the
replicability and generalizability of ESM findings is the problem that a lack of repeatability,
reproducibility, or robustness (for the difference, see Moeller et al., in prep.) can look
deceivingly similar to a lack of replicability and generalizability. ESM researchers have
manifold degrees of freedom in the analytics choices they make when preparing and
analyzing their data (Bastiaansen et al., 2020). It has therefore been suggested that robustness
checks (i.e., sensitivity analyses) be applied across different choice options (like which data to
exclude from the analysis) - a procedure called multiverse multiverse analysis (Weermeijer et
al., 2022).
Some researchers have argued that ESM findings can be expected to be generalizable
if randomly timed surveys are scheduled, based on the idea that randomly scheduled surveys
provide a random sample of everyday life activities, which in turn is expected to be
Book chapter ESM in research on emotion and motivation
20
representative and therefore generalizable. However, due to the many layers of sources of
variation (see Figure 1), a random sample of time points across a day neither guarantees a
representative sample of individuals, not of meso- or macro-level contexts (schools, countries,
etc), which limits the generalizability of ESM findings towards other individuals or macro-
level contexts. Moreover, ESM surveys may be randomly scheduled, but a typical sample of
10 daily surveys risks missing rare occurrences and even 70 weekly surveys of two minutes
each cover only 70*2=140 minutes of a week, which represents only 0.7% of the 20,160
minutes in a working week. Thus, if an ESM survey only asks about the current experiences,
there is a considerable chance of missing any experience that occurs rarer than 99% of the
time. Therefore, it can make sense to a priori limit the contexts and time points of ESM
surveys before scheduling them, to make sure to capture enough of those time points and
contexts most relevant to a given research question.
Book chapter ESM in research on emotion and motivation
21
References
Arslan, R. C., Tata, C. S., & Walther, M. P. (2018). ESM & mobile sensing solutions: Feature
table. Retrieved from: https://comparison-to.formr.org In: Arslan, R.C., Tata, C.S. &
Walther, M.P. (2018). formr: A study framework allowing for automated feedback
generation and complex longitudinal experience sampling studies using R. (version
v0.18.3). https://doi.org/10.5281/zenodo.5211425
Azevedo, R., Bouchet, F., Duffy, M., Harley, J., Taub, M., Trevors, G., Cloude, E., Dever, D.,
Wiedbusch, M., Wortha, F., & Cerezo, R. (2022). Lessons learned and future directions
of MetaTutor: Leveraging multichannel data to scaffold self-regulated learning with an
intelligent tutoring system. Frontiers in Psychology, 13, 813632.
https://doi.org/10.3389/fpsyg.2022.813632
Barrett, L. F. (2017). How emotions are made: The secret life of the brain. Pan Macmillan.
Bastiaansen, J. A., Kunkels, Y. K., Blaauw, F. J., Boker, S. M., Ceulemans, E., Chen, M.,
Chow, S. M., de Jonge, P., Emerencia, A. C., Epskamp, S., Fisher, A. J., Hamaker, E.
L., Kuppens, P., Lutz, W., Meyer, M. J., Moulder, R., Oravecz, Z., Riese, H., Rubel, J.,
Ryan, O., … & Bringmann, L. F. (2020). Time to get personal? The impact of
researchers choices on the selection of treatment targets using the experience sampling
methodology. Journal of Psychosomatic Research, 137, 110211. Advance online
publication. https://doi.org/10.1016/j.jpsychores.2020.110211
Beck, E. D., & Jackson, J. J. (2020). Consistency and change in idiographic personality: A
longitudinal ESM network study. Journal of Personality and Social Psychology, 118(5),
1080–1100. https://doi.org/10.1037/pspp0000249
Beltz, A. M., Wright, A. G. C., Sprague, B. N., & Molenaar, P. C. M. (2016). Bridging the
Nomothetic and Idiographic Approaches to the Analysis of Clinical Data. Assessment,
23(4), 447–458. https://doi.org/10.1177/1073191116648209
Book chapter ESM in research on emotion and motivation
22
Bergman, L. R., Nurmi, J. E., & von Eye, A. A. (2012). I-states-as-objects-analysis (ISOA):
Extensions of an approach to studying short-term developmental processes by analyzing
typical patterns. International Journal of Behavioral Development, 36(3), 237-246.
https://doi.org/10.1177/0165025412440947
Bergman, L. R., Nurmi, J. E., & Eye, A. A. V. (2012). I-states-as-objects-analysis (ISOA):
Extensions of an approach to studying short-term developmental processes by analyzing
typical patterns. International Journal of Behavioral Development, 36(3), 237-246.
https://doi.org/10.1177/0165025412440947
Berweger, B., Kracke, B., & Dietrich, J. (2021). Pre-Service Teachers’ Epistemic Emotions
when confronted with their Misconceptions about Education. Presentation on the
European Society of Psychology Learning and Teaching (ESPLAT) conference
(Organized Symposium “Studentsʼ metacognitive ability, epistemic beliefs, and
epistemic emotions when confronted with conflicting information in psychology and
educational science”, Chair: Belinda Berweger) [digital].
Csikszentmihalyi, M. (1975). Beyond boredom and anxiety. San Francisco: Jossey-Bass
Publishers.
Csikszentmihalyi, M., & Rathunde, K. (1993). The measurement of flow in everyday life:
Toward a theory of emergent motivation. In J. E. Jacobs (Ed.), Nebraska Symposium on
Motivation, 1992: Developmental perspectives on motivation (pp. 57–97). University of
Nebraska Press.
Dietrich, J., Viljaranta, J., Moeller, J., & Kracke, B. (2017). Situational expectancies and task
values: Associations with students' effort. Learning and Instruction, 47, 53–64.
https://doi.org/10.1016/j.learninstruc.2016.10.009
Dietrich, J., Moeller, J., Guo, J., Viljaranta, J., & Kracke, B. (2019). In-the-Moment Profiles
of Expectancies, Task Values, and Costs. Frontiers in Psychology, 10, 1662.
https://doi.org/10.3389/fpsyg.2019.01662
Book chapter ESM in research on emotion and motivation
23
Dietrich, J., Schmiedek, F., & Moeller, J. (2022). Academic Motivation and Emotions are
Experienced in Learning Situations, so let's Study Them. Introduction to the Special
Issue. Learning & Instruction, 81, 101623.
https://doi.org/10.1016/j.learninstruc.2022.101623
Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-
value theory: A developmental, social cognitive, and sociocultural perspective on
motivation. Contemporary Educational Psychology, 61, 101859.
https://doi.org/10.1016/j.cedpsych.2020.101859
Eccles, J. S. (2022). Commentary on within-person designs and motivational science.
Learning and Instruction, 81, 101662.
https://doi.org/10.1016/j.learninstruc.2022.101662
Edwards, O. V., & Taasoobshirazi, G. (2022). Social presence and teacher involvement: The
link with expectancy, task value, and engagement. The Internet and Higher Education,
55, 100869. https://doi.org/10.1016/j.iheduc.2022.100869
Engeser, S. & Rheinberg, F. (2008). Flow, moderators of challenge-skill-balance and
performance. Motivation and Emotion, 32, 158-172. doi:10.1007/s11031-008-9102-4
Eronen, M. I., & Bringmann, L. F. (2021). The theory crisis in psychology: How to move
forward. Perspectives on Psychological Science, 16(4), 779–788.
https://doi.org/10.1177/174569162097058
Fahrenberg, J. (1996). Ambulatory assessment: Issues and perspectives. Ambulatory
assessment: Computer-assisted psychological and psychophysiological methods in
monitoring and field studies, 3-20. In Fahrenberg, J., & Myrtek, M. (Eds.). Ambulatory
assessment: Computer-assisted psychological and psychophysiological methods in
monitoring and field studies. Seattle, WA: Hogrefe and Huber.
Book chapter ESM in research on emotion and motivation
24
Ferreira, D., Kostakos, V. & Dey, A.K. (2015). AWARE: mobile context instrumentation
framework. Frontiers in ICT 2:6. https://doi.org/10.3389/fict.2015.00006
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement
practices and how to avoid them. Advances in Methods and Practices in Psychological
Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
Fleeson, W. (2007). Using experience sampling and multilevel modeling to study person-
situation interactionist approaches to positive psychology. In A. D. Ong & M. H. M. van
Dulmen (Eds.), Oxford handbook of methods in positive psychology (pp. 501–514).
Oxford University Press.
Föhr, T., Tolvanen, A., Myllymäki, T., Järvelä-Reijonen, E., Rantala, S., Korpela, R.,
Peuhkuri, K., Kolehmainen, M., Puttonen, S., Lappalainen, R., Rusko, H. & Kujala, U.
M. (2015). Subjective stress, objective heart rate variability-based stress, and recovery
on workdays among overweight and psychologically distressed individuals: a cross-
sectional study. Journal of Occupational Medicine and Toxicology, 10(1), 1-9.
https://doi.org/10.1186/s12995-015-0081-6
Fried, E.I., Papanikolaou, F., & Epskamp, S. (2020). Mental Health and Social Contact
During the COVID-19 Pandemic: An Ecological Momentary Assessment Study.
Clinical Psychological Science, 10, 340 - 354.
https://doi.org/10.1177/21677026211017839
Fritz, J., Piccirillo, M.L., Cohen, Z.D., Frumkin, M., Kirtley, O., Moeller, J., Neubauer, A.B.,
Norris L.A., Schuurman, N.K., Snippe & E., Bringmann, L.F. (in prep.). So you want to
do ESM? A brief overview on ESM design and implementation. Manuscript in
preparation.
Gates, K.M., Chow, S., & Molenaar, P.C.M. (2023). Intensive Longitudinal Analysis of
Human Processes (1st ed.). Chapman and Hall/CRC .
Book chapter ESM in research on emotion and motivation
25
Goetz, T., Frenzel, A. C., Pekrun, R., & Hall, N. C. (2006). The domain specificity of
academic emotional experiences. Journal of Experimental Education, 75(1), 5–29.
https://doi.org/10.3200/JEXE.75.1.5-29
Goetz, T., Bieg, M., Lüdtke, O., Pekrun, R., & Hall, N. C. (2013a). Do girls really experience
more anxiety in mathematics?. Psychological science, 24(10), 2079-2087.
https://doi.org/10.1177/0956797613486989
Goetz, T., Frenzel, A. C., Hall, N. C., Nett, U. E., Pekrun, R., & Lipnevich, A. A. (2013b).
Types of boredom: An experience sampling approach. Motivation and emotion, 38(3),
401-419. https://doi.org/10.1007/s11031-013-9385-y
Gogol, K., Brunner, M., Goetz, T., Martin, R., Ugen, S., Keller, U., Fischbach, A., & Preckel,
F. (2014). ‘My questionnaire is too long!’ The assessments of motivational-affective
constructs with three-item and single-item measures. Contemporary Educational
Psychology, 39, 188–205. https://doi.org/10.1016/j.cedpsych.2014.04.002
Granic, I., Patterson, G.R. (2006) Toward a comprehensive model of antisocial development:
a dynamic systems approach. Psychological Review. (1):101-131.
https://doi.org/10.1037/0033-295x.113.1.101.
Hall, M., Scherner, P. V., Kreidel, Y., & Rubel, J. A. (2021). A systematic review of
momentary assessment designs for mood and anxiety symptoms. Frontiers in
Psychology, 12, 642044. https:// doi.org/10.3389/fpsyg.2021.642044
Hamaker, E. L., & Wichers, M. (2017). No time like the present: Discovering the hidden
dynamics in intensive longitudinal data. Current Directions in Psychological Science,
26(1), 10-15. https://doi.org/10.1177/0963721416666518
Harley, J., Lajoie, S., Frasson, C. & Hall, N. (2016). Developing Emotion-Aware, Advanced
Learning Technologies: A Taxonomy of Approaches and Features. International
Book chapter ESM in research on emotion and motivation
26
Journal of Artificial Intelligence in Education. 27. https://doi.org/10.1007/s40593-016-
0126-8
Hektner, J.M., Schmidt, J.A., & Csikszentmihalyi, M. (2007). Experience sampling method :
measuring the quality of everyday life. Thousand Oaks, CA : Sage Publications
Inkinen, J., Klager, C., Juuti, K., Schneider, B., Salmela-Aro, K., Krajcik, J.S., & Lavonen, J.
(2020). High school students' situational engagement associated with scientific practices
in designed science learning situations. Science Education, 104, 667-692.
https://doi.org/10.1002/sce.21570
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8),
e124. https://doi.org/10.1371/journal.pmed.0020124
Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspectives on
Psychological Science, 7(6), 645–654. https://doi.org/10.1177/1745691612464056
Ketonen, E. E., Dietrich, J., Moeller, J., Salmela-Aro, K., & Lonka, K. (2018). The role of
daily autonomous and controlled educational goals in students' academic emotion states:
An experience sampling method approach. Learning and Instruction, 53, 10-20.
https://doi.org/10.1016/j.learninstruc.2017.07.003
Kirtley, O., Hiekkaranta, A. P., Kunkels, Y. K., Verhoeven, D., van Nierop, M., & Myin-
Germeys, I. (2018). The Experience Sampling Method (ESM) Item Repository.
Retrieved from: https://osf.io/kg376/#!. https://doi.org/10.17605/OSF.IO/KG376 (See
also https://www.esmitemrepository.com)
Kirtley, O. J., Lafit, G., Achterhof, R., Hiekkaranta, A. P., & Myin-Germeys, I. (2020). A
template and tutorial for (pre-)registration of studies using Experience Sampling
Methods (ESM). https://doi.org/10.17605/OSF.IO/2CHMU
Book chapter ESM in research on emotion and motivation
27
Krkovic, K., Krink, S., & Lincoln, T. (2018). Emotion regulation as a moderator of the
interplay between self-reported and physiological stress and paranoia. European
Psychiatry, 49, 43-49. https://doi.org/10.1016/j.eurpsy.2017.12.002
Kühnel, J., Sonnentag, S., Bledow, R., & Melchers, K. G. (2017). The relevance of sleep and
circadian misalignment for procrastination among shift workers. Journal of
Occupational and Organizational Psychology, 91(1), 110-133.
https://doi.org/10.1111/joop.12191
Larson, R. & Csikszentmihalyi, M. (2014). The Experience Sampling Method. In: M.
Csikszentmihalyi, Flow and the Foundations of Positive Psychology. The Collected
Works of Mihaly Csikszentmihalyi (pp. 21-34). https://doi.org/ 10.1007/978-94-017-
9088-8_1
Luedtke, A., & van der Laan, M.J. (2016). Super-Learning of an Optimal Dynamic Treatment
Rule. The International Journal of Biostatistics, 12, 305 - 332.
https://doi.org/10.1515/ijb-2015-0052
Lundh, L. G. (2019). The crisis in psychological science, and the need for a person-oriented
approach. In J. Valsiner (Ed.), Social Philosophy of Science for the Social Sciences
(pp. 203–233). Springer. https://doi.org/10.1007/978-3
Malmberg, L. E., & Martin, A. J. (2019). Processes of students’ effort exertion, competence
beliefs and motivation: Cyclic and dynamic effects of learning experiences within school
days and school subjects. Contemporary Educational Psychology, 58, 299-309.
Moeller, J., Salmela-Aro, K., Lavonen, J., & Schneider, B. (2015). Does Anxiety in Science
Classrooms Impair Science Motivation? -Gender Differences Beyond the Mean Level.
International Journal of Gender, Science and Technology, 7(2), 229–254. Retrieved
from https://genderandset.open.ac.uk/index.php/genderandset/article/view/398
Book chapter ESM in research on emotion and motivation
28
Moeller, J., Viljaranta, J., Kracke, B., & Dietrich, J. (2020). Disentangling objective
characteristics of learning situations from subjective perceptions thereof, using an
experience sampling method design. Frontline Learning Research, 8(3), 63–84.
https://doi.org/10.14786/flr.v8i3.529
Moeller, J. (2021). Averting the next credibility crisis in psychological science: Within-person
methods for personalized diagnostics and intervention. Journal for Person-Oriented
Research, 7(2), 53–77. https://doi.org/10.17505/jpor.2021.23795
Moeller, J., Viljaranta, J., Tolvanen, A., Kracke, B., & Dietrich, J. (2022a). Introducing the
DYNAMICS framework of moment-to-moment development in achievement
motivation. Learning and Instruction. Advance online publication.
https://doi.org/10.1016/j.learninstruc.2022.101653
Moeller, J., Dietrich, J., Neubauer, A., Brose, A., Kühnel, J., Dehne, M., Jähne, M.,
Schmiedek, F., Bellhäuser, H., Malmberg, L.-E., Stockinger, K., Riediger, M., &
Pekrun, R. (2022b). Generalizability Crisis Meets Heterogeneity Revolution:
Determining Under Which Boundary Conditions Findings Replicate and Generalize.
Pre-print: https://doi.org/10.31234/osf.io/5wsna
Moeller, J., Langener, A., Lafit, G., Karhulahti, V.-M., Bastiaansen, J., & Bergmann, C.
(2022c). The Hypository: Registering Hypotheses for Cumulative Science. Pre-print:
https://doi.org/10.31234/osf.io/5qgj7
Moeller, J., Bergmann, C., Bringmann, L., Bastiaansen, J., Schmiedek, F., Loderer, K.,
Riediger, M., Pekrun, R. and the ManyMoments Consortium (in prep.). ManyMoments -
Improving replicability of experience sampling method studies in multi-lab
collaborations. Manuscript in preparation.
Molenaar, P. C. M. (2004). A manifesto on psychology as idio-graphic science: Bringing the
person back into scientific psy-chology, this time forever. Measurement:
Book chapter ESM in research on emotion and motivation
29
Interdisciplinary Research and Perspectives, 2(4), 201-218.
https://doi.org/10.1207/s15366359mea0204_1
Montoya, L. M., van der Laan, M. J., Luedtke, A. R., Skeem, J. L., Coyle, J. R., & Petersen,
M. L. (2022). The optimal dynamic treatment rule superlearner: considerations,
performance, and application to criminal justice interventions. The International Journal
of Biostatistics. https://doi.org/10.1515/ijb-2020-0127
Murayama, K., Goetz, T., Malmberg, L-E., Pekrun, R., Tanaka, A., & Martin, A.J. (2017).
Within-person analysis in educational psychology: Importance and illustrations. In D.
W. Putwain, & K. Smart (Eds.). British Journal of Educational Psychology Monograph
Series II, 12, 71-87. ISSN: 1476-9808 and ISBN: 978-1-85433-752-8
Myin-Germeys, I., & Kuppens, P. (Eds.). (2022). The open handbook of experience sampling
methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies
(2nd ed.). Leuven: Center for Research on Experience Sampling and Ambulatory
Methods Leuven.
Nakamura, J. & Csikszentmihalyi, M. (2002). The concept of flow. In C. R. Snyder & S. J.
Lopez (Eds.), Handbook of Positive Psychology (pp. 89-105). Oxford, UK: Oxford
University Press.
Nakamura, J., & Csikszentmihalyi, M. (2009). ' Flow Theory and Research'. In Shane J.
Lopez, and C. R. Snyder (Eds), The Oxford Handbook of Positive Psychology (2nd ed.
online ed, Oxford Academic, 18 Sept. 2012).
https://doi.org/10.1093/oxfordhb/9780195187243.013.0018
Neubauer, A. B., Dirk, J., & Schmiedek, F. (2019). Momentary working memory
performance is coupled with different dimensions of affect for different children: A
mixture model analysis of ambulatory assessment data. Developmental Psychology,
55(4), 754–766. https://doi.org/10.1037/dev0000668
Book chapter ESM in research on emotion and motivation
30
Neubauer, A.B., Schmidt, A., Schmiedek, F., & Dirk, J. (2022). Dynamic reciprocal relations
of achievement goals with daily experiences of academic success and failure: An
ambulatory assessment study. Learning and Instruction, 81, 101617.
https://doi.org/10.1016/j.learninstruc.2022.101617
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler,
F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A.
M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S. (2022). Replicability, robustness,
and reproducibility in psychological science. Annual Review of Psychology, 73(1),
719–748. https://doi.org/10.1146/annurev-psych-020821-114157
Parrisius, C., Gaspard, H., Zitzmann, S., Trautwein, U., & Nagengast, B. (2021). The
“situative nature” of competence and value beliefs and the predictive power of
autonomy support: A multilevel investigation of repeated observations. Journal of
Educational Psychology, 114(4), 791–814. https://doi.org/10.1037/edu0000680
Pekrun, R. (2021). Self-appraisals and emotions: A generalized control-value approach. In T.
Dicke, F. Guay, H. W. Marsh, R. G. Craven, & D. M. McInerney (Eds.), Self – A
multidisciplinary concept (pp. 1–30). Information Age Publishing.
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-
regulated learning and achievement: A program of qualitative and quantitative research.
Educational Psychologist, 37(2), 91–106. https://doi.org/10.1207/S15326985EP3702_4
Pekrun, R., & Marsh, H. W. (2022). Research on situated motivation and emotion: Progress
and open problems. Learning and Instruction, 81, 101664.
https://doi.org/10.1016/j.learninstruc.2022.101664
Piccirillo, M. L., Fritz, J., Cohen, Z. D., Frumkin, M. R., Kirtley, O., Moeller, J., Neubauer,
A. B., Norris, L. A., Schuurman, N. K., Snippe, E., Bringmann, L. F. (in prep.). So you
want to do ESM? Part II: A momentary assessment on the future of ESM research.
Manuscript in preparation.
Book chapter ESM in research on emotion and motivation
31
Rauthmann, J. F., & Sherman, R. A. (2018). The description of situations: Towards replicable
domains of psychological situation characteristics. Journal of Personality and Social
Psychology, 114(3), 482–488. https://doi.org/10.1037/pspp0000162
Rauthmann, J.F., Horstmann, K.T., & Sherman, R.A. (2020). The Psychological
Characteristics of Situations: Towards an Integrated Taxonomy. In Rauthmann, J.F.,
Sherman, R.A. & Funder, D.C. (Eds), The Oxford Handbook of Psychological
Situations (online Edn, Oxford Academic), 25, 389–403
https://doi.org/10.1093/oxfordhb/9780190263348.013.19
Reitzle, M., & Dietrich, J. (2019). From between-person statistics to within-person dynamics.
Diskurs Kindheits- und Jugendforschung, 14, 323–342.
https://doi.org/10.3224/diskurs.v14i3.06
Richters, J. E. (2021). Incredible utility: The lost causes and causal debris of psychological
science. Basic and Applied Social Psychology, 43(6), 366–405.
https://doi.org/10.1080/01973533.2021.1979003
Rodebaugh, T. L., Frumkin, M. R., & Piccirillo, M. L. (2020). The long road from person-
specific models to personalized mental health treatment. BMC medicine, 18(1), 1-2.
https://doi.org/10.1186/s12916-020-01838-w
Salmela-Aro, K., Moeller, J., Schneider, B., Spicer, J., & Lavonen, J. (2016). Integrating the
light and dark sides of student engagement using person-oriented and situation-specific
approaches. Learning and Instruction, 43, 61-70.
https://doi.org/10.1016/j.learninstruc.2016.01.001
Schimmack, U. (2021). The validation crisis in psychology. Meta-Psychology, 5, 1-9.
https://doi.org/10.15626/MP.2019.1645
Schmiedek, F., & Neubauer, A. B. (2020). Experiments in the wild: Introducing the within-
person encouragement design. Multivariate Behavioral Research, 55, 256–276.
https://doi.org/10.1080/00273171.2019.1627660
Book chapter ESM in research on emotion and motivation
32
Schmitz, B., & Wiese, B.S. (2006). New perspectives for the evaluation of training sessions in
self-regulated learning: Time-series analyses of diary data. Contemporary Educational
Psychology, 31, 64-96. https://doi.org/10.1016/J.CEDPSYCH.2005.02.002
Schneider, B., Krajcik, J., Lavonen, J., Salmela-Aro, K., Broda, M., Spicer, J., Bruner, J.,
Moeller, J., Linnansaari, J., Juuti, K., & Viljaranta, J. (2016). Investigating optimal
learning moments in U.S. and Finnish science classes. Journal of Research in Science
Teaching, 53(3), 400–421. https://doi.org/10.1002/tea.21306
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment.
Annual Review of Clinical Psychology, 4, 1-32. https://doi.org/
10.1146/annurev.clinpsy.3.022806.091415.
Sonnenberg, B., Riediger, M., Wrzus, C., & Wagner, G. G. (2012). Measuring time use in
surveys–concordance of survey and experience sampling measures. Social Science
Research, 41(5), 1037-1052. https://doi.org/10.1016/J.SSRESEARCH.2012.03.013
Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J., Aschenbrenner, A., Bröder, A., Cox,
G., Criss, A., Curl, R.A., Dobbins, I.G., Dunn, J., Enam, T., Evans, N., Farrell, S.,
Fraundorf, S.H., Gronlund, S.D., Heathcote, A., Heck, D.W., Hicks, J.L., Huff, M.J. …
Wilson, J. (2019). Assessing theoretical conclusions with blinded inference to
investigate a potential inference crisis. Advances in Methods and Practices in
Psychological Science, 2(4), 335-349. https://doi.org/10.1177%2F2515245919869583
Syed, M. (2021). Reproducibility, diversity, and the crisis of inference in psychology. Pre-
print: https://doi.org/10.31234/osf.io/89buj
Takarangi, M. K., Garry, M., & Loftus, E. F. (2006). Dear diary, is plastic better than paper? I
can't remember: Comment on Green, Rafaeli, Bolger, Shrout, and Reis (2006).
Psychological Methods, 11(1), 119–125. https://doi.org/10.1037/1082-989X.11.1.119
Theobald, M., Breitwieser, J., Murayama, K., & Brod, G. (2021). Achievement emotions
mediate the link between goal failure and goal revision: Evidence from digital learning
Book chapter ESM in research on emotion and motivation
33
environments. Computers in Human Behavior, 119, 106726. https://doi.org/
10.1016/j.chb.2021.106726
van Berkel, N.V., Gonçalves, J., Hosio, S.J., & Kostakos, V. (2017). Gamification of Mobile
Experience Sampling Improves Data Quality and Quantity. Proceedings of the ACM on
Interactive, Mobile, Wearable and Ubiquitous Technologies, 1, 1 - 21.
https://doi.org/10.1145/3130972
Vansteenlandt, K., Van Mechelen, I., & Nezlek, J. B. (2005). The co-occurrence of emotions
in daily life: A multilevel approach. Journal of Research in Personality, 39(3), 325-335.
https://doi.org/10.1016/j.jrp.2004.05.006
Weermeijer, J., Lafit, G., Kiekens, G., Wampers, M., Eisele, G., Kasanova, Z., Vaessen, T.,
Kuppens, P., Myin-Germeys, I., & Myin-Germeys, I. (2022). Applying multiverse
analysis to experience sampling data: Investigating whether preprocessing choices affect
robustness of conclusions. Behavior Research Methods, 1-12.
https://doi.org/10.3758/s13428-021-01777-1
Wilson, K.C.M., Hopkins, R.,deVries, M.W., & Copeland, J.R.M. (1992). ‘Research
alliance and the limit of compliance: Experience sampling with the depressed
elderly’, in deVries, M.W. (Ed.): The Experience of Psychopathology: Investigating
Mental Disorders in their Natural Settings (Cambridge University Press, New York).
Wright, A. G., & Zimmermann, J. (2019). Applied ambulatory assessment: Integrating
idiographic and nomothetic principles of measurement. Psychological Assessment,
31(12), 1467. https://doi.org/10.1037/pas0000685.
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, 1–37.
https://doi.org/10.1017/S0140525X20001685
