Conference PaperPDF Available

On the use of implicit methods for evaluating the impact of interactive narratives on attitudes and behavior (Position paper)



While dominant in research on user experience, the use of self-report methods such as questionnaires or surveys involves some significant limitations for scrutinizing the transformative potential of narratives on users' attitudes and behavior. Here I defend the use of implicit methods, which avoid directly asking the user, as a way to obtain a more detailed picture of such effects, allowing researchers to understand dimensions of the transformative processes beyond what can be verbally expressed by users.
On the use of implicit methods for evaluating
the impact of interactive narratives on
attitudes and behavior (Position paper)
Miguel Barreda-Ángeles
Dept. of Communication Science & Network Institute, VU University Amsterdam,
De Boelelaan 1105, 1081HV Amsterdam
Phone: +31 (0)20 5987101
Abstract. While dominant in research on user experience, the use of self-report methods such as
questionnaires or surveys involves some significant limitations for scrutinizing the transformative
potential of narratives on users’ attitudes and behavior. Here I defend the use of implicit methods,
which avoid directly asking the user, as a way to obtain a more detailed picture of such effects,
allowing researchers to understand dimensions of the transformative processes beyond what can be
verbally expressed by users.
Keywords: Interactive narratives; Implicit methods; Attitudes; Behavior
1. Introduction
The transformative aspects of an interactive narrative experience cannot be
comprehensively described without accounting for its impact on our attitudes and
behavior. This is particularly relevant in the case of narratives with an (explicit or
implicit) pro-social persuasive intention: ideally, they should not only move us
and make us think about the topics depicted but also change our attitudes and
behaviors towards them.
The measurement of attitudes and behavior brings important challenges to
researchers in the field. Self-report techniques such as questionnaires, surveys, or
interviews, offer clear advantages, including the easiness of administration and
analysis, their cheap cost, and their intuitive interpretation, which have made them
the most common methodological approach in user experience research. However,
they ground on the assumption that participants are able and willing to report the
targeted constructs, an assumption that is not always met. Self-report taps into the
introspective abilities of the subjects, and trying to access certain aspects of our
mind through introspection may be "rather like attempting to discern the elements
of water by looking at a drop of water." (Cummins, 2000). Indeed, consistent
evidence from social psychology research shows that we usually lack conscious
access to many of our cognitive processes, and several mental and behavioral
events occur in an automated, unconscious, way (Nisbett & Wilson, 1977; Hassin
et al., 2004). Moreover, individuals' inability to report the effects of stimuli (e.g.
narratives) may also appear in combination with some (conscious or unconscious)
motivations for inaccurate reporting (e.g. social desirability bias, acquiescence
bias, or careless responding) that may call into question the validity of the
information provided.
These issues may particularly affect the self-report of attitudes and behavior.
Indeed, theoretical models of attitudes describe the coexistence of both explicit,
consciously accessible, attitudes, together with implicit, unconscious, ones that,
despite having relevant behavioral implications, are not accessible through self-
report (Petty et al. 2007). In turn, the study of behavior is often tackled through
self-declared behavior or behavioral intentions, but the mismatch between
declared and actual behavior has been well known for a long time (Baumeister et
al. 2009). Hence, and without dismissing the utility of self-report, it is clear self-
report techniques alone are not enough to examine all the possible transformative
effects of interactive narratives.
2. Some alternatives to self-report
Given those limitations, researchers in social psychology and affine fields have
developed a battery of measures to help overcome them. With regards to the
measurement of attitudes, probably the most prominent one is the Implicit
Association Test (IAT) (see Greenwald et al. 2009 for a detailed description). The
test involves measuring the response times of participants making associations
between concepts and attributes across several trials and inferring the strength of a
certain (implicit) attitude from the variations in response times. It has been
extensively used in research on prejudice, and meta-analytical evidence shows
that it explains variability in attitudes and behavior beyond the variability
explained by self-report, particularly in socially sensitive topics (e.g. attitudes
towards different races or sexual orientations) (Greenwald et al. 2009).
Other alternatives for the implicit measurement of attitudinal and behavioral
change are related to brain activity measures. A particularly interesting option,
which is gaining popularity in the last years, is the use of functional near-infrared
spectroscopy (fNIRS), a relatively new, cheap, and portable system for brain
imaging. Research has shown that certain patterns of brain activity are associated
with attitudinal and behavioral change (Burns et al., 2018), making clear the
potential of fNIRS for the investigation of the transformative impact of interactive
Finally, another option is the direct observation of participants' behavior after
being exposed to a narrative in an experimental setup. The idea is to offer the
participants the possibility to perform some action (related to the story topic) that
reveals their actual behavior, such as signing a petition or making a donation to a
NGO. Some very creative approaches have been used for this purpose: for
instance, Fonseca and Kraus (2016) showed, to different groups of participants,
narratives either on the environmental consequences of meat consumption or on
non-related topics. Then, by the end of the experiment, they offered some snacks
to the participants and analyzed how many of them in each group went for
vegetarian options.
The use of these approaches may be challenging for researchers in interactive
narratives in several ways. Among them, there is the technical expertise required,
which may make necessary the collaboration of multidisciplinary teams.
Relatedly, the design of studies and the interpretation of the results may not be
straightforward in many cases, and thus may demand additional efforts from
researchers. Finally, practical aspects such as the increase in costs (e.g. due to
equipment acquisition) and time for running studies need also to be taken into
account. However, the additional effort should pay off: methods such as the IAT
or fNIRS have been shown to predict variability in behavioral outcomes
(associated to exposure to media messages) beyond what is reported by the users
(Greenwald et al. 2009; Burns et al. 2018). Hence, a wide picture of how a
narrative may transform the audience can only be achieved by addressing such
methodological and practical challenges.
The effects of narratives on our attitudes and behavior are a central dimension of
its transformational potential. The utility of self-report methods to explore them is
undeniable, but their limitations are also clear: narratives may affect us in ways
that we are not able to express verbally. Despite the challenges they pose to
researchers, the use of implicit methods such as the IAT, fNIRS, or the direct
observation of participants behavior, should be encouraged, since it will allow
researchers to gain deeper insights on how narratives make us think and act, and,
ultimately, how they change who we are.
Baumeister RF, Vohs KD, Funder DC (2007) Psychology as the science of self-reports and finger
movements: Whatever happened to actual behavior?. Perspect. Psychol. Scienc. 2:396-403.
Burns SM, Barnes LN, Katzman PL, Ames DL, Falk EB, Lieberman MD (2018) A functional near
infrared spectroscopy (fNIRS) replication of the sunscreen persuasion paradigm. Soc. Cognit.
affective neuroscience, 13:628-636
Cummins DD (2000) A history of thinking. In: Cummins R, Cummins DD (eds) Minds, Brains,
and Computers: The Foundations of Cognitive Science: An Anthology, Blackwell, pp. 8-19
Fonseca D, Kraus M (2016). A comparison of head-mounted and hand-held displays for 360
videos with focus on attitude and behavior change. In: Proceedings of the 20th International
Academic Mindtrek Conference, AMD, pp. 287-296
Greenwald AG, Poehlman, TA, Uhlmann EL, Banaji, MR (2009) Understanding and using the
Implicit Association Test: III. Meta-analysis of predictive validity. J. Personalit. Soc. Psychol.
Hassin RR, Uleman JS, Bargh JA (Eds.) (2004) The new unconscious. Oxford University Press.
Nisbett, RE, Wilson TD (1977) Telling more than we can know: Verbal reports on mental
processes. Psychol. Rev. 84: 231-259.
Petty, RE, Briñol P, DeMarree KG (2007) The metacognitive model (MCM) of attitudes:
Implications for attitude measurement, change, and strength. Soc. Cognit., 25:657-686.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Activity in medial prefrontal cortex (mPFC) during persuasive messages predicts future message-consistent behavior change, but there are significant limitations to the types of persuasion processes that can be invoked inside an MRI scanner. For instance, real world persuasion often involves multiple people in conversation. Functional near infrared spectroscopy (fNIRS) allows us to move out of the scanner and into more ecologically valid contexts. As a first step, the current study used fNIRS to replicate an existing fMRI persuasion paradigm (i.e. the sunscreen paradigm) to determine if mPFC shows similar predictive value with this technology. Consistent with prior fMRI work, activity in mPFC was significantly associated with message-consistent behavior change, above and beyond self-reported intentions. There was also a difference in this association between previous users and non-users of sunscreen. Activity differences based on messages characteristics were not observed. Finally, activity in a region of right dorsolateral PFC (dlPFC), which has been observed with counterarguing against persuasive messages, correlated negatively with future behavior. The current results suggest it is reasonable to use fNIRS to examine persuasion paradigms that go beyond what is possible in the MRI scanner environment.
Full-text available
Full-text available
We present a Meta-Cognitive Model (MCM) of attitudes. According to the MCM, an attitude object can be associated with both positive and negative evaluations that can be detected with modern implicit (automatic) measures of attitudes. These evaluative associations can be further associated with validity tags that are also consulted when completing deliberative attitude measures. We explain how the MCM accounts for existing findings in the attitudes literature and discuss recent studies and novel predictions derived from the MCM framework. Particular attention is devoted to factors that produce discrepancies between automatic and deliberative measures of attitudes and their consequences. Implications for attitude change, attitude strength, and the domain of prejudice are also considered.
Full-text available
This review of 122 research reports (184 independent samples, 14,900 subjects) found average r = .274 for prediction of behavioral, judgment, and physiological measures by Implicit Association Test (IAT) measures. Parallel explicit (i.e., self-report) measures, available in 156 of these samples (13,068 subjects), also predicted effectively (average r = .361), but with much greater variability of effect size. Predictive validity of self-report was impaired for socially sensitive topics, for which impression management may distort self-report responses. For 32 samples with criterion measures involving Black-White interracial behavior, predictive validity of IAT measures significantly exceeded that of self-report measures. Both IAT and self-report measures displayed incremental validity, with each measure predicting criterion variance beyond that predicted by the other. The more highly IAT and self-report measures were intercorrelated, the greater was the predictive validity of each.
Conference Paper
The present study is designed to test how immersion, presence, and narrative content (with a focus on emotional immersion) can affect one's pro-environmental attitude and behavior with specific interest in 360° videos and meat consumption as a non pro-environmental behavior. This research describes a between-group design experiment that compares two systems with different levels of immersion and two types of narratives, one with and one without emotional content. In the immersive video (IV) condition (high immersion), 21 participants used a Head-Mounted Display (HMD) to watch an emotional 360° video about meat consumption and its effects on the environment; another 21 participants experienced the tablet condition (low immersion) where they viewed the same video but with a 10.1 inch tablet; 22 participants in the control condition viewed a non emotional video about submarines with an HMD. The purpose of the experiment was to test the effect of presence and emotional impact on pro-environmental attitude and behavior. In a questionnaire, self-reported measurements were used to address presence, emotional impact and pro-environmental attitude, while an unobtrusive method evaluated pro-environmental behavior. The results showed that both immersion and emotional impact enhance self-reported presence; higher immersion and emotional impact enhanced pro-environmental attitude; narrative content and emotional immersion (i.e., personal attachment to the characters) enhanced pro-environmental behavior. Higher immersion appeared to increase pro-environmental behavior but not significantly.
Evidence is reviewed which suggests that there may be little or no direct introspective access to higher order cognitive processes. Subjects are sometimes (a) unaware of the existence of a stimulus that importantly influenced a response, (b) unaware of the existence of the response, and (c) unaware that the stimulus has affected the response. It is proposed that when people attempt to report on their cognitive processes, that is, on the processes mediating the effects of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit causal theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response. This suggests that though people may not be able to observe directly their cognitive processes, they will sometimes be able to report accurately about them. Accurate reports will occur when influential stimuli are salient and are plausible causes of the responses they produce, and will not occur when stimuli are not salient or are not plausible causes.
Psychology calls itself the science of behavior, and the American Psychological Association's current "Decade of Behavior" was intended to increase awareness and appreciation of this aspect of the science. Yet some psychological subdisciplines have never directly studied behavior, and studies on behavior are dwindling rapidly in other subdisciplines. We discuss the eclipse of behavior in personality and social psychology, in which direct observation of behavior has been increasingly supplanted by introspective self-reports, hypothetical scenarios, and questionnaire ratings. We advocate a renewed commitment to including direct observation of behavior whenever possible and in at least a healthy minority of research projects. © 2007 Association for Psychological Science.