No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Value-driven attention and associative learning models: a computational simulation analysis
Abstract
Value-driven attentional capture (VDAC) refers to a phenomenon by which stimulus features associated with greater reward value attract more attention than those associated with smaller reward value. To date, the majority of VDAC research has revealed that the relationship between reward history and attentional allocation follows associative learning rules. Accordingly, a mathematical implementation of associative learning models and multiple comparison between them can elucidate the underlying process and properties of VDAC. In this study, we implemented the Rescorla-Wagner, Mackintosh (Mac), Schumajuk-Pearce-Hall (SPH), and Esber-Haselgrove (EH) models to determine whether different models predict different outcomes when critical parameters in VDAC were adjusted. Simulation results were compared with experimental data from a series of VDAC studies by fitting two key model parameters, associative strength (V) and associability (α), using the Bayesian information criterion as a loss function. The results showed that SPH-V and EH- α outperformed other implementations of phenomena related to VDAC, such as expected value, training session, switching (or inertia), and uncertainty. Although V of models were sufficient to simulate VDAC when the expected value was the main manipulation of the experiment, α of models could predict additional aspects of VDAC, including uncertainty and resistance to extinction. In summary, associative learning models concur with the crucial aspects of behavioral data from VDAC experiments and elucidate underlying dynamics including novel predictions that need to be verified.
... Although currently, there is no formal model on how VMAC is learned and influences attentional priority, Le Pelley et al. (2016) proposed a formulation of the Mackintosh model of Pavlovian conditioning 4 in which the attention received by a cue is a direct function of its absolute associative strength. This model can, in principle, predict the acquisition of the VMAC effect (see also Jeong et al., 2023). However, recent research has found that when a distractor is associated with higher reward variability, it also receives increased attentional priority (Cho & Cho, 2021;Le Pelley et al., 2019a, b;Pearson et al., 2024). ...
Particular features of the stimuli that predict significant outcomes tend to capture our attention in a rather automatic and inflexible way. This form of attention has been described as a Pavlovian bias that mimics the phenomenon of sign-tracking described in animals, where reward-predictive cues become motivational magnets. In humans, Value-Modulated Attentional Capture (VMAC) refers to a phenomenon where distractors that signal high-value outcomes receive higher attentional priority. VMAC is particularly difficult to extinguish, showing a similar persistence often described in animal sign-tracking. In the present study, we evaluated to what extent VMAC would persist using a more specific extinction procedure than previous research, where instead of removing the possibility of obtaining rewards, the different discriminant stimuli that signal reward equate its value. Furthermore, we manipulated between experiments whether the high-value distractor predicted high-reward and high-punishment contingent to response accuracy (mimicking previous research; Experiment 1) or only high-reward (Experiment 2), and also explored the association of VMAC and its persistence with measures of emotional impulsivity employed in past research. Our results show that when both rewards and punishments are possible, VMAC does not extinguish after an extensive extinction stage, nor is it associated with measures of emotional impulsivity. When punishments were removed, we showed that VMAC gradually extinguished both in response times and accuracy and that the persistence of VMAC was significantly associated with positive urgency. We discussed these results on the potential of punishments to qualitatively alter learning and response strategies employed by participants.
... Although currently, there is no formal model on how VMAC is learned and influences attentional priority, Le Pelley et al. (2016) proposed a formulation of the Mackintosh model of Pavlovian conditioning 2 in which the attention received by a cue is a direct function of its absolute associative strength. This model can, in principle, predict the acquisition of the VMAC effect (see also Jeong et al., 2023). However, recent research has found that when a distractor is associated with higher reward variability, it also receives increased attentional priority (Le Pelley, Pearson et al., 2019;Cho & Cho, 2021;Pearson et al., 2024). ...
Particular features of the stimuli that predict significant outcomes tend to capture our attention in a rather automatic and inflexible way. This form of attention has been described as a Pavlovian bias that mimics the phenomenon of sign-tracking described in animals, where reward-predictive cues become motivational magnets. In humans, Value-Modulated Attentional Capture (VMAC) refers to a phenomenon where distractors that signal high-value outcomes receive higher attentional priority. VMAC is particularly difficult to extinguish, showing a similar persistence often described in animal sign-tracking. In the present study, we evaluated to what extent VMAC would persist using a more specific extinction procedure than previous research, where instead of removing the possibility of obtaining rewards, the different discriminant stimuli that signal reward equate its value. Furthermore, we manipulated between experiments whether the high-value distractor predicted high-reward and high-punishment contingent to response accuracy (mimicking previous research; Experiment 1) or only high-reward (Experiment 2), and also explored the association of VMAC and its persistence with measures of emotional impulsivity employed in past research. Our results show that when both rewards and punishments are possible, VMAC does not extinguish after an extensive extinction stage, nor is it associated with measures of emotional impulsivity. When punishments were removed, we showed that VMAC gradually extinguished both in response times and accuracy and that the persistence of VMAC was significantly associated with positive urgency. We discussed these results on the potential of punishments to qualitatively alter learning and response strategies employed by participants.
Visual features previously associated with reward can capture attention even when task-irrelevant, a phenomenon known as value-driven attention capture (VDAC). VDAC persists without reinforcement, unlike other forms of learning, where removing reinforcement typically leads to extinction. In five experiments, factors common to many studies were manipulated to examine their impact on VDAC and its extinction. All experiments included learning and test phases. During learning, participants completed a visual search task during which one of two target colors was associated with a reward, and the other with no reward. During test, 1 week later, participants completed another visual search task in which the reward association was not reinforced. When a rewarded feature remained task-relevant (Experiment 1), VDAC was observed. When the rewarded feature was made task-irrelevant (Experiments 2–5) there was no evidence of a VDAC effect, except when the target feature was physically salient and there was a reduction in the frequency of exposure to the reward-associated feature (Experiment 5). We failed to find evidence of VDAC in Experiments 2–4, suggesting that VDAC may depend on the demands of the task resulting in vulnerability to VDAC. When VDAC was observed, extinction was also observed. This indicates that VDAC is subject to extinction as would be expected from an effect driven by reinforcement learning.
Kim and Beck (2020b) demonstrated that value-driven attention is based on relative value rather than absolute value, suggesting that prospect theory is relevant to our understanding of value-driven attention. To further this understanding, the present study investigated the impacts of diminishing sensitivity on value-driven attention. According to diminishing sensitivity, changes in outcomes have greater impacts nearer the reference point of 0 than farther from the point. Thus, the difference between 100 looms larger than that between 1000, due to their different ratios (100/1 > 1000/901). However, according to the absolute difference hypothesis, the differences should have similar impacts due to the absolute differences being the same (100 - 1 = 1000 - 901). Experiment 1 investigated whether diminishing sensitivity operates in the modified value-driven attention paradigm while controlling the impact of absolute differences. In the training phase, 100-point and 1000-point color targets had references of 1-point and 901-point color targets, respectively. In the test phase, 100-point color distractors attracted attention more than 1000-point color distractors, supporting the diminishing sensitivity hypothesis. Experiment 2 examined the absolute difference hypothesis while controlling the impact of diminishing sensitivity. Contrary to the absolute difference hypothesis, the test phase showed that 1000-point color distractors (compared with 10-point colors for a 990 absolute difference in the training phase) failed to attract attention more than 100-point color distractors (compared with 1-point colors, for a 99 absolute difference). These results suggest that diminishing sensitivity rather than absolute difference influences value-driven attention, further supporting the relevance of prospect theory to value-driven attention.
Previously reward-associated stimuli persistently capture attention. We attempted to extinguish this attentional bias through a reversal learning procedure where the high-value color changed unexpectedly. Attentional priority shifted during training in favor of the currently high-value color, although a residual bias toward the original high-value color was still evident. Importantly, during a subsequent test phase, attention was initially more strongly biased toward the original high-value color, counter to the attentional priorities evident at the end of training. Our results show that value-based attentional biases do not quickly update with new learning and lag behind the reshaping of strategic attentional priorities by reward.
The majority of previous studies on the value modulation of attention have shown that the magnitude of value-driven attentional bias correlates with the strength of reward association. However, relatively little is known about how uncertainty affects value-based attentional bias. We investigated whether attentional capture by previously rewarded stimuli is modulated by the uncertainty of the learned value without the influence of the strength of reward association. Participants were instructed to identify the line orientation in the target color circle. Importantly, each target color was associated with a different level of uncertainty by tuning the variation in reward delivery (Experiment 1) or reward magnitude (Experiment 2). Attentional interference for uncertainty-related distractors was greater than that for certainty distractors in Experiments 1 and 2. In addition, uncertainty-induced attentional bias disappeared earlier than attentional bias for certainty. The study demonstrated that uncertainty modulates value-based attentional capture in terms of strength and persistence, even when the effect of expected value remains constant.
Stimuli in our sensory environment differ with respect to their physical salience but moreover may acquire motivational salience by association with reward. If we repeatedly observed that reward is available in the context of a particular cue but absent in the context of another cue the former typically attracts more attention than the latter. However, we also may encounter cues uncorrelated with reward. A cue with 50% reward contingency may induce an average reward expectancy but at the same time induces high reward uncertainty. In the current experiment we examined how both values, reward expectancy and uncertainty, affected overt attention. Two different colors were established as predictive cues for low reward and high reward respectively. A third color was followed by high reward on 50% of the trials and thus induced uncertainty. Colors then were introduced as distractors during search for a shape target, and we examined the relative potential of the color distractors to capture and hold the first fixation. We observed that capture frequency corresponded to reward expectancy while capture duration corresponded to uncertainty. The results may suggest that within trial reward expectancy is represented at an earlier time window than uncertainty.
We conducted a human fear conditioning experiment in which three different color cues were followed by an aversive electric shock on 0, 50, and 100% of the trials, and thus induced low (L), partial (P), and high (H) shock expectancy, respectively. The cues differed with respect to the strength of their shock association (L < P < H) and the uncertainty of their prediction (L < P > H). During conditioning we measured pupil dilation and ocular fixations to index differences in the attentional processing of the cues. After conditioning, the shock-associated colors were introduced as irrelevant distracters during visual search for a shape target while shocks were no longer administered and we analyzed the cues’ potential to capture and hold overt attention automatically. Our findings suggest that fear conditioning creates an automatic attention bias for the conditioned cues that depends on their correlation with the aversive outcome. This bias was exclusively linked to the strength of the cues’ shock association for the early attentional processing of cues in the visual periphery, but additionally was influenced by the uncertainty of the shock prediction after participants fixated on the cues. These findings are in accord with attentional learning theories that formalize how associative learning shapes automatic attention.
Recent evidence shows that distractors that signal high compared to low reward availability elicit stronger attentional capture, even when this is detrimental for task-performance. This suggests that simply correlating stimuli with reward administration, rather than their instrumental relationship with obtaining reward, produces value-driven attentional capture. However, in previous studies, reward delivery was never response independent, as only correct responses were rewarded, nor was it completely task-irrelevant, as the distractor signaled the magnitude of reward that could be earned on that trial. In two experiments, we ensured that associative reward learning was completely response independent by letting participants perform a task at fixation, while high and low rewards were automatically administered following the presentation of task-irrelevant colored stimuli in the periphery (Experiment 1) or at fixation (Experiment 2). In a following non-reward test phase, using the additional singleton paradigm, the previously reward signaling stimuli were presented as distractors to assess truly task-irrelevant value driven attentional capture. The results showed that high compared to low reward-value associated distractors impaired performance, and thus captured attention more strongly. This suggests that genuine Pavlovian conditioning of stimulus-reward contingencies is sufficient to obtain value-driven attentional capture. Furthermore, value-driven attentional capture can occur following associative reward learning of temporally and spatially task-irrelevant distractors that signal the magnitude of available reward (Experiment 1), and is independent of training spatial shifts of attention towards the reward signaling stimuli (Experiment 2). This confirms and strengthens the idea that Pavlovian reward learning underlies value driven attentional capture.
This article presents a comprehensive survey of research concerning interactions between associative learning and attention in humans. Four main findings are described. First, attention is biased toward stimuli that predict their consequences reliably (). This finding is consistent with the approach taken by Mackintosh (1975) in his attentional model of associative learning in nonhuman animals. Second, the strength of this attentional bias is modulated by the value of the outcome (). That is, predictors of high-value outcomes receive especially high levels of attention. Third, the related but opposing idea that may result in increased attention to stimuli (Pearce & Hall, 1980), receives less support. This suggests that hybrid models of associative learning, incorporating the mechanisms of both the Mackintosh and Pearce-Hall theories, may not be required to explain data from human participants. Rather, a simpler model, in which attention to stimuli is determined by how strongly they are associated with significant outcomes, goes a long way to account for the data on human attentional learning. The last main finding, and an exciting area for future research and theorizing, is that and modulate both deliberate attentional focus, and more automatic attentional capture. The automatic influence of learning on attention does not appear to fit the traditional view of attention as being either or . Rather, it suggests a new kind of "derived" attention. (PsycINFO Database Record
It is now well established that the visual attention system is shaped by reward learning. When visual features are associated with a reward outcome, they acquire high priority and can automatically capture visual attention. To date, evidence for value-driven attentional capture has been limited entirely to the visual system. In the present study, I demonstrate that previously reward-associated sounds also capture attention, interfering more strongly with the performance of a visual task. This finding suggests that value-driven attention reflects a broad principle of information processing that can be extended to other sensory modalities and that value-driven attention can bias cross-modal stimulus competition.
When stimuli are associated with reward outcome, their visual features acquire high attentional priority such that stimuli possessing those features involuntarily capture attention. Whether a particular feature is predictive of reward, however, will vary with a number of contextual factors. One such factor is spatial location: for example, red berries are likely to be found in low-lying bushes, whereas yellow bananas are likely to be found on treetops. In the present study, I explore whether the attentional priority afforded to reward-associated features is modulated by such location-based contingencies. The results demonstrate that when a stimulus feature is associated with a reward outcome in one spatial location but not another, attentional capture by that feature is selective to when it appears in the rewarded location. This finding provides insight into how reward learning effectively modulates attention in an environment with complex stimulus-reward contingencies, thereby supporting efficient foraging.
Salient stimuli and stimuli associated with reward have the ability to attract both
attention and the eyes. The current study exploited the effects of reward on the wellknown
global effect in which two objects appear simultaneously in close spatial
proximity. Participants always made saccades to a predefined target, while the colour of
a nearby distractor signalled the reward available (high/low) for that trial. Unlike
previous reward studies, in the current study these distractors never served as targets. We
show that participants made fast saccades towards the target. However, saccades landed
significantly closer to the high compared to the low reward signalling distractor. This
reward effect was already present in the first block and remained stable throughout the
experiment. Instead of landing exactly in between the two stimuli (i.e., the classic global
effect), the fastest eye movements landed closer towards the reward signalling distractor.
Results of a control experiment, in which no distractor-reward contingencies were
present, confirmed that the observed effects were driven by reward and not by physical
salience. Furthermore, there were trial-by-trial reward priming effects in which saccades
landed significantly closer to the high instead of the low reward signalling distractor
when the same distractor was presented on two consecutive trials. Together the results
imply that a reward signalling stimulus that was never part of the task set has an
automatic effect on the oculomotor system.
Attention provides the gateway to cognition, by selecting certain stimuli for further analysis. Recent research demonstrates that whether a stimulus captures attention is not determined solely by its physical properties, but is malleable, being influenced by our previous experience of rewards obtained by attending to that stimulus. Here we show that this influence of reward learning on attention extends to task-irrelevant stimuli. In a visual search task, certain stimuli signaled the magnitude of available reward, but reward delivery was not contingent on responding to those stimuli. Indeed, any attentional capture by these critical distractor stimuli led to a reduction in the reward obtained. Nevertheless, distractors signaling large reward produced greater attentional and oculomotor capture than those signaling small reward. This counterproductive capture by task-irrelevant stimuli is important because it demonstrates how external reward structures can produce patterns of behavior that conflict with task demands, and similar processes may underlie problematic behavior directed toward real-world rewards. (PsycINFO Database Record (c) 2014 APA, all rights reserved).
Previously rewarded stimuli involuntarily capture attention. The learning mechanisms underlying this value-driven attentional capture remain less understood. We tested whether theories of prediction-based associative reward learning explain the conditions under which reward feedback leads to value-based modulations of attentional priority. Across 4 experiments, we manipulated whether stimulus features served as unique predictors of reward outcomes. Participants received monetary rewards for correctly identifying a color-defined target in an initial search task (training phase) and then immediately completed a second, unrewarded visual search task in which color was irrelevant (test phase). In Experiments 1-3, monetary reward followed correct target selection during training, but critically, no target-defining features carried uniquely predictive information about reward outcomes. Under these conditions, we found no evidence of attentional capture by the previous target colors in the subsequent test phase. Conversely, when target colors in the training phase of Experiment 4 carried uniquely predictive information about reward magnitude, we observed significant attentional capture by the previously rewarded color. Our findings show that value-based attentional priority only develops for stimulus features that carry uniquely predictive information about reward, ruling out a purely motivational account and suggesting that mechanisms of reward prediction play an important role in shaping attentional priorities. (PsycINFO Database Record (c) 2014 APA, all rights reserved).
Recent studies have shown that reward history acts as a powerful attentional bias, even overcoming top-down goals. This has led to the suggestion that rewards belong to a class of attentional cues based on selection history, which are defined by past outcomes with a stimulus feature. Selection history is thought to be separate from traditional attentional cues based on physical salience and voluntary goals, but there is relatively little understanding of how selection history operates as a mechanism of attentional selection. Critically, it has yet to be understood how multiple sources of selection history interact when presented simultaneously. For example, it may be easier to find something we like if it also appears in a predictable location. We therefore pitted spatial probabilities against reward associations and found that the two sources of information had independent and additive effects. Additionally, the strength of the two sources in biasing attentional selection could be equated. In contrast, while a nonpredictive but perceptually salient cue also exhibited independent and additive effects with reward, reward associations dominated the perceptually salient cue at all levels. Our data indicate that reward associations are part of a class of particularly potent attentional cues that guide behavior through learned expectations. However, selection history should not be thought of as a unitary concept but should be understood as a collection of independent sources of information that bias attention in a similar fashion.
Classic spatial cueing experiments have demonstrated that salient cues have the ability to summon attention as evidenced by performance benefits when the cue validly indicates the target location and costs when the cue is invalid. Here we show that nonsalient cues that are associated with reward also have the ability to capture attention. We demonstrate performance costs and benefits in attentional orienting towards a nonsalient cue that acquired value through reward learning. The present study provides direct evidence that stimuli associated with reward have the ability to exogenously capture spatial attention independent of task-set, goals and salience.
Attentional theories of associative learning and categorization propose that learning about the predictiveness of a stimulus influences the amount of attention that is paid to that stimulus. Three experiments tested this idea by looking at the extent to which stimuli that had previously been experienced as predictive or nonpredictive in a categorization task were able to capture attention in a dot probe task. Consistent with certain attentional theories of learning, responses to the dot probe were faster when it appeared in a location cued by a predictive stimulus compared to a location cued by a nonpredictive stimulus. This result was obtained only with short (250-ms or 350-ms) but not long (1,000-ms) delays between onset of the stimuli and the dot probe, suggesting that the observed spatial cuing effect reflects the operation of a relatively rapid, automatic process. These findings are consistent with the approach to the relationship between attention and learning taken by the class of models exemplified by Mackintosh's (1975) theory. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
In psychology research studies, the goals of the experimenter and the goals of the participants often do not align. Researchers are interested in having participants who take the experimental task seriously, whereas participants are interested in earning their incentive (e.g., money or course credit) as quickly as possible. Creating experimental methods that are pleasant for participants and that reward them for effortful and accurate data generation, while not compromising the scientific integrity of the experiment, would benefit both experimenters and participants alike. Here, we explored a gamelike system of points and sound effects that rewarded participants for fast and accurate responses. We measured participant engagement at both cognitive and perceptual levels and found that the point system (which invoked subtle, anonymous social competition between participants) led to positive intrinsic motivation, while the sound effects (which were pleasant and arousing) led to attentional capture for rewarded colors. In a visual search task, points were awarded after each trial for fast and accurate responses, accompanied by short, pleasant sound effects. We adapted a paradigm from Anderson, Laurent, and Yantis (Proceedings of the National Academy of Sciences 108(25):10367-10371, 2011b), in which participants completed a training phase during which red and green targets were probabilistically associated with reward (a point bonus multiplier). During a test phase, no points or sounds were delivered, color was irrelevant to the task, and previously rewarded targets were sometimes presented as distractors. Significantly longer response times on trials in which previously rewarded colors were present demonstrated attentional capture, and positive responses to a five-question intrinsic-motivation scale demonstrated participant engagement.
Attention selects stimuli for perceptual and cognitive processing according to an adaptive selection schedule. It has long been known that attention selects stimuli that are task relevant or perceptually salient. Recent evidence has shown that stimuli previously associated with reward persistently capture attention involuntarily, even when they are no longer associated with reward. Here we examine whether the capture of attention by previously reward-associated stimuli is modulated by the processing of current but unrelated rewards. Participants learned to associate two color stimuli with different amounts of reward during a training phase. In a subsequent test phase, these previously rewarded color stimuli were occasionally presented as to-be-ignored distractors while participants performed visual search for each of two differentially rewarded shape-defined targets. The results reveal that attentional capture by formerly rewarded distractors was the largest when both recently received and currently expected reward were the highest in the test phase, even though such rewards were unrelated to the color distractors. Our findings support a model in which value-driven attentional biases acquired through reward learning are maintained via the cognitive mechanisms involved in predicting future rewards.
Stimuli that have previously been associated with the delivery of reward involuntarily capture attention when presented as unrewarded and task-irrelevant distractors in a subsequent visual search task. It is unknown how long such effects of reward learning on attention persist. One possibility is that value-driven attentional biases are plastic and constantly evolve to reflect only recent reward history. According to such a mechanism of attentional control, only consistently reinforced patterns of attention allocation persist for extended periods of time. Another possibility is that reward learning creates enduring changes in attentional priority that can persist indefinitely without further learning. Here we provide evidence for an enduring effect of reward learning on attentional priority: stimuli previously associated with reward in a training phase capture attention when presented as irrelevant distractors over half a year later, without the need for further reward learning. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
Review of the literature indicates that, according to theories of selective attention, learning about a stimulus depends on attending to that stimulus; this is represented in 2-stage models by saying that Ss switch in analyzers as well as learning stimulus-response associations. It is argued that this assumption, however, is equally well represented in a formal model by the incorporation of a stimulus-specific learning-rate parameter, a, into the equations describing changes in the associative strength of stimuli. Previous theories of selective attention have also assumed that (a) Ss learn to attend to and ignore relevant and irrelevant stimuli (i.e., that a may increase or decrease depending on the correlation of a stimulus with reinforcement); and (b) there is an inverse relationship between the probabilities of attending to different stimuli (i.e., that an increase in a to one stimulus is accompanied by a decrease in a to others). The first assumption has been used to explain the phenomena of acquired distinctiveness and dimensional transfer, the second to explain those of overshadowing and blocking. It is argued that although the first assumption is justified by the data, the second is not: Overshadowing and blocking are better explained by the choice of an appropriate rule for changing a, such that a decreases to stimuli that signal no change from the probability of reinforcement predicted by other stimuli. (65 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
explicate the evolution of a theory of Pavlovian conditioning [SOP] / concentrate . . . on how it attempts to capture, in abstract formulation, what might be entertained to be the basic episodic . . . processes responsible for the acquisition and performance of conditioned responding
the working version of the theory that is the subject of our current research is referred to as AESOP / it portends an elaboration and extension of SOP, but has not yet been rendered in comparable quantitative terms / indicate the manner of challenge to SOP that leads us to think that some extension is necessary / show how AESOP is designed to increase the scope and empirical adequacy of the theory (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Covert shifts of attention precede and direct overt eye movements to stimuli that are task relevant or physically salient. A growing body of evidence suggests that the learned value of perceptual stimuli strongly influences their attentional priority. For example, previously rewarded but otherwise irrelevant and inconspicuous stimuli capture covert attention involuntarily. It is unknown, however, whether stimuli also draw eye movements involuntarily as a consequence of their reward history. Here, we show that previously rewarded but currently task-irrelevant stimuli capture both attention and the eyes. Value-driven oculomotor capture was observed during unconstrained viewing, when neither eye movements nor fixations were required, and was strongly related to individual differences in visual working memory capacity. The appearance of a reward-associated stimulus came to evoke pupil dilation over the course of training, which provides physiological evidence that the stimuli that elicit value-driven capture come to serve as reward-predictive cues. These findings reveal a close coupling of value-driven attentional capture and eye movements that has broad implications for theories of attention and reward learning.
Visual attention is captured by physically salient stimuli (termed salience-based attentional capture), and by otherwise task-irrelevant stimuli that contain goal-related features (termed contingent attentional capture). Recently, we reported that physically nonsalient stimuli associated with value through reward learning also capture attention involuntarily (Anderson, Laurent, & Yantis, PNAS, 2011). Although it is known that physical salience and goal-relatedness both influence attentional priority, it is unknown whether or how attentional capture by a salient stimulus is modulated by its associated value. Here we show that a physically salient, task-irrelevant distractor previously associated with a large reward slows visual search more than an equally salient distractor previously associated with a smaller reward. This magnification of salience-based attentional capture by learned value extinguishes over several hundred trials. These findings reveal a broad influence of learned value on involuntary attentional capture.
Theories of selective attention in associative learning posit that the salience of a cue will be high if the cue is the best available predictor of reinforcement (high predictiveness). In contrast, a different class of attentional theory stipulates that the salience of a cue will be high if the cue is an inaccurate predictor of reinforcement (high uncertainty). Evidence in support of these seemingly contradictory propositions has led to: (i) the development of hybrid attentional models that assume the coexistence of separate, predictiveness-driven and uncertainty-driven mechanisms of changes in cue salience; and (ii) a surge of interest in identifying the neural circuits underpinning these mechanisms. Here, we put forward a formal attentional model of learning that reconciles the roles of predictiveness and uncertainty in salience modification. The issues discussed are relevant to psychologists, behavioural neuroscientists and neuroeconomists investigating the roles of predictiveness and uncertainty in behaviour.
Valuable stimuli receive attentional priority. However, it is unknown whether the mechanism of the attentional priority is based on relative (e.g., higher) or absolute (e.g., 45 points) values. Therefore, we manipulated the relative and absolute values independently in a modified value-driven attentional capture paradigm. In the training phase, where associative learning occurs between color and reward value, two test target colors were each presented with another different target color (reference target colors) in separate context blocks. Therefore, each test target color had different reference points. In the test phase, the two test target colors were used as singleton distractor colors. In the training phase of Experiment 1, the absolute reward value of the test target colors was the same, but one had a higher value than its reference target color and the other had a lower value. In the test phase, the high relative value color distractor captured attention more, suggesting that the relative value of stimuli influenced selective attention. In Experiment 2 the relative value of the test target colors was the same, but the absolute value was higher for one. The high and low absolute value color distractors captured attention equally in the test phase, indicating little impact of the absolute value on selective attention. These findings suggest that the relative value, rather than absolute value, plays a critical role in the allocation of attention. Accordingly, the present study suggests that prospect theory (Kahneman & Tversky, Econometrica, 47 (2), 363-391, 1979) can be extended to earlier cognitive stages such as selective attention.
Attention is shaped by our prior experiences with stimuli, and in particular by learning about their relationship with motivationally significant events: rewards and punishments. While it is typically adaptive to prioritize detection of signals of reward and punishment, recent evidence suggests that attentional prioritization of motivationally relevant information can be involuntary and inflexible, which can be counterproductive when circumstances change, and these signals are no longer the focus of a person’s goals. We review this literature, which suggests that attentional capture is promoted by learning about both rewards and punishments, though further research is required to probe for differences in the temporal dynamics of these processes. We also highlight the clinical relevance of interactions between appetitive and aversive motivation and perceptual-cognitive processes.
Past research in animals has suggested that attention is distributed to exploit known relationships between stimuli (e.g., Mackintosh, 1975) and explore stimuli whose consequences are uncertain (e.g., Pearce & Hall, 1980). The resulting changes in attention influence how animals learn new information involving those stimuli. While there is strong support for exploitative attention and its effects on learning in humans, the evidence for exploratory attention is less well developed. Two experiments examined whether preferential allocation of attention (as measured by eye-gaze) to cues associated with uncertainty leads to more rapid learning of new associations involving these cues in the future. In each experiment, participants first learned about compounds containing one predictive cue and one non-predictive cue. The level of uncertainty during this first stage of training was also manipulated: cue-outcome relationships were either deterministic (certain) or probabilistic (uncertain). In a second stage, new cue-outcome relationships were trained and the uncertainty of these relationships could be resolved by learning about the previously non-predictive cues. As a result of the manipulation of uncertainty in the first stage, some participants experienced a sudden onset of uncertainty at the start of this second stage, while others experienced a stable level of uncertainty throughout the experiment. Experiment 1 showed that participants who experienced an onset of uncertainty learned novel cue-outcome associations faster than participants for whom uncertainty was constant. Furthermore, participants experiencing unexpected uncertainty showed a greater increase in attention to cues in Stage 2. When the first stage of training was extended in Experiment 2 a larger difference in the rate of learning between the two conditions was observed in the second stage. We argue that this represents evidence for an effect of exploratory attention on rate of learning in humans.
A large body of research has shown that learning about relationships between neutral stimuli and events of significance-rewards or punishments-influences the extent to which people attend to those stimuli in future. However, different accounts of this influence differ in terms of the critical variable that is proposed to determine learned changes in attention. We describe two experiments using eye-tracking with a rewarded visual search procedure to investigate whether attentional capture is influenced by the predictiveness of stimuli (i.e., the extent to which they provide information about upcoming events) or by their absolute associative value (that is, the expected incentive value of the outcome that a stimulus predicts). Results demonstrated a clear influence of associative value on the likelihood that stimuli will capture eye-movements, but the evidence for a distinct influence of predictiveness was less compelling. The results of these experiments can be reconciled within a simple account under which attentional prioritization is a monotonic function of the expected, subjective value of the reward that is signalled by a stimulus.
Findings from an increasingly large number of studies have been used to argue that attentional capture can be dependent on the learned value of a stimulus, or value-driven. However, under certain circumstances attention can be biased to select stimuli that previously served as targets, independent of reward history. Value-driven attentional capture, as studied using the training phase-test phase design introduced by Anderson and colleagues, is widely presumed to reflect the combined influence of learned value and selection history. However, the degree to which attentional capture is at all dependent on value learning in this paradigm has recently been questioned. Support for value-dependence can be provided through one of two means: (1) greater attentional capture by prior targets following rewarded training than following unrewarded training, and (2) greater attentional capture by prior targets previously associated with high compared to low value. Using a variant of the original value-driven attentional capture paradigm, Sha and Jiang (Attention, Perception, and Psychophysics, 78, 403–414, 2016) failed to find evidence of either, and raised criticisms regarding the adequacy of evidence provided by prior studies using this particular paradigm. To address this disparity, here we provided a stringent test of the value-dependence hypothesis using the traditional value-driven attentional capture paradigm. With a sufficiently large sample size, value-dependence was observed based on both criteria, with no evidence of attentional capture without rewards during training. Our findings support the validity of the traditional value-driven attentional capture paradigm in measuring what its name purports to measure.
Stimuli associated with monetary reward can become powerful cues that effectively capture visual attention. We examined whether such value-driven attentional capture can be induced with monetary feedback in the absence of an expected cash payout. To this end, we implemented images of U.S. dollar bills as reward feedback. Participants knew in advance that they would not receive any money based on their performance. Our reward stimuli—20 bill images—were thus dissociated from any practical utility. Strikingly, we observed a reliable attentional capture effect for the mere images of bills. Moreover, this finding generalized to Monopoly money. In two control experiments, we found no evidence in favor of nominal or symbolic monetary value. Hence, we claim that bill images are special monetary representations, such that there are strong associations between the defining visual features of bills and reward, probably due to a lifelong learning history. Together, we show that the motivation to earn cash plays a minor role when it comes to monetary rewards, while bill-defining visual features seem to be sufficient. These findings have the potential to influence human factor applications, such as gamification, and can be extended to novel value systems, such as the electronic cash Bitcoin being developed for use in mobile banking. Finally, our procedure represents a proof of concept on how images of money can be used to conserve expenditures in the experimental context.
Most modern theories of associative learning emphasize a critical role for prediction error (PE, the difference between received and expected events). One class of theories, exemplified by the Rescorla-Wagner (1972) model, asserts that PE determines the effectiveness of the reinforcer or unconditioned stimulus (US): surprising reinforcers are more effective than expected ones. A second class, represented by the Pearce-Hall (1980) model, argues that PE determines the associability of conditioned stimuli (CSs), the rate at which they may enter into new learning: the surprising delivery or omission of a reinforcer enhances subsequent processing of the CSs that were present when PE was induced. In this mini-review we describe evidence, mostly from our laboratory, for PE-induced changes in the associability of both CSs and USs, and the brain systems involved in the coding, storage and retrieval of these altered associability values. This evidence favors a number of modifications to behavioral models of how PE influences event processing, and suggests the involvement of widespread brain systems in animals' responses to PE.
There is growing consensus that reward plays an important role in the control of attention. Until recently, reward was thought to influence attention indirectly by modulating task-specific motivation and its effects on voluntary control over selection. Such an account was consistent with the goal-directed (endogenous) versus stimulus-driven (exogenous) framework that had long dominated the field of attention research. Now, a different perspective is emerging. Demonstrations that previously reward-associated stimuli can automatically capture attention even when physically inconspicuous and task-irrelevant challenge previously held assumptions about attentional control. The idea that attentional selection can be value driven, reflecting a distinct and previously unrecognized control mechanism, has gained traction. Since these early demonstrations, the influence of reward learning on attention has rapidly become an area of intense investigation, sparking many new insights. The result is an emerging picture of how the reward system of the brain automatically biases information processing. Here, I review the progress that has been made in this area, synthesizing a wealth of recent evidence to provide an integrated, up-to-date account of value-driven attention and some of its broader implications.
Rewards affect the deployment of visual attention in various situations. Evidence suggests that the stimulus associated with reward involuntary captures attention (value-driven attentional capture; VDAC). Recent studies report VDAC even when the reward-associated feature does not define the target (i.e., task-irrelevant). However, these studies did not conduct the test phase without reward, thus the effect may be qualitatively different from those in the previous studies. In the current study, we tested if task-irrelevant features induce VDAC even in the test phase with no reward. We used a flanker task during reward learning to create color-reward associations (training phase), and then tested the effect of color during visual search (test phase). Reward learning with no spatial uncertainty in the flanker task induced VDAC, even when reward signaling color was associated with both target and distractor (Experiments 1 and 2). In Experiment 3, a significant VDAC with a color for all letters indicated that target-distractor discrimination is not necessary for VDAC. Finally, a significant VDAC (Experiment 4) with color rectangular frames around the letters indicated binding reward-associated features to task-relevant letters is not necessary for VDAC. All these effects were obtained in the test phase without reward, thus VDAC in the current study is comparable to previous studies using target-defining features. These findings indicate that task-relevance is not a necessary condition for VDAC from reward-associated features, suggesting that reward-associated learning in VDAC is more indirect.
Moore and Stickney (1980) described a real-time computational version of Mackintosh’s (1975) attentional model of associative learning. By assuming that hippocampal lesions affect computations that control the rate of learning, they were able to simulate impairments of latent inhibition and blocking, as reported in studies of classical conditioning. Schmajuk (1984a) proposed that hippocampal lesions affect computations of stimulus associabilities, as defined in Pearce and Hall’s (1980) model of learning. A revised version of the Moore-Stickney model and a real-time version of Pearce and Hall’s (1980) model, both incorporating the proposed modifications for the effect of hippocampal lesions, were applied to different classical conditioning paradigms. Simulation experiments with both models were carried out for the following protocols: acquisition under simultaneous, delay, and trace conditioning; partial reinforcement; noncontingent training; conditioned inhibition; differential conditioning; extinction; latent inhibition; blocking; overshadowing; and discrimination reversal. Although some discrepancies between simulation experiments and relevant literature were noted, both models proved capable of simulating most hippocampal lesion effects.
Translated by Gleb Vassilievitch von Anrep
Eye movements provide a direct link to study the allocation of overt attention to stimuli in the visual field. The initiation of saccades towards visual stimuli is known to be influenced by the bottom-up salience of stimuli as well as the motivational context of the task. Here, we asked whether the initiation of saccades is also influenced by the intrinsic motivational salience of a stimulus. Face stimuli were first associated with positive or negative motivational salience through instrumental learning. The same faces served as target stimuli in a subsequent saccade task, in which their motivational salience was no longer task-relevant. Participants performed either voluntary saccades, which required the selection of the saccade target out of two simultaneously presented stimuli (experiment 1), or reactive saccades, where only the target stimulus was presented (experiment 2). We found a specific effect of learned positive stimulus value on the latencies of voluntary saccades: For faces with high versus low positive motivational salience, saccadic latencies were significantly reduced. No such difference was observed for previously punished faces. In contrast, reactive saccades to both previously rewarded and punished faces were unaffected by learned stimulus value. Our findings show for the first time that saccadic preparation is susceptible to the acquired intrinsic motivational salience of visual stimuli. Based on the observation that only voluntary saccades but not reactive saccades were modulated, we conclude that the recruitment of neural processes for target identification is required to allow for an influence of motivational stimulus salience on saccadic preparation.
Proposes a formal conditioning model that adds a dynamic attention rule and a novel response mapping rule to the R. A. Rescorla–A. R. Wagner (1972) associative axiom. This model retains the virtues of its predecessor and, in addition, accurately simulates many conditioning phenomena that are not encompassed by the original Rescorla–Wagner model: (a) the acquisition function is –S-shaped, and abrupt shifts in responsiveness occur during acquisition and extinction; (b) reacquisition is characteristically more rapid than the initial acquisition; (c) the behavioral effect of a set of conditioning operations is not necessarily independent of the S's prior conditioning history. In addition, the new model seems to cope more efficiently with latent inhibition and with the ineffectiveness of nonreinforcing a conditioned inhibitor. (61 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Argues that J. Cohen's (see record
1981-04405-001) critique of the present authors' work (1979) is unfounded, and that his Baconian formalism has little normative and descriptive appeal. (2 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
The function of the hippocampus in conditioning is portrayed in terms of an extension of N. J. Mackintosh's (see record
1975-26802-001) attention theory, which describes the evolution of the salience (associability) of each stimulus in the situation, including the context, and its predictive associative relationship to itself and all other stimuli. In terms of the model, the hippocampus is essential for computations that reduce salience when a stimulus is presented in the context of other stimuli that are better predictors of events. The model is applied to the phenomena of latent inhibition and blocking. (15 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
A formal account of the relationship between attention and associative learning is presented within the framework of a configural theory of discrimination learning. The account is based on a connectionist network in which the entire pattern of stimulation presented on a trial activates a configural unit that then enters into an association with the trial outcome. Attention is assumed to have two roles within this network. First, the salience of the stimuli at the input to the network can be increased if they are relevant to the occurrence of reinforcement and decreased if they are irrelevant. Second, the associability of configural units can increase on trials when the outcome is surprising and decrease when the outcome is not surprising.
It is well known that salient yet task irrelevant stimuli may capture our eyes independent of our goals and intentions. The present study shows that a task-irrelevant stimulus that is previously associated with high monetary reward captures the eyes much stronger than that very same stimulus when previously associated with low monetary reward. We conclude that reward changes the salience of a stimulus such that a stimulus that is associated with high reward becomes more pertinent and therefore captures the eyes above and beyond its physical salience. Because the stimulus capture the eyes and disrupts goal-directed behavior we argue that this effect is automatic not driven by strategic, top-down control.