ArticlePublisher preview available

How Selection in the Mind Is Different From Attention to the World

Journal of Experimental Psychology: General

September 2021
151(3)

DOI:10.1037/xge0001098

Authors:

Daryl Fougnie

New York University Abu Dhabi

Attentional mechanisms allow us to focus on objects that would help us achieve our goals while ignoring those that would distract us. Attention can also be focused internally toward specific items in memory. But does selection within memory work similarly to selection within perception? Perceptual attention is fast and effective at selecting regions of space. Across five experiments, we used a memory search task to investigate whether spatial selection is also efficient for selection in memory. Participants remembered four items on a grid before being asked to access their memory of one item and update one of its features. We found that it took longer to access an item when referenced by its spatial location than by its color, despite memory accuracy for location being superior. We conclude that there must be multiple, distinct memory representations in the brain and that selection in memory is different from perceptual selection. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A preview of this full-text is provided by American Psychological Association.

Learn more

Content available from Journal of Experimental Psychology: General

This content is subject to copyright. Terms and conditions apply.

How Selection in the Mind Is Different From Attention to the World

Garry Kong

1, 2

and Daryl Fougnie

Department of Psychology, New York University Abu Dhabi

Waseda Institute for Advanced Study, Waseda University

Attentional mechanisms allow us to focus on objects that would help us achieve our goals while ignor-

ing those that would distract us. Attention can also be focused internally toward speciﬁc items in mem-

ory. But does selection within memory work similarly to selection within perception? Perceptual

attention is fast and effective at selecting regions of space. Across ﬁve experiments, we used a memory

search task to investigate whether spatial selection is also efﬁcient for selection in memory. Participants

remembered four items on a grid before being asked to access their memory of one item and update one

of its features. We found that it took longer to access an item when referenced by its spatial location

than by its color, despite memory accuracy for location being superior. We conclude that there must be

multiple, distinct memory representations in the brain and that selection in memory is different from

perceptual selection.

Keywords: visual working memory, attention, visual search

Supplemental materials: https://doi.org/10.1037/xge0001098.supp

Our visual world is often cluttered and chaotic, full of objects

that need to be identiﬁed and processed. Attentional mechanisms

help us deal with this overload of information by steering us toward

speciﬁc items or areas of interest at the expense of others. But how

does our visual system ﬁnd and select objects that are relevant?

Several decades of research in visual search and attentional cuing

have revealed fundamental insights on what visual properties are ef-

ﬁcient (or less efﬁcient) for orienting attention, and these ﬁndings

have revealed basic properties of perceptual representations (Wolfe

& Horowitz, 2004,2017).

Critically, selection among competing representations occurs not

just during perception but can also occur once those representations

arestoredinourminds—for example, when information is kept online

in visual working memory. We can preferentially hold (Grifﬁn&

Nobre, 2003;Landman et al., 2003), manipulate (Garavan, 1998;

Oberauer, 2002), and/or drop items from working memory (Williams

et al., 2013;Williams & Woodman, 2012) based on our current goals.

For example, after stimulus offset, if a cue indicates that a particular

item is more task relevant, then that item will be held in a preferential

status. Recent research has catalogued the behavioral and neural

consequences of selection (for a review, see Souza & Oberauer, 2016).

However, while the consequences of selective processes have been

well studied, how such selection processes arise is not well understood.

How are our minds able to ﬁnd and select a single object representa-

tion among many? Does the ease of this selection depend on the repre-

sentational properties of the to-be-selected object? To put this another

way,howdowesearchinourmemories,andisthissimilartosearch

during perception?

While there are a number of paradigms demonstrating that we

can select information in working memory, only a paucity of

research has examined which representational properties assist or

hinder this selection process. This is unfortunate. In the same way

that the efﬁciencies of visual search have been leveraged to under-

stand perception and attention (Nakayama & Martini, 2011;Wolfe

& Horowitz, 2004,2017), the factors that afford efﬁcient memory

access can tell us about the properties of memory representations

(at least at the level they are easily accessed in memory) and how

they may be similar to or different from those in perception. Fol-

lowing this logic, we recently developed a memory search para-

digm aimed at better understanding the extent to which memory

selection and visual search are inﬂuenced by common factors

(Kong & Fougnie, 2019). We identiﬁed what we believed to be

three fundamental ﬁndings in visual search that, together, reveal

and constrain theories on perceptual representation—conjunction

effects (Treisman & Gelade, 1980), target distracter similarity

effects (Duncan & Humphreys, 1989), and search asymmetries

(Treisman & Souther, 1985). We uniformly found similar selec-

tion properties in memory that are found during visual search, sug-

gesting that there is strong overlap in the representational

properties through which selection is enacted in the two processes.

Indeed, these ﬁndings are consistent with previous research and

theories that paint the two processes as necessarily intertwined

(Cowan, 1999;Oberauer, 2002).

This article was published Online First September 9, 2021.

Garry Kong https://orcid.org/0000-0002-6832-8016

We are grateful to Geoff Woodman for his helpful comments on an earlier

version of this article. Data collected and used in the reported experiments can

be found at https://osf.io/2qauv/?view_only=9b2b698a2e9d4e468ddc5bf3cbe

b78ec.

Correspondence concerning this article should be addressed to Garry

Kong, Waseda Institute for Advanced Study, Waseda University,

Nishiwaseda Building, 103 1-21-1 Nishiwaseda, Shinjuku, Tokyo 169-

0051, Japan. Email: kong.garry@aoni.waseda.jp

542

Journal of Experimental Psychology: General

ISSN: 0096-3445 https://doi.org/10.1037/xge0001098

This document is copyrighted by the American Psychological Association or one of its allied publishers.

This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

The influence of depth on object selection and manipulation in visual working memory within a 3D context

Article

Mar 2024

Recent studies have examined whether the internal selection mechanism functions similarly for perception and visual working memory (VWM). However, the process of how we access and manipulate object representations distributed in a 3D space remains unclear. In this study, we utilized a memory search task to investigate the effect of depth on object selection and manipulation within VWM. The memory display consisted of colored items half positioned at the near depth plane and the other half at the far plane. During memory maintenance, the participants were instructed to search for a target representation and update its color. The results showed that under object-based attention (Experiments 1, 3, and 5), the update time was faster for targets at the near plane than for those at the far plane. This effect was absent in VWM when deploying spatial attention (Experiment 2) and in visual search regardless of the type of attention deployed (Experiment 4). The differential effects of depth on spatial and object-based attention in VWM suggest that spatial attention primarily relied on 2D location information irrespective of depth, whereas object-based attention seemed to prioritize memory representations at the front plane before shifting to the back. Our findings shed light on the interaction between depth perception and the selection mechanisms within VWM in a 3D context, emphasizing the importance of ordinal, rather than metric, spatial information in guiding object-based attention in VWM.

Object Features, Object Locations, and Object Files: Which Does Selective Attention Activate and When?

Article

Full-text available

Aug 2000

The representation within which attention operates was investigated in 3 experiments. The task was similar to that of R. Egly, J. Driver, and R. D. Rafal (1994). Participants had to detect the presence of a target at 1 of 4 ends of 2 shapes, differing in color and form. A precue appeared at 1 of the 4 possible corners. The 2 shapes occupied either the same or different locations in the cuing and target displays. The results showed that the cued object location was attended whether or not space was task relevant, whereas the cued object features (color and form) were attended only when these were task relevant. Moreover, when object file continuity was maintained through continuous movement, attention was found to follow the cued object file as it moved while also accruing to the cued location.

Search Asymmetry: A Diagnostic for Preattentive Processing of Separable Features

Article

Full-text available

Sep 1985

The search rate for a target among distractors may vary dramatically depending on which stimulus plays the role of target and which that of distractors. For example, the time required to find a circle distinguished by an intersecting line is independent of the number of regular circles in the display, whereas the time to find a regular circle among circles with lines increases linearly with the number of distractors. The pattern of performance suggests parallel processing when the target has a unique distinguishing feature and serial self-terminating search when the target is distinguished only by the absence of a feature that is present in all the distractors. The results are consistent with feature-integration theory (Treisman & Gelade, 1980), which predicts that a single feature should be detected by the mere presence of activity in the relevant feature map, whereas tasks that require subjects to locate multiple instances of a feature demand focused attention. Search asymmetries may therefore offer a new diagnostic to identify the primitive features of early vision. Several candidate features are examined in this article: Colors, line ends or terminators, and closure (in the sense of a partly or wholly enclosed area) appear to be functional features; connectedness, intactness (absence of an intersecting line), and acute angles do not.

Spontaneous imagery scanning in mental extrapolation

Article

Full-text available

Mar 1982

After 12 college students inspected a simple dot pattern, the pattern was removed, and Ss were then shown an arrow at an unexpected location. The task was to judge as quickly as possible whether the arrow pointed at any of the dots in the previously observed pattern. Although Ss were never instructed to form or scan mental images, most reported having done so to make their judgments. RTs were directly proportional to the distances separating the dots and the arrows, as typically found in image-scanning experiments. Imagery scanning may therefore serve a useful function when judging spatial relations between the positions of remembered objects and newly specified locations. (17 ref)

Premembering Experience: A Hierarchy of Time-Scales for Proactive Attention

Article

Full-text available

Oct 2019

Memories are about the past, but they serve the future. Memory research often emphasizes the former aspect: focusing on the functions that re-constitute (re-member) experience and elucidating the various types of memories and their interrelations, timescales, and neural bases. Here we highlight the prospective nature of memory in guiding selective attention, focusing on functions that use previous experience to anticipate the relevant events about to unfold-to "premember" experience. Memories of various types and timescales play a fundamental role in guiding perception and performance adaptively, proactively, and dynamically. Consonant with this perspective, memories are often recorded according to expected future demands. Using working memory as an example, we consider how mnemonic content is selected and represented for future use. This perspective moves away from the traditional representational account of memory toward a functional account in which forward-looking memory traces are informationally and computationally tuned for interacting with incoming sensory signals to guide adaptive behavior.

Attention induces surround suppression in visual working memory

Article

Full-text available

Jun 2019

Controversy currently exists regarding whether visual working memory (VWM) maintains sensory or non-sensory representations. Here, we tested the nature of VWM representations by leveraging a perceptual surround suppression effect when an item is attended. Participants performed a delayed-estimation task in which they memorized an array of six colors. A cue indicated which location was most likely probed. In separate experiments, we manipulated external attention (via a precue) or internal attention (via a retrocue). Both types of attention elicited a surround suppression effect, such that memory performance showed a Mexican-hat profile as a function of cue-probe offsets. Given the sensory origin of the surround suppression effect, our results thus provide compelling evidence that VWM maintenance relies on sensory mechanisms.

Visual Search Within Working Memory

Article

Full-text available

Jan 2019

Attention and working memory are 2 key pillars of cognition. Despite much research, there are important aspects about the relationship between the 2 constructs that are not well understood. Here we explore the similarity in the mechanisms that select and update working memory to those that guide attention during perception, such as in visual search. We use a memory search task where participants memorize a display of objects on a grid. During memory maintenance, participants are instructed to update the spatial positions of a subset of objects. The speed of the updating process should reflect the accessibility of the to-be-updated subset. Using this task, we explored whether landmark findings in visual search would hold true for memory search. In Experiment 1, we found a search asymmetry-it was easier to access memory representations defined by a feature than defined by the lack of a feature. In Experiment 2, we found target-distractor similarity effects-updating a single target was easier when the distractors were farther away in feature space. In Experiment 3, we found a feature versus conjunction benefit-access times were much faster for instructions to move objects defined by only 1 feature (e.g., all triangles) as opposed to a conjunction of features (e.g., all red triangles). In Experiment 4, we find a set-size effect-update times increased with the number of items in memory, particularly for conjunctive stimuli. Taken together, our results suggest a common coding and selection scheme for working memory and perceptual representations.

Access to Information in Working Memory: Exploring the Focus of Attention

Article

Full-text available

May 2002

Klaus Oberauer

Participants memorized briefly presented sets of digits, a subset of which had to be accessed as input for arithmetic tasks (the active set), whereas another subset had to be remembered independently of the concurrent task (the passive set). Latencies for arithmetic operations were a function of the setsize of active but not passive sets. Object-switch costs were observed when successive operations were applied to different digits within an active set. Participants took 2 s to encode a passive set so that it did not affect processing latencies (Experiment 2). The results support a model distinguishing 3 states of representations in working memory: the activated part of long-term memory, a capacity limited region of direct access, and a focus of attention.

Visual Memories Bypass Normalization

Article

Full-text available

Mar 2018

How distinct are visual memory representations from visual perception? Although evidence suggests that briefly remembered stimuli are represented within early visual cortices, the degree to which these memory traces resemble true visual representations remains something of a mystery. Here, we tested whether both visual memory and perception succumb to a seemingly ubiquitous neural computation: normalization. Observers were asked to remember the contrast of visual stimuli, which were pitted against each other to promote normalization either in perception or in visual memory. Our results revealed robust normalization between visual representations in perception, yet no signature of normalization occurring between working memory stores—neither between representations in memory nor between memory representations and visual inputs. These results provide unique insight into the nature of visual memory representations, illustrating that visual memory representations follow a different set of computational rules, bypassing normalization, a canonical visual computation.

The role of location in visual feature binding

Article

Jan 2019

Location appears to play a vital role in binding discretely processed visual features into coherent objects. Consequently, it has been proposed that objects are represented for cognition by their spatiotemporal location, with other visual features attached to this location index. On this theory, the visual features of an object are only connected via mutual location; direct binding cannot occur. Despite supporting evidence, some argue that direct binding does take over according to task demands and when representing familiar objects. The current study was developed to evaluate these claims, using a brief memory task to test for contingencies between features under different circumstances. Participants were shown a sequence of three items in different colours and locations, and then asked for the colour and/or location of one of them. The stimuli could either be abstract shapes, or familiar objects. Results indicated that location is necessary for binding regardless of the type of stimulus and task demands, supporting the proposed structure. A follow-up experiment assessed an alternate explanation for the apparent importance of location in binding; eye movements may automatically capture location information, making it impossible to ignore and suggesting a contingency that is not representative of cognitive processes. Participants were required to maintain fixation on half of the trials, with an eye tracker for confirmation. Results indicated that the importance of location in binding cannot be attributed to eye movements. Overall, the findings of this study support the claim that location is essential for visual feature binding, due to the structure of object representations.

Five Factors that Guide Attention in Visual Search

Article

Mar 2017
Nat. Hum. Behav.

How do we find what we are looking for? Even when the desired target is in the current field of view, we need to search because fundamental limits on visual processing make it impossible to recognize everything at once. Searching involves directing attention to objects that might be the target. This deployment of attention is not random. It is guided to the most promising items and locations by five factors discussed here: bottom-up salience, top-down feature guidance, scene structure and meaning, the previous history of search over timescales ranging from milliseconds to years, and the relative value of the targets and distractors. Modern theories of visual search need to incorporate all five factors and specify how these factors combine to shape search behaviour. An understanding of the rules of guidance can be used to improve the accuracy and efficiency of socially important search tasks, from security screening to medical image perception.