ArticlePDF Available

Correspondence problems cause repositioning costs in visual working memory

Authors:

Abstract and Figures

Visual working memory performance often declines when objects are tested in new positions from those they were observed. We report an asymmetry in repositioning costs for orientation compared to colour memory (Experiment 1). Follow-up experiments demonstrated a similar asymmetry for line length memory compared to shape memory (Experiment 2). When different shades of the same colour category were used, however, repositioning costs emerged for colour as well (Experiment 3). Finally, a direct comparison experiment demonstrated an asymmetry for orientation compared to categorical colours, but in a task with no explicit memory demands (Experiment 4). These results challenge previous accounts of repositioning costs, suggesting that they emerge not due to the contents of visual working memory, but naturally because of correspondence procedures that must be executed in order to use a memory to judge the present.
Content may be subject to copyright.
This article was downloaded by: [Florent Levillain]
On: 11 December 2013, At: 08:03
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,
UK
Visual Cognition
Publication details, including instructions for authors
and subscription information:
http:/ / www.tandfonline.com/ loi/ pvis20
Correspondence problems cause
repositioning costs in visual
working memory
Florent Levillain
a
& Jonathan I. Flombaum
a
a
Department of Psychological and Brain Sciences ,
Johns Hopkins University , Baltimore , MD , USA
Published online: 29 May 2012.
To cite this article: Florent Levillain & Jonathan I. Flombaum (2012) Correspondence
problems cause repositioning costs in visual working memory, Visual Cognition, 20:6,
669-695, DOI: 10.1080/ 13506285.2012.683050
To link to this article: http:/ / dx.doi.org/ 10.1080/ 13506285.2012.683050
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the
information (the Content) contained in the publications on our platform.
However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or
suitability for any purpose of the Content. Any opinions and views expressed
in this publication are the opinions and views of the authors, and are not the
views of or endorsed by Taylor & Francis. The accuracy of the Content should
not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions,
claims, proceedings, demands, costs, expenses, damages, and other liabilities
whatsoever or howsoever caused arising directly or indirectly in connection
with, in relation to or arising out of the use of the Content.
This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan, sub-
licensing, systematic supply, or distribution in any form to anyone is expressly
forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Correspondence problems cause repositioning
costs in visual working memory
Florent Levillain and Jonathan I. Flombaum
Department of Psychological and Brain Sciences, Johns Hopkins
University, Baltimore, MD, USA
Visual working memory performance often declines when objects are tested in new
positions from those they were observed. We report an asymmetry in repositioning
costs for orientation compared to colour memory (Experiment 1). Follow-up
experiments demonstrated a similar asymmetry for line length memory compared
to shape memory (Experiment 2). When different shades of the same colour category
were used, however, repositioning costs emerged for colour as well (Experiment 3).
Finally, a direct comparison experiment demonstrated an asymmetry for orientation
compared to categorical colours, but in a task with no explicit memory demands
(Experiment 4). These results challenge previous accounts of repositioning costs,
suggesting that they emerge not due to the contents of visual working memory, but
naturally because of correspondence procedures that must be executed in order to use
a memory to judge the present.
Keywords: Capacity limit; Change detection; Correspondence problem; Visual
working memory.
Human visual working memory is critical for a variety of daily activities and
it is known to predict myriad outcomes including intelligence and forms of
mental impairment (Cowan, 2000; Jarrold & Towse, 2006; Kane & Engle,
2002). Commensurate with its importance, a great deal of research has
explored the nature of human visual working memory. The lions share of
this work has focused on the contents of visual working memory, that is what
gets stored in memory*bound objects, unbound features, or noisy
distributions in feature space, for instance*and how much of it can get
Please address all correspondence to Florent Levillain, Department of Psychological and
Brain Sciences, Johns Hopkins University, Ames Hall, 3400 N. Charles St., Baltimore MD
21218, USA. E-mail: flevillain@mac.com
This research was supported by a Fyssen Foundation postdoctoral fellowship to FL.
VISUAL COGNITION, 2012, 20 (6), 669695
#
2012 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business
http://www.psypress.com/viscog http://dx.doi.org/10.1080/13506285.2012.683050
Downloaded by [Florent Levillain] at 08:03 11 December 2013
stored*a discrete number of items, a total amount of information, or an
infinite set of items with declining precision, for instance. Vigorous debate
surrounds these questions (Bays & Husain, 2008; Luck & Vogel, 1997;
Zhang & Luck, 2008). As a result of this intense interest in the contents of
visual working memory, another aspect of its mechanisms has been relatively
neglected. Namely, we know surprisingly little about the procedures through
which a memory is used in order to judge the world faced presently. Both in
life and in experimental tasks we do not remember items for the pure sake of
it; instead, we use visual working memory in order to eventually make
judgements about new observations. Are they the same as those seen before?
Is something missing? Has something been added? And so on. Making these
judgements requires not only that a memory with some contents is stored,
but also some mechanisms for comparing two representations, the one in
memory and the one of the world observed. These mechanisms have been
taken for granted and left unspecified. As a result, some observations about
human performance in memory tasks have likely been attributed to the
nature of the contents of visual working memory, when in fact they are due
to the procedures involved in comparing memories and observations. In the
current study we suggest that correspondence mechanisms are a crucial
procedure supporting the use of visual working memory. Correspondence
mechanisms support inferences about which objects in memory correspond
to which observed objects.
In order to demonstrate the importance of correspondence mechanisms,
we explored the repositioning of items between sample and test, a
manipulation that is known to impair human change detection performance.
In several studies, such repositioning has resulted in costs to performance,
and these costs have been taken to reflect the contents of visual working
memory and its spatial nature (Jiang & Kumar, 2004; Jiang, Olson, & Chun,
2000; Logie, Brockmole, & Jaswal, 2011). Contrary to these prevailing
theories, we suggest that repositioning costs emerge naturally because
knowing which items in memory correspond to which of the ones observed
is more difficult when items do not occupy the same locations.
REPOSITIONING ITEMS IMPAIRS WORKING MEMORY
Consider the following typical working memory experiment. An observer
needs to remember the colours of several observed squares. They all
disappear briefly, and when they reappear the observer needs to judge
whether one of them has undergone a colour change. It is well known that, as
the number of memory items increases, participants become more likely to
make mistakes. But now consider a variant of the same task: When the
squares reappear they all occupy new and previously unoccupied locations.
670
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
What impact will this have on task performance? We will return to this
specific question in Experiment 1 for both colour and orientation memory.
But first, we consider prior evidence on the role of spatial position in
working memory.
Unsurprisingly, the majority of work on the role of spatial position in
working memory has considered effects on memory for position itself, in
other words, effects on spatial working memory. Jiang and colleagues (2000)
conducted a variety of experiments exploring how relocation of nontarget
items may influence memory performance for the position of a target item.
In their Experiment 2a, for instance, participants memorized the locations of
up to 12 squares. After the squares disappeared for 900 ms, participants were
required to assess if one of the previous items, a square enclosed by an
outline box, had moved or not. This target square was presented under three
conditions: (1) in isolation, (2) with all the previously presented items
occupying their previously occupied positions, or (3) with all the previously
presented items occupying new and randomly assigned locations. By far the
worst performance obtained in this latter condition, and the best perfor-
mance obtained when all the items appeared in the same positions as before.
When the nontargets appeared in different positions than originally
encountered, the ability to detect a location change was impaired consider-
ably, even when as few as three items were memorized. Generally, Jiang and
colleagues interpreted these and related results as evidence that participants
automatically encode relative positions and configurations into working
memory, leading to impairments when these relative features are disrupted.
Thus, their interpretation focuses on the contents of visual working memory.
There is also evidence that spatial repositioning influences memory for
object features (as opposed to object locations). Treisman and Zhang (2006,
Exp. 3) asked participants to remember feature conjunctions (e.g., red
triangle), detecting instances when remembered items swapped features.
Performance declined considerably when display items appeared in new and
previously unoccupied locations. In other words, memory for feature
bindings eroded when object locations were altered between memory and
test, which was taken as evidence that feature bindings depend on spatial
information. Several studies, including this one, have also looked for effects
of spatial position on memory for individual features (as opposed to
conjunctions). But these results have been less conclusive, with several
studies failing to find effects (Treisman & Zhang, 2006; Wheeler & Treisman,
2002), though at least one recent study did find an effect (Logie et al., 2011).
A variety of other kinds of results also point to the importance of spatial
features in perception and memory. For instance, we seem to have rapid and
automatic access to the spatial layout of a visual scene (Biederman, 1981;
Metzger & Antes, 1983; Sanocki & Epstein, 1997), and spatial changes are
usually much more easily detected than changes to surface properties such as
REPOSITIONING COSTS IN VWM 671
Downloaded by [Florent Levillain] at 08:03 11 December 2013
colours (Aginsky & Tarr, 2000; Simons, 1996). In visual working memory,
changes to nonspatial properties usually have a modest or no impact on
memory for other nonspatial properties (but see Logie et al., 2011; Vidal,
Gauchou, Tallon-Baudry, & O’Regan, 2005). For instance, Jiang and
colleagues (2000, Exp. 4) asked participants to judge whether the colour of
a target item changed, and then found that turning the colour of all the
context items grey (though they were colourful at test) produced only a 7%
decrease in accuracy (compared to a condition in which the nontargets
remained unchanged), whereas a difference of almost 20% was observed
when context locations changed.
REPRESENTATIONS OF CONFIGURATION?
What might explain why changes in spatial context impair performance in
memory tasks for target locations and possibly features? Jiang, among
others, has proposed that locations and configurations are encoded in a
mandatory way into the contents of working memory, accounting for the
influence and importance of spatial position in terms of disruptions (or
preservations) of these configurations (Jiang et al., 2000; Logie et al., 2011;
Vidal et al., 2005). No specific model has detailed the exact nature of a
configuration representation*and it is worth noting that configurations
must be different for different numbers of items and, of course, depending on
their exact spatial arrangement. The same shape cannot be used to
remember three items as four, and four items can take on multiple shape
configurations, including squares, diamonds, rectangles, trapezoids, and also
irregular shapes. A complete theory would need to specify representational
formats for different configurations and mechanisms for their acquisition.
Thus, to some extent, a configurational hypothesis is fairly unspecific, at
present, and able to account for nearly any results.
Moreover, the literature has not been specific with respect to how and why
encoding configurations ends up impairing memory performance. In other
words, just because a configuration is encoded, performance need not
decline when spatial context is altered. Perhaps the operating hypothesis
includes some assumptions about how configurational encodings might
induce a conflict or compatibility effect when spatial context changes are
noticed. But the literature is not clear. Overall, several studies simply argue
(Jiang et al., 2000; Logie et al., 2011) that if changing a feature from sample
to test has an impact on performance, it should be taken as evidence that the
feature is encoded in a mandatory way into the contents of visual working
memory.
But drawing such conclusions about the contents of memory is not
warranted without first considering whether changing aspects of a test
672
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
display will have an impact on unavoidable comparison procedures that are
used when a memory is consulted to make a judgement. For example,
consider again the experiment by Jiang and colleagues (2000) in which
participants were asked to judge whether a cued item moved or not.
Performance was impaired when the item appeared in a new spatial context
than originally observed. In order to accomplish that task, a participant
needed to compare a representation of the cued item’s position with the
position of the newly observed item. But how would a participant know
which item in memory corresponded with the cued one? This correspondence
was something that the participant had to infer using all the knowledge she
stored about the memory display, and evaluating it in light of all the items
redisplayed. Surely such inferences fail more frequently when many of the
redisplayed items appear in new and random locations. In this experiment,
and generally, repositioning may not exert its impact by impairing the ability
to judge whether a particular item and a particular memory representation
share some feature value (e.g., a position), but instead by impairing the
ability to know which memory representation corresponds with a particular
observed item. To be sure, human observers very likely describe scenes in
terms of configurations and employ such representational formats to their
advantage. But the question of interest in the present study is whether
configurations are the reason for repositioning costs in change detection. We
present a series of results that cannot be explained entirely on the basis of
configurations, and which emphasize the neglected role of correspondence
procedures in visual working memory.
INFERENCES ABOUT CORRESPONDENCE
The current set of experiments was designed to test a specific hypothesis
about the causes of repositioning costs in visual working memory. We
suggest that whenever working memory is used to make a judgement about
the present, correspondences must be inferred between remembered objects
and observed ones. When items are repositioned, these inferences become
considerably more challenging since position no longer supplies a source of
evidence over which to make the inferences. Thus, when observed items are
in the same positions as their stored representations, solving correspondence
problems should be relatively easier. In contrast, when items are repositioned
from the time they were remembered, the challenge may be severe,
potentially accounting for the consequences of spatial rearrangement on
memory performance.
One prediction that follows from the theory that correspondence
challenges account for effects of spatial position is that spatial repositioning
should have a more severe effect in situations where correspondences are
REPOSITIONING COSTS IN VWM 673
Downloaded by [Florent Levillain] at 08:03 11 December 2013
especially hard to infer on the basis of properties other than position. But
when other properties readily facilitate correspondences, then spatial
position should play a less critical role. For instance, in a standard change
detection experiment with highly distinct colours, one may detect a change
simply by noting that a feature present in memory is absent at test, or vice
versa. Thus, one could infer a mismatch in featural correspondences between
memory and test without recourse to spatial position. In contrast, if items in
memory and at test are relatively similar to one another on a featural basis,
then one would not be able to merely note the presence or absence of a
feature in order to detect a change. A good spatial basis for knowing which
items in memory to compare with which observations could greatly ease this
challenge. Consider an analogy. If you need to pick up someone at the
airport, and you know only what they look like, finding them in the crowd
will depend greatly on how similar they look to individuals in the crowd. In
contrast, if you set a definite meeting place, finding your target should be
independent of her similarity in appearance to other individuals. In
Experiment 1, we explored the effects of spatial position on working
memory for two different surface features, one for which we expected easy
feature-based correspondences*and thus little or no effect of spatial
repositioning*and one for which we expected feature-based correspon-
dences to be very challenging*and thus a substantial impairment caused by
repositioning.
EXPERIMENT 1: A COST FOR REPOSITIONING
We contrasted working memory for colour and orientation. In the case of
colour, we used a handful of highly discriminable colours and expected that,
in a change detection paradigm, participants would not be affected by spatial
repositioning since colour categories should not be very confusable. In
contrast, precision for orientation is known to be quite low (especially when
participants need to remember several items; Bays & Husain, 2008), and so
we expected that items with different orientations would be relatively
confusable with one another, and, as a result, that working memory
performance should decline considerably when items were repositioned.
In every trial, participants observed three coloured boxes, or three
oriented black bars. At test, the coloured boxes or the oriented bars
reappeared, either in the same exact locations they occupied previously, or in
new, previously unoccupied locations. The participants’ task was to report
whether one of the previously seen objects had failed to reappear and had
been instead replaced by a new object (which happened in half of the trials)
or not. Figure 1 supplies a schematic depiction of ‘‘change’’ trials. Inspecting
the first two rows*the ones depicting colour changes*it becomes
674
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Figure 1. Examples of sample and test conditions in Experiment 1a: (A) Colour/same location, (B)
colour/new location, (C) orientation/same location, (D) orientation/new location. To view this figure in
colour, please see the online issue of the Journal.
REPOSITIONING COSTS IN VWM 675
Downloaded by [Florent Levillain] at 08:03 11 December 2013
immediately apparent that a yellow item has appeared in the test display,
whereas no such item was present in the memory display. This seems to be
true even when the test items occupy new locations (Figure 1B). In contrast,
inspecting the orientation panels (C and D), it seems far more difficult to
find changes when the objects occupy new locations (D). We suggest that this
difficulty arises because drawing correspondences between the items in each
of the panels is difficult. That is, deciding which item on the left to compare
with which item on the right is challenging. As far as such correspondences,
a memory task is really not so different from the comparison task the reader
just engaged in. One of the panels is simply in memory instead of on the left
(indeed, Experiment 4 will make this point directly). Therefore, we predicted
that correspondence uncertainty should cause location changes to have a
large impact on memory performance for orientation, whereas a relative lack
of such uncertainty for colour would result in no impact of location changes
in the current experiment.
Method
Participants. Twelve Johns Hopkins University undergraduates partici-
pated in this experiment, receiving either course credit or a small monetary
compensation. All participants had normal or corrected-to-normal visual
acuity. The protocol for this experiment was approved by the Johns Hopkins
University Homewood IRB.
Stimuli and procedure. Visual stimuli were displayed on a 21.5-inch iMac
running MATLAB 7.6.0 with Psychophysics toolbox (Brainard, 1997; Pelli,
1997). Participants were seated at a viewing distance of 60 cm such that the
display subtended approximately 39.56 8 25.358 of visual angle.
On each trial, two arrays were presented, separated by a blank screen
(Figure 1). The memory array was presented for 1 s, followed by a retention
interval of 900 ms and then the test array, which remained on screen until a
participant gave a response. A new trial started as soon as the response was
registered. The items were randomly scattered on a grey background, across
a 20.48 20.48 region, with the constraint that no two items were within 4.88
of each other.
The memory array consisted of either three coloured squares (0.88 0.88)
or three oriented black bars (0.18 1.78). The coloured squares were selected
at random without replacement from a set of eight different hues (blue,
yellow, green, cyan, brown, orange, purple, and red). In the same way, the
oriented bars were selected at random without replacement from a set of
eight orientations (208,408,608,808, 1008, 1208, 1408, 1608).
The test array was identical (‘‘no change’’) to the memory array on 50% of
trials, and the colour or the orientation of one item changed on the
676 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
remaining 50% (‘‘change’’). When coloured squares were displayed, the new
colour was randomly selected from the remaining hues in the colour set.
Likewise, a new oriented bar was randomly selected from the remaining
orientations in the orientation set. In 50% of trials, all items maintained their
locations between sample and test (‘‘same location trials’’). In the other 50%
of trials, all items occupied new and randomly assigned locations at test
(‘‘new location trials’’).
We instructed participants to report (by keypress) whether the test array
contained a new item. We told them explicitly that the location changes
would take place, but that these were irrelevant to the task.
Each participant completed 20 trials for each change condition (change/
no change) by location condition (same/new) by feature (colour/orientation),
totalling 160 trials.
Analysis. In all experiments we measured performance in ter ms of A?.
We favoured A? over d? since A? is considered a more accurate measure of
sensitivity when yes/no paradigms are considered (Donaldson, 1993), and
because A? can be applied when false alarm rates are equal to 0, which they
were for some of our subjects in some conditions. A? was calculated for each
participant in each condition, following the formula by Grier (1971):
A
0
¼ :5 þ H FðÞ
1 þ H FðÞ= 4
H
1 FðÞ½;
In this formula H is the hit rate and F is the false alarm rate. When the false
alarm rate was greater than the hit rate, we used the following formula
instead:
A
0
¼ :5 þ F HðÞ
1 þ F HðÞ= 4
F
1 HðÞ½:
To compare performance in same versus new location trials as a function of
feature, we conducted a two-way repeated measures ANOVA on A?
individual means, and all post hoc tests were alpha adjusted with Bonferroni
correction.
Results
Mean A? as a function of memory feature and location condition is displayed
in Figure 2. Inspection of the data suggests that location changes impaired
performance for orientation memory, but not for colour memory. Statistics
confirmed these intuitions. We found a main effect of feature type,
F(1, 11) 207.05, p B.0001, h
2
.4, and location, F(1, 11) 91.86,
pB.0001, h
2
.25.
Crucially, the interaction between the two factors was significant,
F(1, 11) 12.12, pB.01, h
2
.08. Regarding the difference between same
location and new location trials, we found a significant difference in
REPOSITIONING COSTS IN VWM 677
Downloaded by [Florent Levillain] at 08:03 11 December 2013
performance when oriented bars were remembered, post hoc test,
t(11) 6.85, p B.001, d2.52, but not when coloured squares were
remembered, post hoc test, t(11) 1.92, p.39, d0.97. Testing the items
at new locations only made an impact when participants had to remember
orientations.
Discussion
In a change detection task, repositioning items after the retention interval
frequently impairs working memory performance (Jiang et al., 2000; Logie et
al., 2011; Treisman & Zhang, 2006). Indeed, we found that in a very simple
memory task, in which only three items had to be retained, identifying
memorized orientations became more difficult when the bars reappeared at
different locations. However, when coloured boxes were tested, no impair-
ment was observed.
Prevailing theories argue that the effects of spatial relocation on memory
performance are due to the storage and use of spatial configurations. But
these theories should not predict the observed asymmetry between colour
and orientation, at least not without a post hoc accommodation (i.e., that
Figure 2. Experiment 1: Mean A? as a function of the type of feature to remember and the location
of the items in the test array. Error bars show the standard deviation.
678 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
orientation is more ‘‘configurational’’). We argue instead that the asymmetry
arises because location changes preclude the ability to make correspondence
inferences on the basis of location, and correspondence inferences are much
more challenging when made on the basis of orientation than colour. Simply
imagine knowing that a red item was present in one display, and then seeing
a red item present in another display, but in a new location; you would know
with certainty*at least in the current experiment*that these two are the
same item. Now imagine that you saw a 208 bar in a memory display and,
moreover, that your representation of that bar includes imprecision with a
standard deviation of as much as 108 (a conservative estimate; Bays &
Husain, 2008). In the test display there may be several bars that appear to be
reasonable candidates for ‘‘the same’’ bar as the one you saw before. We
argue that the very large error rate for orientation trials with location
changes reveals the impact of such correspondence challenges.
One possible alternative to this account may appeal to a role of verbal
encoding in these experiments, perhaps arguing that verbal encoding is more
available for colours than orientations. To control for this possibility, we
conducted Experiment 1b (N 9, results in Figure 3), which included a
verbal shadowing task. At the start of each trial, participants saw a random
Figure 3. Experiment 1b: Mean A? as a function of the type of feature to remember and the location
of the items in the test array. Error bars show the standard deviation.
REPOSITIONING COSTS IN VWM 679
Downloaded by [Florent Levillain] at 08:03 11 December 2013
number between 1 and 90. They then proceeded to count out loud from that
number, incrementally, for the duration of the trial until test. After the test
display, a second random number was shown, and participants reported
whether this number was larger than or greater than the cardinal value at
which their counting had ended. The results of this experiment were identical
to the results of Experiment 1. We found a main effect of both feature type,
F(1, 8) 26.12, pB.001, h
2
.3, and location, F(1, 8) 28.53, p B.001,
h
2
.22, and a marginal interaction between the two, F(1, 8) 4.97, p.06,
h
2
.04. There was a significant difference between same location and new
location when oriented bars were remembered, post hoc test, t(8) 5.34,
pB.01, d2.22, but not when coloured squares were remembered, post hoc
test, t(8) 2.19, p.31, d0.7.
EXPERIMENT 2: SHAPE AND LENGTH
We have so far presented a repositioning cost for orientation memory, but
not for colour memory. We have argued that this cost arises from the
challenges of making correspondence inferences about test items and
memory representations. Having evidenced the finding that motivates the
next three experiments, we take a moment, here, to more explicitly explain
the logic of our analysis. In any change detection trial, for a judgement to be
made, an observer must decide which memory item corresponds with which
observed item. In a standard experiment, for instance with identically shaped
but differently coloured items (e.g., Luck & Vogel, 1997), optimal logic
should lead observers to rely entirely on location to make correspondences,
since they know that colour is the feature that may change (and there are no
other features that distinguish items). But when locations change, and
especially when they change randomly*as they have here*location can no
longer supply a basis for correspondences. Now, the changing feature
dimension (in this case colour) must be used to make correspondence
inferences, and also, of course, to determine if any individual item has
changed its feature value. Change detection and correspondence inferences
become folded into one especially difficult problem. As a result, reposition-
ing costs may emerge.
Whether they emerge depends on the difficulty of making correspondence
inferences on the basis of the available feature dimension. Given noisy
representations of orientation (Anderson, Vogel, & Awh, 2011; Bays &
Husain, 2008), for example, correspondence inferences should be error
prone, leading to repositioning costs. In colour trials, in contrast, reposition-
ing costs did not emerge simply because the colours that we used in
Experiments 1*which were always easily distinguishable and selected from
a small palette of easily nameable colours*supplied a firm basis for
680
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
correspondence inferences, even in the face of location changes. A red item in
memory, in these experiments, could easily be inferred to correspond with a
red observed item, whatever locations each of them occupied. In summary,
then, repositioning leads to costs by aggravating correspondence challenges.
A perfect storm arises when repositioning takes place and items are easily
confusable with one another on the basis of the feature values they occupy.
But if they can be readily distinguished on a feature basis, performance may
withstand the effects of repositioning.
As further evidence of this perspective, in the current experiment we
explored two new object properties for which we expected an asymmetry in
repositioning costs. Specifically, in one third of trials we asked participants
to detect shape changes. By selecting stimuli from a set of easily
distinguishable shapes (squares, circles, etc.), we expected that shape would
behave like colour, revealing immunity to repositioning costs. In contrast we
expected that line length should behave like orientation, evidencing
repositioning costs, because we chose individual line lengths from within a
relatively narrow range and with differences between exemplars as small as
20 pixels. As a baseline for comparison we also included colour trials in this
experiment.
Method
Participants. Ten new participants were tested.
Stimuli, procedure, and analysis. We used similar methods to those of
Experiment 1. Participants had to remember either three coloured squares,
three shapes, or three vertical bars. The vertical bars were selected at random
without replacement from a set of eight bars with different lengths (0.698,
0.978, 1.268, 1.438, 1,728,28, 2.298, 2.588) (Figure 4a). The shapes were
selected randomly without replacement from a set of eight different shapes
(Figure 4b).
Each participant completed 20 trials for each change condition (change/
no change) by location condition (same/new) by feature (colour/shape/
length), totalling 240 trials.
Results
Mean A? as a function of memory feature and location condition is displayed
in Figure 5. Inspection of the data suggests that location changes impaired
performance for length, but not for colour or shape. Statistics confirmed
these intuitions. We found a main effect of both feature type, F(2, 9) 31.44,
pB.0001, h
2
.41, and location, F(1, 9) 8.31, p .018, h
2
.06, and a
marginal interaction between the two factors, F(1, 9) 3.16, p .06,
REPOSITIONING COSTS IN VWM 681
Downloaded by [Florent Levillain] at 08:03 11 December 2013
h
2
.04. Post hoc tests revealed a significant difference in performance
between same location and new location trials when lengths were remem-
bered, t(18) 3.78, pB.05, d1.2, but not when the other features were
remembered: Colour, t(18) 0.6, p.99, d0.29; shape, t(18) 0.82,
p.99, d 0.31.
Discussion
As in Experiment 1, colour was immune to any repositioning costs in this
experiment. As predicted, memory for shape behaved similarly. However,
memory for line length evidenced a significant cost, just as memory for
orientation did in Experiment 1. We suggest that this was caused by the
relative confusability of specific values within the dimensions of length and
orientation, contrasted with the ease of discriminating specific exemplars
from the shape and colour sets.
Figure 4. Schematic depiction of stimuli used in Experiment 2 for (a) length (vertical bars) trials and
(b) shape (geometric shapes) trials. Stimuli drawn to scale.
682 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
We add that this experiment can also be taken as additional evidence
against a configurational or relational account of repositioning costs. There
are clear differences between feature dimensions, where some evidence costs
and some do not. An account of the causes of these costs should make a
principled distinction between the features that do and do not incur costs.
We have supplied one such principled distinction*namely, is the feature
readily used as a basis for correspondence matching? It is not obvious what
kind of principled distinction would be pointed to by a configurational
account of these asymmetries.
EXPERIMENT 3: A CONFUSABLE COLOUR SPACE
We have argued so far that repositioning costs arise because of errors that
emerge when location is rendered unusable as a basis for correspondence
inferences. We have also claimed that this perspective helps to explain
observed asymmetries in repositioning costs. Costs should only arise when
objects’ feature values are relatively confusable, making correspondence
matching error prone. We have shown this to be the case, for instance, in
Figure 5. Experiment 2: Mean A? as a function of the type of feature to remember and the location
of the items in the test array. Error bars show the standard deviation.
REPOSITIONING COSTS IN VWM 683
Downloaded by [Florent Levillain] at 08:03 11 December 2013
memory for orientation and line length, but not for easily distinguishable
colours and shapes. Crucially, our theory predicts that asymmetries do not
reveal inherent properties of feature spaces and their representation, but
instead, they arise when the individual feature values used are relatively
confusable. Thus, any feature space should evidence repositioning costs if the
right feature values from within that space are assigned to sample items.
Experiment 3 sought to provide direct evidence of this prediction by
demonstrating a repositioning cost for colour memory. In each trial, all
the items were blue, but different shades of blue were selected from a set of
eight possibilities
1
(Figure 6). We expected to find repositioning costs for
colour memory in this experiment since we expected that individual shades
of blue provide a less firm basis for correspondence matching than colours
from distinctly nameable categories (as in Experiments 1 and 2). As a
baseline for comparison, we included trials with the same orientation stimuli
used in Experiment 1.
Method
Participants. Nine new participants were tested.
Stimuli, procedure, and analysis. This experiment was identical to
Experiment 1, except that in colour trials a new colour set composed of
Figure 6. Shades of blue (with RGB values) used in Experiment 3. To view this figure in colour,
please see the online issue of the Journal.
1
We thank James Brockmole for suggesting this experiment.
684 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
different shades of blue (Figure 6) replaced the previous colour set. (RGB
values for each shade of blue are included in Figure 6.)
Each participant completed 20 trials for each change condition (change/
no change) by location condition (same/new) by feature (colour/orientation),
totalling 160 trials.
Results
Mean A? as a function of feature type and location condition is displayed in
Figure 7. Inspection of the data suggests that location changes impaired
performance for orientation memory, but, contrary to Experiment 1, also for
colour memory. Statistics confirmed these intuitions. We found a main effect
of location, F(1, 8) 57.41, p B.0001, h
2
.55, whereas feature type
revealed no significant effect, F(1, 8) 0.88, p.38, h
2
.01. The interac-
tion between the two factors was significant, F(1, 8) 84.18, pB.0001,
h
2
.08. Regarding the difference between same location and new location
trials, we found a significant difference in performance both when oriented
bars were remembered, post hoc test, t(8) 18.18, pB.0001, d 5.86, and
when coloured squares were remembered, post hoc test, t(8) 5.2, pB.01,
d0.99.
Figure 7. Experiment 3: Mean A? as a function of the type of feature to remember and the location
of the items in the test array. Error bars show the standard deviation.
REPOSITIONING COSTS IN VWM 685
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Discussion
This experiment, unlike Experiments 1 and 2, evidenced a repositioning cost
for colour. This was because we chose only colours from a single category.
Explaining the asymmetry between within category and between category
colours should be relatively difficult for any theory that explains repositioning
costs on the basis of the contents of visual working memory, for instance a
theory that appeals to configurations. The particular feature values chosen
should not have an impact on what contents are encoded into memory, at least
not without a post hoc accommodation. In contrast, our suggestion that
repositioning costs emerge because of correspondence challenges predicts the
effects of Experiment 3. We argue that repositioning costs for shades of blue
emerged because individual exemplars of blue are relatively confusable with
one another, as are individual bars whose orientations may only differ by 208,
but unlike colours that are selected from distinct regions of colour space.
EXPERIMENT 4: DIRECT COMPARISON TASK
We have argued that a loss of accuracy is caused by repositioning items at test,
but only when the items are not sufficiently distinct to make direct
correspondence matches easily without a spatial anchor. At the core of the
argument is the idea that repositioning effects are not really memory effects,
but comparison or inference effects. Repositioning items at test does not alter
the contents of visual working memory. Instead, repositioning just makes it
more challenging to use representations in visual working memory in order to
make a judgement about the world faced presently. To directly test this
hypothesis, we designed a task in which spatial repositioning should have
consequences for a comparison process, but with no explicit memory
demands. Specifically, we asked participants to visually examine two
simultaneous displays, both containing either three coloured squares or three
oriented bars, with the goal of assessing whether the two displays included the
same set of items (Egeth, 1966). We predicted that the time necessary to make
such a judgement would be longer for oriented bars when they appeared in
different relative positions in the two displays (as opposed to the same relative
positions). In contrast, we predicted that for colour, relative position
differences in the two displays should not have an effect on time to judgement.
Method
Participants. Eight new participants were tested.
Stimuli, procedure, and analysis. This experiment was identical to
Experiment 1, with the exception that there was no retention interval
686
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
between the ‘‘sample’’ and ‘‘test’’ displays. Instead, the two displays were
presented side by side on the screen, and participants were instructed to
judge, as quickly as possible, whether the two arrays included the same
individual items. The displays remained on screen until the participant gave
an answer. Response latencies were recorded from the moment the displays
appeared until a keypress was made. In half the trials the two displays
included all the same items (‘‘no change’’), and in half the trials one of the
items was different across the two displays (‘‘change’’). In 50% of trials,
items’ relative locations were the same in the two displays (same location). In
the other 50% of trials, items occupied different relative locations in the two
displays (new location).
Each participant completed 20 trials for each change condition (change/
no change) by location condition (same/new) by feature (colour/orientation),
totalling 160 trials.
Results
Figure 8 displays response latencies as a function of feature type and
location condition. Inspection suggests a large effect of location changes on
orientation judgements, but not colour judgements. These intuitions were
confirmed statistically. A two-way repeated measures ANOVA on individual
reaction time means revealed a main effect of feature type, F(1, 7) 509.29,
pB.0001, h
2
.71, and location, F(1, 7) 67.36, pB.0001, h
2
.07, as well
as a significant interaction between the two factors, F(1, 7) 39.19, pB.001,
h
2
.05. A post hoc test revealed that same location trials produced
significantly faster responses than new location trials when oriented bars
were compared, post hoc test, t(7) 9.59, pB.001, d2.06, but not when
coloured squares were compared, post hoc test, t(7) 0.74, p.98,
d0.22.
Discussion
When asked to directly compare two displays containing oriented bars,
participants incurred a reliable latency cost when those bars occupied
different relative positions in the two displays. In other words, they
incurred the same kind of cost that they did for accuracy in Experiment 1,
a working memory experiment. This suggests that the costs in that
experiment were not memory costs per se. Instead they reveal that, for
orientation comparisons, deciding which of two bars to compare*whether
both are currently observed, or one is in memory*is considerably easier
when this decision can be made on the basis of a shared location, as opposed
to when the decision must be made on the basis of the orientation feature
itself.
REPOSITIONING COSTS IN VWM 687
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Experiment 4 created a task that demanded exactly the correspon-
dence and comparison procedures we argue are necessary in a working
memory change detection task, but without the same memory demands.
The experiment may well involve some memory demands*participants
may store and compare and then store and c ompare repeatedly until
they find an answer. But the demands are not the same as in a standard
working memory experiment because participants may choose how
many items to store at any given moment or for any given comparison.
But the same asymmetries emerged as i n the prior experiments.
Thus, we were able to isolate correspondence aspects of processing
and assign them responsibility for repositioning costs. Given the
procedural and stimulus similarity between this experiment and typical
memory experiments such as Experiment 1, one should like to explain
both kinds of experiments with the same theoretical model. The
challenges associated with correspondence inferences supply such a
framework.
Figure 8. Experiment 4: Mean reaction time as a function of the type of feature to remember and the
location of the items in the test array. Error bars show the standard deviation.
688 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
GENERAL DISCUSSION
The current study sought to explore the hypothesis that repositioning costs
in visual working memory tasks emerge by creating correspondence
problems for decision making*an inability to know which representations
to compare with which observed objects in order to make a judgement. We
predicted that if correspondence errors are the causes of repositioning costs,
then those costs should only emerge when exemplars within a memory
sample are relatively confusable with one another, but not when they are
readily distinguishable. Thus, we predicted asymmetries in repositioning
costs depending on the set of exemplars used to construct a memory sample.
Experiments 13 all demonstrated reliable repositioning costs with predicted
asymmetries. Specifically, we found costs for orientation memory and length
memory, but not for highly distinguishable colours and shapes. However, we
did find a cost when we rendered colour less distinguishable, by selecting
only shades of blue. Finally, to demonstrate that repositioning costs are
caused by the challenges of inferring correspondences, Experiment 4
demonstrated the same costs and asymmetries in a comparison task with
no explicit memory requirements. This collection of evidence supports the
claim that spatial repositioning costs arise because of the correspondence
inferences that one must make when comparing two displays.
Correspondence inferences and repositioning costs in visual
working memory
We have suggested that repositioning can create a sever challenge for
correspondence inferences, and that this challenge accounts for the costs
observed in this study and in related reports (Jiang & Kumar, 2004; Jiang
et al., 2000; Logie et al., 2011). Surprisingly, no other study has implicated
correspondence inferences in explaining repositioning costs. Yet correspon-
dence inferences are an inescapable step in completing any change detection
trial. When items are repositioned, one simply cannot make a change
detection judgement without making correspondence inferences. A thought
experiment makes this clear. Imagine that, after a sample display disappears,
we could print the contents of a participant’s visual working memory, and
then ask a different observer to use the printout in order to make a change
detection decision about the test display. Even if the printout included a
perfect, high-fidelity replica of each item in the sample display, the second
observer would still need to draw correspondences between items in the
printout and the items in the test display in order to compare them and
report a summary judgement. The process of assigning correspondences
could be error prone. Thus, errors could arise even with perfect memory
encoding and storage. In fact, errors may arise independent of the contents
REPOSITIONING COSTS IN VWM 689
Downloaded by [Florent Levillain] at 08:03 11 December 2013
of working memory (as demonstrated by our direct comparison experiment).
Accordingly, we should not draw any conclusions about the contents of
working memory without first accounting for errors that may be caused by
correspondence mechanisms.
How might correspondence mechanisms operate in repositioned trials?
Mathematical implementations of such mechanisms abound (Doucet, de
Freitas, Murphy, & Russell, 2000; Gustafson et al., 2002; Ma & Huang,
2009; Vul, Frank, Alvarez, & Tenenbaum, 2010), but the fundamental issues
are as follows. An item must be selected from among memory representa-
tions and compared to each new observed item until a high probability
match is found. When such a match is found, the next memory item is
selected, and that one is compared to the remaining observations. By
identifying a pairwise match for each item in memory with one individual in
a test display, change detection judgements can now be made. The critical
issue that will determine how successful such an operation is*i.e., how
accurately it assigns correspondences*will be the distinguishability of items
in a sample. When items are highly distinguishable, a high probability match
can be assumed to be correct, and the pair of items in any given match can be
excluded from further analysis. In contrast, when items are not highly
distinguishable, all pairwise considerations should be made in light of all
possible pairwise assignments. Doing this to find a mutually exclusive set of
pairwise assignments is computationally intensive and highly error prone.
Indeed, even for a set size as small as three (as in the current experiments),
three different correspondence assignments are possible for each memory
item and the set of test items.
This analysis naturally supplies an explanation for observed asymmetries
in repositioning costs. Among the features that we tested, only those where
inter-item confusability was high were affected, namely, orientation, length,
and shades of blue. As we noted, by repositioning items, location becomes
unusable as a basis for correspondence, leaving the features themselves as the
necessary substrate. For highly confusable features, correspondence mechan-
isms should fail frequently because the goodness of a match between a
memory and test item may be hard to assess, and will often require reference
to other possible matches. This is particularly true if the precision of a
representation in that dimension is inherently poor relative to the differences
between objects. For example, say a person remembers an orientation of
about 208 in a sample display. We know from other work that precision in
orientation representations can be rather low, and that it may even decline
precipitously with set size (Anderson et al., 2011; Bays & Husain, 2008). If
the standard deviation of her representation is for example, 208 (conservative
by reported estimates at set size 3), what should she think when she observes
a test item that is actually oriented at 408? Whether or not she counts the
item as a correspondence for the remembered 208 item should clearly depend
690
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
on the other items she encounters in that display. Thus, any final decisions
about whether a trial includes a change or not depends on which set of
correspondence assignments a participant makes, and this in turn depends
on the composition of the memory set, as well as on the precision of a
participant’s representations.
In contrast, if the memory and test sets are made up of highly
distinguishable individuals, then correspondence mechanisms should be
relatively successful, motivating good change detection judgements. In our
colour experiments (apart from Experiment 3), when a red item was
remembered, and a red item was observed at test, it was fair for participants
to assume that those items amounted to a correspondence match, and then
to exclude those items from consideration when evaluating matches among
other items.
Also consistent with our findings, correspondence mechanisms should
not lead to featural asymmetries when items are not repositioned between
sample and test. When items maintain their position, their locations supply a
feature-independent basis for making correspondence assignments. In a
typical colour memory task, for example, the process might go something
like this: ‘‘The object in memory at 3,5 is red and the object observed at 3,5 is
red as well. But the object in memory at 7,12 is green while the object
observed at that location is blue.’’ Crucially, for highly confusable features,
such as orientation, the same kind of procedure can apply. Imprecise
representations may lead change detection judgements to fail, but they
should not have an impact on correspondence judgements. The observer no
longer needs to ask, for instance, ‘‘which item corresponds to the one I saw
at 3,5, before?’’ She can now simply ask, ‘‘Is the orientation I remember at
3,5 the same as the one I see at 3,5 now?’’
Finally, two unique predictions emerge by analysing repositioning costs in
light of correspondence mechanisms. First, location should not be special.
Any feature that uniquely identifies individuals in a scene and remains
invariant in a trial can facilitate correspondences and should be used by
observers. For instance, if an experiment demanded change detection for
colours, but included items with unique shapes that remained invariant over
the course of a trial, participants should use shape as an additional basis for
correspondence. If shape changed from sample to test, however (i.e., if there
were ‘repositioning’ in shape space), then we would expect a cost associated
with this manipulation, since it should produce a cost to correspondence
mechanisms. Two previous reports include results which may be consistent
with this prediction. In one of several experiments, Vidal and colleagues
(2005) found a cost to colour change detection when nontarget items
changed shape from sample to test. Similarly, Logie and colleagues (2011)
contrasted colour change detection when items either maintained their
shapes, or took on new, random shapes between sample and test. They found
REPOSITIONING COSTS IN VWM 691
Downloaded by [Florent Levillain] at 08:03 11 December 2013
a cost (at shorter exposures) when shape was not maintained. However, in
contrast with our results, these studies did generally find costs to colour
performance, whereas we found none for colour. These differences may have
arisen because of a variety of other differences between our studies.
Specifically, both sets of studies included larger set sizes than we did, and
of course, correspondence problems become considerably more error prone
with increasing set siz e (because the number of possible correspondences
grows exponentially). And Vidal and colleagues used a cue procedure that
may also interact with set size and feature changes. Overall, future work
should further explore the role of nonspatial dimensions in correspondence
inferences.
The second prediction that arises from an analysis of correspondence
challenges is that repositioning costs are not actually memory costs per se.
We confirmed this prediction in Experiment 4. In our direct comparison
task, participants were asked to directly compare, in full view, images
previously used as sample and test displays, determining whether they
comprised the same individuals (Egeth, 1966). We observed the same
repositioning cost for orientation as in our prior experiments, as well as
the same asymmetry with colour, albeit through a reaction time measure.
Previous studies have observed repositioning costs in working memory tasks
and drawn conclusions about the contents of working memory, the limits on
those contents, and the features encoded into those contents in a mandatory
way. But of course, in the direct comparison task, one should not appeal to
any of these factors to explain the results. The similarity between the direct
comparison task and standard change detection begs for a single explana-
tory framework. Correspondence challenges supply such a framework,
moreover, one that begins with first principles. Correspondence decisions
are an ineluctable computational step when accomplishing either task.
Are configurations encoded?
Throughout this paper we have argued that repositioning costs arise because
of correspondence challenges, and not because of difficulties associated with
configurational representations. We want to emphasize that this is not to say
that configurations are not, or cannot be stored. Simply, that they are not the
causes of these particular kinds of effects. Indeed, considerable evidence
supports the idea that we perceive and can report configurations, that
configurations can be emphasized or disrupted by task manipulation, and,
moreover, an inchoate but quickly growing literature has identified a variety
of group-based, or summary-representations that seem to play a role in
visual cognition (Alvarez, 2011; Ariely, 2001).
692
LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
In fact, we must acknowledge that on trials where items are
not repositioned, participants may use configurational representations to
support correspondence inferences. The issue, however, is not what causes
them to perform well when no position change takes place, but what causes
them to perform poorly when a change does. If position changes render
configurations unusable as a basis for correspondence matching, then the
general point remains that correspondence failures are the proximate cause
of repositioning costs. Future research should explore the relationship
between correspondence procedures and configurational representation.
Imprecision and inference in visual working memory
A recent wave of research has applied a signal detection approach to
understanding the limits of visual working memory (Bays & Husain, 2008;
Wilken & Ma, 2004). At the core of this research is the idea that there is
intrinsic imprecision in the representation of object features, imprecision that
results from neural noise. As a consequence, comparing observed and
remembered objects amounts to probabilistic inference. This emphasis on
imprecision and inference has been very productive in accounting for a
variety of phenomena and limits associated with standard visual working
memory tasks (Bays & Husain, 2008; Bays, Catalao, & Husain, 2009). We
have attempted to apply the benefits of this emphasis to a phenomenon
previously described, but not well accounted for, costs to memory
performance associated with item repositioning. We suggest that conceiving
of visual working memory in terms of imprecision and inference may be the
crucial catalyst for understanding performance limits in a variety of contexts
under a single explanatory paradigm.
REFERENCES
Aginsky, V., & Tarr, M. J. (2000). How are different properties of a scene encoded in visual
memory? Visual Cognition, 7, 147162.
Alvarez, G. A. (2011). Representing multiple objects as an ensemble enhances visual cognition.
Trends in Cognitive Sciences, 15(3), 122131.
Anderson, D. E., Vogel, E. K., & Awh, E. (2011). Precision in visual working memory reaches a
stable plateau when individual item limits are exceeded. Journal of Neuroscience, 31,
11281138.
Ariely, D. (2001). Seeing sets: Representation by statistical properties. Psychological Science,
12(2), 157162.
Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is
set by allocation of a shared resource. Journal of Vision, 9,111.
Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in
human vision. Science, 321, 851854.
REPOSITIONING COSTS IN VWM
693
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Biederman, I. (1981). On the semantics of a glance at a scene. In M. Kubovy & J. R. Pomerantz
(Eds.), Perceptual organization (pp. 213253). Hillsdale, NJ: Lawrence Erlbaum Associates,
Inc.
Brainard, D. H. (1997). The Psychophysics Toolbox. Spatial Vision, 10, 433436.
Cowan, N. (2000). The magical number 4 in short-term memory: A reconsideration of mental
storage capacity. Behavioral and Brain Sciences, 24,87185.
Donaldson, W. (1993). Accuracy of d? and A? as estimates of sensitivity. Bulletin of the
Psychonomic Society, 31, 271274.
Doucet, A., de Freitas, N., Murphy, K., & Russell, S. (2000). Rao-Blackwellised particle filtering
for dynamic Bayesian networks. In Proceedings of Uncertainty in AI. (pp. 184191).
San Francisco, CA: Morgan Kaufmann Publishers.
Egeth, H. E. (1966). Parallel versus serial processes in multidimensional stimulus discrimina-
tion. Perception and Psychophysics, 1, 245252.
Grier, J. B. (1971). Nonparametric indexes for sensitivity and bias: Computing formulas.
Psychological Bulletin, 75, 424429.
Gustafsson, F., Gunnarsson, F., Bergman, N., Forssell, U., Jansson, J., Karlsson, R., &
Nordlund, P. (2002). Particle filters for positioning, navigation, and tracking. IEEE
Transactions on Signal Processing, 50, 200.
Jarrold, C., & Towse, J. N. (2006). Individual differences in working memory. Neuroscience, 139,
3950.
Jiang, Y., & Kumar, A. (2004). Visual short-term memory for two sequential arrays: One
integrated representation or two separate representations. Psychonomic Bulletin and Review,
11, 495500.
Jiang, Y., Olson, I. R., & Chun, M. M. (2000). Organization of visual short-term memory.
Journal of Experimental Psychology: Learning, Memory,and Cognition, 26, 683702.
Kane, M. J., & Engle, R. W. (2002). The role of prefrontal cortex in working-memory capacity,
executive attention, and general fluid intelligence: An individual-differences perspective.
Psychonomic Bulletin and Review, 9, 637671.
Logie, R. H., Brockmole, J. R., & Jaswal, S. (2011). Feature binding in visual short-term
memory is unaffected by task-irrelevant changes of location, shape, and color. Memory and
Cognition, 39,2436.
Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and
conjunctions. Nature, 390, 279281.
Ma, W. J., & Huang, W. (2009). No capacity limit in attentional tracking: Evidence for
probabilistic inference under a resource constraint. Journal of Vision, 9,130.
Metzger, R. L., & Antes, J. R. (1983). The nature of processing early in picture perception.
Psychological Research, 45, 267274.
Pelli, D. G. (1997). The VideoToolbox software for visual psychophysics: Transforming numbers
into movies. Spatial Vision, 10, 437442.
Sanocki, T., & Epstein, W. (1997). Priming spatial layout of scenes. Psychological Science, 8,
374378.
Simons, D. J. (1996). In sight, out of mind: When object representations fail. Psychological
Science, 7, 301305.
Treisman, A., & Zhang, W. (2006). Location and binding in visual working memory. Memory
and Cognition, 34, 17041719.
Vidal, J. R., Gauchou, H. L., Tallon-Baudry, C., & O’Regan, J. K. (2005). Relational
information in visual short-term memory: The structural gist. Journal of Vision, 5, 244256.
Vul, E., Frank, M. C., Alvarez, G. A., & Tenenbaum, J. B. (2010). Explaining human multiple
object tracking as resource-constrained approximate inference in a dynamic probabilistic
model. Advances in Neural Information Processing Systems, 22, 19551963.
694 LEVILLAIN AND FLOMBAUM
Downloaded by [Florent Levillain] at 08:03 11 December 2013
Wheeler, M. E., & Treisman, A. M. (2002). Binding in short-term visual memory. Journal of
Experimental Psychology: General, 131,4864.
Wilken, P., & Ma, W. J. (2004). A detection theory account of change detection. Journal of
Vision, 4, 11201135.
Zhang, W., & Luck, S. J. (2008). Discrete fixed-resolution representations in visual working
memory. Nature, 453, 233235.
Manuscript received February 2012
Revised manuscript received March 2012
First published online May 2012
REPOSITIONING COSTS IN VWM 695
Downloaded by [Florent Levillain] at 08:03 11 December 2013
... Adding another non-spatial feature to a spatial cue had little to no effectas expected if that additional feature could only be used to retrieve the item location already provided. The disruptive effect of spatial scrambling of the test array in a change detection task (Kondo & Saiki, 2012;Treisman & Zhang, 2006) can be explained as depriving observers of the use of a spatial cue to directly access and compare the features of each memory item (Levillain & Flombaum, 2012; but see also Gilchrist & Cowan, 2014), forcing them instead to use, for example, the shape of an item to indirectly access its colour via their shared location in memory. Related to this, it has been found that adding spatial structure to a memory display improves children's performance for detecting binding changes (Simmering & Wood, 2017). ...
... Arguing against a necessary role of location for memorizing visual sample arrays, Woodman, Vogel, and Luck (2012) reported that performance in a colour change detection task was unimpaired under various types of task-irrelevant location and size changes between sample and test. Notably however, this task did not require any binding of features (and see Levillain & Flombaum, 2012 for further limitations). Saiki (2016) argued that a location-independent storage mechanism exists for colour-shape bindings, in addition to a mechanism that binds features to locations. ...
Article
How does visual working memory (WM) store the binding between different features of a visual object (like colour, orientation, and location), and does memorizing these bindings require additional resources beyond memorizing individual features? These questions have traditionally been addressed by comparing performance across different types of change detection task. More recently, experimental tasks such as analogue (cued) recall, combined with analysis methods including Bayesian hypothesis testing and formal model comparison, have shed new light on the properties of WM. A significant new perspective is that noise in neural representation limits the precision of recall, and several recent models incorporate this view to account for failures of binding in WM. We review the literature on feature binding with a focus on these new developments and discuss their implications for the interpretation of classical findings.
... The goal of the current study was to identify the triggering conditions for the updating and resetting processes, to better understand the capability of VWM to cope with the dynamic environment. We argue that the updating process relies on a mapping between each VWM representation and a specific object in the world (Bae and Flombaum, 2013;Kahneman et al., 1992;Levillain and Flombaum, 2012). This one-to-one mapping allows the appropriate representation to be accessed and altered when the object changes. ...
... The theoretical notion of correspondence relates to the conceptualization of the object file theory (Kahneman et al., 1992). In this framework, findings of a behavioral benefit for consecutive presentations of stimuli sharing all feature were interpreted as stemming from a pointer-like system that connects objects in the environment to temporary representations that include their different features (see also Bae and Flombaum, 2013;Levillain and Flombaum, 2012). The central factor driving this correspondence was assumed to be location. ...
... Others have suggested that multiple cognitive processes are involved in the detection of object identity change, so that subjects may first establish the spatial correspondences between an initial memory array and a current test array, by comparing the spatial configuration of the displays, and then retrieve the object identity from the corresponding location . Thus, when one object changes its location in the test array, the spatial index of that changing object needs to be updated to form a new correspondence between the memory and the test array, a procedure which is known as resolving the "correspondence problem" (Hollingworth and Rasmussen, 2010;Levillain and Flombaum, 2012). Conversely, when the memory array and the test array share the same location, subjects respond faster and more accurately, a process known as the "spatial congruency advantage" (Boduroglu et al., 2009;Hollingworth, 2007;Hollingworth and Rasmussen, 2010). ...
... Specifically, when the perceptual load is low, subjects have sufficient VWM capacity or resources to set up spatial correspondences (resolving the "correspondence problem") between the current test array and memorial representations, and are thus able to detect subsequent object color change more accurately. However, when the VWM capacity or attentional resources are occupied by the attention-based rehearsal of the memorized representations and/or by the current perceptual load (load 4 condition), subjects performed worse in the color change detection and further worse in the accompanied location change condition, in which an additional spatial correspondence process was needed (Hollingworth and Rasmussen, 2010;Levillain and Flombaum, 2012). Thus, the observed interaction between VWM load and object location in the present study extends previous findings by showing that the spatial correspondence problem (i.e. the opposite of the spatial congruency advantage) was diminished when VWM capacity is adequate (Hollingworth, 2007;Jiang et al., 2000;Jiang and Makovski, 1992). ...
... One demonstration of VWM's vulnerability is that VWM performance can already be affected by relatively minor changes to the test context or test method (Hollingworth, 2006(Hollingworth, , 2007Jiang et al., 2000;Levillain & Flombaum, 2012;Makovski, Sussman, & Jiang, 2008;Makovski, Watson, Koutstaal, & Jiang, 2010;Vidal, Gauchou, Tallon-Baudry, & O'Regan, 2005). For instance, Makovski et al. (2010) found that using a two alternative forced choice (2AFC) method yielded a lower assessment of VWM capacity than the canonical "same/different" approach. ...
... Together, the current experiments uncover at least two factors that determine the functioning of VWM. First, the spatiotemporal context at encoding should match the context at test, which is in line with a large body of previous work (Hollingworth, 2006(Hollingworth, , 2007Jiang et al., 2000;Levillain & Flombaum, 2012;Makovski et al., 2008Makovski et al., , 2010Vidal et al., 2005). As the performance impairment that results from a mismatch could not be compensated by a cue (Experiment 2), this role of context specificity has an obligatory element to it. ...
Article
Full-text available
Visual working memory (VWM) has been found to support a very limited representation of visual information and yet relatively little is known about the mechanisms that underlie this important cognitive construct. Prior investigations have revealed that VWM performance can be affected by relatively minor changes in the test method as well as the method of encoding. In the present two experiments, we separately investigated these two factors. The results suggest that sequential object displays can improve VWM performance significantly but that a lack of context relatedness between encoding and retrieval impairs performance. This impairment seems to be caused by a mismatch in the spatiotemporal configuration of the memory and test displays, and, importantly, cannot be compensated by selective attention. These findings suggest that spatiotemporal configuration information may be a fundamental component of the information that is stored in VWM as suggested by a number of influential theories.
... The underlying principle is based on the unchanging nature of the object location index and/or attentional focus in spatially congruent conditions, thereby facilitating the detection of changes in object identity. Conversely, in spatially incongruent conditions, individuals tend to perform two processing steps: first, they establish a new spatial correspondence between the object location patterns in their mind and VWM retrieval, resulting in an object location repositioning cost; subsequently, individuals evaluate whether object identities have changed or remain the same, resulting in poorer performance than that in spatially congruent conditions (Jiang et al. 2000;Hollingworth and Rasmussen 2010;Levillain and Flombaum 2012). Yang and colleagues reported the presence of the spatial congruency advantage only for high, but not low, VWM loads. ...
Article
Full-text available
According to the spatial congruency advantage, individuals exhibit higher accuracy and shorter reaction times during the visual working memory (VWM) task when VWM test stimuli appear in spatially congruent locations, relative to spatially incongruent locations, during the encoding phase. Functional magnetic resonance imaging studies have revealed changes in right inferior frontal gyrus (rIFG) and right supra-marginal gyrus (rSMG) activity as a function of object location stability. Nevertheless, it remains unclear whether these regions play a role in active object location repositioning or passive early perception of object location stability, and demonstrations of causality are lacking. In this study, we adopted an object identity change-detection task, involving a short train of 10-Hz online repetitive transcranial magnetic stimulations (rTMS) applied at the rIFG or rSMG concurrently with the onset of VWM test stimuli. In two experimental cohorts, we observed an improved accuracy in spatially incongruent high VWM load conditions when the 10 Hz-rTMS was applied at the rIFG compared with that in TMS control conditions, whereas these modulatory effects were not observed for the rSMG. Our results suggest that the rIFG and rSMG play dissociable roles in the spatial congruency effect, whereby the rIFG is engaged in active object location repositioning, while the rSMG is engaged in passive early perception of object location stability.
... This assumption may not be true in some groups of participants. In addition, some experiments implement the partial report task by means of a test display that contains only the one item to be compared with memory, and this may lead to difficulty in knowing exactly which item from the sample array corresponds to the one item in the test display (Levillain & Flombaum, 2012). It should also be noted that location binding errors, in which the features are remembered but their locations are lost (Bays, Catalao, & Husain, 2009), will contribute to capacity estimates in the partial report task but should have little or no impact on performance in the multiple change detection task. ...
Article
The change detection task is a common method for assessing the storage capacity of working memory, but estimates of memory capacity from this task can be distorted by lapses of attention. When combined with appropriate mathematical models, some versions of the change detection task make it possible to separately estimate working memory and the probability of attentional lapses. In principle, these models should allow researchers to isolate the effects of experimental manipulations, group differences, and individual differences on working memory capacity and on the rate of attentional lapses. However, the present research found that two variants of a widely accepted model of the change detection task are not mathematically identified.
... An important aspect of VWM's online status is its ability to modify its representations following the dynamic status of each item. We argue that this updating process relies on an ongoing correspondence between each representation and a specific object in the environment (Kahneman, Treisman, & Gibbs, 1992;Levillain & Flombaum, 2012), allowing the correct representations to be accessed and modified. While the updating mechanism is flexible enough to handle a range of changes (e.g., location: Drew & Vogel, 2008;form: Blaser, Pylyshyn, & Holcombe, 2000; and even interpretation: , we suggest that certain changes cannot be integrated into existing representations. ...
Article
Full-text available
The visual working memory (VWM) resetting process is triggered when the mapping between an object in the environment and its corresponding VWM representation becomes irrelevant. Resetting involves discarding the no longer relevant representations, and encoding novel representations and mappings. We examined how resetting operates on VWM’s contents. Specifically, we tested whether losing only part of the encoded mappings led to resetting all of the VWM representations. Subjects monitored moving polygons for an abrupt shape-change. Occasionally, a polygon separated into two halves that continued to move independently, making the original single mapping irrelevant. This loss of mapping triggered a resetting process, producing a performance cost: subjects missed shape-changes when they occurred during resetting, but not when the changes occurred before or after resetting. Critically, the cost was (1) specific to the separated item, (2) larger when more mappings were lost, and (3) unaffected by the set-size. This suggests that resetting is a “local” process: VWM removes only the representations whose mappings are lost.
... In principle, however, the delayed- 108 estimation task requires selective reporting of one of multiple objects, often with multiple 109 features (e.g., Fougnie & Alvarez, 2011), which would require selecting among candidate 110 memory representations. Relatedly, Flombaum and colleageues (Levillain & Flombaum, 2012; 111 Bae & Flombaum, 2013) have shown that task-irrelevant featural overlap between objects can 112 lead to correspondence errors; if objects differ on features that are integral to those being 113 reported (e.g., objects of different hues in a context where luminance memory is tested) 114 decrements in memory precision can be eliminated. The authors argued that reducing 115 correspondence problems led to this improvement in performance, although it is not clear what 116 stage, or stages, of memory were affected by their stimulus manipulation (see also Bays, Catalao, 117 & Husain, 2009). ...
Article
Full-text available
In this article, we demonstrate limitations of accessibility of information in visual working memory (VWM). Recently, cued-recall has been used to estimate the fidelity of information in VWM, where the feature of a cued object is reproduced from memory (Bays, Catalao, & Husain, 2009; Wilken & Ma, 2004; Zhang & Luck, 2008). Response error in these tasks has been largely studied with respect to failures of encoding and maintenance; however, the retrieval operations used in these tasks remain poorly understood. By varying the number and type of object features provided as a cue in a visual delayed-estimation paradigm, we directly assess the nature of retrieval errors in delayed estimation from VWM. Our results demonstrate that providing additional object features in a single cue reliably improves recall, largely by reducing swap, or misbinding, responses. In addition, performance simulations using the binding pool model (Swan & Wyble, 2014) were able to mimic this pattern of performance across a large span of parameter combinations, demonstrating that the binding pool provides a possible mechanism underlying this pattern of results that is not merely a symptom of one particular parametrization. We conclude that accessing visual working memory is a noisy process, and can lead to errors over and above those of encoding and maintenance limitations.
... Tracking moving objects requires correspondence mechanisms (Bae & Flombaum, 2012;Vul, Frank, & Tenebaum, 2009). And even basic visual working memory tasks require operations that place objects in memory in correspondence with new observations, providing a basis for comparison (Levillain & Flombaum, 2012). Thus, the current results emphasize that a computational challenge faced at every level in the visual system involves the merging of distinct encounters to produce a coherent visual experience of objects. ...
Article
Full-text available
Object files (OFs) play an important role in theories of mid-level vision. On some influential views, OFs operate and persist only via spatiotemporal continuity. One open question concerns what occurs when direct spatiotemporal continuity is absent: Do OFs move in accordance with any motion correspondence ultimately resolved? Specifically, do OFs accord with apparent motion (AM) correspondences, which arise despite a lack of continuous spatiotemporal stimulation? In Experiment 1, subjects were presented with an AM display consisting of two circles that, across two frames, were seen as moving between two noncontiguous locations. The two objects were primed with two symbols and were then moved in a single step; a third symbol appeared, and could either match the symbol from the closer or the further object. We found a robust object specific preview benefit (OSPB) for the shorter path, in other words, the path along which AM was perceived. In order to control for the possibility that priming occurs at any nearby object, in Experiment 2, the original two objects never disappeared, but two new objects appeared in the would-be AM locations. No AM was perceived, and no OSPB obtained. In the third experiment the OSPB effect persisted even when motion along the shorter path included an unlikely featural transformation (circles turning into squares). In Experiment 4, which was nearly identical to Experiment 2, no OSPB obtained despite unique featural matches between the initial and new objects, seemingly because no AM was perceived. In Experiment 5, we failed to find an effect of featural priming, even when distance between the objects was equated. Finally, we extended the OSPB effect to two additional kinds of AM—line motion (Experiment 6) and phi motion (Experiment 7), supplying strong evidence that AM correspondences and OF correspondences are controlled by the same basic rules.
Article
In the ongoing debate about the efficacy of visual working memory for more than three items, a consensus has emerged that memory precision declines as memory load increases from one to three. Many studies have reported that memory precision seems to be worse for two items than for one. We argue that memory for two items appears less precise than that for one only because two items present observers with a correspondence challenge that does not arise when only one item is stored-the need to relate observations to their corresponding memory representations. In three experiments, we prevented correspondence errors in two-item trials by varying sample items along task-irrelevant but integral (as opposed to separable) dimensions. (Initial experiments with a classic sorting paradigm identified integral feature relationships.) In three memory experiments, our manipulation produced equally precise representations of two items and of one item.
Article
Full-text available
In signal detection theory, sensitivity can be indexed as either the distance between distributions or the area under the isosensitivity curve. The statistic d′ is usually identified as a distance measure of sensitivity, while A′ is an estimate of the area. This article considers both statistics as estimates. As such, they can be evaluated against actual distances and areas calculated under specific assumptions about the variances of the distributions. The analysis shows that d′ is a better estimate of distance than A′ is of area only under the assumption that the variances of the hypothetical distributions are equal (i.e., the slope of the z-coordinate ROC is 1.0). When the slope takes on values other than 1.0, A′ is generally a more accurate estimate of area than d′ is of distance.
Conference Paper
Full-text available
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention. Human participants show a distinctive pattern of suc- cesses and failures in tracking experiments that is often attributed to limits on an object system, a tracking module, or other specialized cognitive structures. Here we use a computational analysis of the task of object tracking to ask which human failures arise from cognitive limitations and which are consequences of inevitable perceptual uncertainty in the tracking task. We find that many human perfor- mance phenomena, measured through novel behavioral experiments, are naturally produced by the operation of our ideal observer model (a Rao-Blackwelized par- ticle filter). The tradeoff between the speed and number of objects being tracked, however, can only arise from the allocation of a flexible cognitive resource, which can be formalized as either memory or attention.
Article
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention. Human participants show a distinctive pattern of successes and failures in tracking experiments that is often attributed to limits on an object system, a tracking module, or other specialized cognitive structures. Here we use a computational analysis of the task of object tracking to ask which human failures arise from cognitive limitations and which are consequences of inevitable perceptual uncertainty in the tracking task. We find that many human performance phenomena, measured through novel behavioral experiments, are naturally produced by the operation of our ideal observer model (a Rao-Blackwelized particle filter). The tradeoff between the speed and number of objects being tracked, however, can only arise from the allocation of a flexible cognitive resource, which can be formalized as either memory or attention.
Article
Observers responded to full-color images of scenes by indicating which of two critical objects was closer in the pictorial space. These target images were preceded by prime images of the same scene sans critical objects, or by control primes or different-scene primes. Reaction times were faster following same-scene primes than following the various control and different-scene primes. Same-scene facilitation was obtained with color primes, line-drawing primes, and primes with shifted views. The effect occurred with natural scenes having gist and simple artificial scenes having little or no gist. The results indicate that prime-induced representations influence the perception of spatial layout in pictures.
Article
Models of human visual memory often presuppose an extraordinary ability to recognize and identify objects, based on evidence for nearly flawless recognition of hundreds or even thousands of pictures after a single presentation (Nickerson, 1965, Shepard, 1967, Standing, Conezio, & Haber, 1970) and for storage of tens of thousands of object representations over the course of a lifetime (Biederman, 1987) However, recent evidence suggests that observers often fail to notice dramatic changes to scenes, especially changes occurring during eye movements (e g, Grimes, 1996) The experiments presented here show that immediate memory for object identity is surprisingly poor, especially when verbal labeling is prevented However, memory for the spatial configuration of objects remains excellent even with verbal interference suggesting a fundamental difference between representations of spatial configuration and object properties
Article
Although considerable effort has been devoted to the description of processes underlying discriminations along single dimensions, there have been few attempts to determine whether or how these elementary processes are combined when discrimination requires the consideration of more than one stimulus dimension. In the present experiment, Ss were required to indicate whether two simultaneously presented multidimensional visual stimuli were identical or different. The response measure was reaction time, and Ss had a monetary incentive to respond both quickly and accurately. It was concluded that the most appropriate model for this task is one that assumes that dimensions are compared serially, and that the order in which dimensions are compared varies from trial-to-trial. Further, when a pair differs along several dimensions, Ss do not necessarily examine every dimension before initiating the response “Different.”
Article
The VideoToolbox is a free collection of two hundred C subroutines for Macintosh computers that calibrates and controls the computer-display interface to create accurately specified visual stimuli. High-level platform-independent languages like MATLAB are best for creating the numbers that describe the desired images. Low-level, computer-specific VideoToolbox routines control the hardware that transforms those numbers into a movie. Transcending the particular computer and language, we discuss the nature of the computer-display interface, and how to calibrate and control it.
Article
The visual system can only accurately represent a handful of objects at once. How do we cope with this severe capacity limitation? One possibility is to use selective attention to process only the most relevant incoming information. A complementary strategy is to represent sets of objects as a group or ensemble (e.g. represent the average size of items). Recent studies have established that the visual system computes accurate ensemble representations across a variety of feature domains and current research aims to determine how these representations are computed, why they are computed and where they are coded in the brain. Ensemble representations enhance visual cognition in many ways, making ensemble coding a crucial mechanism for coping with the limitations on visual processing.