No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Search Superiority: Goal-Directed Attentional Allocation Creates More Reliable Incidental Identity and Location Memory Than Explicit Encoding in Naturalistic Virtual Environments
Abstract
We use representations and expectations formed during life-long learning to support attentional allocation and perception. In comparison to traditional laboratory investigations, real-world memory formation is usually achieved without explicit instruction and on-the-fly as a by-product of natural interactions with our environment. Understanding this process and the quality of naturally formed representations is critical to understanding how memory is used to guide attention and perception. Utilizing immersive, navigable, and realistic virtual environments, we investigated incidentally generated memory representations by comparing them to memories for items which were explicitly memorized. Participants either searched for objects embedded in realistic indoor environments or explicitly memorized them for follow-up identity and location memory tests. We show for the first time that memory for the identity of naturalistic objects and their location in 3D space is higher after incidental encoding compared to explicit memorization, even though the subsequent memory tests came as a surprise to participants. Relating gaze behavior to memory performance revealed that encoding time was more predictive of subsequent memory when participants explicitly memorized an item, compared to incidentally encoding it. Our results suggest that the active nature of guiding attentional allocation during proactive behavior allows for behaviorally optimal formation and utilization of representations. This highlights the importance of investigating cognition under ecologically valid conditions and shows that understanding the most natural processes for encoding and maintaining information is critical for understanding adaptive behavior.
... We hypothesized that using a flashlight in dark scenes would lead to decreased accuracy rates and longer search times compared to illuminated scenes. With regards to object memory, we hypothesized that both identity and location memory performance would be better for targets than for distractors, replicating previous findings 6,11,14 . However, the influence of increased search difficulty induced by the flashlight on incidental memory is unclear. ...
... These findings suggest potential differences in cognitive processing in VR compared to the computer screen experiment, which may be due to the immersive nature of VR, resulting in an interaction of actions and incidental memory formations. Moreover, we replicated previous findings 6,11,12,14 showing that memory was increased for objects which were previously searched compared to objects which were simply present in the scenes. Surprisingly, we found that removing visual information did not necessarily compromise location memory. ...
... In both experiments, we used the same set of in-house developed indoor scenes also used in previous studies 11,18,19,21,29,52 . Here, we selected six scenes, two each from three different room categories: bathroom, kitchen, and living room (see Fig. 1a). ...
A popular technique to modulate visual input during search is to use gaze-contingent windows. However, these are often rather discomforting, providing the impression of visual impairment. To counteract this, we asked participants in this study to search through illuminated as well as dark three-dimensional scenes using a more naturalistic flashlight with which they could illuminate the rooms. In a surprise incidental memory task, we tested the identities and locations of objects encountered during search. Importantly, we tested this study design in both immersive virtual reality (VR; Experiment 1) and on a desktop-computer screen (Experiment 2). As hypothesized, searching with a flashlight increased search difficulty and memory usage during search. We found a memory benefit for identities of distractors in the flashlight condition in VR but not in the computer screen experiment. Surprisingly, location memory was comparable across search conditions despite the enormous difference in visual input. Subtle differences across experiments only appeared in VR after accounting for previous recognition performance, hinting at a benefit of flashlight search in VR. Our findings highlight that removing visual information does not necessarily impair location memory, and that screen experiments using virtual environments can elicit the same major effects as VR setups.
... of that object. For instance, Draschkow et al. ( 18 see also 19 ) had participants search for an object or memorize an object in meaningless arrays of pictures or in real-world scenes. Their results indicated that memory for target objects was better when participants had searched for this object in a scene before than when they were instructed to explicitly memorize it. ...
... 18,[37][38][39][40] or virtual reality (e.g. 19,[41][42][43]. Traditional search paradigms and well-established results seem to successfully translate into (immersive) virtual reality (VR) environments, such that there seem to be no additional side effects that might undermine the overall outcome 43,44 . ...
... Furthermore, memory for present and absent targets was tested in a surprise memory test after the end of the search tasks. Based on previous findings from the laboratory, we expected that search should be faster and less accurate in target-present than in target-absent searches 11,12,56 and present targets to be remembered better than absent targets 18,19 . Furthermore, following the findings from laboratory settings (e.g. 25 ), we expected that an executive working memory load should lead to a worse search and memory performance compared to a control condition without any load. ...
We investigated whether and how different types of search distractions affect visual search behavior and target memory while participants searched in a real-world environment. They searched either undistracted (control condition), listened to a podcast (auditory distraction), counted down aloud at intervals of three while searching (executive working memory load), or were forced to stop the search on half of the trials (time pressure). In line with findings from laboratory settings, participants searched longer but made fewer errors when the target was absent than when it was present, regardless of distraction condition. Furthermore, compared to the auditory distraction condition, the executive working memory load led to higher error rates (but not longer search times). In a surprise memory test after the end of the search tasks, recognition was better for previously present targets than for absent targets. Again, this was regardless of the previous distraction condition, although significantly fewer targets were remembered by the participants in the executive working memory load condition than by those in the control condition. The findings suggest that executive working memory load, but likely not auditory distraction and time pressure affected visual search performance and target memory in a real-world environment.
... However, the relationship between search behavior and subsequent memory for the scenes and the objects being searched for is complex. In general, searching for objects leads to memories of those objects and related elements in the scenes they inhabit (Castelhano & Henderson, 2005;Draschkow et al., 2014;Helbing et al., 2020;Williams et al., 2005). Scenes learned incidentally in a change blindness task (i.e., visual search for change) could be recognized as well as explicitly memorized scenes even 2 weeks after exposure (Utochkin & Wolfe, 2018). ...
... Though across the literature there is strong evidence that search can create a robust incidental memory trace for the target and related objects in a scene (Castelhano & Henderson, 2005;Draschkow et al., 2014;Helbing et al., 2020;Utochkin & Wolfe, 2018;Williams et al., 2005), a recent line of research has found evidence to the contrary. Specifically, Chen and Wyble (2015) asked participants to search for a letter among digits and report its location for tens of trials before presenting a surprise trial, in which participants were asked to report the identity of the letter instead of its location. ...
... It has been previously demonstrated that search for objects in natural scenes creates incidental memory of those scenes and the searched objects (Castelhano & Henderson, 2005;Draschkow et al., 2014;Helbing et al., 2020;Utochkin & Wolfe, 2018;Williams et al., 2005). However, we demonstrated that this is not necessarily the case using an AA task coupled with visual recall. ...
Searching within natural scenes can induce incidental encoding of information about the scene and the target, particularly when the scene is complex or repeated. However, recent evidence from attribute amnesia (AA) suggests that in some situations, searchers can find a target without building a robust incidental memory of it’s task relevant features. Through drawing-based visual recall and an AA search task, we investigated whether search in natural scenes necessitates memory encoding. Participants repeatedly searched for and located an easily detected item in novel scenes for numerous trials before being unexpectedly prompted to draw either the entire scene (Experiment 1) or their search target (Experiment 2) directly after viewing the search image. Naïve raters assessed the similarity of the drawings to the original information. We found that surprise-trial drawings of the scene and search target were both poorly recognizable, but the same drawers produced highly recognizable drawings on the next trial when they had an expectation to draw the image. Experiment 3 further showed that the poor surprise trial memory could not merely be attributed to interference from the surprising event. Our findings suggest that even for searches done in natural scenes, it is possible to locate a target without creating a robust memory of either it or the scene it was in, even if attended to just a few seconds prior. This disconnection between attention and memory might reflect a fundamental property of cognitive computations designed to optimize task performance and minimize resource use.
... In real life, we more commonly complete goal-directed behavior, such as setting the table, during which the location and identity representation of the surrounding objects is generated "on the fly." Recent work has shown that the representations generated through natural behavior are more reliable than those generated through explicit memorization (Draschkow, Wolfe, & Võ, 2014;Helbing, Draschkow, & Võ, 2020). ...
... In parallel with the previous sections, we emphasize the importance of studying active natural behavior Foulsham et al., 2011;Malcolm et al., 2016;Tatler, 2014; and how VLTMs are generated as a natural by-product of interactions with the environment (Draschkow & Võ, 2017;Helbing et al., 2020), as these representations support seamless everyday activities. In comparison to memory investigations in which memorization is the explicit task, during ecological behavior it is not necessary to constantly instruct ourselves to remember everything in our surroundings. ...
... Search within naturalistic images created more robust memories for the identity of target objects than representations formed as a result of explicit memorization (Draschkow et al., 2014;Josephs, Draschkow, Wolfe, & Võ, 2016). During immersive searches in virtual reality this search superiority even leads to more reliable incidentally generated spatial representations when compared to memories formed under explicit instruction to memorize (Helbing et al., 2020). Critically, incidental encoding seems to strongly rely on the availability of meaningful scene semantics in the stimulus materials used (Draschkow et al., 2014;Võ et al., 2019). ...
Research within visual cognition has made tremendous strides in uncovering the basic operating characteristics of the visual system by reducing the complexity of natural vision to artificial but well-controlled experimental tasks and stimuli. This reductionist approach has for example been used to assess the basic limitations of visual attention, visual working memory (VWM) capacity, and the fidelity of visual long-term memory (VLTM). The assessment of these limits is usually made in a pure sense, irrespective of goals, actions, and priors. While it is important to map out the bottlenecks our visual system faces, we focus here on selected examples of how such limitations can be overcome. Recent findings suggest that during more natural tasks, capacity may be higher than reductionist research suggests and that separable systems subserve different actions, such as reaching and looking, which might provide important insights about how pure attentional or memory limitations could be circumvented. We also review evidence suggesting that the closer we get to naturalistic behavior, the more we encounter implicit learning mechanisms that operate “for free” and “on the fly.” These mechanisms provide a surprisingly rich visual experience, which can support capacity-limited systems. We speculate whether natural tasks may yield different estimates of the limitations of VWM, VLTM, and attention, and propose that capacity measurements should also pass the real-world test within naturalistic frameworks. Our review highlights various approaches for this and suggests that our understanding of visual cognition will benefit from incorporating the complexities of real-world cognition in experimental approaches.
... As an example, Francis et al. (2017) found that moral judgments on a trolley problem were dramatically different in a virtual reality presentation compared to text vignettes. Other works that we review have found that processes ranging from visual processing (Amme et al., 2024), learning (Rosenberg et al., 2021;Collins, 2024), memory (Helbing et al., 2020) and more are changed when naturalistic elements are introduced. The works that we review in this section thus suggest that experiments that span a broader range of naturalistic settings are essential for drawing correct inferences about the system as a whole. ...
... Example 2: memory performance differs between naturalistic search and explicit memorization. When participants are explicitly instructed to memorize objects in a 3D home environment, their subsequent recall accuracy was significantly lower than when they incidentally encounter the same objects during visual search tasks (Helbing et al., 2020). This effect extends to spatial memory as well. ...
Artificial Intelligence increasingly pursues large, complex models that perform many tasks within increasingly realistic domains. How, if at all, should these developments in AI influence cognitive science? We argue that progress in AI offers timely opportunities for cognitive science to embrace experiments with increasingly naturalistic stimuli, tasks, and behaviors; and computational models that can accommodate these changes. We first review a growing body of research spanning neuroscience, cognitive science, and AI that suggests that incorporating a broader range of naturalistic experimental paradigms (and models that accommodate them) may be necessary to resolve some aspects of natural intelligence and ensure that our theories generalize. We then suggest that integrating recent progress in AI and cognitive science will enable us to engage with more naturalistic phenomena without giving up experimental control or the pursuit of theoretically grounded understanding. We offer practical guidance on how methodological practices can contribute to cumulative progress in naturalistic computational cognitive science, and illustrate a path towards building computational models that solve the real problems of natural cognition - together with a reductive understanding of the processes and principles by which they do so.
... It is known that incidentally encoded information can lead to reliable memory, especially when natural scenes are used (Castelhano & Henderson, 2005;Hollingworth & Henderson, 2002;Olejarczyk, Luke & Henderson, 2014;Williams, Henderson & Zacks, 2005) or when performing natural actions (Tatler & Tatler, 2013). Recent results also suggest a bigger role of this kind of memory to optimize visual search in more ecologically valid settings (Draschkow, Wolfe & Vö, 2014;Guevara-Pinto et al., 2020;Helbing, Draschkow & Vö, 2020;Kristjánsson & Draschkow, 2021). Our experiment supports this claim. ...
... It is also important to point out that our participants did not have explicit instructions requiring memorization of target objects or the virtual environment: knowledge about the different objects and their locations was acquired incidentally. This, paradoxically, seems to increase location memory (see Helbing et al., 2020). ...
Most research on visual search has used simple tasks presented on a computer screen. However, in natural situations visual search almost always involves eye, head, and body movements in a three-dimensional (3D) environment. The different constraints imposed by these two types of search tasks might explain some of the discrepancies in our understanding concerning the use of memory resources and the role of contextual objects during search. To explore this issue, we analyzed a visual search task performed in an immersive virtual reality apartment. Participants searched for a series of geometric 3D objects while eye movements and head coordinates were recorded. Participants explored the apartment to locate target objects whose location and visibility were manipulated. For objects with reliable locations, we found that repeated searches led to a decrease in search time and number of fixations and to a reduction of errors. Searching for those objects that had been visible in previous trials but were only tested at the end of the experiment was also easier than finding objects for the first time, indicating incidental learning of context. More importantly, we found that body movements showed changes that reflected memory for target location: trajectories were shorter and movement velocities were higher, but only for those objects that had been searched for multiple times. We conclude that memory of 3D space and target location is a critical component of visual search and also modifies movement kinematics. In natural search, memory is used to optimize movement control and reduce energetic costs.
... Visual Information. During attention allocation in a dynamic and complex scene, relevant anchor objects-those with a spatial relationship to the target object-can guide attention to a faster reaction time, less scene coverage, and less time between fixating on the anchor and the target object [54,24,2]. Therefore, we need to encode each frame of a given video to extract target and non-target features which an agent needs in order to effectively select the next fixation locations. Next, we describe in detail how this encoding is done (see Figure. ...
... We recruited 20 participants (5 female and 15 male, ages [22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39] with at least three years of driving experience (Mean=9.7, SD=5.8). ...
... The proportions of the rooms' content were similar to real-world scenes; each room was populated with 36 unique objects, 8 anchor objects, and 28 local objects, of which 6 were selected as target objects. For a description of the scenes, please refer to Helbing et al. (2020). Out of the 16 rooms, one living room was set aside for participants to train with and did not appear after the training phase, leaving 15 rooms for the actual task. ...
... The authors thank Jason Helbing for the construction of the complex, indoor scenes (Helbing et al., 2020) that were used in this study. ...
Visual search in natural scenes is a complex task relying on peripheral vision to detect potential targets and central vision to verify them. The segregation of the visual fields has been particularly established by on-screen experiments. We conducted a gaze-contingent experiment in virtual reality in order to test how the perceived roles of central and peripheral visions translated to more natural settings. The use of everyday scenes in virtual reality allowed us to study visual attention by implementing a fairly ecological protocol that cannot be implemented in the real world. Central or peripheral vision was masked during visual search, with target objects selected according to scene semantic rules. Analyzing the resulting search behavior, we found that target objects that were not spatially constrained to a probable location within the scene impacted search measures negatively. Our results diverge from on-screen studies in that search performances were only slightly affected by central vision loss. In particular, a central mask did not impact verification times when the target was grammatically constrained to an anchor object. Our findings demonstrates that the role of central vision (up to 6 degrees of eccentricities) in identifying objects in natural scenes seems to be minor, while the role of peripheral preprocessing of targets in immersive real-world searches may have been underestimated by on-screen experiments.
... We propose that a critical difference is the type or strength of memory that was formed. In fact, it is known that searching for objects produces stronger memories for those objects than do other types of encoding (Draschkow et al., 2014;Helbing et al., 2020). Despite this, the role of memory strength or type during semantically relevant search has not been examined. ...
Episodic memory and semantic knowledge can each exert strong influences on visual attention when we search through real-world scenes. However, there is debate surrounding how they interact when both are present; specifically, results conflict as to whether memory consistently improves visual search when semantic knowledge is available to guide search. These conflicting results could be driven by distinct effects of different types of episodic memory, but this possibility has not been examined. To test this, we tracked participants’ eyes while they searched for objects in semantically congruent and incongruent locations within scenes during a study and test phase. In the test phase containing studied and new scenes, participants gave confidence-based recognition memory judgments that indexed different types of episodic memory (i.e., recollection, familiarity, unconscious memory) for the background scenes, then they searched for the target. We found that semantic knowledge consistently influenced both early and late eye movements, but the influence of memory depended on the type of memory involved. Recollection improved first saccade accuracy in terms of heading towards the target in both congruent and incongruent scenes. In contrast, unconscious memory gradually improved scanpath efficiency over the course of search, but only when semantic knowledge was relatively ineffective (i.e., incongruent scenes). Together, these findings indicate that episodic memory and semantic knowledge are rationally integrated to optimize attentional guidance, such that the most precise or effective forms of information available – which depends on the type of episodic memory available – are prioritized.
... Importantly, the task setting encouraged the formation of reliable LTMs (see Data S1 for preliminary learning experiments) through goal-directed interactions with a stable environment, as opposed to explicit memorization. [62][63][64] Participants thus may have formed memories connected to different aspects of the task environment (e.g., scene context or configuration in model display). Further, contributions of LTM might be dependent on factors such as the strength or explicit availability of memory content, as suggested by the relationship between explicit recall performance and certain LTM contributions (see also Musfeld et al., 24 Musfeld et al., 50 and Ngiam et al. 65 ). ...
... Cognitive Research: Principles and Implications (2025) 10:4 2021; Failing & Theeuwes, 2018;Grégoire et al., 2021;Le Pelley & Newell, 2023;Schmidt et al., 2015). Learning about the behavioural relevance of otherwise neutral objects can occur outside the laboratory, such as through repeated interactions with these objects during everyday behaviour in natural settings (Helbing et al., 2020;Kristjánsson & Draschkow, 2021). Thus, to better capture realistic stimulus-context learning, researchers have demonstrated that real-world stimuli could elicit attentional guidance patterns that relate to participants' previous experiences. ...
Human attention can be guided by semantic information conveyed by individual objects in the environment. Over time, we learn to allocate attention resources towards stimuli that are behaviourally relevant to ongoing action, leading to attention capture by meaningful peripheral stimuli. A common example includes, while driving, stimuli that imply a possibly hazardous scenario (e.g. a pedestrian about to cross the road) warrant attentional prioritization to ensure safe proceedings. In the current study, we report a novel phenomenon in which the guidance of attention is dependent on the stimuli appearing in a behaviourally relevant context. Using a driving simulator, we simulated a real-world driving task representing an overlearned behaviour for licensed drivers. While driving, participants underwent a peripheral cue-target paradigm where a roadside pedestrian avatar (target) appeared following a cylinder cue. Results revealed that, during simulated driving conditions, participants (all with driver’s licenses) showed greater attentional facilitation when pedestrians were oriented towards the road compared to away. This orientation-specific selectivity was not seen if the 3-D context was removed (Experiment 1) or the same visual scene was presented, but participants’ viewpoints remained stationary (Experiment 2), or an inanimate object served as a target during simulated driving (Experiment 3). This context-specific attention modulation likely reflects drivers’ expertise in automatically attending to behaviourally relevant information in a context-dependent manner.
... The benefits of contextual memory associations have also been demonstrated when searching for consistently located targets within repeated scenes [24][25][26][27][28][29] and immersive virtual and real environments [30,31]. The benefits of contextual cueing extend to memory recall of target identity and location in healthy and at-risk populations [32][33][34][35][36][37][38]. Also, studies using learned scenes as attention-guiding memory cues for subsequent target identification have revealed memory-guided signals that proactively prepare anticipatory attention to anticipate target locations [39][40][41][42]. ...
During visual search, we quickly learn to attend to an object’s likely location. Research has shown that this process can be guided by learning target locations based on consistent spatial contextual associations or other statistical regularities. Here, we tested how different types of associations guide learning and the utilisation of established memories for different purposes. Participants learned contextual associations or rule-like statistical regularities that predicted target locations within different scenes. The consequences of this learning for subsequent performance were then evaluated on attention-orienting and memory-recall tasks. Participants demonstrated facilitated attention-orienting and recall performance based on both contextual associations and statistical regularities. Contextual associations facilitated attention orienting with a different time course compared to statistical regularities. Benefits to memory-recall performance depended on the alignment between the learned association or regularity and the recall demands. The distinct patterns of behavioural facilitation by contextual associations and statistical regularities show how different forms of long-term memory may influence neural information processing through different modulatory mechanisms.
... Other studies on visual search using naturalistic scenes have further demonstrated that incidentally learned object-location associations can guide attention and enable the observer to find objects at learned locations much faster in the future (Draschkow et al., 2021;Draschkow & Võ, 2016;Hollingworth, 2006Hollingworth, , 2007Võ & Wolfe, 2015). This is true for contextual information stored in semantic knowledge structures acquired during life-long learning (e.g., if you look for a toaster, you typically find it on a kitchen shelf) as well as recently established episodic memory representations (e.g., in my friend's kitchen, the toaster is next to the fridge) (Helbing et al., 2020;Võ & Wolfe, 2015). ...
... In addition, neutral, arbitrary objects without such inherent property that attract attention can become behaviourally relevant through learning in a speci c context. An abstract experimental stimulus (e.g. a black square) that has been previously associated with a positive or negative outcome (e.g., monetary rewards, electric shock) can gain preferential attention in behavioural tasks when compared to objects that are not exposed to such association (Anderson et Learning about the behavioural relevance of otherwise neutral objects can occur outside the laboratory, such as in naturalistic settings that occur through daily behaviour (Helbing et al., 2020;Kristjánsson & Draschkow 2021). Thus, to better capture realistic stimulus-context learning, researchers have demonstrated that real-world stimuli could elicit attentional guidance patterns that relate to previous experiences by participants in their lifetime. ...
Human attention can be guided by semantic information of individual objects in the environment (e.g., traffic light). In the current study, we reported a novel phenomenon in which certain behavioural contexts could determine whether guidance of attention occurs. Using a driving simulator, we simulated a real-world driving task representing an over-learned behaviour. A peripheral cue-target paradigm was implemented where a roadside pedestrian avatar (target) appeared following a cylinder cue. Results revealed that, during simulated driving, participants (all with driver’s licenses) showed increased attentional facilitation when pedestrians were oriented toward the road compared to away. This orientation-specific selectivity was not seen if the 3-D context was removed, or the same visual scene was presented but participants’ viewpoint remained stationary, or an inanimate object served as a target during simulated driving. This context-specific attention modulation likely reflects drivers’ expertise in automaticallyattending to behaviourally relevant information in a context-dependent manner.
... Moving out of the lab and into real-world settings provides many challenges, such as low reproducibility and limited information on the collected data (Shamay-Tsoory & Mendelsohn, 2019), that might make drawing conclusions difficult. The continuing strive to investigate human behavior beyond the classical lab setups, the challenge of real-world experiments, and a rise in new and better technology have led many researchers to utilize virtual reality (VR) to bridge this gap (Bell et al., 2020;Draschkow et al., 2022;Helbing et al., 2020Helbing et al., , 2022Llanes-Jurado et al., 2020). Virtual reality allows for complex experiments with a high degree of freedom for the subject while maintaining high control and replicability, and providing knowledge about participants' behavior (Bohil et al., 2011;Pan & Hamilton, 2018). ...
Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.
... When participants are asked to create virtual scenes from scratch, they create a layout by first placing anchors and subsequently arranging local objects accordingly 22 , which influences their search and recall behaviour as well. It has also been shown that gaze can fall onto task-relevant and -irrelevant stimuli, the latter of which still play a role in improving an overall spatial layout and memory that can ultimately improve task-relevant performance [38][39][40] . Thus, it is rather surprising that participants would not implicitly relate local object positions to the seemingly irrelevant anchors in the scene, given the supportive role that task-irrelevant anchors might play. ...
Interacting with objects in our environment requires determining their locations, often with respect to surrounding objects (i.e., allocentrically). According to the scene grammar framework, these usually small, local objects are movable within a scene and represent the lowest level of a scene’s hierarchy. How do higher hierarchical levels of scene grammar influence allocentric coding for memory-guided actions? Here, we focused on the effect of large, immovable objects (anchors) on the encoding of local object positions. In a virtual reality study, participants (n = 30) viewed one of four possible scenes (two kitchens or two bathrooms), with two anchors connected by a shelf, onto which were presented three local objects (congruent with one anchor) (Encoding). The scene was re-presented (Test) with 1) local objects missing and 2) one of the anchors shifted (Shift) or not (No shift). Participants, then, saw a floating local object (target), which they grabbed and placed back on the shelf in its remembered position (Response). Eye-tracking data revealed that both local objects and anchors were fixated, with preference for local objects. Additionally, anchors guided allocentric coding of local objects, despite being task-irrelevant. Overall, anchors implicitly influence spatial coding of local object locations for memory-guided actions within naturalistic (virtual) environments.
... Other studies on visual search using naturalistic scenes have further demonstrated that incidentally learned object-location associations can guide attention and enable the observer to find objects at learned locations much faster in the future (Draschkow et al., 2021;Draschkow & Võ, 2016;Hollingworth, 2006Hollingworth, , 2007Võ & Wolfe, 2015). This is true for contextual information stored in semantic knowledge structures acquired during life-long learning (e.g., if you look for a toaster, you typically find it on a kitchen shelf) as well as recently established episodic memory representations (e.g., in my friend's kitchen, the toaster is next to the fridge) (Helbing et al., 2020;Võ & Wolfe, 2015). ...
In a visual environment, objects are encoded within a spatial and temporal context. The present study investigated whether incidental learning of spatial and temporal associations in hybrid visual and memory search enable observers to predict targets in space and time. In three experiments, observers looked for four, previously memorized, target items across many trials. We examined effects of learning target item sequences (e.g., the butterfly always follows the paint box), target location sequences (e.g., a target in the lower middle-left always follows a target in the right corner), target item-location associations (the butterfly is always in the right corner), and target item-location sequences (the butterfly in the right corner always follows the paint box in the lower middle-left). We found only weak incidental learning for the sequences of target items or target locations alone. By contrast, we found good learning of target item-location associations. Furthermore, we did find a reliable effect of sequence learning for target item-location associations. These findings suggest that spatiotemporal learning in hybrid search is hierarchical: Only if spatial and non-spatial target features are bound, temporal associations can bias attention dynamically to the task-relevant features expected to occur next.
... The various XR behavioral data streams enable relating where someone is looking to where their head is directed in 3D space within the same frame of reference (Bischof et al., 2023;David et al., 2020;Thom et al., 2023). Transporting cognitive and behavioral research from the flat screen to the immersive environment has already led to foundational insights into how people perceive and navigate their environment (Cohen et al., 2020;Doeller et al., 2010;Finnegan et al., 2023;Sato et al., 2006;Warren et al., 2001), how they allocate attention to find things (Helbing et al., 2022;Li et al., 2016), and how people use memory to complete tasks Helbing et al., 2020;Laurent et al., 2016)-just to name a few. ...
Extended reality (XR, including augmented and virtual reality) creates a powerful intersection between information technology and cognitive, clinical, and education sciences. XR technology has long captured the public imagination, and its development is the focus of major technology companies. This article demonstrates the potential of XR to (1) deliver behavioral insights, (2) transform clinical treatments, and (3) improve learning and education. However, without appropriate policy, funding, and infrastructural investment, many research institutions will struggle to keep pace with the advances and opportunities of XR. To realize the full potential of XR for basic and translational research, funding should incentivize (1) appropriate training, (2) open software solutions, and (3) collaborations between complementary academic and industry partners. Bolstering the XR research infrastructure with the right investments and incentives is vital for delivering on the potential for transformative discoveries, innovations, and applications.
... For example, Choe et al. (2017) found that searching for an object in scenes TO SEARCH IS TO SEE 1171 produced less effective memory for that scene than evaluating one's aesthetic preference for the scene. Our findings are also consistent with Helbing et al. (2020) who observed a search superiority effect where tasking participants with searching for an object in 3D virtual reality space led to greater memory for that object and its location than an intentional memorization task did. Thus it may be particularly interesting to consider the possibility that search-based integration may vary substantially not only based on the kinds of integration being tested (e.g., integration of identification and representation vs. integration of representation and comparison) but also the event structure associated with the scenes being searched (e.g., searching a sequence of scenes for an object target, vs. searching a room for an object, vs. searching an ongoing event for an agent). ...
Participants in incidental change detection studies often miss large changes to visually salient or conceptually relevant objects such as actor substitutions across video cuts, but there are competing explanations of why participants fail to detect these changes. According to an integrative processing account, object-based attention typically induces integrated representation and comparison processes sufficient to detect changes to that object. On this view, participants miss changes in incidental paradigms because those paradigms fail to elicit the level of attention necessary to trigger integrated representation and comparison processes. In contrast, a selective processing account posits that representation and comparison processes needed to detect changes do not occur by default, even for attended objects, but are only elicited in response to specific functional needs. In four experiments, we tested detection of actor substitutions when participants engaged in tasks that required actor identity processing but did not necessarily require the combination of processes necessary to detect changes. Change blindness for actor substitutions persisted when participants counted the number of actors in the video and sometimes persisted when participants were instructed to remember the substituted actor for later recall. Change blindness consistently diminished, however, when participants were shown the prechange actor before or during the video and instructed to search for that actor in the video. Our results refine the contrast between selective and integrative processing by specifying how task demands to create durable visual representations can remain independent of comparison processes, while search demands can induce integrative comparison processes in a naturalistic setting. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
... Here, our aim was to test whether these findings generalize to immersive 360-degree visual search and to demonstrate the feasibility of studying scene inversion effects on behavior, including eye movements, in virtual reality (VR). By embracing more unconstrained and naturalistic task settings, VR promises to increase the external validity of findings Helbing et al., 2020Helbing et al., , 2022Parsons, 2015). In some cases, effects demonstrated in laboratory setups were weaker or absent when tested in more realistic settings which often engage behavior that is multimodal, immersive, and selfreferential (David, Beitner, et al., 2020;Johnsdorf et al., 2023;Kisker et al., 2021;Li et al., 2016;Rubo et al., 2021;Schöne et al., 2019Schöne et al., , 2021Zhang & Pan, 2022). ...
Image inversion is a powerful tool for investigating cognitive mechanisms of visual perception. However, studies have mainly used inversion in paradigms presented on two-dimensional computer screens. It remains open whether disruptive effects of inversion also hold true in more naturalistic scenarios. In our study, we used scene inversion in virtual reality in combination with eye tracking to investigate the mechanisms of repeated visual search through three-dimensional immersive indoor scenes. Scene inversion affected all gaze and head measures except fixation durations and saccade amplitudes. Our behavioral results, surprisingly, did not entirely follow as hypothesized: While search efficiency dropped significantly in inverted scenes, participants did not utilize more memory as measured by search time slopes. This indicates that despite the disruption, participants did not try to compensate the increased difficulty by using more memory. Our study highlights the importance of investigating classical experimental paradigms in more naturalistic scenarios to advance research on daily human behavior.
... Finally, by using VR, we were able to measure the heading-direction bias alongside the gaze bias as participants' head, eye, and body were unconstrained. To date, the benefits of VR have been appreciated most prominently by researchers studying naturalistic human navigation, ethology, and long-term memory (Mobbs et al., 2021;Helbing, Draschkow, & Võ, 2020;Stangl et al., 2020;Topalovic et al., 2020;Draschkow & Võ, 2017;Li, Aivar, Kit, Tong, & Hayhoe, 2016). Our present findings further highlight the benefits of using VR (combined with eyeand head-tracking) to study bodily orienting behavior (Draschkow et al., 2021(Draschkow et al., , 2022 related to internal cognitive processes, as showcased here for internal attentional focusing in working memory. . ...
We shift our gaze even when we orient attention internally to visual representations in working memory. Here, we show the bodily orienting response associated with internal selective attention is widespread as it also includes the head. In three virtual reality experiments, participants remembered 2 visual items. After a working memory delay, a central color cue indicated which item needed to be reproduced from memory. After the cue, head movements became biased in the direction of the memorized location of the cued memory item—despite there being no items to orient toward in the external environment. The heading-direction bias had a distinct temporal profile from the gaze bias. Our findings reveal that directing attention within the spatial layout of visual working memory bears a strong relation to the overt head orienting response we engage when directing attention to sensory information in the external environment. The heading-direction bias further demonstrates common neural circuitry is engaged during external and internal orienting of attention.
... Recent studies investigating visual search using head-mounted displays highlight, in particular, that active behavior recruits memory to aid search performance in naturalistic settings. Active exploration of virtual environments prior to search has been shown to improve search performance by engaging spatial memory 46,69 , a benefit not seen for explicit memorization 70 . For example, one study demonstrates that spatial memory aids search by restricting attention to relevant areas of the scene 47 : when the location of a target object was changed from a learned location, participants continued to initially fixate on the learned location. ...
Visual search is a ubiquitous activity in real-world environments. Yet, traditionally, visual search is investigated in tightly controlled paradigms, where head-restricted participants locate a minimalistic target in a cluttered array that is presented on a computer screen. Do traditional visual search tasks predict performance in naturalistic settings, where participants actively explore complex, real-world scenes? Here, we leverage advances in virtual reality technology to test the degree to which classic and naturalistic search are limited by a common factor, set size, and the degree to which individual differences in classic search behavior predict naturalistic search behavior in a large sample of individuals (N = 75). In a naturalistic search task, participants looked for an object within their environment via a combination of head-turns and eye-movements using a head-mounted display. Then, in a classic search task, participants searched for a target within a simple array of colored letters using only eye-movements. In each task, we found that participants’ search performance was impacted by increases in set size—the number of items in the visual display. Critically, we observed that participants’ efficiency in classic search tasks—the degree to which set size slowed performance—indeed predicted efficiency in real-world scenes. These results demonstrate that classic, computer-based visual search tasks are excellent models of active, real-world search behavior.
... Methodologically, our study joins the rapidly growing list of efforts to investigate search in realistic virtual reality scenes Bennett et al., 2021;David et al., 2020David et al., , 2021Draschkow & Võ, 2017;Enders et al., 2021;Figueroa et al., 2017;Hadnett-Hunter et al., 2019;Helbing et al., 2020;Kit et al., 2014;T. Kristjánsson et al., 2022;Li et al., 2016Li et al., , 2018Lukashova-Sanz & Wahl, 2021;Olk et al., 2018). ...
Successful adaptive behavior requires efficient attentional and locomotive systems. Previous research has thoroughly investigated how we achieve this efficiency during natural behavior by exploiting prior knowledge related to targets of our actions (e.g., attending to metallic targets when looking for a pot) and to the environmental context (e.g., looking for the pot in the kitchen). Less is known about whether and how individual nontarget components of the environment support natural behavior. In our immersive virtual reality task, 24 adult participants searched for objects in naturalistic scenes in which we manipulated the presence and arrangement of large, static objects that anchor predictions about targets (e.g., the sink provides a prediction for the location of the soap). Our results show that gaze and body movements in this naturalistic setting are strongly guided by these anchors. These findings demonstrate that objects auxiliary to the target are incorporated into the representations guiding attention and locomotion.
... Objects undeniably define the scene (Bar, 2004;Võ et al., 2019) and guide our actions and memories (Draschkow & Võ, 2017;Helbing et al., 2020;Torralba et al., 2006). For this reason, there are numerous scientific questions for which an object-based measurement is preferable. ...
Psychophysical paradigms measure visual attention via localized test items to which observers must react or whose features have to be discriminated. These items, however, potentially interfere with the intended measurement, as they bias observers’ spatial and temporal attention to their location and presentation time. Furthermore, visual sensitivity for conventional test items naturally decreases with retinal eccentricity, which prevents direct comparison of central and peripheral attention assessments. We developed a stimulus that overcomes these limitations. A brief oriented discrimination signal is seamlessly embedded into a continuously changing 1/ f noise field, such that observers cannot anticipate potential test locations or times. Using our new protocol, we demonstrate that local orientation discrimination accuracy for 1/ f filtered signals is largely independent of retinal eccentricity. Moreover, we show that items present in the visual field indeed shape the distribution of visual attention, suggesting that classical studies investigating the spatiotemporal dynamics of visual attention via localized test items may have obtained a biased measure. We recommend our protocol as an efficient method to evaluate the behavioral and neurophysiological correlates of attentional orienting across space and time.
... As immersive environments aim to enhance user experience in gallery and museum settings, many exploratory studies have started to investigate visitors' experience and the feasibility of these VR applications (Hoang and Cox 2017;Petrelli 2019;Parker and Saker 2020;Gulhan et al. 2021). Experiments have mostly focussed so far on the general cognitive implications of using these environments, for example, crowd movement on navigation decisions in VR (Zhao et al. 2020), mental imagery and eye movements in VR (Chiquet et al. 2020), visual search in 3D scenes (Helbing et al. 2020), replication of findings from a lab-based inattentional blindness paradigm in VR (Schöne et al. 2021), or episodic memory in virtual museum rooms (van Helvoort et al. 2020). Experimental aesthetics research in VR remains to be explored. ...
Empirical aesthetics is beginning to branch off from conventional laboratory-based studies, leading to in-situ, immersive, often more accessible experiments. Here, we explored different types of aesthetic judgments of three-dimensional artworks in two contexts: virtual reality (VR), aiming for an immersive experience, and online settings aiming for an accessible setup for a remote audience. Following the pilot experiment conducted to select a set of 3D artworks, in the first experiment, participants freely engaged with virtual artworks via an eye-tracking-enabled VR headset and provided evaluations based on subjective measures of aesthetic experience such as ratings on liking, novelty, complexity, perceived viewing duration; and the objective viewing duration was also recorded. Results showed positive, linear, and mostly moderate correlations between liking and the other perceived judgment attributes. Supplementary eye-tracking data showed a range of viewing strategies and variation in viewing durations between participants and artworks. Results of the second experiment, adapted as a short online follow-up, showed converging evidence on correlations between the different aspects contributing to aesthetic judgments and suggested similarity of judgment strategies across contexts. In both settings, participants provided further insights via exit questionnaires. We speculate that both VR and online settings offer ecologically valid experimental contexts, create immersive visual arts experience, and enhance accessibility to cultural heritage.
... Finally, by using VR, we were able to measure the head-direction bias alongside the gaze bias as participants' head, eye, and body were unconstrained. To date, the benefits of VR have been appreciated most prominently by researchers studying naturalistic human navigation, ethology, and long-term memory [59][60][61][62][63][64] . Our present findings further highlight the benefits of using VR (combined with eye-and head-tracking) to study bodily orienting behaviour 11,31 related to internal cognitive processes, as showcased here for internal attentional focusing in working memory. ...
We shift our gaze even when we orient attention internally to visual representations in working memory. Here, we show the bodily orienting response associated with internal selective attention is widespread as it also includes the head. In three virtual reality (VR) experiments, participants remembered two visual items. After a working memory delay, a central colour cue indicated which item needed to be reproduced from memory. After the cue, head movements became biased in the direction of the memorised location of the cued memory item, despite there being no items to orient towards in the external environment. The head-direction bias had a distinct temporal profile from the gaze bias. Our findings reveal that directing attention within the spatial layout of visual working memory bears a strong relation to the overt head orienting response we engage when directing attention to sensory information in the external environment. The head-direction bias further demonstrates common neural circuitry is engaged during external and internal orienting of attention.
... In our view, it is central to the paradigm that learning is incidental during the study phase, else the effects in recognition memory will not be attributable to the anticipation of novelty, but instead will be the anticipation of an important stimulus that should be remembered later. A recent study showed that longer viewing time increased memory performance to a greater degree in an intentional learning condition than in an incidental condition (Helbing, Draschkow & Võ, 2020). Thus, anticipation of a stimulus to learn in an intentional learning paradigm could possibly improve performance. ...
Novelty is defined as the part of an experience that is not yet represented by memory systems. Novelty has been claimed to exert various memory‐enhancing effects. A pioneering study by Wittmann et al. (2007) has shown that memory formation may even benefit from the expectation of novelty. We aimed to replicate this assumed memory effect in four behavioral studies. However, our results do not support the idea that anticipated novel stimuli are more memorable than unexpected novelty. In our experiments, we systematically manipulated the novelty predicting cues to ensure that the expectations were correctly formed by the participants, however, the results showed that there was no memory enhancement for expected novel pictures in any of the examined indices, thus we could not replicate the main behavioral finding of Wittmann et al. (2007). These results call into question the original effect, and we argue that this fits more into current thinking on memory formation and brain function in general. Our results are more consistent with the view that unexpected stimuli are more likely to be retained by memory systems. Predictive coding theory suggests that unexpected stimuli are prioritized by the nervous system and this may also benefit memory processes. Novel stimuli may be unexpected and thus recognized better in some experimental setups, yet novelty and unexpectedness do not always coincide. We hope that our work can bring more consistency in the literature on novelty, as educational methods in general could also benefit from this clarification.
... Another type of continuous behavior of interest to researchers is eye movement data (Clay et al., 2019), which is becoming more widely accessible through increasingly available eye tracking solutions for VR, such as in the FOVE (FOVE Ltd., Torrance, CA, USA) or HTC Vive Pro Eye HMDs (HTC Corp., Xindian, New Taipei, Taiwan). VR eye tracking allows to capture an observer's gaze behavior during exploration of highly realistic and complex environments (Clay et al., 2019;Hayhoe & Rothkopf, 2011;Rothkopf et al., 2007), visual search (Helbing et al., 2020;Kit et al., 2014;Marek & Pollmann, 2020) or visual working memory tasks in VR (Draschkow et al., 2021(Draschkow et al., , 2022, or to directly compare the deployment of gaze while walking in a real building versus navigating its virtual twin (Drewes et al., 2021). Other continuous measures might come from mobile physiological sensors or EEG devices (e.g., Banaei et al., 2017;Gramann et al., 2014). ...
Virtual reality (VR) is a powerful tool for researchers due to its potential to study dynamic human behavior in highly naturalistic environments while retaining full control over the presented stimuli. Due to advancements in consumer hardware, VR devices are now very affordable and have also started to include technologies such as eye tracking, further extending potential research applications. Rendering engines such as Unity, Unreal, or Vizard now enable researchers to easily create complex VR environments. However, implementing the experimental design can still pose a challenge, and these packages do not provide out-of-the-box support for trial-based behavioral experiments. Here, we present a Python toolbox, designed to facilitate common tasks when developing experiments using the Vizard VR platform. It includes functionality for common tasks like creating, randomizing, and presenting trial-based experimental designs or saving results to standardized file formats. Moreover, the toolbox greatly simplifies continuous recording of eye and body movements using any hardware supported in Vizard. We further implement and describe a simple goal-directed reaching task in VR and show sample data recorded from five volunteers. The toolbox, example code, and data are all available on GitHub under an open-source license. We hope that our toolbox can simplify VR experiment development, reduce code duplication, and aid reproducibility and open-science efforts.
... The applications of combined cortical EEG recordings (Jiang et al., 2021) and specific gait parameters (Pieruccini-Faria et al., 2021) can independently predict the development of cognitive impairment in older and cognitive impaired persons. However, the integration of such MoBI applications into exergames is limited, but will be elaborated on (Helbing et al., 2020). At the same time, XR-MoBI exergames with the appropriate unit-architecture of telerehabilitation are limitly published. ...
A major concern of public health authorities is to also encourage adults to be exposed to enriched environments (sensory and cognitive-motor activity) during the pandemic lockdown, as was recently the case worldwide during the COVID-19 outbreak. Games for adults that require physical activity, known as exergames, offer opportunities here. In particular, the output of the gaming industry nowadays offers computer games with extended reality (XR) which combines real and virtual environments and refers to human-machine interactions generated by computers and wearable technologies. For example, playing the game in front of a computer screen while standing or walking on a force plate or treadmill allows the user to react to certain infrastructural changes and obstacles within the virtual environment. Recent developments, optimization, and minimizations in wearable technology have produced wireless headsets and sensors that allow for unrestricted whole-body movement. This makes the virtual experience more immersive and provides the opportunity for greater engagement than traditional exercise. Currently, XR serves as an umbrella term for current immersive technologies as well as future realities that enhance the experience with features that produce new controllable environments. Overall, these technology-enhanced exergames challenge the adult user and modify the experience by increasing sensory stimulation and creating an environment where virtual and real elements interact. As a therapy, exergames can potentially create new environments and visualizations that may be more ecologically valid and thus simulate real activities of daily living that can be trained. Furthermore, by adding telemedicine features to the exergame, progress over time can be closely monitored and feedback provided, offering future opportunities for cognitive-motor assessment. To more optimally serve and challenge adults both physically and cognitively over time in future lockdowns, there is a need to provide long-term remote training and feedback. Particularly related to activities of daily living that create opportunities for effective and lasting rehabilitation for elderly and sufferers from chronic non-communicable diseases (CNDs). The aim of the current review is to envision the remote training and monitoring of physical and cognitive aspects for adults with limited mobility (due to disability, disease, or age), through the implementation of concurrent telehealth and exergame features using XR and wireless sensor technologies.
... Moreover, with the help of eye-tracking, it is possible to identify higher nervous functions such as emotions, regret, and disappointment (Nakhaeizadeh et al., 2020), as well as memorization processes (Bault et al., 2016). It has also been shown that eye-tracking technologies can be used to control decision-making (Helbing et al., 2020) for example by increasing its value (Fridman et al., 2018). A recent study (Smith and Krajbich, 2019) showed the importance of the multiplicative model, which states that greater attention to options when making a choice, has higher influences on choices. ...
The concept of using eye-tracking in virtual reality for education has been researched in various fields over the past years. With this review, we aim to discuss the recent advancements and applications in this area, explain the technological aspects, highlight the advantages of this approach and inspire interest in the field. Eye-tracking has already been used in science for many decades and now has been substantially reinforced by the addition of virtual and augmented reality technologies. The first part of the review is a general overview of eye-tracking concepts, technical parts, and their applications. In the second part, the focus shifted toward the application of eye-tracking in virtual reality. The third part, first the description of the recently emerged concept of eye-tracking in virtual reality is given, followed by the current applications to education and studying, which has not been thoroughly described before. We describe the main findings, technological aspects, and advantages of this approach.
... In recent years, VR 47 has become a go-to tool for investigating long-term memory and spatial navigation 21,[48][49][50][51][52][53] . By contrast, the use of VR for investigating working memory remains rare 5 . ...
As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood because most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realization that multiple spatial frames support working memory in natural behaviour. The results also illustrate how virtual reality can be a critical experimental tool to characterize this core memory system.
... Behavioral data were analyzed using R (R Core Team, 2018) using the approach described in Helbing et al. (2020) and Draschkow and Võ (2017). Differences in analysis procedures between experiments are highlighted in the corresponding section. ...
Visual search is a fundamental element of human behavior and is predominantly studied in a laboratory setting using static displays. However, real-life search is often an extended process taking place in dynamic environments. We have designed a dynamic-search task in order to incorporate the temporal dimension into visual search. Using this task, we tested how participants learn and utilize spatiotemporal regularities embedded within the environment to guide performance. Participants searched for eight instances of a target that faded in and out of a display containing similarly transient distractors. In each trial, four of the eight targets appeared in a temporally predictable fashion with one target appearing in each of four spatially separated quadrants. The other four targets were spatially and temporally unpredictable. Participants’ performance was significantly better for spatiotemporally predictable compared to unpredictable targets (Experiments 1–4). The effects were reliable over different patterns of spatiotemporal predictability (Experiment 2) and primarily reflected long-term learning over trials (Experiments 3, 4), although single-trial priming effects also contributed (Experiment 4). Eye-movement recordings (Experiment 1) revealed that spatiotemporal regularities guide attention proactively and dynamically. Taken together, our results show that regularities across both space and time can guide visual search and this guidance can primarily be attributed to robust long-term representations of these regularities.
... Objects undeniably define the scene (Bar, 2004;Võ, Boettcher, & Draschkow, 2019) and guide our actions and memories (Torralba, Oliva, Castelhano, & Henderson, 2006;Draschkow & Võ, 2017;Helbing, Draschkow, & Võ, 2020). For this reason, there are numerous scientific questions for which an object-based measurement is preferable. ...
Psychophysical paradigms measure visual attention via localized test items to which observers must react or whose features have to be discriminated. These items, however, potentially interfere with the intended measurement as they bias observers’ spatial and temporal attention to their location and presentation time. Furthermore, visual sensitivity for conventional test items naturally decreases with retinal eccentricity, which prevents direct comparison of central and peripheral attention assessments. We developed a stimulus that overcomes these limitations. A brief oriented discrimination signal is seamlessly embedded into a continuously changing 1/f noise field, such that observers cannot anticipate potential test locations or times. Using our new protocol, we demonstrate that local orientation discrimination accuracy for 1/f filtered signals is largely independent of retinal eccentricity. Moreover, we show that items present in the visual field indeed shape the distribution of visual attention, suggesting that classical studies investigating the spatiotemporal dynamics of visual attention via localized test items may have obtained a biased measure. We recommend our protocol as an efficient method to evaluate the behavioral and neurophysiological correlates of attentional orienting across space and time.
Significance statement
Where (and when) we pay attention can be experimentally quantified via visual sensitivity: Attending to a certain visual signal results in better detection and feature discrimination performance. This approach is widely used, but poses an unrecognized dilemma: The test signal itself, typically a grating or letter stimulus, biases observers’ perception and expectations – and thus also the attention measurement. We developed a stimulus that manages without test items. The signal to measure attention is seamlessly embedded in a dynamic 1/f noise field, so that neither spatial nor temporal information about signal presentation is conveyed. Unlike with conventional approaches, perception and expectations in this new protocol remain unbiased, and the undistorted spatial and temporal spread of visual attention can be measured.
... Considering the role of scene-object consistencies on memory has important real-world implications for how we design scenes, and how we test memory. Some of the first seminal work looking at scene consistency and memory tested memory for real graduate student offices (Pezdek et al., 1989), and recent work has brought questions about scene memory into virtual reality (Helbing et al., 2020). It will be exciting to see whether our findings can help guide the design of realworld scenes, based on what aspects we wish to be memorable (Bainbridge, 2019): a key object, all objects, or the gist of the scene. ...
Humans are highly sensitive to the statistical relationships between features and objects within visual scenes. Inconsistent objects within scenes (e.g., a mailbox in a bedroom) instantly jump out to us and are known to catch our attention. However, it is debated whether such semantic inconsistencies result in boosted memory for the scene, impaired memory, or have no influence on memory. Here, we examined the relationship of scene–object consistencies on memory representations measured through drawings made during recall. Participants (N = 30) were eye-tracked while studying 12 real-world scene images with an added object that was either semantically consistent or inconsistent. After a 6-minute distractor task, they drew the scenes from memory while pen movements were tracked electronically. Online scorers (N = 1,725) rated each drawing for diagnosticity, object detail, spatial detail, and memory errors. Inconsistent scenes were recalled more frequently, but contained less object detail. Further, inconsistent objects elicited more errors reflecting looser memory binding (e.g., migration across images). These results point to a dual effect in memory of boosted global (scene) but diminished local (object) information. Finally, we observed that participants fixate longest on inconsistent objects, but these fixations during study were not correlated with recall performance, time, or drawing order. In sum, these results show a nuanced effect of scene inconsistencies on memory detail during recall.
... However, thanks to ever improving display technology, decreasing costs and ease-of-use, over the recent years, VR systems have become a research tool in many fields. This includes -besides the entertainment market -highly regulated fields like medicine [e.g., Dentistry (Huang et al., 2018), education and training (Bernardo, 2017;Izard et al., 2018), simulation, diagnosis and rehabilitation of visual impairments (Baheux et al., 2005;Jones et al., 2020)] and psychotherapy (e.g., autism therapy: Georgescu et al., 2014;Lahiri et al., 2015, fear andanxiety disorders: Hong et al., 2017;Kim et al., 2018;Matthis et al., 2018), as well as in areas directly relevant to psychophysical research such as attentional allocation (Helbing et al., 2020). As fears of long-term negative effects of VR use have so far not been confirmed (e.g., Turnbull and Phillips, 2017), and the recent VR goggles approach photorealistic capabilities while being more and more comfortable to wear, we are now in a position to ask, to what extent a HMD can be used as a proxy for a real-world setting in the context of gaze tracking -a question that has previously only been addressed in a limited scope. ...
How vision guides gaze in realistic settings has been researched for decades. Human gaze behavior is typically measured in laboratory settings that are well controlled but feature-reduced and movement-constrained, in sharp contrast to real-life gaze control that combines eye, head, and body movements. Previous real-world research has shown environmental factors such as terrain difficulty to affect gaze; however, real-world settings are difficult to control or replicate. Virtual reality (VR) offers the experimental control of a laboratory, yet approximates freedom and visual complexity of the real world (RW). We measured gaze data in 8 healthy young adults during walking in the RW and simulated locomotion in VR. Participants walked along a pre-defined path inside an office building, which included different terrains such as long corridors and flights of stairs. In VR, participants followed the same path in a detailed virtual reconstruction of the building. We devised a novel hybrid control strategy for movement in VR: participants did not actually translate: forward movements were controlled by a hand-held device, rotational movements were executed physically and transferred to the VR. We found significant effects of terrain type (flat corridor, staircase up, and staircase down) on gaze direction, on the spatial spread of gaze direction, and on the angular distribution of gaze-direction changes. The factor world (RW and VR) affected the angular distribution of gaze-direction changes, saccade frequency, and head-centered vertical gaze direction. The latter effect vanished when referencing gaze to a world-fixed coordinate system, and was likely due to specifics of headset placement, which cannot confound any other analyzed measure. Importantly, we did not observe a significant interaction between the factors world and terrain for any of the tested measures. This indicates that differences between terrain types are not modulated by the world. The overall dwell time on navigational markers did not differ between worlds. The similar dependence of gaze behavior on terrain in the RW and in VR indicates that our VR captures real-world constraints remarkably well. High-fidelity VR combined with naturalistic movement control therefore has the potential to narrow the gap between the experimental control of a lab and ecologically valid settings.
... By contrast, Draschkow and Võ (2016) had participants perform visual search in the physical world, in one of two conditions: In the "Find" condition, participants merely looked for objects without physically handling them (much like standard computer-based search tasks) and in the "Handle" condition, participants collected the objects as they progressed through the task. The results showed that search times (i.e., time to first fixation of the target) were not different between conditions, but that the memories retained for objects in the rooms followed different patterns for the Find and Handle conditions (see also Draschkow & Võ, 2017;Helbing et al., 2020). Taken together, this work suggests that what we find in well-controlled laboratory settings may or may not translate to search behavior in settings wherein other senses and/or tasks are employed. ...
In many important search tasks, observers must find what they are looking for using only visual information (e.g., X-ray baggage screening/medical screening). However, numerous other search tasks can only be effectively completed when the searcher uses their hands to find what they are looking for (e.g., “rummage” search). Unfortunately, it is not currently well understood how observers conduct such “interactive” searches, nor what the best strategies might be for doing so. Here, we first review the limited literature on interactive search. We then present a novel methodology for the study of interactive search that involves having observers seek out Lego targets in a cluttered tray of assorted bricks. In our Validation Task, we confirm the validity of this approach by demonstrating that it produces sensible patterns of diminishing returns in response time as targets are removed from the set, as well as hastened search times for larger targets. In our Experiment, we modify the approach, refining its systematicity and experimental control. We also build on prior work exploring strategy use in visual search by investigating the extent to which active and passive strategy use impacts performance in interactive search. In contrast to our prior findings in hybrid visual search (Madrid & Hout, 2019), our current findings suggest that in interactive search, an active search strategy can be superior to a passive one. We close by offering a conceptual model (the Interactive Multiple Decision Model; i-MDM) that explicates the steps involved in a search task of this nature, and we then provide suggestions for how to further refine the task to achieve higher internal validity and to delve deeper into questions of theoretical importance in the field of interactive search.
Objects are always situated within a scene context and have specific relationships with their environment. Understanding how scene context and the relationships between objects and their context affect object identification is crucial. Previous studies have indicated that scene-incongruent objects are detected faster than scene-congruent ones, and that “context cueing” can enhance object identification. However, no study has directly tested this relationship while considering the effects of bottom-up and top-down attention processes on object judgment. In our research, we explored the influence of context and its relationships by incorporating “context cueing” and categorizing these relationships into two types: semantic and syntactic, within an object judgment task. The behavioral results from Experiment 1 revealed that the recognition accuracy for syntactically incongruent objects was higher, with shorter response times. Eye-tracking data indicated that when semantic congruence was present, the first fixation duration on syntactically incongruent objects was shorter; conversely, when semantic incongruence was present, the first fixation duration on syntactically congruent objects was longer. In Experiment 2, which introduced context cueing, we found that the recognition accuracy for semantically congruent objects was higher, and they received more fixations. Notably, when syntactic incongruence was present, the first fixation duration on semantically congruent objects was longer. These findings suggest that under conditions without background cueing, syntactic processing has priority in scene processing. We interpret these results as evidence that top-down attention biases object processing, leading to reduced processing of scene-congruent objects compared to scene-incongruent ones. Thus, “context cueing” activates top-down attention, playing a pivotal role in object identification.
A current focus in sensorimotor research is the study of human perception and action in increasingly naturalistic tasks and visual environments. This is further enabled by the recent commercial success of virtual reality (VR) technology, which allows for highly realistic but well-controlled three-dimensional (3D) scenes. VR enables a multitude of different ways to interact with virtual objects, but only rarely are such interaction techniques evaluated and compared before being selected for a sensorimotor experiment. Here, we compare different response techniques for a memory-guided action task, in which participants indicated the position of a previously seen 3D object in a VR scene: pointing, using a virtual laser pointer of short or unlimited length, and placing, either the target object itself or a generic reference cube. Response techniques differed in availability of 3D object cues and requirement to physically move to the remembered object position by walking. Object placement was the most accurate but slowest due to repeated repositioning. When placing objects, participants tended to match the original object's orientation. In contrast, the laser pointer was fastest but least accurate, with the short pointer showing a good speed-accuracy compromise. Our findings can help researchers in selecting appropriate methods when studying naturalistic visuomotor behavior in virtual environments.
During visual search, we quickly learn to attend to an object’s likely location. Research has shown that this process can be guided by learning target locations based on consistent spatial contextual associations or statistical regularities. Here, we tested how these different types of learning aid the utilisation of established memories for different purposes. Participants learned contextual associations or statistical regularities that predicted target locations within different scenes. The consequences of this learning for subsequent performance were then evaluated on attention-orienting and memory-recall tasks. Participants demonstrated facilitated attention-orienting and recall performance based on both contextual associations and statistical regularities. Contextual associations facilitated attention orienting with a different time course compared to statistical regularities. Benefits to memory-recall performance depended on the alignment between the learned association or regularity and the recall demands. The distinct patterns of behavioural facilitation by contextual associations and statistical regularities show how different forms of long-term memory may influence neural information processing through different modulatory mechanisms.
Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans’ ability to navigate complex, multidimensional real-world environments so successfully.
Viewpoint effects on object recognition interact with object-scene consistency effects. While recognition of objects seen from “noncanonical” viewpoints (e.g., a cup from below) is typically impeded compared to processing of objects seen from canonical viewpoints (e.g., the string-side of a guitar), this effect is reduced by meaningful scene context information. In the present study we investigated if these findings established by using photographic images, generalize to strongly noncanonical orientations of three-dimensional (3D) models of objects. Using 3D models allowed us to probe a broad range of viewpoints and empirically establish viewpoints with very strong noncanonical and canonical orientations. In Experiment 1, we presented 3D models of objects from six different viewpoints (0°, 60°, 120°, 180° 240°, 300°) in color (1a) and grayscaled (1b) in a sequential matching task. Viewpoint had a significant effect on accuracy and response times. Based on the viewpoint effect in Experiments 1a and 1b, we could empirically determine the most canonical and noncanonical viewpoints from our set of viewpoints to use in Experiment 2. In Experiment 2, participants again performed a sequential matching task, however now the objects were paired with scene backgrounds which could be either consistent (e.g., a cup in the kitchen) or inconsistent (e.g., a guitar in the bathroom) to the object. Viewpoint interacted significantly with scene consistency in that object recognition was less affected by viewpoint when consistent scene information was provided, compared to inconsistent information. Our results show that scene context supports object recognition even when using extremely noncanonical orientations of depth rotated 3D objects. This supports the important role object-scene processing plays for object constancy especially under conditions of high uncertainty.
Viewpoint effects on object recognition interact with object-scene consistency effects. While recognition of objects seen from "accidental" viewpoints (e.g., a cup from below) is typically impeded compared to processing of objects seen from canonical viewpoints (e.g., the string-side of a guitar), this effect is reduced by meaningful scene context information. In the present study we investigated if these findings established by using photographic images, generalise to 3D models of objects. Using 3D models further allowed us to probe a broad range of viewpoints and empirically establish accidental and canonical viewpoints. In Experiment 1, we presented 3D models of objects from six different viewpoints (0°, 60°, 120°, 180° 240°, 300°) in colour (1a) and grayscaled (1b) in a sequential matching task. Viewpoint had a significant effect on accuracy and response times. Based on the performance in Experiments 1a and 1b, we determined canonical (0°-rotation) and non-canonical (120°-rotation) viewpoints for the stimuli. In Experiment 2, participants again performed a sequential matching task, however now the objects were paired with scene backgrounds which could be either consistent (e.g., a cup in the kitchen) or inconsistent (e.g., a guitar in the bathroom) to the object. Viewpoint interacted significantly with scene consistency in that object recognition was less affected by viewpoint when consistent scene information was provided, compared to inconsistent information. Our results show that viewpoint-dependence and scene context effects generalize to depth rotated 3D objects. This supports the important role object-scene processing plays for object constancy.
This systematic review study synthesizes research findings pertaining to the use of eye-tracking technology in immersive virtual reality (IVR) learning environments created by using head mounted displays. Research published between January 2012 and May 2022 has been explored and specific inclusion and exclusion criteria have been applied to select articles. Fifty articles were selected and then reviewed with respect to (1) devices employed in research; (2) learning domains; (3) the number and academic level of participants, data collection methods, and duration of IVR activities; and (4) indicators and themes of visual attention. The main findings from this review demonstrate Tobii and HTC Vive are the most popular tools in research on eye-tracking technology usage in IVR learning environments. What's more, cognitive science and educational technology were found to be the most frequent domains in research on eye-tracking technology usage in IVR. The number of participants varied across reviewed articles but tertiary education was the most common academic level of participating students. Scholars used questionnaires and tests frequently to collect their research data. Fixation duration was the most frequently used indicator. The most frequent themes in reviewed research were task performance, teaching and learning strategies, and learning tools. The study concludes with suggestions for future research.
Previous research raised the counterintuitive hypothesis that searching for multiple potential targets leads to increased incidental encoding of distractors. Are these previously reported findings due to increased visual working memory (VWM) engagement, or less precise target templates? In four experiments, we examined the effect of VWM load during visual search on incidental encoding of distractors. Consecutive target repetitions indirectly reduce template-related VWM demands but failed to reduce recognition for distractors relative to conditions where the targets were novel. Distractors that were subsequently recognized attracted longer cumulative dwell time, regardless of search condition. When placed in a dual-task situation where search was performed while holding a working memory load, recognition for distractors was marginally improved relative to a search task without additional VWM demands. We ruled out the possibility that the dual-task was not sufficiently difficult to trigger the scrutiny of distractors required for significant encoding benefits by showing a decrement to encoding when search time was limited. This suggests that widening the attentional set is the crucial factor in improved incidental encoding given that observers can assign differential status to various contents of VWM. Thus, utilizing VWM resources in general appears insufficient to meaningfully improve incidental memory.
Memories are about the past, but they serve the future. Memory research often emphasizes the former aspect: focusing on the functions that re-constitute (re-member) experience and elucidating the various types of memories and their interrelations, timescales, and neural bases. Here we highlight the prospective nature of memory in guiding selective attention, focusing on functions that use previous experience to anticipate the relevant events about to unfold-to "premember" experience. Memories of various types and timescales play a fundamental role in guiding perception and performance adaptively, proactively, and dynamically. Consonant with this perspective, memories are often recorded according to expected future demands. Using working memory as an example, we consider how mnemonic content is selected and represented for future use. This perspective moves away from the traditional representational account of memory toward a functional account in which forward-looking memory traces are informationally and computationally tuned for interacting with incoming sensory signals to guide adaptive behavior.
Understanding the content of memory is essential to teasing apart its underlying mechanisms. While recognition tests have commonly been used to probe memory, it is difficult to establish what specific content is driving performance. Here, we instead focus on free recall of real-world scenes, and quantify the content of memory using a drawing task. Participants studied 30 scenes and, after a distractor task, drew as many images in as much detail as possible from memory. The resulting memory-based drawings were scored by thousands of online observers, revealing numerous objects, few memory intrusions, and precise spatial information. Further, we find that visual saliency and meaning maps can explain aspects of memory performance and observe no relationship between recall and recognition for individual images. Our findings show that not only is it possible to quantify the content of memory during free recall, but those memories contain detailed representations of our visual experiences.
The arrangement of the contents of real-world scenes follows certain spatial rules that allow for extremely efficient visual exploration. What remains underexplored is the role different types of objects hold in a scene. In the current work, we seek to unveil an important building block of scenes-anchor objects. Anchors hold specific spatial predictions regarding the likely position of other objects in an environment. In a series of three eye tracking experiments we tested what role anchor objects occupy during visual search. In all of the experiments, participants searched through scenes for an object that was cued in the beginning of each trial. Critically, in half of the scenes a target relevant anchor was swapped for an irrelevant, albeit semantically consistent, object. We found that relevant anchor objects can guide visual search leading to faster reaction times, less scene coverage, and less time between fixating the anchor and the target. The choice of anchor objects was confirmed through an independent large image database, which allowed us to identify key attributes of anchors. Anchor objects seem to play a unique role in the spatial layout of scenes and need to be considered for understanding the efficiency of visual search in realistic stimuli.
Visual long-term memory capacity appears massive and detailed when probed explicitly. In the real world, however, memories are usually built from chance encounters. Therefore, we investigated the capacity and detail of incidental memory in a novel encoding task, instructing participants to detect visually distorted objects among intact objects. In a subsequent surprise recognition memory test, lures of a novel category, another exemplar, the same object in a different state, or exactly the same object were presented. Lure recognition performance was above chance, suggesting that incidental encoding resulted in reliable memory formation. Critically, performance for state lures was worse than for exemplars, which was driven by a greater similarity of state as opposed to exemplar foils to the original objects. Our results indicate that incidentally generated visual long-term memory representations of isolated objects are more limited in detail than recently suggested.
An important issue of psychological research is how experiments conducted in the laboratory or theories based on such experiments relate to human performance in daily life. Immersive virtual reality (VR) allows control over stimuli and conditions at increased ecological validity. The goal of the present study was to accomplish a transfer of traditional paradigms that assess attention and distraction to immersive VR. To further increase ecological validity we explored attentional effects with daily objects as stimuli instead of simple letters. Participants searched for a target among distractors on the countertop of a virtual kitchen. Target–distractor discriminability was varied and the displays were accompanied by a peripheral flanker that was congruent or incongruent to the target. Reaction time was slower when target–distractor discriminability was low and when flankers were incongruent. The results were replicated in a second experiment in which stimuli were presented on a computer screen in two dimensions. The study demonstrates the successful translation of traditional paradigms and manipulations into immersive VR and lays a foundation for future research on attention and distraction in VR. Further, we provide an outline for future studies that should use features of VR that are not available in traditional laboratory research.
Search is a central visual function. Most of what is known about search derives from experiments where subjects view 2D displays on computer monitors. In the natural world, however, search involves movement of the body in large-scale spatial contexts, and it is unclear how this might affect search strategies. In this experiment, we explore the nature of memory representations developed when searching in an immersive virtual environment. By manipulating target location, we demonstrate that search depends on episodic spatial memory as well as learnt spatial priors. Subjects rapidly learned the large-scale structure of the space, with shorter paths and less head rotation to find targets. These results suggest that spatial memory of the global structure allows a search strategy that involves efficient attention allocation based on the relevance of scene regions. Thus spatial memory may allow less energetically costly search strategies.
Predictions of environmental rules (here referred to as “scene grammar”) can come in different forms: seeing a toilet in a living room would violate semantic predictions, while finding a toilet brush next to the toothpaste would violate syntactic predictions. The existence of such predictions has usually been investigated by showing observers images containing such grammatical violations. Conversely, the generative process of creating an environment according to one’s scene grammar and its effects on behavior and memory has received little attention. In a virtual reality paradigm, we either instructed participants to arrange objects according to their scene grammar or against it. Subsequently, participants’ memory for the arrangements was probed using a surprise recall (Exp1), or repeated search (Exp2) task. As a result, participants’ construction behavior showed strategic use of larger, static objects to anchor the location of smaller objects which are generally the goals of everyday actions. Further analysis of this scene construction data revealed possible commonalities between the rules governing word usage in language and object usage in naturalistic environments. Taken together, we revealed some of the building blocks of scene grammar necessary for efficient behavior, which differentially influence how we interact with objects and what we remember about scenes.
People perform visual search tasks every day: from trivial tasks to emergencies. Classical research on visual search used artificial stimuli to identify factors that affect search times and accuracy. Recent studies have explored visual search in real scenes by simulating them on two-dimensional displays. The scientific community continues to use new technology to formulate better methods and practices. Virtual reality is a new technology that offers its users immersivity and elicits “real” responses. The purpose of this study is to compare search efficiencies in real scenes on 2-D displays and Virtual Reality. A visual search experiment measuring reaction times and accuracy was conducted to evaluate both methods. Results suggest that visual search in real scenes is significantly faster and more accurate in Virtual Reality than in 2-D Displays. These findings could open up new opportunities for visual search research on real scenes and real life scenarios.
One of the frequent questions by users of the mixed model function lmer of the lme4 package has been: How can I get p values for the F and t tests for objects returned by lmer? The lmerTest package extends the 'lmerMod' class of the lme4 package, by overloading the anova and summary functions by providing p values for tests for fixed effects. We have implemented the Satterthwaite's method for approximating degrees of freedom for the t and F tests. We have also implemented the construction of Type I - III ANOVA tables. Furthermore, one may also obtain the summary as well as the anova table using the Kenward-Roger approximation for denominator degrees of freedom (based on the KRmodcomp function from the pbkrtest package). Some other convenient mixed model analysis tools such as a step method, that performs backward elimination of nonsignificant effects - both random and fixed, calculation of population means and multiple comparison tests together with plot facilities are provided by the package as well.
The role of memory in guiding attention allocation in daily behaviors is not well understood. In experiments with two-dimensional (2D) images, there is mixed evidence about the importance of memory. Because the stimulus context in laboratory experiments and daily behaviors differs extensively, we investigated the role of memory in visual search, in both two-dimensional (2D) and three-dimensional (3D) environments. A 3D immersive virtual apartment composed of two rooms was created, and a parallel 2D visual search experiment composed of snapshots from the 3D environment was developed. Eye movements were tracked in both experiments. Repeated searches for geometric objects were performed to assess the role of spatial memory. Subsequently, subjects searched for realistic context objects to test for incidental learning. Our results show that subjects learned the room-target associations in 3D but less so in 2D. Gaze was increasingly restricted to relevant regions of the room with experience in both settings. Search for local contextual objects, however, was not facilitated by early experience. Incidental fixations to context objects do not necessarily benefit search performance. Together, these results demonstrate that memory for global aspects of the environment guides search by restricting allocation of attention to likely regions, whereas task relevance determines what is learned from the active search experience. Behaviors in 2D and 3D environments are comparable, although there is greater use of memory in 3D.
Looking for as well as actively manipulating objects that are relevant to ongoing behavioral goals are intricate parts of natural behavior. It is, however, not clear to what degree these two forms of interaction with our visual environment differ with regard to their memory representations. In a real-world paradigm, we investigated if physically engaging with objects as part of a search task influences identity and position memory differently for task-relevant versus irrelevant objects. Participants equipped with a mobile eye tracker either searched for cued objects without object interaction (Find condition) or actively collected the objects they found (Handle condition). In the following free-recall task, identity memory was assessed, demonstrating superior memory for relevant compared to irrelevant objects, but no difference between the Handle and Find conditions. Subsequently, location memory was inferred via times to first fixation in a final object search task. Active object manipulation and task-relevance interacted in that location memory for relevant objects was superior to irrelevant ones only in the Handle condition. Including previous object recall performance as a covariate in the linear mixed-model analysis of times to first fixation allowed us to explore the interaction between remembered/forgotten object identities and the execution of location memory. Identity memory performance predicted location memory in the Find but not the Handle condition, suggesting that active object handling leads to strong spatial representations independent of object identity memory. We argue that object handling facilitates the prioritization of relevant location information, but this might come at the cost of deprioritizing irrelevant information.
The analysis of experimental data with mixed-effects models requires
decisions about the specification of the appropriate random-effects structure.
Recently, Barr et al. (2013) recommended fitting 'maximal' models with all
possible random effect components included. Estimation of maximal models,
however, may not converge. We show that failure to converge typically is not
due to a suboptimal estimation algorithm, but is a consequence of attempting to
fit a model that is too complex to be properly supported by the data,
irrespective of whether estimation is based on maximum likelihood or on
Bayesian hierarchical modeling with uninformative or weakly informative priors.
Importantly, even under convergence, overparameterization may lead to
uninterpretable models. We provide diagnostic tools for detecting
overparameterization and guiding model simplification. Finally, we clarify that
the simulations on which Barr et al. base their recommendations are atypical
for real data. A detailed example is provided of how subject-related
attentional fluctuation across trials may further qualify statistical
inferences about fixed effects, and of how such nonlinear effects can be
accommodated within the mixed-effects modeling framework.
An essential first step in planning a confirmatory or a replication study is to determine the sample size necessary to draw statistically reliable inferences using power analysis. A key problem, however, is that what is available is the sample-size estimate of the effect size, and its use can lead to severely underpowered studies when the effect size is overestimated. As a potential remedy, we introduce safeguard power analysis, which uses the uncertainty in the estimate of the effect size to achieve a better likelihood of correctly identifying the population effect size. Using a lower-bound estimate of the effect size, in turn, allows researchers to calculate a sample size for a replication study that helps protect it from being underpowered. We show that in most common instances, compared with nominal power, safeguard power is higher whereas standard power is lower. We additionally recommend the use of safeguard power analysis to evaluate the strength of the evidence provided by the original study.
© The Author(s) 2014.
Holding recently experienced information in mind can help us achieve our current goals. However, such immediate and direct forms of guidance from working memory are less helpful over extended delays or when other related information in long-term memory is useful for reaching these goals. Here we show that information that was encoded in the past but is no longer present or relevant to the task also guides attention. We examined this by associating multiple unique features with novel shapes in visual long-term memory (VLTM), and subsequently testing how memories for these objects biased the deployment of attention. In Experiment 1, VLTM for associated features guided visual search for the shapes, even when these features had never been task-relevant. In Experiment 2, associated features captured attention when presented in isolation during a secondary task that was completely unrelated to the shapes. These findings suggest that long-term memory enables a durable and automatic type of memory-based attentional control. (PsycINFO Database Record
MEMORIZING CRITICAL OBJECTS AND THEIR LOCATIONS IS AN ESSENTIAL PART OF EVERYDAY LIFE IN THE PRESENT STUDY, INCIDENTAL ENCODING OF OBJECTS IN NATURALISTIC SCENES DURING SEARCH WAS COMPARED TO EXPLICIT MEMORIZATION OF THOSE SCENES: TO INVESTIGATE IF PRIOR KNOWLEDGE OF SCENE STRUCTURE INFLUENCES THESE TWO TYPES OF ENCODING DIFFERENTLY, WE USED MEANINGLESS ARRAYS OF OBJECTS AS WELL AS OBJECTS IN REAL-WORLD, SEMANTICALLY MEANINGFUL IMAGES SURPRISINGLY, WHEN PARTICIPANTS WERE ASKED TO RECALL SCENES, THEIR MEMORY PERFORMANCE WAS MARKEDLY BETTER FOR SEARCHED OBJECTS THAN FOR OBJECTS THEY HAD EXPLICITLY TRIED TO MEMORIZE, EVEN THOUGH PARTICIPANTS IN THE SEARCH CONDITION WERE NOT EXPLICITLY ASKED TO MEMORIZE OBJECTS: THIS FINDING HELD TRUE EVEN WHEN OBJECTS WERE OBSERVED FOR AN EQUAL AMOUNT OF TIME IN BOTH CONDITIONS: CRITICALLY, THE RECALL BENEFIT FOR SEARCHED OVER MEMORIZED OBJECTS IN SCENES WAS ELIMINATED WHEN OBJECTS WERE PRESENTED ON UNIFORM, NON-SCENE BACKGROUNDS RATHER THAN IN A FULL SCENE CONTEXT THUS, SCENE SEMANTICS NOT ONLY HELP US SEARCH FOR OBJECTS IN NATURALISTIC SCENES, BUT APPEAR TO PRODUCE A REPRESENTATION THAT SUPPORTS OUR MEMORY FOR THOSE OBJECTS BEYOND INTENTIONAL MEMORIZATION:
Maximum likelihood or restricted maximum likelihood (REML) estimates of the
parameters in linear mixed-effects models can be determined using the lmer
function in the lme4 package for R. As for most model-fitting functions in R,
the model is described in an lmer call by a formula, in this case including
both fixed- and random-effects terms. The formula and data together determine a
numerical representation of the model from which the profiled deviance or the
profiled REML criterion can be evaluated as a function of some of the model
parameters. The appropriate criterion is optimized, using one of the
constrained optimization functions in R, to provide the parameter estimates. We
describe the structure of the model, the steps in evaluating the profiled
deviance or REML criterion, and the structure of classes or types that
represents such a model. Sufficient detail is included to allow specialization
of these structures by users who wish to write functions to fit specialized
linear mixed models, such as models incorporating pedigrees or smoothing
splines, that are not easily expressible in the formula language used by lmer.
Visual memory has been demonstrated to play a role in both visual search and attentional prioritization in natural scenes. However, it has been studied predominantly in experimental paradigms using multiple two-dimensional images. Natural experience, however, entails prolonged immersion in a limited number of three-dimensional environments. The goal of the present experiment was to recreate circumstances comparable to natural visual experience in order to evaluate the role of scene memory in guiding eye movements in a natural environment. Subjects performed a continuous visual-search task within an immersive virtual-reality environment over three days. We found that, similar to two-dimensional contexts, viewers rapidly learn the location of objects in the environment over time, and use spatial memory to guide search. Incidental fixations did not provide obvious benefit to subsequent search, suggesting that semantic contextual cues may often be just as efficient, or that many incidentally fixated items are not held in memory in the absence of a specific task. On the third day of the experience in the environment, previous search items changed in color. These items were fixated upon with increased probability relative to control objects, suggesting that memory-guided prioritization (or Surprise) may be a robust mechanisms for attracting gaze to novel features of natural environments, in addition to task factors and simple spatial saliency.
Selecting and remembering visual information is an active and competitive process. In natural environments, representations are tightly coupled to task. Objects that are task-relevant are remembered better due to a combination of increased selection for fixation and strategic control of encoding and/or retaining viewed information. However, it is not understood how physically manipulating objects when performing a natural task influences priorities for selection and memory. In this study, we compare priorities for selection and memory when actively engaged in a natural task with first-person observation of the same object manipulations. Results suggest that active manipulation of a task-relevant object results in a specific prioritization for object position information compared with other properties and compared with action observation of the same manipulations. Experiment 2 confirms that this spatial prioritization is likely to arise from manipulation rather than differences in spatial representation in real environments and the movies used for action observation. Thus, our findings imply that physical manipulation of task relevant objects results in a specific prioritization of spatial information about task-relevant objects, possibly coupled with strategic de-prioritization of colour memory for irrelevant objects.
In sentence processing, semantic and syntactic violations elicit differential brain responses observable in event-related potentials: An N400 signals semantic violations, whereas a P600 marks inconsistent syntactic structure. Does the brain register similar distinctions in scene perception? To address this question, we presented participants with semantic inconsistencies, in which an object was incongruent with a scene's meaning, and syntactic inconsistencies, in which an object violated structural rules. We found a clear dissociation between semantic and syntactic processing: Semantic inconsistencies produced negative deflections in the N300-N400 time window, whereas mild syntactic inconsistencies elicited a late positivity resembling the P600 found for syntactic inconsistencies in sentence processing. Extreme syntactic violations, such as a hovering beer bottle defying gravity, were associated with earlier perceptual processing difficulties reflected in the N300 response, but failed to produce a P600 effect. We therefore conclude that different neural populations are active during semantic and syntactic processing of scenes, and that syntactically impossible object placements are processed in a categorically different manner than are syntactically resolvable object misplacements.
A popular model of visual perception states that coarse information (carried by low spatial frequencies) along the dorsal stream is rapidly transmitted to prefrontal and medial temporal areas, activating contextual information from memory, which can in turn constrain detailed input carried by high spatial frequencies arriving at a slower rate along the ventral visual stream, thus facilitating the processing of ambiguous visual stimuli. We were interested in testing whether this model contributes to memory-guided orienting of attention. In particular, we asked whether global, low-spatial frequency (LSF) inputs play a dominant role in triggering contextual memories in order to facilitate the processing of the upcoming target stimulus. We explored this question over four experiments. The first experiment replicated the LSF advantage reported in perceptual discrimination tasks by showing that participants were faster and more accurate at matching a low spatial frequency version of a scene, compared to a high spatial frequency version, to its original counterpart in a forced-choice task. The subsequent three experiments tested the relative contributions of low versus high spatial frequencies during memory-guided covert spatial attention orienting tasks. Replicating the effects of memory-guided attention, pre-exposure to scenes associated with specific spatial memories for target locations (memory cues) led to higher perceptual discrimination and faster response times to identify targets embedded in the scenes. However, either high or low spatial frequency cues were equally effective; LSF signals did not selectively or preferentially contribute to the memory-driven attention benefits to performance. Our results challenge a generalized model that LSFs activate contextual memories, which in turn bias attention and facilitate perception.
A powerful way of improving one's memory for material is to be tested on that material. Tests enhance later retention more than additional study of the material, even when tests are given without feedback. This surpris- ing phenomenon is called the testing effect, and although it has been studied by cognitive psychologists sporadically over the years, today there is a renewed effort to learn why testing is effective and to apply testing in educational settings. In this article, we selectively review laboratory studies that reveal the power of testing in improving re- tention and then turn to studies that demonstrate the basic effects in educational settings. We also consider the related concepts of dynamic testing and formative assess- ment as other means of using tests to improve learning. Finally, we consider some negative consequences of testing that may occur in certain circumstances, though these negative effects are often small and do not cancel out the large positive effects of testing. Frequent testing in the classroom may boost educational achievement at all levels of education. In contemporary educational circles, the concept of testing has a dubious reputation, and many educators believe that testing is overemphasized in today's schools. By ''testing,'' most com- mentators mean using standardized tests to assess students. During the 20th century, the educational testing movement produced numerous assessment devices used throughout edu- cation systems in most countries, from prekindergarten through graduate school. However, in this review, we discuss primarily the kind of testing that occurs in classrooms or that students engage in while studying (self-testing). Some educators argue
Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structure justified by the design. The generalization performance of LMEMs including data-driven random effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separate F1 and F2 tests, and in many cases, even worse than F1 alone. Maximal LMEMs should be the ‘gold standard’ for confirmatory hypothesis testing in psycholinguistics and beyond.
When looking at a scene, observers feel that they see its entire structure in great detail and can immediately notice any changes in it. However, when brief blank fields are placed between alternating displays of an original and a modified scene, a striking failure of perception is induced: identification of changes becomes extremely difficult, even when changes are large and made repeatedly. Identification is much faster when a verbal cue is provided, showing that poor visibility is not the cause of this difficulty. Identification is also faster for objects mentioned in brief verbal descriptions of the scene. These results support the idea that observers never form a complete, detailed representation of their surroundings. In addition, results also indicate that attention is required to perceive change, and that in the absence of localized motion signals it is guided on the basis of high-level interest.
To see or not to see: The need for attention to perceive changes in scenes. Available from: https://www.researchgate.net/publication/236170014_To_see_or_not_to_see_The_need_for_attention_to_perceive_changes_in_scenes [accessed Jun 15, 2017].
Recent results from Võ and Wolfe (2012b) suggest that the application of memory to visual search may be task specific: Previous experience searching for an object facilitated later search for that object, but object information acquired during a different task did not appear to transfer to search. The latter inference depended on evidence that a preview task did not improve later search, but Võ and Wolfe used a relatively insensitive, between-subjects design. Here, we replicated the Võ and Wolfe study using a within-subject manipulation of scene preview. A preview session (focused either on object location memory or on the assessment of object semantics) reliably facilitated later search. In addition, information acquired from distractors in a scene-facilitated search when the distractor later became the target. Instead of being strongly constrained by task, visual memory is applied flexibly to guide attention and gaze during visual search.
Showed 2560 photographic stimuli for 10 sec. each to 21 undergraduates, and tested their recognition memory using a 2-alternative forced-choice task. Performance exceeded 90% retention, even when up to 3 days elapsed between learning and testing. Variants showed that the presentation time could be reduced to 1 sec/picture without seriously affecting performance, and that the stimuli could be reversed in orientation in the test situation without impairing recognition performance appreciably. The orientation of the stimuli could also be learned, although not as well as the identity of the pictures. Results indicate the vast memory for pictures possessed by human beings and emphasize the need to determine mechanisms by which this is accomplished. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
In everyday situations, we often rely on our memories to find what we are looking for in our cluttered environment. Recently, we developed a new experimental paradigm to investigate how long-term memory (LTM) can guide attention and showed how the pre-exposure to a complex scene in which a target location had been learned facilitated the detection of the transient appearance of the target at the remembered location [Summerfield, J. J., Rao, A., Garside, N., & Nobre, A. C. Biasing perception by spatial long-term memory. The Journal of Neuroscience, 31, 14952–14960, 2011; Summerfield, J. J., Lepsien, J., Gitelman, D. R., Mesulam, M. M., & Nobre, A. C. Orienting attention based on long-term memory experience. Neuron, 49, 905–916, 2006]. This study extends these findings by investigating whether and how LTM can enhance perceptual sensitivity to identify targets occurring within their complex scene context. Behavioral measures showed superior perceptual sensitivity (d′) for targets located in remembered spatial contexts. We used the N2pc ERP to test whether LTM modulated the process of selecting the target from its scene context. Surprisingly, in contrast to effects of visual spatial cues or implicit contextual cueing, LTM for target locations significantly attenuated the N2pc potential. We propose that the mechanism by which these explicitly available LTMs facilitate perceptual identification of targets may differ from mechanisms triggered by other types of top–down sources of information.
Three experiments were conducted to investigate the existence of incidentally acquired, long-term, detailed visual memory for objects embedded in previously viewed scenes. Participants performed intentional memorization and incidental visual search learning tasks while viewing photographs of real-world scenes. A visual memory test for previously viewed objects from these scenes then followed. Participants were not aware that they would be tested on the scenes following incidental learning in the visual search task. In two types of memory tests for visually specific object information (token discrimination and mirror-image dis-crimination), performance following both the memorization and visual search conditions was reliably above chance. These results indicate that recent demon-strations of good visual memory during scene viewing are not due to intentional scene memorization. Instead, long-term visual representations are incidentally generated as a natural product of scene perception. What is the nature of the representation that is created during ongoing, natural scene perception? Intuitively, it seems that the visual system generates a com-plete and highly detailed model of the external world. The perceptual experience of a stable and detailed visual world has led many vision researchers in the past to conclude that the visual representation formed for a scene is veridical and complete (McConkie & Rayner, 1976; Neisser, 1967). Such a detailed visual This research was submitted in partial fulfilment of the requirements for the Masters of Arts degree by Monica Castelhano, who was supported by the MSU graduate school on behalf of the NSF IGERT program in Cognitive Science. The authors wish to thank Jeremy Athy, for his help with data collection, and the members of the masters committee, Erik Altmann, Thomas Carr, and Fred Dyer, for their guidance and comments.
Past experience provides a rich source of predictive information about the world that could be used to guide and optimize ongoing perception. However, the neural mechanisms that integrate information coded in long-term memory (LTM) with ongoing perceptual processing remain unknown. Here, we explore how the contents of LTM optimize perception by modulating anticipatory brain states. By using a paradigm that integrates LTM and attentional orienting, we first demonstrate that the contents of LTM sharpen perceptual sensitivity for targets presented at memory-predicted spatial locations. Next, we examine oscillations in EEG to show that memory-guided attention is associated with spatially specific desynchronization of alpha-band activity over visual cortex. Additionally, we use functional MRI to confirm that target-predictive spatial information stored in LTM triggers spatiotopic modulation of preparatory activity in extrastriate visual cortex. Finally, functional MRI results also implicate an integrated cortical network, including the hippocampus and a dorsal frontoparietal circuit, as a likely candidate for organizing preparatory states in visual cortex according to the contents of LTM.
Human perception is highly flexible and adaptive. Selective processing is tuned dynamically according to current task goals and expectations to optimize behavior. Arguably, the major source of our expectations about events yet to unfold is our past experience; however, the ability of long-term memories to bias early perceptual analysis has remained untested. We used a noninvasive method with high temporal resolution to record neural activity while human participants detected visual targets that appeared at remembered versus novel locations within naturalistic visual scenes. Upon viewing a familiar scene, spatial memories changed oscillatory brain activity in anticipation of the target location. Memory also enhanced neural activity during early stages of visual analysis of the target and improved behavioral performance. Both measures correlated with subsequent target-detection performance. We therefore demonstrated that memory can directly enhance perceptual functions in the human brain.
Linear mixed models (LMMs) provide a still underused methodological perspective on combining experimental and individual-differences research. Here we illustrate this approach with two-rectangle cueing in visual attention (Egly et al., 1994). We replicated previous experimental cue-validity effects relating to a spatial shift of attention within an object (spatial effect), to attention switch between objects (object effect), and to the attraction of attention toward the display centroid (attraction effect), also taking into account the design-inherent imbalance of valid and other trials. We simultaneously estimated variance/covariance components of subject-related random effects for these spatial, object, and attraction effects in addition to their mean reaction times (RTs). The spatial effect showed a strong positive correlation with mean RT and a strong negative correlation with the attraction effect. The analysis of individual differences suggests that slow subjects engage attention more strongly at the cued location than fast subjects. We compare this joint LMM analysis of experimental effects and associated subject-related variances and correlations with two frequently used alternative statistical procedures.
One might assume that familiarity with a scene or previous encounters with objects embedded in a scene would benefit subsequent search for those items. However, in a series of experiments we show that this is not the case: When participants were asked to subsequently search for multiple objects in the same scene, search performance remained essentially unchanged over the course of searches despite increasing scene familiarity. Similarly, looking at target objects during previews, which included letter search, 30 seconds of free viewing, or even 30 seconds of memorizing a scene, also did not benefit search for the same objects later on. However, when the same object was searched for again memory for the previous search was capable of producing very substantial speeding of search despite many different intervening searches. This was especially the case when the previous search engagement had been active rather than supported by a cue. While these search benefits speak to the strength of memory-guided search when the same search target is repeated, the lack of memory guidance during initial object searches-despite previous encounters with the target objects-demonstrates the dominance of guidance by generic scene knowledge in real-world search.
Despite many recent technical advances, the human efficacy of naturalistic scene processing is still unparalleled. What guides attention in real world environments? How does scene context affect object search and classification? And how are the many predictions we have with regards to our visual environment structured? Here, we review the latest findings that speak to these questions, ranging from traditional psychophysics, eye movement, and neurocognitive studies in the lab to experiments in virtual reality (VR) and the real world. The growing interest and grasp of a scene's inner workings are enriching our understanding of the efficiency of scene perception and could inspire new technological and clinical advances.
A long line of research has shown that vision and memory are closely linked, such that particular eye movement behaviour aids memory performance. In two experiments, we ask whether the positive influence of eye movements on memory is primarily a result of overt visual exploration during the encoding or the recognition phase. Experiment 1 allowed participants to free-view images of scenes, followed by a new-old recognition memory task. Exploratory analyses found that eye movements during study were predictive of subsequent memory performance. Importantly, intrinsic image memorability does not explain this finding. Eye movements during test were only predictive of memory within the first 600 ms of the trial. To examine whether this relationship between eye movements and memory is causal, Experiment 2 manipulated participants' ability to make eye movements during either study or test in a new-old recognition task. Participants were either encouraged to freely explore the scene in both the study and test phases, or had to refrain from making eye movements in either the test phase, the study phase, or both. We found that hit rate was significantly higher when participants moved their eyes during the study phase, regardless of what they did in the test phase. False alarm rate, on the other hand, was affected only by eye movements during the test phase: it decreased when participants were encouraged to explore the scene. Taken together, these results reveal a dissociation of the role of eye movements during the encoding and recognition of scenes. Eye movements during study are instrumental in forming memories, and eye movements during recognition support the judgment of memory veracity.
There are ever more compelling tools available for neuroscience research, ranging from selective genetic targeting to optogenetic circuit control to mapping whole connectomes. These approaches are coupled with a deep-seated, often tacit, belief in the reductionist program for understanding the link between the brain and behavior. The aim of this program is causal explanation through neural manipulations that allow testing of necessity and sufficiency claims. We argue, however, that another equally important approach seeks an alternative form of understanding through careful theoretical and experimental decomposition of behavior. Specifically, the detailed analysis of tasks and of the behavior they elicit is best suited for discovering component processes and their underlying algorithms. In most cases, we argue that study of the neural implementation of behavior is best investigated after such behavioral work. Thus, we advocate a more pluralistic notion of neuroscience when it comes to the brain-behavior relationship: behavioral work provides understanding, whereas neural interventions test causality.
To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties.
In the analysis of data it is often assumed that observations y1, y2, …, yn are independently normally distributed with constant variance and with expectations specified by a model linear in a set of parameters θ. In this paper we make the less restrictive assumption that such a normal, homoscedastic, linear model is appropriate after some suitable transformation has been applied to the y's. Inferences about the transformation and about the parameters of the linear model are made by computing the likelihood function and the relevant posterior distribution. The contributions of normality, homoscedasticity and additivity to the transformation are separated. The relation of the present methods to earlier procedures for finding transformations is discussed. The methods are illustrated with examples.
Previous work has shown that recall of objects that are incidentally encountered as targets in visual search is better than recall of objects that have been intentionally memorized (Draschkow, Wolfe, & Võ, 2014). However, this counter-intuitive result is not seen when these tasks are performed with non-scene stimuli. The goal of the current paper is to determine what features of search in a scene contribute to higher recall rates when compared to a memorization task. In each of four experiments, we compare the free recall rate for target objects following a search to the rate following a memorization task. Across the experiments, the stimuli include progressively more scene-related information. Experiment 1 provides the spatial relations between objects. Experiment 2 adds relative size and depth of objects. Experiments 3 and 4 include scene layout and semantic information. We find that search leads to better recall than explicit memorization in cases where scene layout and semantic information are present, as long as the participant has ample time (2500ms) to integrate this information with knowledge about the target object (Exp. 4). These results suggest that the integration of scene and target information not only leads to more efficient search, but can also contribute to stronger memory representations than intentional memorization.
The manner in which we sample visual information from the world is constrained by the spatial and temporal sampling limits of the human eye. High acuity vision is restricted to the small central foveal region of the retina, which is limited to just a few degrees of visual angle in extent. Moreover, visual sampling is effectively limited to when the retinal image is relatively stabilised for periods of fixation (Erdmann and Dodge 1898)
, which last on average around 200–400 ms when viewing text, scenes or real environments (Land and Tatler 2009; Rayner 1998)
. It is clear from these severe spatiotemporal constraints on visual sampling that high acuity vision is a scarce resource and, like any scarce resource, it must be distributed carefully and appropriately for the current situation.
Many daily activities involve looking for something. The ease with which these searches are performed often allows one to forget that searching represents complex interactions between visual attention and memory. Although a clear understanding exists of how search efficiency will be influenced by visual features of targets and their surrounding distractors or by the number of items in the display, the role of memory in search is less well understood. Contextual cueing studies have shown that implicit memory for repeated item configurations can facilitate search in artificial displays. When searching more naturalistic environments, other forms of memory come into play. For instance, semantic memory provides useful information about which objects are typically found where within a scene, and episodic scene memory provides information about where a particular object was seen the last time a particular scene was viewed. In this paper, we will review work on these topics, with special emphasis on the role of memory in guiding search in organized, real-world scenes.
© 2015 New York Academy of Sciences.
Certain results obtained in an experiment 'On the Analysis of the Memory Function in Orthography' conducted in the psychological laboratory of the University of Illinois in the year 1907-08, led to the conclusion that the opportunity for recall, during or immediately after the learning process, was of great benefit to the individual. It has been the aim of the present experiment to determine more carefully the nature of the influence of this recall and the conditions under which it could be used most favorably. In other words, to determine, when a definite length of time is given in which to learn a given amount of material, whether it is of the greater advantage to spend all the time in actual perception of the material, or part of the time in perception and part in recall; and also whether the recall should be interspersed with the perception or should follow it immediately or after an interval. We have also attempted to make some analysis and explanation of the factors which are of influence in this recall period and particularly to determine the effect of localization. The work already done along the line of economy of learning has shown that the relative value of many of the so-called methods of learning depend in a large measure on the memory type of the individual who is to use the method. A condition which would be of great advantage to a visually-minded individual might prove positively distracting to one of the auditory-motor type and again might have no appreciable effect on the individual of a mixed type. In view of this, in the present experiment care was taken to determine from introspective analysis the type of imagery of the subjects and in general the mental processes which they went through in learning the material presented to them.
This book encompasses and weaves together the common threads of the four major topics that comprise the core of false memory research: theories of false memory, adult experimental psychology of false memory, false memory in legal contexts, and false memory in psychotherapy. By integrating material on all four of these topics, the book provides a comprehensive picture of our current understanding of human false memory.
The visual environment is extremely rich and complex, producing information overload for the visual system. But the envi- ronment also embodies structure in the form of redundancies and reg- ularities that may serve to reduce complexity. How do perceivers internalize this complex informational structure? We present new evi- dence of visual learning that illustrates how observers learn how objects and events covary in the visual world. This information serves to guide visual processes such as object recognition and search. Our first experiment demonstrates that search and object recognition are facilitated by learned associations (covariation) between novel visual shapes. Our second experiment shows that regularities in dynamic visual environments can also be learned to guide search behavior. In both experiments, learning occurred incidentally and the memory rep- resentations were implicit. These experiments show how top-down visual knowledge, acquired through implicit learning, constrains what to expect and guides where to attend and look.
The representations that are formed as we view real environments are still not well characterized. We studied the influence of task instructions on memory performance and fixation allocation in a real-world setting, in which participants were free to move around. Object memories were found to be task sensitive, as was the allocation of foveal vision. However, changes in the number of fixations directed at objects could not fully explain the changes in object memory performance that were found between task instruction conditions. Our data suggest that the manner in which information is extracted and retained from fixations varies with the instructions given to participants, with strategic prioritization of information retention from fixations made to task-relevant objects and strategic deprioritization of information retention from fixations directed to task-irrelevant objects.
It seems intuitive to think that previous exposure or interaction with an environment should make it easier to search through it and, no doubt, this is true in many real-world situations. However, in a recent study, we demonstrated that previous exposure to a scene does not necessarily speed search within that scene. For instance, when observers per- formed as many as 15 searches for different objects in the same, unchanging scene, the speed of search did not decrease much over the course of these multiple searches (Võ & Wolfe, 2012). Only when observers were asked to search for the same object again did search become considerably faster. We argued that our naturalistic scenes provided such strong ‘‘semantic’’ guidance—e.g., knowing that a faucet is usually located near a sink—that guidance by incidental episodic memory—having seen that faucet previously—was ren- dered less useful. Here, we directly manipulated the availability of semantic information provided by a scene. By monitoring observers’ eye movements, we found a tight coupling of semantic and episodic memory guidance: Decreasing the availability of semantic infor- mation increases the use of episodic memory to guide search. These findings have broad implications regarding the use of memory during search in general and particularly during search in naturalistic scenes.
Attention is strongly influenced by both external stimuli and internal goals. However, this useful dichotomy does not readily capture the ubiquitous and often automatic contribution of past experience stored in memory. We review recent evidence about how multiple memory systems control attention, consider how such interactions are manifested in the brain, and highlight how this framework for 'memory-guided attention' might help systematize previous findings and guide future research.
Prior research into the impact of encoding tasks on visual memory (Castelhano & Henderson, 2005) indicated that incidental and intentional encoding tasks led to similar memory performance. The current study investigated whether different encoding tasks impacted visual memories equally for all types of objects in a conjunction search (e.g., targets, colour distractors, object category distractors, or distractors unrelated to the target). In sequences of pictures, participants searched for prespecified targets (e.g., green apple; Experiment 1), memorized all objects (Experiment 2), searched for specified targets while memorizing all objects (Experiment 3), searched for postidentified targets (Experiment 4), or memorized all objects with one object prespecified (Experiment 5). Encoding task significantly improved visual memory for targets and led to worse memory for unrelated distractors, but did not influence visual memory of distractors that were related to the target's colour or object category. The differential influence of encoding task indicates that the relative importance of the object both positively and negatively influences the memory retained.
Research on contextual cueing has demonstrated that with simple arrays of letters and shapes, search for a target increases in efficiency as associations between a search target and its surrounding visual context are learned. We investigated whether the visual context afforded by repeated exposure to real-world scenes can also guide attention when the relationship between the scene and a target position is arbitrary. Observers searched for and identified a target letter embedded in photographs of real-world scenes. Although search time within novel scenes was consistent across trials, search time within repeated scenes decreased across repetitions. Unlike previous demonstrations of contextual cueing, however, memory for scene-target covariation was explicit. In subsequent memory tests, observers recognized repeated contexts more often than those that were presented once and displayed superior recall of target position within the repeated scenes. In addition, repetition of inverted scenes, which made the scene more difficult to identify, produced a markedly reduced rate of learning, suggesting semantic information concerning object and scene identity are used to guide attention.
This paper provides an introduction to mixed-effects models for the analysis of repeated measurement data with subjects and items as crossed random effects. A worked-out example of how to use recent software for mixed-effects modeling is provided. Simulation studies illustrate the advantages offered by mixed-effects analyses compared to traditional analyses based on quasi-F tests, by-subjects analyses, combined by-subjects and by-items analyses, and random regression. Applications and possibilities across a range of domains of inquiry are discussed.
How do people distribute their visual attention in the natural environment? We and our colleagues have usually addressed this question by showing pictures, photographs or videos of natural scenes under controlled conditions and recording participants' eye movements as they view them. In the present study, we investigated whether people distribute their gaze in the same way when they are immersed and moving in the world compared to when they view video clips taken from the perspective of a walker. Participants wore a mobile eye tracker while walking to buy a coffee, a trip that required a short walk outdoors through the university campus. They subsequently watched first-person videos of the walk in the lab. Our results focused on where people directed their eyes and their head, what objects were gazed at and when attention-grabbing items were selected. Eye movements were more centralised in the real world, and locations around the horizon were selected with head movements. Other pedestrians, the path, and objects in the distance were looked at often in both the lab and the real world. However, there were some subtle differences in how and when these items were selected. For example, pedestrians close to the walker were fixated more often when viewed on video than in the real world. These results provide a crucial test of the relationship between real behaviour and eye movements measured in the lab.