No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Embodied Sentence Comprehension
Abstract
There are two views of cognition in general and of language comprehension in particular. According to the traditional view (Chomsky, 1957; Fodor, 1983; Pylyshyn, 1986), the human mind is like a bricklayer, or maybe a contractor, who puts together bricks to build structures. The malleable clay of perception is converted to the neat mental bricks we call words and propositions, units of meaning, which can be used in a variety of structures. But whereas bricklayers and contractors presumably know how bricks are made, cognitive scientists and neuroscientists have no idea how the brain converts perceptual input to abstract lexical and propositional representations – it is simply taken as a given that this occurs (Barsalou, 1999). According to an alternative and emerging view, there are no clear demarcations between perception, action, and cognition. Interactions with the world leave traces of experience in the brain. These traces are (partially) retrieved and used in the mental simulations that make up cognition. Crucially, these traces bear a resemblance to the perceptual/action processes that generated them (Barsalou, 1999) and are highly malleable. Words and grammar are viewed as a set of cues that activate and combine experiential traces in the mental simulation of the described events (Zwaan, 2004). The main purpose of this chapter is to provide a discussion of this view of language comprehension. To set the stage for this discussion we first analyze a series of linguistic examples that present increasingly larger problems for the traditional view.
... Some researchers have proposed that language may play an important role in the generation of complex perceptual representations (e.g. Borghi & Binkofski, 2014;Carruthers, 1996;Gomila, Travieso, & Lobo, 2012;Lupyan & Bergen, 2016;Paivio, 1986;2007;Spelke, 2003;Zwaan & Madden, 2005). For example, Zwaan and Madden (2005) view linguistic sequences as a series of cues that can activate and combine previously stored perceptual features. ...
... Borghi & Binkofski, 2014;Carruthers, 1996;Gomila, Travieso, & Lobo, 2012;Lupyan & Bergen, 2016;Paivio, 1986;2007;Spelke, 2003;Zwaan & Madden, 2005). For example, Zwaan and Madden (2005) view linguistic sequences as a series of cues that can activate and combine previously stored perceptual features. If this is the case, then the temporal concatenation of a sequence of cues may play an important role in the activation of feature combinations. ...
... Although our original rationale was based on the idea that language may play a key role in the incremental construction of perceptual representations (cf. Zwaan & Madden, 2005), Experiments 2a-b and 3a-b did not explicitly manipulate linguistic structure in the verbal cues (i.e., they manipulated word pairs that could be interpreted as having linguistic structure, but could also be interpreted as merely disconnected words). In Experiments 4a-b we tested whether the linguistic structure of verbal cues influences the way feature combinations are represented in a visualization task. ...
Although many studies have investigated the activation of perceptual representations during language comprehension, to our knowledge only one previous study has directly tested how perceptual features are combined into representations during comprehension. In their classic study, Potter and Faulconer [(1979). Understanding noun phrases. Journal of Verbal Learning and Verbal Behavior, 18, 509–521.] investigated the perceptual representation of adjective-noun combinations. However, their non-orthogonal design did not allow the differentiation between conjunctive vs. disjunctive representations. Using randomized orthogonal designs, we observe evidence for disjunctive perceptual representations when participants represent feature combinations simultaneously (in several experiments; N = 469), and we observe evidence for conjunctive perceptual representations when participants represent feature combinations sequentially (In several experiments; N = 628). Our findings show that the generation of conjunctive representations during comprehension depends on the concatenation of linguistic cues, and thus suggest the construction of elaborate perceptual representations may critically depend on language.
... According to the experiential trace model (Zwaan and Madden, 2005), this grounding of word meanings is established through systematic and repeated co-occurrence between sensorimotor and linguistic experience (for example, hearing the word balloon while seeing a balloon in the sky), which establishes associative relations between the two (see also Hauk et al., 2004). The word balloon then serves as a cue to re-activate this sensorimotor experience, enabling the recipient to comprehend its meaning. ...
... This congruency effects for vertical hand movements during the processing of vertically related words have been well-replicated and firmly established in a series of studies (Dudschig et al., 2014a;; Thornton et al., 2013). Even more direct evidence for the predictions of the experiential trace model (Zwaan and Madden, 2005) was provided by Öttl et al. (2017), who observed this automatic congruency effect even for novel words, after participants learned them as labels for novel object referents presented in a specific vertical location. ...
... At this point, one can argue that a major difference still lies in the amount and quality of learning experience participants had with the novel compounds, which in the previous studies by Günther et al. (2018) was rather limited. We know that connections between experiential traces are assumed to be formed through repeated experience (Hauk et al., 2004;Zwaan and Madden, 2005), as is the case for associative learning in general. Furthermore, since learning was always immediately followed by the test phase, there was no possibility to consolidate the learned associations in memory. ...
While a number of studies have repeatedly demonstrated an automatic activation of sensorimotor experience during language processing in the form of action-congruency effects, as predicted by theories of grounded cognition, more recent research has not found these effects for words that were just learned from linguistic input alone, without sensorimotor experience with their referents. In the present study, we investigate whether this absence of effects can be attributed to a lack of repeated experience and consolidation of the associations between words and sensorimotor experience in memory. To address these issues, we conducted four experiments in which (1 and 2) participants engaged in two separate learning phases in which they learned novel words from language alone, with an intervening period of memory-consolidating sleep, and (3 and 4) we employed familiar words whose referents speakers have no direct experience with (such as plankton ). However, we again did not observe action-congruency effects in subsequent test phases in any of the experiments. This indicates that direct sensorimotor experience with word referents is a necessary requirement for automatic sensorimotor activation during word processing.
... According to proposals in the field of grounded cognition, language comprehension involves the reactivation of the sensorimotor (i.e., perceptual and motor) states acquired during everyday experience (Barsalou, 2008;Fischer, 2012;Zwaan & Madden, 2005). In this perspective, sensorimotor processing would play a crucial role as a constitutive part of the meaning-making stream. ...
... Contextual information, hence, has a decisive impact on the training of such text-based models. It is also worth noting that such a computational system can find its cognitive underpinnings in learning theories of similarity-based generalization in language (Anderson & Milson, 1989;Rescorla & Wagner, 1972;see Landauer & Dumais, 1997;Mandera et al., 2017;Hollis, 2017, for detailed arguments) as well as within grounded-framework approaches that centrally rely on the notion of co-occurrence between and within units of linguistic and sensorimotor experience as a meaning-forming mechanism (Zwaan & Madden, 2005; see Günther et al., 2019). ...
... In addition to on-line processing, context also plays a vital role in the acquisition of concepts and word meanings (Zwaan, 2014;Zwaan & Madden, 2005). This aligns perfectly with our modelling approach, where contextual information has a decisive impact on the training of both the language-and the vision-based models used here. ...
In their strongest formulation, theories of grounded cognition claim that concepts are made up of sensorimotor information. Following such equivalence, perceptual properties of objects should consistently influence processing, even in purely linguistic tasks, where perceptual information is neither solicited nor required. Previous studies have tested this prediction in semantic priming tasks, but they have not observed perceptual influences on participants’ performances. However, those findings suffer from critical shortcomings, which may have prevented potential visually grounded/perceptual effects from being detected. Here, we investigate this topic by applying an innovative method expected to increase the sensitivity in detecting such perceptual effects. Specifically, we adopt an objective, data-driven, computational approach to independently quantify vision-based and language-based similarities for prime-target pairs on a continuous scale. We test whether these measures predict behavioural performance in a semantic priming mega-study with various experimental settings. Vision-based similarity is found to facilitate performance, but a dissociation between vision-based and language-based effects was also observed. Thus, in line with theories of grounded cognition, perceptual properties can facilitate word processing even in purely linguistic tasks, but the behavioural dissociation at the same time challenges strong claims of sensorimotor and conceptual equivalence.
... This consideration lies at the core of theories of embodied cognition (e.g. Barsalou, 1999;Barsalou, Santos, Simmons, & Wilson, 2008;Fischer, 2012;Glenberg & Robertson, 2000;Glenberg, 2015;Zwaan & Madden, 2005), which have taken a central place in the debate on concept acquisition and representation. In this view, concepts such as DOG are formed through the interplay between linguistic experience (hearing or reading the word dog) as well as sensorimotor experience (seeing a dog, or hearing it bark) (Zwaan & Madden, 2005). ...
... Barsalou, 1999;Barsalou, Santos, Simmons, & Wilson, 2008;Fischer, 2012;Glenberg & Robertson, 2000;Glenberg, 2015;Zwaan & Madden, 2005), which have taken a central place in the debate on concept acquisition and representation. In this view, concepts such as DOG are formed through the interplay between linguistic experience (hearing or reading the word dog) as well as sensorimotor experience (seeing a dog, or hearing it bark) (Zwaan & Madden, 2005). Consequently, in language processing, the linguistic stimuli would act as cues to re-activate this sensorimotor experience, or the representation formed from it. ...
... We thus assume that, when a compound is processed, the concepts related to its constituent meanings are accessed. This operation would involve the re-activation of sensorimotor experiential traces linked to the constituents (Zwaan & Madden, 2005), including representations formed from visual experience. These multi-modal representations, in turn, constitute the building blocks of a conceptual combination process, in which they are combined into a new representation (Spalding et al., 2010). ...
Previous studies found that an automatic meaning-composition process affects the processing of morphologically complex words, and related this operation to conceptual combination. However, research on embodied cognition demonstrates that concepts are more than just lexical meanings, rather being also grounded in perceptual experience. Therefore, perception-based information should also be involved in mental operations on concepts, such as conceptual combination. Consequently, we should expect to find perceptual effects in the processing of morphologically complex words. In order to investigate this hypothesis, we present the first fully-implemented and data-driven model of perception-based (more specifically, vision-based) conceptual combination, and use the predictions of such a model to investigate processing times for compound words in four large-scale behavioral experiments employing three paradigms (naming, lexical decision, and timed sensibility judgments). We observe facilitatory effects of vision-based compositionality in all three paradigms, over and above a strong language-based (lexical and semantic) baseline, thus demonstrating for the first time perceptually grounded effects at the sub-lexical level. This suggests that perceptually-grounded information is not only utilized according to specific task demands but rather automatically activated when available.
... Many scholars have shown that both the processing and production of gestures are based on the instantaneous, automatic and unconscious mechanism of sensorimotor simulation (Gallese & Goldman, 1998;Gallese, 2001;Moors & De Houwer, 2006;Hostetter & Alibali, 2008;Shtyrov, Butorina, Nikolaeva & Stroganova, 2014). This mechanism has its roots in the kinesthetic imitation (Meltzoff & Gopnik, 1993;Chartland & Bargh, 1999;Dijksterhuis & Bargh, 2001), is influenced by experience (Zwaan & Madden, 2005;Zwaan & Taylor, 2006;Pecher & Winkielman, 2013) and is considered one of the main mechanisms underlying embodiment effects found in cognitive processing (Dijkstra & Post, 2015). ...
... This association is generally studied through experimental designs consisting of conditions in which participants are instructed to perform behaviors compatible or not with certain stimuli, so that the recording of short response times in compatible conditions is considered indicative of a facilitation effect in cognitive processing, while longer response times in incompatible conditions are indicative of an interference effect. This is because the involved cognitive process entails a mental simulation that reactivates the same sensorimotor states that were active while experiencing the processed stimulus (Goldstone & Barsalou, 1998;Zwaan & Taylor, 2006;Zwaan & Madden, 2005;Barsalou, Santos, Simmons & Wilson, 2008;Guan, Meng, Yao & Glenberg;. ...
... Such aspects have been put in light only recently from the embodiment perspective, thanks to the important discovery of mirror neurons (Rizzolatti et al., 1988;Gallese, Fadiga, Fogassi, & Rizzolatti, 1996;Rizzolatti & Fadiga, 1998;Rizzolatti & Sinigaglia, 2006). This perspective assumes that the mechanisms that give rise to sensorimotor simulations are based on previous experience and had the original function of allowing information to be represented in the absence of physical stimuli (Glenberg, 1997;Barsalou, 1999;2010;Borghi, Glenberg & Kaschak, 2004;Gallese & Lakoff, 2005;Zwaan & Madden, 2005;Casasanto, 2009;Pecher, Van Dantzig, Boot, Zanolie, & Huber, 2010;Gallese & Sinigaglia, 2011). Therefore, top-down motor compatibility effects can be generated even when cognitive processes occur without a direct connection to the environment that had allowed the acquisition or activation of sensorimotor patterns (Niedenthal et al., 2005;Körner, Topolinski, & Strack, 2015). ...
Our recent study within the embodiment perspective showed that the evaluation of true and false information activates the simulation of vertical and horizontal head movements involved in nodding and shaking of the head (Moretti & Greco, 2018). This result was found in an explicit evaluation task where motion detection software was deployed to enable participants to assess a series of objectively true or false statements by moving them with the head vertically and horizontally on a computer screen, under conditions of compatibility and incompatibility between simulated and performed action. This study replicated that experiment, but with subjective statements about liked and disliked food, in both explicit and implicit evaluation tasks. Two experiments, plus one control experiment, were devised to test the presence of a motor-affective compatibility effect (vertical-liked; horizontal-disliked) and whether the motor-semantic compatibility found with objective statements (vertical-true; horizontal-false) could be a sub-effect of a more general and automatic association (vertical-accepted; horizontal-refused). As expected, response times were shorter when statements about liked foods and disliked foods were moved vertically and horizontally respectively by making head movements, even when participants were not explicitly required to evaluate them. In contrast, the truth compatibility effect only occurred in the explicit evaluation task. Overall results support the idea that head-nodding and shaking are simulated approach-avoidance responses. Different aspects of the meaning of these gestures and the practical implications of the study for cognitive and social research are discussed.
... This is especially true for the area of language comprehension. Embodied cognition views of human language comprehension (e.g., Barsalou, 1999;Bergen, 2012;Glenberg & Kaschak, 2002;Zwaan & Madden, 2005) propose that comprehenders grasp the meaning of a word by mentally simulating the word's referent. Concretely, upon hearing or reading a word, comprehension is effected by reactivating sensorimotor experiences that are associated with its referent. ...
... Over the last few years, embodied accounts of cognition have become increasingly important in theories and empirical research on human language comprehension. Crucially, these approaches (e.g., Barsalou, 1999;Glenberg & Kaschak, 2002;Zwaan & Madden, 2005) propose that we conceive the meaning of language via mental simulations, which are created by means of activating and combining sensorimotor experiences related to the referents of the linguistic input (e.g., words, phrases, or idioms). As of yet, however, there has been little focus on the processes underlying meaning composition during embodied sentence comprehension. ...
Language comprehenders activate mental representations of sensorimotor experiences related to the content of utterances they process. However, it is still unclear whether these sensorimotor simulations are driven by associations with words or by a more complex process of meaning composition into larger linguistic expressions, such as sentences. In two experiments, we investigated whether comprehenders indeed create sentence-based simulations. Materials were constructed such that simulation effects could only emerge from sentence meaning and not from word-based associations alone. We additionally asked when during sentence processing these simulations are constructed, using a garden-path paradigm. Participants read either a garden-path sentence (e.g., "As Mary ate the egg was in the fridge") or a corresponding unambiguous control with the same meaning and words (e.g., "The egg was in the fridge as Mary ate"). Participants then judged whether a depicted entity was mentioned in the sentence or not. In both experiments, picture response times were faster when the picture was compatible (vs. incompatible) with the sentence-based interpretation of the target entity (e.g., both for garden-path and control sentence: an unpeeled egg), suggesting that participants created simulations based on the sentence content and only operating over the sentence as a whole.
... It is also considered essential to develop complex sensorimotor skills, facilitating the representation of behaviors and actions. Embodied Language Grounding [50] is the field that studies how agents can align language with their behaviors in order to extract the meaning of linguistic constructions. Early approaches in In (a) the agent grows an object which is described with spatial (underlined) or attribute (highlighted) reference. ...
... The idea that agents should learn to represent and ground language in their experience of the world has a long history in developmental robotics [50,39,40,7] and was recently extended in the context of Language Conditioned Deep Reinforcement Learning [11,22,31,3]. These recent approaches often consider navigation [10,9] or object manipulation [1,22] tasks and are always using instructive language. ...
Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents. We also release our code under open-source license as well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.
... Embodied Language Grounding [50] is the field that studies how agents can align language with their behaviors in order to extract the meaning of linguistic constructions. Early approaches in In (a) the agent grows an object which is described with spatial (underlined) or attribute (highlighted) reference. ...
... The idea that agents should learn to represent and ground language in their experience of the world has a long history in developmental robotics [50,39,40,7] and was recently extended in the context of Language Conditioned Deep Reinforcement Learning [11,22,31,3]. These recent approaches often consider navigation [10,9] or object manipulation [1,22] tasks and are always using instructive language. ...
Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents. We also release our code under open-source license as well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.
... Subsequently, word learning is often accompanied by a direct association to objects, actions and properties of their referents in the environment (Carey and Bartlett, 1978;Bloom, 2000;Zwaan and Madden, 2005). These sensory and motor experiences become reactivated when the child and later the adult, encounters the word. ...
... Moreover, children also learn the syntactical construction allowing transferring concrete experiences to an abstract meaning, such as in "tell them a story". This would be associated with a speech motor program and support embodiment effects when processing metaphors and abstract language (Zwaan and Madden, 2005). ...
Accumulating behavioral and neurophysiological evidence supports the idea of language being grounded in sensorimotor processes, with indications of a functional role of motor, sensory and emotional systems in processing both concrete and abstract linguistic concepts. However, most of the available studies focused on native language speakers (L1), with only a limited number of investigations testing embodied language processing in the case of a second language (L2). In this paper we review the available evidence on embodied effects in L2 and discuss their possible integration into existing models of linguistic processing in L1 and L2. Finally, we discuss possible avenues for future research towards an integrated model of L1 and L2 sensorimotor and emotional grounding.
... These simulations are based on previously acquired information and are supposed to be the result of the evolution of mechanisms which originally allowed individuals to make inferences and represent information in absence of physical stimuli. Thus, the effects of this grounding occur even when cognition is disconnected from the environment in which the sensorimotor patterns were acquired or activated [2][3][4]. These effects, generally, show up as a facilitation or interference in cognitive processing, based on whether bodily states and cognitive states are compatible or not. ...
... In Western culture, indeed, the vertical movement of nodding is used to communicate agreement or acceptance and it typically accompanies positive and affirmative responses (Yes), while the horizontal movement of shaking commonly accompanies negative responses (No) and it is used to communicate dissent or denial (Fig. 1). A relevant idea within the embodiment perspective is that gestures are simulated actions [10] and that sensorimotor simulation is one of the main mechanisms underlying compatibility effects [11] which occur because the affected cognitive processing entails a mental simulation that reactivates the same neuronal paths that were active while experiencing the situation expressed verbally [4]. ...
Research within the embodiment perspective has found that cognitive processing proceeds easier when bodily actions (mostly arms motion) are compatible with the conceptual meaning of verbal expressions (concrete or abstract, or with positive and negative values). Facilitation effects involving head motion, however, have not yet been investigated. The present work aims to test the motor compatibility hypothesis between directional head movements, usually performed to communicate agreement and disagreement, and truth evaluation. Five experiments were designed: participants were asked to assess a series of sentences as true or false, according to their meaning (objectively) or on the basis of personal preferences (subjectively), in compatible and incompatible motion conditions and with different response modalities. Response times were shorter only when true sentences, or about a liked content, were moved vertically, and when false sentences, or about a disliked content, were moved horizontally, with the head. Results confirm the hypothesis that higher cognitive processing is grounded in bodily motion, and shed light on the possibility to manipulate vertical and horizontal head movements in order to reveal attitudes.
... The Theory of Experiential Traces suggests that bodily experience plays an important role in language comprehension. Each time we interact with the world, we produce traces similar to this experience, which are then activated when we encounter the same object or concept, i.e. the understanding of concepts and language is based on the activation of experiential traces [5]. Some studies on language have shown that people engage their sensory-motor systems and even emotional neural mediators when they engage in language comprehension [6]. ...
... We propose that DVS operates as a type of editor that uses language to activate mental models and can then insert, delete, modify or replace aspects of their contents. During SR, DVS activates perceptual and sensorimotor representations, which in turn activate corresponding lexical representations (Zwaan and Madden 2005) via reciprocally connected brain systems (Pulvermüller 2005). We discuss DVS at greater length below. ...
Recent information technologies such as virtual reality (VR) and augmented reality (AR) allow the creation of simulated sensory worlds with which we can interact. Using programming language, digital details can be overlaid onto displays of our environment, confounding what is real and what has been artificially engineered. Natural language, particularly the use of direct verbal suggestion (DVS) in everyday and hypnotic contexts, can also manipulate the meaning and significance of objects and events in ourselves and others. In this review, we focus on how socially rewarding language can construct and influence reality. Language is symbolic, automatic and flexible and can be used to augment bodily sensations e.g. feelings of heaviness in a limb or suggest a colour that is not there. We introduce the term ‘suggested reality’ (SR) to refer to the important role that language, specifically DVS, plays in constructing, maintaining and manipulating our shared reality. We also propose the term edited reality to encompass the wider influence of information technology and linguistic techniques that results in altered subjective experience and review its use in clinical settings, while acknowledging its limitations. We develop a cognitive model indicating how the brain’s central executive structures use our personal and linguistic-based narrative in subjective awareness, arguing for a central role for language in DVS. A better understanding of the characteristics of VR, AR and SR and their applications in everyday life, research and clinical settings can help us to better understand our own reality and how it can be edited.
... Where do symbolic representations of language get their meaning from? It has been argued both from a theoretical and an empirical perspective that knowledge is grounded in perceptual experience (Barsalou, 2008;Lakoff, 1987;Langacker, 1999;Zwaan and Madden, 2005). Evidence for this embodied view of knowledge comes from a range of scientific domains such as neuroimaging (e.g. ...
Language grounding to vision is an active field of research aiming to enrich text-based representations of word meanings by leveraging perceptual knowledge from vision. Despite many attempts at language grounding, it is still unclear how to effectively inject visual knowledge into the word embeddings of a language in such a way that a proper balance of textual and visual knowledge is maintained. Some common concerns are the following. Is visual grounding beneficial for abstract words or is its contribution only limited to concrete words? What is the optimal way of bridging the gap between text and vision? How much do we gain by visually grounding textual embeddings? The present study addresses these questions by proposing a simple yet very effective grounding approach for pre-trained word embeddings. Our model aligns textual embeddings with vision while largely preserving the distributional statistics that characterize word use in text corpora. By applying a learned alignment, we are able to generate visually grounded embeddings for unseen words, including abstract words. A series of evaluations on word similarity benchmarks shows that visual grounding is beneficial not only for concrete words, but also for abstract words. We also show that our method for visual grounding offers advantages for contextualized embeddings, but only when these are trained on corpora of relatively modest size. Code and grounded embeddings for English are available at https://github.com/Hazel1994/Visually_Grounded_Word_Embeddings_2.
... We should note that the above interpretation of the model results is based on a key assumption: we are postulating that a decrease in the number of responsive extrasylvian CA cells corresponds to a de facto deficit in the word recognition ability of the network. A large body of experimental data suggests that (extrasylvian) sensorimotor areas convey information on aspects of word meaning (71)(72)(73); thus, even if parts of the CA circuits in extrasylvian areas are still activating in response to a presentation of an auditory stimulus, we submit that this partial activation contains less semantic information about the word, and therefore this information deficit translates directly into a corresponding observable word recognition deficit. The results of the simulations also enable us to make predictions and adjudicate between competing theories on the question of whether SD patients have category-specific or category-general recognition problems. ...
The nature of semantic knowledge – conceptual information stored in the brain – is highly debated in the field of cognitive science. Experimental and clinical data specify various cortical regions involved in the processing of meaning. Those include semantic hubs that take part in semantic processing in general as well as sensorimotor areas that process specific conceptual categories according to their modality. Biologically inspired neurocomputational models can help adjudicate between different theories about the exact roles of those regions in the functioning of the semantic system.
Here, we used an existing neuroanatomically constrained model of frontotemporal brain areas implicated in language acquisition and grounding. We adapted it to replicate and explain the effects of semantic dementia on word processing abilities. Semantic dementia is a disease characterized by semantic knowledge deterioration that correlates with neural damage in the anterior temporal lobe. The behavior of our model is in accordance with clinical data – namely, word recognition performance decreases as SD lesions progress, whereas word repetition abilities remain preserved, or are less affected. Furthermore, our model makes novel predictions about category-specific effects of SD – namely, our simulation results indicate that word processing should be more impaired for object-than for action-related words, and that white matter degradation should lead to more severe consequences than the same proportion of grey matter degradation.
The present results provide a mechanistic, cortical-level explanatory account of a range of language impairments as observed in clinical populations during the onset and progress of semantic dementia.
... As part of cognition, language processing itself might be grounded in perception and action, and exploit the same kind of representations as those involved in perception/action (Barsalou, 1999;Glenberg, 1997;Zwaan, 2004). Specifically, embodied-simulation views of language comprehension suggest that understanding words and sentences involves mentally simulating the actions, events and referents being described through the reactivation of experiential traces (Zwaan & Madden, 2005). For example, understanding a sentence like John pets the dog would reactivate perceptual traces of direct experience with dogs, as well as motor traces of petting. ...
The current study originates from inconsistent findings within the framework of embodied language processing, specifically in the reading-by-rotating literature: whereas some studies report a match advantage (e.g., Zwaan and Taylor (J Exp Psychol 135:1, 2006)), i.e., shorter reading times when the direction of a linguistically conveyed manual rotation matched rather than mismatched the direction of an actually to be performed manual rotation Claus (Acta Psychol 156:104–113, 2015) found a mismatch advantage. The current study addresses two explanations that were previously discussed as potentially responsible for this inconsistency: on the one hand, differences in the knob devices employed; on the other hand, differences in the perspectives adopted by the readers depending on the number of characters involved in the narratives. Concurrently, the study exploits individual differences in motoric experience to explore the experiential basis of action-sentence compatibility effects. The results are inconclusive with respect to the two explanations. However, in their overall picture, they contribute suggestive considerations for the ongoing debate on action-simulation effects by pointing to the potential role of interindividual variation in motoric experience.
... We based our work on research using the vertical Stroop task. For example, Lachmair et al. (2011) employed such a setup (see also Thornton et al., 2013) for verifying assumptions that were derived from the experiential-simulations view of language comprehension (Barsalou, 1999;Glenberg & Kaschak, 2002;Zwaan & Madden, 2005). In their vertical Stroop task, Lachmair et al. (2011, Experiment 2) presented participants with nouns referring to entities that are typically located in lower or upper vertical space (e.g., "worm" and "bird", respectively). ...
The number of web-based studies in experimental psychology has been growing tremendously throughout the last few years. However, a straightforward web-based implementation does not exist for all types of experimental paradigms. In the current paper, we focus on how vertical response movements—which play a crucial role in spatial cognition and language research—can be translated into a web-based setup. Specifically, we introduce a web-suited counterpart of the vertical Stroop task (e.g., Fox & Shor, in Bull Psychon Soc 7:187–189, 1976; Lachmair et al., in Psychon Bull Rev 18:1180–1188, 2011; Thornton et al., in J Exp Psychol Hum Percept Perform 39:964–973, 2013). We employed nouns referring to entities typically located in lower or upper vertical space (e.g., “worm” and “bird”, respectively) in Experiments 1 and 2, and emotional valence words associated with a crouched or an upward bodily posture (e.g., “sadness” and “excitement”, respectively) in Experiment 3. Depending on the font color, our participants used their mouse to drag the words to the lower or upper screen location. Across all experiments, we consistently observed congruency effects analogous to those obtained with the lab paradigm using actual vertical arm movements. Consequently, we conclude that our web-suited paradigm establishes a reliable approach to examining vertical spatial associations.
... Con el término «corporeización» (en inglés, embodiment), la lingüística cognitiva se refiere a «la motivación más o menos directa del lenguaje en nuestra experiencia corpórea, física, social y cultural» (Ibarretxe-Antuñano, 2013: p. 254, con referencia a Johnson, 1987 Desde esta perspectiva, el lenguaje llega a tener significado porque está anclado en nuestra experiencia sensomotora (Glenberg y Robertson, 2000;Zwaan y Madden, 2005). La perspectiva corpórea de la producción de significados se basa en la hipótesis de la simulación corporeizada, según la cual, cuando nos comunicamos mediante el lenguaje, construimos mentalmente las experiencias que corresponderían a la percepción de los objetos y acciones descritos por el discurso (Bergen, 2015: p. 142). ...
Desde una perspectiva corpórea de la cognición, los gestos representacionales se han descrito como creaciones espontáneas que emergen de la producción de imágenes mentales durante los procesos de construcción de significados. El objetivo de este trabajo es explorar el papel que desempeñan este tipo de gestos en los procesos de construcción de significados de los intérpretes simultáneos. Con este fin, se han estudiado las relaciones entre los gestos icónicos realizados espontáneamente por cuatro intérpretes en cabina y las imágenes mentales que recordaban haber producido durante la interpretación. Los resultados ofrecen indicios convergentes de una vinculación entre los gestos analizados y las imágenes mentales descritas por las participantes y permiten formular algunas hipótesis sobre el origen y las funciones de los gestos icónicos producidos por los intérpretes durante la interpretación simultánea.
... In contrast to propositional and computational theories (Chomsky, 1957;Fodor, 1975;Pylyshyn, 1986) that propose abstract, amodal, and arbitrary symbols; grounded-embodied cognition theories assume that there are no strict delimitations between perception and cognition and that mental representations are sensorimotor in nature (Barsalou, 1999(Barsalou, , 2008Paivio, 1986;Pulvermüller, 1999). A grounded-embodied treatment of language suggests that upon reading or hearing a word, traces of previous perceptual experiences (e.g., visual, auditory, olfactory, gustatory or tactile) related to the meaning of the word are automatically reactivated and give rise to a sensorimotor simulation of its referent (Bergen, 2016;Zwaan & Madden, 2005;Zwaan & Radvansky, 1998). For example, reading a spatial word, that is a word with a spatial association such as bird which elicits a mental image, can give rise to a perceptual experience of an upward location based on the typical location of birds in the physical world. ...
Previous evidence shows that words with implicit spatial meaning or metaphorical spatial associations are perceptually simulated and can guide attention to associated locations (e.g., bird – upward location). In turn, simulated representations interfere with visual perception at an associated location. The present study investigates the effect of spatial associations on short-term verbal recognition memory to disambiguate between modal and amodal accounts of spatial interference effects across two experiments. Participants in both experiments encoded words presented in congruent and incongruent locations. Congruent and incongruent locations were based on an independent norming task. In Experiment 1, an auditorily presented word probed participants’ memory as they were visually cued to either the original location of the probe word or a diagonal location at retrieval. In Experiment 2, there was no cue at retrieval but a neutral encoding condition in which words normed to central locations were shown. Results show that spatial associations affected memory performance although spatial information was neither relevant nor necessary for successful retrieval: Words in Experiment 1 were retrieved more accurately when there was a visual cue in the congruent location at retrieval but only if they were encoded in a non-canonical position. A visual cue in the congruent location slowed down memory performance when retrieving highly imageable words. With no cue at retrieval (Experiment 2), participants were better at remembering spatially congruent words as opposed to neutral words. Results provide evidence in support of sensorimotor simulation in verbal memory and a perceptual competition account of spatial interference effect.
... Related work. The idea that language understanding is grounded in one's experience of the world and should not be secluded from the perceptual and motor systems has a long history in Cognitive Science [31,80]. This vision was transposed to intelligent systems [71,54], applied to humanmachine interaction [24,52] and recently to deep RL via frameworks such as BabyAI [17]. ...
Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.
... As we said before, the simulation mechanism is a sort of a "re-instantiation" or "re-experiencing" of past sensorimotor experiences, constrained by, and therefore adapted to, the present situation. Central to the idea of embodied comprehension is the claim that linguistic constructs (word forms) are linked to sensorimotor representations (which denote the meaning of words) that form the basis of the simulation mechanism (Zwaan & Kaschak, 2008;Zwaan & Madden, 2005). For example, Zwaan, Stanfield, & Yaxley (2002) asked their participants to read sentences which evoked an object or animal in a certain position, with the shape of the animal or object being different depending on the position (for example, an eagle in flight or perched on its nest). ...
The aim of the present study was to demonstrate the multisensory nature of vocabulary knowledge by using learning designed to encourage the simulation of sensorimotor experiences.
Forty participants were instructed to learn pseudowords together with arbitrary definitions, either by mentally experiencing (sensorimotor simulation) the definitions, or by mentally repeating them. A test phase consisting of three tasks was then administered: in a recognition task, participants had to recognize learned pseudowords among distractors. In a categorization task, they had to categorize pseudowords as representing either living or non-living items. Finally, in a sentence completion task, participants had to decide whether pseudowords were congruent with context sentences. As expected, the sensorimotor simulation condition induced better performances only in the categorization task and the sentence completion task. The results converge with data from the literature in demonstrating that knowledge emergence implies sensorimotor simulation and showing that vocabulary learning can benefit from encoding that encourages the simulation of sensorimotor experiences.
... 7 We thank an anonymous reviewer for this suggestion. 8 The limit of 3000 concepts was not entirely arbitrary but rather reflected a very rough approximation of the number of concepts in long-term memory that may retain trace activation in the context of an ongoing task, such as when maintaining the plot of a novel in a situation model (e.g., Zwaan & Madden, 2005) or when asked to remember large sets of pictured objects (e.g., Brady, Konkle, Alvarez, & Oliva, 2008). 9 Note that because the optimal stopping point for the model was not at the maximum performance for rounded mean rank (see section "Choosing the Optimal Model"), it meant the model was technically capable of better performance on this measure. ...
The human conceptual system comprises simulated information of sensorimotor experience and linguistic distributional information of how words are used in language. Moreover, the linguistic shortcut hypothesis predicts that people will use computationally cheaper linguistic distributional information where it is sufficient to inform a task response. In a pre-registered category production study, we asked participants to verbally name members of concrete and abstract categories and tested whether performance could be predicted by a novel measure of sensorimotor similarity (based on an 11-dimensional representation of sensorimotor strength) and linguistic proximity (based on word co-occurrence derived from a large corpus). As predicted, both measures predicted the order and frequency of category production but, critically, linguistic proximity had an effect above and beyond sensorimotor similarity. A follow-up study using typicality ratings as an additional predictor found that typicality was often the strongest predictor of category production variables, but it did not subsume sensorimotor and linguistic effects. Finally, we created a novel, fully grounded computational model of conceptual activation during category production, which best approximated typical human performance when conceptual activation was allowed to spread indirectly between concepts, and when candidate category members came from both sensorimotor and linguistic distributional representations. Critically, model performance was indistinguishable from typical human performance. Results support the linguistic shortcut hypothesis in semantic processing and provide strong evidence that both linguistic and grounded representations are inherent to the functioning of the conceptual system. All materials, data, and code are available at https://osf.io/vaq56/.
... According to embodied cognition accounts, cognitive processes operate on meaning representations that are grounded in experience and are therefore sensorimotor in nature (e.g., Zwaan and Madden, 2005;Barsalou, 2008). One of the central issues plaguing these accounts are abstract concepts, such as justice or imaginary numbers. ...
The present study examines whether deictic time and valence are mentally associated, with a link between future and positive valence and a link between past and negative valence. We employed a novel paradigm, the two-choice-sentence-completion paradigm, to address this issue. Participants were presented with an initial sentence fragment that referred to an event that was either located in time (future or past) or of different valence (positive or negative). Participants chose between two completion phrases. When the given dimension in the initial fragment was time, the two completion phrase alternatives differed in valence (positive vs. negative). However, when the given dimension in the initial fragment was valence, the two completion phrase alternatives differed in time (future vs. past). As expected, participants chose completion phrases consistent with the proposed association between time and valence. Additional analyses involving individual differences concerning optimism/pessimism revealed that this association is particularly pronounced for people with an optimistic attitude.
... Finally, another possible explanation for the lack of SR compatibility effects in our opaque conditions is that of experiential traces (cf. Ahlberg et al., 2018;Zwaan & Madden, 2005), which posits that, during language acquisition, new words and prepositions such as an, ab, and auf might be learned with corresponding actions, for instance, pointing downward when learning the word ab, and that every subsequent encounter with ab reactivates the experiential trace of a downward motion (embodied cognition), whereas words used in our opaque condition would not activate an experiential trace associated with an upward or downward movement. ...
We investigated sensitivity for the vertical meaning of the German particle ab by means of stimulus-response compatibility effects. In German, the particle ab is ambiguous and can take on a vertical meaning (downward) as in Auf und Ab (engl. up and down), but it can also take on nonvertical or nonspatial meanings as in Ab und An (engl. from time to time). We show that the particle ab only creates a spatial compatibility effect relative to the German particle auf (Experiment 1) but not relative to the particle an (Experiment 2). Furthermore, as participants executed upward versus downward responses in both Experiments 1 and 2, the mere vertical antagonism of the responses was insufficient to instill a verticality-based compatibility effect. In addition, the compatibility effect was restricted to the transparent version of the particle. If a letter sequence corresponding to the particles was presented in a semantically and morphologically opaque way (e.g., the letters ab were embedded in the German word kn
ab
e, engl. boy), no compatibility effect was found, underlining that the effect was due to word meanings rather than visual features. The results underscore the boundary conditions for using compatibility effects in investigating lexical and semantic spatial processing in humans.
... answer this query by simulating the scenario of Sally, the floor, and the action of pounding using a hammer, and then retrieving the answer from the simulation (Zwaan and Madden, 2005). Although the question and answer are presented in natural language, most of the information for performing this simulation are in the visual and motor systems. ...
Despite the recent progress in AI powered by deep learning in solving narrow tasks, we are not close to human intelligence in its flexibility, versatility, and efficiency. Efficient learning and effective generalization come from inductive biases, and building Artificial General Intelligence (AGI) is an exercise in finding the right set of inductive biases that make fast learning possible while being general enough to be widely applicable in tasks that humans excel at. To make progress in AGI, we argue that we can look at the human brain for such inductive biases and principles of generalization. To that effect, we propose a strategy to gain insights from the brain by simultaneously looking at the world it acts upon and the computational framework to support efficient learning and generalization. We present a neuroscience-inspired generative model of vision as a case study for such approach and discuss some open problems about the path to AGI.
... Indem nun die ökonomischen Standardlehrbücher den Begriff der Nachfrage sprachlich mit jenem des ‚Fallens' koppeln und zugleich durch die ‚fallende' Nachfragekurve visualisieren, kann esüberträgt man Erkenntnisse empirischer Studien der Kognitionswissenschaften (vgl. etwa Bransford et al. 1972;Zwaan/Pecher 2012;Zwaan/Madden 2005) auf diesen Falldazu kommen, dass die ‚Nachfrage' implizit auch mit Vorstellungen des Gesetzmäßigen, Unweigerlichen und jenseits der eigenen Kontrolle ablaufenden Geschehnissen verbunden wird. Wäre dies der Fall, so würde damit die Rede vom ‚Gesetz der Nachfrage' eben nicht mehr nur durch Frames auf sprachlicher Ebene erfasst werden (Lakoff und Wehling sprechen hier von surface frames), sondern auch auf Ebene der deep seated frames: ...
Standard economics teaching has been subject to increasing scholarly critique claiming it to be either one-sided, detached from reality or an instrument of indoctrination. The following study attempts to systematically address and analyze possible forms of indoctrination. Drawing from two standard textbooks –Economics by Samuelson and Nordhaus as well as Mankiw’s Eco-nomics– a language and text-based analysis, based primarily on cognitive research methodol-ogy, provides a detailed elucidation of examples of unconscious forms of persuasion students are subjected to which do not match the neoclassical ideal of scientific objectivity. In addition, the following discusses whether a manipulation of students, in the sense of deliberate and covert influence of thought and perception processes is in fact taking place, while identifying future fields of research as well as possible new directions in economics education.
... A critical question, then, is how this meaning reaches our fingertips. One possibility arises from the embodied-cognition framework which proposes that all language is grounded in sensorimotor experiences (Zwaan and Madden, 2005;Glenberg and Gallese, 2012). In this view, the gestures we produce reflect sensorimotor experiences and arise from rich memory representations of the world around us. ...
When people talk, they gesture. Gesture is a fundamental component of language that contributes meaningful and unique information to a spoken message and reflects the speaker’s underlying knowledge and experiences. Theoretical perspectives of speech and gesture propose that they share a common conceptual origin and have a tightly integrated relationship, overlapping in time, meaning, and function to enrich the communicative context. We review a robust literature from the field of psychology documenting the benefits of gesture for communication for both speakers and listeners, as well as its important cognitive functions for organizing spoken language, and facilitating problem-solving, learning, and memory. Despite this evidence, gesture has been relatively understudied in populations with neurogenic communication disorders. While few studies have examined the rehabilitative potential of gesture in these populations, others have ignored gesture entirely or even discouraged its use. We review the literature characterizing gesture production and its role in intervention for people with aphasia, as well as describe the much sparser literature on gesture in cognitive communication disorders including right hemisphere damage, traumatic brain injury, and Alzheimer’s disease. The neuroanatomical and behavioral profiles of these patient populations provide a unique opportunity to test theories of the relationship of speech and gesture and advance our understanding of their neural correlates. This review highlights several gaps in the field of communication disorders which may serve as a bridge for applying the psychological literature of gesture to the study of language disorders. Such future work would benefit from considering theoretical perspectives of gesture and using more rigorous and quantitative empirical methods in its approaches. We discuss implications for leveraging gesture to explore its untapped potential in understanding and rehabilitating neurogenic communication disorders.
... That is, each conversational participant determines a point of view and selects some details of the utterance to focus on, ignoring others. Repeating this process over and over in conversing allows language participants to arrive at a particular understanding of an utterance (Zwaan, 2016;Zwaan & Madden, 2005). We therefore further propose that viewers use our seven construals to comprehend activity and the passing of time in sign language storytelling. ...
Cohn (2016 Cohn, N. (2016). Sequential images are not universal, or Caveats for using visual narratives in experimental tasks. In A. Papafragou, D. Grodner, D. Mirman, & J. C. Trueswell (Eds.), Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 2057–2062). Cognitive Science Society. [Google Scholar]) posits two constraints critical to understanding sequential static images as narrative – the continuity constraint, assuring that elements in one frame are coreferential with elements in other frames, and the activity constraint, assuring that states in each frame are related with respect to passing of time, shifting of vantage point, or other factors relevant to activity. We identify seven construals used in ordinary human understanding of the physical world that help account for how viewers interpret activity in static visual narratives and that we propose form the foundation of the activity constraint. In an elicitation experiment, these construals also supported understanding of depicted activity in dynamic storytelling in Irish Sign Language, suggesting that a range of action information in sign language narratives is conveyed via reliance on visuo-spatial construals independent of language.
... Embodied language theories propose that linguistic representations are grounded in sensorimotor experiences: that words evoke sights, sounds, and movements in the mind, and those features constitute the representations of the words (e.g., Glenberg, 1997;Barsalou, 1999;Glenberg and Kaschak, 2002;Gallese & Lakoff, 2005;Barsalou, 2008). In many cases, theories of language embodiment have focused on how sensorimotor experiences are reactivated and recombined in mental simulations (Zwaan and Madden, 2005;Barsalou, 2009). Simulations are AFFORDANCE COMPATIBILITY EFFECT FOR WORD LEARNING 3 thought to constrain ongoing sensorimotor processes, perturbing the actions and perceptions of language listeners. ...
... One common criticism of many embodied theories of language is that they are ill-equipped to deal with abstract information (Zwaan & Madden, 2005; see also Borghi & Caruana, 2015). Several criticisms of EC have been noted, including that the theories offer nothing new, or are unfalsifiable (Mahon, 2015). ...
Cognitive psychology has undergone a paradigm shift in the ways we understand how knowledge is acquired and represented within the brain, yet the implications for how this impacts students’ learning of material across disciplines has yet to be fully applied. In this article, we present an integrative review of embodied cognition, and demonstrate how it differs from previously held theories of knowledge that still influence the ways in which many subjects are taught in the classroom. In doing so, we review the literature of embodied learning in the areas of reading instruction, writing, physics, and math. In addition, we discuss how these studies can lead to the development of new learning strategies that utilized the principles of embodied cognition.
After having read this chapter, our readers…
There has been a lot of recent interest in the way that language might enhance embodied cognition. This interest is driven in large part by a growing body of evidence implicating the language system in various aspects of semantic memory—including, but not limited to, its apparent contribution to abstract concepts. In this essay, I develop and defend a novel account of the cognitive role played by language in our concepts. This account relies on the embodied nature of the language system itself, diverges in significant ways from traditional accounts, and is part of a flexible, multimodal and multilevel view of our conceptual system.
This article is part of the theme issue ‘Concepts in interaction: social engagement and inner experiences’.
Poetry evokes mental imagery in its readers. But how is mental imagery precisely related to poetry? This article provides a systematic treatment. It clarifies two roles of mental imagery in relation to poetry—as an effect generated by poetry and as an efficient means for understanding and appreciating poetry. The article also relates mental imagery to the discussion on the ‘heresy of paraphrase’. It argues against the orthodox view that the imagistic effects of poetry cannot be captured by prosaic paraphrase, but points to features of poetry that can shape aspects of mental imagery that are liable to be lost in paraphrase.
We know little about the factors influencing which words we choose during lexical selection. In two experiments we investigated whether (re)activations of experiential traces of space have an impact on language production. Participants performed up- and downward head movements while listening to sentence fragments describing situations (e.g. ‘You are at the beach and you see…’). When reaching the upward/downward head position they completed the sentence with a freely chosen noun. A different group of participants rated the spatial location of the produced word’s referents. We found that the head movements influenced participant’s lexical choices. After upward movements the produced words were rated to be located higher up in space compared to downward movements. Furthermore, higher scores in interoceptive sensibility as measured using the attention regulation scale from the Multidimensional Assessment of Interoceptive Awareness questionnaire (Mehling et al., 2018) lead to an increased effect of head movement on the spatial properties of the produced nouns. We conclude that sensorimotor activations are among the meaning facets that guide which words we chose when speaking. The tendency to verbally express embodied meaning is enhanced with higher levels of interoceptive sensibility, suggesting that interoception may be a key to understand interindividual differences in how we express our experiences and feelings when we speak.
The role of meaning facets based on sensorimotor experiences is well-investigated in comprehension but has received little attention in language production research. In two experiments, we investigated whether experiential traces of space influenced lexical choices when participants completed visually-presented sentence fragments (e.g., ‘You are at the sea and you see a ...’) with spoken nouns (e.g., ‘dolphin’, ‘palm tree’). The words were presented consecutively in an ascending or descending direction, starting from the center of the screen. These physical spatial cues did not influence lexical choices. However, the produced nouns met the spatial characteristics of the broader sentence contexts such that the typical spatial locations of the produced noun referents were predicted by the location of the situations described by the sentence fragments (i.e., upper or lower sphere). By including distributional semantic similarity measures derived from computing cosine values between sentence nouns and produced nouns using a web-based text corpus, we show that the meaning dimension of ‘location in space’ guides lexical selection during speaking. We discuss the relation of this spatial meaning dimension to accounts of experientially grounded and usage-based theories of language processing and their combination in hybrid approaches. In doing so, we contribute to a more comprehensive understanding of the many facets of meaning processing during language production and their impact on the words we select to express verbal messages.
Substantial evidence indicates that first language (L1) comprehension involves embodied visual simulations. The present study tested the assumption that a formally learned second language (L2), which is less related to real-life experiences, is processed in a less embodied manner relative to a naturally acquired L1. To this end, bilingual participants completed the same task in their L1 and L2. In the task, they read sentences and decided immediately after each sentence whether a pictured object had been mentioned in the preceding sentence. Responses were significantly faster when the shape of the object in the picture matched rather than mismatched the sentence-implied shape, but only in the L1, and only when the L1 block was performed before the L2 block. These findings suggest that embodied visual simulations are reduced in a formally learned L2 and may be subjected to cross-language influences.
Theories of grounded cognition assume that conceptual representations are grounded in sen-sorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants' judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don't have.
Language comprehenders have been arguing to mentally represent the implied orientation of objects. However, compared to the effects of shape, size, and color, the effect of orientation is rather small. We examined a potential explanation for the relatively low magnitude of the orientation effect: Object size moderates the orientation effect. Theoretical considerations led us to predict a smaller orientation effect for small objects than for large objects in a sentence–picture verification task. We furthermore investigated whether this pattern generalizes across languages (Chinese, Dutch, and English) and tasks (picture-naming task). The results of the verification task show an orientation effect overall, which is not moderated by object size (contrary to our hypothesis) and language (consistent with our hypothesis). Meanwhile, the preregistered picture–picture verification task showed the predicted interaction between object size and orientation effect. We conducted exploratory analyses to address additional questions.
Reaction times for a translation recognition study are reported where novice to expert English–ASL bilinguals rejected English translation distractors for ASL signs that were related to the correct translations through phonology, semantics, or both form and meaning (diagrammatic iconicity). Imageability ratings of concepts impacted performance in all conditions; when imageability was high, participants showed interference for phonologically related distractors, and when imageability was low participants showed interference for semantically related distractors, regardless of proficiency. For diagrammatically related distractors high imageability caused interference in experts, but low imageability caused interference in novices. These patterns suggest that imageability and diagrammaticity interact with proficiency – experts process diagrammatic related distractors phonologically, but novices process them semantically. This implies that motivated signs are dependent on the entrenchment of language systematicity; rather than decreasing their impact on language processing as proficiency grows, they build on the original benefit conferred by iconic mappings.
Im Gegensatz zu klassischen, amodalen Kognitionstheorien, die das Gehirn als zentrale Instanz mentaler Repräsentationen und Kognition ansehen, postulieren Embodied-Cognition-Ansätze, dass Denkprozesse nicht unabhängig von Wahrnehmungs- und Bewegungsprozessen, sondern multimodal verkörperlicht sind. Im folgenden Kapitel werden die Ursprünge der Embodied-Cognition-Perspektive dargestellt und verschiedene neuere Embodied-Cognition-Ansätze mit ihrem jeweiligen Bezug zur Sportpsychologie beschrieben. Darauf aufbauend werden empirische Embodied-Cognition-Studien vorgestellt, die die Interaktionen zwischen Bewegung und Kognition sowie zwischen Bewegung und Wahrnehmung untersuchen. Zum Abschluss wird skizziert, wie Embodied-Cognition-Effekte spezifiziert und quantifiziert werden können. Außerdem werden Grenzen der Embodied-Cognition-Forschung aufgezeigt und Vorhersagen zur Zukunft von Embodied Cognition im Kontext der Sportpsychologie gemacht.
Scholars in sociology and social psychology typically represent creativity as an imaginative and deliberate mental activity. Such a perspective has led to a view of creativity as disconnected from the body and the senses as well as from nonanalytic cognition. In this article, we demonstrate that creativity is more grounded in bodily and sensory experience and more reliant on a combination of cognitive processes than has been typically recognized. We use literature on social cognition and embodiment to build our arguments, specifically, the embodied simulation perspective and tripartite process models. We draw from data on elite chefs to show how actors rely on embodied simulations, continually switch between heu-ristic and analytical thinking, and monitor and control their cognitive processing during the creative process. We outline the implications of this study for the understanding of creativity and extant models of cognition and action more generally.
Models representing meaning as high-dimensional numerical vectors (such as LSA, HAL, BEAGLE, Topic Models, GloVe or word2vec) have been introduced as extremely powerful machine-learning proxies for human semantic representations, and have seen an explosive rise in popularity over the last two decades. However, despite their considerable advancements and spread in the cognitive sciences, one can observe problems associated with the adequate presentation and understanding of some of their features. Indeed, when these models are examined from a cognitive perspective, a number of unfounded arguments tend to appear in the psychological literature. In the present article, we review the most common of these arguments, directed at (1) what exactly these models represent at the implementational level and their plausibility as a cognitive theory, (2) how they deal with various aspects of meaning such as polysemy or compositionality, and (3) how they relate to the debate on embodied and grounded cognition. We identify common misconceptions arising due to incomplete descriptions, outdated arguments, and unclear distinctions between theory and implementation of the models. We clarify and amend these points, to provide a theoretical basis for future research and discussions on vector models of semantic representation.
In this chapter the human embrained systems starting with the memory, the multimodal processes (visual, auditory, haptic, proprioceptive, vestibular olfactive, gustative, spatial), and the action via the neuron mirror system are analysed. These structure-function systems, which are considered to be in permanent interaction with each other are non verbal processes and are the basis of oral, written language, number and calculation action, i.e. the verbal processes. Their interaction with emotional process is given from a neurocognitive point of view. Self-consciousness and consciousness are analysed as emerging high level processes.
Cet article présente une application pédagogique subjectiviste en classe de Français langue étrangère (FLE) cherchant à utiliser la richesse de la pluralité culturelle des apprenants pour maximiser les performances d’apprentissage scolaire. La pédagogie subjectiviste émane de la pédagogie constructiviste fondée sur la reconnaissance que tout apprenant construit son savoir sur ses propres expériences. Elle repose sur deux processus fondamentaux ancrés anthropologiquement dans l’enculturation et la responsabilisation. L’enculturation est le processus à travers lequel un apprenant de FLE développe naturellement ses capacités linguistiques et sa compréhension socioculturelle. La responsabilisation est le processus naturel par lequel un apprenant développe une identité socioculturelle positive. Dans la classe subjectiviste de FLE, ces processus subjectivistes sont systématiquement conçus en activités permettant aux apprenants d’intérioriser les compétences et la culture de la langue. Cet article présente la théorie et la méthodologie subjectivistes, puis décrit une application pédagogique en classe de FLE.
Many people report experiencing mental imagery (visual, auditory and/or kinetic) when they comprehend verbal metaphors. The question whether imagery is merely an incidental side-effect of processes of metaphor understanding or plays a key role in comprehension remains unresolved, with diametrically opposed views expressed among psychologists, philosophers and literary theorists. I survey a wide array of evidence (behavioural and neurocognitive) concerning the activation and use of imagery in language processing generally and metaphor processing in particular. I conclude that mental imagery is not an essential component in the comprehension of language, whether literal or metaphorical, but it is often automatically activated in the minds of hearers or readers as a by-product of their linguistic and pragmatic processes. Writers may exploit this fact and enhance the impact of imagery by producing extended and/or creative metaphors, so that, although not essential, imagistic effects can be as significant as propositional cognitive effects. 2
A growing body of evidence suggests that cognition is embodied and grounded. Abstract concepts, though, remain a significant theoretical challenge. A number of researchers have proposed that language makes an important contribution to our capacity to acquire and employ concepts, particularly abstract ones. In this essay, I critically examine this suggestion and ultimately defend a version of it. I argue that a successful account of how language augments cognition should emphasize its symbolic properties and incorporate a view of embodiment that recognizes the flexible, multimodal and task-related nature of action, emotion and perception systems. On this view, language is an ontogenetically disruptive cognitive technology that expands our conceptual reach.
This article is part of the theme issue ‘Varieties of abstract concepts: development, use and representation in the brain’.
The close integration between visual and motor processes suggests that some visuomotor transformations may proceed automatically and to an extent that permits observable effects on subsequent actions. A series of experiments investigated the effects of visual objects on motor responses during a categorisation task. In Experiment 1 participants responded according to an object's natural or manufactured category. The responses consisted in uni-manual precision or power grasps that could be compatible or incompatible with the viewed object. The data indicate that object grasp compatibility significantly affected participant response times and that this did not depend upon the object being viewed within the reaching space. The time course of this effect was investigated in Experiments 2-4b by using a go-nogo paradigm with responses cued by tones and go-nogo trials cued by object category. The compatibility effect was not present under advance response cueing and rapidly diminished following object extinction. A final experiment established that the compatibility effect did not depend on a within-hand response choice, but was at least as great with bi-manual responses where a full power grasp could be used. Distributional analyses suggest that the effect is not subject to rapid decay but increases linearly with RT whilst the object remains visible. The data are consistent with the view that components of the actions an object affords are integral to its representation.
Deriving representations of meaning has been a long-standing problem in cognitive psychology and psycholinguistics. The lack of a m odel for representing semantic and grammatical knowledge has been a handicap in attempting to model the effects of semantic constraints in hum an syntactic processing. A computational model of high-dim ensional context space, the Hyperspace A nalogue to Language (H AL), is presented with a series of simulations modelling a variety of human empirical results. HAL learns its representations from the unsupervised processing of 300 million words of conversational text. W e propose that HAL's high-dim ensional context space can be used to (1) provide a basic categorisation of semantic and grammatical concepts, (2) model certain aspects of morphological ambiguity in verbs, and (3) provide an account of semantic context effects in syntactic processing. W e propose that the distributed and contextually derived representations that HAL acquires provide a basis for the subconceptual knowledge that can be used in accounting for a diverse set of cognitive phenomena.
Similarity is a central construct in models of cognitive processing, and as such has been the focus of psychological inquiry. This work has revealed that similarity and difference judgments are not always inverses. One explanation for this effect is that similarity judgments focus on matching relations between the items, while difference judgments focus on the mismatching attributes. A second explanation is that both similarity and difference judgments involve a process of structural alignment, and that they use the output of this process differently. These views are contrasted by using the one-shot mapping technique that places attribute similarity and relational similarity in competition. The results suggest that similarity and difference judgments both involve structural alignment.
How do my mental states compare to yours? We suggest that, while we may not be able to compare experiences, we can compare neural representations, and that the correct way to compare neural representations is through analysis of the distances between them. In this paper, we present a tech- nique for measuring the similarities between representations at various layers of neural networks. We then use the measure to demonstrate empirically that different artificial neural net- works trained by backpropagation on the same categorization task, even with different representational encodings of the in- put patterns and different numbers of hidden units, reach states in which representations at the hidden units are similar.
We evaluate a fundamental assumption of Lakoff and Johnson's (1980a, 1980b) view that people routinely use conceptual mappings to understand conventional expressions in ordinary discourse. Lakoff and Johnson argue that people rely on mappings such as ARGUMENT IS WAR in understanding expressions such as his criticism was right on target. We propose that people need not rely on conceptual mappings for conventional expressions, although such mappings may be used to understand nonconventional expressions. Three experiments support this claim. Experiments 1 and 2 used a reading-time measure and found no evidence that readers used conceptual mappings to understand conventional expressions. In contrast, the experiments did reveal the use of such mappings with nonconventional expressions. A third experiment ruled out lexical or semantic priming as an explanation for the results. Our findings call into question Lakoff and Johnson's central claim about the relationship between conventional expressions and conceptual mappings.
Referents ("topics") serve as file labels in the episodic memory for stored text. The grammar of topic marking is a set of processing instructions that cue two major cognitive systems: attention and episodic memory. The cataphoric elements in the grammar tell the hearer/reader whether the referent is important and thus needs to be activated and then used as file label for a newly opened storage file in the episodic memory. Unimportant referents are not activated but rather are filed as new information in the currently active memory file. The anaphoric elements in the grammar tell the hearer/reader where to search for the topic in the existing storage structure of episodic memory. Coding a referent by zero or unstressed pronoun signals that it is the currently active topic, that it should retain its activation, and that incoming information should continue to be filed under its label. When a currently inactive referent is coded as definite and important, various grammatical devices tell the hearer/reader where to search for it in episodic memory. Following search and retrieval, the referent is then activated, and new incoming information is filed under its label.
Two separate issues were looked at in this experimental study of the semantics of spatial prepositions. In the context of work to specify general factors of a functional geometry mediating the use of spatial prepositions (Garrod & Sanford 1989; Coventry 1992, 1993), object-specific effects were investigated. Subjects described video scenes of various objects and their responses of in, on, over, and beside were monitored. The independent variables involved the manipulation of functionality specific to various types of objects. It was concluded that knowledge about how particular objects interact with each other contributes to the representation of functional relations which determine preposition usage. Therefore a specification of functional geometries cannot proceed without a prior formulation of our knowledge about the physical and social worlds. Additionally two different experimental measures of prepositional covariance with the scenes were used: Lickert-scale judgements and sentence completions. Responses from two separate groups were compared. The findings indicated some agreement between the two measures, but also some differences in patterns of response. It is suggested that the measures are tapping different processes, and that a variety of methods need to be used to abstract to lexical representation.
The use of spatial relational terms requires the selection of a reference frame and the construction of a spatial template. The reference frame divides up space, indicating above/below, front/back, and left/right directions. Spatial templates are applied to reference frames and define regions in space that correspond to good, acceptable, and bad uses of particular spatial relations. In two experiments we examined whether reference frame selection influences the spatial templates that are constructed for the spatial relations “above” and “below.” Results indicated two such influences, one operating across trials and one operating within a trial. Across trials, the preferences for using the different types of reference frames directly influenced the parsing of space, such that when multiple spatial templates were constructed, they were combined to form a composite template. Spatial templates constructed for the different reference frames were very similar, indicating that the type of reference frame did not alter the overall shape or relative sizes of the regions within the spatial template. Within a trial, the activation of multiple reference frames during reference frame selection resulted in the construction of multiple spatial templates, even when instructions were given to respond on the basis of a single reference frame.
Several accounts of the semantic system posit that function information plays a critical role in the representations of man-made objects. Alternative possibilities are that man-made objects such as tools are organized according to manner of manipulation, or that both function and manipulation information figure importantly and distinctly in man-made object representations. An agnosic patient, FB, reported by Sirigu et al. (Brain 1991; 114: 727-41) provides support for the latter view. FB was able to demonstrate how to manipulate objects whose function he did not recognize. We now report two severely apraxic patients whose performance, together with that of FBI indicates that function and manipulation information may doubly dissociate. On ‘declarative’ semantic tests not requiring gesture production, our subjects demonstrate severely impaired manipulation knowledge in the context of relatively intact (and, in one subject, perfectly unimpaired) function knowledge. The double dissociation provides further support for the notion that function and manipulation knowledge are critical and distinct features of the representations of manipulable man-made objects.
Perceptual symbol systems assume an analogue relationship between a symbol and its referent, whereas amodal symbol systems assume an arbitrary relationship between a symbol and its referent. According to perceptual symbol theories, the complete representation of an object, called a simulation, should reflect physical characteristics of the object. Amodal theories, in contrast, do not make this prediction. We tested the hypothesis, derived from perceptual symbol theories, that people mentally represent the orientation of an object implied by a verbal description. Orientation (vertical-horizontal) was manipulated by having participants read a sentence that implicitly suggested a particular orientation for an object. Then recognition latencies to pictures of the object in each of the two orientations were measured. Pictures matching the orientation of the object implied by the sentence were responded to faster than pictures that did not match the orientation. This finding is interpreted as offering support for theories positing perceptual symbol systems.
The question, "What is Cognitive Science?" is often asked but seldom answered to anyone's satisfaction. Until now, most of the answers have come from the new breed of philosophers of mind. This book, however, is written by a distinguished psychologist and computer scientist who is well-known for his work on the conceptual foundations of cognitive science, and especially for his research on mental imagery, representation, and perception. In Computation and Cognition, Pylyshyn argues that computation must not be viewed as just a convenient metaphor for mental activity, but as a literal empirical hypothesis. Such a view must face a number of serious challenges. For example, it must address the question of "strong equivalents" of processes, and must empirically distinguish between phenomena which reveal what knowledge the organism has, phenomena which reveal properties of the biologically determined "functional architecture" of the mind. The principles and ideas Pylyshyn develops are applied to a number of contentious areas of cognitive science, including theories of vision and mental imagery. In illuminating such timely theoretical problems, he draws on insights from psychology, theoretical computer science, artificial intelligence, and psychology of mind. A Bradford Book
This article can be viewed as an attempt to explore the consequences of two propositions. (I) Intentionality in human beings (and animals) is a product of causal features of the brain. I assume this is an empirical fact about the actual causal relations between mental processes and brains. It says simply that certain bran processes are sufficient for intentionality. (2) Instantiating a computer program is never by itself a sufficient condition of intentionality. The main argument of this paper is directed at establishing this claim. The form of the argument is to show how a human agent could instantiate the program and still not have the relevant intentionality. These two propositions have the following consequences: (3) The explanation of how the brain produces intentionality cannot be that it does it by instantiating a computer program. This is a strict logical consequence of 1 and 2. (4) Any mechanism capable of producing intentionality must have causal powers equal to those of the brain. This is meant to be a trivial consequence of 1. (5) Any attempt literally to create intentionality artificially (strong AI) could not succeed just by designing programs but would have to duplicate the causal powers of the human brain. This follows from 2 and 4. 'Could a machine think?' On the argument advanced here only a machine could think, and only very special kinds of machines, namely brains and machines with internal causal powers equivalent to those of brains. And that is why strong AI has little to tell us about thinking, since it is not about machines but about programs, and no program by itself is sufficient for thinking.
This paper offers a detailed survey of occurrences of English reflexive pronouns which are marked with respect to the binding theory of Chomskyan generative grammar. Through a large corpus of attested examples found in contemporary nonlinguistic prose, English reflexive pronouns are shown to violate Chomsky's Binding Principle A in a productive way. The proposed grammar of these violations draws a clear-cut line between syntax and discourse, and shows Chomsky's Binding Principle A to be (a) essentially correct for English if it is clearly defined as a theory of sentence-internal, discourse-independent anaphora; but also (b) crucially incomplete, since it ignores a whole component of the grammar of reflexives and thus fails to account for an open set of data.
Eighty-two participants listened to sentences and then judged whether two sequentially presented visual objects were the same. On critical trials, participants heard a sentence describe the motion of a ball toward or away from the observer (e.g., “The pitcher hurled the softball to you”). Seven hundred and fifty milliseconds after the offset of the sentence, a picture of an object was presented for 500 ms, followed by another picture. On critical trials, the two pictures depicted the kind of ball mentioned in the sentence. The second picture was displayed 175 ms after the first. Crucially, it was either slightly larger or smaller than the first picture, thus suggesting movement of the ball toward or away from the observer. Participants responded more quickly when the implied movement of the balls matched the movement described in the sentence. This result provides support for the view that language comprehension involves dynamic perceptual simulations.
*
I would like to thank Bill Batchelder, David Laberge, and Ken Wexler for a number of interesting discussions which helped me in writing this review.
Imagery and Text, Second Edition extends the first edition's unified theory of cognition in literacy from the perspective of Dual Coding Theory (DCT), one of the most influential and empirically sound theories of cognition ever developed. This theory provides a comprehensive, systematic account of all major aspects of literacy including decoding, comprehension, and response in reading and composing in writing. The Second Edition updates DCT as a scientific theory, a cognitive theory, an embodied theory, and a constructivist theory of literacy. New content includes a detailed account of the decoding process and its integral connection to comprehension, a new program of research on DCT in composing text, a review of neuroscientific support, and increased attention to multimedia literacy, socio-cultural influences, and recent educational applications. More than any other theory, DCT explains how both verbal and nonverbal cognition are woven together through all aspects of literacy. Written in concise chapters with illustrative examples, Imagery and Text is approachable for both students and advanced scholars in the field of literacy.
Spatial relations indicate the location of an object by specifying its direction with respect to a reference object whose location is known. It has been largely assumed that specific features of these objects are generally ignored in descriptions of spatial location. Within such a view, spatial relations can be described in terms of geometric or topological features that are not dependent on object details such as function. We demonstrate in two experiments that a geometric feature (center of mass) is not sufficient. Using two different tasks, we show that the function of the reference object and its relationship with the other object significantly influence the way in which vertical spatial relations (“above” and “below”) are applied to spatial configurations.
Five new or returning midlife university students were asked to provide metaphors describing their experience of midlife change, change indicated by their attending university at this point in their lives. Five resulting portraits (S. Lawrence-Lightfoot & J. H. Davis, 1997) were created, based on participant-invoked metaphors: personal rebellion against colonization, stage productions, collection and accumulation, journey home, and alchemy and metamorphosis. The portraits indicate that even from within widely divergent metaphoric conceptualizations, the respondents all find midlife to be an active, positive, and hopeful period of life. For them midlife is decidedly neither crisis nor dull plateau. Further, given the richness of the metaphor use, the study demonstrates the desirability of listening carefully to and moreover expressly soliciting metaphoric usage, in order to more fully understand people's inner realities. Finally, the portraits reveal that the participants' metaphoric conceptions are deeply imbedded in their lives, past and current, and as such identify life themes.
Performance of bilingual Russian–English speakers and monolingual English speakers during auditory
processing of competing lexical items was examined using eye tracking. Results revealed that both bilinguals and monolinguals experienced competition from English lexical items overlapping phonetically with an English target item (e.g., spear and speaker). However, only bilingual speakers experienced competition from Russian competitor items overlapping crosslinguistically with an English target (e.g., spear and spichki, Russian for matches). English monolinguals treated the Russian competitors as they did any other filler items. This difference in performance between bilinguals and monolinguals tested with exactly the same sets of stimuli suggests that eye movements to a crosslinguistic competitor are due to activation of the other language and to between-language competition rather than being an artifact of stimulus selection or experimental design.
Bilingualism provides a unique opportunity for exploring hypotheses about how the human brain encodes language. For example, the “input switch” theory states that bilinguals can deactivate one language module while using the other. A new measure of spoken language comprehension, headband-mounted eyetracking, allows a firm test of this theory. When given spoken instructions to pick up an object, in a monolingual session, late bilinguals looked briefly at a distractor object whose name in the irrelevant language was initially phonetically similar to the spoken word more often than they looked at a control distractor object. This result indicates some overlap between the two languages in bilinguals, and provides support for parallel, interactive accounts of spoken word recognition in general.
Studies are reported which show that concrete and abstract words of equal objective frequency (based on available , word counts) are not perceived as being equal. The abstract word has greater perceived frequency than the concrete word. The judged variety of contexts in which a word appears correlates very highly with perceived frequency. The results have relevance to the design of learning studies in which concrete and abstract words are used. and also to the interpretation of such experiments.
Four theories of the human conceptual system—semantic memory, exemplar models, feed‐forward connectionist nets, and situated simulation theory—are characterised and contrasted on five dimensions: (1) architecture (modular vs. non‐modular), (2) representation (amodal vs. modal), (3) abstraction (decontextualised vs. situated), (4) stability (stable vs. dynamical), and (5) organisation (taxonomic vs. action–environment interface). Empirical evidence is then reviewed for the situated simulation theory, and the following conclusions are reached. Because the conceptual system shares mechanisms with perception and action, it is non-modular. As a result, conceptual representations are multi-modal simulations distributed across modality‐specific systems. A given simulation for a concept is situated, preparing an agent for situated action with a particular instance, in a particular setting. Because a concept delivers diverse simulations that prepare agents for action in many different situations, it is dynamical. Because the conceptual system’s primary purpose is to support situated action, it becomes organised around the action–environment interface.
Previous research has shown that abstract words are comprehended more slowly than concrete words. In two experiments, the authors investigated the influence of a sentence context on the processing of abstract and concrete words. In the first experiment, 17 college students completed a word naming task in which abstract and concrete words were presented in either a meaningful or a neutral sentence context. Abstract words took longer to name than concrete words when presented in a neutral context, suggesting that concreteness effects occur early in the processing of words. In the second experiment, 21 college students judged whether abstract and concrete words were meaningful sentence completions; the words were presented in either a meaningful or a nonmeaningful context. Meaningfulness judgments were made more slowly for abstract than for concrete words when the words were presented in a nonmeaningful context. However, in both the naming and the meaningfulness judgment tasks, concreteness effects in processing were eliminated when the words were presented in supportive contexts. The authors discuss these results as supporting a context availability view - that abstract words are comprehended more slowly than concrete words when presented in nonsupportive contexts because it is more difficult to retrieve the associated contextual knowledge necessary for comprehension. /// [French] Les recherches antérieures montrent que les mots abstraits sont compris plus lentement que les mots concrets. Le présent article rapporte deux recherches qui visent à identifier les causes de ces différences. Dans une première expérience, 17 étudiants du niveau universitaire ont été soumis à une tâche de dénomination dans laquelle des mots abstraits et concrets étaient présentés soit dans un contexte signifiant soit dans un contexte neutre. Les sujets prirent plus de temps à nommer les mots abstraits que les mots concrets en contexte neutre, confirmant l'effet de la concrétude sur le traitement de mots. Dans la seconde expérience, 22 étudiants du niveau universitaire ont été soumis à une tâche de jugement qui consistait à dire si oui ou non les phrases présentées avaient du sens. Les phrases se distinguaient selon qu'elles incluaient des mots abstraits ou concrets et qu'elles étaient présentées dans un contexte signifiant ou non signifiant. Dans les deux tâches cependant, l'effet du type de mots était annulé lorsque les mots étaient présentés dans des contextes signifiants. Pour les auteurs, ces résultats confirment la nécessité, de pouvoir activer des connaissances contextuelles pour traiter les mots. Ainsi, si les mots abstraits sont traités plus lentement que les mots concrets en contexte non signifiant, c'est que les mots abstraits activent plus difficilement les connaissances contextuelles stokées en mémoire. /// [Spanish] La investigación previa ha demostrado que las palabras abstractas son comprendidas más lentamente que las palabras concretas. En dos experimentos, los autores investigaron el punto focal de tales efectos de concreción. En el primer experimento, 17 estudiantes universitarios completaron una tarea de nombramiento de palabras en la que palabras abstractas y concretas fueron presentadas en una frase con contexto significativo o en una frase con contexto neutral. Les tomó más tiempo nombrar las palabras abstractas que las concretas cuando éstas fueron presentadas en un contexto neutral, sugiriendo así un punto focal temprano para los efectos de concreción en el procesamiento de palabras. En el segundo experimento, 21 estudiantes universitarios juzgaron la significancia de oraciones en las que palabras abstractas y concretas fueron presentadas en un contexto significativo o no significativo. Los juicios de significancia fueron hechos más lentamente para las palabras abstractas que para las concretas cuando las palabras se presentaban en un contexto no significativo. Sin embargo, tanto en la tarea de nombramiento como en la de juicio de significancia, los efectos de concreción en el procesamiento fueron eliminados cuando las palabras se presentaron en contextos de apoyo. Los autores discuten estos resultados como evidencia para sustentar una teoría basada en la disponibilidad del contexto - en que las palabras abstractas son comprendidas más lentamente que las concretas cuando son presentadas en contextos sin soporte porque es más difícil accesar el conocimiento contextual asociado con ellas, y este proceso es necesario para la comprensión. /// [German] Frühere forschungen haben gezeigt, daß abstrakte Wörter langsamer verstanden werden als konkrete. In zwei Experimenten untersuchten die Autoren dieses Beitrages die Lokalisierung dieser Konkretheitseffekte. Im ersten Experiment führten siebzehn Studenten eine Wortbenennungsaufgabe durch, in der abstrakte und konkrete Wörter entweder in einem sinnvollen oder neutralen Satzkontext präsentiert wurden. In einem neutralen Satzkontext dauerte die Benennung bei abstrakten Wörtern länger als bei konkreten, was bei der Wortverabeitung auf eine frühe Lokalisierung der Konkretheitseffekte hindeutet. Im zweiten Experiment beurteilten einundzwanzig Studenten den Sinngehalt von Sätzen, in denen abstrakte und konkrete Wörter entweder in einem sinnvollen oder nicht-sinnvollen Kontext präsentiert wurden. Beurteilungen des Sinngehaltes bei Wörtern in einem nicht-sinnvollen Kontext wurden für abstrakte Wörter langsamer vorgenommen als für konkrete. Sowohl in der Benennungsaufgabe als auch in der Sinngehaltsbeurteilung wurden die Konkretheitseffekte bei der Verarbeitung jedoch eliminiert, wenn die Wörter in einem sinnvollen Kontext präsentiert wurden. Die Autoren besprechen diese Resultate zur Unterstützung einer Ansicht über die Kontextverfügbarkeit - daß nämlich abstrakte Wörter langsamer verstanden werden als konkrete, wenn sie in nicht-sinnvollen Kontexten präsentiert werden, weil es schwieriger ist, das zur Verständigung notwendige assoziative Kontextwissen abzufragen.
In this article, Ipropose STRAIGHT äs an image Schema, discuss t he evidence for this claim, and examine the relationship between some of the specific properties of this image Schema in our experience and how they are extended into abstract domains. Relying on Johnson s (1987) criteria for an image Schema, I review research on the special role of straight lines in visual perception, and consider the relations between spatial and forcedynamic properties of straight bodily forms and movements, and of straight objects which we commonly encounter in our experience, particularly in industrialized societies. An examination of metaphorical expressions in English and Russian shows that domains in these languages commonly characterized by STRAIGHTNESS—or its opposites—include time, events, discourse, thought, control, social norms, morality, truth, and law. The examples support findings from previous research that a duality commonly exists in Systems of metaphors between OBJECT and LOCATION versions. Cultural models of the target domains are also discussed as a factor which motivates the metaphorical characterization of a domain as STRAIGHT or not, and whether that STRAIGHTNESS is evaluated as positive or negative.
This paper reports four experiments designed to examine the role that recurring bodily experiences have in motivating people's understandings of different senses of the polysemous word stand. Different patterns of recurring bodily experiences, called image schemas, emerge throughout sensorimotor activity and from our perceptual understanding of actions and events in the real world. The present claim is that each use of stand is motivated by a complex pattern of different image schemas. Experiment 1 revealed five major image schemas that are primarily to people's bodily experiences of standing. Experiment 2 looked at people's judgements of similarity for different uses of stand. Experiment 3 first examined people's intuitions about the relative importance of five image schemas for different senses of stand. We then attempted to predict the pattern of data from Experiment 2 using the image schema profiles obtained for the different senses of stand in Experiment 3. Finally,-Experiment 4 considered an alternative hypothesis for people's judgements of similarity for different uses of stand. The data from these studies generally suggest that people tacitly believe there are significant connections between their recurring bodily experiences and the various meanings of the polysemous word stand. We argue that theories of psychological semantics should account not only for the organization of polysemous words in the mental lexicon, but must also be capable of explaining why different senses of a word make sense to people in the way they do.
A word may have the identical conventional meaning in different descriptions and yet be taken as denoting very different things. The proposal we tested is that the denotation of such a word is what the addressee believes it must be in order to contribute to the model of the situation that the speaker intended the addressee to create. We tested this proposal on the verb approach in descriptions schematised by “Figure F is just approaching landmark L for reason R.” The distance from F to L was judged to be larger, all else being equal, the larger the landmark, the larger the figure, the faster the figure was moving, and the farther away the figure could be and still fulfil his or her purpose. We argued that these and other results about word interpretation are best accounted for by listeners creating the intended situation models.
We examined the hypothesis that muscle contractions in the face influence subjective emotional experience. Previously, researchers have been critical of experiments designed to test this facial feedback hypothesis, particularly in terms of methodological problems that may lead to demand characteristics. In an effort to surmount these methodological problems Strack, Martin, and Stepper (1988) developed an experimental procedure whereby subjects were induced to contract facial muscles involved in the production of an emotional pattern, without being asked to actually simulate an emotion. Specifically, subjects were required to hold a pen in their teeth, which unobtrusively creates a contraction of the zygomaticus major muscles, the muscles involved in the production of a human smile. This manipulation minimises the likelihood that subjects are able to interpret their zygomaticus contractions as representing a particular emotion, thereby preventing subjects from determining the purpose of the experiment. Strack et al. (1988) found support for the facial feedback hypothesis applied to pleasant affect, in that subjects in the pen-in-teeth condition rated humorous cartoons as being funnier than subjects in the control condition (in which zygomaticus contractions were inhibited). The present study represents an extension of this nonobtrusive methodology to an investigation of the facial feedback of unpleasant affect. Consistent with the Strack et al. procedure, we wanted to have subjects furrow their brow without actually instructing them to do so and without asking them to produce any emotional facial pattern at all. This was achieved by attaching two golf tees to the subject's brow region (just above the inside comer of each eye) and then instructing them to touch the tips of the golf tees together as part of a “divided-attention” experiment. Touching the tips of the golf tees together could only be achieved by a contraction of the corrugator supercilii muscles, the muscles involved in the production of a sad emotional facial pattern. Subjects reported significantly more sadness in response to aversive photographs while touching the tips of the golf tees together than under conditions which inhibited corrugator contractions. These results provide evidence, using a new and unobtrusive manipulation, that facial feedback operates for unpleasant affect to a degree similar to that previously found for pleasant affect.
New insight into how the brain makes a decision has come from a study of the effects of the decision-making process on an eye movement evoked by electrical stimulation of the frontal cortex. The accumulation of sensory evidence was found to cause a gradual commitment toward a choice.
How can the semantic interpretation of a formal symbol system be made
intrinsic to the system, rather than just parasitic on the meanings in our
heads? How can the meanings of the meaningless symbol tokens, manipulated
solely on the basis of their (arbitrary) shapes, be grounded in anything but
other meaningless symbols? The problem is analogous to trying to learn Chinese
from a Chinese/Chinese dictionary alone. A candidate solution is sketched:
Symbolic representations must be grounded bottom-up in nonsymbolic
representations of two kinds: (1) "iconic representations," which are analogs
of the proximal sensory projections of distal objects and events, and (2)
"categorical representations," which are learned and innate feature-detectors
that pick out the invariant features of object and event categories from their
sensory projections. Elementary symbols are the names of these object and event
categories, assigned on the basis of their (nonsymbolic) categorical
representations. Higher-order (3) "symbolic representations," grounded in these
elementary symbols, consist of symbol strings describing category membership
relations (e.g., "An X is a Y that is Z").
Conventional wisdom and previous research suggest that similarity judgments and difference judgments are inverses of one another . An excep- tion to this rule arises when both rela- tional similarity and attributional simi- larity are considered . When presented with choices that are relationally or at- tributionally similar to a standard, hu- man subjects tend to pick the relationally similar choice as more similar to the standardandas more different from the standard. These results not only rein- force the general distinction between at- tributes and relations but also show that attributes and relations are dynamically distinct in the processes that give rise to similarity and difference judgments .
Expressions describing the spatial relation between two objects can be interpreted from a perspective defined by the objects (intrinsic) or by the viewer or the environment (deictic-extrinsic). Identifying the factors that govern the selection of one perspective over another is an important step in understanding the interaction between language and perception. In two experiments, we explored the influence of a functional relation between two objects on the formulation of a spatial expression relating the objects. Both rating and production tasks showed that people preferred to use intrinsic descriptions in the presence of a functional relation and deictic-extrinsic descriptions in the absence of such a relation. These results suggest that contextual aspects of the scene influence spatial term selection.
Four experiments explored readers' mental models of described scenes. Environments were described from one of two perspectives, an internal perspective of an observer within the scene, surrounded by objects, or an external perspective of an observer outside the scence, with objects in front. Subjects read narratives describing a scene and were probed for locations of objects. In the general case, reaction times to identify objects were fastest for the head/feet (above/below) axis, then the front/back (front/behind) axis, and then the left/right axis, conforming to the spatial framework analysis which reflects people's conceptions of space based on typical interactions in space. For the internal spatial framework, readers were faster to questions of front than back, reflecting the perceptual and biological asymmetries that favor an observer's front. For the external spatial framework, all objects were in front of the observer and readers were equally fast to questions of front and behind. The difference between internal and external spatial framework reflects the different perceptual experience of observers in the two perspectives. The two variants of the spatial framework allowed us to infer readers' spatial perspective for narratives with unspecified perspectives.