ChapterPDF Available

Iconic and representational gestures

Authors:

Abstract and Figures

The construct of iconic gestures, those gestures understood as sharing certain form features with the object, action or scene they represent, has traditionally proven to be a useful tool for scholars to classify this subset of gestures, distinguishing them from other types such as indexical or emblematic gestures. More recent approaches prefer to avoid discrete categories and rather speak in terms of dimensions or principles, such as iconicity or indexicality, in order to highlight the fact that gestures tend to perform multiple functions at once. Iconic co-speech gestures are semiotically conditioned not only by the particular language spoken, but also by the pragmatics of situated, multimodal language use, thus being cognitively, intersubjectively and socio-culturally motivated. Iconic patterns of gesture production identified within individual as well as across various languages and language families have provided valuable insights into the intimate interrelation of thought, gesture and speech in face-to-face interaction as well as other kinds of multimodal communication. This chapter reviews both production- and comprehension-oriented research on iconic gestures, including examples from cross-cultural, clinical, and forensic studies. Ways in which iconic gestures pertain to related terms, such as representational and referential gestures, are also addressed.
Content may be subject to copyright.
VIII. Gesture and language1732
Müller, Cienki, Fricke, Ladewig, McNeill, Teßendor (eds.) 2014, Body  Language  Communication (HSK 38.2), de Gruyter, 17321746
Wilcox, Sherman 2004. Conceptual spaces and embodied actions: Cognitive iconicity and signed
languages. Cognitive Linguistics 15(2): 119147.
Wilcox, Sherman in press. Signed languages. In: Dagmar Divjak and Ewa Dabrowska (eds.), Hand-
book of Cognitive Linguistics. Berlin: Mouton de Gruyter.
Wittgenstein, Ludwig 1953. Philosophical Investigations. Oxford: Blackwell.
Zlatev, Jordan 2005. What’s in a schema? Bodily mimesis and the grounding of language. In: Beate
Hampe (ed.), From Perception to Meaning: Images Schemas in Cognitive Linguistics, 313342.
Berlin: Mouton de Gruyter.
Irene Mittelberg, Aachen (Germany)
131. Iconic and representational gestures
1. Introduction
2. Iconic gestures, dimensions, and patterns
3. Mimicry: Intersubjective alignment and understanding
4. Representational and referential gestures
5. Concluding remarks
6. References
Abstract
The construct of iconic gestures, those gestures understood as sharing certain form features
with the object, action or scene they represent, has traditionally proven to be a useful tool
for scholars to classify this subset of gestures, distinguishing them from other types such as
indexical or emblematic gestures. More recent approaches prefer to avoid discrete catego-
ries and rather speak in terms of dimensions or principles, such as iconicity or indexicality,
in order to highlight the fact that gestures tend to perform multiple functions at once. Iconic
co-speech gestures are semiotically conditioned not only by the particular language spoken,
but also by the pragmatics of situated, multimodal language use, thus being cognitively,
intersubjectively and socio-culturally motivated. Iconic patterns of gesture production iden-
tified within individual as well as across various languages and language families have
provided valuable insights into the intimate interrelation of thought, gesture and speech in
face-to-face interaction as well as other kinds of multimodal communication. This chapter
reviews both production- and comprehension-oriented research on iconic gestures, including
examples from cross-cultural, clinical, and forensic studies. Ways in which iconic gestures
pertain to related terms, such as representational and referential gestures, are also ad-
dressed.
1. Introduction
Iconicity, in broader terms, is understood as the relationship between a sign and an
object in which the form the sign takes is perceived and interpreted to be similar in some
131. Iconic and representational gestures 1733
way to the object it is representing (Peirce 1960). Because representation tends to be
partial, iconicity interacts with the principles of metonymy (i.e., a part stands for a
whole; see Mittelberg and Waugh this volume).
Since gesture is characterized by the extraordinary affordance of spatially and dynam-
ically encoding visual information and kinetic action, questions gesture researchers have
dealt with have been whether people produce iconic gestures
(i) a) based on what visual information they have available,
(ii) b) if they are motivated by the particular language they cogesture with, or
(iii) c) a combination of the two.
Moreover, because the driving principle of iconicity is generally assumed to be based on
similarity (and not conventionality), the role that social and individual practices play
in the creation and use of these semiotic forms can be easily misunderstood. Semiotic
foundations of iconicity are discussed in detail in Mittelberg this volume (see also, e.g.,
Andre
´n 2012; Fricke 2012; Lücking 2013; Sonesson 2008). This chapter will focus on
(predominantly) iconic gestures, both from a production and a comprehension perspec-
tive, as they are linked to speech, to social practices, and how they pertain to related
terms, i.e., representational and referential gestures, in the literature.
2. Iconic gestures, dimensions, and patterns
Among the established gesture typologies, the one most strongly associated with the
notion of iconic gestures is the one proposed by McNeill and Levy (1982) and then
extended by McNeill (1992: 12). According to this Peirce-inspired taxonomy, iconics en-
compass gestures illustrating aspects of what is conveyed in speech through actional and
visuo-spatial imagery primarily based on memories and other kinds of mental represen-
tations. Iconic gestures imply a correspondence between the form a gesture takes, e.g.,
a body posture, hand shape and/or the trajectory and manner of a hand movement, and
the person, concrete object, action, or motion event it depicts. Put differently, “in an
iconic gesture there is a certain degree of isomorphism between the shape of the gesture
and the entity that is expressed by the gesture” (Kita 2000: 162). Iconics also reflect the
viewpoint from which the speaker portrays a scene, e.g., character or observer viewpoint.
Metaphorics are related to iconics in that they are “like iconic gestures in that they are
pictorial, but the pictorial content presents an abstract idea rather than a concrete object
or event” (McNeill 1992: 14). Both sets of representational gestures tend to be produced
in a more central gesture space, as opposed to others, for example, indexical gestures
which are produced more peripherally (McNeill 1992: 8994), which arguably accentu-
ates their relatedness. Examples of the kinds of abstract ideas referred to here are
“knowledge, language itself, the genre of the narrative, etc.” (1992: 80). In a well-known
example of an iconic gesture, a speaker describes a scene in the animated cartoon “The
Canary Row” (McNeill 1992: 12). When saying he grabs a big oak tree and he bends it
way back, the speaker simultaneously performs with his right arm and hand a grabbing
and pulling action backward. According to McNeill (2005: 67), “the gesture has clear
iconicity the movement and the handgrip; also a locus (starting high and ending
low) all creating imagery that is analogous to the event being described in speech at
the same time”. As becomes apparent in this quote, in his more recent work McNeill
VIII. Gesture and language1734
(2005: 4143) has moved away from the original categories (i.e. iconics, deictics, meta-
phorics, beats, and cohesives) in preference to dimensions such as iconicity, indexicality,
and metaphoricity (see also Duncan, McNeill, and McCullough 1995).
2.1. Production o iconic gestures
Exploring patterns in the production of iconic gestures has allowed valuable insights
into the intimate interrelation of thought, gesture and speech, positing “gesture and the
spoken utterance as different sides of a single underlying mental process” (McNeill 1992:
1). Iconic gestures have been shown to enhance both speaking and thinking, in particular
analytical and spatio-motoric thinking (e.g., Kita 2000). In the Vygotsky-inspired con-
cept of growth point, gestural imagery also plays an important role, “since it grounds
sequential linguistic categories in an instantaneous visuospatial context” (McNeill
2005: 115).
Fig. 131.1: Enactment of
figure’s posture
Fig. 131.2: Paul Klee, Dance
of a Mourning Child (1922)
Not unlike pointing and other indexical gestures, iconic gestures can be produced to fill
a semantic gap in speech, especially when representing spatial imagery like size, shape,
motion, or other schematic, partial images which take advantage of the affordances of
gestures versus speech. For example, the participant shown in Fig. 133.1 ephrastically
describes a painting by Paul Klee (Dance of a Mourning Child, 1922, Fig. 133.2; adapted
from Mittelberg 2013) through a full-body enactment of the figure’s stance including the
tilted head and arm configuration, as well as its position of the legs and the eye-gaze
directed downward. She also evokes the flowing skirt by repeated manual up-and-down
movements around her hips and upper legs. Keeping character viewpoint all through
the sequence, she extends her arms to the side while saying her head was turned to this
side … if I were mirroring what she was doing … and her arms were like this …. Then, on
and her mouth was almost in the shape of a heart, she draws an icon of the figure’s heart-
shaped mouth onto her own lips and two lines imitating its eye slits onto her own eyes
(I kept trying to see if her eyes were open or closed, and it looked like they were just slits).
Although in the speech modality she’s using a deictic in “like this”, bringing the listener’s
attention to her body and gestures, the full meaning of the speech-gesture utterance is
131. Iconic and representational gestures 1735
understood via her iconic bodily gestures, an image clearly easier to produce via gesture
than via speech (see Fricke 2007 and Streeck (2009: 108118) on gestures performing
attributive and adverbial functions).
A host of studies involving typologically different languages have provided ample
insights into how processes of thinking for speaking (Slobin 1996) may on a moment-to-
moment basis shape not only the linguistic, i.e. lexical and grammatical, encoding of
motion events, but also the tightly interwoven gestural imagery especially exhibiting
manner and/or path information. Initial observations put into relief a “high degree of
cross-linguistic similarity” in gestures about the same content and “accompanying lin-
guistic segments of an equivalent type, in spite of major lexical and grammatical differ-
ences between the languages” (McNeill 1992: 212). Subsequent research has produced
converging “evidence for language specificity of representational gestures” (Kita 2000:
167; see e.g., Kita and Özyürek 2007; McNeill and Duncan 2000; Müller 1998a; Özyürek
et al. 2005). For example Kita and Özyürek (2007) found that speakers of satellite-
framed languages (e.g., Germanic and Slavic languages, which encode path separately
from the main motion verb) were more likely to iconically represent path and manner
of motion actions conflated in the same gestures as opposed to speakers of verb-framed
languages (e.g., Romance and Semitic languages, where path is expressed in the main
verb and manner of motion expressed by other means) who tend to gesture manner and
path as separate gestures. Following this line of research, these gestures appear to be
motivated by the iconicity of the linguistic-conceptual representation and not of the
visual-spatial imagery. However, as Duncan (2002: 204) points out regarding gestural
imagery and verb aspect, this claim “is not the same as saying that gestures merely
mirror linguistically codified aspect contrasts. Rather, different verb aspects appear ex-
pressive of fundamental distinctions in the ways we can ‘cognize’ an event during acts
of speaking” (see also Cienki in press on representational gestures’ connection to
grammar).
Besides single gestures exhibiting a structural resemblance with the entities or actions
they portray, and supporting the argument that gestures are semiotically linked not only
to the language used but to language use, there are also discourse-internal iconic pat-
terns, so-called catchments: “[a] catchment is recognized from a recurrence of gesture
features over a stretch of discourse. It is a kind of thread of consistent visuo-spatial
imagery running through a discourse segment that provides a gesture-based window into
discourse cohesion” (McNeill 2000: 316; the notion is inspired by Kendon’s (1972) idea
of locution clusters). Hence, iconicity here pertains to how gestures resemble (in part)
other, preceding gestures in the semiotic neighborhood (see also Jakobson 1960 on the
principle of equivalence and the poetic function in language).
A common denominator for this research strand, which has resulted in a wealth of
iconic gestures, is the employed semi-experimental method of data elicitation: partici-
pants are asked to retell the aforementioned animated cartoon “The Canary Row”, in
which the protagonists Tweety Bird and Sylvester undergo all kinds of adventures while
chasing each other around town. This particular kind of stimulus consisting of two-
dimensional cartoon action movies with numerous motion events unfolding up, down
and along various kinds of spatial structures is reflected in the iconic gestures produced
by a large and diverse group of study participants. While this approach limits the range
of gestures as well as the kind of iconic gestures (i.e., based on the medium cartoon)
that might occur, it has the advantage, as opposed to naturalistic conversations for
VIII. Gesture and language1736
instance, that based on the stimulus material the gesture analyst is able to reconstruct
scene by scene what the participants’ gestures are iconic of. This also allows researchers
to compare gesture production patterns not only across speakers of a single language or
across different languages, but also across different age and clinical groups. Investiga-
tions into language acquisition have revealed particular stages in cognitive and language
development, including transition points and gesture-speech mismatches (e.g., Goldin-
Meadow 2003; McNeill 2005; McNeill and Duncan 2000). Generally, work on aphasia
and other communication disorders evidences their impact on forms and functions of
iconic gestures and also provide a window onto the workings of the non-disturbed
multimodal language system (e.g., Caldognetto, Magno, and Poggi 1995; Cocks et al.
2011; Cocks et al. 2013; Duncan and Pedelty 2007; Goodwin 2011; Hogrefe et al. 2012;
McNeill 2005). The large body of work reviewed above has presented ample evidence
that iconic gestures are cognitively and communicatively extremely versatile, fulfilling a
broad range of functions that go well beyond facilitating lexical retrieval during word-
searching processes (e.g., Hadar and Butterworth 1997; see Krauss, Chen, and Gottes-
man 2000: 263 on the category of lexical gestures).
2.2. Comprehension o iconic gestures
Taking the perspective of gesture comprehension, a body of research has evidenced the
communicative significance of iconic gestures, that is, their contribution to the address-
ee’s understanding of what the speaker is conveying multimodally. In an intercultural
study, Calbris (1990) explores how iconic and cultural facets of a set of French gestures
ranging from highly motivated examples to others implying a cultural cliche
´was inter-
preted by a group of Hungarian and a group of Japanese speakers respectively. Some of
what is called “cliche
´” here compares to the kinds of culturally-defined gestures now
known as emblems (McNeill 1992). In reference to Saussure’s (1986) notion of the arbi-
trariness, the author stresses the point that “gestures are not arbitrary signs, but conven-
tional and motivated (Fo
´nagy 1956, 19611962)” (Calbris 1990: 38). The more conven-
tional gestures, such as the cliche
´Ceinture, evoked by a transverse line drawn at waist
level to indicate privation, were not understood equally well by the two groups: the
Hungarians were better at guessing and reconstructing their meaning than the Japanese.
More universal motivations appear to facilitate intercultural comprehension, as in a
gesture consisting of a hand placed on the belly expressing, in conjunction with a corre-
sponding facial expression, disgust or nausea (Calbris 1990: 39). It is concluded that
“[l]ess linked to a cliche
´, less symbolic, less polyvalent, motivation seems to be all the
more natural and transparent as it approaches depiction, or simple reproduction of
movement. It seems all the more direct as it is narrowly linked with what is concrete”
(Calbris 1990: 40; see also Andre
´n (2010) and Bouvet (1997, 2001) on the transparency
of iconic gestures and signs).
In a serious of experimental studies investigating the communicative functions of co-
speech gestures, Beattie and Shovolton (1999) found, for instance, that participants who
had listened to retellings of a cartoon story gave a more accurate summary by ten per-
cent if they could see the iconic gestures accompanying the verbal retellings. In a study
focusing on gestures presented without speech (Beattie and Shovolton 2002), a correla-
tion was found between the viewpoint with which a scene was portrayed multimodally
and the communicative effectiveness of the gestures. Gestures produced from character
viewpoint were more informative than those embodying observer viewpoint (see e.g.,
131. Iconic and representational gestures 1737
McNeill 1992; Dancygier and Sweetser 2012). Looking at the interaction of speech and
gesture in the communication of specific semantic features, it was further demonstrated
that character viewpoint gestures were more communicative when conveying features
pertaining to relative position and character viewpoint gestures where more effective in
conveying speed and shape features (Beattie and Shovolton 2001). Moreover, it was
suggested that the effectiveness of TV advertisements may be increased by integrating
spontaneous gestures considering their temporal and semantic properties (Beattie and
Shovolton 2007; see also Beattie 2003). Studies on iconic gestures and speech integration
in aphasics have shown that if comprehension is impaired on the verbal level, gestures
are more heavily relied upon to decode messages. In addition, aphasia may have a dis-
turbing effect on the multimodal integration of information presented in speech and
iconic gestures (Cocks et al. 2009). Eye-movement studies are a way to find out what
kinds of gestures addressees tend to notice more than others, and what they note about
them. Gullberg (2003) found that listener-observers pay particular attention to gestures
representing objects or actions and that the attentive direction of the participants eye-
gaze on the gestures had both cognitive and social motivations (see also Gullberg and
Holmqvist 2006; Gullberg and Kita 2009).
With regards to gestures in the field of forensics, Evola and Casonato (2012) have
suggested that legal transcripts of interviews and interrogations can be compromised by
not taking into account the gestures produced (both by the interviewer and the inter-
viewee) in the interrogation setting. Indeed, gestures are not usually transcribed in de-
posed transcripts. In particular iconic gestures (for example, ones produced during state-
ments of physical descriptions), if properly interpreted, are useful in forensic and psycho-
logical evaluations, in that they may reveal extra information not encoded in speech;
however, ultimately this information often goes unnoticed or unrecorded in the legal
deposition. Moreover, children being interviewed may tend to prefer gesturing over ver-
balizing, especially with regards to taboo topics. In one dispute, for example, a pre-teen
girl being interviewed in an alleged child molestation case is asked by the adult in-
terviewer to describe “what she felt”. Upon insistent questioning, the girl hedges the
question and repeatedly touches her forehead with her straight index finger for almost
four minutes before verbally admitting she felt “a big finger” against her head. By paying
more attention to the interviewee’s gestures, especially iconic ones, the authors suggest
that “hidden” information is revealed, and the child’s own way of communicating is re-
spected.
3. Mimicry: Intersubjective alignment and understanding
A kind of socially oriented, intersubjective iconicity in co-speech gestures may reside in
the ways in which speakers interpret and partly imitate the gestural behavior of their
interlocutors (e.g., McNeill 2005: 160162; see also Calbris 1990: 104153 on the moti-
vated, conventional and cultural aspects of mimetic gestures and Müller (2010a) on the
notion of mimesis as applied to gesture). Kimbara (2006: 41) defines gestural mimicry
as the “recurrence of the same or similar gestures across speakers” and “as an instance
of jointly constructed meaning” (Kimbara 2006: 42). Gesture, like speech, contributes
both form and meaning as shared cognitive and semiotic resources on the bases of which
co-participants build up common ground and unify cultural patterns (Clark 1996). Ges-
tural mimicry is not an automatic or exact duplication of an interlocutor’s behavior, but
VIII. Gesture and language1738
a collaboratively achieved “representational action mediated by meaning” (Kimbara
2006: 58), reinforcing one’s identity of inclusion or exclusion within a social and cultural
setting. In the process, the reoccurrence of particular gesture form features may “make
salient […] those aspects of what is being talked about, and […] influence the way in
which the interlocutor comes to represent and so to conceive of the same referent”
(Kimbara 2006: 58; see also Evola 2010; Parrill and Kimbara 2006). A study by Mol,
Kramer, and Swerts (2009: 4) investigates whether speakers mimic gestures of their inter-
locutors that are inconsistent with the accompanying speech (evidence for “perception-
behavior link”) or those consistent with the representations in speech (evidence for “lin-
guistic alignment”). Results show that almost only gestures that matched the concurrent
speech were repeated. Moreover, participants who had seen inconsistent gestures per-
formed fewer gestures overall. This indicates that “the copying of a gesture’s form is
more likely a case of convergence in linguistic behavior (alignment) than a general in-
stance of physical mimicry” (Mol, Kramer, and Swerts 2009: 7).
Comparing gestural mimicry in face-to-face situations to situations with an invisible
interlocutor, Holler and Wilkin (2011) not only consider shared formal and semantic
features as criteria for gestural mimicry, but also the use of the same mode of representa-
tion. The authors further posit three functions of mimicked gestures: “presentation, ac-
ceptance, and displaying incremental understanding” (Holler and Wilkin 2011: 141).
They conclude that mimicked gestures assume crucial functions in the incremental cre-
ation of mutually shared understanding and “are both part of the common ground
interactants accrue, as well as part of the very process by which they do so” (Holler and
Wilkin 2011: 148; see also Bergmann und Kopp 2010 and Kopp, Bergmann, and Wachs-
muth 2008 on questions of alignment and iconic gestures from the perspective of com-
puter modeling).
4. Representational and reerential gestures
Gesture scholars have proposed various other terms to capture as well as highlight cer-
tain nuances of the kinds of semiotic processes referred to as iconic gestures above. Here,
a selective overview of some of the prominent accounts will be provided in chronological
order, not laying out the complete taxonomies, but focusing on underlying questions of
iconicity and representation instead.
Early on Wundt (1921) divided referential gestures into two different kinds:a) gestures
imitating an object or concept or gestures mimicking an action, for instance by drawing
with the index finger its contours in the air or by evoking through a specific hand config-
uration the plasticity of its characteristic shape (e.g., a cupped hand imitating a small
bowl; andb) connotative gestures that pick out a characteristic feature to refer to the
object or action in its entirety. While both of these processes imply partial representation
and thus metonymy (Mittelberg and Waugh this volume), it is Wundt’s category of sym-
bolic gestures in which figurative aspects and especially metaphor come to the fore (see
also Wundt 1973). Efron ([1941] 1972: 96) distinguished between several types of object-
related gestural behaviors, only some of which have the capacity for pictorial, physio-
graphic representation: “depicting either the form of a visual object or a spatial relation-
ship (iconographic gesture), or that of a bodily action (kinetographic gesture)”. Hence, a
difference is made between what could also be called object images and bodily motor
images; Efron assigns the function of a true icon only to the iconographic type. Building
131. Iconic and representational gestures 1739
on Efron (1972), Ekman and Friesen (1969) attribute considerable importance to the
idea of representation. In their classification of nonverbal behaviors, they distinguish,
inter alia, gestural acts that stand, either iconically or arbitrarily, for something else (i.e.
extrinsically coded acts) from those being significant in and of themselves (i.e. intrinsi-
cally coded acts). Among the various subtypes of speech-accompanying illustrators, three
may fulfill iconic functions (i.e. are extrinsically coded): pictographs, spatial illustrators
and kinetographs; however, only the first type always is iconic: “Pictographs (…) are
iconic because by definition a picture must resemble but cannot be its significant” (1969:
77; see also Fricke 2007; Kendon 2004; Müller 1998a for overviews).
In her work of mimic representations, Calbris (1990: 104115) demonstrates that re-
gardless of the motivated, i.e. iconic, nature of mimetic gestures, they are always also
conventional in the sense that they portray cognitive schemata or cultural practices. That
is, there are cultural differences in which features of a reference object or scene are
selected and encoded for gestural representation, and how exactly one imitates an every
action involving objects such picking up the phone or raising a glass. The author also
draws attention to the schematicity of such gestures afforded by the “powers of abstrac-
tion. […] Even in evoking a concrete situation, a gesture does not reproduce the concrete
action, but the idea abstracted from the concrete reality” (Calbris 1990: 115). Motivation
may also manifest itself in the form of analogous relationships between the meaning (the
signified) and the gesture (the signifier) through isomorphism (see also Fricke 2012;
Lücking 2013; Mittelberg 2006, this volume).
In her functional classification system, Müller (1998a: 89f.) draws on Bühler’s ([1934]
1982) model of communication with its three functions: expressive, referential, and ap-
pellative.Müller (1998a: 110113) accounts for predominantly referential gestures by
making a distinction between those that refer to concrete reference objects, such as physi-
cal entities, behaviors and events, and those that refer to abstracta such as timelines or
financial transactions. She further stresses the fact that the same kind of gesture, e.g., a
tracing gesture outlining a rectangular-shaped structure, may, depending on the concur-
rent speech content, refer to a physical picture frame or a theoretical framework (see
also Müller 2010a; Müller and Cienki 2009). Whether concrete or abstract reference
objects and actions, referential gestures involve abstraction of relevant aspects or a gene-
ral idea. In addition, hand movements may shape and create referential gestures in dif-
ferent ways, thus bringing about iconicity in gestural gestalts. To account for this, Müller
(1998a: 114126; 1998b: 323327) introduced four modes of representation in gesture:
drawing (e.g., tracing the outlines of a picture frame), molding (e.g., sculpting the form
of a crown); acting (e.g., pretending to open a window), and representing (e.g., a flat
open hand with the palm turned up stands for a piece of paper).
According to Kendon (2004: 160), referential gestures may point to what the utterance
is about or represent certain aspects of the propositional content of an utterance. In the
group of manual actions that serve purposes of representation, Kendon (2009) distin-
guishes between two distinct uses. First, there are uses of manual action that “provide
dynamic movement information about the properties of objects or actions the speaker
is talking about”. These may fulfill an adverbial or adjectival function communicating
aspects of the manner of an action or the shape or relative dimensions of a given object
(see Fig. 131.1). Second, manual actions may suggest the presence of concrete objects,
e.g., by placing the items being talked about in space or highlighting aspects of their
relationships (comparable to diagrams or drawings); these uses do not add anything to
VIII. Gesture and language1740
the propositional content of the utterance. In addition, Kendon (2004) distinguishes
different techniques of representation, namely modeling a body part to stand for some-
thing else, enacting certain features of an action pattern, or depicting objects in the air
through movements recognized as sketching or sculpting the shape of something (see
also Streeck 2008, 2009 on depicting by gesture; see also Mittelberg this volume).
While responding to the need of categorizing gestures for the purpose of analysis,
many gesture scholars have come to realize that working with categories, even if seen as
not absolute, poses problems in light of the dynamic, polysemous and multifunctional
nature of gestural forms (cf. Kendon 2004; McNeill 2005; Müller 2010b). This implies,
in principle, that there are no iconic or metaphoric gestures as such, but that these
semiotic principles interact to a certain degree in a given gestural sign and that one needs
to establish, in conjunction with the concurrent speech and other contextual factors,
which one actually determines its local function. The latter understanding expresses, in
alignment with Peirce (1960) and Jakobson (1987), a hierarchical view on processes of
association and signification (e.g., Mittelberg 2008, volume 1; Mittelberg and Waugh
2009, this volume). One should also keep in mind that motivated iconic signs tend to
involve habit and conventionality and could not unfold their meaning if not understood
as indexically embedded in utterance formation, performances, and intricate structures
of embodied interaction with the material and cultural world (e.g., Calbris 1990, 2011;
Streeck, Goodwin, and LeBaron 2011; Sweetser 2012).
5. Concluding remarks
In light of the research reviewed in this chapter, it is useful to take note that the terms
iconic gestures and iconics come from traditional semiotics and as such have their own
specific, and at times complex, meaning in the history of ideas (for details see Mittelberg
this volume; see also Jakobson 1966; Jakobson and Waugh 2002; Sonesson 2008). The
subjective and interpretative nature of iconic gestures, as with icons in other modalities,
is important to keep in mind in order to understand that, when attempting to classify
gestures as such, there can be ambiguity. As to their interpretative aspect, gestures can
be classified as iconics, or as predominantly iconic bodily signs, when there is some form
of similarity between the sign-vehicle (the gesture) and the object which can be seized
by and is salient to an interpretant. The examples provided in the chapter dealing with
cultural iconics, which although are socially motivated, address to what extent they are
habitual, conventional or conventionalized albeit based on similarity. The way people
use their language(s) in their diverse settings motivates their thinking for speaking and
for gesturing (e.g., Slobin 1996; Cienki and Müller 2008; McNeill 2005). As evidenced
by the work reviewed in this chapter, bodily iconic signs metonymically foreground cer-
tain aspects of an object, an idea, or even another gesture and translate them cognitively
and gesturally in a way that may enhance both the speaker’s own understanding of what
s/he is trying to convey as well as the interlocutor’s interpretation. Iconic kinetic action
features, often only consisting of minimal motion onsets or schematic images furtively
traced in the air, thus help co-participants to arrive at shared understandings in dynami-
cally evolving “contextures of action” (Goodwin 2011: 182; see also Enfield 2009, 2011).
In this fashion, different meanings expressed in diverse sign systems (e.g. speech, art, a
scene on the street) become multimodally comprehensible and mutually interpreting,
also allowing people to refer to something outside their own gestural system (cf. Mann-
heim 1999).
131. Iconic and representational gestures 1741
Although iconic gestures occupy a struggled place in the literature, the following
intermodal, cross-modal,interpersonal, intramodal, and intertextual iconic relations and
patterns can be devised: iconic relations between
(i) an individual gestural sign carrier and what it evokes or represents (e.g., iconic
gestures, representational gestures);
(ii) gestures and the concurrent speech content as well as prosodic contours;
(iii) gestural behavior of interlocutors (e.g., mimicry (Kimbara 2006)); as well as iconic
patterns emerging from gestural forms recurring
(iv) within the same discourse (e.g., catchments (McNeill and Duncan 2000) or locution
clusters (Kendon 1972));
(v) across discourses and speakers (e.g., recurrent gestures (Bressem volume 1; Lade-
wig 2011, this volume; Müller 2010b) and geometric and image-schematic patterns
in gesture space (Cienki 2005; Mittelberg 2010, 2013);
(vi) across different languages; and
(vii) across different age groups, clinical groups, social groups, or cultures (see sec-
tion 2).
It seems worthwhile to bring into the picture additional theoretical approaches that
might account for certain properties and functions of gestures more effectively than
similarity and iconicity can (see Fricke 2012; Lücking 2013; Streeck 2009). Contiguity
relations between the communicating body and its material and social habitat also play
an important role in sensing and interpreting the meaning of bodily signs which most of
the time subsume iconic and indexical (and symbolic) functions (Peirce 1960; see Mittel-
berg and Waugh this volume). It thus seems crucial to further examine how gestural
icons are indexically anchored in their semiotic, material and social environment, thus
revealing their subjective and intersubjective dimensions (e.g. Haviland 1993; Streeck,
Goodwin, and LeBaron 2011; Sweetser 2012). One way to do this is to continue compar-
ative semiotic studies investigating the interplay of iconic and other semiotic principles
interacting in both co-speech gesture and signed languages to arrive at a fuller under-
standing of the cognitive, physical, pragmatic and socio-cultural forces that drive proc-
esses of conventionalization and grammaticalization in bodily signs and their grounded,
richly contextualized usage (e.g., Andre
´n 2012; Goldin-Meadow 2003; Grote and Linz
2003; Kendon 2004; Perniss et al. 2010; Sweetser 2009; Wilcox in press; Zlatev 2005).
Considering the interdependent factors of communicative human action and interac-
tion addressed throughout this chapter, the following observations Le
´vi-Strauss made
decades ago may serve as an interim conclusion, since they not only provide historical
anchorage, but may also inspire future work:
both the natural and the human sciences concur to dismiss an out-moded philosophical
dualism. Ideal and real, abstract and concrete, ‘emic’ and ‘etic’ can no longer be opposed to
each other. What is immediately ‘given’ to us is neither the one nor the other, but something
which is betwixt and between, that is, already encoded by the sense organs as by the brain.
(Le
´vi-Strauss [1972] cited in Jakobson and Waugh 2002: 51)
Acknowledgements
The preparation of the article was supported by the Excellence Initiative of the German
State and Federal Governments and the Bonn-Aachen International Center for Informa-
tion Technology (B-IT).
VIII. Gesture and language1742
6. Reerences
Andre
´n, Mats 2010. Children’s Gestures from 18 to 30 Months. Lund: Centre for Languages and
Literatures, Lund University.
Beattie, Geoffrey and Heather Shovolton 1999. Mapping the range of information contained in
the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social
Psychology 18: 438462.
Beattie, Geoffrey 2003. Visible Thought: The New Psychology of Body Language. London:
Routledge.
Beattie, Geoffrey and Heather Shovolton 2001. An experimental investigation of the role of different
types of iconic gesture in communication: A semantic feature approach. Gesture 1: 129149.
Beattie, Geoffrey and Heather Shovolton 2002. An experimental investigation of some properties
of individual iconic gestures that affect their communicative power. British Journal of Psychology
93: 473492.
Beattie, Geoffrey and Heather Shovolton 2007. The role of iconic gesture in semantic communica-
tion and its theoretical and practical implications. In: Susan Duncan, Justine Cassell and Elena
T. Levy (eds.), Gesture and the Dynamic Dimension of Language, 221241. Amsterdam: John
Benjamins.
Bergmann, Kirsten and Stefan Kopp 2010. Systematicity and idiosyncrasy in iconic gesture use:
Empirical analysis and computational modeling. In: Stefan Kopp and Ipke Wachsmuth (eds.),
Gesture in Embodied Communication and Human-Computer Interaction, 182194. Berlin:
Springer.
Bouvet, Danielle 1997. Le Corps et la Me
´taphore dans les Langues Gestuelles: A la Recherche des
Modes de Production des Signes. Paris: L’Harmattan.
Bouvet, Danielle 2001. La Dimension Corporelle de la Parole. Les Marques Posturo-Mimo-Gestuelles
de la Parole, leurs Aspects Me
´tonymiques et Me
´taphoriques, et leur Ro
ˆle au Cours d’un Re
´cit.
Paris: Peeters.
Bressem, Jana volume 1. A linguistic perspective on the notation of form features in gestures. In:
Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill and Sedinha Tessen-
dorf (eds.), Body Language Communication. An International Handbook on Multimodality
in Human Interaction. (Handbooks of Linguistics and Communication Science 38.1.), 1079
1098. Berlin/Boston: De Gruyter Mouton.
Bühler, Karl 1982. Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart, New York: Fi-
scher. First published [1934].
Calbris, Genevie
`ve 1990. The Semiotics of French Gesture: Advances in Semiotics. Bloomington:
Indiana University Press.
Caldognetto, Emanuela Magno and Isabella Poggi 1995. Creative iconic gestures: some evidence
from aphasics. In: Rafaele Simone (ed.), Iconicity in Language, 257276. Amsterdam: John Ben-
jamins.
Cienki, Alan 2005. Image schemas and gesture. In: Beate Hampe (ed.), From Perception to Meaning:
Image Schemas in Cognitive Linguistics, 421442. Berlin: Mouton de Gruyter.
Cienki, Alan in press. Gesture, space, grammar, and cognition. In Peter Auer and Martin Hilpert
(eds.), Space in Language and Linguistics: Geographical, Interactional, and Cognitive Perspectives.
Berlin: Mouton de Gruyter.
Cienki, Alan and Cornelia Müller 2008. Metaphor, gesture, and thought. In Raymond W. Gibbs,
Jr. (ed.), The Cambridge Handbook of Metaphor and Thought, 483501. Cambridge: Cambridge
University Press.
Clark, Herbert H. 1996. Using Language. Cambridge: Cambridge University Press.
Cocks, Naomi, Laetitia Sautin, Sotaro Kita, Gary Morgan and Sally Zlotowitz 2009. Gesture and
speech integration: an exploratory study of a man with aphasia. International Journal of Lan-
guage and Communication Disorders 44: 795804.
131. Iconic and representational gestures 1743
Cocks, Naomi, Lucy Dipper, Ruth Middleton and Gary Morgan 2011. What can iconic gestures
tell us about the language system? A case of conduction aphasia. International Journal of Lan-
guage and Communication Disorders 46: 423436.
Cocks, Naomi, Lucy Dipper, Madeleine Pritchard and Gary Morgan 2013. The impact of impaired
semantic knowledge on spontaneous iconic gesture production. Aphasiology 27 (9): 10501069.
Dancygier, Barbara and Eve E. Sweetser (eds.) 2012. Viewpoint in Language: A Multimodal Perspec-
tive. Cambridge: Cambridge University Press.
Duncan, Susan and Laura Pedelty 2007. Discourse focus, gesture, and disfluent aphasia. In: Susan
Duncan, Justine Cassell and Elena T. Levy (eds.), Gesture and the Dynamic Dimension of Lan-
guage, 269283. Amsterdam: John Benjamins.
Duncan, Susan D. 2002. Gesture, verb aspect, and the nature of iconic imagery in natural discourse.
Gesture 2 (2): 183206.
Duncan, Susan, David McNeill and Karl-Erik McCullough 1995. How to transcribe the invisible
and what we see. In Daniel O’Connell, Sabine Kowal and Roland Posner (eds.), Zeichen für Zeit:
Zur Notation und Transkription von Bewegungsabläufen (special issue of KODIKAS/CODE) 18,
7594. Tübingen: Günter Narr.
Efron, David 1972. Gesture, Race and Culture. The Hague: Mouton and Co. First published [1941].
Ekman, Paul and Wallace Friesen 1969. The repertoire of nonverbal behavior: Categories, origins,
usage and coding. Semiotica 1: 4998.
Enfield, N.J. 2009. The Anatomy of Meaning. Speech, Gestures, and Composite Utterances. Cam-
bridge: Cambridge University Press.
Enfield, N.J. 2011. Elements of formulation. In: Jürgen Streeck, Charles Goodwin and Curtis Le-
Baron, (eds.), Embodied Interaction: Language and the Body in the Material World, 5966. Cam-
bridge: Cambridge University Press.
Evola, Vito 2010. Multimodal cognitive semiotics of spiritual experiences: Beliefs and metaphors in
words, gestures, and drawings. In: Fey Parrill, Vera Tobin and Mark Turner (eds.), Form, Mean-
ing, and Body,4160. Stanford: CSLI Publications.
Evola, Vito and Marco Casonato 2012. Gesture studies on trial:
Applying gesture studies to forensic interrogations and interviews. Presentation at the Interna-
tional Society for Gesture Studies (ISGS) 2012, Lund University, Sweden.
Fo
´nagy, Iva
´n 1956. Über die Eigenart des sprachlichen Zeichens. Lingua 6: 6788.
Fo
´nagy, Iva
´n 196162. Le signe conventionnel motive
´.La Linguistique 7: 5584.
Fricke, Ellen 2007. Origo, Geste und Raum Lokaldeixis im Deutschen. Berlin, New York: de
Gruyter.
Fricke, Ellen 2012. Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin: Mou-
ton de Gruyter
Goldin-Meadow, Susan 2003. Hearing Gesture: How our Hands Help us Think. Cambridge, MA:
Harvard University Press.
Goodwin, Charles 2011. Contextures of action. In: Jürgen Streeck, Charles Goodwin and Curtis
LeBaron (eds.), Embodied Interaction: Language and the Body in the Material World, 182193.
Cambridge: Cambridge University Press.
Grote, Klaudia and Erika Linz 2003. The influence of sign language iconicity on semantic conceptu-
alization. In: Wolfgang G. Müller and Olga Fischer (eds.), From Sign to Signing: Iconicity in
Language and Literature 3, 2340. Amsterdam: John Benjamins.
Gullberg, Marianne 2003. Eye movements and gestures in human interaction. In: Jukka Hyönä,
Ralf Radach and Heiner Deubel (eds.), The Minds’s Eye: Cognitive and Applied Aspects of Eye
Movements, 685703. Oxford: Elsevier.
Gullberg, Marianne and Kenneth Holmquivst 2006. What speakers do and what listeners look at.
Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition
14: 5382.
Gullberg, Marianne and Sotaro Kita 2009. Attention to speech-accompanying gestures: Eye move-
ments and information uptake. Journal of Nonverbal Behavior 33(4): 251277.
VIII. Gesture and language1744
Hadar, Uri and Brian Butterworth 1997. Iconic gestures, imagery and word retrieval in speech.
Semiotica 115: 147172.
Haviland, John 1993. Anchoring, iconicity and orientation in Guugu Yimithirr pointing gestures.
Journal of Linguistic Anthropology 3: 345.
Hogrefe, Katharina, Wolfram Ziegler, Nicole Weidinger and Georg Goldenberg 2012. Non-verbal
communication in severe aphasia: Influence of aphasia, apraxia, or semantic processing? Cortex
48: 952962.
Holler, Judith and Katie Wilkin 2011. Co-speech gesture mimicry in the process of collaborative
referring during face-to-face dialogue. Journal of Nonverbal Behavior 35: 133153.
Jakobson, Roman 1960. Linguistics and poetics. In: Krystyna Pomorska and Stephen Rudy (eds.),
Roman Jakobson Language in Literature,6294. Cambridge, MA: Harvard University Press.
Jakobon, Roman 1966. Quest for the essence of language. In: Linda R. Waugh and Monique Mon-
ville-Burston (eds.), Roman Jakobson: On Language, 407421. Cambridge, MA: Harvard Uni-
versity Press.
Jakobson, Roman 1987. On the relation between auditory and visual signs. In: Krystyna Pomorska
and Stephen Rudy (eds.), Roman Jakobson, Language in Literature, 467473. Cambridge, M.A.:
Harvard University Press.
Jakobson, Roman and Linda R. Waugh 2002. The Sound Shape of Language. Berlin, New York:
Mouton de Gruyter, 3rd. ed. First published [1979].
Kendon, Adam 1972. Some relationships between body motion and speech. An analysis of an
example. In: Aaron Siegman and Benjamin Pope (eds.), Studies in Dyadic Communication, 177
210. Elmsford, NY: Pergamon Press.
Kendon, Adam 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.
Kendon, Adam 2009. Kinesic components of multimodal utterances. Berkeley Linguistics Society
Proceedings. Berkeley, CA: Berkeley Linguistics Society.
Kimbara, Irene 2006. On gestural mimicry. Gesture 6(1): 3961.
Kita, Sotaro 2000. How representational gestures help speaking. In: David McNeill (ed.), Language
and Gesture, 162185. Cambridge: Cambridge University Press.
Kita, Sotaro and Özyürek, Asli 2003. What does cross-linguistic variation in semantic coordination
of speech and gesture reveal? Evidence for an interface representation of spatial thinking and
speaking. Journal of Memory and Language 48: 1632.
Kita, Sotaro and Asli Özyürek 2007. How does spoken language shape iconic gestures? In: Susan
Duncan, Justine Cassell, Elena T. Levy (eds.), Gesture and the Dynamic Dimension of Language,
6781. Amsterdam: John Benjamins.
Kopp, Stefan, Kirsten Bergmann and Ipke Wachsmuth 2008 Multimodal Communication from
Multimodal Thinking Towards an integrated model of speech and gesture production. In-
ternational Journal of Computing 2(1): 115136.
Krauss, Robert M., Chen Yihsiu and Rebecca Gottesman 2000. Lexical gestures and lexical access:
A process Model. In: David McNeill (ed.), Language and Gesture, 261283. Cambridge: Cam-
bridge University Press.
Ladewig, Silva H. 2011. Putting the Cyclic Gesture on a Cognitive Basis. CogniTextes 6.
Ladewig, Silva H. this volume. Recurrent gestures. In: Cornelia Müller, Ellen Fricke, Alan Cienki,
Silva H. Ladewig, David McNeill and Jana Bressem (eds.), Body Language Communication.
An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics
and Communication Science 38.2.). Berlin/ Boston: De Gruyter Mouton.
Lücking, Andy 2013. Ikonische Gesten.Grundzüge einer linguistischen Theorie. Berlin: Mouton de
Gruyter.
Mannheim, Bruce 1999. Iconicity. Journal of Linguistic Anthropology 9(12): 107110.
McNeill, David 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago: Chicago
University Press.
McNeill, David (ed.) 2000. Language and Gesture. Cambridge: Cambridge University Press.
McNeill, David 2005. Gesture and Thought. Chicago: Chicago University Press.
131. Iconic and representational gestures 1745
McNeill, David and Elena Levy 1982. Conceptual representations in Language activity and gesture.
In Robert J. Jarvella and Wolfgang Klein (eds.), Speech, Place, and Action, 271296. Chichester:
John Wiley and Sons.
McNeill, David and Susan Duncan 2000. Growth points in thinking-for-speaking. In: David
McNeill (ed.), Language and Gesture, 141161. Cambridge: Cambridge University Press.
Mittelberg, Irene 2006. Metaphor and Metonymy in Language and Gesture: Discourse Evidence for
Multimodal Models of Grammar (Ph.D. Dissertation, Cornell University). Ann Arbor, MI: UMI.
Mittelberg, Irene 2008. Peircean semiotics meets conceptual metaphor: Iconic modes in gestural
representations of grammar. In: Alan Cienki and Cornelia Müller (eds.), Metaphor and Gesture,
115154. Amsterdam: John Benjamins.
Mittelberg, Irene 2010. Geometric and image-schematic patterns in gesture space. In: Vyvyan Evans
and Paul Chilton (eds.), Language, Cognition, and Space: The State of the Art and New Direc-
tions, 351385. London: Equinox.
Mittelberg, Irene 2013. Balancing acts: Image schemas and force dynamics as experiential essence
in pictures by Paul Klee and their gestural enactments. In: Barbara Dancygier, Mike Bokrent
and Jennifer Hinnell (eds.), Language and the Creative Mind. Stanford: Center for the Study of
Language and Information.
Mittelberg, Irene volume 1. The exbodied mind: Cognitive-semiotic principles as motivating forces
in gesture. In: Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill and
Sedinha Teßendorf (eds.), Body Language Communication: An International Handbook on
Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science
38.1.), 750779. Berlin/Boston: De Gruyter Mouton.
Mittelberg, Irene this volume. Gesture and iconicity. In: Cornelia Müller, Alan Cienki, Ellen Fricke,
Silva H. Ladewig, David McNeill and Jana Bressem (eds.), Body Language Communication.
An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics
and Communcation Science 38.2.). Berlin: De Gruyter Mouton.
Mittelberg, Irene and Linda R. Waugh 2009. Metonymy first, metaphor second: A cognitive-semio-
tic approach to multimodal figures of thought in co-speech gesture. In: Charles Forceville and
Eduardo Urios-Aparisi (eds.), Multimodal Metaphor, 329356. Berlin: Mouton de Gruyter.
Mittelberg, Irene and Linda R. Waugh this volume. Gesture and metonymy. In: Cornelia Müller,
Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill and Jana Bressem (eds.),
Body Language Communication. An International Handbook on Multimodality in Human
Interaction (Handbooks of Linguistics and Communcation Science 38.2.). Berlin: De Gruyter
Mouton.
Mol, Lisette, Emiel Kramer and Marc Swerts 2009. Alignment in iconic gestures, does it make
sense? Proceedings of ACSP 2009 International Conference on Audio-Visual Speech Processing,
University of East Anglia, Norwich, UK, September, 2009.
Müller, Cornelia 1998a. Redebegleitende Gesten. Kulturgeschichte Theorie Sprachvergleich. Ber-
lin Verlag.
Müller, Cornelia 1998b. Iconicity and gesture. In: Serge Santi et al. (eds.), Oralite
´et gestualite
´:
Communication multimodale et interaction, 321328. Montre
´al/Paris: L’Harmattan.
Müller, Cornelia 2010a. Mimesis und Gestik. In: Gertrude Koch, Christiane Voss and Martin
Vöhler, (eds.), Die Mimesis und ihre Künste, 149187. München: Fink.
Müller, Cornelia 2010b. Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische
Perspektive. Sprache und Literatur 41(1): 3768.
Müller, Cornelia and Alan Cienki 2009. Words, gestures and beyond: Forms of multimodal meta-
phor in the use of spoken language. In: Charles Forceville and Eduardo Urios-Aparisi (eds.),
Multimodal Metaphor, 297328. Berlin: Mouton de Gruyter.
Özyürek, Asli, Sotaro Kita, Shanley Allen, Reyhan Furman and Amanda Brown 2005. How does
linguistic framing of events influence co-speech gestures? Insights from cross-linguistic variations
and similarities. Gesture 5(1): 251237.
VIII. Gesture and language1746
Parrill, Fey and Irene Kimbara 2006. Seeing and hearing double: The influence of mimicry in speech
and gesture on observers. Journal of Nonverbal Behavior 30: 157166.
Peirce, Charles Sanders 1960. Collected Papers of Charles Sanders Peirce (19311958). Vol. I.:
Principles of Philosophy, Vol. II: Elements of Logic. Charles Hartshorne und Paul Weiss (eds.).
Cambridge: The Belknap of Harvard University Press.
Perniss, Pamela, Thompson, Robin and Gabriella Vigliocco 2010. Iconicity as a general property
of language: evidence from spoken and signed languages. Frontiers in Psychology 1: 115.
Saussure, Ferdinand de 1986. Course in General Linguistics. 3rd edition. Translated by Roy Harris.
Chicago: Open Court.
Slobin, Dan J. 1996. From ‘thought and language’ to thinking for speaking. In: John J. Gumperz
and Steven Levinson (eds.), Rethinking Linguistic Relativity,7096. Cambridge: Cambridge
University Press.
Sonesson, Göran 2008. Prolegoma to a general theory of iconicity: Considerations of language,
gesture, and pictures. In Klaas Willems and Ludovic De Cuypere (eds.), Naturalness and Iconic-
ity in Language,4772. Amsterdam: John Benjamins.
Streeck, Jürgen 2008. Depicting by gesture. Gesture 8(3): 285301.
Streeck, Jürgen 2009. Gesturecraft: The Manu-Facture of Meaning. Amsterdam: John Benjamins.
Streeck, Jürgen, Charles Goodwin and Curtis D. LeBaron 2011. Embodied Interaction: Language
and Body in the Material World: Learning in Doing: Social, Cognitive and Computational Perspec-
tives. New York: Cambridge University Press.
Sweetser, Eve E. 2009. What does it mean to compare language and gesture? Modalilties and con-
trasts. In: Jiansheng Guo, Elena Lieven, Nancy Budwig, Susan Ervin-Tripp, Keiko Nakamura,
Seyda Özcaliskan (eds.), Crosslinguistic Approaches to the Psychology of Language: Studies in
the Tradition of Dan Isaac Slobin, 357366. New York: Psychology Press.
Sweetser, Eve E. 2012. Viewpoint and perspective in language and gesture. Introduction to Barbara
Dancygier and Eve Sweetser (eds.), Viewpoint in Language: A Multimodal Perspective,122.
Cambridge: Cambridge University Press.
Wilcox, Sherman in press. Signed languages. In: Ewa Dabrowska and Dagmar Divjak (eds.), Hand-
book of Cognitive Linguistics. Berlin: Mouton de Gruyter.
Wundt, Wilhelm 1921. Völkerpsychologie. Eine Untersuchung der Entwicklungsgesetze von Sprache,
Mythos und Sitte, Vol. 1, 4th edition. Stuttgart: Kröner Verlag.
Wundt, Wilhelm 1973. A Psychology of Gesture. Translated by J.S. Thayer, C.M. Greenleaf and
M.D. Silberman from Völkerpsychology, First Volume, Fourth Edition, First Part Chapter 2.
Stuttgart: Alfred Kröner Verlag, 1921. The Hague: Mouton.
Zlatev, Jordan 2005. What’s in a schema? Bodily mimesis and the grounding of language. In: Beate
Hampe (ed.), From Perception to Meaning: Images Schemas in Cognitive Linguistics, 313342.
Berlin: Mouton de Gruyter.
Irene Mittelberg, Aachen (Germany)
Vito Evola, Aachen (Germany)
... It is possible, and indeed often the case, for a communicative form to signal meaning in more than one way: A depiction of someone finger-pointing at something, for example, is both depictive and indicative; conventionalized ideophones such as meow and whack are descriptive as well as depictive (Dingemanse, 2017a); likewise, depicting constructions in signed languages exhibit properties associated with both descriptions and depictions (Ferrara and Hodge, 2018). Depicting, indicating, and describing are therefore better considered, not as discrete categorical notions, but as properties or dimensions of communicative signals, a view that finds advocates in more recent studies (McNeill, 2005;Mittelberg and Evola, 2014). In this sense, a prototypical depiction is really a communicative signal whose depictive property is more salient than its indicative and descriptive properties. ...
... Also covered by Clark's notion of depicting are manual gestures with an iconic form-meaning relation, a topic that has been explored by a great number of researchers, though not necessarily using the term depicting (e.g., Calbris, 1990;McNeill, 1992McNeill, , 2005Gullberg, 1998;Kendon, 2004;Sowa, 2006;Cienki and Müller, 2008). Depending whether the denotation is something concrete in the material world, these gestures are often divided into two separate groups: "iconic" (or "representational, " "referential, " "imagistic") and "metaphoric" (or "conceptual, " "ceiving") (see the reviews in Kendon, 2004;Streeck, 2008;Mittelberg and Evola, 2014). On a more schematic level, Clark's notion of depicting covers also phenomena which are often approached separately and independently, but which share the same defining property of iconicity. ...
Article
Full-text available
The framework of depicting put forward by Clark (2016) offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from American TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance.
... Scholars have argued for different categories of gesture, for example based on their function (referential vs. pragmatic, Kendon 2004;representational vs. discursive, McNeill 1992;Cienki 2013; see also Mittelberg & Evola 2014). Referential or representational gestures have been described as iconics, in the case where a semiotic relationship of similarity is created between the form of the gesture and its referent (the "vase" example); metaphorics, where gestures have a metaphorical relationship with their referent (e.g., pointing up to tell someone to turn "up" the volume"); deictics or pointing gestures to spatially locate concrete objects or persons but also abstract ones (as when assigning discourse referents to different locations in the gesture space, e.g., saying "Ella was here and Farin over there" and pointing to two different locations in space to disambiguating anaphoras of the grammatical sentence) or pointing to objects or persons that used to be present in the physical space ("phantom pointing", e.g., pointing to the chair of someone who left the room while continuing to talk about him). ...
Preprint
Full-text available
The contemporary interdisciplinary domain of gesture studies is heavily rooted in the cognitive linguistics enterprise (Evans, Bergen & Zinken, 2007), especially concerning its “cognitive commitment” (Lakoff, 1990: 40), aligning gesture research within research on cognition. There is a growing body of literature highlighting that gesture may very well be part of language competence, playing a decisive role in language evolution and development (see infra). The dual cognitive and social function of gestures is the primary focus in gesture studies, with gestural phenomena being viewed as offering “a window into the mind” and playing a vital role in social interactions. After providing a historical perspective and a brief overview of some fundamentals of gesture studies, this chapter will focus on functional and formal aspects found in gesture studies highlighting their relationship with cognitive linguistics. The chapter will then discuss the need for research on gestural meaning-making, or “gesture phonology”, and gesture categorisation before highlighting future directions in the area of human computer interaction.
... McNeill (2005) stated that metaphoric gestures or "metaphoricity" is one of the four gestural dimensions along with "iconicity", "deixis" and "beats" that are often mixed in the same gesture (pp. 38-43), and Mittelberg and Evola (2014) asserted that iconic gestures, like metaphoric ones, depict pictorial image schemas, but unlike metaphoric gestures, they portray concrete rather than abstract entities. ...
... Ladewig suggested that distinctions of gesture based on their links to speech are not supported by an analysis of multimodal communication; the form and function of gestural production must be analyzed in its communicative context. Mittelberg and Evola (2014) extend this analysis with their review of the many factors, such as linguistic, discourse, and sociocultural contexts, that can influence the interpretation of the iconicity found in gestures. ...
Article
Full-text available
Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of 30 monolingual English-speaking children between the ages of 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture.
... This includes the analysis of gestures and signs across different naturalistic but also experimental contexts. It presupposes a close semiotic, interactional, and linguistic analysis of all the gestural forms we observe 'in the wild' (Müller, 2010(Müller, , 2016(Müller, , 2017Bressem et al., 2013, see also Mittelberg, 2013Mittelberg and Evola, 2014;Mittelberg and Waugh, 2014) and the multitude of ways in which they are integrated with speech or sign creating simultaneous structures (Vermeerbergen and Demey, 2007), composite utterances (Enfield, 2009(Enfield, , 2013Janzen, 2017), gesture-speech ensembles (Kendon, 2004), or multimodal utterances (Ladewig, 2014a;Ladewig, in press). It also starts from a broader notion of the term gesture than the one suggested by McNeill and Goldin-Meadow. ...
Article
Full-text available
The goal of the article is to offer a framework against which relations between gesture and sign can be systematically explored beyond the current literature. It does so by (a) reconstructing the history of the discussion in the field of gesture studies, focusing on three leading positions (Kendon, McNeill, and Goldin-Meadow); and (b) by formulating a position to illustrate how this can be achieved. The paper concludes by emphasizing the need for systematic cross-linguistic research on multimodal use of language in its signed and spoken forms.
Article
From an enactivist perspective, cognition can be described as embodied, since it is determined by our bodily, multisensory, affective interaction with the environment, in particular by our social interaction. In recent years, interpreting has been defined as a multimodal, embodied cognitive activity of inter-lingual mediation, and research on gestures in conference interpreting has found that simultaneous interpreters, although not visible for their audience, do gesture in the booth. However, gestures in interpreting are yet understudied. This paper presents an exploratory, in-depth descriptive study with the aim of generating hypotheses about the cognitive functions of gestures in simultaneous interpreting. To this end, we investigate the different types of gesture that emerge throughout a whole process of simultaneous interpreting, in conjunction with the concurrent speech, the interpreter’s interaction with her environment and her own description of her production of mental images and gestures. The research question guiding our investigation is: What functions do the different types of gestures play in the interpreting process? The results suggest that, in the analyzed material, referential gestures tend to support the construction of meanings, while the main role of pragmatic gestures consists in helping to manage the progress of the interpreting process.
Preprint
Full-text available
CHAPTER TO APPEAR (2022, in press) in Jamin Pelkey & Paul Cobley (eds), Semiotic Movements (Bloomsbury Semiotics 4). London: Bloomsbury Academic. As semiotic beings, we build our understanding of the discourses we participate in, and more generally of the world around us, by integrating signs in different modalities into dynamic meaningful ensembles. From the first exchanges between infants and caregivers, dialogue typically involves articulations in multiple modalities such as vocalizations, spoken or signed language, manual gestures, eye gaze, body posture and facial expressions. While listening to our conversational partners, we observe their facial expressions; we sense emotional qualities and inner dispositions in the ways they articulate themselves-both verbally and gesturally-and we might respond to them-both verbally and gesturally-or hold ourselves back. Put differently, spontaneous interactive discourse typically consists of concerted multimodal semiotic acts of contextualized meaning-making. In this chapter, we place the human body and its communicative behaviour at the centre of studying multimodal processes of semiosis (e.g., Peirce (e.g., Peirce 1875: CP 1.337; 1907: 5.472, 5.484). We characterize various ways in which the human body functions as sign and sign-creator, bringing together semiotic and related accounts that help us understand how interlocutors gesturally indicate objects, ideas, locations, or events and enact physical habits such as interpersonal interaction, movement patterns, or object manipulation, as well as more abstract, yet deeply embodied, schemata of experience. Gestures here are broadly understood as discourse-embedded, kinesic actions that are performed with the hands and arms, head, shoulders, torso or entire body, and have semiotic function(s). Imagine a conversation in which a friend explains to you that her plan for the weekend is to work. While she speaks, she performs a typing action in mid-air, simulating typing on an imagined keyboard. From her gesture you will likely infer that she will spend her weekend typing on an actual keyboard at an actual desk: i.e., you will infer that, here, "working" means on a computer. You may also imagine the written text that will result from the work, as well as other ideas and states (e.g., your friend's future mental or emotional state when she submits her final manuscript) resulting from her weekend of writing. This decontextualized example nonetheless shows how a gesture can evoke not only an implied object or tool, but also the associated pragmatically structured context of experience. In this chapter, we focus on aspects of the study of gesture most highly relevant to semiotics. After an overview of pioneering research in modern gesture studies, we provide a semiotic characterization of gesture, especially as compared to more highly codified linguistic signs. Drawing on Peirce, we then demonstrate how in gestural signs basic semiotic modes such as iconicity, indexicality, and conventionality/habit interact in modality-specific ways. We further highlight the inherently metonymic nature of gestures and their tendency to schematize experience. We close the chapter with a forward-looking perspective on the field and an overview of new technologies being put to use in semiotic gesture analyses.
Article
Full-text available
This study presents a microanalysis of what information performers “give” and “give off” to each other via their bodies during a contemporary dance improvisation. We compare what expert performers and non-performers (sufficiently trained to successfully perform) do with their bodies during a silent, multiparty improvisation exercise, in order to identify any differences and to provide insight into nonverbal communication in a less conventional setting. The coordinated collaboration of the participants (two groups of six) was examined in a frame-by-frame analysis focusing on all body movements, including gaze shifts as well as the formal and functional movement units produced in the head–face, upper-, and lower-body regions. The Methods section describes in detail the annotation process and inter-rater agreement. The results of this study indicate that expert performers during the improvisation are in “performance mode” and have embodied other social cognitive strategies and skills (e.g., endogenous orienting, gaze avoidance, greater motor control) that the non-performers do not have available. Expert performers avoid using intentional communication, relying on information to be inferentially communicated in order to coordinate collaboratively, with silence and stillness being construed as meaningful in that social practice and context. The information that expert performers produce is quantitatively less (i.e., producing fewer body movements) and qualitatively more inferential than intentional compared to a control group of non-performers, which affects the quality of the performance.
Article
Full-text available
This study examines the form and function of gestural depictions that develop over extended stretches of concept explanation by a philosopher. Building on Streeck’s (2009) explorations of depiction by gesture, we examine how this speaker’s process of exposition involves sequences of multimodal, analogical depiction by which the philosophical concepts are not only expressed through gesture forms, but also dynamically analyzed and construed through gestural activity. Drawing on perspectives of gesture as active meaning making ( Müller 2014 , 2016 , Streeck 2009 ), we argue that the build-up of gestures in depiction sequences, activated through a multimodal metaphor ( Müller & Cienki 2009 ), engages the wider philosophical standpoint of the speaker. Using video analysis supported by interview data, we demonstrate how examination of gestures within and across discourse can lead to understanding of how dynamic, embodied, and subjective processes of conceptualization contribute to philosophical theorizing.
Article
Full-text available
Embodied image schemas are central to experientialist accounts of meaning-making. Research from several disciplines has evidenced their pervasiveness in motivating form and meaning in both literal and figurative expressions across diverse semiotic systems and art forms (e.g., Gibbs and Colston; Hampe; Johnson; Lakoff; and Mandler). This paper aims to highlight structural similarities between, on the one hand, dynamic image schemas and force schemas and, on the other, hand shapes and gestural movements. Such flexible correspondences between conceptual and gestural schematicity are assumed to partly stem from experiential bases shared by incrementally internalized conceptual structures and the repeated gestural (re-) enacting of bodily actions as well as more abstract semantic primitives (Lakoff). Gestures typically consist of evanescent, metonymically reduced hand configurations, motion onsets, or movement traces that minimally suggest, for instance, a PATH, the idea of CONTAINMENT, an IN-OUT spatial relation, or the momentary loss of emotional BALANCE. So, while physical in nature, gestures often emerge as rather schematic gestalts that, as such, have the capacity to vividly convey essential semantic and pragmatic aspects of high relevance to the speaker. It is further argued that gesturally instantiated image schemas and force dynamics are inherently meaningful structures that typically underlie more complex semantic and pragmatic processes involving, for instance, metonymy, metaphor, and frames. First, I discuss previous work on how image schemas, force gestalts, and mimetic schemas may underpin hand gestures and body postures. Drawing on Gibbs’ dynamic systems account of image schemas, I then introduce an array of tendencies in gestural image schema enactments: body-inherent/self-oriented (body as image-schematic structure; forces acting upon the body); environment-oriented (material culture including spatial structures), and interlocutor-oriented (intersubjective understanding). Adopting a dynamic systems perspective (e.g.,Thompson and Varela) thus puts the focus on how image schemas and force gestalts that operate in gesture may function as cognitive-semiotic organizing principles that underpin a) the physical and cognitive self-regulation of speakers; b) how they interact with the (virtual) environment while talking; and c) intersubjective instances of resonance and understanding between interlocutors or between an artwork and its beholder. Examples of these patterns are enriched by video and motion-capture data, showing how numeric kinetic data allow one to measure the temporal and spatial dimensions of gestural articulations and to visualize movement traces.
Conference Paper
Full-text available
Abstract: The Influence of Sign Language Iconicity on Semantic Conceptualization The empirical work described in this paper explores the influence of sign language iconicity on semantic conceptualization processes of deaf and hearing signers of German Sign Language (GSL). The main question addressed is whether the meaning of a sign is independent of iconic aspects of the sign-signifier. This research question is based on the observation that in many cases the forms of the signs of GSL resemble forms of the object, action or event they denote. In fact, the most important way to introduce new signs to the lexicon of GSL is by highlighting one or another aspect of a certain referent. However, it is not clear if the iconicity of a sign-signifier preserves during its lexicalization process an influence on the sense of the sign, i.e. on the strength of semantic relations to other concepts in the cognitive system. In a Peirceian framework, our main theoretical question can be reformulated as follows: 1. Does a sign loose its function as an 'icon' through transformation into a linguistic 'legisign' and 2. does the iconic relation between the sign and its referent retain an influence on the interpretant after the sign´s integration into the language system. In the first experiment it was examined whether deaf and hearing signers of German Sign Language (GSL) and hearing speakers of German Spoken Language (GSL) (n=60) show different response times (RT) in a verification task where they had to judge the presence or absence of a semantic relation between a reference item (sign vs. spoken word) and a target item (picture). All reference items in Sign Language were iconic and highlighted a certain aspect of the referent they denoted, whereas their translational equivalents in spoken language were arbitrary. The combined target items (3 different pictures: e.g. window / door / roof) were all semantically related to the reference item (e.g. house), but only one of the pictures actually resembled the prior presented sign in GSL (e.g. roof). The experimental material consisted of 120 pairs composed of 20 reference items (signs vs. spoken words) combined with 20 x 3 semantically related pictures and 20 x 3 semantically non-related pictures (distractor items). In the second study it was examined whether the deaf and hearing subjects show different choices in a task asking them to decide which of two presented pictures has a stronger semantic relation to a target item (sign vs. spoken word). The relative number of choices in favor for a specific picture were measured. The results of both studies support the hypothesis that the iconicity of a sign has an influence on the structure and organization of the semantic network of deaf and hearing signers. To be more specific, the features and properties of a certain object which are highlighted by physical and spatial aspects of the sign, are more central in the semantic network in which the meaning of the sign is embedded. The results will be discussed with respect to theoretical implications about the relation of iconicity and arbitrariness of linguistic signs.
Chapter
Based on spoken academic discourse and its accompanying gestures, this chapter presents a cognitive-semiotic approach to multimodal communication that assigns equal importance to metaphor and metonymy. Combining traditional semiotics with contemporary cognitivist theories, we demonstrate how these two figures of thought jointly structure multimodal representations of grammatical concepts and structures. We discuss Jakobson's view of metaphor and metonymy, and particularly his distinction between internal and external metonymy, thus discerning various principles of sign constitution and indirect reference within metaphoric gestures (whether or not the concurrent speech is metaphorical). We then introduce a dynamic two-step interpretative model suggesting that metonymy leads the way into metaphor: in order to infer the imaginary objects or traces that gesturing hands seem to hold or draw in the air, a metonymic mapping between hand (source) and imaginary object (target) is a prerequisite for the metaphorical mapping between that very object (source) and the abstract idea (target) it represents.