Irene Mittelberg

RWTH Aachen University · Institute of English Studies & Center for Sign Language and Gesture

Zusammenfassung. Dieser Beitrag versucht eine zeichentheoretische Untersuchung von Diagrammen, wie sie im Rahmen der motion-capture-gestützten, empirischen Ges­tenforschung zur Repräsentation und Analyse redebegleitender Gesten erzeugt und verwendet werden. Nach Vorüberlegungen zum Diagrammbegriff und einer Darlegung herkömmlicher Mittel der visuel...

Bloomsbury Semiotics offers a state-of-the-art overview of the entire field of semiotics by revealing its influence on a wide range of disciplinary perspectives. With four volumes spanning theory, method and practice across the disciplines, this definitive reference work emphasizes and strengthens common bonds shared across intellectual cultures, a...

This paper investigates how embodied image schemas and force dynamics to operate and interact in the computer game Antichamber (Bruce (2013)). Our main aim is to show how some of the most pervasive image and force schemata may serve as cognitive-semiotic tools to discern different kinds of dynamicorientationprocessesbasedoncentralproblem-solvingstr...

CHAPTER TO APPEAR (2022, in press) in Jamin Pelkey & Paul Cobley (eds), Semiotic Movements (Bloomsbury Semiotics 4). London: Bloomsbury Academic. As semiotic beings, we build our understanding of the discourses we participate in, and more generally of the world around us, by integrating signs in different modalities into dynamic meaningful ensemb...

Cognitive bias in depression may increase sensitivity to judgmental appraisal of communicative cues. Nonverbal communication encompassing co-speech gestures is crucial for social functioning and is perceived differentially by men and women, however, little is known about the effect of depression on the perception of appraisal. We investigate if a c...

Generating natural embodied conversational agents within virtual spaces crucially depends on speech sounds and their directionality. In this work, we simulated directional filters to not only add directionality, but also directionally adapt each phoneme. We therefore mimic reality where changing mouth shapes have an influence on the directional pro...

During naturalistic production, sign language users regulate their hand velocity, slowing down for lexical productions compared to transitions. We asked signers and nonsigners to produce grooming gestures and pseudosigns (i.e., nonexistent but permissible DGS signs) in tandem with videos and measured hand velocity for item production versus transit...

This paper presents an account of how Peirce’s Universal Categories (UCs) of perception and experience may, as heuristic principles, inform gesture theory and multimodal analysis. Peirce’s UCs – Firstness (possibility), Secondness (actuality), and Thirdness (law, habit) – constitute the core of his phenomenology and thus also the foundation of his...

This paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of g...

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce’s Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers w...

Presentation on how interlocutors modify their head movement, such as nods and shakes in response to their dialogue partner’s utterances. Employing analysis of motion-capture data we look for for timing, intensity and frequency of head movements to differentiate phases of discourse.

Embodied image schemas are central to experientialist accounts of meaning-making. Research from several disciplines has evidenced their pervasiveness in motivating form and meaning in both literal and figurative expressions across diverse semiotic systems and art forms (e.g., Gibbs and Colston; Hampe; Johnson; Lakoff; and Mandler). This paper aims...

Redebegleitende Gesten sind spontane Bewegungen, insbesondere von Armen und Händen, die das Sprechen begleiten und auf verschiedenen sprachlichen Ebenen eng mit ihm verbunden sein können. Der Anthropologe Adam Kendon (1980: 207) nennt diese Art von Bewegungen gesticulation. Nach Kendon handelt es ich dabei um zwei Manifestationen desselben zugrunde...

This paper lays out the foundations of a frame-based account of gesture pragmatics through detailing how frames and metonymy interact not only in motivating gestural sign formation, but also in guiding crossmodal processes of pragmatic inferencing. It is argued that gestures recruiting frame structures tend to profile deeply embodied, routinized as...

In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substit...

Face-to-face communication is multimodal; it encompasses spoken words, facial expressions, gaze, and co-speech gestures. In contrast to linguistic symbols (e.g., spoken words or signs in sign language) relying on mostly explicit conventions, gestures vary in their degree of conventionality. Bodily signs may have a general accepted or conventionaliz...

In light of the growing importance of digital design tools, this paper puts back into focus the human body’s internalized knowledge of the material and social world. It exploits the semiotic potential of operative manufacts, understood as encompassing both sketches on paper and gestures evolving in real time and space. Operative manufacts are seen...

Taking an Emergent Grammar (Hopper 1998) approach to multimodal usage events in face-to-face interaction, this paper suggests that basic scenes of experience tend to motivate entrenched patterns in both language and gesture (Fillmore 1977; Goldberg 1998; Langacker 1987). Manual actions and interactions with the material and social world, such as gi...

Metaphor theory has shifted from asking whether metaphor is 'conceptual' or 'linguistic' to debating whether it is 'embodied' or 'discursive'. Although recent work in the social and cognitive sciences has yielded clear opportunities to resolve that dispute, the divide between discourse- and cognition-oriented approaches has remained. To unite the f...

In language switching, it is assumed that in order to produce a response in one language, the other language must be inhibited. In unimodal (spoken-spoken) language switching, the fact that the languages share the same primary output channel (the mouth) means that only one language can be produced at a time. In bimodal (spoken-signed) language swit...

Viewpoint has been shown to be a powerful construal mechanism in multimodal spoken and signed discourse, as well as in various other modalities and genres. This paper investigates embodied viewpoint strategies that have been observed when speakers combine speech, gestures, postures, gaze, and simulated action to describe their interaction with spat...

One of the most fundamental challenges when accessing gestural patterns in 3D motion capture databases is the definition of spatiotemporal similarity. While distance-based similarity models such as the Gesture Matching Distance on gesture signatures are able to leverage the spatial and temporal characteristics of gestural patterns, their applicabil...

The question of how to model spatiotemporal similarity between gestures arising in 3D motion capture data streams is of major significance in currently ongoing research in the domain of human communication. While qualitative perceptual analyses of co-speech gestures, which are manual gestures emerging spontaneously and unconsciously during face-to-...

The construct of iconic gestures, those gestures understood as sharing certain form features with the object, action or scene they represent, has traditionally proven to be a useful tool for scholars to classify this subset of gestures, distinguishing them from other types such as indexical or emblematic gestures. More recent approaches prefer to a...

The approach presented in this paper aims to contribute to an account of the three-dimensionality of gesture space. Here, gesture space is assumed to be dynamically constructed and adaptive to the communicative situation. Making use of an optical motion-capture system, volumetric representations of gesture spaces were generated, based on gesture da...

This paper explores if people perceive and perform touchless gestures differently when communicating with technology vs. with humans. Qualitative reports from a lab study of 10 participants revealed that people perceive differences in the speed of performing gestures, sense of enjoyment, feedback from the communication target. Preliminary analysis...

Hand, Hirn und Werkzeug - eine komplexe Interaktion. Dieser aus der Architekturpraxis initiierte Band untersucht aus der Perspektive verschiedener Wissensdisziplinen, inwiefern die manuelle Intelligenz des Menschen kreative Prozesse befördert. Besteht in Zeiten programmierter Umgebungen ein Widerspruch zwischen instrumentellem Handeln und intuitive...

This paper explores how na?ve observers recognize and interpret transitive actions (actions involving manipulation of objects) without accompanying speech, in order to derive guidelines for the design of gesture interpretation systems. Semi-structured interviews with 11 observers, interpreting 106 video clips of transitive actions elicited unstaged...

This paper builds on a strand of gesture research that regards form as the basis of any meaningful interpretation of gesture. Gesture form is considered to be derived from body part (ar-ticulator) configurations and movement. We propose that gesture form is multi-layered and can be described economically by form primitives (zero to three dimensions...

The importance of bodily movements in the production and perception of communicative actions has been shown for the spoken language modality and accounted for by a theory of communicative actions (Cogn. Process. 2010;11:187–205). In this study, the theory of communicative actions was adapted to the sign language modality; we tested the hypothesis t...

This paper explores how interaction with systems using touchless gestures can be made intuitive and natural. Analysis of 912 video clips of gesture production from a user study of 16 subjects communicating transitive actions (manipulation of objects with or without external tools) indicated that 1) dynamic pantomimic gestures where imagined tool/ob...

Based on spoken academic discourse and its accompanying gestures, this chapter presents a cognitive-semiotic approach to multimodal communication that assigns equal importance to metaphor and metonymy. Combining traditional semiotics with contemporary cognitivist theories, we demonstrate how these two figures of thought jointly structure multimodal...

Based on spoken American academic discourse and its accompanying gestures videotaped during linguistics courses, this paper presents a cognitive-semiotic approach to multimodal communication that assigns equal importance to metaphor and metonymy. Combining traditional semiotics (Peirce [1960] and Jakobson [1956, 1961, 1963]) with contemporary cogni...

Methods in Cognitive Linguistics is an introduction to empirical methodology for language researchers. Intended as a handbook to exploring the empirical dimension of the theoretical questions raised by Cognitive Linguistics, the volume presents guidelines for employing methods from a variety of intersecting disciplines, laying out different ways of...

This study represents an interdisciplinary attempt to trace the origins of metaphorical expressions as they are found in English grammatical terminology. The perspective chosen combines cognitive and art-historical approaches to the meaning of abstract concepts: cognitive semantics and iconography. It takes into account the historical dimension of...

System und Rede Die Gestenforschung ist ein vergleichsweise junges Feld, das sich vor allem in und zwischen den Disziplinen Linguistik, Kognitive Psychologie, Anthropologie und Semiotik bewegt. Geht man von einem primär linguistischen Ansatz aus, dann ist das sprachlich Konkrete eine Äußerung, die raumzeitlich situiert und intersubjektiv wahrnehmba...

Thesis (Ph.D.)--Cornell University, Jan., 2006. Includes bibliographical references.