No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Discourse markers in relation tonon-verbal behavior: How do speech and body language correlate?
Abstract
The research proposed in this paper focuses on pragmatic interlinks between discourse markers and non-verbal behavior. Although non-verbal behavior is recognized to add non-redundant information and social interaction is not merely recognized as the transmission of words and sentences, the evidence regarding grammatical/linguistic interlinks between verbal and non-verbal concepts are vague and limited to restricted domains. This is even more evident when non-verbal behavior acts in the foreground but contributes to the structure and organization of the discourse. This research focuses on investigating the multimodal nature of discourse markers by observing their linguistic and paralinguistic properties in informal discourse. We perform a quantitative analysis with case studies for representative cases. The results show that discourse markers and background non-verbal behavior tend to follow a similar functionality in interaction. Therefore, by examining them together, one gains more insight into their true intent despite the high multifunctionality of both non-verbal behavior and DMs.
... To quote Cooperrider, "They are in the background of the speaker's awareness, in the background of the listener's awareness and the background of the interaction" (Cooperrider, 2017, p. 7). Despite this, they convey conversational intent (Cooperrider, 2017, p. 191) or meaning which helps the addressee understand the message as the sender intended (e.g., a hesitant and slow nod of the head accompanying the utterance "right" suggest to the recipient that it should not be interpreted as an affirmative answer) (Mlakar et al., 2021). Background gestures, in particular, serve in providing suggestive influences and give a certain degree of clarity to the overall human-human discourse. ...
... To facilitate our motivation, we first apply a modular and extendable EVA (embodied virtual agent) Scheme (Mlakar et al., 2019) using the annotation tools ELAN (Wittenburg et al., 2006) 1 and WebAnno(Eckart de Castilho et al., 2016). 2 The conversational phenomena are observed on a comprehensive multimodal corpus that contains real-life, near authentic, multiparty discourse with spontaneous responses. In this way, we quantify the conversational phenomena into verbal and non-verbal cues into action items (Riggio & Riggio, 2012): (i) linguistic annotations, i.e., segmentation (token, utterance, turn), sentence type, sentiment, parts of speech tags (POS), syntax, and discourse markers (DMs); (ii) paralinguistic signals, i.e., communicative intent, emotions, prosodic phrases, and accentuation (primary accent (PA) and secondary accent (NA)); (iii) management and social signals, i.e., person/relation, dialog acts, and (iv) visual cues and non-verbal communicative intent (NCI) (Mlakar et al., 2021). ...
This paper focuses on gaining new knowledge through observation, qualitative analytics, and cross-modal fusion of rich multi-layered conversational features expressed during multiparty discourse. The outlined research stems from the theory that speech and co-speech gestures originate from the same representation; however, the representation is not solely limited to the speech production process. Thus, the nature of how information is conveyed by synchronously fusing speech and gestures must be investigated in detail. Therefore, this paper introduces an integrated annotation scheme and methodology which opens the opportunity to study verbal (i.e., speech) and non-verbal (i.e., visual cues with a communicative intent) components independently, however, still interconnected over a common timeline. To analyse this interaction between linguistic, paralinguistic, and non-verbal components in multiparty discourse and to help improve natural language generation in embodied conversational agents, a high-quality multimodal corpus, consisting of several annotation layers spanning syntax, POS, dialogue acts, discourse markers, sentiment, emotions, non-verbal behaviour, and gesture units was built and is represented in detail. It is the first of its kind for the Slovenian language. Moreover, detailed case studies show the tendency of metadiscourse to coincide with non-verbal behaviour of non-propositional origin. The case analysis further highlights how the newly created conversational model and the corresponding information-rich consistent corpus can be exploited to deepen the understanding of multiparty discourse.
As in most other languages, clicks in Hebrew are not phonemes but still occur very frequently in speech. This Interactional Linguistic study, based on a corpus of casual Hebrew conversation, explores clicks at discourse unit junctures-prefacing topics, sequences, and their parts. We argue that the function of such clicks is best understood through Goffman's notion of 'frame shifts,' signalling changes in footing. Some frame-shifting clicks are byproducts of swallowing and breathing; at the same time, we show that click use is regular in certain contexts and interactionally significant, and that such clicks qualify as discourse markers. The tension between being non-linguistic epiphenomena and having a syntactic status is what makes frame-shifting clicks 'liminal signs.' Finally, we hypothesize that frame-shifting clicks are explainable as the result of a reanalysis in specific bridging contexts, whereby clicks-as-byproducts are conventionalized into discourse markers. This process may explain why non-phonemic clicks are so widespread across the world, and it may serve as a starting point for describing the development of additional interactional functions of clicks.
Native listeners benefit from both visible speech and iconic gestures to enhance degraded speech comprehension (Drijvers & Ozyürek, 2017). We tested how highly proficient non-native listeners benefit from these visual articulators compared to native listeners. We presented videos of an actress uttering a verb in clear, moderately, or severely degraded speech, while her lips were blurred, visible, or visible and accompanied by a gesture. Our results revealed that unlike native listeners, non-native listeners were less likely to benefit from the combined enhancement of visible speech and gestures, especially since the benefit from visible speech was minimal when the signal quality was not sufficient.
Actions may be used to directly act on the world around us, or as a means of communication. Effective communication requires the addressee to recognize the act as being communicative. Humans are sensitive to ostensive communicative cues, such as direct eye gaze (Csibra & Gergely, 2009). However, there may be additional cues present in the action or gesture itself. Here we investigate features that characterize the initiation of a communicative interaction in both production and comprehension. We asked 40 participants to perform 31 pairs of object-directed actions and representational gestures in more- or less- communicative contexts. Data were collected using motion capture technology for kinematics and video recording for eye-gaze. With these data, we focused on two issues. First, if and how actions and gestures are systematically modulated when performed in a communicative context. Second, if observers exploit such kinematic information to classify an act as communicative. Our study showed that during production the communicative context modulates space-time dimensions of kinematics and elicits an increase in addressee-directed eye-gaze. Naïve participants detected communicative intent in actions and gestures preferentially using eye-gaze information, only utilizing kinematic information when eye-gaze was unavailable. Our study highlights the general communicative modulation of action and gesture kinematics during production but also shows that addressees only exploit this modulation to recognize communicative intention in the absence of eye-gaze. We discuss these findings in terms of distinctive but potentially overlapping functions of addressee directed eye-gaze and kinematic modulations within the wider context of human communication and learning.
Language Related Episodes occur when speakers explicitly question lexical and grammatical aspects of the language they are using, resulting in collaborative discourse and assisted performance from peers. This paper demonstrates how such negotiation and repair may occur in relation to the gestural component of a speaker's expression, leading us to introduce the parallel term ‘Gesture Related Episodes’. Our single case analysis reveals a range of issues that have received little attention, including the problems that people experience with gestures, what constitutes struggling in the gestural modality, and how people help each other to gesture more effectively in collaborative discourse. Our discussion links these issues to the L2 concerns and knowledge asymmetries in our data, as well as to conceptual and conventional features of gestures more generally.
Though the positive effects of iconic gestures on word recall and comprehension by children have been clearly established, less is known about the benefits of beat gestures (rhythmic hand/arm movements produced together with prominent prosody). This study investigated (a) whether beat gestures combined with prosodic information help children recall contrastively focused words as well as information related to those words in a child-directed discourse (Experiment 1) and (b) whether the presence of beat gestures helps children comprehend a narrative discourse (Experiment 2). In Experiment 1, fifty-one 4-year-old children were exposed to a total of three short stories with contrastive words presented in three conditions, namely with prominence in both speech and gesture, prominence in speech only, and non-prominent speech. Results of a recall task showed (a) that children remembered more words when exposed to prominence in both speech and gesture than in either of the other two conditions, and (b) that children were more likely to remember information related to those words when the words were associated with beat gestures. In Experiment 2, fifty-five 5- and 6-year-old children were presented with six narratives with target items either produced with prosodic prominence but no beat gestures or produced with both prosodic prominence and beat gestures. Results of a comprehension task demonstrated that stories told with beat gestures were comprehended better by the children. Together, these results constitute evidence that beat gestures help preschoolers not only to recall but also to comprehend discourse information.
Recent comparative semantics suggests that sign language makes use of the same logical resources as spoken language, but has richer mechanisms of iconic enrichment. It is often thought that when speech is analyzed in tandem with iconic gestures, ‘expressive parity’ is regained between speech and sign. But this comparison between sign-with-iconicity and speech-with-gestures turns out to be complex because iconic enrichments often have a non-at-issue status, as attested by their interaction with logical operators. We argue that the status of iconic enrichments is constrained by two parameters: ±internal, ±separate_time_slot. If an enrichment is effected by the internal modification of an expression (+internal)—e.g. by lengthening the word loooong in English, or the sign GROW in ASL—it can have any semantic status and can in particular be at-issue. If an enrichment is an external addition to an expression (-internal)—as is the case of co-speech gestures in English—it does not make an at-issue contribution, but it may have the status of a presupposition or of a supplement. If an enrichment has a separate time slot (+separate_time_slot), it may not be trivial (= presupposed), and must thus be at-issue or supplemental. The generalization is assessed on the basis of vocal iconicity in spoken language, iconic modulations in sign language, co-speech/co-sign as well as post-speech/post-sign gestures and facial expressions in spoken and sign language, and also gestures that fully replace words in spoken language. Our typology suggests that there are systematic expressive differences between sign-with-iconicity and speech-with-gestures, and also that the semantic status of iconic enrichments can in part be predicted by parameters pertaining to their form.
With expanding evidence on the formulaic nature of human communication, there is a growing need to extend discourse marker research to functionally analogue multi-word expressions. In contrast to the common qualitative approaches to discourse marker identification in corpora, this paper presents a corpus-driven semi-automatic approach to identification of multi-word discourse markers (MWDMs) in the reference corpus of spoken Slovene. Using eight statistical measures, we identified 173 structurally fixed discourse-marking MWEs, distinguished by a high number of tokens, a large proportion of grammatical words and semantic heterogeneity. This is a significantly longer list than would have been gained by manual inspection of smaller corpus samples. Although frequency-based methods produced satisfactory results, best precision in MWDM identification was achieved using the t-score association measure, while the overall poor performance of the mutual information suggests its inadequacy for extraction of MWDMs and other MWEs with similar lexical and distributional features.
This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-recorded first acquaintance conversations between Danish speakers. The annotation includes speech transcription including word boundaries, and formal as well as functional coding of gestural behaviours, specifically head movements, facial expressions, and body posture. The corpus has served as the empirical basis for a number of studies of communication phenomena related to turn management, feedback exchange, information packaging and the expression of emotional attitudes. We describe the annotation scheme, procedure, and annotation results. We then summarise a number of studies conducted on the corpus. The corpus is available for research and teaching purposes through the authors of this article.
Co-speech gestures are ubiquitous: when people speak, they almost always produce gestures. Gestures reflect content in the mind of the speaker, often under the radar and frequently using rich mental images that complement speech. What are gestures doing? Why do we use them? This book is the first to systematically explore the functions of gesture in speaking, thinking, and communicating – focusing on the variety of purposes served for the gesturer as well as for the viewer of gestures. Chapters in this edited volume present a range of diverse perspectives (including neural, cognitive, social, developmental and educational), consider gestural behavior in multiple contexts (conversation, narration, persuasion, intervention, and instruction), and utilize an array of methodological approaches (including both naturalistic and experimental). The book demonstrates that gesture influences how humans develop ideas, express and share those ideas to create community, and engineer innovative solutions to problems.
Do speakers intend their gestures to communicate? Central as this question is to the study of gesture, researchers cannot seem to agree on the answer. According to one common framing, gestures are an "unwitting" window into the mind (McNeill, 1992); but, according to another common framing, they are designed along with speech to form "composite utterances" (Enfield, 2009). These two framings correspond to two cultures within gesture studies—the first cognitive and the second interactive in orientation—and they appear to make incompatible claims. In this article I attempt to bridge the cultures by developing a distinction between foreground gestures and background gestures. Foreground gestures are designed in their particulars to communicate a critical part of the speaker's message; background gestures are not designed in this way. These are two fundamentally different kinds of gesture, not two different ways of framing the same monolithic behavior. Foreground gestures can often be identified by one or more of the following hallmarks: they are produced along with demonstratives; they are produced in the absence of speech; they are co-organized with speaker gaze; and they are produced with conspicuous effort. The distinction between foreground and background gestures helps dissolve the apparent tension between the two cultures: interactional researchers have focused on foreground gestures and elevated them to the status of a prototype, whereas cognitive researchers have done the same with background gestures. The distinction also generates a number of testable predictions about gesture production and understanding, and it opens up new lines of inquiry into gesture across child development and across cultures.
Gestures are fundamental to the way we communicate, yet our understanding of this communicative impulse is clouded by a number of ingrained assumptions. Are gestures merely ornamentation to speech? Are they simply an 'add-on' to spoken language? Why do we gesture? These and other questions are addressed in this fascinating book. McNeill explains that the common view of language and gesture as separate entities is misinformed: language is inseparable from gesture. There is gesture-speech unity. Containing over 100 illustrations, Why We Gesture provides visual evidence to support the book's central argument that gestures orchestrate speech. This compelling book will be welcomed by students and researchers working in linguistics, psychology and communication.
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
This study studies how gestures (here defined in a wide sense, including all body movements which have a communicative function) are used for Own Communication Management (OCM), an interesting and not completely well described part of the language system. OCM concerns how a speaker continuously manages the planning and execution of his/her own communication and is a basic function in face-to-face interaction. It has two main functions, i.e. "choice" and "change". The study investigates how much of OCM involves gestures and whether there is a difference between choice and change OCM in this respect. It also concerns what kinds of gestures are used in OCM and what the relation is between vocal and gestural OCM. Some of the main findings are that roughly 50% of all speech based OCM cooccurs with gestures and that most of the OCM involving gestures (about 90%) is choice directed. Gestures occurring with OCM can illustrate the content of a sought after word, but also more generally induce word activation. They can also signal to an interlocutor that the speaker needs time. Gestures are often multifunctional and, thus, both choice and change are often integrated with more interactive functions. A final observation is that gestural OCM either precedes or occurs simultaneously with verbal OCM.
Sign language descriptions that use an analytic model borrowed from spoken language structural linguistics have proved to be not fully appropriate. Pictorial and action-like modes of expression are integral to how signed utterances are constructed and to how they work. However, observation shows that speakers likewise use kinesic and vocal expressions that are not accommodated by spoken language structural linguistic models, including pictorial and action-like modes of expression. These, also, are integral to how speaker utterances in face-to-face interaction are constructed and to how they work. Accordingly, the object of linguistic inquiry should be revised, so that it comprises not only an account of the formal abstract systems that utterances make use of, but also an account of how the semiotically diverse resources that all languaging individuals use are organized in relation to one another. Both language as an abstract system and languaging should be the concern of linguistics.
Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic gestures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages.
A taxonomy of hand gestures and a digital tool (CodGest) are proposed in order to describe different types of gesture used by speaker during speech in different social contexts. It is an exhaustive and mutually exclusive categories system to be shared within the scientific community to study multimodal signals and their contribute to the interaction. Classical taxonomies from gesture literature were integrated within a comprehensive taxonomy, which was tested in five different social contexts and its reliability was measured across them through inter-observer agreement indexes. A multi-media tool was realized as digital support for coding gestures in observational research.
In this chapter we provide evidence that speech-synchronized gestures are major sources of discourse cohesion. The evidence comes from a range of gesture types, including representational, beat, and pointing gestures. Our analyses suggest that gestures mark information that is newsworthy relative to what has come before, and thus that synchronous, co-expressive gesture-speech units are equally discourse units; they have absorbed meaningful context as a matter of their formation. Of special importance to our analyses are catchments, threads of visuospatial action imagery that run through a discourse, and are embodied in gesture features such as space, handedness, and hand shape. Catchments provide a gesture-based window onto discourse cohesion, and our analyses of particular instances suggest they are a locus around which discourse is organized. We conclude by exploring the origins of speech-gesture integration—early gestural catchments—in a young child’s spontaneous discourse.
Discourse markers, a group of pragmatic elements, are used widely in our daily communication. Among these markers, well is found to be the most frequently used one in conversation. Although much research has been carried out on its theoretical bases such as its definition, functions and categorizations, few studies have taken a multimodal approach in analyzing this marker. The aim of the present study was to provide a multimodal description of the use of well as a discourse marker in spontaneous speech. Firstly, we wanted to investigate whether the use of this marker in conversation, regardless of its function, was accompanied with any kind of non-verbal behavior (posture shift, head shift, gaze and eyebrow movement) and secondly, we wanted to determine whether any of these non-verbal cues (if any) were specific to one pragmatic function of well and if they could be used as a signal to disambiguate the pragmatic function of well. The study of discourse markers is considered important due to the fact that it can contribute both to dialogue interpretation and generation resulting in a more natural dialogue modeling useful for human-infocommunications interface.
The relationships between text or talk and the context are among the basic fields of pragmatic research and an insight into their nature may contribute to a better understanding of language use. In this article, we use the results of an analysis of discourse marker use in two different conversational genres (telephone conversation and television interviews) in an attempt to examine the impact of context on the use of discourse markers, generalized for each analysed genre. In the first stage of the analysis, we observe important differences between the two genres: discourse markers are far more frequently used in telephone conversations than in television interviews. In the second stage of the analysis, we identify several contextual factors which contribute to the differences in the use of discourse markers. In this way, we obtain insight into this particular aspect of genre context-talk relationships, and identify some of the characteristics of the genres in question.
A context-of-use study is reported of the ‘head shake’. A large number of examples are described and compared, drawn from video recordings of naturally occasioned interactions in the circumstances of everyday life, made in Campania, Italy, central England and the Eastern United States. Eight different kinds of uses for the head shake are illustrated. It is concluded that the head shake is not to be understood simply as the kinesic equivalent of a unit of verbal expression. It appears as an expression in its own right which, furthermore, the speaker uses as a component in the construction of an utterance which, it seems, is so often a
multimodal
construction in which the different modalities of expression available are
deployed
by the speaker in the course of building a unit of expression according to the rhetorical needs of the interactive moment.
This paper aims to determine whether iconic tracing gestures produced while speaking constitute part of the speaker’s communicative intention. We used a picture description task in which speakers must communicate the spatial and color information of each picture to an interlocutor. By establishing the
necessary minimal content
of an intended message, we determined whether speech produced with concurrent gestures is less explicit than speech without gestures. We argue that a gesture must be communicatively intended if it expresses necessary information that was nevertheless omitted from speech. We found that speakers who produced iconic gestures representing spatial relations omitted more required spatial information from their descriptions than speakers who did not gesture. These results provide evidence that speakers intend these gestures to communicate. The results have implications for the cognitive architectures that underlie the production of gesture and speech.
The kappa statistic is frequently used to test interrater reliability. The importance of rater reliability lies in the fact that it represents the extent to which the data collected in the study are correct representations of the variables measured. Measurement of the extent to which data collectors (raters) assign the same score to the same variable is called interrater reliability. While there have been a variety of methods to measure interrater reliability, traditionally it was measured as percent agreement, calculated as the number of agreement scores divided by the total number of scores. In 1960, Jacob Cohen critiqued use of percent agreement due to its inability to account for chance agreement. He introduced the Cohen's kappa, developed to account for the possibility that raters actually guess on at least some variables due to uncertainty. Like most correlation statistics, the kappa can range from -1 to +1. While the kappa is one of the most commonly used statistics to test interrater reliability, it has limitations. Judgments about what level of kappa should be acceptable for health research are questioned. Cohen's suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable. Kappa and percent agreement are compared, and levels for both kappa and percent agreement that should be demanded in healthcare studies are suggested.
Co-speech gestures are ubiquitous: when people speak, they almost always produce gestures. Gestures reflect content in the mind of the speaker, often under the radar and frequently using rich mental images that complement speech. What are gestures doing? Why do we use them? This book is the first to systematically explore the functions of gesture in speaking, thinking, and communicating – focusing on the variety of purposes served for the gesturer as well as for the viewer of gestures. Chapters in this edited volume present a range of diverse perspectives (including neural, cognitive, social, developmental and educational), consider gestural behavior in multiple contexts (conversation, narration, persuasion, intervention, and instruction), and utilize an array of methodological approaches (including both naturalistic and experimental). The book demonstrates that gesture influences how humans develop ideas, express and share those ideas to create community, and engineer innovative solutions to problems.
This volume brings together a series of studies that focus on multimodality in various communicative settings, with special attention to how non-verbal elements reinforce and/or add meaning to verbal expressions. Following an introductory chapter that presents an overview of the topic and summarizes the individual contributions, the volume is structured into two parts. The five chapters in the first part explore issues related to the use of multimodal resources in educational interactions and English language classroom teaching, also involving learners with disabilities. In the five chapters of the second part, multimodality is investigated as a key component of communication that takes place in different specialized domains (e.g., political discourse, legal discourse) and genres (e.g., fictional genres such as live-action and animated films vs. authentic forms of communication, such as TED Talks, political interviews and courtroom trials). The contributions reflect a variety of methodological approaches that are grounded in both quantitative and qualitative techniques. These include multimodal discourse analysis, multimodal transcription, and multimodal annotation software capable of representing the interplay of different semiotic modes, i.e. speech, intonation, direction of gaze, facial expressions, hand/arm gesturing, and spatial positioning of interlocutors.
The research of the contributors to this volume highlights the increasingly important role of multimodality in communication across different genres and communicative contexts, and offers new perspectives on how to exploit multimodal resources to enhance the learning of English for both general and specific purposes.
Discourse markers - the particles oh, well, now, then, you know and I mean, and the connectives so, because, and, but and or - perform important functions in conversation. Dr Schiffrin's approach is firmly interdisciplinary, within linguistics and sociology, and her rigourous analysis clearly demonstrates that neither the markers, nor the discourse within which they function, can be understood from one point of view alone, but only as an integration of structural, semantic, pragmatic, and social factors. The core of the book is a comparative analysis of markers within conversational discourse collected by Dr Schiffrin during sociolinguistic fieldwork. The study concludes that markers provide contextual coordinates which aid in the production and interpretation of coherent conversation at both local and global levels of organization. It raises a wide range of theoretical and methodological issues important to discourse analysis - including the relationship between meaning and use, the role of qualitative and quantitative analyses - and the insights it offers will be of particular value to readers confronting the very substantial problems presented by the search for a model of discourse which is based on what people actually say, mean, and do with words in everyday social interaction.
There are few aspects of language which are more problematic than its discourse particles. The present study of discourse particles draws upon data from the London-Lund Corpus to show how the methods and tools of corpora can sharpen their description. The first part of the book provides a picture of the state of the art in discourse particle studies and introduces the theory and methodology for the analysis in the second part of the book. Discourse particles are analysed as elements which have been grammaticalised and as a result have certain properties and uses. The importance of linguistic and contextual cues such as text type, position in the discourse, prosody and collocation for analysing discourse particles is illustrated. The following chapters deal with specific discourse particles (now, oh, just, sort of, and that sort of thing, actually) on the basis of their empirical analysis in the London-Lund Corpus. Examples and extended extracts from many different text types are provided to illustrate what discourse particles are doing in discourse.
The present paper describes a corpus for research into the pragmatic nature of how information is expressed synchronously through language, speech, and gestures. The outlined research stems from the ‘growth point theory’ and ‘integrated systems hypothesis’, which proposes that co-speech gestures (including hand gestures, facial expressions, posture, and gazing) and speech originate from the same representation, but are not necessarily based solely on the speech production process; i.e. ‘speech affects what people produce in gesture and that gesture, in turn, affects what people produce in speech’ ([1]: 260). However, the majority of related multimodal corpuses ‘ground’ non-verbal behavior in linguistic concepts such as speech acts or dialog acts. In this work, we propose an integrated annotation scheme that enables us to study linguistic and paralinguistic interaction features independently and to interlink them over a shared timeline. To analyze multimodality in interaction, a high-quality multimodal corpus based on informal discourse in a multiparty setting was built.
Discourse markers have a central role in planning and repairing processes of speech production. They relate with fluency and disfluency phenomena such as pauses, repetitions and reformulations. Their polyfunctionality is challenging and few form-function mappings are stable cross-linguistically. This study combines a functional and a structural approach to discourse markers and their combination with and within repetitions and self-repairs in native English, French and Spanish, in order to establish the inter-relation between these three fluency-related devices and to find potentially universal patterns of use. Qualitative coding and quantitative analyses of categories of markers and repairs allowed us to identify discourse markers which are specific to repair sequences and others which are much more pervasive. Combinations with repetitions vary across languages and repair types. Our findings fill a gap in cross-linguistic fluency research, disentangle the overlap between discourse markers, repairs and repetitions, and can be integrated into pedagogical materials.
This corpus-based study examines the widely-used discourse marker well in Chinese-speaking learners’ speech and compares its frequencies in native speaker data and Swedish learners. While Swedish learners overuse well, Chinese-speaking learners (predominantly at the upper-intermediate level) significantly underuse it. The positions and functions of well are further examined using a functional framework. One-fourth of the Chinese-speaking learners who use well manipulate its positions in utterances in a similar way to native speakers. In terms of functions, well is employed for speech management much more frequently than for attitudinal purposes. The greater use of the former does not generally create negative effects, but the under-representation of the latter may suggest that Chinese-speaking learners sound too direct in certain contexts. The paper concludes by considering pedagogical implications for different first languages and proficiency levels and their possible applications to the instruction of well.
The study investigates how the brain processes self-adaptors, semantically-unrelated emblems, and iconic gestures along with speech. The three types of gestures give rise to a continuum of semantic distinctions in relation to the accompanying speech. The overall N400 component occurred between 500 and 800 msec after the simultaneous gesture and speech onsets. In comparison to the speech-only condition, the reduced N400 evidenced the facilitation effect of iconic gestures at the centro-parietal sites. The meaningful yet non-speech-related emblems elicited enhanced N400 s at the left frontal-parietal sites; the meaningless self-adaptors produced the largest N400 effect over the scalp at the frontal-parietal sites. Self-adaptors had produced a larger negativity of N400 than emblems did at the centro-parietal regions. The results evidence the automatic integration of gesture and speech, and the diverse influence of gesture on processing. Only iconic gestures facilitate the semantic integration with speech. For a linguistic meaning to integrate with a semantically-unrelated emblem is less effortful than with a self-adaptor, suggesting that the processing of meanings proceeds more readily than the processing of a meaningless gesture occurring at the same time with speech.
Keevallik’s impressive survey of how body movements affect grammatical choices is a timely reminder that language use in social interaction does not occur in a vacuum. Yet although body movements can be intercalated in complex ways with the grammatical structure of utterances, I argue here that they are not part of grammar in a strict sense of the word. In “composite” utterances they fill slots that grammatical structures create, without being grammatical elements themselves.
This article navigates the findings of conversation analysis, interactional linguistics, and related multimodal studies to summarize what we know about the grammar-body interface. It shows how grammar is fitted to sequences and trajectories of embodied activities, as well as deployed interchangeably with bodily displays, resulting in truly multimodal patterns that emerge in real time. These findings problematize both the paradigmatic and syntagmatic structures documented in verbal-only linguistics. They call for a reconceptualization of grammar as an assembly of routinized methods for the organization of vocal conduct, capable of incorporating aspects of participants’ bodily behavior. Data are in Estonian, French, German, Italian, Japanese and Swedish.
While recent research has shown that iconic gestures and the so-called pitch gestures (or gestures that mimic melody in speech) favour word learning in a second language, little is known about (a) the potential benefits of beat gestures (or hand gestures that accompany prosodic prominence) for second language novel word learning, and (b) the contribution of prosodic prominence (independently or in combination with gestural prominence) to this effect. This study investigates the effects of prosodic prominence (e.g. focal pitch accent) and visual prominence (e.g. beat gesture) on L2 novel vocabulary learning. In a within-subjects design, 96 Catalan-dominant native speakers were asked to learn 16 Russian words in four conditions, namely the presence or absence of prosodic prominence in speech (L+H* pitch accent) combined with the presence or absence of visual prominence (beat gesture). The results of recall and recognition tasks conducted after a training session showed that the strongest effect corresponded to target words presented with visual prominence together with prosodic prominence; by contrast, the condition involving visual prominence with no prosodic prominence triggered smaller effects than the condition involving prosodic prominence alone. Thus, beat gestures produced naturally (that is, accompanied by focal pitch accent in speech) favour second language vocabulary learning. The results have implications for second language instruction practices and multisensory integration and working memory models.
This paper sketches the use of simple discourse-deictic there in the history of English and shows that – in contrast to the frequent and varied employment of discourse-deictic there in there-compounds such as therein, thereby or textual therefore in written genres – simple there was only rarely and restrictedly used with discourse-deictic reference until the 19th century. In Present-Day English, discourse-deictic there, as in you are wrong there, is almost exclusively found in face-to-face interaction of a particular type, which may be labelled ‘issue(s)-at-debate discussion type’, such as TV and radio broadcast discussions or council or staff meetings. For these communicative situations, two particular pragmatic functions are identified, both of them discourse-organizational in nature: Speakers may signal their wish to expand on or enforce an argument which has been neglected or misinterpreted in the immediately preceding discourse or they index their wish of ‘end of topic (or even discourse)’ by simple discourse-deictic there. These functions are linked to there being deictic, an element of the ‘field of pointing’, and its inherent pointing functions to a distal place/space.
This study investigates face-to-face interaction among Taiwanese, Indonesian, and Indian speakers utilizing a multimodal corpus linguistics approach to examine semantic categories of speech that most frequently co-occur with gestures, and whether the gesture-speech relationship is to a certain extent influenced by language/culture backgrounds or English proficiency levels of a speaker. The analysis of the semantic categories of the co-gesture speech demonstrates that speech most commonly co-occurs with gestures in the categories of moving, coming and going, general objects, numbers, location and direction, and time. The findings demonstrate similar preferences of gesture-speech production by speakers despite different cultural and linguistic backgrounds. The gesture-speech relationship was shown to fall into six discrete categories: reinforcing, integrating, supplementary, complementary, contradictory, and others. While results show that the gesture-speech relationship is not significantly influenced by different language backgrounds of a speaker, speakers at a high proficiency level tended to use significantly more gestures that serve reinforcing and integrating functions, whereas less proficient speakers produced more gestures as complements and other gestures that have no obvious relationship to the conceptual content of their accompanying speech.
The concept of discourse markers (DMs) is far from being settled in the linguistic community. There are two main subsets of linguistic expressions that are usually labeled by the term: expressions that index the utterances to the speaker and/or the hearer; and expressions that signal a relation between two discourse segments. This chapter uses the label DMs to refer to the second subset of expressions. It assumes that DMs are relevant in the recognition of discourse relations, that is, relations that hold together different segments of a discourse, building up its coherence. The chapter also assumes that DMs link utterances, i.e., discourse units, and not clauses within a complex sentence. It aims to sketch a synthesis of the most widely discussed properties of DMs, to relate classes of DMs with discourse relations, through clear examples from the Portuguese language, and to outline future research in the area.
Co-speech gestures are ubiquitous: when people speak, they almost always produce gestures. Gestures reflect content in the mind of the speaker, often under the radar and frequently using rich mental images that complement speech. What are gestures doing? Why do we use them? This book is the first to systematically explore the functions of gesture in speaking, thinking, and communicating – focusing on the variety of purposes served for the gesturer as well as for the viewer of gestures. Chapters in this edited volume present a range of diverse perspectives (including neural, cognitive, social, developmental and educational), consider gestural behavior in multiple contexts (conversation, narration, persuasion, intervention, and instruction), and utilize an array of methodological approaches (including both naturalistic and experimental). The book demonstrates that gesture influences how humans develop ideas, express and share those ideas to create community, and engineer innovative solutions to problems.
Discourse markers, comprising items like yes, well, but, uh, oh, and uh-huh, are notoriously difficult to define, and the functional spectrum suggested for these items is huge, including functions as diverse as marking discourse boundaries, managing the turn-taking system, signaling speaker attitude, connecting text segments, and managing repair.
This chapter reviews three influential perspectives in research on discourse markers: Schiffrin's discourse perspective, Fraser's pragmatic approach, and Maschler's functional interactional linguistics perspective. These three approaches are compared with respect to the ways they account for the sources of discourse markers, metalanguage, prosody, the boundary between discourse markers and conjunctions, the relationship between discourse markers and contextual realms, and the integration of discourse-marker analysis into the study of language. This is followed by a review of a considerable subset of recent studies dealing with discourse markers as they are employed across different contexts, across languages (including code-switching at discourse markers in bilingual discourse), and over time (including both diachronic and synchronic studies of grammaticization processes undergone by discourse markers). The chapter concludes with a consideration of what different approaches to the study of discourse markers teach us about conceptions of discourse and grammar
Given its usage-oriented character, Cognitive Grammar (CG) can be expected to be consonant with a multimodal, rather than text-only, perspective on language. Whereas several scholars have acknowledged this potential, the question as to how speakers’ gestures can be incorporated in CG-based grammatical analysis has not been conclusively addressed. In this paper, we aim to advance the CG-gesture relationship. We first elaborate on three important points of convergence between CG and gesture research: (1) CG’s conception of grammar as a prototype category, with central and more peripheral structures, aligns with the variable degrees to which speakers’ gestures are conventionalized in human communication. (2) Conceptualization, which lies at the basis of grammatical organization according to CG, is known to be of central importance for gestural expression. In fact, all of the main dimensions of construal postulated in CG (specificity, perspective, profile-base relationship, conceptual archetypes) receive potential gestural expression. (3) CG’s intensive use of diagrammatic notation allows for the incorporation of spatial features of gestures. Subsequently, we demonstrate how CG can be applied to analyze the structure of multimodal, spoken-gestured utterances. These analyses suggest that the constructs and tools developed by CG can be employed to analyze the compositionality that exists within a single gesture (between conventional and more idiosyncratic components) as well as in the grammatical relations that may exist between gesture and speech. Finally, we raise a number of theoretical and empirical challenges.
I use the term the embodied turn to mean the point when interest in the body became established among researchers on language and social interaction, exploiting the greater ease of video recording. This review article tracks the growth of “embodiment” in over 400 articles published in Research on Language and Social Interaction from 1987 to 2013. I consider closely two areas where analysts have confronted challenges and how they have responded: settling on precise and analytically helpful terminology for the body, and transcribing and representing the body, particularly its temporality and manner.
The article reports on a mixed-methods study evaluating the use of a three-dimensional digital game-based language learning (3D-DGBLL) environment to teach German two-way prepositions and specialized vocabulary within a simulated real-world context of German recycling and waste management systems. The study assumed that goal-directed player activity in this environment would configure digital narratives, which in turn would help study participants in the experimental group to co-configure story maps for ordering and making sense of the problem spaces encountered in the environment. The study further assumed that these participants would subsequently rely on the story maps to help them structure written L2 narratives describing an imagined personal experience closely resembling the gameplay of the 3D-DGBLL environment. The study found that immersion in the 3D-DGBLL environment influenced the manner in which the second language was invoked in these written narratives: Participants in the experimental group produced narratives containing more textual indicators describing the activity associated with the recycling and waste management systems and the spaces in which these systems are located. Increased usage of these indicators suggest that participants in the experimental group did indeed rely on story maps generated during 3D gameplay to structure their narratives, although stylistic and grammatical features of the narratives suggest, however, that changes could be made to the curricular implementation of the 3D-DGBLL environment. The study also puts forward ideas for instructional best practices based on research findings and suggests future areas of development and investigation.
Because many studies of small talk (and talk in general) focus on the input of main speakers, the verbal behavior of listeners is often underrepresented in descriptions of interaction. The notion of small talk as talk superfluous to transactional exigencies enables us to encompass a variety of phenomena, including phatic exchanges, relational language, and various types of insertion sequence. This article adds to this range of phenomena by examining a set of high-frequency short listener response tokens that fulfill the criteria of being superfluous to transactional needs, of being focused on the interpersonal plane of discourse, and of having social functions that seem to overlap with those of phatic and relational episodes in different types of talk. Probably because the items involved are themselves "small" (in that their position is often difficult to locate on the cline from back-channels to full turns), their relational importance is easily overlooked.
Interactional analysts have long argued for the importance of tying techniques, which function to connect the current speaker's utterance to the actions of a previous speaker, in the organization of turns at talk (M. H. Goodwin, 1990; Sacks, 1992c). The organization of embodied actions through such dialogic tying, however, has received far less attention, a gap addressed by this article in its examination of one nonverbal tying technique: dialogic embodied action. In this phenomenon, coparticipants purposefully take up and selectively reproduce particular features of one another's gestures and instrumental actions. Drawing on data from instructional interactions at a bicycle-repair shop, the analysis demonstrates that focusing on the selectivity of such reproductions elucidates two functions of these dialogic actions: (a) to organize intersubjective engagement, facilitating coparticipants' enactment of aligned participant roles and (b) to structure sequential organization through actions that are visibly constituted as prior to other actions.[Supplementary materials are available for this article. Go to the publisher's online edition of Research on Language and Social Interaction for the following free supplemental resource(s): video clips.]
Conversational involvement refers to the degree to which participants in a communicative exchange are cognitively and behaviorially engaged in the topic, relationship, and/or situation. It is argued that involvement should be viewed from a functional perspective and conceptualized as entailing fiue dimensions: immediacy, expressiveness, interaction management, altercentrism, and social anxiety. Specific nonverbal behaviors that are actually encoded to express involvement along these five dimensions are examined within an interview context. Unacquainted dyads (N=52) engaged in baseline interviews followed by a second interview in which one participant was asked to increase or decrease involvement significantly. Tiuentyone kinesic, proxemic, and vocalic behaviors were rated during five intervals. Change scores from baseline to manipulations shouted numerous differences between high and low involvement, as did correlations between magnitude of involvement and nonverbal behaviors. The behaviors that most strongly discriminated high from low involvement were general kinesic/proxemic attentiveness, forward lean, relaxed laughter, coordinated speech, fewer silences and latencies, and fewer object manipulations. Behaviors most predictive of magnitude of involvement change were facial animation, vocal warmth/interest, deeper pitch, less random movement, and more vocal attentiveness.
This paper presents an integrative approach to the study of discourse coherence which follows the observation of Bühler (1934) and others that language use always involves both the representation of propositional content and the expression of attitudes and intentions. Consideration of only one of these functions is shown to be insufficient for an adequate account of discourse coherence.Coherence is regarded as arising from semantic relations between the ideas states and pragmatic relations between the actions performed in speaking or writing. Empirical evidence for this view is provided by the use of pragmatic and ideational structuring devices in film descriptions. Speakers who were describing a film to a friend used more markers of pragmatic structure than those whose listener was a stranger. At the same time, they were less explicit in indicating the ideational structure of their discourse. This trade-off between pragmatic and ideational structuring occurred not only in dialogues (two-way auditory channel where the friends gave more feedback, but also in monologues (one-way channel), where the two conditions differed only in the speaker's knowledge that the listener was a friend or a stranger.
A rapidly expanding body of research deals with a functionally related class of connective expressions commony referred to as discourse markers. The items typically treated in this research include non-truth-conditional uses of forms such as English well, so, and now. While it is widely agreed that such expressions play a variety of important roles in utterance interpretation, there is disagreement in regard to such fundamental issues as how the discourse marker class should be delimited, whether the items in question comprise a unified grammatical category, what type of meaning they express, and the sense in which such expressions may be said to relate elements of discourse. This paper reviews the principal issues in this research area with reference to several prominent frameworks in which discourse markers and closely related items have been studied.