Chapter

Multimodal (Inter)action Analysis

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Multimodal (Inter)action Analysis was developed to study social interaction based upon the theoretical notion of mediated action. Building on this core concept, Multimodal (Inter)action Analysis includes several theoretical/methodological tools. These tools facilitate analysis which moves flexibly between micro-level moments of interaction and macro-level practices and discourses. In this chapter, the application of mediated action to multimodal analysis is discussed, before the central theoretical/methodological tools are introduced. Tight links are made between the tools used in Multimodal (Inter)action Analysis and the core theoretical tenets, to support robust multimodal interaction research.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... People or things around us form the basis of any relationship between two or more people from birth till we are no more. Jesse [5] postulated that social interaction is one of the most critical factors for the health and development of everybody, from young children to the elderly and that positive social interactions have a wide range of both physical and mental benefits, including increased cognitive ability, good mental health, communication skills, independence, and improved physical health, especially in older people. ...
Article
Full-text available
Poor-quality Early Childhood Education Centre (ECEC) can be detrimental to the development of children, as they may lead to poor social, emotional, educational, health, physical, and behavioral outcomes and the general well-being of children. However, this research examined the influence of quality ECEC centres on the social interaction of pre-primary school pupils in Kwara State. This study examines evidence on the influence of the quality of the ECE Centre on the Social interaction of pre-primary school pupils in Kwara State. The quantitative approach was adopted for the study while data was collected through a Questionnaire on the Quality of the ECE Centre and Observation rating scales on social interaction. Data collected were analyzed through descriptive statistics of frequency count and percentage scores. The results revealed that quality ECE Centre positively influenced social interaction between pre-primary school pupils and the overall development of children. At the same time, the study recommends that schools provide excess hands-on materials and allocate more play time and well enough space for play activities to foster children’s social interaction through quality ECE Centre, especially in Kwara State.
Article
Full-text available
The paper explores the concept of multimodality in contemporary stylistics of virtual communication, mainly focusing on the process of interpretation. After outlining the basics of multimodal meaning-making theory, it further develops it by incorporating ideas from the theory of media or other related disciplines. The next part describes the of immersive potential of virtual environment, tying it to the multisensorial nature of human epistemic existence. Finally, the last part introduces these ideas to the process of stylistic interpretation, emphasizing the changes and problems, which multimodal virtual communication introduces.
Article
Full-text available
FULL TEXT HERE: https://rfmv.fr/numeros/5/articles/11-visual-transcription/ The communication and language sciences have decidedly taken a multimodal turn and a proliferation of work in previously language-dominated fields is focusing on the contribution of non-verbal communicative modes in social interaction. While this has proven empirically and theoretically fruitful, it throws an additional kink in an already complicated issue: transcription practices and the interpretive and representational process accompanying said practices. In this article, we review and champion a methodological framework that provides analytical tools and a transcription protocol for the generation of visual transcripts. Visual transcription, we contest, is far more congruent with the multimodal ethos and we both detail the method while applying it to a data segment from a larger project investigation teamwork and collaborative problem-solving in dyadic videoconferencing interactions. The analytical focus on action coupled with the accompanying transcription method reveals complex and fluctuating distributions of attention and interactional awareness. Non-verbal actions reveal contributions to the task at hand which are not realized in talk and the analysis details an intricate ebb and flow of attentional orientations which are realized through non-verbal means.
Article
Full-text available
Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse . Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development . Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics . Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . Routledge, London. https://doi.org/10.4324/9780203379493 ; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis . de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280 ; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality . Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77 ) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.
Article
Full-text available
Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse . Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development . Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics . Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . Routledge, London. https://doi.org/10.4324/9780203379493 ; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis . de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280 ; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality . Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77 ) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.
Chapter
Full-text available
This article problematizes the notion of selecting micro-data pieces to shed light upon the focus of participants. The issue presented is two-fold: 1. The article shows that selecting micro-analytical data pieces does not allow a researcher to determine the focus of a participant; and 2. The article demonstrates that language use of a participant does not necessarily mean that the participant is focused upon a conversation. Both, purely working with micro-analytical data pieces and the presumption that language use indicates focus of the speaker, are problematized and it is shown with an example from a relatively large study of family Skype conversation that includes 82 participants that: 1. Focused attention can only be analyzed correctly when crossing micro-analytical boundaries; and 2. A participant can utilize language without paying focused attention to an interaction.
Conference Paper
Full-text available
This paper reports on analysis from a corpus of audio-video recorded interactions during a collaborative building task. The task generates distinct knowledge asymmetries which motivate interaction toward acquiring shared understandings. The analysis suggests that the convergence of the communicative modes of posture and gaze is crucial to producing shared knowledge. These findings support claims that there are no fixed norms for gaze distribution and postural orientation in interaction, but that these are heavily influenced by the environment and task. Furthermore, the findings suggest participants prioritise producing communicative intersubjectivity over perceptual intersubjectivity. The implications of these findings for the nature of intersubjectivity and research into teamwork are considered.
Article
Full-text available
ABSTRACT Recent research has highlighted several areas where pragmatics plays a central role in the process of acquiring a first language. In talking with their children, adults display their uses of language in each context, and offer extensive feedback on form, meaning, and usage, within their conversational exchanges. These interactions depend critically on joint attention, physical co-presence, and conversational co-presence - essential factors that help children assign meanings, establish reference, and add to common ground. For young children, getting their meaning across also depends on realizing language is conventional, that words contrast in meaning, and that they need to observe Grice's cooperative principle in conversation. Adults make use of the same pragmatic principles as they solicit repairs to what children say, and thereby offer feedback on both what the language is and how to use it.
Article
Full-text available
Gesture, or visible bodily action that is seen as intimately involved in the activity of speaking, has long fascinated scholars and laymen alike. Written by a leading authority on the subject, this 2004 study provides a comprehensive treatment of gesture and its use in interaction, drawing on the analysis of everyday conversations to demonstrate its varied role in the construction of utterances. Adam Kendon accompanies his analyses with an extended discussion of the history of the study of gesture - a topic not dealt with in any previous publication - as well as exploring the relationship between gesture and sign language, and how the use of gesture varies according to cultural and language differences. Set to become the definitive account of the topic, Gesture will be invaluable to all those interested in human communication. Its publication marks a major development, both in semiotics and in the emerging field of gesture studies.
Book
Full-text available
From emails relating to adoption over the Internet to discussions in the airline cockpit, the spoken or written texts we produce can have significant social consequences. The area of mediated discourse analysis considers the actions individuals take with texts – and the consequences of those actions. Discourse in Action brings together leading scholars from around the world in the area of mediated discourse analysis and reveals ways in which its theory and methodology can be used in research into contemporary social situations. Each chapter explores real situations and draws on real data to show how the analysis of concrete social actions broadens our understanding of discourse. Taken together, the chapters provide a comprehensive overview to the field and offer a range of current studies that address some of the most important questions facing students and researchers in linguistics, education, communication studies and other fields.
Article
Full-text available
Using data from more than ten years of research, David McNeill shows that gestures do not simply form a part of what is said and meant but have an impact on thought itself. Hand and Mind persuasively argues that because gestures directly transfer mental images to visible forms, conveying ideas that language cannot always express, we must examine language and gesture together to unveil the operations of the mind.
Book
A guide that offers a step-by-step process to data-driven qualitative multimodal discourse analysis Systematically Working with Multimodal Data is a hands-on guide that is theoretically grounded and offers a step-by-step process to clearly show how to do a data-driven qualitative Multimodal Discourse Analysis (MDA). This full-color introductory textbook is filled with helpful definitions, notes, discussion points and tasks. With illustrative research examples from YouTube, an Experimental and a Video Ethnographic Study, the text offers many examples of how to deal with small to large amounts of data, including information on how to transcribe video data multimodally, including online videos, and how to analyze the data. This textbook contains ample theory, directions for literature, and a teaching guide to help with a clear understanding of how to work with multimodal data. Contains new research data, exceptional illustrations and diagrams Offers step-by-step processes of working through examples, transcriptions and online videos Goes into great depth so that students can use the book as hands-on material to engage with their own data analysis Designed to be easy-to-use with color-coded definitions, tasks, discussion points and notes Written for advanced undergraduate, graduate and PhD level students, as well as participants in research workshops, Systematically Working with Multimodal Data is an authoritative guide to understanding data-driven qualitative Multimodal Discourse Analysis.
Article
In this article, I detail incremental microgenetic alterations in the development of one particular socio-interactive aptitude: making a relevant interactive contribution. Taking heed of Clark’s (2014) call for the need to reorient our attention to investigate the pragmatics of interaction by accounting for the multiple communicative modes through which this is acccomplished I detail the ways in which parental facilitation and a flexible participatory configuration, made possible by video-conferencing technology, create conditions enabling the agentive re-introduction of a psycho-socially relevant topic. Paramount are the ways in which residual interactive specificities in introduction, co-production and multimodal configurations re-manifest suggesting a more symbiotic relationship between traditional notions of ‘message’ and ‘production’. During the microgenesis of interactive aptitudes, children are not just learning what constitutes psycho-socially relevant topoi, they also acquire an understanding of exactly how to make the contribution through multimodal ensembles.
Article
Building on multimodal (inter)action analysis as a theoretical and methodological framework, this article introduces and develops the theoretical/methodological tool called primary agency. Taking the mediated action as a unit of analysis, agency can be analysed as a feature of action. However, there is a lack of empirical approaches for the study of agency, and an overemphasis on language as the most important site for identifying agentive action. I develop primary agency through an analysis of three co-produced higher-level actions from a research project into high school tutoring. These are the higher-level actions of conducting research, tutoring and reading a text. Applying co-production and the modal density foreground/background continuum I explore how the researcher, the tutor and the student co-produce these higher-level actions. Through this analysis, I identify the most significant mediational means for each higher-level action, and the social actor with ownership or agency over these mediational means. I define this social actor as the one with primary agency over the co-produced higher-level action. Finally, my analysis outlines the implications of primary agency for co-produced higher-level actions, including the role of the researcher, the attention/awareness participants pay to overarching research projects, and links between primary agency and successful learning.
Article
This article provides a preliminary answer to exactly why video-conferencing is evaluated as better than traditional telephony for long-distance familial interaction by allocating analytical attention to the showing of objects during interaction. While it is acknowledged that ‘showing’ constitutes an interactive move less contingent on linguistic maturation, more importantly, the showing of objects, artefacts or entities during video-conferencing interactions exemplifies an agentive and volitional production of identity elements on behalf of young children. Thus, while some have pointed to shortcomings of conversation-like activities mediated by video-conferencing in favour for more activity-driven tasks, I make a case for drawing upon pre-existing components of the material surround as a means to more comprehensively and longitudinally engage younger children in video-conferencing interaction.
Article
This article takes a multimodal approach to examine how two young men communicate knowledge, shift attention, and negotiate a disagreement via videoconferencing technology. The data for the study comes from a larger ongoing project of participants engaging in various tasks together. Linking micro, intermediate and macro analyses through the various methodological tools employed, the article presents multimodal (inter)action analysis (Norris, 2004, 2011, 2013a, 2013b) as a methodology to gain new insight into the complexity of knowledge communication via videoconferencing technology, which is relevant to many settings from education to employment, from organizations to gaming.
Article
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
Article
Researchers seeking to analyse how intersubjectivity is established and maintained face significant challenges. The purpose of this article is to provide theoretical/methodological tools that begin to address these challenges. I develop these tools by applying several concepts from multimodal (inter)action analysis to an excerpt taken from the beginning of a tutoring session, drawn from a wider data set of nine one-to-one tutoring sessions. Focusing on co-produced higher-level actions as an analytic site of intersubjectivity, I show that lower-level actions that co-constitute a higher-level action can be delineated into tiers of materiality. I identify three tiers of materiality: durable, adjustable and fleeting. I introduce the theoretical/methodological tool tiers of material intersubjectivity to delineate these tiers analytically from empirical data, and show how these tiers identify a multimodal basis of material intersubjectivity. Building on this analysis I argue that the durable and adjustable tiers of material intersubjectivity produce the interactive substrate, which must be established in order for actions that display fleeting materiality to produce intersubjectivity. These theoretical/methodological tools extend the framework of multimodal (inter)action analysis, and I consider some potential applications beyond the example used here.
Chapter
This chapter introduces the primary notions of multimodal (inter)action analysis (MIA) and demonstrates the approach with an example. MIA was developed by Norris (Norris 2004, 2011) as a way to help her understand identity production during research with participants in Germany. Since then MIA has been useful in studies that involve social action ranging from marketing, to kite surfing, to vegetarianism. Researchers continually develop the methodology through use, and it has proven especially practical for studies seeking to describe human action that go beyond the verbal mode to include all the modes involved in social action. MIA draws upon the theoretical notions explicated by Scollon in mediated discourse analysis (MDA)(Scollon 1998, 2001). As with MDA, the notion of mediated action forms the central unit of analysis, and MIA adds further methodological tools to apply the theoretical principles of MDA to the analysis of social action. These developments include delineating social action into higher and lower levels, and introducing the concept of the frozen action. Norris (2004) also developed the concept of modal density to describe more clearly where higher-level actions are positioned on a foreground-background continuum of attention. These methodological tools are introduced below using an example from research into business coaching.
Article
Contemporary research investigating the phenomena of lifestyle sport has highlighted the centrality of space, spatiality and spatial appropriation. Lifestyle sports tend to manifest in liminal and/or unbounded spaces with practitioners drawing upon the affordances of the natural environment in new and unique ways. Simultaneously, practitioners employ continuously evolving technical tools (equipment) in the undertaking of these activities. This article articulates the ways in which the specialised equipment employed as mediational means affect the perception, interpretation and valuation of physical components of the natural and man-made environment. In this article, I introduce the notion of actionary pertinence and concept of locational element through the analysis of the ways in which general geographic areas (spaces) become actual kitesurfing locations (places) for specific social actors through mediated action and in direct connection to the mediational means through which action occurs. Drawing upon data generated through a year-long video ethnography, I argue that components of the physical environment become locational elements for specific social actors in and through mediated action. This explicitly extends theorisation regarding how mediational means affect and structure the nature of the action they mediate (Vygotsky 1978, 1987; Wertsch 1991, 1998), and extends this argument by exemplifying how mediational means also affect and structure the perception and interpretation of physical components of the environment which have a bearing on mediated action. Furthermore, I articulate the ways in which these locational elements are considered, read, re-read and interpreted in terms of their actionary pertinence. Thus, the materiality of locational elements manifests as salient and/or relevant through the bearing an element has on specific mediated actions and practices.
Article
Moving towards multimodal mediated theory, I propose to define a mode as a system of mediated action that comes about through concrete lower-level actions that social actors take in the world. In order to explain exactly how a mode is a system of mediated action, I turn to a perfume blog and use one blog entry as my starting point. The mode that I primarily focus on in this article is the mode of smell, explicating that the mode of smell is not synonymous with olfactory perception, even though modal development of smell is certainly partially dependent upon olfactory perception. As I am ostensibly focusing on the one mode, I once again problematize this notion of countability and delineate the purely theoretical and heuristic unit of mode (Norris, 2004). I clarify that modes a) do not exist in the world as they are purely theoretical in nature; b) that modes can be delineated in various ways; and c) that modes are never singular. Even though the concept of mode is problematical – and in my view needs to always be problematized – I argue that the term and the notion of mode is theoretically useful as it allows us to talk about and better understand communication and (inter)action in three respects: 1. The notion of mode allows us to investigate regularities as residing on a continuum somewhere between the social actor(s) and the mediational means; 2. The theoretical notion of mode embraces socio-cultural and historical as well as individual characteristics, never prioritising any of these and always embracing the tension that exists between social actor(s) and mediational means; and 3. The theoretical notion of mode demonstrates that modal development through concrete lower-level actions taken in the world, is transferable to other lower-level actions taken.
Chapter
Multimodality is an innovative approach to representation, communication and interaction which looks beyond language to investigate the multitude of ways we communicate: through images, sound and music to gestures, body posture and the use of space. The Routledge Handbook of Multimodal Analysis is the first comprehensive ‘research tool kit’ for multimodal analysis, with 22 chapters written by leading figures in the field on a wide range of theoretical and methodological issues. It clarifies terms and concepts, synthesizes the key literature with in-depth exploration and illustrative analysis, and tackles challenging methodological issues. The Handbook includes chapters on key factors for Multimodality such as technology, culture, notions of identity and macro issues such as literacy policy. The handbook takes a broad look at multimodality and engages with how a variety of other theoretical approaches have looked at multimodal communication and representation, including visual studies, anthropology, conversation analysis, socio-cultural theory, socio-linguistics and new literacy studies. Detailed multimodal analysis case studies are also included, along with an extensive glossary of key terms, to support those new to multimodality and allow those already engaged in multimodal research to explore the fundamentals further. The Routledge Handbook of Multimodal Analysis is essential reading for undergraduate and postgraduate students as well as researchers involved in the study of multimodal communication.
Book
In this monograph, the author offers a new way of examining the much discussed notion of identity through the theoretical and methodological approach called multimodal interaction analysis. Moving beyond a traditional discourse analysis focus on spoken language, this book expands our understanding of identity construction by looking both at language and its intersection with such paralinguistic features as gesture, as well as how we use space in interaction. The author illustrates this new approach through an extended ethnographic study of two women living in Germany. Examples of their everyday interactions elucidate how multimodal interaction analysis can be used to extend our understanding of how identity is produced and negotiated in context from a more holistic point of view.
Book
Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences. Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work. Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
Forthcoming). The role of modal intensity in promoting students
  • C Fogarty-Bourget
  • J Pirini
  • N Artemeva
Fogarty-Bourget, C., Pirini, J., & Artemeva, N. (Forthcoming). The role of modal intensity in promoting students' involvement in learning.
Kitesurfing: Action, (inter)action and mediation
  • J Geenen
Geenen, J. (2013b). Kitesurfing: Action, (inter)action and mediation (Unpublished doctoral thesis), Auckland University of Technology, Auckland, New Zealand.
Concepts in multimodal discourse analysis with examples from video conferencing
  • S Norris
Norris, S. (2016). Concepts in multimodal discourse analysis with examples from video conferencing. Yearbook of the Poznań Linguistic Meeting, 2(1), 141-165.