Chapter

Multimodal (inter)action analysis: An integrative methodology

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

This article is an introduction to the theoretical and methodological backgrounds of multimodal (inter)action theory. The aim of this theory is to explain the complexities of (inter)action, connecting micro- and macro levels of analysis, focusing on the social actor. The most important theoretical antecedent, mediated discourse analysis (see Scollon 1998, 2001b), is presented with its key concepts mediated action and modes. It is shown how action is used as the unit of analysis and how modes are understood in multimodal (inter)action analysis – as complex cultural tools, as systems of mediated action with rules and regularities and different levels of abstractness. Subsequently, methodological basics are introduced, such as lower-level, higher-level and frozen action; modal density, which specifies the attention/awareness of the social actor; and horizontal and vertical simultaneity of actions. Horizontal simultaneity can be plotted on the heuristic model of foreground-background continuum of attention/ awareness. Vertical simultaneity of actions comprises the central layer of discourse (immediate actions), the intermediate layer (long-term actions) and the outer layer (institutional or societal contexts). In short, it is sketched how multimodal (inter)action analysis aims to answer questions about the interconnection of the different modes on a theoretical as well as on a practical level.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... The methodological approach employed herein originally derives from the adoption of a Vygotskyan approach to genetic development and the incorporation of Wertsch's mediated action theory as theoretical precursors to Scollon's (2001) mediated discourse theory/analysis and Norris's extension of these principles in Multimodal (Inter)action Analysis (2004, 2011, 2013a, 2013b) and Multimodal Mediated Theory (Norris 2013a(Norris , 2013bGeenen 2013). ...
... Each frame of the visual transcription pictorially represents the lower-level action/s of analytical focus, coupled with the time signature of their unfolding. 1 While natural communicative activity appears propagated through multimodal configurations (Norris 2009(Norris , 2011(Norris , 2013a, there has been considerable contention regarding what exactly constitutes a communicative mode. Social semiotic perspectives champion the idea of semiosis whereby a mode is typically defined as a semiotic system with socio-culturally instantiated regularities attached to them. ...
... This provides insight into the ways in which the more complex pragmatic functions of deictic gestures are propagated collaboratively in naturally unfolding interaction, thus providing support for Norris's (2013a) claim that individual modal development occurs through concrete lower-level actions taken in the world which become transferable across individual sites of engagement. Thus, the comparative analysis shows the 'trying on' of gestural specification which is appropriated from the efforts of an older sibling and an acknowledgment of the need for specification due to a communicative breakdown manifesting previously. ...
Article
In this article, I detail incremental microgenetic alterations in the development of one particular socio-interactive aptitude: making a relevant interactive contribution. Taking heed of Clark’s (2014) call for the need to reorient our attention to investigate the pragmatics of interaction by accounting for the multiple communicative modes through which this is acccomplished I detail the ways in which parental facilitation and a flexible participatory configuration, made possible by video-conferencing technology, create conditions enabling the agentive re-introduction of a psycho-socially relevant topic. Paramount are the ways in which residual interactive specificities in introduction, co-production and multimodal configurations re-manifest suggesting a more symbiotic relationship between traditional notions of ‘message’ and ‘production’. During the microgenesis of interactive aptitudes, children are not just learning what constitutes psycho-socially relevant topoi, they also acquire an understanding of exactly how to make the contribution through multimodal ensembles.
... Multimodal (inter)action analysis (Norris 2004(Norris , 2009(Norris , 2011a(Norris , 2011b(Norris , 2013) is a theory of human communication with an abundance of methodological tools to empirically * investigate interaction. Growing out of applied linguistics, anthropological linguistics, sociolinguistics, discourse analysis, and socio-cultural psychology (Goffman 1963(Goffman , 1974Gumperz 1982;Tannen 1984;Schiffrin 1987;Hamilton 1998;Scollon 1997;2001;van Lier 1996;Wertsch 1998;Wodak 1989) and strongly influenced by social semiotic thought (van Leeuwen 1999;Kress 2000;Kress andVan Leeuwen 1998, 2001), multimodal (inter)action analysis (Norris 2004(Norris , 2011a) is a multimodal discourse approach. ...
... Through the underlying concept of mediation in all respects of action, the framework allows for the simultaneous theoretical inclusion of social actors and their psychological make-up, objects, and the environment. The notion of mediation in this framework facilitates the resolution of differences between human actors, the things they use, and the world that they inhabit (Norris 2013). Thus, in multimodal (inter)action analysis, the notion of mediation is a theoretical concept that allows for the theoretically comprehensively bringing together of cognitive and socio-psychological, embodied physical, object physical, and environmental physical aspects into one framework. ...
... Multimodal (inter)action analysis (Norris 2004(Norris , 2009(Norris , 2011a(Norris , 2011b(Norris , 2013(Norris , 2014(Norris , 2015 originating from mediated discourse analysis (Scollon 1998(Scollon , 2001 is based in the sociological interest of humans acting in the world that we find in the work of Goffman (1963); incorporates the interest in intercultural interaction that we find in the work of Gumperz (1982); includes an interest in power in interaction that we find in the work of Wodak (1989); delves into the microanalysis of interaction that we find in the work of Tannen (1984), Schiffrin (1987), or Hamilton (1998); has a strong interest in applied linguistics that we find in the work of van Lier (1996); is strongly influenced by socio-cultural psychology as we find in the work of Wertsch (1998); and is grounded in social semiotic thought that we find in the writings of van Leeuwen and Kress (van Leeuwen 1999;Kress 2000;Kress andvan Leeuwen 1998, 2001). With these foundations, multimodal (inter)action analysis (Norris 2004(Norris , 2011 has developed into a strong theoretical framework with an abundance of methodological tools (Norris 2004(Norris , 2009(Norris , 2011(Norris , 2013a(Norris , 2013b(Norris , 2014Geenen 2013;Makboon 2015;Pirini 2015Pirini , 2016) that make the analysis of (always) multimodal (inter)action possible, opening up research into new and promising directions. ...
Article
Full-text available
This article presents theoretical concepts and methodological tools from multimodal (inter)action analysis that allow the reader to gain new insight into the study of discourse and interaction. The data for this article comes from a video ethnographic study (with emphasis on the video data) of 17 New Zealand families (inter)acting with family members via skype or facetime across the globe. In all, 84 social actors participated in the study, ranging in age from infant to 84 years old. The analysis part of the project, with data collected between December 2014 and December 2015, is ongoing. The data presented here was collected in December 2014 and has gone through various stages of analysis, ranging from general, intermediate to micro analysis. Using the various methodological tools and emphasising the notion of mediation, the article demonstrates how a New Zealand participant first pays focused attention to his engagement in the research project. He then performs a semantic/pragmatic means, indicating a shift in his focused attention. Here, it is demonstrated that a new focus builds up incrementally: As the participant begins to focus on the skype (inter)action with his sister and nieces, modal density increases and he establishes an emotive closeness. At this point, the technology that mediates the interaction is only a mundane aspect, taken for granted by the participants.
... In this article I use multimodal (inter)action analysis (Norris, 2004(Norris, , 2009(Norris, , 2013a) as a methodological framework, and the associated tools of modal density, and higher and lower-level action. Multimodal (inter)action analysis takes the mediated action as the unit of analysis, delineated into higher and lower levels. ...
... This article examines various movements of the body, and the mode of spoken language and demonstrates the usefulness of applying a multimodal approach to business coaching. Using multimodal (inter)action analysis (Norris, 2004(Norris, , 2011b(Norris, , 2013a as my methodological framework, this article illustrates the utility of modal density and higher and lower-level actions as methodological tools which can be employed to articulate the complexity of a business coaching interaction. ...
... Multimodal (inter)action analysis (Norris, 2004(Norris, , 2011b(Norris, , 2013a draws upon the theoretical principles of mediated discourse analysis (Scollon 1998(Scollon , 2001, with an explicit focus on social action. All actions are inherently social, and all actions are mediated by mediational means (Norris, 2004;Wertsch, 1998). ...
Chapter
In this article I present an analysis of three extracts from a business coaching session captured on video. In business coaching the coach aims to help the client generate solutions to their own issues, often by finding different perspectives. However, there has been a lack of empirical studies focusing on the coaching interaction. Here I set out firstly to describe how the coach carried out the act of coaching, and to illustrate the use of higher and lower-level actions and modal density to focus on the detail of an (inter)action, while not losing sight of the whole. I analyse all the relevant communicative modes (Norris, 2011a), and focus closely on specific lower-level actions in the interaction. I use modal density (Norris, 2004) as a methodological tool to consider these lower-level actions as constituents of higher-level actions, and as a measure of participants’ relative engagement in various higher-level actions. Overall, I show that modal density and lower and higher-level actions can be used as useful tools for the analysis of business coaching at the level of the interaction.
... In the given example, participating in a class, a student may engage in multiple, simultaneous higher-level actions; for example, taking notes and reading slides. Moreover, the higher-level action participating a class can be split into consecutive higher-level actions such as setting up a computer for note-taking and engaging in a group discussion (Norris, 2013). 'Frozen action' refers to disembodied modes such as furniture or a drawing, which are the results of previously performed actions visible in material objects (Norris, 2004). ...
... MDA has been said to have an activist stance, aiming to achieve social change by examining and making explicit the links between mundane, concrete actions, and broader structures in relation to power (Scollon & Scollon, 2004). Norris (2013) refers to the interplay of micro and macro levels of discourse as a "vertical simultaneity of actions." In other words, each higher-level action is embedded simultaneously in the immediate level or central layer of discourse, in the intermediate layer of discourse, and in the outer layer of discourse that comprises the "actions performed within the institutional or societal contexts" (Norris, 2013, p. 283). ...
Article
This paper focuses on how signing students organise themselves spatially in social interactions in a university lecture hall. One may view space as a concrete location, a social construct, and a normative actor with historical trajectories. The study addresses the question, 'What are the mediated actions through which the students and teacher (re)configure space for participating in a class?' Following a methodological framework of Mediated Discourse Analysis and multimodal interaction analysis, I approach this question by examining the social actions occurring when entering a lecture hall. The primary data includes video recordings, photos, and participatory observations, documented by field notes. The analysis shows how the architectural specifications of a space pose restrictions on visual-embodied interactions. However, the participants configure and reconfigure the space to some extent to suit visual-embodied interaction through explicit and implicit negotiation.
... Some instances in the examples, however, are particularly relevant to the action of using video conferencing technology. Overall, the article will add to the literature on video conferencing as well as to the literature of knowledge communication and the literature on negotiating disagreements. 2 MULTIMODAL (INTER)ACTION ANALYSIS: THEORY AND METHODOLOGY Multimodal (inter)action analysis is a holistic analytical framework that understands the multiple modes in (inter)action as all together building one system of communication (Norris, 2004Norris, , 2011Norris, , 2013aNorris, , 2013bNorris, , 2014a, forthcoming). (Inter)action in this analytical framework is written with parenthesis around 'inter' in order to highlight that all actions, no matter if they are taken with another human being, or objects within the setting, or the environment, are interactions. ...
... In this article, we have focused upon how the participants multimodally communicate and accept knowledge , how they shift their attention, and how they work through a disagreement via video conferencing technology. The multimodality discussed here, is certainly just as present in face-to-face (inter)action (Norris, 2004Norris, , 2009Norris, , 2011Norris, , 2013aNorris, , 2013bNorris, , 2013cNorris, , 2014, forthcoming). However, there certainly are also differences between video conferencing (inter)actions and face-to-face (inter)actions. ...
Article
This article takes a multimodal approach to examine how two young men communicate knowledge, shift attention, and negotiate a disagreement via videoconferencing technology. The data for the study comes from a larger ongoing project of participants engaging in various tasks together. Linking micro, intermediate and macro analyses through the various methodological tools employed, the article presents multimodal (inter)action analysis (Norris, 2004, 2011, 2013a, 2013b) as a methodology to gain new insight into the complexity of knowledge communication via videoconferencing technology, which is relevant to many settings from education to employment, from organizations to gaming.
... Besides, the notion of genre informs one's observation of different linguistic and semiotic structures in diverse communication affairs (Bateman 2013). It is also the genre that influences or determines, to a considerable extent, the texture-cum-cohesion of text, as being an image, a word, a gaze and a colour (Jewitt 2009;Baldry & Thibault 2006;Norris 2013;Jewitt, Bezemer & O'Halloran 2016). That is a probable motive for Bhatia (2008) to describe genre as a conventionalised semiotic resource employed to express specific goals in a certain environment. ...
Article
Full-text available
This study examined the interplay of pictorial and written modes that position advertising as a multimodal genre, explainable through a social semiotic perspective. Eight advertisements of the financial, telecommunications, and beverage products functioned as devices of analysis. Nevertheless, multimodal communicative acts served as the processing tool, elucidating the meaning potentials of the advertising configurations. Having deployed a system of multimodal interacts, tables and graphs assisted in accounting for the frequency of the semiotic resources of the written modes. The analysis indicated large and highlighted fonts (Celebrating the world’s no. 1 fixer), repetitions (Guinness, Maltina, real deal), and deviant constructs (EazyLoans, GTWorld) as elements of propagating intended messages. The deployment of codes (*966*11#, 737) and fragmented clauses (Over N100 million worth of airtime) played some roles in the meaning-making operations. Of significance is the Guinness’ conceptual “digits” of 17:59, contextualising the year, time, and channel of promotional benefits. Though questions (Have you called mum today?), offer (It can be), and minor clauses (Welcome to Guinness time) were parts of the communicative systems, statements (Terms and condition apply) and commands (Enjoy the complete richness of Maltina) dominated the entire dialogues. One might suggest that communicators should endeavour to deploy apt constructions and create eye-lines between participants as means of sensitising readers into consumption.
... Multimodal (inter)action analysis is thus a coherent and comprehensive research framework for the analysis of qualitative video-based data. 2 All the pieces in this framework fit together (Norris 2012;Pirini 2014b), allowing the researcher to build a coherent picture of whatever human action, interaction or identity is being studied. In this way, we have made strides in examining space and place or children's acquisition (Geenen 2013;Geenen 2017Geenen , 2018; identity (Norris 2005(Norris , 2007(Norris , 2008(Norris , 2011Norris and Makboon 2015;Matelau-Doherty and Norris 2021); video conferences (Norris 2017a;Norris and Pirini 2017); business coaching, high school tutoring and intersubjectivity (Pirini 2013(Pirini , 2014a(Pirini , 2016, to name but a few areas in which the framework has been used. ...
Article
Full-text available
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... In order to fully understand the roles that this combination of modes play in human activity it is necessary to complement WAGR with an analytical approach that allows researchers to investigate the complex multimodal nature of the lab book and its contributions to knowledge making. We have adapted MIA (Bernad-Mechó, 2021;Norris, 2004Norris, , 2013Norris, , 2019 to inform our study, as "this is the only approach that was especially developed because of and for the analysis of multimodal action and interaction" (Pirini et al., 2018, p. 640). The unit of analysis in MIA is mediated action, which includes "a social actor acting with/through mediational means" (pp. ...
Article
Full-text available
Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation.
... Modalities belong to different systems of representation, and they translate into actions which mediate between the concrete and the abstract. Modalities are complex cultural tools, to the extent that they are socially acquired or learnt and used in interaction (Norris 2013). Nonreferential gestures and body movements can have different functions. ...
Article
Full-text available
Linguistic humour studies have been undertaken from different perspectives. The present paper offers a review of the most influential theories seeking synergies and convergence between them under the umbrella of cognitive linguistics, and, more specifically, resorting to Langacker’s (2001) current discourse space (CDS) as the overall framework which can accommodate and encompass those perspectives, along with Fauconnier and Turner’s (2003) Conceptual Integration Theory. A sketch of various theories is included (Raskin, 1985; Attardo, 1994; Coulson, 2005a; Veale, 2015, etc.), along with an analysis of points of convergence and similarities as the rationale for bringing them together against the backdrop of the CDS.
... Modalities belong to different systems of representation, and they translate into actions which mediate between the concrete and the abstract. Modalities are complex cultural tools, to the extent that they are socially acquired or learnt and used in interaction (Norris 2013). Nonreferential gestures and body movements can have different functions. ...
Article
Full-text available
This paper assesses the results of a multimodal analysis of humorous instances found in a collection of 14 interviews from The Late Show with Stephen Colbert. The interviews have been imported and annotated in ELAN for head movements and face gestures. Only the interviewees’ remarks are studied in order to ensure the analysis of spontaneous speech, avoiding mostly pre-scripted host’s comments and questions. The results demonstrate that there are no gestures consistently and recurrently paired with humour. Therefore, none of the multimodal cues found in the sample can be considered markers of humour. Furthermore, the functions of gestures identified are consistent with the uses reported in previous multimodal studies of humorous and non-humorous communication found in the literature.
... The analysis will also take into account the evaluative properties of the texts under scrutiny (Hunston and Thompson 2000;Bednarek 2009aBednarek , 2009b, by investigating the linguistic markers signalling the legal appraisal of claimants' stories and the values emerging from how immigration judges handle these cases. This paper takes a multimodal perspective (Norris, 2004(Norris, , 2011a(Norris, , 2012a(Norris, , 2013a(Norris, , 2013b(Norris, , 2013cGeenen, 2013aGeenen, , 2013bGeenen, Norris & Makboon, 2015;Makboon, 2015;Pirini, 2014Pirini, , 2015Pirini, , 2016, building upon mediated discourse analysis (Scollon, 1998Norris and Jones, 2005a, b) to examine how identities are produced in family videoconferencing (inter)actions. ...
Conference Paper
Book of Abstracts of the 6th Critical Approaches to Discourse Analysis across Disciplines Conference (CADAAD 2016)
... Multimodal mediated discourse analysis (Norris, 2013) embraces those areas of multimodality that originate in a mediated approach to discourse (Scollon, 2001;Vygotsky, 1978;Wertsch, 1998). This area was hailed by Kress (2005) as indicating a major paradigm shift that changed from treating language as the organizing system for interaction to viewing action as the organizing system of human interaction. ...
Chapter
Multimodal discourse analysis is an emergent field that began around the verge of the millennium with books such as Reading Images: A Grammar of Visual Design (Kress & van Leeuwen, 1996/2006), Mediated Discourse as Social Interaction (Scollon, 1998), Multimodal Discourse (Kress & van Leeuwen, 2001), and Analyzing Multimodal Interaction: A Methodological Framework (Norris, 2004). Initially, multimodal discourse analysis was primarily the domain of mediated discourse analysts, social semioticians, and systemic functional linguists. While early developments were somewhat overlapping in time, these works resulted from, and aligned with, two separate major paradigm shifts stemming from previous work in discourse analysis...
... I utilise multimodal (inter)action analysis (MIA) (Norris, 2004(Norris, , 2011(Norris, , 2013) as a methodological framework. MIA focuses on action, treating all actions as potentially important. ...
Article
During the activities of everyday life social actors always produce multiple simultaneous higher level actions. These necessarily operate at different levels of attention and awareness. Norris (2004, 2011) introduces modal density as a tool for analysing the attention/awareness of social actors in relation to higher level actions they produce, positioning actions in the foreground, midground and background of attention. Using modal density to analyse an opening and a closing in high school tutoring sessions, I show social actors transitioning into and out of producing the same higher level actions at the foreground of their attention/awareness. Through this analysis I identify two potentially unique aspects of one-to-one tutoring. Firstly I show one way that a tutor helps a student take on the practices of being a good student, and secondly I show the influence that students have over tutoring. I argue that movements into and out of a shared focus of attention are potentially useful sites for analysis of social interaction.
Article
Full-text available
The paper explores the concept of multimodality in contemporary stylistics of virtual communication, mainly focusing on the process of interpretation. After outlining the basics of multimodal meaning-making theory, it further develops it by incorporating ideas from the theory of media or other related disciplines. The next part describes the of immersive potential of virtual environment, tying it to the multisensorial nature of human epistemic existence. Finally, the last part introduces these ideas to the process of stylistic interpretation, emphasizing the changes and problems, which multimodal virtual communication introduces.
Book
Full-text available
Band 17 der Reihe Giessener Fremdsprachendidaktik: online versammelt Studien zu kulturellen Fragen aus den Bereichen Fremdsprachendidaktik, Linguistik und Pädagogik. Alle Beiträge wurden auf der ersten internationalen Tagung des Print-Projekts der Universidade do Estado do Rio de Janeiro zum Thema „Sprachen lehren und lernen aus interkultureller Perspektive“ im September 2019 am Instituto de Letras dieser Universität präsentiert. Das Buch beginnt mit einer Diskussion über das Verhältnis von Kultur und Sprachenlernen. Die folgenden Texte zeigen die Ergebnisse aus Studien, die mit angehenden Lehrenden und Lernenden von Portugiesisch und Deutsch als Fremdsprache in universitären und schulischen Kontexten in Brasilien, Deutschland und Frankreich durchgeführt wurden.
Article
Full-text available
Through detailed multimodal analyses, this article shows how participants of an English language course create and manage sites of attention for achieving collaboration across multiple spaces. The ethnographic data for the study comes from an English language course ‘Academic reading’ offered for university students majoring in Finnish Sign Language. The paper examines a classroom situation where the participants interact simultaneously with each other in a chatroom environment and physical environment via several mediational means, such as Finnish Sign Language and typed text in a chatroom that appears on students’ tablets and on a large screen in the classroom. The emerging interactional patterns coordinate fragmented interaction into mutual sites of attention so that the participants are able to collaborate. Further, the paper reflects how gaze plays a crucial role in coordinating actions together. Gaze itself is seen as a mediated action which begins to display regularities, and ‘comes into being’ through long and short life trajectories. The longer trajectory of the mode of gaze is traced to the patterned ways of using gaze in signing communities of practice.
Article
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
ResearchGate has not been able to resolve any references for this publication.