Chapter

Multimodal Interaction Analysis

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

This concise encyclopaedia entry summarizes the theoretical and analytical framework of multimodal interaction analysis and highlights to most current developments in the field.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... Nosotros hemos preferido una traducción más literal de intersemiotic relations, ya que creemos que el término 'orquestación' nos da la idea de algo organizado, mientras que en muchos casos las relaciones entre texto e imagen (por ejemplo entre emoticonos y mensaje verbal en nuestro corpus) no son fruto de una elección voluntaria. y van Leeuwen (2006) y el análisis de la interacción multimodal (Norris, 2004(Norris, , 2011, que enfatiza el contexto de la interacción y la identidad, que ha sido aplicada especialmente a la comunicación cara a cara 54 . Debido al tipo de transcripciones y a nuestro objeto de análisis, al aplicar los conceptos de la multimodalidad al estudio de nuestro corpus, nos referiremos principalmente a la semiótica social. ...
... En nuestro análisis del corpus, además, intentaremos no perder de vista que los mensajes son intercambiados entre personas de carne y hueso, a través de una herramienta real y en circunstancias concretas, por lo que integraremos también otro aspecto de la materialidad: la vida real de los usuarios. 55 También se inspira en la multimodalidad el estudio de Lyons sobre los SMS (Lyons, 2014), en el que la autora estudió cómo se recrea el contexto real y la persona concreta de los usuarios en la comunicación por SMS, haciendo referencia no tanto a la semiótica social, como a los enfoques del análisis de la interacción multimodal (Norris, 2011). ...
Thesis
Full-text available
Emoticons are schematic facial expressions or little pictographs, which are added to electronic messages. Following a complete review of the literature on Computer-Mediated Communication and emoticons, this dissertation analyses the history of emoticons, their typologies and discusses their current spread. The empirical part of this doctoral work is based on an exploratory survey and the analysis of a corpus. The survey was answered by a convenience sample of 226 people. It considered some understudied aspects in research on emoticons, as the settings in which emoticons are mostly used, the most recurring contexts in which they appear and the possible reasons for the current success of emoji. The survey also investigated the agreement of respondents regarding the meaning they would attribute to a set of 19 graphical emoticons, in order to assess the interpretation of these pictographs and the emergence of a conventional meanings. The corpus is composed by a sample of WhatsApp chats. This is a popular instant messaging application for smartphones, which is really popular in Spain. The methodology for the corpus analysis integrates two different approaches: on one hand, Computer-Mediated Discourse Analysis (Herring, 2004), consisting in applying linguistic methods to online corpora, and social semiotics (Kress & Van Leeuwen, 2006), a recent perspective that analyses human communication considering non-linguistic materials, too (Adami, 2016; Jewitt, 2009). Results confirm the popularity of emoji, the preference for the traditional yellow faces and a clear preference for the use of emoticons in informal registers and positive contexts. From an intersemiotic point of view, the relationship between text and images is widely metonymic. The functions of emoticons may be varied (politeness, enhancement of the message, signaling informality, replacing words, managing turn-taking, etc.). Their use seldom provoke misunderstandings. Moreover, the dissertation considered social and cultural aspects related to the use and spread of emoticons, WhatApp and the current digital culture.
... The frst was that interactional moves need not be spoken or written. From the perspective of MMDA, a framework that could only be applied to spoken (or conversation-like written) interaction would be narrow, because interaction is invariably multimodal (Norris 2013). The second assumption was that an exchange may be multimodal in the particular sense that it begins with a turn that draws on one set of semiotic resources and is completed by a turn that draws on another. ...
Chapter
Full-text available
... Given these characteristics, the visual stance was explored by conducting an inductive thematic analysis across the videos focusing on what health-related content was displayed in each shot (Gleeson, 2011). A specific analytical grid adapted from existing digital and visual approaches allowed to complete such analysis by examining how the style was constructed in each shot through meanings via the display of specific objects, environments and materialities (Knoblauch, 2009;Norris, 2012). Due to copyright protection of YouTubers' non-CCL verbal and visual material, we used fictitious names, paraphrased their accounts, and illustrated the visual content through drawings. ...
Article
Full-text available
Professional YouTubers have become highly popular in producing video content through self-mediation. Objective. The present article aimed to study ways in which lifestyle YouTubers construct health practices in their videos within the YouTube media culture. Design. We conducted a narrative and visual analysis across a selection of 15 videos. Results. Results showed that YouTubers’ practices and recommendations for a better life were structured around three themes: Eating to live well; Exercising to live well; Resting to live well and, a fourth cross-cutting theme on Practices aimed at self-development to achieve health and happiness. YouTubers were mainly female presenting, as well as middle/upper-class and white appearing. An overall optimistic tone characterised their health stories, as they delivered personal experiences of success on becoming healthy, happy, and better persons, while encouraging viewers to act similarly. Our findings suggested that YouTubers actively contribute to construct unprecedented definitions of health, enhanced by the social media culture and broader societal logics of healthism and postfeminism. Conclusion. Our study constitutes an original contribution to critical health psychology by examining some of the paradoxes raised by social media influencers like YouTubers regarding health and wellbeing.
... Çokkipli çevriyazı (multimodal transcription), videoyu çeşitli şekillerde yazıya, görüntü biçimlerine ve çeşitli düzenlere dönüştürmek için kullanılır (Mavers, 2012). Çokkipli çevriyazı, konuşma, ifade, bakış, jest, hareket ve vücut pozisyonunun yazı veya görüntü olarak nasıl temsil edilebileceği ile ilgilenir (Norris, 2004). Yazıya dökülecek veriler, insanların nasıl davrandığı ile ilgili olarak davranışsal veriler altında, insanların ne istediği ve neye inandığı ile ilgili olarak da tutumsal veriler altında düzenlenebilir (Goodman vd., 2012). ...
Article
Full-text available
Web 2.0 tools such as blogs, vlogs, wikis, instant messaging and podcasts allow internet users to create, share and collaborate on content, and offer creative opportunities to use them for educational and research purposes. Video sharing platforms are one of the online environments that provide user-generated content (UGC). The rapid increase in the use and sharing of videos in many different fields, from private life to working life and the public sphere, from scientific and technical areas to education, has led researchers to consider online video sharing platforms as rich data sources. Despite their many benefits, the use of such media for educational and research purposes also brings various technical and ethical challenges. This study focuses on the use of user-generated videos (UGVs) on Web 2.0 platforms in design studio education for research purposes. It discusses the technical and ethical aspects of using UGVs as a data source in qualitative research in design studio education and offers recommendations.
... The sections that follow present excerpts selected from the case study of Calia, a plurilingual student, to illustrate the process of MIA grounded in dialogical theorizations of communication (Bakhtin 1986). This approach draws from the work of Norris (2004Norris ( , 2020 and honors the agentic employment of modal ensembles in interaction within socially contextualized events. The MIA analytical approach allows for the examination of research questions such as, which modes and sign complexes do students and teachers employ as they engage in science? ...
Article
Full-text available
Science teaching and learning are discursive practices, yet analysis of these practices has frequently been grounded in theorizations that place language at the forefront of interaction and meaning-making. Such language-centric analytic approaches risk overlooking key embodied, enacted aspects of students’ engagement in science practices. This manuscript presents a case of a plurilingual student’s participation in science inquiry to demonstrate how multimodal interaction analysis can be used to examine the highly diverse array of communicative resources that she draws upon while participating in science, including gestures, facial expressions, vocal intonations, and languages. Grounded in dialogic theorizations of language, we first detail the multimodal interaction approach, and second, we show how multimodal interaction analysis beginning first with her embodied engagement, then coupled with her subsequent written and spoken engagement, reveals robust views of her engagement in science practices. Key to this methodological approach is multilayered analysis that backgrounds verbal or spoken communication to allow for an identification of embodied interaction resources employed. We emphasize how this analytical method allows us to conceptualize science as a practice that unfolds through and in interaction, as compared to a static body of concepts to be learned.
... Kress and van Leeuwen (2001) define four 'strata' (based on Halliday, 1985) of analysis: discourse, i.e., "socially constructed knowledges of (some aspect of) reality" (Kress & van Leeuwen, 2001: 4); design (the arrangement, the composition, of discursive materials); production (the material realization of a semiotic event or object); and distribution (which in the late twentieth and into the twenty-first century often adds or changes meaning, 2001: 7). Interest in multimodality led to work by others influenced by SocSem and/or SFL, who saw the exciting consequences of this new point of view for their own research and publications (e.g., Lemke, 1998;Scollon, 2001;Norris, 2004;and others). Multimodal SocSem (see van Leeuwen, 2005;Andersen et al., 2015 21 ) became even more interdisciplinary in nature and a way of understanding the practice of meaning-making across a range of texts and institutions, and also allowing different theoretical voices, all concerned with the process and practice of semiotic meaning-making, to dialogue with each other-for example, in the journal Social Semiotics (founded in 1991). ...
Chapter
This chapter describes precursors to CDA, and important foundational concepts and theories. We first review briefly the ideas of the British linguist, John Rupert Firth, and his anthropologist colleague, Bronislaw Malinowski, and then discuss Michael Halliday’s Systemic Functional Linguistics (SFL), Systemic Functional Grammar (SFG) and Language as Social Semiotic (SocSem) and some ideas originated by James Martin—including his stratal-functional model, notions of text and context (and register and genre), the three metafunctions, grammatical metaphor and ‘appliable linguistics’. Next, we describe critical linguistics (CritLing): its relation to other approaches, its definition, important works such as Language and Control and Language as Ideology, its interdisciplinarity and ‘useability’ as an approach, and its further development in Kress (1985b/1989), Linguistic Processes in Sociocultural Practice. We then discuss the complex relationship between CritLing and SocSem, the further development of SocSem in Hodge and Kress (1988) and especially in Kress and Theo van Leeuwen’s work on the ‘grammar’ of visual design in Reading Images (1996), including the three metafunctions and other facets of the visual. We conclude with the development of multimodality—and a short discussion of Kress and van Leeuwen’s Multimodal Discourse in relation to CritLing, SocSem, and CDA.
... Data analysis method. Data analysis for this study was influenced by the methodological framework referred to as 'multimodal interactional analysis' (Norris, 2004). 'Multimodality' characterises research data that is not primarily words or numbers. ...
Article
Full-text available
Introduction Opportunities to participate with enjoyable activities is one of the most frequently reported unmet needs by the person living with dementia. Enabling and intuitive technologies may offer accessible ways to engage with such activities. Objectives To explore how tablet computers might encourage participation in enjoyable activities by people with moderate levels of dementia and to consider how such technologies might be incorporated into the repertoire of activities currently provided through day care settings. Methods A focused visual ethnographic approach was developed specifically to meet the research objectives. Twelve participants attending a community day care centre and nine supporters (both volunteers and paid staff) consented to take part in the research. Technology facilitated group activity sessions took place twice a week for a period of four weeks and all were video recorded. Findings: Video analysis demonstrated that the majority of people with dementia found the technology an effective means of participating in enjoyable activities. Analysis also revealed the extent to which participation relies on the existence of effective support. It showed how maintaining focus on retained strengths and abilities enabled the group overall to meet and often exceed their own and others perceived capacity to participate. Finally, analysis confirmed the importance of enjoyment of activities 'in the moment' and the need for those supporting people in the moderate stages of dementia to acknowledge and work with this. Conclusion The use of tablet computers to enhance participation in sociable and enjoyable activities in day care settings is realistic and achievable if supported appropriately.
... Methodologically, analyzing the details of how screens are implemented within reality TV formats is done using multimodal video analysis (Norris 2004) and television analysis (i.a. Mikos 2003). ...
... Other sign systems may include gesture, body language, drawings, mathematics, or any other objects that have symbolic meaning. A mediated action perspective often focuses attention on how these other modes are integrated into language for communicating and thinking (Norris, 2004). Language and other symbol systems mediate our communicating and thinking, just as pencil and paper (or a keyboard and monitor) mediate the act of writing. ...
Chapter
One of the interesting challenges facing science education today is learning how to collaborate across different teaching and learning contexts, such as museum and classrooms. The out-of-school setting poses specific challenges to teachers, among them: the novelty of out-of-school settings, the possibility of unexpected events, teachers who are insecure about their pedagogical content knowledge, management issues and physical challenges (especially in the outdoors). All of these challenges place the teacher in a stressful situation, and yet she is expected to function as well as she does in her classroom.
Article
Full-text available
Professional YouTubers are developing a new culture through which they stage health content in pervasive ways. The present article aims to provide a methodological approach to analyse this type of content by adopting a critical perspective in health psychology. To achieve this, we first define our theoretical framework. Then, we formulate a multi-method design combining narrative and visual analyses, as well as automatised linguistic procedures. We then illustrate the potential of our methodology through concrete examples. Implications of this methodology in health psychology are discussed with regard to healthism, a contemporary trend that highly values the individualised pursuit of health.
Chapter
This chapter investigates the use of organizational metadiscourse across three different lecturing styles: reading, conversational and rhetorical (Dudley-Evans 1994). Despite the fact that organizational metadiscourse has not traditionally been considered as conveying engagement, we claim that all metadiscourse can perform this function. Although metadiscourse in spoken lectures has received increasing interest in the last decades, to our knowledge there is no study comparing the use of metadiscourse throughout lecturing styles from a multimodal perspective and highlighting engagement. Following Ädel's (2010) taxonomy and conducting multimodal analyses on a dataset extracted from lectures in Social Sciences at Yale University's OpenCourseWare, we set out to research organizational metadiscourse instances and demonstrate their multimodal use across lecturing styles and the engagement they convey.
ResearchGate has not been able to resolve any references for this publication.