Systematically Working with Multimodal Data: Research Methods in Multimodal Discourse Analysis.
Abstract
A guide that offers a step-by-step process to data-driven qualitative multimodal discourse analysis
Systematically Working with Multimodal Data is a hands-on guide that is theoretically grounded and offers a step-by-step process to clearly show how to do a data-driven qualitative Multimodal Discourse Analysis (MDA). This full-color introductory textbook is filled with helpful definitions, notes, discussion points and tasks. With illustrative research examples from YouTube, an Experimental and a Video Ethnographic Study, the text offers many examples of how to deal with small to large amounts of data, including information on how to transcribe video data multimodally, including online videos, and how to analyze the data.
This textbook contains ample theory, directions for literature, and a teaching guide to help with a clear understanding of how to work with multimodal data.
Contains new research data, exceptional illustrations and diagrams
Offers step-by-step processes of working through examples, transcriptions and online videos
Goes into great depth so that students can use the book as hands-on material to engage with their own data analysis
Designed to be easy-to-use with color-coded definitions, tasks, discussion points and notes
Written for advanced undergraduate, graduate and PhD level students, as well as participants in research workshops, Systematically Working with Multimodal Data is an authoritative guide to understanding data-driven qualitative Multimodal Discourse Analysis.
... This chapter examines miscommunication and the interactive nonalignment of an intercultural team working on a task via video conferencing technology using English. We utilize multimodal (inter)action analysis (Norris 2004(Norris , 2011(Norris , 2019(Norris , 2020 as our theoretical and methodological framework to shed new light on how participants appear to co-construct common ground, while they in fact do not achieve conceptual convergence. ...
... We utilize multimodal (inter)action analysis (Norris 2004(Norris , 2011(Norris , 2019(Norris , 2020 as our theoretical and methodological framework to shed new light on how participants appear to co-construct common ground, while they in fact do not achieve conceptual convergence because they each produce their own higher-level actions, drawing on different social practices as they go about the task in different ways. Multimodal (inter)action analysis is a theoretical and methodological framework that allows the analysis of human actions and interactions in their vast complexity. ...
... In this chapter, we use multimodal transcription conventions (Norris 2004(Norris , 2011(Norris , 2019(Norris , 2020. Full transcripts show the lower-level actions that individuals perform. ...
This chapter examines miscommunication and the interactive nonalignment of an intercultural team working on a task via videoconferencing technology using English. We utilize Multimodal (Inter)action Analysis (Norris, 2004, 2011, 2019, 2020) as our theoretical and methodological framework to shed new light on how participants appear to co-construct common ground, while they in fact do not achieve conceptual convergence.
... This paper thus presents a unique comparison between the actual communication during the game versus the live broadcasted commentary later that night, applying both Conversation Analysis (Sacks 1992; Ten Have 2004) and Multimodal (Inter)action Analysis (Norris 2004(Norris , 2019. It uncovers the structured sequentiality of talk, such as requests and compliances within the team communication and the commentary itself, and highlights synthetic and parallel actions within the complex technical workplace settings. ...
... She includes the above-mentioned non-verbal modes for analysis, and takes videography as the basis for her transcription system on which she lays language. Multimodal (Inter)action Analysis is the second methodological framework used for the present study, as it enables an analysis of modes beyond language such as gaze, posture, proximity as the focus of the transcription (Norris 2004(Norris , 2019. Also, modes work together and they represent different levels of materiality, and in this sense the multiactivity in a workplace setting can be captured more accurately (Norris 2019). ...
... Multimodal (Inter)action Analysis is the second methodological framework used for the present study, as it enables an analysis of modes beyond language such as gaze, posture, proximity as the focus of the transcription (Norris 2004(Norris , 2019. Also, modes work together and they represent different levels of materiality, and in this sense the multiactivity in a workplace setting can be captured more accurately (Norris 2019). ...
This paper discusses the linguistic features of the production of football commentary within its macrosocial contexts. It contrasts the communication of the TV production team during a football game with the live football commentary aired on TV. Applying a mixed methods analysis, it reveals how football commentary is prepared and produced in the technical setting of the commentator’s booth in a stadium. This study reveals how a medial reality of the same events concerning a goal are broadcasted in the 11 min highlight video football commentary. The author and her co-principal investigator video recorded a total of 36 h in four work settings. Finally, retrospective interviews with the commentator and head of sports of the TV station place the study in the broader social context of football as a media production. In this paper, I show how live football reporting is prepared and conducted using two different theoretical and methodological frameworks: Conversation Analysis (CA) and Multimodal (Inter)action Analysis (MMIA). Both provide insight into the synthetic verbal, non-verbal and parallel actions for the successful production of the football commentary and they also reveal the sequential structure of naturally occurring talk (CA).
... Our work is innovative and makes contributions to the fi eld in several ways. Firstly, our choice of qualitative analysis method to study online multimodal interactions, Multimodal (Inter)action Analysis (MIA) ( Norris, 2004( Norris, , 2019( Norris, , 2020, is novel particularly in Applied Linguistics. More specifi cally, in online language learning and teaching literature, several researchers (e.g., Balaman & Sert, 2017 ;Cappellini & Azaoui, 2017 ) have utilised Multimodal Conversation Analysis for a micro-analysis of the sequential organisation of talkin-interaction in multimodal computer-mediated pedagogical interactions. ...
... With MIA, we used the construct of a mode to examine interaction in its multimodal complexity and, in transcript production, we were guided by modal shifts in the LLAs performed. As Norris (2019 ) argues, this enables greater replicability in the practice of transcript production: should one researcher place a particular movement (e.g. a head movement) in one mode during transcription (head movement) but another researcher place the same movement in another mode (posture), the movement will be produced in both researchers' fi nal transcripts in the same manner because the movement (LLA) guides the transcript, with mode providing a theoretically-founded way to describe the action. The transcription of multimodal CALL data is time-consuming and often requires dividing annotation, transcription, and transcript production practices across several researchers ( Guichon, 2017 ). ...
... The measures implemented only related to the provision of lesson materials and learners and, thus, did not impede collection of naturally-occurring data with respect to freedom teachers had to adapt the material to suit their everyday teaching practices. No further interventions were imposed and participants were not guided to (inter)act in any specifi c way ( Norris, 2019 ). This book, Contributions, pedagogical refl ections, and future perspectives 145 thus, demonstrates the potential to examine variety in teacher practices in an exploratory fashion, while preserving an interest in naturally-occurring data. ...
... Multimodal (Inter)action Analysis (MIA) established and described by Norris (2004Norris ( , 2019Norris ( , 2020 constitutes the methodological framework of this study. MIA enables researchers to focus on a single actor's actions (in our case, the teacher) as well as the mediational qualities of these actions through multiple modes. ...
... MIA (Norris, 2004(Norris, , 2019(Norris, , 2020 is "a holistic analytical framework that understands the multiple modes in (inter)action as all together building one system of communication" ( Norris & Pirini, 2016 , p. 24). It considers interactions between social actors and other social actors, objects, or the environment as actions. ...
... In identifying the multimodal organisation of LLAs, our guiding principle was to discern "what is absolutely necessary to perform this very action and what is not," as well as modal density to determine the place of the HLAs within the social actors' attention/awareness ( Norris, 2019 , p. 246). The analytical tools of modal density, modal configurations, and the foreground-background continuum of attention/awareness ( Norris, 2019 ) were employed. ...
... Such videos often powerfully resonate with multilingual students' experiences, prompting the examination of social biases and "-isms" (i.e., linguicism, racism) existing in their lives. Our chapter includes a multimodal microanalysis (Norris, 2019) of one equity video introduced to Dulari in the classroom, which draws attention to the film's "interconnected semiotic" nature, revealing how technology-enhanced modes function within it to create potentially powerful resources for composing (Strauss et al., 2009, p. 190). ...
... Our constructed case study draws on multimodal analytic methods (Norris, 2019), and considers Dulari's critical writing practices in U4 as a "telling case" (Mitchell, 1984) of re-semiosis. Our purpose is to establish "theoretically valid connections" between the equity video Dulari watched and her subsequent resemiotization process. ...
... We began our multimodal analysis by selecting segments from the equity video, which Dulari had emphasized in her own writing samples for a finegrained microanalysis (see Norris, 2019). While the boundaries of what may qualify as a semiotic mode are limitless (i.e., music, images, color tones, lighting), our own analysis focused on Lyiscott's direct speech in combination with perspective shifts. ...
In the era of rapidly advancing technology, multilingual writers’ semiotic landscapes are increasingly multimodal, such that semiotics other than the written and spoken word – images, lighting, movement – are implicated in their composing processes. While some multimodal scholarship focuses on the writer's process of converting traditional prose into multimodal forms, this chapter considers the multimodal pedagogical activities using technology that supported one South Asian writer in generating print-based prose and engaging in critical writing practices in the context of a university developmental English writing course in the Northeastern US. We consider the writer's interaction with an equity video introduced during class, which prompted her examination of linguistic hierarchization in India. The chapter features a multimodal analysis (Norris, 2019) of the video to deconstruct its interconnected semiotic nature and illuminate its potential resources for composing. Subsequently, our case analysis triangulates multiple data sources (writing samples, transcripts) which were generated with the focal learner (who participated in dissertation research by Author 1) to understand the “resemiotization” process which transpired through the learner's interactions with the video (Iedema, 2003). Ultimately, we offer a fine-grained perspective on multimodal idea development and a vision for multilingual pedagogies that integrate videos to support L2 writing development.
... However, while the definition of modal complexity is apparent, definitions of modal intensity are less clear and difficult to apply analytically. Norris (2004Norris ( , 2019 offers three strategies to identify a mode that is intense: (1) A mode that structures a higherlevel action is of high intensity. For example, the activities of people dancing while watching a band are structured by the mode of music. ...
... From this list and with reference to the video, actions wherein the tutor provided some form of guidance to the student were identified. Those that included lowerlevel actions produced with high modal intensity were selected using the same qualitative features applied for the study of mathematics lectures, and were then transcribed according to Norris (2004Norris ( , 2019. In brief, this process involves producing a transcript for each participant by capturing a screenshot of each change in lowerlevel action for each mode (e.g., gesture, gaze, posture). ...
... Even though these studies and others (e.g., Callow, 2020;Serafini, 2014) all point to the multimodal nature of reader responses, I have yet to find a study that explicitly considers the movements of teachers and/or children as they engage in reading nonfiction books. To understand this more, I employed a multimodal analysis (Norris, 2019) to examine the embodied teaching practices of an experienced teacher as she conducted read-alouds using nonfiction picture books. ...
... Twentyfour read-aloud videos were captured in total, but for the purposes of analysis, I selected recordings of nonfiction books with strong instances of the students' multimodal responses (14 in total). I conducted a multimodal analysis for each of the 14 interactive read-alouds (Norris, 2019). I began by documenting the low-level mediated actions of Mrs. Burnette and the students in the class. ...
Few studies examine young children's multimodal responses to nonfiction picture books, and even fewer examine teaching practices that encourage these responses. This case study of a kindergarten class illustrates how one teacher regularly conducted interactive whole‐group read‐alouds using nonfiction picture books. After examining the multimodal teaching strategies employed by this teacher, the findings of this study highlight the social and physical nature of reading with young children. A discussion of this teacher's multimodal teaching practices suggests that understanding information from nonfiction picture books for young children requires readers to continually negotiate multiple modes of information over time and with the actions and opinions of those around them. Considerations for teachers reading nonfiction picture books are provided.
... Multimodal (inter)action analysis involves two phases of analysis: analysis of actions and analysis of modes. The analysis begins with identifications of different types of mediated actions: higher-, lower-, and frozen actions (Norris, 2004(Norris, , 2019. Higher-level actions are complex actions that have identifiable boundaries. ...
... The second phase of analysis in multimodal (inter)action analysis is to explore how modes accomplish higher-level actions in hierarchical and non-hierarchical ways (Norris, 2019). This is determined by analyzing modal intensity (i.e., the weight that a mode carries in a higher-level action) and modal complexity (i.e., the relationships between modes that rely on each other for meaning) (Norris, 2017). ...
This study examines how English-as-lingua-franca (ELF) learners employ semiotic resources, including head movements, gestures, facial expression, body posture, and spatial juxtaposition, to negotiate for meaning in an immersive virtual reality (VR) environment. Ten ELF learners participated in a Taiwan-Spain VR virtual exchange project and completed two VR tasks on an immersive VR platform. Multiple datasets, including the recordings of VR sessions, pre-and post-task questionnaires, observation notes, and stimulated recall interviews, were analyzed quantitatively and qualitatively with triangulation. Built upon multimodal interaction analysis (Norris, 2004) and Varonis and Gass' (1985a) negotiation of meaning model, the findings indicate that ELF learners utilized different embodied semiotic resources in constructing and negotiating meaning at all primes to achieve effective communication in an immersive VR space. The avatar-mediated representations and semiotic modalities were shown to facilitate indication, comprehension, and explanation to signal and resolve non-understanding instances. The findings show that with space proxemics and object handling as the two distinct features of VR-supported environments, VR platforms transform learners' social interaction from plane to three-dimensional communication, and from verbal to embodied, which promotes embodied learning. VR thus serves as a powerful immersive interactive environment for ELF learners from distant locations to be engaged in situated languacultural practices that goes beyond physical space. Pedagogical implications are discussed.
... The methodological approach chosen for the analysis in this research is Multimodal (Inter)action Analysis (Norris 2004(Norris , 2019(Norris , 2020. This methodological framework gathers concepts from different theoretical approaches such as mediated discourse (Scollon 2001), studies of social interaction and social semiotics (Kress andvan Leeuwen 2001, 2020). ...
... Although people construct their identities through many different modes of interaction [18,50], this study focuses on the transcribed excerpts of spoken discourse. The students' interaction has been transcribed, translated from Swedish to English and adapted for readability and comprehension. ...
This methodological article focuses on how to effectively map pre-service teachers’ use of discursive resources in professional identity production. By adopting a discourse analytical approach, this study views identity construction as a situational, real-time process occurring in interaction. The aim is to contribute knowledge about how to systematically map and analyze the resources that pre-service teachers use to construct their teacher identities during their education. Drawing on the framework of Mediated Discourse Analysis, this article presents a model that integrates two key concepts: discourse domains, which refer to the types of discourse commonly used in teacher education, and layers of discourse, which address societal levels in identity construction. The results suggest that using these concepts to map students’ use of discursive resources highlights how their knowledge of the teaching profession, their education and everyday experiences can be assets when constructing their teacher identities. While the model can be further refined and developed to better show the complexity of discursive resources in identity construction processes, it shows promise as a fruitful approach. By mapping and visualizing discursive resources through this model, this study offers valuable methodological insights into how to approach professional identity development among pre-service teachers.
... While traditional discourse analysis often falls short in capturing this complexity, as it overlooks the multimodal nature of human (inter)actions. MIA addresses this gap by enabling the analysis of both verbal and non-verbal actions, thereby providing a more comprehensive understanding of how identities are produced and negotiated in multimodal actions and interactions (Norris 2011(Norris , 2019. ...
The journey from a novice to an established educator is fraught with challenges that significantly impact the development of a professional identity. This study examines the experiences of novice university English language teachers in China, focusing on Caroline, an early career teacher who navigates the challenge of navigating a high power culture and integrating into established teaching communities. Employing Multimodal (Inter)action Analysis (MIA), this research offers a nuanced examination of the interplay between Caroline’s imagined and practiced identities. The findings reveal the complexities of her identity formation, highlighting struggles with exclusion, resistance from senior colleagues, and the reconciliation of her aspirations with professional realities. The study advocates for a comprehensive approach to teacher training and support, emphasizing the need for emotional resilience, personal growth, and the integration of theory with practice. It suggests targeted mentorship, the creation of supportive ecosystems, and the adaptation of educational policies to better prepare novice educators for the multifaceted challenges of the teaching profession. This case study contributes to the global discourse on teacher identity formation, offering insights that can inform the development of more effective support structures for novice teachers, thus enhancing the quality of education.
... I approached this limitation by accounting for more evidence to support my interpretations, by taking a multimodal approach to communication. This entails seeing communication as occurring in and through more than one modality, systematically including nonverbal elements of communication during analysis (Norris, 2019). Teaching has indeed been described as a multimodal orchestration (Bourne & Jewitt, 2003), making an embodied view of communication important when researching teaching. ...
This article investigates the roles of the terms agreement and disagreement in teachers’ talk in Norwegian Grades 1-4 classrooms. Through an exploration of what teachers said and did when they used these terms, five different themes were identified in the teachers’ talk. The teachers tended to use the terms in relation to the process of discussion, the outcome of these discussions, and nuancing the idea of the nature of this outcome; as a function in conversation; and how agreement and disagreement are valuable in different ways. The key finding across these themes and patterns was that the lessons tended to be oriented toward consensus. This is problematized in relation to exploration and elaboration of perspectives, which is crucial for deliberation.
... The meme's text labels vary chronically in response to sociopolitical change and they are completely independent conceptual relations concernced with situations during the pandemic. Abstract concepts such as Covid variants or the media are personified and visualized with human facial expressions and body alignments (Norris 2019), which index the human emotions of neglect, attention and jealousy. As Covid-19 developed Delta and Omicron variants and as people shifted their attention to constantly changing social conditions and circumstances, the meme's canvas remained similar as its text labels changed. ...
Digital humor expressed through memes plays a vital role in facilitating social interaction in online environments. This paper reports on age and gender-related differences in multimodal digital meme humor from users aged 14 to 93. Its aim is to investigate how visual and textual elements, psychological humor types and digital humor types vary among smartphone users of different age and gender groups. The multimodal corpus was compiled from memes that were shared by 250 BA English and senior citizen students at a German university and their relatives and friends. The memes were annotated with the Covid-19 topic (e.g. lockdown, mask requirement, vaccination) and the sources of visual origin (e.g. cartoon characters, animals, politicians). Also, a previously non-multimodal psychological humor framework (Martin, Rod A., Patricia Puhlik-Doris, Gwen Larsen, Jeanette Gray & Kelly Weir. 2003. Individual differences in uses of humor and their relation to psychological well-being: Development of the Humor Styles Questionnaire. Journal of Research in Personality 37(1) 48-75) was applied to the memes. To determine the linguistic multimodal digital humor types, Vásquez, C. 2019. Language, creativity and humour online. London). Terminology was employed for annotation. The statistical analysis reveals only slight gender differences concerning all variables. Significant differences were found concerning both multimodal humor frameworks, language usage, canvas and humor themes across all age groups. The interconnectedness of the codes is captured in a qualitative analysis of select memes to underscore the semiotic dynamics of multimodal humor.
... It is here that the work carried out within the field of multimodal analysis in the past thirty years comes to the rescue (Stöckl 2024). Whether from a social semiotic perspective (Kress and van Leeuwen 2021), an analytically and empirically driven perspective (Bateman et al. 2017), or an interactional perspective (Norris 2019), to name three prominent paradigms, multimodality research accounts for the not negligible and often significant part paraverbal and nonverbal means of communication play in expressing arguments and realizing argumentative discourses. The paraverbal includes, for instance, voice qualities, intonation/pausing, co-speech gestures or typography and layout etc., whereas the nonverbal subsumes, for example, image, sound (music and noise) or facial expressions etc. Multimodality research thus provides frameworks and methods for studies in multimodal meaning-making, which rhetoric and argumentation scholars simply cannot ignore in their efforts to model multimodal argumentation. ...
... According to the multi-perspectival outreach of our approach (Blasch, 2021), we combine quantitative and qualitative procedures. The basis for the quantitative analytic account is, in large parts, qualitative coding with the codebook developed in a data-driven ethos (see Norris, 2019) and an abductive-cyclical procedure: ...
In this article, we investigate (partly guided) conceptualisations of "peace" (and "war") in children's school drawings and their accompanying textual framings. We draw on a transdisciplinary framework grounded in ethnography and metapragmatics, combining tools from socio-pragmatic (critical) approaches to multimodal discourse. Our data consists of authentically generated, photographed image-text worksheets that were publicly displayed on the fence of a primary school in a small town in Northern Italy in April 2022. Combining qualitative and quantitative analytical procedures, the (textual and multimodal) conceptualisations range from peace as a very concrete mode of secure-relaxed experience of basic relationships, of home and togetherness, and of self, to peace as care and unity on a more (global-)political scale. Contrary to ideologies on children's drawings as naïve-unmediated "windows" to inner states, our analysis shows how the trans-/locally re-/produced repertoire(s) of multimodal frozen mediated actions (including emblematic patterns such as emojis, peace-flags, comics-speech bubbles, etc.) are deployed ranging from realistic scenes to abstract and complex visual designs. Thereby, children show themselves as literate and often humorous-creative practitioners of visual communication.
... Utifrån det totala filmmaterialet skapades initialt en översikt med teman och tillhörande stillbilder. Från översikten skapades tematiska multimodala transkriptioner, det vill säga transkript där bild, text och samtalsanalytiska konventioner samspelar (Broth & Keevallik, 2020;Norris, 2019). Skapandet av transkripten är en filtreringsprocess (Ochs, 1979, s. 44) där ögonblick väljs ut som representationer för skeendet (Duranti, 2006, s. 309). ...
Multimodal kommunikation kring estetiska aspekter i ett slöjduppdrags idéutvecklingsfas är fokus i den här artikeln. Kommunikativa aspekter vid undervisning i slöjd har tidigare undersökts ur olika perspektiv: hur läraren förmedlar hantverkskunskaper och att elever lär av varandra. Även materialets roll i skapande processer har varit av intresse: hur materialet inspirerar, samarbetar eller ger motstånd i hantverksprocesser. Empirin som ligger till grund för artikeln kommer från en videodokumenterad forskningslektion i slöjd med 8 ̶ 9 år gamla elever. Intresset riktas mot hur lärarens inledande undervisning, materialet, rummet och elevernas estetiska erfarenheter bidrar i idéutvecklingen. I analysen används begreppet collaborative imagining, att föreställningsförmåga är en gemensam resurs för kommunikation vid utveckling av idéer. Vi kunde se att föreställningsförmåga gav eleverna både imaginära och konkreta kommunikativa resurser i idéutvecklingen. Resultatet visar att eleverna använde föreställningsförmåga kollaborativt när de i samarbete utvecklade idéer i handling genom samspel med materialet och med inspiration från egna och gemensamma estetiska erfarenheter. Keywords: slöjd, idéutveckling, embodied learning, kollaborativ föreställningsförmåga, estetiska erfarenheter, multimodal transkription
... Methodologies often involve integrating video, audio, sensor, and survey data, yet these studies commonly report inconsistent data quality, ethical concerns, and difficulties in cross-modal analysis. Results indicate that while there is recognition of the importance of data governance, few have outlined comprehensive frameworks tailored to the specific challenges of multimodal data [19,20]. Limitations include a lack of clear guidelines for ethical data use, standardization across modalities, and efficient data processing pipelines. ...
In the digital era, multimodal behavioral research has emerged as a pivotal discipline, integrating diverse data sources to comprehensively understand human behavior. This paper defines and distinguishes data governance from mere data management within this context, highlighting its centrality in assuring data quality, ethical handling, and participant protection. Through a meticulous review of the literature and empirical experience, we identify key implementation strategies and elucidate the benefits and risks of data governance frameworks in multimodal research. A demonstrative case study illustrates the practical applications and challenges, revealing enhanced data reliability and research integrity as tangible outcomes. Our findings underscore the critical need for robust data governance, pointing to future advancements in the field, including the development of adaptive governance frameworks, innovative big data analytics solutions, and user-friendly tools. These enhancements are poised to amplify the utility of multimodal data, propelling behavioral science forward.
... After multimodal discourse analysis was initially put forward (Kress and van Leeuwen 1996;O'Halloran 1999), a fundamental issue is to reconsider the statuses of different modalities as the semogenic resources in meaning-making processes. The default condition is that all modalities have an equal stance in making meanings to produce discourses in context (Kress 2010;Norris 2019). This viewpoint was coined "modal democracy" (Krug and Frenk 2006), which meant the "democratic stance that all modalities are equal" (Page 2010: 4). ...
This article presents a fresh perspective on the interplay among visual, verbal, and auditory modalities, positing that these modalities, as semogenic resources, compete to express dynamic meanings. The theoretical paradigm emphasizes that whether a modality or an element within a modality gets or loses semantic status, it will elicit an additional layer of social meaning to depict a comprehensive picture of a story together with an explicit semiotic meaning. The article adopts a qualitative method to analyze the data, which are drawn from The Good Wife and My Roommate is a Gumiho and annotated in ELAN 6.3. It was found that modal competition can shed light on the dynamic meaning-making processes in semiotic and societal orientations. Modal competition may distort space and time of different stories, and reconstruct a different discursive spatio-temporal dimension in the TV world. It can diversify the dynamic orientations from New to Given in visual, verbal, and auditory texts of multimodal discourses to tell stories. Modal competition provides a lens to understand the multidimensional reality and to appreciate the aesthetics of a modern TV series.
... The concept of multimodal (inter)action analysis as a framework was initially introduced by Norris (2004Norris ( , 2011Norris ( , 2019. It refers to "a holistic analytical framework that understands the multiple modes in (inter)action as all together building one system of communication" (Norris & Pirini, 2016, p. 24). ...
Technological advancement has enabled language learners to employ verbal and nonverbal cues in computer-mediated communication (CMC). These cues can support language use for learners wishing to communicate more effectively in English. Interactive alignment is one phenomenon that shows how humans tend to collaborate in their language use by adapting, priming, and reusing verbal and nonverbal cues to achieve mutual understanding. Informed by a sociocognitive framework, this study explored and documented English language learners' multimodal interactive alignment during their CMC task engagement through Instagram. We collected data from 30 first-year Indonesian business school learners who participated in seven online CMC tasks using Instagram chat features: text chat, voice chat, and video chat. To examine various interactive alignments (e.g., how interlocutors adapt, prime, and reuse verbal and nonverbal cues to achieve mutual understanding) that occurred during multimodal task communication, we employed multimodal (inter)action analysis. Findings revealed that learners adapted and reused various nonverbal features (e.g., emojis, GIFs, facial expressions, gestures) and verbal cues (e.g., expression, lexical) to convey and comprehend meaning during CMC task completion. Caveats about using various nonverbal alignment patterns for supporting better English online communication were also noted. The study highlights how language learners use the full repertoire of semiotic resources in CMC to maximize their online language learning.
... They can also be connected to larger academic discourses on the connection between micro and macro discourses and actions as argued by the Scollon and Scollon (2004), Ron Scollon and Scollon (2004), and by Jan Blommaert's work on chronotopes from 2017 until 2019 (Blommaert 2015a(Blommaert , 2015b(Blommaert , 2018a(Blommaert , 2018b(Blommaert , 2018c(Blommaert , 2018dBlommaert and De Fina 2015;Blommaert and Maly 2019). Finally, this may then make us wonder about the relation of macrosemantics and macro-acts with discussions on lower and higher-level acts or actions as presented by Jay Lemke (2000) and Sigrid Norris (2004, 2019) (see also Al Zidjay 2019). ...
... This was done through the development of a carefully designed annotation approach and with the help of trained students. The texts in the corpus can be considered as collections of language and communication in context and can be considered suitable for corpus-based multimodal discourse analysis(O'Halloran 2011, Norris 2019, Baldry & Kantz 2022 ...
This article presents a preliminary analysis of a corpus of texts relating to the 2022 Australian Tennis Open using a multimodal appraisal framework. The study utilises quantitative and qualitative content analysis to examine media reports, official statements, and public reactions to the incident, which centred around Novak Djokovic's vaccination status. The analysis focusses on assessing how evaluative language contributes to community-building and identifies the underlying values, beliefs, and evaluations that shape stakeholders' emotional, cognitive, and behavioural responses.The appraisal framework, encompassing attitude, engagement, and graduation, serves as a comprehensive tool for categorising resources that express evaluation. Furthermore, the article delves into the application of appraisal analysis within the context of multimodal and online discourse, encompassing various platforms such as newspapers, television, radio, YouTube, Twitter, Instagram, blogs, official political statements, and court rulings. By examining these diverse media, the study seeks to investigate the dynamic discourse interplay surrounding the 2022 Australian Open, highlighting the pivotal role of evaluative communication in fostering alignment among readers through shared values and attitudes.The preliminary findings suggest that access to greater semiotic recourses increases consensus. The gains from using this interpretative framework are an asset, facilitating the coding of a large data set and attending the different manifestations of discourses around the player’s participation. As discourse continues to shape societal narratives, this multimodal appraisal investigation contributes to our understanding of the complex dynamics inherent in discourse construction and the influence of evaluative language in shaping collective perception.
... This definition of modality is more relevant to our study. This is illustrated by the way our human brain receives information from the world and processes it into an understanding of a scenario [29]. In detail, we perceive the world through our five senses (sight, hearing, taste, smell and touch). ...
Prognostics and health management (PHM) is a crucial enabler to reduce maintenance costs and enhance the availability and reliability of manufacturing systems. In the context of Industry 4.0, these systems become more complex and can be monitored by different types of sensors. The quality and completeness of data are crucial factors for the success of any PHM task in this paradigm. Here, we investigate the possibility of exploiting additional data sources in manufacturing besides monitoring sensors, e.g. production line cameras or maintenance reports. We first present the terminologies of multimodal learning and the potential it holds for industrial PHM. We then further explore the development and notable works in this field applied to other domains, look at the relevant works in PHM, and finally present a case study to demonstrate how multimodal learning can be performed to improve PHM processes.KeywordsMultimodal dataMultimodal learningPrognostics and health managementDeep learning
... Multimodal discourse analysis suggests that language and images are interrelated systems of meaning that communicate ideational meaning, interpersonal meaning, and textual meaning (O'Halloran et al. 2018: 460). Norris (2019) notes that much research into multimodality has examined texts or images or the interplay between both texts and images in the production, transmission, and reception of meaning. Interest in music, semiotic modes such as color, and the online dimension are growing, and this case study of the BBC Radio platform is intended to contribute to this. ...
... This was due to a greater number of modal shifts performed in a shorter time. Here, in addition to Norris's (2009Norris's ( , 2019 identification of modal density (achieved through modal intensity or complexity), we propose that modal density can also be achieved through brisk modal shifts. 4 Teachers' own perspectives, as represented in the interviews, corroborated the observations above. ...
... After collecting and organizing the data, I conducted a multimodal (inter)action analysis to identify how children were communicating with one another in a variety of ways and how these communications shaped their responses and understandings of the world. Norris (2019) established an "interdisciplinary approach that has been developed specifically for the analysis of multimodal action and interaction" (p. 2). ...
Traditionally, literacy in elementary classrooms has been defined as reading, writing, and talk. Literature responses for assessment purposes, then, are often limited to what children say, draw, or write after experiencing a text. Although some work examining students’ dramatic responses to literature has been explored by literacy scholars, few studies have examined spontaneous moments of literacy learning that occur. This chapter examines the physical ways early elementary students respond to course content and provides examples of movement being an integral component of the meaning making process in whole group, small group, and individual settings. These examples illuminate the physical, embodied nature of literacy learning and provide alternative considerations for assessing student’s literacy development, especially in areas of vocabulary development, writing, and comprehension. These examples demonstrate the need for a reconceptualization of what counts as literacy in the elementary classroom and highlights the numerous ways students respond to texts that are often overlooked, ignored, or in some cases, discouraged in traditional classrooms. Instructional strategies that encourage alternative responses in the elementary classroom are offered, including a checklist to assist elementary teachers to consider the physical and embodied responses of students as a form of assessment in their classrooms.KeywordsLiteracyMultimodalityReader responsePicturebooks
... Pedagogical communication as subject to multimodal discourse analysis has become increasingly popular during the last decade (Firmansyah, 2018). Another trend refers to the research activities and their outcomes dissemination where visual and verbal tools serve to systematise knowledge (Norris, 2019), to specify the state of affairs in a particular area, for instance, financial issues (Höllerer et al., 2018), regional politics (Cvetkovic & Pantic, 2018). ...
The paper explores the conceptual vision of BRICS in the contemporary world. The study focuses on language and images that are used within BRICS-related institutional communication. We argue that the research is important because of the increasing impact of BRICS on the development of the multilateral and multipolar world. The research aims to offer preliminary considerations with regard to key topics, features and tools of multimodal discourse that comes from the BRICS nations and representatives of other international/regional organisations. This area has not been subject to academic analysis so far. This confirms the novelty of the present study.
The research material includes 600 image-text correlated items from BRICS official sources of information and from organisation and institutions, which are not affiliated with the BRICS and refer to national or international actors. The research combined theoretical analysis of literature, empirical investigation of materials within qualitative paradigm, through content-based analysis and manual coding on thematic and pragmatic criteria.
The findings reveal different approaches to BRICS that are introduced by different actors through specific coordination of verbal and visual tools, in explicit and implicit ways. The findings show that BRICS sources contain proportioned use of texts and photos of high-ranking official events, socio-cultural features of BRICS countries, and pictures of youth with regard to BRICS mission, values, goals, and policies. This strengthens the concept of equality and human rights provision in the modern world in general and leads to the understanding of the need to include the issues of youth rights and their equality on the BRICS agenda in an explicit way.
... This multisite case study (Yin, 2018) of two early childhood teachers and their classes used a qualitative, multimodal design (Norris, 2019) including educator interviews at the beginning and end of the year as well as fortnightly researcher visits to the class, which were audio recorded and later transcribed. On completion of the year of visits and interviews, data were analysed. ...
Environmental education across the early years has become increasingly important in Australia since the implementation of the Early Years Learning Framework and the Australian Curriculum. These documents promote a connection to nature for young children as well as environmental responsibility. In Western Australia, large areas of natural environments are bush spaces, accessible by young children, families and schools. There is no existing research investigating early childhood teacher’s knowledge of plants in these bush spaces and the utilisation of these spaces in teaching botany as part of their teaching practice. The discussion in this article examines part of a larger year-long multi-site case study of the changes in the botanical understanding of two early childhood teachers of children aged 5–8 years, in Western Australian schools both before and after the Mosaic Approach, botanical practices and Indigenous knowledges were incorporated into their teaching practice. This article focuses on the changes of botanical literacies of the early childhood teachers specifically. The findings suggest that using inquiry-based and place-based methods and including First Nations Peoples’ perspectives about plants whilst teaching in the bush can significantly increase the plant knowledge and understanding of teachers, as well their own scientific and botanical literacies.
... According to this integrated logic, disability could be re-assembled as the materialization of the social into the local order of the senses, the body and the surrounding materiality (Schillmeier, 2007); 3) Finally, the analysis employed the concept of "voices" as this is appropriated within mediated discourse research (Blommaert, 2005;Scollon & Scollon, 2004) to capture the dialogical activity of the activist which occurs within a profound heteroglossic context. The hybrid multimodal model described is mainly borrowed from multimodal discourse analysis (MMDA) (Kress, 2001) and also multimodal analysis of (inter)action as this is outlined by Norris (2004Norris ( , 2019. The central analytic interest Finally, the analysis was also permeated by the notion of nexus, which allows for an integrated attention into the micro and the macro-level of discourse (Scollon & Scollon, 2003). ...
... All communication modes were also used in the analysis which poses certain risks, something which might amplify the element of interpretation. There are, however, methods to analyse non-verbal communication modes, 39 which we used when further analysing three of the interviews, confirming of the themes described in this paper. 40 ...
Background
The impact of the COVID-19 pandemic on people with disabilities has been described as a ‘triple jeopardy’. Not only have they experienced the negative social impacts of disease control measures, but access to required health services has been affected, and, not least, they are at increased risk of severe outcomes from COVID-19. This study aimed to determine how children with disabilities have experienced the pandemic in Sweden and its impact on their lives.
Methods
Six children (5–13 years) were interviewed via video conferencing. An interview guide was adapted based on the children’s communicative abilities and included augmentative and alternative communication support. Reflective field notes were included in the analysis. The data were analysed using qualitative content analysis.
Results
Two themes were identified: The child’s knowledge of Corona raises anxiety and fear; and Boring Corona makes the child even lonelier. The children had knowledge about and were worried about COVID-19, primarily about illness and death of their grandparents. The children longed for their grandparents and other social contacts at school, and life was described as boring and lonely. Many families lacked adequate tools to communicate with their children about the pandemic.
Conclusion
Given adequate support, children with disabilities and communication difficulties can give insights to their unique life situations. The interviewed children reported significant impact on their life and school life. Children were worried about their grandparents based on their knowledge about the virus. The enthusiasm with which the children engaged in the interviews is testament to the need and right of all children, regardless of communicative competence, to voice their experiences
Multimodal (inter)action analysis offers a powerful and robust methodology for the study of action and interaction between social actors, their environment, and the objects and tools within. Yet its implementation in the analysis of synchronous multimodal online data sets, e.g. (inter)actions via videoconferencing, is limited. Drawing on our research in understanding teacher-learner (inter)actions in instruction-giving fragments in synchronous multimodal online language lessons, we describe and illustrate the ways in which we adapted and extended some of the methodological and analytical tools. These include (1) the use of a grounded theory approach in delineating and identifying higher-level actions, (2) the embodiment and disembodiment of frozen actions, (3) electronic print mode, (4) semiotic lag, (5) semiotic (mis)alignment, (6) modal density (mis)alignment, and (7) how modal density can be achieved by brisk modal shifts in addition to through modal intensity and complexity. We conclude by a call for further educational research in online teaching platforms using the framework to have richer understandings of the (inter)actions between social actors with particular roles and identities (teachers-learners), their environment, and the objects and tools within, which bring their “own material properties, feel and techniques of use, affordances and limitations” (Chun, Dorothy, Richard Kern & Bryan Smith. 2016. Technology in language use, language teaching, and language learning. The Modern Language Journal 100. 64–80: 65).
Résumé
L’un des objectifs de la conception des modules en ligne est
d’engager les étudiants dans le processus d’apprentissage. Plusieurs
travaux de recherche (Casimiro, 2016 ; Oh et Kim, 2016 ; Zhu, 2006)
montrent que le niveau d’engagement cognitif avec des ressources en
ligne est directement lié aux compétences acquises grâce à ce mode
d’apprentissage. Cependant, engager les étudiants en ligne et mesurer
cet engagement reste un processus complexe. Par ailleurs, l’un des défis
de la recherche est d’identifier une méthodologie appropriée qui permet
à la fois aux chercheurs et aux praticiens de développer ou de trouver
des stratégies adaptées, de les mettre en place et de tester leur efficacité.
L’objectif de cet article est de présenter les résultats d’une recherche sur
l’engagement étudiant en ligne dans le cadre de l’approche de
Recherche Orientée par la Conception (ROC), qui a été adoptée pour
mettre en oeuvre un module en ligne ALIVE (Apprentissage en Langues
et Interculturalité Via un Environnement numérique) au sein de
l’EHESP dans le cadre du projet DESIR (Développement d’un
Enseignement Supérieur Innovant à Rennes). Nous cherchons à
comprendre : 1). L'impact du ROC sur la conception des cours et sur la
pratique professionnelle des chercheurs et des praticiens. 2) Dans quelle
mesure le module en ligne proposé, basé sur le modèle itératif de la
ROC, favorise l'engagement des étudiants (n = 51). Nos résultats
permettent de montrer les limites des outils utilisés pour mesurer l’engagement étudiant et reflètent l’impact de la méthodologie du ROC sur les dynamiques de collaboration chercheurs-praticiens et sur la conception du module.
Mots clés :
Engagement, apprentissage en ligne, approche orientée par la
conception.
Esta investigación busca analizar el uso de la red social Twitter (X) por parte de las candidaturas presidenciales en el contexto de la primera ronda de la campaña electoral de 2022 en Costa Rica. Esta es una propuesta que se articula bajo un alcance exploratorio y descriptivo, por lo que en primer lugar pretende comprender las características bajo las cuales las y los políticos ejercen presencia en entornos digitales y hacen uso general de Twitter en el contexto de una campaña electoral, en segunda instancia, busca identificar la dimensión programática en las publicaciones compartidas por los perfiles de las candidaturas con mayor intención de voto. El trabajo realiza un aporte a partir del antecedente metodológico del Manifesto Research on Political Representation (MARPOR), para indagar en torno a la comunicación electoral ejercida por medio de una red social, desde un enfoque multimodal. Los hallazgos comprueban la existencia de una clara dimensión programática dentro de la actividad de las candidaturas en la plataforma que se articula no solo con los planes de gobierno, sino también con elementos coyunturales de la campaña como los debates. Asimismo, el trabajo explora en torno algunos usos no programáticos y ofrece algunas nuevas líneas de investigación para el campo de la comunicación política y los entornos digitales en Costa Rica.
Children often prefer nonfiction to fiction books but historically, teachers have neglected nonfiction books during reads alouds. The present study examined how young readers collectively make meaning of nonfiction picturebooks with the help of the teacher and their peers during a whole group interactive read-aloud in one kindergarten classroom. Using Bakhtin’s dialogism and Rosenblatt’s reader response theory, this study captured videos of nonfiction read-alouds, interviews, and formal observations to examine how children make sense of nonfiction picturebooks during whole group read-alouds. This study exposes the social nature of learning. Findings indicate that readers of nonfiction consider the responses of those around them in their takeaways, that making sense of nonfiction is a continual and discursive process, and that children used nonfiction books as a way to connect with one another. Implications for conducting nonfiction read-alouds with young children are discussed. This research exposes the power and potential for interactive read-alouds using nonfiction picturebooks with kindergarteners.
The Covid-19 pandemic was a testbed for crisis communication, leading to recommendations on how to meet communicative goals and several individual case studies. This paper contributes to the latter by engaging in a detailed three-level analysis of an early, pivotal address to the nation by Ghana’s president Nana Akufo-Addo. In terms of infection rates and deaths, Ghana has been much less severely impacted by the pandemic than other countries, making it worthwhile to look at the role of official communications. This study investigates how the president addressed the public at the beginning of the Covid-19 pandemic, in what ways the linguistic features of his address reflected the specific political context, and what potential impact his language use had on the behaviour of the public. Findings show that linguistic and, to a lesser extent, visual elements represent the president as powerful, authoritative, but somewhat detached from the audience. However, this is balanced by direct appeals to the same audience, whose cooperation he seeks to win rather than enforce. This balance reflects the political and socio-cultural context of the text, as further evidenced by comments on the address on Akufo-Addo’s Facebook page.
This thesis studies web homepages to understand the complex social practice of organizational identity communication on a digital medium. It examines how designs of web homepages realize discourses of identity through the mobilization and orchestration of various semiotic resources into multimodal ensembles, addressing critical organizational visual identity elements (‘logo,’ ‘corporate name,’ ‘color,’ ‘typography,’ ‘graphic shapes,’ and ‘images’), communicative content of the page, and navigation structures. By examining these three ‘strata’ of organizational identity communication, it investigates how a homepage uses formal design elements and more abstract principles of composition, such as spatial positioning and content ordering, as resources for making meaning.
The data consists of three complementary sets drawn from thirty-nine web homepages of Australian university websites in 2020. Data set #1 includes four homepages for an in-depth study of organizational identity designs; data set #2 consists of 400 images from the ‘above the fold’ web area as the most strategic space on four homepages between the years 2015 and 2021; data set #3 is comprised of eight historical versions of a selected web homepage between the years 2000 and 2021, with three most representative designs for an in-depth investigation. Grounded in the discourse-analytic approach informed by multimodal social semiotics, the thesis adopts a mixed-method approach to data analysis. It applies multimodal discourse analysis combining the Genre and Multimodality model (Bateman, 2008; Bateman et al., 2017) to document the structural design patterns and social semiotic (metafunctional) approach to address the meaning potentials of the identified patterns; (Kress & van Leeuwen, 2021); content analysis (Bell, 2001; Rose, 2016) and visual social actor framework (van Leeuwen, 2008) to identify key representational tropes and visual personae.
The study reveals the role of design as a mediating tool between the participants of discourse – the rhetor-institution/designer and envisaged audiences – and offers systematic insights into the uses of semiotic resources, both material (e.g., formal design elements and navigation structures) and nonmaterial (e.g., spatial considerations and content structuring), all contributing to the production of meanings and fostering identification with such meanings in the form of association with the university’s identity. Addressing the subtle differences and shifts in the form and function of key layout structures and strategies of viewer engagement, the study concludes that is plural – each university constantly revises semiotic choices and their multimodal composition to achieve specific rhetorical purposes. Together with several visual design choices, five identified strategies of viewer engagement – proximation, alignment, equalization, objectivation, and subjectivation – promote the university as a place of opportunity, achievement, sociality, and intellectual growth for a student as an individual and as a member of the community.
The current research contributes to the emerging collaboration between multimodality, organization studies, and branding, recognizing the complexities and importance of multimodal communication in web-mediated texts amidst the critically increased roles of marketization and social presence in the current higher education landscape.
Red Chinese movies depict heroic individuals who make significant contributions to the nation, and these films play a vital role in constructing the national image of new China. This paper explores the images of heroic individuals in red Chinese movies using multimodal action theory. The findings indicate that these individuals are characterized by their selfless sacrifices for their country, fearless perseverance, embodiment of international communist ideals, and unwavering devotion and warmth. The study also reveals that facial expressions, body movements, and language serve as critical mediational means in shaping these characters, while other forms of mediation are underutilized. Recommendations include employing more mediational means to create more nuanced and complex characters in future red Chinese movies.
This research seeks to analyze the use of the social network Twitter by presidential candidacies in the context of the first round of the 2022 electoral campaign in Costa Rica. This is a proposal that is articulated under an exploratory and descriptive scope, so that in the first place it tries to understand the characteristics under which politicians exercise a presence in digital environments and make general use of Twitter in the context of an electoral campaign, Secondly, it seeks to identify the programmatic dimension in the publications shared by the profiles of the candidacies with the highest voting intentions. The work makes a contribution based on the methodological background of the Manifesto Research on Political Representation (MARPOR), to inquire about electoral communication exercised through a social network, from a multimodal approach. The findings prove the existence of a clear programmatic dimension within the activity of the candidacies on the platform that is articulated not only with government plans, but also with conjunctural elements of the campaign such as debates. Likewise, the work explores some non-programmatic uses and offers new lines of research within the field of political communication in Costa Rica.
The construction of Samoan identity in New Zealand is shaped by complex historical and contemporary social, economic, cultural and political factors. In addition, New Zealand-born Samoans are negotiating an ethnic and identity that incorporates their experience with the intergenerational stories and cultural knowledge of their ancestors. Such cultural and identity negotiations are occurring through the practice of Siva Samoa, Samoan dance. Using Multimodal (Inter)action Analysis we conduct a micro analysis of two excerpts of video data involving female Samoan dancers rehearsing for a dance showcase. Vertical identity production (Norris, S. (2011). Identity in (inter)action: introducing multimodal (inter)action analysis. Berlin and Boston: Mouton: 179, 2020:85) is used as a framework to analyze the multiple layers of discourse within each site of engagement that shape the construction of Samoan Identity for the participants involved.
Tarafların aynı dili ve ekinsel artalanı paylaşmadığı, bir sözlü çevirmen aracılığında gerçekleşen bildirişim ortamında, çevirmenin rolü katılımcılardan birinin kaynak sözcelerinin erek dile aktarılması ile sınırlandırılamaz. Bu sebepten, diller arası çeviri kadar diliçi ve göstergelerarası çeviri ediminde de bulunan çevirmene biçilen rol, durağan bir çerçevede değil, o bildirişim ortamının devingenliğinde ele alınmalıdır. Böylesi devingen bir yapıya yönelik araştırmalarda, dildışı olana kıyasla ekseriyetle dilsel olanın öncelendiği düşünüldüğünde, etkileşimin dildışı bileşenlerinin bildirişim ortamına verdiği katkıya yönelik çalışmalara ihtiyaç duyulmaktadır. Her iki gösterge dizgesini de bütüncül bir bakış açısıyla eşit ölçüde merkezine alan çalışmalarda, çoklu ortam yaklaşımının (multimodal approach) benimsenmesi önem arz eder. Çoklu ortam yaklaşımı, çevirmenin nasıl ve hangi ölçüde diğer eyleyenlerin davranışlarına müdahil olduğunun gözlenmesine imkân tanır. Bu çalışmada, çoklu ortam yaklaşımı çerçevesinde, sözlü çevirmenin muhataplarının dildışı eylemlerine verdiği tepkilerin etkileşim ortamına olan katkıları incelenecektir. Bu doğrultuda, Ekman ve Friesen’in (1969; 1981) dilsel iletişim öğelerinin dildışı iletişim öğeleri ile etkileşimine dair ortaya koydukları altı parçalı sınıflandırma, sözlü çevirmenin rolünün göstergelerarası çeviri bağlamında irdelenmesinde yol gösterici olacaktır. Çalışmanın derlemini görsel işitsel kayıt altına alınan iki ayrı diş ve saç estetiği özgün diyalog ortamının çevriyazısı oluşturmaktadır. Sözlü çevirmenin göstergelerarası aracılığının katılımcıların sözcelemsel varlığının tesis edilmesinde önemli bir işlevinin olduğu ortaya koyulmuştur.
Politeness and impoliteness are not just expressed by words. People communicate polite and impolite attitudes towards each other through their intonation, tone of voice, their facial expressions, their gestures, the positioning of their bodies towards each other, and so on. This volume brings together eleven empirical studies that investigate these various modalities of im/politeness across signed, spoken and written languages, plus a detailed introductory chapter that establishes a framework for the multimodal investigation of im/politeness. The papers cover a range of languages and cultures, including Swiss German Sign Language, Catalan Sign Language, English (as a native language and as a lingua franca), Korean, Catalan, Persian, Japanese and Spanish. Using a range of data sources and state-of-the art methodologies, the papers reveal that these multimodal features are essential aspects of im/politeness across different languages, cultures and modes of interaction. Put together, the findings from these studies lay the groundwork for a new understanding of im/politeness which is fundamentally multimodal.
This paper is a critical appreciation of some of Gunther Kress’s central (social) semiotic notions: i.e., motivation, materiality, rhetorical aptness and semiotic mode versus medium. These will be discussed in relation to four landmark models of sign-making and semiosis by Saussure, Peirce, Bühler and Jakobson. Based on these comments, the paper identifies the persistent difficulties current multimodality research faces in defining mode and in devising linguistically unbiased grammars of non-verbal modes. Finally, the argument is advanced that multimodal genre and discourse interpretation in particular deserve to be re-developed. The paper critiques Kress’s insistence on motivation as a universal principle of sign use and his overemphasis on materiality to the detriment of grammar, while praising his overall (social) semiotic legacy for multimodality research as far-sighted and lastingly influential.
This study examines the different meaning in the visual analysis of Bli-Bli 12.12 and Lazada 12.12 advertisements. This study aims to determine the difference in meaning in visual analysis using the theory of reading images from Kress and Leeuwen (2006). Visual data analysis on these two advertisements is divided into three levels, namely representational, interactional, and compositional. The data is in the form of screenshots taken from the ad videos of bli-bli 12.12 and lazada 12.12. This research method uses descriptive qualitative research. The results showed some similarities and differences between the two advertisements in the theory of Reading images by Kress and Leeuwen (2006). The similarities in the visual data of the two advertisements can be seen in the dominance of advertisements by primary announcements and the difference in the different amounts of data in each part of the analysis point.
In October 2018, a collaboration between young rap artists in Thailand’s Indy rap scene, Rap Against Dictatorship (RAD), launched a video criticizing the ruling Junta that went viral within days of publication. The Junta soon after released its own video as a response to RAD. The production and publication of both videos are what Scollon (2001) calls social actions mediated by a distinct cultural toolkit. This study analyzed how modes such as music, text, color, camera angle, gestures, voice, image and iconicity emerged in both videos to realize scalar differences in civic participation. The Junta’s video represents a high sociolinguistic scale, whereas RAD realizes a lower scale. In a time of political unrest in Thailand, sociolinguistic scale and the semiotic resources that people employ to realize scales are a lens to analyze how different stakeholders address various perspectives of the political situation and appeal to different levels of civic participation.
As has been pointed out in previous research, teacher-led learning plays an important role in developing preschool children's technological skills and technological self-esteem. What is missing in research are more detailed analysis of how the children’s and teachers’ actions and interactions shape the learning process. In order to study this within the field of construction, an action research project was conducted, where construction activities were developed, implemented and revised in an iterative procedure. Data from the second cycle were analyzed for this article using graphic transcriptions and multimodal analysis, with a focus on action, interaction and experience from a pragmatist perspective. Our findings show that children who quickly and decisively engage with the material, the teachers and their peers in suggesting which material to use and/or how the material can be used, end up in a central role in the design process. These children (or their actions) often get legitimized by the teachers. Thus, in order to give children access to equal opportunities in the construction activities, it is important for teachers to understand how the children’s construction-focused actions become constitutive and what their role in that process is.
Este artigo tem como objetivo investigar a (co)construção metafórico-multimodal que advém do impacto discursivo de ações e de concepções capacitistas em relação ao autismo. Para tanto, adoto uma aproximação entre a Análise de Discurso Crítica, orientado por Fairclough (2003, 2006), os estudos da metáfora conceptual, influenciado por Fauconnier e Turner (2003, 2008) e por Vereza (2010, 2017), a metáfora multimodal, encaminhada por Forceville (1988, 2009) e por Sperandio (2015), e a Sociolinguística Interacional, em consonância com Goffman (1998 [1979]) e com Tannen e Wallet (1998 [1987]). No âmbito da pesquisa qualitativa, o corpus desta pesquisa foi gerado a partir de discussões com pessoas autistas sobre um texto multimodal publicado na revista Saúde, em 2019, por meio da plataforma WhatsApp. Os resultados apontam que metáforas socioculturalmente situadas, instanciadas em práticas sociais que representam e que identificam pessoas autistas como ‘solitárias’, ‘seres de outro planeta’, ‘anjo azul’, subjazem a metáfora conceptual AUTISMO É ESTAR FORA, colaborando com a manutenção do capacitismo em estruturas sociais.
In semiotics there is an analysis called multimodal analysis. This study aimed to analyze the advertisement of Colgate in audiovisual formed, when Colgate advertisement is one of the Super Bowl commercial breaks in USA. This study applied on semantic approach focused on the multimodal discourse analysis. The method applied in this study was descriptive qualitative method. The result of this study was that Colgate Total advertisement covers five aspects in a multimodal semiotic system, namely linguistic, visual, audio, gestural, and location aspects. These five aspects come together to make the ad appearance very attractive and easy to remember. A product that is advertised, if many people are able to remember the words, visuals, accompanying music, and actors in the advertisement, it means that the message in the product has successfully reached the audience. It could be predicted that these products would be able to survive in the market. This ad for Colgate Total toothpaste products displays clear and interesting information about what people want from a product, namely its ingredients and the benefits of using it. Messages in the form of language carried out through spoken language and written language make the audience understand more about what is being promoted. This ad makes a unique impression on the audience because of its short duration but important overall message. The linguistic, visual, audio, gestural, and location aspects of a multimodal semiotic system were all addressed in this advertisement. These five elements combine to create a visually appealing and memorable advertisement. Keywords: multimodality, advertisement, analysis
This study aims to examine the difference in meaning in verbal analysis in Gojek and Grab advertisements. The purpose of this research is to find the difference in meaning in verbal analysis at the level of Systemic Functional Linguistics, namely ideational, interpersonal, and textual metafunction. The data were analyzed using the theory of Systemic Functional Linguistics Halliday (2006). The data is the utterances contained in the Gojek and Grab advertising videos. The method of this research uses descriptive qualitative research. The results showed that there were some differences in the two advertisements in using Systemic Functional Linguistics, namely mood analysis, attitude, modality, graduation, and theme. The five differences are dominated by Gojek advertisements.Key words: systemic functional linguistics, semiotics, advertisementAbstarkPenelitian ini bertujuan untuk mengkaji perbedaan makna dalam analisis verbal pada iklan Gojek dan Grab. Tujuan dari penelitian ini adalah untuk mengetahui perbedaan makna dalam analisis verbal pada tingkat Systemic Functional Linguistics yaitu ideational, interpersonal, dan textual metafunction. Data dianalisis dengan menggunakan theory Systemic Functional Linguistics Halliday (2006) dengan pedekatan semiotik. Datanya adalah ujaran-ujaran yang terdapat didalam vidio iklan Gojek dan Grab. Metode dari penelitian ini menggunakan descriptive qualitative research. Hasil penelitian menunjukkan bahwa terdapat beberapa perbedaan dalam kedua iklan dalam menggunakan Systemic Functional Linguistics, yaitu mood analysis, attitude, modality, graduation, dan theme. Lima perbedaan itu didominasi oleh iklan Gojek.
ResearchGate has not been able to resolve any references for this publication.