Book

Systematically Working with Multimodal Data: Research Methods in Multimodal Discourse Analysis.

Authors:

Abstract

A guide that offers a step-by-step process to data-driven qualitative multimodal discourse analysis Systematically Working with Multimodal Data is a hands-on guide that is theoretically grounded and offers a step-by-step process to clearly show how to do a data-driven qualitative Multimodal Discourse Analysis (MDA). This full-color introductory textbook is filled with helpful definitions, notes, discussion points and tasks. With illustrative research examples from YouTube, an Experimental and a Video Ethnographic Study, the text offers many examples of how to deal with small to large amounts of data, including information on how to transcribe video data multimodally, including online videos, and how to analyze the data. This textbook contains ample theory, directions for literature, and a teaching guide to help with a clear understanding of how to work with multimodal data. Contains new research data, exceptional illustrations and diagrams Offers step-by-step processes of working through examples, transcriptions and online videos Goes into great depth so that students can use the book as hands-on material to engage with their own data analysis Designed to be easy-to-use with color-coded definitions, tasks, discussion points and notes Written for advanced undergraduate, graduate and PhD level students, as well as participants in research workshops, Systematically Working with Multimodal Data is an authoritative guide to understanding data-driven qualitative Multimodal Discourse Analysis.
... My methodological concerns primarily derive from the developing field of multimodal analysis of (inter)action, as this is described by Norris (2004Norris ( , 2014Norris ( , 2019. The analysis commits to the study of mediated actions and the way that these are distributed across the activity of video. ...
... Beyond early considerations, that followed an unproblematic logic with regards to transcription, I decided to take a fine-grained approach to convert multiple data into text. Baldry and Thibault, (2006), Hammersley (2010), Norris (2019) and Ochs (1979), advocate for a transcription method that comes as a process of active choices which on the whole are driven by the motivations, theoretical origins and aims of the researcher. O'Reily and Parker ...
... C) Finally, one of the major concerns in the transcription strategy revolved around the identification of modes that build up the multimodal landscape. Drawing from the theoretical discussion which treats mode as a social-cultural artefact (Kress, 2010) I tried to identify different modes as constructive, situated resources for meaning-making that are manipulated by the social actor in the context of particular interactions and within specific social environments (Norris, 2019;Pirini & Norris, 2018). I entered in a recursive reading of the audio-visual material, in order to initially become accustomed to the overall layout of the film. ...
... What makes MIA coherent is that all parts from philosophy to theory, method and methodology are interconnected. On the overarching philosophical strata, MIA posits that human action, interaction and identity come about through a primacy of perception and a primacy of embodiment (Norris, 2019). On the theoretical strata, MIA follows MDA (Scollon, 1998(Scollon, , 2001, insisting on the principles of social action (including communication) and history (Norris, 2020). ...
... The method part is divided into four phases (Norris, 2019), all of which depend on audio-visual technology, but none are technology-specific. In other words, researchers can use the kind of technology they have easy access to, without having to download any kind of specific software. ...
... In Phase III, interviews, videos as well as all other data are transcribed into higher-level action tables and the higherlevel actions are then bundled. By doing this, we systematically work through all of the data that has been collected in a consistent manner, allowing us to work in a datadriven way that moves beyond common interpretive paradigms (Norris 2019(Norris , 2020. As with Phases I & II, easily accessible computer software chosen by the researcher is used. ...
Article
Full-text available
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... Social actors coproduce their identity through social actions, mediational means and the environment. MIA allows for the micro analysis of concrete mediated actions (Scollon, 1998;Wertsch, 1998;Norris, 2002Norris, , 2004Norris, , 2006Norris, , 2009Norris, , 2010Norris, , 2011Norris, , 2012aNorris, , 2012bNorris, , 2012cNorris, , 2013Norris, , 2014Norris, , 2016Norris, , 2017Norris, , 2019. Using the site of engagement to analyse these concrete mediated actions highlights the interrelationship between the mediated actions, the mediational means, the practices and discourses (Jones, 2005;Norris & Jones, 2005;Norris, 2011Norris, , 2014Scollon, 2001). ...
... Using the site of engagement to analyse these concrete mediated actions highlights the interrelationship between the mediated actions, the mediational means, the practices and discourses (Jones, 2005;Norris & Jones, 2005;Norris, 2011Norris, , 2014Scollon, 2001). MIA also offers a systematic approach of working with the data from data collection to data analysis (Norris, 2019). Furthermore, the transcription conventions utilised within MIA do not privilege one mode over others, enabling the analysis of the many embodied and disembodied modes that make up interaction. ...
... Therefore, using both video recordings of social action and interviews as data collection methods will reveal self-identity claims as well as the identity-telling mediated actions that the participants perform. 2.4 Introduction to data analysis Norris (2019) outlines five systematic phases of analysis when working with multimodal data. These phases include data collection, delineating data, selecting data pieces for micro analysis, transcribing data and using analytical tools. ...
Thesis
Full-text available
This thesis examines the construction of hybrid and fluid ethnic identity elements as produced by Māori and Pacific female artists. Ethnic identity studies within New Zealand reveal different types of ethnic identities, and although there is research being conducted into hybrid and fluid Māori and Pacific identity elements, no studies have been done examining the construction of these identities through mediated action. This thesis attempts to fill this gap. Using video ethnography and socio-linguistic interviews, data were collected and analysed utilising multimodal (inter)action analysis (MIA) as the theoretical and methodological framework. Vertical identity production and site of engagement are analytical tools within MIA that allow for the study of the intersection between discourses and mediated actions performed by social actors. These analytical tools were applied to interview and video transcripts selected from the data, following a systematic process of data cataloguing. Analysis of the data is presented in three chapters which show the ethnic and creative identity production of the participants as constructed through the central, intermediary and outer layers of discourse. The first analysis chapter demonstrates the way the participants create art by blending traditional and contemporary features and diverse knowledge, in turn constructing their immediate ethnic and creative identity elements. This analysis is compared to the way the participants verbalise these identity elements within their interviews. The second analysis chapter examines the way experiences of exclusion and inclusion from within their networks shape their continuous ethnic and creative identity elements. The third analysis chapter explores moments of exclusion and inclusion but within larger communities such as mainstream New Zealand, and their ethnic communities. It also illustrates the way in which the participants’ art creates inclusion and shapes the general ethnic and creative identity development of other social actors. Following this, wider discourses and practices are examined using the site of engagement as the analytical tool. This chapter demonstrates the way in which wider discourses such as colonial, superiority/inferiority and racism discourse intersect with practices such as superiority/inferiority, gratitude, and marginalisation and with the mediated actions performed by the participants. This analysis highlights the negative impact these discourses and practices can have on ethnic identity construction for Māori and Pacific social actors. To this end, numerous recommendations are made within the conclusion with the intention of changing these wider discourses and practices. This thesis contributes to knowledge in the area of Māori and Pacific identity studies by utilising multimodal (inter)action analysis to study identity production. It also contributes to the theoretical and methodological framework of multimodal (inter)action analysis.
... In response to Hiippala's (2014) call, we propose one such theoretically grounded analytical approach. Specifically, we draw on a combination of Writing, Activity, and Genre Research (WAGR 1 ; Russell, 1997Russell, , 2009 and Multimodal Interaction Analysis (MIA; Norris, 2004Norris, , 2019 to develop a framework for the study of multimodal genres and their involvement in knowledge-making activities of science. We then use an illustrative example of the multimodal genre of the laboratory notebook, or lab book (Latour & Woolgar, 1986;Wickman, 2010), to apply this approach to the investigation of the disciplinary enculturation of a doctoral medical physics student-an emerging scientist, to use a term coined by Emerson (2016), and further our understanding of the rhetorical action the genre performs and the role it plays in mediating knowledge construction in a medical physics laboratory. ...
... In order to fully understand the roles that this combination of modes play in human activity it is necessary to complement WAGR with an analytical approach that allows researchers to investigate the complex multimodal nature of the lab book and its contributions to knowledge making. We have adapted MIA (Bernad-Mechó, 2021;Norris, 2004Norris, , 2013Norris, , 2019 to inform our study, as "this is the only approach that was especially developed because of and for the analysis of multimodal action and interaction" (Pirini et al., 2018, p. 640). The unit of analysis in MIA is mediated action, which includes "a social actor acting with/through mediational means" (pp. ...
... An important concept in MIA is that of attention; that is, the degree of awareness that participants have of higher and lower level actions during a multimodal interaction (e.g., Norris, 2004Norris, , 2019Pirini et al., 2018). To trace how participants' attention to higher and lower level actions is mediated through modes, MIA uses an attention foreground/background continuum (Norris, 2019, Pirini, 2014. ...
Article
Full-text available
Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation.
... Data collection occurred in the New Zealand participants' homes by one to three researchers at a time (depending upon availability) with a research laptop that had a screen recording software installed and one to two tripod-standing video cameras (depending on need and possibility) recording the interactions of the family members in the homes around the Skype interactions. A few weeks after the recording, (usually) a follow-up phone interview with at least one of the New Zealand adult family members was conducted (Norris 2019). The data in this article, however, comes from the video recorded data. ...
... The data in this article, however, comes from the video recorded data. This particular data piece was then multimodally transcribed following Norris's transcription conventions in order to ensure replicability and reliability and further analyzed in detail (Norris 2004(Norris , 2011(Norris , 2019. ...
... Through a systematic and detailed analysis (Norris 2019), it becomes evident on the one hand that specific micro data pieces selected by researchers from a large amount of data, when a larger point of view is disregarded, can lead to incorrect or partial findings. When, on the other hand, micro-analytical boundaries are crossed, groundbreaking findings can be discovered and exact shifts in a participant's focused attention can be determined (Pirini 2014(Pirini , 2015(Pirini , 2017. ...
Chapter
Full-text available
This article problematizes the notion of selecting micro-data pieces to shed light upon the focus of participants. The issue presented is two-fold: 1. The article shows that selecting micro-analytical data pieces does not allow a researcher to determine the focus of a participant; and 2. The article demonstrates that language use of a participant does not necessarily mean that the participant is focused upon a conversation. Both, purely working with micro-analytical data pieces and the presumption that language use indicates focus of the speaker, are problematized and it is shown with an example from a relatively large study of family Skype conversation that includes 82 participants that: 1. Focused attention can only be analyzed correctly when crossing micro-analytical boundaries; and 2. A participant can utilize language without paying focused attention to an interaction.
... The analytical framework utilised in this study is that of multimodal (inter)action analysis (MIA) (Norris, 2004(Norris, , 2019. MIA is based on the concept of mediated action rooted in the works of Scollon (1998a) and Wertsch (1998). ...
... An important aspect is that social interaction is co-produced through mediational means/cultural tools in which language is not necessarily the centre of attention. Norris (2004Norris ( , 2019 has developed analytical tools that are especially suitable for investigating social actors' engagement in simultaneous activities. ...
... The activities are analysed as higher-level mediated actions (actions with an opening and a closing, such as a meeting) and lower-level mediated actions (pragmatic meaning units of modes, such as an utterance or gaze shift, which has a start and an end). Pirini (2016Pirini ( , 2017 has demonstrated that the analytical tool of modal density as an indication of attention/ awareness (Norris, 2004(Norris, , 2019 can be further developed to determine intersubjectivity and agency. I will return to this in the analytical framework section. ...
Article
Full-text available
This paper presents an analysis of three roleplayed interpreted institutional meetings in which sight translation is part of the interaction. The analysis is based on multimodal (inter)action analysis and utilises the analytical tool of modal density as indication of attention/awareness. This analytical framework is novel in interpreting studies. The data include filmed material from an experimental setting and participants’ reflections about the situation. The findings show variations in sight translation practices and that the shift from interpreting to sight translation affects interactional patterns, particularly social actors’ attention and agency. In my discussion of agency in sight-translated interaction, I argue that interpreters, in addition to translating, need to pay attention to interactional issues related to attention and agency caused by the interpreting method.
... However, few studies explore experienced online teachers' practices in videoconferencing particularly while giving instructions, which are key to success in task-based language teaching (Markee, 2015). Adopting multimodal (inter)action analysis (Norris, 2004;2019) to investigate the multimodal construction of instructions in a single case study, we examine instruction-giving as a social practice demonstrated in a specific site of engagement (a synchronous online lesson recorded for research purposes). Drawing on the higher-level actions (instruction-giving fragments) we have identified elsewhere (Satar & Wigham, 2020), in this paper we analyse the lower-level actions (modes) that comprise these higher-level actions, specifically focusing on the print mode (task resource sheets, URLs, textchat, and online collaborative writing spaces) wherein certain higher-level actions become frozen. ...
... This case study examines how experienced online language teachers harness the print mode (resource sheets, URLs, textchat, Google Docs [online document hereafter]) in instructiongiving practices. Adopting multimodal (inter)action analysis (Norris, 2004;2019) we explore the videoconferencing interface as a site of engagement or "window opened up through practices that make concrete mediated actions possible" (Scollon, 1998in Norris, 2019. We then examine the print mode and its role in mediated actions. ...
... This case study examines how experienced online language teachers harness the print mode (resource sheets, URLs, textchat, Google Docs [online document hereafter]) in instructiongiving practices. Adopting multimodal (inter)action analysis (Norris, 2004;2019) we explore the videoconferencing interface as a site of engagement or "window opened up through practices that make concrete mediated actions possible" (Scollon, 1998in Norris, 2019. We then examine the print mode and its role in mediated actions. ...
Article
Online language teaching is gaining momentum worldwide and an expanding body of research analyses online pedagogical interactions. However, few studies have explored experienced online teachers' practices in videoconferencing particularly while giving instructions, which are key to success in task-based language teaching (Markee, 2015). Adopting multimodal (inter)action analysis (Norris, 2004, 2019) to investigate the multimodal construction of instructions in a single case study, we examine instruction-giving as a social practice demonstrated in a specific site of engagement (a synchronous online lesson recorded for research purposes). Drawing on the higher-level actions (instruction-giving fragments) we have identified elsewhere (Satar & Wigham, 2020), in this paper we analyse the lower-level actions (modes) that comprise these higher-level actions, specifically focusing on the print mode (task resource sheets, URLs, text chat, and online collaborative writing spaces) wherein certain higher-level actions become frozen. Our findings are unique in depicting the modal complexity of sharing task resources in synchronous online teaching due to semiotic misalignment and semiotic lag that precludes the establishment of a completely shared interactional space. We observe gaze shifts as the sole indicator for learners that the teacher is multitasking between different higher-level actions. Further research is needed to fully understand the interactional features of online language teaching via videoconferencing to inform teacher training policy and practice.
... For example, the objects present in a classroom construct the mode of layout, which gives off messages about the social actor and structure the interaction somehow. According to Norris (2004Norris ( , 2019Norris ( , 2020, the higher-level actions are also fluid and develop in real-time, and each higher-level action is bracketed by social openings and closings that are at least in part ritualized. Jewitt (2014, p. 36) holds that multimodal interaction focuses on the mediate interaction in a given context, that is, how a variety of modes are brought into and constitutive of social interaction. ...
... The researchers follow the procedures suggested by Norris (2004Norris ( , 2019Norris ( , 2020 to analyze two EFL teachers' multimodal pedagogic discourse during classroom lead-ins. According to Norris (2004, 2019, 2020), the first step to a multimodal interaction analysis is to understand an array of communicative modes. ...
Article
Full-text available
Classroom lead-in is the initial stage for motivating students to become engaged in-class interaction. However, little research, to our knowledge, has analyzed the role of teachers’ multimodal competence reflected through their multimodal pedagogic discourse in the realization of the ultimate goals of classroom lead-ins. Based on the data collected from a teaching contest in China, this paper explores how two-winner teachers utilize their multimodal ensembles of communicative modes to engage students during classroom lead-ins. The analysis shows that different communicative modes construct the higher-level action of lead-in, and they are orchestrated into multimodal ensembles for the specific function of each lead-in move. The findings indicate that EFL teachers’ high multimodal competence plays a decisive role in performing classroom lead-ins, and different lead-ins strategies influence the different orchestration of communicative modes. In constructing multimodal pedagogic discourse, teachers build up their professional image and display their personal charm as well. Future research for multimodal discourse analysis and pedagogic research is suggested in the paper.
... Examples of multimodal analysis approaches. Modified from Jewitt (2017) Kress (2010) Systemic functional multimodal discourse analysis Linguistic-based approach focused on artefacts Understanding how modes are organized and used to fulfill a range of social functions O'Halloran and Lim (2014) Multimodal (inter)action analysis Action-based approach focused on interactions Exploring how interactions, identities, and social relations are constructed through modes Norris (2004Norris ( , 2011Norris ( , 2019 Mediated discourse analysis (cf. nexus analysis, geo-semiotic analysis) ...
... Data collection methods in MIA analysis are often ethnographic and include multiple sources of data for gaining a deeper understanding of the complexity of communication (Pirini et al. 2018). The primary source of data is often video-recordings of the interaction and, depending on the research question, may include a large data set of similar interactions for the purposes of contrast and comparison (Norris 2019). Other sources of data may include field notes taken at the time of the interaction and other artefacts that played a role in the interaction. ...
Article
Full-text available
The contexts and methods for communicating in healthcare and health professions education (HPE) profoundly affect how we understand information, relate to others, and construct our identities. Multimodal analysis provides a method for exploring how we communicate using multiple modes—e.g., language, gestures, images—in concert with each other and within specific contexts. In this paper, we demonstrate how multimodal analysis helps us investigate the ways our communication practices shape healthcare and HPE. We provide an overview of the theoretical underpinnings, traditions, and methodologies of multimodal analysis. Then, we illustrate how to design and conduct a study using one particular approach to multimodal analysis, multimodal (inter)action analysis, using examples from a study focused on clinical reasoning and patient documentation. Finally, we suggest how multimodal analysis can be used to address a variety of HPE topics and contexts, highlighting the unique contributions multimodal analysis can offer to our field.
... Nexus analysis maintains that a social action intersects at three discourse aggregates: the discourse in place (e.g. a narration of a monument in a schoolscape including the text on the monument); the interaction order (e.g. a conversation during a walking interview);, and the historical body (e.g. the life experiences of the social actors that might emerge during that interview) ( Scollon & Scollon, 2004 , pp. 18-23). Signs and objects in the material world are the result of someone's previous social action and this notion is conceptualized in the notion of the frozen action ( Norris, 2004( Norris, , 2019Norris & Makboon, 2015 ). Frozen actions range in meaning from moving an object onto a table to the emplacement of a public sign or memorial. ...
... As has been noted by Norris (2019) and Pietikäinen et al., (2011) , nexus analysts are not always able to be present when signs and material objects are placed in the material world. However, the history of those objects can be captured through the narrations of social actors who interact with them, or who have been socialized to think about the traces of these frozen actions in particular ways. ...
Article
Full-text available
The present study reports on the geosemiotics (Scollon & Scollon, 2003) of a Thai University. Walking interview tours (Lou, 2017; Stroud & Jegels, 2014a) of Thammasat University’s Tha Prachan campus were conducted. These interviews reveal how students narrate and take stances towards the geosemiotic artifacts that are found on their campus. The purpose of the study was twofold: 1) to gain an understanding of how students react to the geosemiotics on campus, and 2) to get a sense of their understanding of Thai history. For the latter the university has been the site of several historical events pertaining to Thailand’s spotty relationship with democracy, most notably the Thammasat massacre, and much of this history has been repressed (Huebner, 2017; Winichakul, 2002, 2020). Using the geosemiotic framework to discuss the multimodal make-up of this university’s signs and space, I illustrate how the narrations that emerged during walking interviews serve as lenses through which we are offered a glimpse of how students are socialized to think about the material environment of their campus. We can observe how students take different stances to the signs and places of their campus as well as Thailand’s history. Such narrations reveal traces of socialization on the one hand and the emergence of an affective regime of reverence on the other (Wee & Goh, 2019). The findings contribute to the growing body of literature on schoolscapes in multilingual educational settings (Gorter, 02018; Gorter & Cenoz, 2015).
... were partially inspired by mediated discourse analysis(Scollon et al, 2011) and studies of action-oriented approaches to multimodal interactional analysis for classroom observations and examinations of video recorded transcriptions(Cortez, 2008;Jewitt, Bezemer, & O'Halloran, 2016;Norris, 2019;Scollon, ...
Conference Paper
University EFL programs in expanding circle nations often pressure instructors and students to use globally published EFL textbooks for reasons more socio-political than pedagogical. While some critical studies underscore multimodal discourse to be an under-appreciated source of dominant social narratives in EFL textbooks, few have investigated their live negotiation in classrooms. The present study proposes that a two-step framework for achieving a zone of prioritized curricularivity (ZPC) can inform reflexive teaching practices by the instructor. The framework requires an understanding of the curricular commonplaces of a particular EFL program and an understanding of the power and ideologies in the multimodal discourse (Machin & Mayr, 2012) of their textbooks, to mitigate perceived social injustices in lesson contents textbook lessons as they are negotiated “in situ.” Demonstrated in vignettes featuring two EFL courses at Chung-Buk National University in Cheong Ju city, Korea, two instructors used the ZPC framework to inform their reconstruction of multimodal discourses in their EFL textbook lessons to inculcate student involvement and participation. A novel, multimodal interactional analysis (Norris, 2019) of video recordings in two EFL classrooms discovered that each instructor recontextualized, neutralized, or skipped much of the multimodal discourse in the lessons. The findings suggest that a ZPC is achieved when the efforts by instructors to recontextualizing textbook lessons in situ is met with positive feedback from students in the classroom – noted as heightened attentiveness, happy or cheerful participation, and enthusiastic discussion. The implications suggest a ZPC can help instructors and students and in EFL programs in any expanding circle culture because it can simultaneously improve student learning/acquisition in the classroom, diminish dominant, culturally marginalizing textbook content, while raising the value of student investment in EFL learning.
... All communication modes were also used in the analysis which poses certain risks, something which might amplify the element of interpretation. There are, however, methods to analyse non-verbal communication modes, 39 which we used when further analysing three of the interviews, confirming of the themes described in this paper. 40 ...
Article
Full-text available
Background The impact of the COVID-19 pandemic on people with disabilities has been described as a ‘triple jeopardy’. Not only have they experienced the negative social impacts of disease control measures, but access to required health services has been affected, and, not least, they are at increased risk of severe outcomes from COVID-19. This study aimed to determine how children with disabilities have experienced the pandemic in Sweden and its impact on their lives. Methods Six children (5–13 years) were interviewed via video conferencing. An interview guide was adapted based on the children’s communicative abilities and included augmentative and alternative communication support. Reflective field notes were included in the analysis. The data were analysed using qualitative content analysis. Results Two themes were identified: The child’s knowledge of Corona raises anxiety and fear; and Boring Corona makes the child even lonelier. The children had knowledge about and were worried about COVID-19, primarily about illness and death of their grandparents. The children longed for their grandparents and other social contacts at school, and life was described as boring and lonely. Many families lacked adequate tools to communicate with their children about the pandemic. Conclusion Given adequate support, children with disabilities and communication difficulties can give insights to their unique life situations. The interviewed children reported significant impact on their life and school life. Children were worried about their grandparents based on their knowledge about the virus. The enthusiasm with which the children engaged in the interviews is testament to the need and right of all children, regardless of communicative competence, to voice their experiences
... Multimodal transcription involves many challenging theoretical and practical decisions for determining what should be transcribed and how (Norris, 2019). This is partly due to the high dimensionality of the data that means that it can be approached simultaneously from many different perspectives (Zappavigna, 2019). ...
Article
This article explores how digital intimacy is construed through ambient embodied copresence in ‘personal attention’ role play videos, a type of Autonomous Sensory Meridian Response (ASMR) video that has become popular on YouTube. ASMR is the experience of positive sensations in response to visual and aural stimuli. Online video sharing platforms have provided a way for people who experience these ASMR sensations to watch, produce, and disseminate ASMR-invoking material. In ASMR role play videos the YouTuber constructs a conceit (e.g. a beautician visit) and uses visual and aural resources to encourage the feeling in the ambient viewers that they are there with the YouTuber experiencing the interaction. This article considers how these videos forge an immersive faux-interactional context, and invoke the visual and aural perspectives and embodiment of ambient viewers. The dataset explored is a playlist of 116 role play videos from the GentleWhispering ASMR YouTube channel, the most popular ASMR channel at the time of writing.
... З огляду на прагматичну мету висловлення, тобто показати на прикладі основні зміни технічних параметрів нового приладу, мовець використовує ширший тональний діапазон, ніж у фонаційному відрізку, представленому на Рис. 4, який визначається абсолютною тривалістю в 21107 мс, налічує 16 Закарпатські філологічні студії напряму ЧОТ відповідає показникові в 5,92, а частотний діапазон -2,14. Такий фонаційний відрізок не містить емоційно-забарвлених лексичних одиниць, тому автор вважає, що значна кількість інтонаційних груп пояснюється необхідністю донести до уваги аудиторії основну ідею повідомлення, а саме пояснення технічних показників, коментуючи дії, які демонструються на моніторі. ...
... an analysis of multimodal data collected from different sources is established in social sciences as well as the analytical strategy chosen [43]. Data collection and analysis will be thus performed by a multidisciplinary team led by social science researchers experienced in both. ...
Preprint
BACKGROUND Cohort studies represent a strong methodology for increasing understanding of human life-course development and aetiological mechanisms. Retention of participants, especially during long follow-up periods, is, however, a major challenge. A better understanding of motives for participation and for participants’ reluctance to continue in cohort studies in diverse socio-geographic and cultural settings is needed as this information is most useful in developing effective retention strategies. OBJECTIVE We present a study design aiming to improve understanding of participation and attrition phenomena in European cohort of very preterm and/or very low birth weight (VPT/VLBW) studies in various socio-geographic and cultural settings in order to understand variability and to ultimately contribute to develop novel and/or more ‘in context’ strategies to improve retention. METHODS This study uses a triangulation of multi-situated methods to collect data on RECAP Preterm’s cohorts, which include: focus groups and individual semi-driven interviewing with adult participants, parents (caregivers), cohort staff, health-care professionals as well as other relevant key actors and a collaborative reflexive visual methodology (participant-generated VideoStories) with parents and adults participating in the cohorts. The methodological strategy aims to provide a shared flexible framework of various qualitatively-driven methods to collect data on cohorts of VPT/VLBW adults and children, from which local research partners may choose and combine those most adequate to apply in their own specific contexts. Data from all sources and sites will be submitted to a triangulation of phenomenological thematic analysis with discourse analysis. RESULTS This paper focuses only on the research approach used in the study. Findings will be reported in detail in future publications. At the time of publication, the process of data collection is underway. CONCLUSIONS Qualitative research methods are a useful complement for enriching and illuminating quantitative results. Some of the distinctive features of our study approach are the use of open, exploratory research questions and the employment of meaning-based forms of data analysis, which tend to potentiate the emergence of something new. We expect that opting for a multi-situated flexible approach to garner feedback from the various key actors involved with cohorts in different settings will contribute towards filling some gaps in the understanding of participation and attrition phenomena. Moreover, health research subjects have traditionally been positioned as passive objects of study rather than active participants, even though they have the greatest stake in improving health-care and practices. The use of collaborative methods allows to counteract the ‘top down’ model by handing some research control to the very people who are providing the data on which research findings will be based, while also acknowledging the value of their involvement. Findings of this study will be of great value to foster retention of participants in longitudinal cohort studies.
... This study of interaction among deaf and hearing interlocutors makes an original contribution to the study of meaning-making within the field of multimodal interaction analysis (e.g. Norris, 2017) and multimodal conversation analysis (e.g. Mondada, 2012), where shared auditory and oral resources are more usually assumed. ...
Article
Full-text available
Interaction between deaf and hearing interlocutors is examined to demonstrate how understanding (and misunderstanding) can be expressed and inspected through the situated use of multimodal resources. In this communicative situation participants have asymmetrical experiences of being deaf and being hearing and ‘codified’ (either speech or sign-language) resources are little shared among participants. The multimodal analysis of an interactional sequence between a young deaf child, her deaf friend and her hearing mother demonstrates ways in which participants use semiotic resources to take,execute and give turns in the presence of sensory asymmetries. The organisation of turn taking in this sequence provides insights into the ways in which understanding (or lack of it) can be demonstrated, monitored and co-constructed by participants throughout the interaction. The findings demonstrate that turns offer a useful point of analysis for the recognition and inspection of signs of understanding in the context of sensory asymmetries but there needs to be a qualitative orientation to assessing this. This contribution to the research on situated multimodal sign-making underlines the need for the development of multimodal frameworks that can account for, and effectively document, situated meaning-making beyond ‘codified/linguistic’ resources.
... For instance, circling around in small steps is used to indicate inner anxiety in Snow in August. Proxemics refer to the physical distance or closeness between Huineng and other characters (Norris, 2004(Norris, , 2019, which usually indicates the interpersonal relationship. Finally, voice can be used to indicate the gender, age, or mental status of the character. ...
Article
Adopting a multimodal social semiotic approach, this study investigates the resemiotization of the story of Huineng in two theatrical performances, the opera Snow in August performed in Taipei in 2002, and the musical theater The Sixth Patriarch Huineng performed in Shanghai in 2007. Although both claim to have been based on the historical record of Huineng, especially the Platform Sutra, significant differences exist. Focusing on the differences, the study investigates: (1) how semiotic resources of language, music and mise-en-scène are adopted to present Huineng in a particular way; and (2) why the semiotic resources are mobilized to present Huineng in such ways. Analysis shows that while Snow in August presents Huineng as a rebellious, independent thinker through episodes concerning Huineng’s innate enlightenment, breaking the alms-bowl and burning the robe, and his decline of the imperial invitation, The Sixth Patriarch Huineng presents Huineng as a cultural icon by emphasizing his reconciliation with former enemy Shenxiu, and the harmonious relationship between Huineng and the other characters. Each performance is contextualized to reveal ways in which resemiotization is socially situated and ideologically positioned.
... A series of high-inference questions (Dornyei, 2007), detailed in Table 3, underpinning the nature of the EFL textbook use in class served as key points of observation in the analysis of the video recordings. The questions listed in Table 3 were partially inspired by mediated discourse analysis (Scollon et al, 2011) and studies of action-oriented approaches to multimodal interactional analysis for classroom observations and examinations of video recorded transcriptions (Cortez, 2008;Jewitt, Bezemer, & O'Halloran, 2016;Norris, 2019;Scollon, Scollon, & Jones, 2011;Wohlwend, 2011). ...
Article
Full-text available
English as a foreign language (EFL) university programs in expanding circle nations often pressure instructors and students to use globally published textbooks for reasons more socio-political than pedagogical. While some critical studies underscore multimodal discourse to be an under-appreciated source of dominant social narratives in EFL textbooks, few have investigated their live negotiation in classrooms. To address the challenges negotiating potentially harmful social narratives in EFL textbooks, the present study proposes a two-step model for achieving a zone of prioritized curricularivity (ZPC). The model informs reflexive teaching practice in EFL instruction because it necessitates an understanding of a) the curricular commonplaces of a particular EFL program and b) the power and ideologies in the multimodal discourse of their textbooks, to mitigate perceived social injustices in the textbook lessons as they are negotiated "in situ." Demonstrated in vignettes, featuring two EFL courses at Chung-Buk National University in Cheong Ju city, Korea, two instructors used the ZPC framework to inform their reconstruction of multimodal discourses in their EFL textbooks to inculcate student involvement and participation. A novel, multimodal interactional analysis of video recordings looked at proxemics, gaze, spoken language, head movement, auditory emphasis, and gesture and discovered that each instructor recontextualized, neutralized, or skipped much of the multimodal discourse in the lessons. The findings suggest that a ZPC is achieved when the efforts by instructors to recontextualize textbook lessons in situ is met with positive feedback from students in the classroom. The implications suggest a ZPC can help instructors and students and in EFL programs in any expanding circle culture because it can simultaneously improve student learning/acquisition in the classroom, diminish dominant, culturally marginalizing textbook content, while raising the value of student investment in EFL learning.
... For transcription of spoken language, I have employed Jefferson's (2004) well-established conventions. In transcription of other modes besides speech, I have been inspired byNorris's (2011Norris's ( , 2019 way of working with image-transcription, together withMondada's (2016) conventions for detailed multimodal transcription. For further notes on transcription, see Studies I-III. ...
Thesis
Full-text available
This thesis comprises three separate studies that together explore how Swedish student teachers construct or produce professional identity in interaction while navigating different institutional and professional instances of teacher education. As a discourse analytical contribution to research on teacher identity, the main theoretical framework is mediated discourse theory (e.g. Scollon 2001a). For data construction and analysis in the studies, different parts of the two related methodologies of nexus analysis (Scollon & Scollon 2004) and multimodal (inter)action analysis (Norris 2011) are employed. Constructed through an ethnographic approach, the interactional data consist of audio and video recordings of interaction in instances from three different components of a Swedish teacher education program: a rhetoric course, a bachelor thesis course in history and teaching placement. Furthermore, the data include observational field notes and interviews, as well as resources used by the participants, primarily written texts. Taking place early on in teacher education, Study I focuses on student teachers performing oral presentations under the fictitious presumption that they are speaking as teachers. Employing the notion of communicative project (Linell 1998), the empirical aim of the study is to shed light on how student teachers manage institutional affordances and constraints affecting interactional role shifts from student teacher to teacher. In Study II, three student teachers are writing their bachelor theses in the subject of history, and the study focuses on the interactional production of teacher identity of one of the students during seminars. While partly being a methodological study, Study II empirically explores how student teachers interactionally relate to their future profession in an academic disciplinary setting, highlighting which actors and institutions are involved in the production of professional identity. Finally, Study III concentrates on a student teacher during his final teaching placement. Focusing on previous experiences resemiotized as stories, Study III highlights how discourse re-emerging from the historical body (Nishida 1958) can be used in interaction in producing identity. The results suggest that the production of teacher identity by the student teachers is a co-operative and communicative task, where previous experiences as well as an anticipatory perspective on the teaching profession are important features. The three studies identify different resources that can be used and adapted by students to suit different purposes in professional identity production, described as textual resources, embodied resources, and narrative resources. In turn, the different uses of such resources motivate the need for studying identity in interaction with an approach where ethnographic and sociocultural knowledge is part of the analysis. The creative use of resources in identity production highlights that students use knowledge and experience linked to academic and professional as well as everyday discourse in producing professional identity. Presuming an interest in opportunities for student teachers to develop professional identity during their education, it appears fruitful to reflect upon how potential resources are designed and implemented in teacher education, and how institutional affordances and constraints affect the possibilities of using them.
... A mode is a means for making meaning, such as speech, writing, image, sound, color, or gesture, so multimodality refers to people using multiple means of meaning making (Jewitt, Bezemer, & O'Halloran, 2016;Kress, 2010;Norris, 2019). To be considered a mode, a semiotic system of meaning-making requires a community to recognize it as such (Jewitt et al., 2016), like a handshake, or a bow. ...
Article
The present study identifies and maps the reflexive praxis of two experienced English as a foreign language (EFL) instructors as they reconstruct and negotiate textbook material in situ. An abundance of critical studies underscoring social injustices in the contents of globally published EFL textbooks do not sufficiently address the negotiation of their multimodal discourses during class time. Although reflexive teaching practice in language learning classrooms has a robust pool of research, limited scholarly attention has been given to the active negotiation of a textbook's multimodal discourse in Korean university classrooms. The present study asks: (1) How do two instructors at different Korean universities negotiate the contents of an EFL textbook with their students during class? (2) How do the students react to the multimodal discourse negotiated in their textbooks? (3) What pedagogical implications do the findings lend to EFL textbook instruction in Korean university contexts? Using Norris' (2004) framework for video transcription of multimodal interaction in two Korean university English communication classes, the findings reveal that reflexive negotiation between students and instructors is a kind of rhetorical accomplishment that lessens the potential for cultural marginalization in the multimodal discourse of EFL textbooks. Implications suggest that textbook reflexivity in situ raises the value of student EFL learning investments.
... They can also be connected to larger academic discourses on the connection between micro and macro discourses and actions as argued by the Scollon and Scollon (2004), Ron Scollon and Scollon (2004), and by Jan Blommaert's work on chronotopes from 2017 until 2019 (Blommaert 2015a(Blommaert , 2015b(Blommaert , 2018a(Blommaert , 2018b(Blommaert , 2018c(Blommaert , 2018dBlommaert and De Fina 2015;Blommaert and Maly 2019). Finally, this may then make us wonder about the relation of macrosemantics and macro-acts with discussions on lower and higher-level acts or actions as presented by Jay Lemke (2000) and Sigrid Norris (2004, 2019) (see also Al Zidjay 2019). ...
Article
Semantic macrostructures, although strangely ignored or ruled outside most formal linguistics and even some methods of discourse analysis, define the general, overall meanings of discourse, informally called ‘topic,’ ‘theme,’ ‘gist,’ or ‘upshot’ and on the basis of the higher levels of the mental models of the social actors. Macro-structural (topical, crucial) information, according to Teun van Dijk, plays a fundamental role in discourse comprehension and recall― although other factors (such as remarkableness, vividness, etc.) may also affect attention, prominent representation, and hence recall. One can summarise a sequence of pictures by a title, like a summary of a movie. Similarly, a trailer serves to summarise the movie and to indicate what makes it different from other movies on the cinema circuit. But can we summarise a sequence of images by another (‘macro’) image or image-topic? This article discusses semantic macro-structures and processes of multimodal discourse comprehension, formulating the mapping rules underlying the global interpretation of political cartoons. The rules for multimodal discourse processing (involving group-specific knowledge) formulated in this study apply to Arabic-speaking audiences, which should be of interest for scholars in intercultural communication and cognition.
... The multimodal media analysis informing this study integrated semiotic analysis and critical discourse analysis methods (Fairclough, 1995;Kress and Van Leeuwen, 2002;Norris, 2019). The analysis was undertaken in sequential stages to interpret how the multimodal visual and discursive components of multiple media data sources came together to construct leadership frames. ...
Article
Full-text available
Over time, the relevance of heroic leadership to contemporary corporate environments has been questioned, with media coverage arguing there is a need for alternate, post-heroic forms of leadership. Using a multimodal media analysis, we show how two leading Australian business magazines frame leadership in response to this debate, identifying three distinct frames of leadership. The first frame emphasizes masculinized heroic leadership as normative which reinforces gendered assumptions through differential framing of men and women’s leadership. We then argue media (re)frames post-heroic leadership as a variation of heroic leadership through two further frames; by subsuming feminized attributes into the repertoire of heroic leadership as ‘softer masculinities’ and through the construction of a masculinized post-heroic hero, both applied exclusively to men’s leadership. This (re)framing of heroic leadership has significant implications for perceptions of credible contemporary business leadership.
... One such omission has been the analytical protocol and development of multimodal transcripts which go hand-in-hand in the analytical procedure. For a complete and comprehensive guide to analytical protocol and transcript generation, see Norris (2018). We could not deal comprehensively with topics like the mediational interrelationship (Geenen, 2013b(Geenen, , 2014, the notion of practice (Norris, 2004(Norris, , 2011Scollon, 2001) or other developments, such as what constitutes a communicative mode and how they develop and are during ontogenesis (see Norris, 2013). ...
Chapter
Multimodal (Inter)action Analysis was developed to study social interaction based upon the theoretical notion of mediated action. Building on this core concept, Multimodal (Inter)action Analysis includes several theoretical/methodological tools. These tools facilitate analysis which moves flexibly between micro-level moments of interaction and macro-level practices and discourses. In this chapter, the application of mediated action to multimodal analysis is discussed, before the central theoretical/methodological tools are introduced. Tight links are made between the tools used in Multimodal (Inter)action Analysis and the core theoretical tenets, to support robust multimodal interaction research.
Article
Full-text available
This contribution is the introduction for the special issue of Gesture entitled “Anthropology of Gesture”. As such, it raises two main questions: how do gestures contribute to the field of anthropology? And, inversely, how anthropology can improve our understanding of gesture and gestural behaviours? Of particular importance for this special issue, is the emphasis on what Lempert called “the anthropological sensibility” which aims at taking a more cultural and ethnographic approach to the study of gesture, especially but not only in cross-cultural contexts. The last part of this introduction presents all the contributions of this special issue.
Article
Speakers in research dissemination talks are challenged with the need to connect with an audience that does not necessarily share their knowledge and expertise. This communicative situation can be particularly challenging for speakers using English both as a foreign language and for academic purposes. This study combines multimodal and ethnographic methods to explore how speakers of dissemination talks engage with their public. It focuses on four presenters’ use and combination of language, paralanguage, kinesics, proxemics and gaze during intensive moments of engagement. The results show that these interpersonal rich points consist of dense multimodal ensembles that serve to shorten the distance between presenters and their audiences. The findings suggest that a skilful orchestration of modes can be greatly beneficial to achieve the desired level of audience engagement. Therefore, developing speakers' multimodal communicative competence should be a priority in English for specific and academic purposes (ESP/ EAP) training.
Article
Full-text available
This paper offers two contributions to the multimodal discourse analysis of tourism advertising videos. First it shows how Thainess has been resemiotized to be accessible to non-Thai people and second it illustrates how the mediated action can be used as a unit of analysis to analyze advertising videos. Thainess has been a national identity strategy that has evolved into numerous polysemic. In this discussion five discursive stances to Thainess are identified: Thainess as a national identity, Thainess as a form other-ness, the Thai-ification of Thai people, Thainess as popular culture, and Thainess as commodified consumption. The analysis uses an integrative framework based in mediated discourse theory, multimodal discourse analysis and social semiotics to discuss and analyze the discursive stances of Thainess that emerge in two videos from the 2015 Discover Thainess campaign. Using the mediated action as the unit of analysis the multimodal features of Thainess that emerge from these videos are analyzed. Through this analysis Thainess is shown to be resemiotized in the videos where it was once inaccessible to westerners, the Thainess represented in the videos has been resemiotized to be both accessible and authentic to western tourists.
Article
У статті розкрито алгоритм агрегації програмних метрик і її застосування при тестуванні програмного забезпечення. Визначено генезис формування наукової думки, щодо агрегації програмних метрик. Розкрито методологію тестування програмного забезпечення з відокремленням схеми процесу тестування програмного забезпечення. Наголошено, що в якості базису для визначення рівнів агрегації програмних метрик при тестуванні програмного забезпечення спочатку слід визначити процес і складові блоки на прикладі системи тестування. Підкреслено, що агрегація програмних метрик може проводитися на рівні винесення рішень, на рівні значень відповідності та на рівні ознак і зразків. Відзначено, що агрегація на першому та другому рівнях відбувається після залучення засобу порівняння, в той час як рівні третій та четвертий проводять операції до того, як пристрій порівняння видасть результуючі дані. Описано математичні властивості методів агрегації, а саме, домен, діапазон, інваріантність та розкладання. Представлено алгоритм агрегації програмних метрик до рейтингів, використовуючи порогові значення на основі еталонних показників. Покроково описано реалізацію алгоритму та визначено параметричні значення процесу агрегації. Наголошено, що зведення окремих вимірювань до рейтингів здійснюється за допомогою дворівневого процесу, заснованого на двох типах порогів, а окремі вимірювання об’єднуються в профілі ризиків за допомогою метричних порогів. При цьому, профілі ризику агрегуються за 5-бальною зірковою шкалою за допомогою порогових значень. Агрегація дворівнева, на першому рівні агрегація здійснюється шляхом обчислення відносного розміру системи, що підпадає під кожну категорію ризику, на другому рівні об’єднання профілів ризику в рейтинг здійснюється шляхом визначення мінімального рейтингу, для якого сукупний відносний розмір усіх категорій профілю ризику не перевищує набору порогів 2-го рівня. Здійснено тестування програмного забезпечення Dia. Профіль ризику для Dia містить 73,3% коду у низькому ризику, 8,2% помірного ризику, 7.9% високого ризику та 10.7% дуже високого ризику. Використання інтерпольованої функції дає рейтингове значення 2,99, рейтинг для Dia має три зірки.
Book
Full-text available
Classic pragmatic theories emphasize the linguistic aspect of illocutionary acts and forces. However, as multimodality has gained importance and popularity, multimodal pragmatics has quickly become a frontier of pragmatic studies. This book adds to this new research trend by offering a perspective of situated discourse in the Chinese context. Using the multimodal corpus approach, this study examines how speakers use multiple devices to perform illocutionary acts and express illocutionary forces. Not only does the author use qualitative analysis to study the types, characteristics, and emergence patterns of illocutionary forces, he also performs a quantitative, corpus-based analysis of the interaction of illocutionary forces, emotions, prosody, and gestures. The results show that illocutionary forces are multimodal in nature while meaning in discourse is created through an interplay of an array of modalities. Students and scholars of pragmatics, corpus linguistics, and Chinese linguistics will benefit from this title.
Article
This paper focuses on how signing students organise themselves spatially in social interactions in a university lecture hall. One may view space as a concrete location, a social construct, and a normative actor with historical trajectories. The study addresses the question, 'What are the mediated actions through which the students and teacher (re)configure space for participating in a class?' Following a methodological framework of Mediated Discourse Analysis and multimodal interaction analysis, I approach this question by examining the social actions occurring when entering a lecture hall. The primary data includes video recordings, photos, and participatory observations, documented by field notes. The analysis shows how the architectural specifications of a space pose restrictions on visual-embodied interactions. However, the participants configure and reconfigure the space to some extent to suit visual-embodied interaction through explicit and implicit negotiation.
Article
Full-text available
This article presents theoretical concepts and methodological tools from multimodal (inter)action analysis that allow the reader to gain new insight into the study of discourse and interaction. The data for this article comes from a video ethnographic study (with emphasis on the video data) of 17 New Zealand families (inter)acting with family members via skype or facetime across the globe. In all, 84 social actors participated in the study, ranging in age from infant to 84 years old. The analysis part of the project, with data collected between December 2014 and December 2015, is ongoing. The data presented here was collected in December 2014 and has gone through various stages of analysis, ranging from general, intermediate to micro analysis. Using the various methodological tools and emphasising the notion of mediation, the article demonstrates how a New Zealand participant first pays focused attention to his engagement in the research project. He then performs a semantic/pragmatic means, indicating a shift in his focused attention. Here, it is demonstrated that a new focus builds up incrementally: As the participant begins to focus on the skype (inter)action with his sister and nieces, modal density increases and he establishes an emotive closeness. At this point, the technology that mediates the interaction is only a mundane aspect, taken for granted by the participants.
ResearchGate has not been able to resolve any references for this publication.