Book

Analyzing Multimodal Interaction: A Methodological Framework

Authors:

Abstract

Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences. Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work. Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
... Each utterance is a language unit that has meaning in a social context when there is communication interaction. All communication interactions are what are called multimodal (Norris, 2004). ...
... (7) Visual awareness and multimodal skills emphasize the importance of visual literacy and skills in using various modes of communication in contemporary society. Kress views multimodal pragmatics as an understanding of communication that involves more than just words, encouraging speakers and interlocutors to analyze how various modes of communication interact to form meaning in diverse contexts.Based onNorris (2004), there are eight characteristics of multimodality, namely (1) Multimodality, which emphasizes various ways of communication such as verbal, visual, audio, kinesthetic, and other modalities. (2) Interaction mode, where various modes interact to form a complete meaning in communication.(3) ...
... (8) Multimodal literacy skills, emphasizing the importance of the ability to understand and produce messages using various communication modes. Norris explains his contribution to the understanding of multimodal pragmatics by exploring the contribution of different modes of communication to forming meaning in everyday situations and diverse sociocultural contexts.A synthesis of the opinions ofKress & van Leeuwen (2001) andNorris (2004) produces a more complete formulation of multimodal characteristics. These characteristics include (1) Multimodality and its interactions. ...
Conference Paper
Full-text available
The language context in the study of pragmatics in the digital era has changed from initially using linguistic aspects to now using audio and visuals. This research aims to collect and analyze several sources related to multimodality as a new perspective in pragmatic analysis in the digital era. The research design used is a literature review by collecting various sources from books and relevant scientific articles. Based on the books and articles collected and reviewed, it can be concluded that the research results show (1) The multimodal concept is a theory that observes the use of various modes by individuals to communicate and express themselves, which involves writing, gestures, body posture, eye gaze, font choice and colours, images, videos, and interactions between individuals. (2) Multimodal characteristics include multimodality and its interactions, the role of technology, visual modality, social and cultural context, analysis of social interactions, multimodal literacy skills, semiotic analysis, and case studies. (3) The multimodal aspect includes the giving and creation of meaning, the multiple modes involved, and multimodal practices. (4) Multimodal pragmatics is a new perspective in pragmatic analysis that examines linguistic or verbal modes and other modes, such as audio and visual, which are used simultaneously to communicate with each other. Thus, pragmatic studies using a multimodal perspective are important because understanding a speech analyzes linguistic aspects and pays attention to multimodal visual and audiovisual aspects to strengthen the meaning or purpose of the speech act.
... Multimodal interaction analysis (Norris, 2004) accentuates the interpretation and representation of social interactions as they evolve over a range of coalescing communicative channels. These communicative modes cover anything from spoken language, proximity, posture, gesture, head movements, gaze, print, and even layout (or the setting and objects therein; Norris, 2004). ...
... Multimodal interaction analysis (Norris, 2004) accentuates the interpretation and representation of social interactions as they evolve over a range of coalescing communicative channels. These communicative modes cover anything from spoken language, proximity, posture, gesture, head movements, gaze, print, and even layout (or the setting and objects therein; Norris, 2004). The researcher homes in on a brief (10−40 second) interaction of significance recorded by video camera. ...
... Multimodal transcripts might powerfully add to research into L+ learning emotions by affording insights into how (emotional) "meaning is made in all modes separately, and at the same time" and the ways in which (emotional) "meaning is an effect of all modes acting jointly" (Kress & van Leeuwen, 2001, p. 1). However, as Norris (2004) cautioned, the task of a multimodal transcript is not to analyze the images that are incorporated, but rather to use the images to describe the dynamic unfolding of specific moments in time, in which the layout and modes like posture, gesture, and gaze play as much a part as the verbal. (p. ...
Article
Empirical work exploring additional language (L+) learning emotions has both proliferated and expanded its focus over the past 15 years. The current article explores one possibility for responding to the challenge of capturing and describing emotions in order to furnish a more contextual-ized, multidimensional picture of emotions in L+ learning: the small-lens approach. From the perspective of the author as a practitioner-researcher, this article draws on data from an L+ discussion activity. Via the activity, the practitioner-researcher identified an emotional outcome of interest and examined the historical buildup to this phenomenon. A narrative of the research process applied aims to illustrate the ways in which multimodal analysis and interrogation of psychological timescales might illuminate intersections between the social and individual. Rather than delineating a complete description of the research, the article strives to intimate possibilities and stimulate more nuanced, situated , and dynamic empirical work into the emergence of emotions in instructed L+ learning. While not the primary focus, some of the teleological (functional) aspects of emotions and emotional expression are also unearthed.
... The methodological approach chosen for the analysis in this research is Multimodal (Inter)action Analysis (Norris 2004(Norris , 2019(Norris , 2020. This methodological framework gathers concepts from different theoretical approaches such as mediated discourse (Scollon 2001), studies of social interaction and social semiotics (Kress andvan Leeuwen 2001, 2020). ...
... Teachers' use of gestures, their different movements through the classroom space and the tools and resources employed, complement each other with languages and create a multimodal set of meanings (Lim 2021). Given Multimodal (Inter)action Analysis' potential to explore interaction (Norris 2004), it will provide the methodological framework for data analysis. ...
... language, gesture, gaze or modes that are performed with the human body), disembodied modes (e.g. music, print, layout or modes that do not belong to a human body and are understood through a frozen action rather than real-time action, an action that has been performed previously) and the site of engagement (the social encounters in the diverse spaces of the classrooms where interaction takes place) (Norris 2004(Norris , 2011. ...
... algunos de estos estudios se han centrado más en el discurso escrito. Sin embargo, en el campo del análisis del discurso oral es probablemente donde más se ha trabajado la multimodalidad (Kress y van Leeuwen, 1996;Norris, 2004;O'Halloran, 2004;Jewitt et al., 2016), sobre todo en contextos académicos, centrándose en presentaciones en congresos (Querol-Julián y Fortanet-Gómez, 2012) y clases universitarias (Bernad-Mechó, 2017; Crawford ISSN 2696-676X Fortanet-Gómez, Inmaculada; Bernad-Mechó, Edgard; Valerias-Jurado, Julia (2024). GRAPE-MARS: una nueva herramienta para el análisis multimodal en la investigación sobre segundas lenguas. ...
... Un análisis multimodal proporciona una visión más completa de cómo tiene lugar una comunicación y qué elementos juegan un papel en ella. Hay tres enfoques básicos para la investigación multimodal del discurso oral (Bernad-Mechó, 2021;Jewitt, 2014): la Semiótica Social Multimodal (SSM) (Kress y van Leeuwen, 1996, 2001, el Análisis del Discurso Multimodal (ADM) (O'Halloran, 2004(O'Halloran, , 2011, y el Análisis de (Inter)acción Multimodal (AIM) (Norris, 2004). Cada uno de estos enfoques ofrece herramientas analíticas específicas y difiere en sus teorías subyacentes, el énfasis dado al contexto y la interrelación entre modos, así como en la atención prestada al creador de signos (Jewitt, 2014). ...
... En esencia, Scollon (2001) propone un enfoque para analizar el discurso donde el analista se desvincula del texto -lo que se dice -y se enfoca en las acciones realizadas por los actores sociales -los hablantes. Este enfoque es aplicado por Norris (2004) al estudio multimodal de la comunicación. Así, el AIM se interesa por cómo los actores sociales interactúan en contextos específicos y situados; según Norris (2014a), el trabajo del actor lo es todo. ...
Article
Full-text available
El objetivo de este artículo es presentar el software GRAPE-MARS (Multimodal Analysis Research Software) (https://www.grape.uji.es/mars), un programa de código abierto y uso gratuito desarrollado por el grupo de investigación GRAPE (Group for Research on Academic and Professional English) para el análisis multimodal de vídeos, en el contexto particular de la investigación en segundas lenguas. La investigación multimodal ha avanzado mucho en los últimos años, especialmente en el análisis del discurso, aunque también se han realizado estudios relacionados con la traducción o la pragmática, entre otros. GRAPE-MARS es un programa de interfaz sencilla y usabilidad asequible que pueden utilizar tanto investigadores e investigadoras, como profesores o estudiantado y cuyo objetivo es facilitar la tarea de identificar los elementos o modos que conforman el discurso y facilitar su representación. En una época en que los géneros digitales, con una multimodalidad inherente, cada vez cobran más protagonismo, creemos que GRAPE-MARS puede ser una herramienta esencial para desgranar los elementos que los componen y contribuir a análisis multimodales más completos que ayuden en la investigación de las segundas lenguas.
... Despite multimodal approaches being on the rise, persistent analytical constraints have been noted, including systematising methods in multimodal research and creating 'stable analytical inventories' of multimodal semiotic resources (Jewitt, 2013); bringing various semiotic modes together under a cohesive analytical framework (Kress & Van Leeuwen, 2001); and integrating different tools and techniques for analysing multimodal interactions (Norris, 2004). Against this background, I consider the methodological affordances of each approach here, with a particular focus on their application in the health communication domain. ...
... A key theoretical underpinning of MDA is that ''the focus of mediated discourse analysis is not discourse per se, but the whole intersection of social practices of which discourse is a part'' (Jones & Norris, 2005, p. 4): this distinguishes MDA from turn-taking approaches, such as CA (see following section). Compared to other multimodal discourse approaches outlined here, MDA is the least centred on language per se, without, at the same time, denying that language often plays the central role in interaction (Norris, 2004). Core concepts underpinning MDA are sites of engagement and historical bodies: sites of engagement place emphasis on context and are defined by Scollon (2001, p. 4) as a ''window that is opened up through the intersection of social practices and meditational means (cultural tools) that make that action the focal point of attention of the relevant participants''. ...
Article
Full-text available
Background: Multimodal discourse approaches have only recently gained consistent prominence in health communication research. The theoretical and methodological underpinnings of each approach, and their contribution to the health communication domain, require further articulation. Aim: This article aims to sketch out the field, showcasing the methodological strengths and limitations of multimodal discourse approaches, and their potential contribution to health research. Methods: The article reviews four established and emerging multimodal discourse approaches used in health communication research. A comparative lens is taken, scrutinising each approach in terms of its theoretical underpinnings, methodological implications, and analytical constraints. Findings: Key points of convergence and divergence among the approaches are identified, with all approaches sharing a commitment to investigating multiple modes and their relationships in creating meaning within health research. The main point of differentiation lies in what each approach considers the unit of analysis: Systemic Functional Multimodal Discourse Analysis focuses on semiotic resources, Mediated Discourse Analysis on action, Conversation Analysis on conversational order, and Multimodal Critical Discourse Analysis on power and social structures. Conclusions: Future directions include a focus on materiality, the integration of emerging technologies, and the development of new analytical tools for investigating crisis communication. All these can offer deeper insights into health communication and enhance professional practices and patient outcomes.
... Building on this emerging body of work, our study engages with the growing call to address the role of materiality in meaning-making. While prior research in applied linguistics and social semiotics has examined how multimodal resources interact in communication (Jewitt et al. 2016;Kress and van Leeuwen 2001;Mondada 2019;Norris 2004), there has been a tendency to treat these resources as either supplementary to language or as separate semiotic systems. Mondada (2016, p. 338) critiques this approach, arguing for an understanding of modalities as "constitutively intertwined", with language being one among many resources in a network of meaning-making. ...
... An essential aspect of this multimodal perspective is the view that objects themselves have agency, as they embody past social interactions and cultural significance. Norris (2004) refers to objects as "frozen actions" emphasising that their uses and interpretations are constructed by the social practices they have been part of. Objects, therefore, play an active role in influencing immediate interactions through their cultural and historical connotations. ...
Article
Within current social science discourse, understanding the complex relationship between materiality, agency and discourse is crucially important for grasping the dynamics of social practices and action. This study employs a linguistic ethnography approach to examine the nuanced interplay between discourse, objects and participants within social interactions. Drawing from theoretical frameworks including mediated discourse analysis (MDA), actor-network theory and historical materialism, we explore how materiality and discourse form and coverage one another. Our analysis reveals that objects and discourse are not fixed and static entities but rather dynamic processes that mutually influence and constitute each other within social situations. We argue that while objects possess agency, they are mediated by human participants and broader social forces which emphasises the need for context-specific analysis and examination. Through a dialectical lens, we unpack the intricate ways in which material conditions inform and reflect social relations, demonstrating the role of objects as tangible manifestations of social processes. Our findings enhance a deeper understanding of the complex interplay between agency, materiality and discourse with implications for theories of practice and social interaction. By recognising the role of objects in shaping cultural practices and social dynamics, we enhance our understanding of the processes everyday social life.
... This contextual understanding is crucial for interpreting the significance and implications of findings (Silverman, 2013). For instance, the Multimodal Communication Theory contextualizes emojis as part of a broader set of communicative modes, helping to explain how they complement and enhance textual communication (Norris, 2004). Furthermore, the use of multiple theoretical frameworks allows for interdisciplinary insights, enriching the analysis by integrating perspectives from linguistics, psychology, sociology, and communication studies (Denzin & Lincoln, 2011). ...
... Multimodal Communication Theory: This theoretical framework examines communication as a multimodal phenomenon involving the integration of multiple semiotic modes, such as language, visuals, gestures, and paralanguage (Kress & van Leeuwen, 2006). Emojis are conceptualized as one of the modalities through which meaning is conveyed in digital communication, complementing and enhancing textual messages with non-verbal cues and emotional expressions (Norris, 2004). (Tajfel & Turner, 1986). ...
Article
Full-text available
This research investigated the evolving role of emojis in shaping paralanguage within social media communication, with a particular focus on how these visual symbols enhance linguistic and emotional expression. The study examined the nuanced ways in which emojis contribute to the overall tone, context, and meaning in written discourse, while also considering cultural and contextual variations in emoji usage among individuals from diverse linguistic backgrounds. Conducted with a sample of 55 Year-3 Bachelor of Education students at a university in Fiji, where English is a second language, the research employed a mixed methods approach to analyze data on the usage and perception of emojis in social media communication. This paper argues that emojis play a crucial role in conveying emotions, fostering engagement, and adding personality to messages, thereby augmenting textual communication with emotional nuance and clarity, leading to more empathetic and meaningful online interactions. The findings indicate that emojis are widely used across various communication channels, including text messages, comments, and direct messages, reflecting their versatility and widespread adoption. Respondents overwhelmingly perceive emojis as positive tools that enhance the effectiveness of digital communication by conveying emotions, fostering engagement, and adding personality to messages. The study also highlights the significant role emojis play in nonverbal communication, particularly in expressing emotions that may be difficult to articulate through text alone. By providing emotional nuance, clarity, and depth, emojis contribute to more empathetic and meaningful interactions in online spaces. Overall, this research provides valuable insights into the dynamic interplay between emojis and paralanguage in contemporary digital communication, signifying the importance of these symbols in enriching text-based exchanges and fostering more engaging and emotionally resonant social media interactions.
... In this context, modes are understood as semiotic systems with rules and regularities -such as images, gestures, speech, music, layout, writing, proxemics, and posture (Kress and van Leeuwen 2001). Norris (2004) categorizes the use of modes as embodied and disembodied. The former consists of resources produced by the human body (e.g. ...
... Therefore, the selection of the segments was made on the basis of their persuasive communicative aim and their multimodal nature. I hypothesized that these fragments, interpreted as using pressure tactics, would be modally dense (Norris 2004), since they would use a greater variety of semiotic modes and more complex interrelations among them. ...
Article
Full-text available
With the rapid shift from print to digital modes of communication, new genres have emerged. One example is the video resume (VR). This novel professional digital genre enables job seekers to enhance their visibility and connect with broader audiences through various semiotic modes. The VR has a clear communicative aim based on a strong persuasive component: to secure a job by convincing the audience of their qualities. The premise of this paper is that VRs are not only informative but also inherently persuasive. Accordingly, the study aim is to explore the use of multimodal semiotic modes as a way of realizing persuasive strategies (i.e. attention-getting, anticipation and control of responses, rapport, emphasis and processing aids). The methodological approach uses video-based analysis and computer-aided multimodal discourse analysis (MDA) to analyze a dataset of 8 VRs from the online platform YouTube, considering the following criteria: limited duration, impact, year uploaded and editing. The results suggest that verbal and non-verbal realizations of persuasive strategies are present in the fragments analyzed, and such strategies are encoded by heterogeneous combinations of semiotic modes. Although the strategies and modes are not always consistent, they contribute to achieving the communicative purpose of the genre. As for pedagogical implications, this genre and its features can be incorporated into English for Specific Purposes (ESP) teaching materials, enhancing learners’ persuasive strategies and developing their multimodal communicative competence.
... The works of Forceville (1994Forceville ( , 2011Forceville ( , 2024, Van Leeuwen (1996, 2001), Kress (2010Kress ( , 2015, Van Leeuwen (1995, Norris (2004Norris ( , 2006Norris ( , 2011, Machin and Mayr (2012), Bateman et al. (2017), Ledin and Machin (2018), O'Halloran et al. (2018), Fernandez-Fontecha et al. (2020), Jewitt et al. (2021), and Kress and Bezemer (2023), among others, have significantly enriched the discourse on multimodality. These scholars have collectively pushed the boundaries of discourse analysis, emphasizing the importance of considering the full spectrum of communicative modes and their integration within social contexts. ...
... The study's novelty lies in its exploration of diverse interaction modes in online courses, a concept that was gaining traction and influencing teaching and learning. Hampel and Stickler (2012) emphasized that video meetings enrich media and modal density, a concept previously defined by Norris (2004). The rapid evolution of videoconferencing and its integration into daily communication has spurred a notable increase in MDS research since 2012, focusing on meaning construction through multimodal resources. ...
Article
Full-text available
In today’s world of multimedia communication, the use of multiple modes of discourse is prevalent in various fields. The international academic community has taken an interest in studying multimodality from different perspectives. This paper uses CiteSpace 6.1.R6 to visually analyze literature on multimodal discourse studies (MDS) in the Web of Science (WoS) Core Collection from 1997 to 2023. The results show a significant increase in English research papers on MDS over the past two decades, a development made possible through collaborative efforts and knowledge-sharing among scholars from different regions and institutions. MDS conducted by scholars from across the world reveals diverse research categories and highlights hot issues and social practices. Consequently, the discipline has produced significant social value. The analyses of keywords and literature further reveal that MDS mainly explores language education, politics, children’s education, identity construction, and media. Additionally, the most cited literature on MDS plays a fundamental role in guiding scholars across the world to understand the key concepts of multimodality and to conduct multimodal discourse analysis from different perspectives. This research provides valuable insights for scholars engaged in or interested in multimodal discourse analysis worldwide, helping them understand MDS’s evolution while offering significant implications for conducting research on multimodal discourse.
... Vemos, en nuestros primeros trabajos, la limitación al lenguaje verbal como un inconveniente mayor en términos de nuestro propósito principal. El significado reside tan fuerte y permanentemente en otros sistemas de significado, en una multiplicidad de códigos visuales, auditivos y de comportamiento, que una concentración solo en las palabras no es suficiente Kress, 1988, p. 8) 1 A partir de allí, se han profundizado los estudios multimodales desde una perspectiva crítica (Hart, 2010(Hart, , 2014Norris, 2004Norris, , 2006Norris, , 2019O'halloran, 2004;Van Leeuwen, 2008). ...
... 5 Dato obtenido de https://www.un.org/es/global-issues/migration (fecha de consulta: 30 de mayo de 2024). discursos sobre migrantes han sido un tema fundamental para el ACD desde sus inicios (Martín Rojo, 2000, 2004Reisigl;Wodak, 2001;Van Dijk, 1984, 1997, 2003Wodak;Matouschek, 1993). En particular, en América Latina podemos citar los trabajos de Bolívar; Fontaines-Ruiz (2021), Aedo; Farías (2009), Taboada (2016), Ramírez Lasso (2018), Vega (2022), entre otros. ...
Article
Full-text available
Resumen: En este trabajo, nos proponemos profundizar en la importancia de la utilización de herramientas de la lingüística cognitiva tanto para describir como para explicar el funcionamiento de los discursos desde una perspectiva crítica. En particular, vamos a analizar cómo los conceptos de marcos, metáforas, metonimias y esquemas de imagen pueden ser utilizados de manera productiva para el análisis del modo verbal y de los gestos manuales. Para ello, analizaremos una entrevista que brindó el expresidente chileno Sebastián Piñera en el año 2016 en ocasión de la campaña electoral que permitiría su posterior reelección. Concluimos que ciertos gestos manuales realizan significado ideacional que refuerza las estrategias discursivas presentes en el modo verbal. Abstract: In this work, we intend to delve into the importance of using cognitive linguistics tools to describe and explain the functioning of discourses from a critical perspective. In particular, we analyse the way in which the concepts of frames, metaphors, metonyms and image schemes can be used productively for the analysis of the verbal mode and manual gestures. In order to do this, we analyse an interview that former Chilean President Sebastián Piñera gave in 2016 during the campaign prior to his re-election. We conclude that certain manual gestures carry ideational meaning and reinforces discursive strategies present in the verbal mode.
... Outra fonte de variação dos conteúdos diz respeito aos suportes que os veiculam, ou seja, os meios empregados para expressar os modos semióticos ou de comunicação (Mortimer, Moro, & Sá, 2018), classificados como incorporados e não incorporados. Fala, gesto e olhar são exemplos de modos de comunicação incorporados, enquanto escrita, imagem e leiaute são classificados como modos não incorporados (Norris, 2004). O papel central da mediação das ferramentas culturais no desenvolvimento humano está em reconhecê-las como constitutivas da linguagem, as quais ampliam as formas de agir no mundo e por conseguinte expandem as possibilidades de engendrar os problemas sociocientíficos, potencializando-os como objetos de atividades criativas porque diversificam os suportes e as formas de comunicação da sala de aula. ...
... Observou-se também por meio da natureza das atividades desenvolvidas na aula uma grande diversidade de suportes (vidrarias, materiais de laboratório, lousa, tela de projeção e computador) e modos de comunicação (fala, escrita, imagem, gesto) empregados, o que indica a mobilização de linguagens e interações multimodais. Ainda que os estudos da linguagem na perspectiva da multimodalidade tenham ganhado destaque nas últimas décadas (Norris, 2004;Kress, 2010;Mortimer e Quadros, 2018), não desenvolveremos um quadro analítico desta ordem, pois nosso enfoque é com a produção verbal na compreensão da dinâmica discursiva. ...
... Even though Kress & Van Leeuwen (2001) do include a range of non-textual modes of discourse, including image, music, sound, gesture or colour, all these modes are not covered in the same depth within the literature, with visuals being particularly favoured at the expense of others, like gesture, posture or movement. The PPAP places itself in the lineage of the work of Norris (2004Norris ( , 2011 who emphasised the embodied and interactional nature of discourse, and other scholars who adopted ...
Article
Full-text available
Despite its versatility as a methodological tool, discourse analysis suffers from a logocentric bias, a tendency to primarily focus on text while ignoring the non-textual components of discourse. Although poststructural approaches to discourse analysis like Discourse Theory (DT) have developed an understanding of discourse going beyond language, there are few practical methodological tools for scholars from these traditions to engage with these non-textual elements. This article fills that gap by providing an original qualitative methodological tool, the Political Performance Analysis Protocol (PPAP) which adapts one of the signature methods of Performance Studies to political performances. It does so by describing the four constitutive elements of political performances — background symbols and foreground scripts, actor, audience and mise-en-scène — and empirically illustrating them through the case of Greta Thunberg’s “How dare you?” performance at a United Nations to show the PPAP’s relevance as a tool complementing other forms of discourse analysis.
... 22 Multimodality challenges the prior dominance of speech (Scollon and Scollon, 2009), which is conceived of as just one of the many components of the so-called "multimodal ensemble" (Jewitt, 2013). Without denying that spoken language often plays a key role in interaction (Norris, 2004), all resources other than speech also have the potential to contribute equally to meaning. ...
Article
Full-text available
This paper presents GFFIntD, a multimodal corpus of interpreting practices performed at the Giffoni Film Festival, with a focus on data collection, transcription and analysis. In particular, the paper draws upon the concept of visibility on screen and discusses the two-step analytical process applied to certain excerpts in which multimodal elements co-construct meaning during guest-initiated vignettes.
... From the various seminal studies that have allowed the methods of discourse analysis to be deepened and applied to various semiotic studies (Kress and van Leeuwen 1990, 1996Norris 2004;O'Halloran 2011;Kress 2012), some linguists have reached a conclusion, which was noted by Scollon and LeVine (2004), and later ratified by Kaltenbacher: Linguists have realized that there is no pure monomodal discourse; that spoken language does not exist without accompanying gestures, grimaces, body posture and eye movement, in the same way that written texts cannot exist without frames, spaces, typography or colour. Therefore, what we [linguists] saw was not, in fact, a change of attention from mono-to multimodality, but the recognition of the completely modal nature of every text (2007, 37; our translation). ...
Article
Communication is multimodal in its very nature. In recent decades, research interest in this topic has grown exponentially, especially from a Multimodal Discourse Analysis (MDA) standpoint. Within academic settings, several studies have looked at lecturers’ combinations of verbal and non-verbal features, but not necessarily in relation to the emotions that are implicitly part of lecturing. We believe that teaching involves transmitting knowledge together with emotions to students, more or less consciously. This may even be more relevant in an English Medium Instruction (EMI) setting, as English is not the instructor’s main language. Thus, our main aim is to analyse an example of EMI teaching practice from an MDA perspective but adding an electronic device—an electroencephalograph—that can help us improve and/or complement the analysis in different ways: adding more objective support and dealing with the emotions lecturers can transmit when delivering their classes. Results show that this combination of observation and technology can potentially enrich results from traditional MDA research.
... Multimodal Discourse Analysis -MDA) (O'Halloran 2004) и мултимодална анализа (интер)акције (енгл. Multimodal (Inter)action Analysis -MIA) (Norris 2004), а оне се разликују према фокусу истраживања и методологији. У овом раду теоријско-методолошки ослонац биће мултимодална анализа дискурса. ...
... Research in media studies has also indicated that complexity emerges when interpreting multimodal content is complex in the sense that it is layered by the interfaced modes of signs, which include linguistic, visual, and also an auditory mode, to give a composite meaning (Kress & van Leeuwen, 2001;Norris, 2004). This complexity becomes further compounded by the rapid flow of information across the digital networks where the messages are quickly disseminated, remixed, and recontextualized by the audiences leading to divergent meanings (Boyd,2014 2024 Abhirup Bhadra et al ...
Article
Full-text available
This paper looks into how quantum theories apply can be applied to mass communication, focusing on how a multimodal media message operates and might be susceptible to similarity with quantum systems. For this purpose, a systematic review of literature is intended to integrate concepts of quantum physics-specifically Superposition, Entanglement, Uncertainty principle, and Decoherence-in offering new ways of conceiving media interpretation across different platforms. These would be used as a conceptual framework with which to systematically explain the layers of media messages.
... . DOI: 10.22456/2238-8915.135007 3 aprofundaram o conhecimento acerca de outras, tão importantes quanto a fala, no processo de interação do sujeito (KENDON, 1988;MCNEILL, 1992;NORRIS, 2004). Tais autores argumentam que a linguagem é sempre multimodal. ...
Article
Full-text available
Estudos sobre a linguagem infantil comumente abordam a captura da criança pela língua, a partir de uma instância principal de interação, a fala. Entretanto, em aquisição da linguagem, o conceito de multimodalidade da linguagem propõe outros elementos linguísticos além da fala, como gestos e olhares. Nesse sentido, os cuidadores interpretam os gestos por meio dos processos de homologia e interpretância. O objetivo do presente estudo é analisar o processo de aquisição de linguagem de crianças na interação com seus cuidadores. Este é um estudo retrospectivo e qualitativo, que analisou duas filmagens da interação cuidador-bebê. Na amostra analisada, os participantes são duas díades, cuidador-bebê, de 8 meses a 10 meses de idade. Foram realizadas entrevistas semiestruturadas e, depois, foi solicitado que os cuidadores interagissem com as crianças. Os encontros foram gravados em vídeo e posteriormente analisados no software ELAN 6.0 (EUDICO Linguistic Annotator). Os resultados apresentados mostram que as crianças utilizam mais de um recurso multimodal na interação. Conclui-se, nos casos analisados, que os cuidadores são fundamentais para o processo de aquisição da linguagem, pois os mesmos podem significar o bebê como sujeito falante, à medida que atribuem significados na interação, a partir dos aspectos multimodais da linguagem.
... Gestural modes: body language, facial expressions, and movement within video content (Norris, 2004). ...
Book
Full-text available
The 4th international scientific-practical conference entitled "HIGHER EDUCATION IN THE REGIONS: REALITIES AND PERSPECTIVES" was organized in a hybrid form by Guba branch of the Azerbaijan State Pedagogical University (ASPU) on October 11-12, 2024, following the education policy of the Republic of Azerbaijan in accordance with the preparation of scientific bases of measures taken by the Ministry of Science and Education, to strengthen the innovation, to attract the potential of natural and geographical conditions to production and practice. The conference was organized by order No. 237 dated July 19, 2024, of the director of Guba Branch of ASPU, Associate Professor Yusif Aliyev, under the relevant decision approved by the Ministry of Science and Education of the Republic of Azerbaijan, the command No. 3/124 of July 10, 2024, of ASPU rector Professor Jafar Jafarov in terms of the development of higher education in the regions, the increase of personnel potential, and the establishment of mutual scientific and international relations.
... Analysoitavaa tekstiä tarkasteltaessa kiinnitimme huomiota kielen mikrotason piirteisiin, kuten sanavalintoihin, sijamuotojen semantiikkaan ja metaforiin sekä lauseenjäsenten tai laajempien kokonaisuuksien rinnastukseen, jotka olivat olennaisia tutkimuskysymysten kannalta (Pietikäinen & Mäntynen 2009: 167). Kirjoitetun kielen lisäksi otimme huomioon myös sivujen visuaalisia elementtejä, sillä diskurssit ilmenevät kaikessa semioottisessa toiminnassa: asioita merkityksellistetään eri tavoilla, esimerkiksi teksteinä, kuvina (Gee 1999;Scollon & Scollon 2003) sekä eleillä ja ilmeillä (Norris 2004). ...
Article
Full-text available
Public discourse in the media about the future demands and requirements of working life on individuals often highlights the ever-increasing importance of interactional skills. Keeping in mind such common and generally-accepted observations, the aim of this study is to explore what might be the suggested points of departure for developing said interactional skills and what kind of discourses might be linked to their development. The materials used in this study consist of service descriptions drawn from the websites of companies that provide training in interactional skills. The sites were identified and selected by using the Google search engine and analysed by using selected discourse-analytic tools. The findings suggest that the service descriptions reflect 1) individual and acquisition-based, and 2) social and participation-based ways of learning and developing interactional skills. In the prior, the significance of standardized tools (e.g. DiSC™) are highlighted. Also, in service of productization, the developing of interactional skills may be linguistically compiled into a ‘sellable package’ that the clients can then acquire by participating in the training by the service provider. In the latter, the training by the service providers is framed as enabling learners’ participation and increasing self-awareness. Additionally, the descriptions of services often draw on scientific, spiritual and self-development discourses to serve the purpose of legitimization. This article problematizes especially the individual and acquisition-based views portrayed in the materials as potentially restrictive for developing interactional skills.
... Produtos audiovisuais digitais são espaços complexos de discurso e são constituídos por diversos elementos multimodais que permitem diferentes sentidos construídos e escolhas de ações múltiplas. Norris (2004) (Iedema, 2001). Inserido na perspectiva semiótica social, o autor leva em consideração, também, o contexto político e a crítica social do produto, não se reduzindo a uma simples produção audiovisual isenta, mas provocando e promovendo a construção de significados a partir dos elementos sociossemióticos presentes no filme, defendendo que as representações de tais discursos "não se concentram nos signos, mas no significado social e nos processos" (Iedema, 2001, p. 187 É importante levar em consideração que essas relações de poder também estão presentes nessa pluralidade de representações masculinas, pois os gêneros se estabelecem a partir das relações sociais que envolvem a sociedade em suas diversas esferas, seja em locais de trabalho, de política ou no ambiente doméstico (Connell, 1995, p. 188 Dessa forma, a partir das variações do tom de voz e do timbre característico individualmente, as representações de masculinidades contra-hegemônicas são construídas e difundidas, firmando-se não somente como forma de expressão e objeto de análise, como também elemento fundador que constitui a construção subjetiva de cada sujeito parte da comunidade em questão. ...
Article
O objetivo deste trabalho é apresentar uma sugestão de percurso metodológico que considere a multimodalidade como principal aporte teórico aliado às masculinidades, que, além de ser o tema de análise, também figura como critério analítico. Dessa forma, pretende-se apresentar uma proposta de metodologia que pode ser aplicado em diversos objetos de estudo similares ao que aqui se sugere. Essa metodologia foi aplicada em uma tese que analisou vídeos de uma série ligada ao projeto Creators for Change, do Youtube. O trabalho metodológico se baseia na relação que se estabelece entre os discursos multimodais aliados às representações das masculinidades negras, culminando na produção de sentido ativista.
... Jewitt compiled the latest research and methods in the field, covering a wide range of applications and theoretical issues [7]. Norris provided a methodological framework for analyzing multimodal interaction, focusing on how different modes collaborate in interaction [8]. Similarly, Forceville explored visual metaphors in advertising, analyzing how images convey complex metaphorical meanings [9]. ...
Article
Full-text available
Contemporary communication studies show that short videos, as the most important form of information dissemination today, have a more multidimensional and immersive communication effect compared to traditional print media and other channels. Since this information medium inherently contains numerous social symbols with expressive characteristics, this paper will analyze the short videos that led to the viral phenomenon of "Tianshui malatang" based on Kress and Leeuwen's visual grammar and Halliday's theories of language as a social semiotic. This paper analyzes the phenomenon of the "Tianshui malatang" a fusion of regional tourism and local specially food that has gone virtual on Chinese short video platforms, using a multimodal discourse analysis approach. The analysis shows that the "Tianshui malatang" video enhances the sensory interaction between the viewer and the screen in terms of visual effects, and in terms of narrative expression, it leads the viewer to generate positive feedback on the video through the presentation of various elements.
... Kress & Van Leeuwen made an expansion of the meaning of multimodality, defining multimodality as the "use of several semiotic modes in the design of a semiotic product or event" (Kress & Van Leeuwen, 2001, p. 20-22). Multimodal analyst Norris (2004) furthered enriching the theoretical body of multimodal discourse analysis by putting forward an integrated methodological framework consisting of a combination of verbal and non-verbal when in communication. John, Bateman & Janina (2014) pointed out a pragmatic model of multimodal discourse resting on linguistics and provided a comprehensive methodology for studying multimodal artifacts. ...
Article
Full-text available
In the context of globalization, audio-visual works produced by ever-evolving information technology provide a quick and convenient channel for bridging cultural divides and communicating with people from all over the world. People harness these productions not only to satisfy their own entertainment needs but also to gain information about other cultures. As manifold cultures and races become increasingly interconnected, the need for intercultural communication is particularly urgent, and with abundant European and American films and cultural productions taking over the world, the significance of subtitle translation cannot be understated. On the surface, subtitle translation transforms one language into another and puts it on screen at the same time. However, at a deeper level, subtitling involves a wide range of cultural and social contexts in both the source and target languages. Translators need to not only use their translation skills to select words and sentences, but also have a deep understanding of the social and cultural aspects of both languages. This is the only way to smoothly accomplish the language transfer and help the audience accurately make sense of the meaning of the words in the film. At the same time, China has entered a new era and is facing unprecedented changes in a century. Only by spreading its own culture, promoting the achievements of excellent Chinese civilization, and gaining widespread international recognition will it be able to stand firm and progress against the tide of globalization. However, even though Chinese film and television products are constantly being produced and innovated every year, there are certain shortcomings in the dissemination of film and television works and the study of subtitle translation. This thesis takes Professor Zhang Delu’s theoretical framework of multimodal discourse analysis as its perspective and selects the subtitle corpus of the film My People, My Country, which was released around National Day in 2019 to celebrate the 70th anniversary of the founding of the People’s Republic of China, as the object of study. Based on the integrated framework of multimodal discourse analysis, the author carefully selects typical examples from My People, My Country to analyze the effects of multimodal discourse analysis on subtitle translation at the cultural, contextual, content, and expression levels, and investigates how different modalities work together to vividly convey the meaning of the film’s discourse. In conclusion, multimodal discourse analysis theory has injected new momentum into the study of subtitle translation applications, breaking away from the traditional monomodal and symbolic studies and contributing to the international dissemination of more domestic audio-visual cultural products.
... The female spectators are engaged with the female models through the use of the frontal angle in these advertisements as their head, body, and eyes are all directed at the target audience (females) to engage them to utilize these products. According to (Norris, 2004), the term "modality" in visual evaluation refers to the extent to which a particular image indicates devices such as -color, depth, represented detail, tone, etc. -is used. The highest modality is typically used in all advertisements except one to create a naturalistic feel so that target spectators can identify with them. ...
Article
Beauty advertisement is one of the most potent and successful tools for significantly influencing consumer purchasing behavior as it modifies individuals’ views of beauty, social conventions, and gender roles. From a Multimodal Critical Discourse Analysis standpoint, this study looks at the beauty advertisements in a few chosen social media (Instagram) posts. The study’s goal is to concentrate on how the idea of beauty is used in these Pakistani advertisements to control and influence consumers through both language as well as images. For this particular reason, four ads from different Instagram pages have been chosen via purposive sampling and they have undergone both linguistic and visual analysis. Fairclough’s Three-Dimensional Model (1995) serves as the foundation for the linguistic examination while Kress and van Leeuwen’s Grammar of Visual Design (2006) is applied for the visual analysis of these ads. The nature of this study is qualitative. The findings reveal that advertisers have employed a variety of discursive techniques like surreal representation, scientific evidence, celebrity endorsement, & self-representation etc., and linguistic devices, such as pronouns, catchy phrases, adjectives, repetitions, etc. in addition to visual tactics, e.g., modality, gaze, social distance, salience, and camera angles etc. to influence women by portraying an idealized version of beauty in the chosen advertisements. Additionally, it shows how advertisers—who are essentially powerful people with vested interests—marginalize and restrict the status of women in society to propagate the ideology of beauty just for boosting the sales of their products. Thus, the companies and creators of these commercials employ images and language as a means of controlling Pakistan’s female consumer base by luring them to get on their items. It is anticipated that this study will increase awareness of the usefulness of multimodal critical discourse analysis and open the door for future researchers to investigate the language and pictures used in advertisements for a variety of other products.
... The proliferation of digital technologies has spurred interest in MDA, particularly in situations where language usage may be peripheral or absent (Jones, 2013). Multimodal Interaction Analysis (MIA), developed by Norris (2004a), analyzes socially situated interactions, encompassing various resources like gesture, gaze, posture, and objects, and emphasizes both embodied and disembodied communicative modes, indicating modal density. This chapter employs MIA for analyzing multimodal actions and interactions among participants, focusing on the interplay between embodied and disembodied modes in specific interactions. ...
... The original text presents classic sequential text-based transcripts and different multimodal transcripts. The interest in transcription formats and procedures has occupied researchers ever since, including experiments with comic transcripts (Norris 2004), multimodal transcripts (Mondada 2007) and chronotopic transcripts (McIlvenny 2014). The first analysis of the interaction is actually performed by the participants themselves in the moment of activity (i.e. in the video material); participants show their understanding of the situation through their multimodal interactions, which can guide the researcher's analysis of the activity. ...
... Research dealing with multimodes in communication and the presence of these multimodes in the classroom has been around since the 1950s (Smith, 1979;Portnoy, 1982). Considering the studies of Halliday's Systemic Functional Linguistics (1978;1985), in which language comes to be considered a network of systems that offer options for performing socially-oriented functions, and social semiotics, which defends language as a means of expressing and establishing social roles and values, multimodality studies are important (Kress & van Leeuwen, 1996;Norris, 2004;Jewitt, 2009;Kress, 2009 and2010). ...
Article
Full-text available
We start from the premise that the representation and the concept are directly related in regard to the teaching and learning of Chemistry. In this article we analyze two episodes of teaching to investigate the meanings produced in the classes in which corporal performances were carried out to represent a Chemical entity. These episodes were extracted from videos produced by two research groups in Chemistry teaching. The analyses evidenced the modal resources used by the teachers in the representation with the body and the possible meanings produced with the students. The analyses indicated that the students noticed the behavior of the chemical entity in the subjects that represented the atoms and that this embodiment presented affordances that helped them understand the concepts explored in the performance. The data lead us to argue that the body and movements also produce learnings. Representing a chemical entity using the body proved to be a rational and affective strategy in the understanding of concepts, which brings implications to the training of teachers, in the sense of overcoming epistemological and affective obstacles.
... A investigação que trata dos multimodos na comunicação e da presença desses multimodos na sala de aula está presente desde a década de 1950 (Smith, 1979;Portnoy, 1982). Considerando os estudos da Linguística Sistêmico-Funcional de Halliday (1978;1985), nos quais a linguagem passa a ser considerada uma rede de sistemas que oferecem opções para desempenhar funções orientadas socialmente, e a semiótica social, que defende a linguagem como um meio para expressar e estabelecer papéis e valores sociais, os estudos de multimodalidade se mostram importantes (Kress & van Leeuwen, 1996;Norris, 2004;Jewitt, 2009;Kress, 2009Kress, e 2010. ...
Article
Full-text available
RESUMO: Partimos da premissa de que a representação e o conceito estão diretamente relacionados quando se trata de ensino e aprendizagem de Química. Neste artigo analisamos dois episódios de ensino nos quais foram executadas performances corporais com o objetivo de investigar os significados produzidos nas aulas em que o corpo foi empregado para representar um ente químico. Esses episódios foram extraídos dos vídeos produzidos por dois grupos de pesquisa em ensino de Química. Foram evidenciados nas análises os recursos modais empregados pelos professores durante a representação com o corpo e os possíveis significados produzidos junto aos estudantes. A análise indicou que os estudantes perceberam o comportamento do ente químico nos sujeitos que representavam os átomos e que essa corporificação apresentou affordances que auxiliaram no entendimento dos conceitos explorados durante a performance. Os dados nos levam a argumentar que o corpo e os movimentos também produzem aprendizagens. Representar um ente químico usando o corpo se mostrou uma estratégia racional e afetiva no entendimento dos conceitos, o que traz implicações para a formação de professores no sentido de superar obstáculos epistemológicos e afetivos.
... Other scholars dive into the topic in social semiotics (O' Toole, 1994; Kress & Van Leeuwen, 1996;Royce, 2002) [16,10,17] . Norris (2004) [15] contends that human interaction is not restricted to language, and some non-linguistic signs like gestures and eye contact can achieve communicative purpose. Gupta (2015) [7] analyzes cultural elements in images and music. ...
Article
This study explores the integration of multimodal teaching, translanguaging, and dialogic pedagogy to improve English literacy among junior students in a Chinese private university. In the context of globalization, proficiency in English is essential for these students, yet many face challenges that hinder their literacy development. This research examines how multimodal mind maps, as tools for translanguaging and multimodal learning, reveal student perspectives on language learning and literacy development in Chinese private universities. The study offers insights with significant potential for shaping educational practices and curriculum design in Chinese private universities (Canagarajah, Suresh. 2012. Translingual practice: Global Englishes and cosmopolitan relations . New York: Routledge). By strategically implementing multimodal teaching techniques, the research introduces innovative strategies to enhance English literacy (García, Ofelia & Li Wei. 2014. Translanguaging: Language, Bilingualism and education . Palgrave Macmillan; Jewitt, Carey & Gunther Kress. 2010. Multimodality, literacy and school English. In D. Wyse, R. Andrews & J. Hoffman (eds.), The Routledge international handbook of English, language and literacy teaching , 342–352. London: Routledge; Vygotsky, Lev S. 1978. Mind in society: The development of higher psychological processes . Cambridge, MA: Harvard University Press). The findings highlight the interplay between linguistic strategies, instructional methods, and student perspectives, offering effective pedagogical interventions that prioritize multimodal engagement. By bridging theory and practice (Canagarajah, Suresh. 2012. Translingual practice: Global Englishes and cosmopolitan relations . New York: Routledge), this research provides actionable insights that address the literacy challenges and aspirations of junior students. Using a multimodal teaching approach (García, Ofelia & Li Wei. 2014. Translanguaging: Language, Bilingualism and education . Palgrave Macmillan), this study not only demonstrates how diverse modes of communication strengthen students English skills and foster their confidence but also addresses the development of their identities as competent members of an English academic community. Drawing on Norton’s (Norton, Bonny. 2013. Identity and Language learning: Extending the conversation . 2nd edn. Bristol: Multilingual Matters) work on identity and investment in language learning, this research underscores the importance of identity construction in the language learning process. It shows how students, through multimodal engagement, can assert their identities and claim the right to participate in academic communities, thereby preparing them for success in a globally interconnected society.
Thesis
Full-text available
This thesis examines the multimodal translation of Critical Role’s D&D campaign Campaign 1: Vox Machina (2015-2017) into an animated series The Legend of Vox Machina (2022-). It also analyses and describes the unique features of The Legend of Vox Machina, which distinguish it from other high-fantasy series, such as intertextuality, humor, gore, sexual content, and the inclusion of characteristic Dungeons&Dragons elements, mechanics, and tropes. Moreover, it describes how this extensive intellectual property has been adapted to a shorter format through the use of different modalities, adaptation operations, and narrative devices. The paper also analyzes the characteristics of the recorded D&D live-streamed campaign (multiple sessions), which is both a linguistic situation (discourse world) and a multimodal, narrative text (text world). Moreover, the presented study underlines the unique challenges posed by the translation of narrative, which was created live during D&D sessions through interactions between the player characters (played by players) and the in-game world including non-player characters (created and controlled by the Dungeon Master). This paper is positioned at the intersection of the fields of study such as multimodality studies, narratology studies, translation studies, game studies, and cognitive linguistic
Chapter
In teaching, learning, and research that focus on a mediated process, screen capture has become a key instrument. Consisting in the recording of multiple channels of information happening simultaneously in‐ and off‐screen (e.g., typing, clicking, navigating, speaking, ambient noises, gestures, eye movements, and props used), screen capture provides a rich multimodal audiovisual rendering of an unfolding event. Once restricted to computer‐based interactions, screen capture is now available on most mobile devices and on virtual or augmented reality headsets. Originally used for step‐by‐step tutorial videos, screen capture has since become an important tool to create pedagogical materials, support asynchronous teaching, or help learners revisit their own learning strategies. Research‐wise, it primarily serves process‐oriented studies by providing access to hidden learning events, allowing to reconstruct a sequence of events, and documenting the multimodality of mediated interactions. Although predominantly employed to investigate text‐based computer‐mediated processes (reading and writing), its use has more recently expanded to also examine oral, aural, and nonverbal activities (eye‐tracking and gesture studies more specifically). Particularly well suited for research grounded in task‐based, ecological, or complex dynamic systems theory approaches, screen capture can be employed for quantitative, qualitative, or mixed methods studies. The greatest challenge of its use resides in the analysis and representation of data due to the sheer volume and richness of information contained in the recordings, requiring overt methodological decisions anchored in theory, and a systematic annotation and transcription system.
Article
This study explores the role of Geographies of Discourse (GoD) (Scollon, Ron. 2013. Geographies of discourse: Action across layered spaces. In Ingrid De Saint-Georges & Jean-Jacques Weber (eds.), Multilingualism and multimodality: Current challenges for educational studies , 183–198. Rotterdam: Sense Publishers) in creating spaces for multilingualism. Building on work that examined the role of multimodality in civic participation (e.g., de Groot and Jocuns 2023. Multimodality as civic participation: The case of Thailand’s rap against dictatorship. Journal of Language and Politics 22(1). 107–128) we show how mapping, analyzing and connecting the multimodal geographies of discourse within language portraits and a mapping task in Kurdistan create insight into the historical, present and future linkages that create a network of mobile language repertoires. We discuss how geographies of discourse (GoD) emerged from how multilingualism in Iraqi Kurdistan map these GoD between the different material objects and create historical and future connections that emerge as small stories. Describing and mapping these intersections and transformations reifies Latour’s (2005. Reassembling the social: An introduction to actor-network-theory . Oxford: Oxford University Press) notion that neither the material nor the immaterial realizes these transformations or gives meaning to them, instead, the transformative actions and meanings indexed are situated in the social relationships between actors that produce these intersections. We argue that this way of understanding the complex networks of discourse practices that produce GoD is important presently when social interaction is situated in a nexus of online and offline spaces.
Article
Full-text available
In this paper, I drew upon the theoretical perspectives on gestures to explore how a teacher used gestures for instructional activities in an early childhood Chinese heritage language classroom. The qualitative data included observations, video recordings of class conversations, and artifacts provided by the participants. All the data were collected from the early childhood classroom at a Chinese heritage language school in a city in the southwestern United States city. The findings show how the teacher used gestures to “write” each Chinese character to prepare the children for their writing practices. The findings also show that the teacher strategically displayed knowledge and engaged the children in meaning-making and reviewing previous lessons through different types of gestures. The conclusions include a review of the teacher’s use of gestures for instructional purposes, such as facilitating meaning-making and preparing the young heritage language learners for their writing practice. The conclusions also included the significance of the research by highlighting the contribution to early childhood heritage language education. Specifically, the study helps contribute to the literature on the use of gestures for supporting children’s language learning and provides recommendations for heritage language teachers to consider using different gestures as pedagogical methods.
Article
This paper studies how internet memes in the context of China are the result of complex semiotic remixing practices and how these multimodal practices offer opportunities for identity work. It offers a situated perspective on the processes that underlie the creation of Chinese memes (Biaoqingbao), the ways in which these memes mediate the digital practices of members in a student community, and how these students find opportunities for identity work. This study takes a social semiotic (Kress, Gunther R. 2010. Multimodality: a social semiotic approach to contemporary communication . Taylor & Francis) and Mediated Discourse Analysis approach (Scollon, Ron. 2001. Mediated discourse: The Nexus of practice . Routledge; Scollon and Scollon 2004; Norris, Sigrid & Rodney H. Jones. 2005. Discourse in action: introducing mediated discourse analysis . London: Routledge) to analyze biaoqingbao as the mediate the social interaction within a light community. The data was collected using a form of digital ethnography. The memes studied in the current paper are unique in two ways. First, they combine both local and global people, places, objects and discourses deploying both Chinese and English texts and effectively create semiotic trajectories between highly localized digital meme practices and more global mainstream meme practices. Second, the practices of producing, sharing and remixing these memes are exclusively situated in a distinct light community and offer important opportunities for multimodal identity performance.
Article
Everyday discourse involves interpreting composite utterances like speech-with-gesture composites. Taking a multimodal and discourse-pragmatic perspective, this single-case study explores the complexity of meaning-making by considering the interweaving of gesture-speech composites and discursive strategies in talk-in-interaction. Specifically, the paper examines various pragmatic functions of kiss-teeth (i.e., a conventionalised sound gesture) when combining it with other multimodal resources. The data for this study was derived from an interview conducted during ethnographic fieldwork in Suriname in 2022. The results show that kiss-teeth has several discourse-pragmatic functions in various contexts, including (1) conveying a reprimand, (2) participating in flirtatious teasing and (3) engaging in gossip. While the research suggests that the meaning of kiss-teeth is closely tied to various multimodal resources, the study also considers the participant’s uses of viewpoint constructions for perspective-taking in demonstrations of kiss-teeth as diverse social actions. The analysis illustrates that the participant indexes distinct gesture functions through viewpoint constructions which rely on descriptions, depictions, and composite utterances. By analysing viewpoint construction, the study does not only aim to scrutinise the gesture’s meanings but also how kiss-teeth contributes to the interaction as a whole. Ultimately, the findings point to the importance of gesture-speech composites and narrative discourse strategies in talk-in-interaction.
Chapter
Full-text available
Der Beitrag „‚The evil of the thriller‘: Multimodalität und Musikvideosemiotik am Beispiel von Michael Jacksons THRILLER“ von Stephan Brössel konturiert die Multimodalität des Musikvideos, wobei er in einem ersten Teil anhand eines Aufrisses der aktuellen Forschungslage eine medientheoretische Fundierung der multimodalen Verfasstheit des Formats entwickelt und diese in einem zweiten Teil am Beispiel von Michael Jacksons Thriller von 1983 in einer detaillierten Analyse illustriert und auf ihre Tragfähigkeit überprüft.
Chapter
This volume contributes to ongoing discussions of ethics in Applied Linguistics scholarship by focusing in depth on several different sub-areas within the field. The book is comprised of four sections: methodological approaches to research; specific participant populations and contexts of research; (language) pedagogy and policy; and personal and interactive aspects of research and scholarship. Moving beyond discussions of how ethics is conceptualized or defined, the chapters in this volume explore ethics-in-practice by examining context-specific ethical challenges and offering guidance for current and future Applied Linguistics scholars. This volume responds to the need to provide context-specific research ethics training for graduate students and novice researchers interested in a variety of contexts and methodological approaches. After engaging with this volume, new and experienced applied linguists alike will gain familiarity with specific ethical challenges and practices within particular sub-disciplines relevant to their work and across the field more broadly.
Article
Despite recent calls to more fully incorporate multimodal perspectives into literacies research, there is still limited scholarship examining how students critically engage in reading activities by drawing on embodied practices. Racially and linguistically minoritized students are particularly disadvantaged by dominant logocentric and developmentalist approaches, which privilege oral and written discourse and often position these students as less capable of performing complex literacy practices. Drawing from three independent ethnographic studies, our multimodal interactional analysis examines how students of a range of ages and raciolinguistic backgrounds use embodied actions and other semiotic resources to agentively navigate text, task, and ideological constraints in activities involving reading and analyzing texts. Our analysis demonstrates the crucial role of students' embodied practices in expanding upon and challenging the constraints of literacy activities, focusing particularly on how students leveraged epistemic stance‐taking, embodied affective responses, and embodied forms of argumentation to negotiate and co‐construct meaning. Through a focus on embodied agency, this paper presents and applies an interactional perspective on the embodied nature of literacy activities; shows how students' creative mobilizations of embodied and other semiotic resources contribute to their critical readings of texts; and offers pedagogical and methodological implications for ways educators and researchers can attend to the intricacies of students' embodied sense‐making in literacy activities.
Article
Full-text available
The paper explores the concept of multimodality in contemporary stylistics of virtual communication, mainly focusing on the process of interpretation. After outlining the basics of multimodal meaning-making theory, it further develops it by incorporating ideas from the theory of media or other related disciplines. The next part describes the of immersive potential of virtual environment, tying it to the multisensorial nature of human epistemic existence. Finally, the last part introduces these ideas to the process of stylistic interpretation, emphasizing the changes and problems, which multimodal virtual communication introduces.
Article
Full-text available
This ethnographic multiple-case study examines how undergraduate students are socialized into the disciplinary norms, values, and practices of a geoscience course at a Canadian university. Transcending logocentric assumptions about academic discourse, this article advances a broader domain of inquiry––multimodal academic discourse socialization––which foregrounds the polysemiotic nature of academic socialization. This approach examines not only linguistic but also a wider range of semiotic resources, including gestural, visual, material, and spatial ones, among others. To understand geoscientists’ disciplinary norms, values, and communicative practices, ethnographic data (classroom observations, semi-structured interviews, course-related artefacts) were thematically analysed. Focal students’ geoscience poster presentation performances were also analysed using multimodal interaction analysis to scrutinize micro-level instantiations of disciplinary practices. Findings highlight how students were socialized into geoscience ‘observations and interpretations’ through a recurrent multimodal classroom activity, which was also reflected in micro-level multimodal practices enacted in students’ geoscience poster presentations. This study emphasizes that multimodal enactments constitute a crucial dimension of disciplinary practices and values connected with learning to think, view, and represent knowledge as geoscientists.
Article
Multimodal research in communication and translation studies is increasingly recognized, yet it remains incompletely explored. Leveraging computational linguistics with both Praat for acoustic analysis and the OpenPose and Rapid Annotator tools for visual analysis, this study delves into the intricate dynamics of the expressive construction thank God , providing a comprehensive examination of both visual and acoustic dimensions. Our objective is to uncover nuanced patterns of multimodal communication embedded within this expression and their implications for Translation and Interpreting. Through an analysis of linguistic features and co-speech gestures present in thank God , we aim to deepen our comprehension of how meaning crisscrosses modalities. Our findings underscore the necessity of a multimodal approach in language studies, emphasizing the requisite to preserve emotional and contextual nuances. The analysis unveils the phonological relevance of the duration of the construction’s second vowel, a key factor for translation. Additionally, data reveals a correlation between the emotion of relief and gestures executed with both hands closer to the chest. Overall, these findings contribute to advancing both multimodal communication research and translation studies, shedding light on the role of multimodal analysis in understanding language and translation dynamics, particularly in the context of constructions like thank God .
Thesis
Full-text available
Frente a la tendencia tradicional de posicionar el lenguaje verbal en el centro de la comunicación, la multimodalidad defiende que la interacción comunicativa se compone, aparte del lenguaje verbal, de otros modos comunicativos que son todos los recursos que los hablantes utilizan para comunicarse o que influyen en el acto comunicativo. Son también los gestos, la mirada, la posición del cuerpo y la proxémica, entre otros, los modos comunicativos que pueden tener el mismo potencial comunicativo que las palabras o que tienen la capacidad de extender o cambiar el sentido de estas. Esta investigación estudia dos de los recursos comunicativos: el lenguaje verbal y los gestos. En concreto, se realiza un análisis multimodal de las perífrasis verbales del español ir + gerundio, dejar de + infinitivo, volver a + infinitivo y ponerse a + infinitivo. De esta manera, el objetivo principal de este estudio consiste en averiguar si existen gestos específicos asociados a estas estructuras perifrásticas.
Article
Shared reading is a crucial site for children’s emerging reading skills when children engage affectively, behaviorally, and cognitively in the reading process. To inform a more holistic, collective, and inclusive view of observable engagement in read aloud (RA) behaviors, authors examined EB preschoolers’ micro-moment engagement behaviors to understand how they recruit their verbal and embodied modes to attend to RAs and how these behaviors align with prior engagement typologies. A phenomenological approach was implemented to examine EB preschool students’ moment-to-moment, multimodal forms of engagement across three video recorded RAs that consisted of 258 units of analysis. Findings reveal that EB preschoolers’ attentiveness and attention divergence during RA activities were showcased across a multimodal continuum but did not necessarily indicate disengagement or inattention. Opportunities for children’s contemplation were constrained by social and behavioral expectations of their compliance, and within less dialogic RAs, these expectations undermined children’s multimodal means of textual engagement in favor of their quiet, undivided display of attention. Findings have implications for recognizing the role of young children’s multimodal expression in supporting their engagement and interactions in RA in ways that may contribute to their developing reading skills and outcomes.
Article
Full-text available
Assessed the relationship between physiological changes (cardiovascular activity) and movement behavior governed by stressors occurring in ongoing communication using reliable physiological and behavioral methods. 13 dyadic and triadic conversations between schizophrenic or bipolar manic patients (aged 17-42 yrs) and their parents, spouse or sibling are analyzed to find evidence for the "stressful" impact of distinct interactional events and to elucidate the effectiveness of the patients' coping ability. The verbal, nonverbal, as well as the psychophysiological domains are described and related to the course of actual interaction. Results suggest that affectively charged statements made by family members or significant others are correlated with (1) specific changes in psychophysiological variables (e.g., decreased period in pulse wave), and (2) distinct changes in psychophysiological variables during specific interactional sequences (e.g., amplification of psychophysiological changes depending on verbal interaction). In contrast to previous findings, there was only a moderate correlation between the nonverbal activity of the patients and the defined verbal stressors. (PsycINFO Database Record (c) 2002 APA, all rights reserved)
Article
Full-text available
Color categories sit at the intersection of 2 central topics in the study of human cognition: (a) the analysis of vision, and (b) the study of semantic categories, or more generally processes of classification. Using as data videotape of archaeologists filling out a coding sheet that requires them to systematically describe the color of the dirt they have excavated, this article describes the practices required to competently classify color within the work life of their profession. The task of color classification is embedded within a situated activity system, which includes not only several different ways of identifying the same color (each designed for alternative uses), but also cognitive artifacts, such as a Munsell color chart and specific embodied practices. The chart creates a historically constituted architecture for perception, a heterotopia that juxtaposes in a single visual field 2 very different kinds of space. As multiple parties fill out the coding sheet together, the full resources of the organization of talk-in-interaction are brought to bear on the contingent tasks they are charged with accomplishing. This investigation of a situated activity system encompassing not only semantic categories, but also physical tools and embodied practices, contrasts with most previous research on color categories, which has focused almost exclusively on mental phenomena, and not on how people perform color classification to pursue a relevant course of action in the consequential settings that make up their lifeworld.
Article
Full-text available
Research in the field of interpersonal behaviour is widely recognised as an important key to understanding the nature of mental illness. In schizophrenic patients especially, deviant social interaction has been proved to be associated with pathological behaviour (see Bellack et al , 1989). Numerous studies have revealed that affective attitudes expressed towards the patient by a relative during hospital admission for an episode of schizophrenia (high expressed emotion (EE)) are predictive of the short-term course of the disorder (Leff & Vaughn, 1985; Jenkins et al , 1986; Nuechterlein et al , 1986; Mintz et al , 1987).
Article
Full-text available
The drawbacks of traditional research into emotional processes have led us to develop a set of methodologies for investigating them in everyday face-to-face communication. The conceptual basis of these procedures is a model of the eliciting conditions of emotional processes as well as a conceptualization of the emotional processes themselves. On the basis of the assumption of conversation as a rule-governed process, one can describe its default temporal, formal, and functional features, for which we use the MAS EDIT and SEQ programs, and the minimal model of communicative exchange, respectively. Violations of these default rules can be identified as unexpected/temporally unpredictable events eliciting emotionalization. The nature of emotionalization is determined by the psychological principle of "standard and deviation." Its investigation under natural conditions requires the following: A noninvasive method of data acquisition (including procedures for rejecting faulty or missing values), measurement (high-resolution recording of physiological, psychomotor, and vocal variables), and the (nonstatistical) construction of an inventory or "relevant effects" (contrastive and template analysis). Finally, we depict three routes of investigating time courses of activation changes as dependent and independent variables and as a target of modification and reflection.
Article
Full-text available
We report on an experiment with a Talking Head, aimed at finding out the relative contributions of pitch accents and rapid eyebrow movements for the perception of focus. For this purpose, we use a "dialogue reconstruction" experiment: subjects have to perform a perceptual task in which they have to determine on the basis of the distributions of pitch accent and eyebrow movements what the focus is of the current utterance. Our results reveal that both pitch accents and eyebrow movements can have a significant effect on the perception of focus, albeit that the effect of pitch is much larger than that of eyebrows.
Article
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
Article
This paper examines the social organization of opening encounters. A base form of opening sequences is suggested, and adaptations of this base form are used to illustrate how the ritual and formal structure of openings interact with variable social and personal circumstances. Some general relationships between recognition, ritual, interaction and social order are suggested; and the implications of these relationships for understanding the structure of ongoing conversation and their relation to social organization are considered.
Article
Discourse analysts need to take into consideration the epistemological dimension of agency on the part of both the actor and the analyst. A focus on action makes it essential to be able to characterize how these social actors orient themselves to different futures through their actions. Conversely, if we want to understand social change, we must theorize how social actors orient themselves to the future. As we argue elsewhere (Scollon & Scollon 2000), we need to develop a theory of anticipatory discourse which includes a theory of human agency as part of a theory of human action that can account for disjunctions between its own theory of human agency and the theory or theories of agency held by the social actors under analysis. Apart from the epistemological dimension of knowledge of the future, a theory of human agency needs to address the dimension of the stance toward the efficacy of individual action in relation to the future. This would range on a continuum from what we call agentive to fatalistic. Mediated Discourse Analysis, like other forms of discourse analysis, has tended to focus on analysis of past action in the context of the present. As R. Scollon declares, 'It remains a task of MDA to develop a workable methodology for capturing the richness of anticipatory discourses.
Article
Focuses on encouraging teachers to become critical and reflective practitioners, researchers of their own professional life and agents of change. Four characteristics defining theory, research, and practice in the context of situated research are listed. Teacher research cannot be limited to quantitative research nor to the classroom. (24 references) (CK)
Chapter
The overarching theme of Discourse and Technology is cutting-edge in the field of linguistics: multimodal discourse. This volume opens up a discussion among discourse analysts and others in linguistics and related fields about the two-fold impact of new communication technologies: The impact on how discourse data is collected, transcribed, and analyzed—and the impact that these technologies are having on social interaction and discourse. As inexpensive tape recorders allowed the field to move beyond text, written or printed language, to capture talk—discourse as spoken language—the information explosion (including cell phones, video recorders, Internet chat rooms, online journals, and the like) has moved those in the field to recognize that all discourse is, in various ways, "multimodal," constructed through speech and gesture, as well as through typography, layout, and the materials employed in the making of texts. The contributors have responded to the expanding scope of discourse analysis by asking five key questions: Why should we study discourse and technology and multimodal discourse analysis? What is the role of the World Wide Web in discourse analysis? How does one analyze multimodal discourse in studies of social actions and interactions? How does one analyze multimodal discourse in educational social interactions? and, How does one use multimodal discourse analyses in the workplace? The vitality of these explorations opens windows onto even newer horizons of discourse and discourse analysis.
Article
In this study, we integrate visual and linguistic research to analyze the relationship between a city's regional, national, transnational, and European identities as the face of four cities depicts it. We explore the cultural identity of Brussels, Luxembourg, Munich, and Dortmund through the study of public signs found in these cities. We examine how the choice of language, layout, typography, accompanying images, and positioning of city signs reflect the tensions involving the change from a national or regional identity to a Pan‐European identity. Our methodology combines an ethnographic approach and neutral observations with interviews of city inhabitants. By combining interpretive analysis with a meta‐discursive approach to how signs are read and perceived, we consider both the production and reception ends of the signs.
Article
This article identifies some limitations of discourse analysis by analyzing interactions between five boys in which the TV and the computer are featured as mediational means. The incorporation of several modalities into transcripts and a shift in focus from primarily language to human action facilitate a better understanding of the multi-modal interaction involved. The use of conventional transcripts with a focus on language demonstrates that movie- and computer-mediated interactions appear fragmented; by contrast, an inclusion of images into the transcripts, representing central interactions and/or images of a movie or computer screen, demonstrates the significant visual modes that are imperative to the ongoing talk. Just as written words correspond to the oral language, images can exemplify the global interaction among the participants, or they can represent the images on the screen. In addition, viewing an image is much faster than reading a description, so that these images also display the fast pace of the movie- and/or computer-mediated interaction.
Article
Suggests that quantitative analyses are useful procedures through which to isolate constraints from different levels of discourse and through which to separate the ways in which structure, meaning, and social action differently influence the production of discourse. The value of quantitative analyses of discourse options is demonstrated by (1) focusing on 2 discourse options for the representation of cause and effect and (2) operationalizing semantic constraints (temporal reference) and pragmatic constraints (discourse topic) and examining the relative effects of each on causal reversibility. Results of a quantitative analysis show that no single level of constraint was able to account for causal reversibility. However, the conclusion that causal reversibility is constrained by factors that crosscut different levels of discourse organization supports views of discourse as an interlocking system of structure, meaning, and action. (47 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
An instance of fruitful cross-disciplinary contacts is examined in detail. The ideas involved include (1) the double-blind hypothesis for schizophrenia, (2) the critique of game theory from the viewpoint of anthropology and psychiatry, and (3) the application of concepts of communication theory and theory of logical types to an interpretation of psychoanalytic practice. The protagonists of the interchange are Gregory Bateson and the two mathematicians Norbert Wiener and John von Neumann; the date, March 1946. This interchange and its sequels are described. While the interchanges between Bateson and Wiener were fruitful, those between Bateson and von Neumann were much less so. The latter two held conflicting premises concerning what is significant in science; Bateson's and Wiener's were compatible. In 1946, Wiener suggested that information and communication might be appropriate central concepts for psychoanalytic theory--a vague general idea which Bateson (with Ruesch) related to contemporary clinical practice. For Bateson, Wiener, and von Neumann, the cross-disciplinary interactions foreshadowed a shift in activities and new roles in society, to which the post World War II period was conducive. Von Neumann became a high-level government advisor; Wiener, an interpreter of science and technology for the general public; and Bateson a counter-culture figure.
Article
Traditional methods of analyzing human peripheral blood flow (PBF) do not account for its continuous nature or small-scale variation. Sequel transformation is presented as a new methodology for measuring these variations. This analysis generates a richer record of aspects of activation of the autonomous nervous system than traditional interbeat-interval (IBI) measurement. Presumably, in addition to heartbeat (i.e., IBI), other information can be extracted that is consistent with the influence of several physiological and psychological factors. This kind of variation is reflected in the PBF signal. After determining and extracting relevant events from vascular volume data generated by sequel transformation, we demonstrate the use of template analysis to analyze patterns of events as time courses of activation. All procedures are included in the computer program VASC Analyzer.
Article
Methodological approaches in which data on nonverbal behavior are collected usually involve interpretative methods in which raters must identify a set of defined categories of behavior. However, present knowledge about the qualitative aspects of head movement behavior calls for recording detailed transcriptions of behavior. These records are a prerequisite for investigating the function and meaning of head movement patterns. A method for directly collecting data on head movement behavior is introduced. Using small ultrasonic transducers, which are attached to various parts of an index person's body (head and shoulders), a microcomputer determines receiver-transducer distances. Three-dimensional positions are calculated by triangulation. These data are used for further calculations concerning the angular orientation of the head and the direction, size, and speed of head movements (in rotational, lateral, and sagittal dimensions). Further analyses determine relevant changes in movements, identify segments of movements, and classify the quantifications of movement patterns. The measured patterns of nonverbal behavior can be accurately related to features of verbal communication and other time-related variables (e.g., psychophysiological measures). To estimate the possible meanings of behavioral patterns, a heuristic is proposed that includes the situational context as the basis of interpretation.