No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Multimodal Discourse Analysis
Abstract
Multimodal discourse analysis is an emergent field that began around the verge of the millennium with books such as Reading Images: A Grammar of Visual Design (Kress & van Leeuwen, 1996/2006), Mediated Discourse as Social Interaction (Scollon, 1998), Multimodal Discourse (Kress & van Leeuwen, 2001), and Analyzing Multimodal Interaction: A Methodological Framework (Norris, 2004). Initially, multimodal discourse analysis was primarily the domain of mediated discourse analysts, social semioticians, and systemic functional linguists. While early developments were somewhat overlapping in time, these works resulted from, and aligned with, two separate major paradigm shifts stemming from previous work in discourse analysis...
... For this purpose, I have selected two video documentaries depicting compassionate educators, one from Arunachal Pradesh, India and the other from San Diego, California, United States. I shall explore the exemplars utilizing the multimodal discourse analysis process as elaborated in Norris (2004). Multimodal discourse analysis investigates various interlinked and interdependent interactions in communication that a person engages simultaneously beyond just speech and action. ...
... Multimodal discourse analysis considers an integral part of its analysis what is usually considered the context in conventional discourse analysis like the layout and environment. According to Norris (2004), an individual uses embodied modes like speech, gaze, gesture, and posture as well as disembodied modes like the layout of the classroom. Each mode of communication is utilized in correspondence with different degrees of awareness and attention by the communicator. ...
... In a focused communication some modes are present while others are in the background. Thus Norris (2004) spoke of a "foregroundbackground continuum" (p. 104) in interactions. ...
This reflective position paper explores teaching and learning of compassion in educational institutions following the theory of enaction proposed by Varela, Thompson, and Rosch (2016). In the enactive view, information processing and cognition are situated in the dynamic relationship between the embodied organism and its environment. Following such a theoretical premise, the paper reviews the stories of two compassionate educators to propose that they enact their teaching space, indirectly transforming it into a learning space of universal compassion. A new model of teaching and learning of compassion, namely, the enactive modelling through non-egocentric responsiveness-embedded stories is suggested through the paper. The study utilizes the multimodal discourse analysis methodology for the investigation of the proposed model.
... For this purpose, I have selected two video documentaries depicting compassionate educators, one from Arunachal Pradesh, India and the other from San Diego, California, United States. I shall explore the exemplars utilizing the multimodal discourse analysis process as elaborated in Norris (2004). Multimodal discourse analysis investigates various interlinked and interdependent interactions in communication that a person engages simultaneously beyond just speech and action. ...
... Multimodal discourse analysis considers an integral part of its analysis what is usually considered the context in conventional discourse analysis like the layout and environment. According to Norris (2004), an individual uses embodied modes like speech, gaze, gesture, and posture as well as disembodied modes like the layout of the classroom. Each mode of communication is utilized in correspondence with different degrees of awareness and attention by the communicator. ...
... In a focused communication some modes are present while others are in the background. Thus Norris (2004) spoke of a "foregroundbackground continuum" (p. 104) in interactions. ...
This reflective position paper explores teaching and learning of compassion in educational institutions following the theory of enaction proposed by Varela, Thompson, and Rosch (2016). In the enactive view, information processing and cognition are situated in the dynamic relationship between the embodied organism and its environment. Following such a theoretical premise, the paper reviews the stories of two compassionate educators to propose that they enact their teaching space, indirectly transforming it into a learning space of universal compassion. A new model of teaching and learning of compassion, namely, the enactive modelling through non-egocentric responsiveness-embedded stories is suggested through the paper. The study utilizes the multimodal discourse analysis methodology for the investigation of the proposed model.
... Multimodal analysis (MA) is based on the premise that identity is continuously shaped at the micro level, yet that it is claimed and contested against the "histories and cultures of the individuals and the societies that they belong to..." (Norris, 2005: 183). The multimodal framework helps to identify all communicative modes at play during a communicative event (Norris, 2004). This framework borrows elements from Scollon and Scollon's (2004) nexus analysis, which claims that within discourse systems, actors have been socialized to act in certain ways and that these actions differentiate one discourse system from another. ...
... Mediated discourse analysis was used in CASE STUDIES II and III, although each case study uses a different focus of MDA. CASE STUDY II uses Norris' (2004) multimodal framework to deconstruct blogger's choices to blog under a pseudonym rather than a name associated with a physical identity. CASE STUDY III uses Scollon and Scollon's (2005) cycles of discourse and historical bodies in order to focus on the ongoing negotiation between the cycles of discourse which gives rise to a tension between the expectations of traditional institutionalized knowledge production and blogging as an alternative mode of academic production. ...
... In CASE STUDY II, the social identity pseudonymous blogger is deconstructed through the mapping of identity elements of two popular bloggers, and compared to the mapping of identity elements of these bloggers by the mass media. Norris' (2004) multimodal framework was used to identify and discuss how social identity is negotiated against cultural projections and societal currents (as performed through the affordances and constraints of the weblog tool). Two well-known bloggers who blogged under a pseudonym were chosen for analysis. ...
... Utilizing a discourse analysis methodology [19], [20], we focused simultaneously on the textual content, the implicit positioning, the participant-researcher interaction, and the broader context for the two interviews. ...
... Many times, the original data is omitted to benefit certain government or capital owners. Through backgrounding of the social actors, the Exclusion becomes less radical (Norris, 2004;Van Leeuwen, 2014). ...
As one of the major venues for articulating and disseminating national agendas and opinion discourse, national newspapers play a critical role in promulgating ideology. Underpinned by Intertextuality and Social Actor Theory, this study explores intertextual aspects of China Daily’s reporting of COVID-19 to unearth hidden ideology behind texts. The analysis reveals diversified voices from multiple actors around the globe, with China’s official leaders appearing most frequently. In the portrayal of social actors, some strategies like impersonalisation, and genericisation are utilised to add impersonal authority or power to an actor’s activity, actant’s engagement, and increase the trustworthiness of news. These reprsentational strategies belies a transformation in Chinese media discourse with a softer approach is used in wielding ideological intentions through journalistic practices of intertextuality. Our findings help to unravel how news texts draw on, echo, and bring together multiple intertextual resources realised in the forms of discourses. The circulations, dissemination and incorporation of these intertextual relations and practices construct specific understandings of ideology consolidation and public relations within the context of China and its response to the COVID-19 pandemic.
... As Norris (2020: 17) reminds us of, the different modes and modalities are interrelated in the users' perception of them ("chains of modes and modalities") and act in support of linguistic and language intakes. Multimodality would participate in users' attention according to different combinations specific to individual strategies and according to communication situations during which learners can modify their "modal density" (Norris, 2004), by favouring certain modes and modalities at a given moment of their simulation. ...
In the last decade, Virtual Reality (VR) has been increasingly studied and used for educational purposes (Mellet-d’Huart & Michel, 2006; Gobin-Mignot & Wolff, 2019). Whilst this technology is highly adapted for technical skills training such as flying an aircraft or performing surgery, simulations through VR with language learners seem less relevant at first sight. However, previous work has highlighted numerous potentials, especially for the development of language skills which requires a rich contextualisation to ensure the necessary input and intake processes for aural/oral skills acquisition (Pekarek Doehler & Berger, 2018). We have sought to identify professional social places virtualized by VR applications, the results of which are presented as a taxonomy for teachers in English Professional Purposes (EPP). As mainstream VR applications are mostly designed for leisure purposes, we will discuss their integration into formal learning through the predominant role of educational scenarios that some applications require if they are used in a language teaching/learning context. We will show the value of VR in supporting situated learning specific to professional situations, as well as their potential for soft skills development. The results open the floor for questions that discuss whether VR applications are indeed a complementary resource among other authentic resources used in the classroom and whether they lead to a renewal of language learning practices.
... There are many new developments and perspectives. The one that we work in at the AUT Multimodal Research Centre is multimodal (inter)action analysis (MIA) (Norris 2004a(Norris , b, 2011a(Norris , 2029(Norris , 2020Geenen et al. 2015), which has grown from theory and methodology to a fully-fledged framework. I will take a moment to set out the state of this framework and discuss the role that audio-visual technology plays in MIA. ...
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... Modes have a meaning potential that is changeable through social interaction, and this potential is actualized and organized based on concrete social, cultural, and historical contexts. (Geenan et al. 2015(Geenan et al. : 1040 In other words, the same kaomoji can communicate different meanings based on the specific context in which it appears and how it interacts with other signs and resources at the moment of its use and interpretation. If this is correct, it is important to investigate kaomoji in a concrete context and in relation to other semiotic resources being employed. ...
This paper investigates multimodal strategies for balancing formality and informality online. The analysis of 300 comment-reply interactions on a recipe sharing site in Japan demonstrates that writers tend to avoid being overly formal or informal in their messages. For example, most comments and replies are written in polite forms but many incorporate some plain forms and colloquial expressions. Linguistic features, however, are not the only way through which the writers manage an appropriate level of formality and informality. The study examines the role of kaomoji or Japanese-style emoticons for socio-relational work online. Some kaomoji function locally as cues for interpreting the sentences featuring kaomoji . All kaomoji , including those with local functions, work to enhance the social presence of the writers on the screen via pictographic gaze and gestures, which increases the perception of intimate rapport. The findings underscore the importance of a multimodal perspective in examining how people handle social relationships online.
... The analysis will also take into account the evaluative properties of the texts under scrutiny (Hunston and Thompson 2000;Bednarek 2009aBednarek , 2009b, by investigating the linguistic markers signalling the legal appraisal of claimants' stories and the values emerging from how immigration judges handle these cases. This paper takes a multimodal perspective (Norris, 2004(Norris, , 2011a(Norris, , 2012a(Norris, , 2013a(Norris, , 2013b(Norris, , 2013cGeenen, 2013aGeenen, , 2013bGeenen, Norris & Makboon, 2015;Makboon, 2015;Pirini, 2014Pirini, , 2015Pirini, , 2016, building upon mediated discourse analysis (Scollon, 1998Norris and Jones, 2005a, b) to examine how identities are produced in family videoconferencing (inter)actions. ...
Book of Abstracts of the 6th Critical Approaches to Discourse Analysis across Disciplines Conference (CADAAD 2016)
... Progressing from co-production and the distribution of attention/awareness, the data suggests that the reason show (and sometimes) tell manifests with such regularity and is recognised as such by the social actors involved in the multiparty interaction is due to the ways in which 'showing' functions as an exemplification of concrete experiences which are embedded in the material state of the object, entity or artefact. While analytical attention has been allocated to the identity producing potential of static objects placed in particular environments and the frozen actions embedded therein (Norris and Makboon 2015), here the relationship between the objects, their interactive relevance and the frozen actions embedded in them is explicitly acknowledged by social actors within the interaction. ...
This article provides a preliminary answer to exactly why video-conferencing is evaluated as better than traditional telephony for long-distance familial interaction by allocating analytical attention to the showing of objects during interaction. While it is acknowledged that ‘showing’ constitutes an interactive move less contingent on linguistic maturation, more importantly, the showing of objects, artefacts or entities during video-conferencing interactions exemplifies an agentive and volitional production of identity elements on behalf of young children. Thus, while some have pointed to shortcomings of conversation-like activities mediated by video-conferencing in favour for more activity-driven tasks, I make a case for drawing upon pre-existing components of the material surround as a means to more comprehensively and longitudinally engage younger children in video-conferencing interaction.
... We adopted a multimodal interactional analysis approach, which is suited to the analysis of stories conveyed through a combination of modalities (Norris, 2004). Multimodal analysis makes a distinction between embodied (gesture, gaze and language) and disembodied (e.g. a book, iPad and picture) resources used for meaning-making (see Flewitt, 2012). ...
... Such expansion thus rallies for paradigm changes to attend to advances in new technologies and institutionalization which have been studied over the past 20 years but are still in need of robust research (Kress, 2013). The second trend focuses on understanding social interactions as multimodal and exploiting non-linguistic devices and instruments to archive and analyze social interactions (Norris, 2004). This complex task involves not only a strong perception of new technologies, media, materials, resources, and organizations but also an interdisciplinary re-imagination of affordances and interactions. ...
Multimodality is an emergent aspect of educational assessment that has received increasing attention from educators and educational researchers. Unfortunately, this new assessment practice has been entrenched in the hegemonic gaze of teaching- and learning-centric pedagogical paradigm from the outset. To reconstruct multimodal educational assessment, this issue paper probes into three essential aspects of assessment, namely, a) ownership and legitimacy, b) knowledge production, and c) participation and productive space. Problematizing these core aspects not only furthers multimodal assessment researchers' understanding of the core issues of learning through assessment practices but also delineates the domain of multimodal educational assessment in terms of theory and practice.
This chapter proposes a solution to open social semiotics to a
particular aspect of political marketing. It illustrates this through a multimodal
analysis that looks at how political actors correlate certain semiotic
resources in discursive interactions from the perspective of their emotional
content. The multimodal choices that political actors make during discursive
interactions allow them to build their political brand and make connections
with the audience on an emotional level.
A partir de los trabajos de Frith, Zargosky-Thomas, Moylan, Juan de Dios Cuartas y Di Cione, hay un conjunto de textos que nos aproximan a la grabación como campo de estudio; sin embargo, aún falta por establecer líneas teóricas que se conecten de manera directa con la episteme del objeto fonográfico y que no se circunscriba de forma exclusiva a los aspectos sociales, antropológicos o culturales de la grabación. En el presente trabajo se hace una disertación sobre la teoría del discurso multimodal desarrollada por Jewitt, Kress y Leeuwen y que en lengua castellana ha tenido una aproximación por Sans. Se hace un resumen de la teoría, sus categorías y bibliografía más relevante y se vincula con los estudios la grabación sonora. Por último, se presenta un estudio de caso focalizado en la figura del cantante de cumbia Pastor López, lo que permitió develar en las grabaciones los distintos modos discursivos
This article explores how social semiotic analysis promotes the field of critical discourse analysis and visual studies in EFL/ESL teaching materials, media, educational software packages, and also other language-related fields of study. At first glance, the article attempts to introduce social semiotics as a systematic methodology and discipline through summarizing theoretical backgrounds and methodological foundations. Secondly, it elaborates close relationships among the language, signs, and cultural values as meaning making resources and how these interrelated concepts are discussed in various studies in the past years. The article finally reviews a variety of studies done in this field and emphasizes that social semiotics is the key to highlight the hidden meanings usually imposed and dictated by higher institutions and cultural backgrounds. This article can be of potential help and use for those researchers who require a quick understanding of social semiotics and critical studies in language-related fields to conduct research in these areas.
The article investigates the Self and Other binaries in wartime visual literature published in Bengali-language children’s periodicals in West Bengal, India during the Bangladesh Liberation Struggle 1971. The study applies a critical multimodal framework using the Social Actors Approach and Social Semiotics within the Discourse-Historical Approach. The binaries are defined by the representation and subsequent differentiation of physical, linguistic, and cultural features of the Bengali and non-Bengali social actors and through their actions in the plots. The representation of social actors in the texts conforms to as well as deviates from typical wartime propaganda.
This thesis examines the construction of hybrid and fluid ethnic identity elements as produced by Māori and Pacific female artists. Ethnic identity studies within New Zealand reveal different types of ethnic identities, and although there is research being conducted into hybrid and fluid Māori and Pacific identity elements, no studies have been done examining the construction of these identities through mediated action. This thesis attempts to fill this gap.
Using video ethnography and socio-linguistic interviews, data were collected and analysed utilising multimodal (inter)action analysis (MIA) as the theoretical and methodological framework. Vertical identity production and site of engagement are analytical tools within MIA that allow for the study of the intersection between discourses and mediated actions performed by social actors. These analytical tools were applied to interview and video transcripts selected from the data, following a systematic process of data cataloguing.
Analysis of the data is presented in three chapters which show the ethnic and creative identity production of the participants as constructed through the central, intermediary and outer layers of discourse. The first analysis chapter demonstrates the way the participants create art by blending traditional and contemporary features and diverse knowledge, in turn constructing their immediate ethnic and creative identity elements. This analysis is compared to the way the participants verbalise these identity elements within their interviews. The second analysis chapter examines the way experiences of exclusion and inclusion from within their networks shape their continuous ethnic and
creative identity elements. The third analysis chapter explores moments of exclusion and inclusion but within larger communities such as mainstream New Zealand, and their ethnic communities. It also illustrates the way in which the participants’ art creates inclusion and shapes the general ethnic and creative identity development of other social actors.
Following this, wider discourses and practices are examined using the site of engagement as the analytical tool. This chapter demonstrates the way in which wider discourses such as colonial, superiority/inferiority and racism discourse intersect with practices such as superiority/inferiority, gratitude, and marginalisation and with the mediated actions performed by the participants. This analysis highlights the negative impact these discourses and practices can have on ethnic identity construction for Māori and Pacific social actors. To this end, numerous recommendations are made within the conclusion with the intention of changing these wider discourses and practices.
This thesis contributes to knowledge in the area of Māori and Pacific identity studies by utilising multimodal (inter)action analysis to study identity production. It also contributes to the theoretical and methodological framework of multimodal (inter)action analysis.
This study analyzed the issue of insecurity and self-acceptance portrayed in Samsung Galaxy A8 advertisement entitled “Lets You Be You”. The main theory for this study is the theory of multimodal discourse analysis by Paltridge (2012) and the theory of body language by Pease (2004) as the supporting theory. Qualitative approach is applied because this study involves the interpretation of both verbal and nonverbal data. The findings showed how the issue of insecurity is reflected through the reaction of the participants in the video before and after receiving a compliment. Furthermore, the construction of the changing process from insecurity to self-acceptance can be seen from the genuine reaction of the compliment, as well as the change of facial expression throughout the video. Key words: Insecurity, Self-acceptance, Multimodal Discourse Analysis, Advertisement, Samsung
This concise guide outlines core theoretical and methodological developments of the growing field of Multimodal (Inter)action Analysis.
The volume unpacks the foundational relationship between multimodality and language and the key concepts which underpin the analysis of multimodal action and interaction and the study of multimodal identity. A focused overview of each concept charts its historical development, reviews the essential literature, and outlines its underlying theoretical frameworks and how it links to analytical tools. Norris illustrates the concept in practice via the inclusion of examples and an image-based transcript, table, or graph.
The book provides a succinct overview of the latest research developments in the field of Multimodal (Inter)action Analysis for early career scholars in the field as well as established researchers looking to stay up-to-date on core developments.
In recent years there has been increased academic and professional interest and awareness in approaches to English language teaching (ELT) that take a plurilingual approach. This is often combined with a multimodal stance. The outcome of this combination is an approach to English language teaching that integrates multiple languages and multiple semiotic resources. This paper examines how a plurilingual approach to ELT can be viewed through a multimodal lens by analyzing the construction of a plurilingual talking book created as a student project in an elementary public school. The analysis uses multimodal analysis software to map the interaction of languages and images, in order to determine how these function as meaning-making resources in a multimodal, multiple-language text created by linguistically diverse students with high ELT needs. The findings indicate how combinations of different semiotic resources work together to create meaning, delineates the role of English in meaning-making, and illustrates the children’s multilingual interactions in the creation of their collaboratively composed multimodal talking book.
Integral captions and subtitles are specific forms of captions and subtitles that are designed to be essential elements of videos in coordination with sound, signs, and other modes of communication. Integral captions reflect the importance of embodied rhetorics in Deaf culture, particularly in the kinetic language of ASL and Deaf Space design practices. Designing a (Deaf) space for integral captions that embody multimodal and multilingual communication is an essential multimodal literacy practice that benefits d/Deaf and hearing composers and viewers. Five criteria that characterize integral captions provide instructors and scholars with a tool for captions and embodied rhetorics.
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
Excerpts available on Google Books. For more info, go to publisher's website : http://ukcatalogue.oup.com/product/9780195117530.do
Discourses in Place is essential reading for anyone with an interest in language and the way we communicate. Written by leaders in the field, this text argues that we can only interpret the meaning of public texts like road signs, notices and brand logos by considering the social and physical world that surrounds them. Drawing on a wide range of real examples, from signs in the Chinese mountains, to urban centres in Austria, Italy, North America and Hong Kong, this textbook equips students with the methodology and models they need to undertake their own research in 'geosemiotics', the key interface between semiotics and the physical world. Discourses in Place is highly illustrated, containing real examples of language in the material world, including a 'how to use this book' section, group and individual activities, and a glossary of key terms. © 2003 Ron Scollon and Suzie Wong Scollon. All rights reserved.
Nexus Analysis presents an exciting theory by two of the leading names in discourse analysis and provides a practical guide to its application. The authors argue that discourse analysis can itself be a form of social action. If the discourse analyst is part of the nexus of practice under study, then the analysis can itself transform that nexus of practice. Focussing on their own involvement with and analysis of pioneering communication technologies in Alaska they identify moments of social importance in order to examine the links between social practice, culture and technology. Media are identified not only as means of expressing change but also as catalysts for change itself, with the power to transform the socio-cultural landscape. In this intellectually exciting yet accessible book, Ron Scollon and Suzie Wong Scollon present a working example of their theory in action and provide a personal snapshot of a key moment in the history of communication technology, as the Internet transformed Alaskan life. © 2004 Ron Scollon and Suzie Wong Scollon. All rights reserved.
This article is an introduction to the theoretical and methodological backgrounds of
multimodal (inter)action theory. The aim of this theory is to explain the complexities
of (inter)action, connecting micro- and macro levels of analysis, focusing on the social
actor. The most important theoretical antecedent, mediated discourse analysis (see
Scollon 1998, 2001b), is presented with its key concepts mediated action and modes.
It is shown how action is used as the unit of analysis and how modes are understood
in multimodal (inter)action analysis – as complex cultural tools, as systems of
mediated action with rules and regularities and different levels of abstractness. Subsequently,
methodological basics are introduced, such as lower-level, higher-level and frozen
action; modal density, which specifies the attention/awareness of the social actor; and
horizontal and vertical simultaneity of actions. Horizontal simultaneity can be plotted on
the heuristic model of foreground-background continuum of attention/ awareness. Vertical
simultaneity of actions comprises the central layer of discourse (immediate actions), the
intermediate layer (long-term actions) and the outer layer (institutional or societal contexts).
In short, it is sketched how multimodal (inter)action analysis aims to answer questions
about the interconnection of the different modes on a theoretical as well as on a
practical level.
This article has the following two overarching aims. First, it traces the development of multimodal discourse analysis and sets out its main descriptive and analytical parameters; in doing so, the article highlights the specific advantages which the multimodal approach has to offer and exemplifies its application. The article also argues that the hierarchical arrangement of different semiotics (in the way common sense construes this) should not be lost from sight. Second, and related to this last point, the article will advance a complementary perspective to that of multimodality: resemiotization. Resemiotization is meant to provide the analytical means for (1) tracing how semiotics are translated from one into the other as social processes unfold, as well as for (2) asking why these semiotics (rather than others) are mobilized to do certain things at certain times. The article draws on a variety of empirical data to exemplify these two perspectives on visual communication and analysis.
In this paper, I look at what might be gained and what might be lost as we move from representation primarily through writing to representation primarily though image. In so doing, I also consider issues related to learning, knowledge, and human agency.
In this monograph, the author offers a new way of examining the much discussed notion of identity through the theoretical and methodological approach called multimodal interaction analysis. Moving beyond a traditional discourse analysis focus on spoken language, this book expands our understanding of identity construction by looking both at language and its intersection with such paralinguistic features as gesture, as well as how we use space in interaction. The author illustrates this new approach through an extended ethnographic study of two women living in Germany. Examples of their everyday interactions elucidate how multimodal interaction analysis can be used to extend our understanding of how identity is produced and negotiated in context from a more holistic point of view.
Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences.
Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work.
Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
This article identifies some limitations of discourse analysis by analyzing interactions between five boys in which the TV and the computer are featured as mediational means. The incorporation of several modalities into transcripts and a shift in focus from primarily language to human action facilitate a better understanding of the multi-modal interaction involved. The use of conventional transcripts with a focus on language demonstrates that movie- and computer-mediated interactions appear fragmented; by contrast, an inclusion of images into the transcripts, representing central interactions and/or images of a movie or computer screen, demonstrates the significant visual modes that are imperative to the ongoing talk. Just as written words correspond to the oral language, images can exemplify the global interaction among the participants, or they can represent the images on the screen. In addition, viewing an image is much faster than reading a description, so that these images also display the fast pace of the movie- and/or computer-mediated interaction.
A theory of action must come to terms with both the details of language use and the way in which the social, cultural, material and sequential structure of the environment where action occurs figure into its organization. In this paper it will be suggested that a primordial site for the analysis of human language, cognition, and action consists of a situation in which multiple participants are attempting to carry out courses of action in concert with each other through talk while attending to both the larger activities that their current actions are ambedded within, and relevant phenomena in their surround. Using as data video recordings of young girls playing hopscotch and archaeologists classifying color, it will be argued that human action is built throught the simultaneous deployment of a range of quite different kinds of semiotic resources. Talk itself contains multiple sign systems with alternative properties. Strips of talk gain their power as social action via their placement within larger sequential structures, encompassing activities, and participation frameworks constituted through displays of mutual orientation made by the actors' bodies. The body is used in a quite different way to perform gesture, again a class of phenomena that encompasses structurally different types of sign systems. Both talk and gesture can index, construe or treat as irrelevant, entities in the participants' surround. Moreover, material structure in the surround, such as graphic fields of various types, can provide semiotic structure without which the constitution of particular kinds of action being invoked through talk would be impossible. In brief it will be argued that the construction of action through talk within situated interaction is accomplished through the temporally unfolding juxtaposition of quite different kinds of semiotic resources, and that moreover through this process the human body is made publicly visible as the site for a range of structurally different kinds of displays implicated in the constitution of the actions of the moment.
El autor ofrece un acercamiento al mundo de las historietas desde el punto de vista de que se trata de un medio de comunicación con una estructura y un lenguaje propio. Analizan los diversos elementos de las historietas, desde el vocabulario hasta su diseño y su retórica.
Multimodality, resemiotization: Extending the analysis of discourse as multisemiotic practice
- R Iedema
Iedema, R. (2003). Multimodality, resemiotization: Extending the analysis of discourse as
multisemiotic practice. Visual Communication, 2(1), 29-57. doi:
10.1177/1470357203002001751
Reading science: Critical and functional perspectives on discourses of science
- J L Lemke
Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J.
R. Martin & R. Veel (Eds.), Reading science: Critical and functional perspectives on
discourses of science (pp. 87-113). London, UK: Routledge.