Multimodality and Practice: Investigating Theory-in-Practice-through-Method
Abstract
In this wide-ranging collection, leading scholars, researchers, and emergent researchers from around the world come together and present examples of multimodal discourse analysis in practice. The book illustrates new theoretical, methodological and empirical research into new technologies such as the internet, software, CD-ROM, video, and older technologies such as film, newspapers, brands or billboards. Each chapter demonstrates how aspects of multimodal theory and method can be used to conduct research into these and other multimodal texts. Taken together, the book advances the field of multimodal discourse analysis and offers a range of current studies that address some of the most important questions facing researchers and students when investigating new or old technologies multimodally in linguistics, education, communication studies, sociology, anthropology and other fields.
... In multimodal analysis, human learning processes and the learning environments are considered complex systems (Elen & Clark, 2006). To understand the way humans learn and the way they make meaning of their physical and social lives, multimodal analysis takes into account many different variables, multiple perspectives, varieties of information retrieval and processing, and mixed-methods (Jewitt, 2009;Norris, 2011). Multimodal analysis can function as a framework as well as the methodology to study human learning processes, perceived affordances of technology, and semiotic resources that facilitate learners to achieve their goals (Baldry & Thibault, 2010;. ...
... Norris's approach is a combination of the model applied by Goodwin (2007) and the multimodal transcription techniques of Baldry and Thibault (2010). In a book chapter entitled Collecting and Transcribing Ethnographic Data: A Case Study of Two Woman in Germany, Norris (2011) explained the extensive fieldwork and transcription processes that she went through in order to extract meaning potential of multimodal texts. In this study, she collected rich data including "audiotapes, diaries, magazines, newspapers, books, TV and radio programs, field notes, videotapes, photos, family and social networks, and creative writings" (Norris, 2011, pp. ...
... The example of analyzing giving-receiving using the concepts of systemic functional linguistic and semiotic analysis means to show that these concepts are relevant for many fields and not only for linguistic and language studies. In alignment with the new media, semiotic analysis and social semiotics have been applied widely in multimodal analysis of the mathematics classroom (O'Halloran, 2008; Thomas, Yoon, & Dreyfus, 2009), literacy studies (Bomer, Zoch, David, & Ok, 2010; Kabuto, 2009), science education (Kress, Charalampos, & Ogborn, 2001; Varelas, Pappas, & Arsenault, 2013), medical treatment (Daldrup-Link & Gooding, 2010; Skarin & Alexander, 2000),and media studies(Bateman, Judy, & Renate, 2006;Knox, Patpong, & Piriyasilpa, 2010).Multimodal analysis is the study of the use of semiotic resources of technology to enrich the quality of communication and education (LeVine & Scollon, 2004;Norris, 2011;. Humans use their senses and their bodies to receive and deliver information and process these in their mind. ...
This mixed method research applied multimodal analysis, sociogram analysis, content analysis, discourse analysis, statistics descriptive, and hypothesis testing to study the process of learning Kindle Fire's usability through the online discussion forum. The key aspects discussed in this study are the semiotic resources of Kindle Forum, the meaning-making process, the affordances and usability of Kindle Fire, the affordances and usability of the forum, and the methodological implications for this study. Two settings of this study were (a) the webpage as an environment for users to experience elements of reading paths and (b) the discussion forum, as a place to post experiences using Kindle Fire. Simple random and purposive sampling were implemented to select, as samples, 437 postings from 247 participants. Data were primarily collected using observations, and MAXQDA 11 was the main software used to analyze the data. Findings and analysis affirmed that: (1) the Kindle Fire Discussion Forum has multisemiotic resources that enable users to quickly perform acts of transferring meaning of the discussion content; (2) there were two categories of action potential of the forum: (a) actions made to the page and feedback from the page, and (b) interactions between users in the discussion itself; and, (3) discussions in the forum could not reveal the affordances but can be used to explain the conventions and constraints of the Kindle Fire. However, the methodology applied in this study would benefit from further testing in a similar, forthcoming study. As a starting point for this study, though, it began with a discussion on technology mediated assistance and peer support in learning new technology.
... For example, we speak of 'creating a video' or of 'changing a law', but typically do not speak of creating videos as taking a sequence of steps. Such a problem could be momentarily resolved by following Lemke (2000aLemke ( , 2000b and Norris (2002Norris ( , 2004Norris ( , 2011, who construe actions vertically as belonging to multiple levels, including higher level actions (e.g. , Yahya creates videos) and lower level actions (e. g., Yahya downloads music and scans photos as he designs the slides that make up the videos). In taking this approach, I keep complexities alive while simultaneously marking the terms of use clearly. ...
... Comparing both bodies of discourse illustrates that identities are complex, for the enlightener identity requires the construction of two different local identities, depending upon who is talking and whom is receiving the message. Norris (2004Norris ( , 2011 Similarly notes this finding in her analysis of the multimodal interactions of two German women. This demonstrates the necessity of conducting intertextual and longitudinal analysis of identity construction if one is to understand the complex nature of identities. ...
... Con base en estos elementos, el análisis del discurso político multimodal puede ayudar a comprender los procesos de significación que se gestan en los procesos electorales en México. aproxImacIón tEórIca la semiótica social, el análisis del discuRso multimodal y la nueva RetóRica En el campo del análisis del discurso multimodal (ADM), las aportaciones recientes (Jewitt, Bezemer y O'Halloran, 2016;Leeuwen, 2005;Machin, 2007;Norris, 2012) han evidenciado la necesidad de considerar el nivel de la composición y la unidad textual, en relación con el marco comunicativo en que las manifestaciones discursivas se realizan (Fontanille, 2001(Fontanille, [1998; Renkema, 1999Renkema, [1993; Kress y Leeuwen, 2001). ...
... El ADM (planificación de proyectos) ha generado acercamientos teóricos y aplicaciones metodológicas para su estudio; comprende una variedad de manifestaciones discursivas en que participan varias modalidades del lenguaje (Jewitt, Bezemer y O'Halloran, 2016;Leeuwen, 2005;Machin, 2007;Norris, 2012). El análisis propuesto retoma el enfoque de Leeuwen (2005), quien considera cuatro aspectos centrales para la determinación de los niveles de cohesión semiótica: 1) ritmo, 2) composición, 3) información vinculada y 4) diálogo. ...
The predominant form of advertising during Mexico’s 2015 mid-term elections was the use of television spots. This study analyzes the construction of the multimodal discourse focusing on persuasion in the pre-campaign spots aired by left Mexican parties. Using contributions from social semiotics, multimodal discourse analysis, and the new rhetoric, the author identifies the axiological content prevalent in visual broadcast publicity as a function of democratic values and symbolic-discursive positioning. The research results propose classifying the spots based on the oppositional relations of their content in the framework of the federal elections political communication.
... For example, we speak of 'creating a video' or of 'changing a law', but typically do not speak of creating videos as taking a sequence of steps. Such a problem could be momentarily resolved by following Lemke (2000aLemke ( , 2000b and Norris (2002Norris ( , 2004Norris ( , 2011, who construe actions vertically as belonging to multiple levels, including higher level actions (e.g. , Yahya creates videos) and lower level actions (e. g., Yahya downloads music and scans photos as he designs the slides that make up the videos). In taking this approach, I keep complexities alive while simultaneously marking the terms of use clearly. ...
... Comparing both bodies of discourse illustrates that identities are complex, for the enlightener identity requires the construction of two different local identities, depending upon who is talking and whom is receiving the message. Norris (2004Norris ( , 2011 Similarly notes this finding in her analysis of the multimodal interactions of two German women. This demonstrates the necessity of conducting intertextual and longitudinal analysis of identity construction if one is to understand the complex nature of identities. ...
... Kress (2003, p. 35) claimed that "language alone cannot give us access to the meanings of the multimodally constituted messages" and emphasized the importance of utilizing semiotic resources beyond the use of language. Multimodality combines different kinds of meaning-making into "an integrated, multimodal whole" (Jewitt et al., 2016, p. 2), such as in the fusion of various communicative modes like gestures, gaze, facial expression, and head movements to enrich the meaning of the spoken discourse (Kress & van Leeuwen, 2001;Norris, 2012;van Leeuwen & Kress, 2011). According to McNeill (1992, p. 11), language in its written and spoken forms is a "linear" code, while communicative modes like gesture are "multi-dimensional". ...
... In performances, which are multimodal in terms of having various signifiers of island meaning (e.g. written, visual, oral, and musical - Norris, 2012), the centre of the nō stage has a prop in the shape of a tall shrine. This prop also represents the boat the Emperor and his fellow travellers use to travel to the island. ...
The uninhabited yet socially active and culturally important Japanese island of Chikubu (Chikubu-shima) is situated towards the northern end of Lake Biwa in the Kansai region of Japan’s largest island of Honshū. Chikubu Island is linked to Shintō and Buddhist ritualistic culture and hosts tens of thousands of day-tripper pilgrims who travel there each year. But its cultural significance is also carried beyond its aquatic margins through multimodal signification in Japanese traditional performing arts where meaning connected with the island is portrayed through visual and sonic media. Extending discourse on islands and performing arts, this article shows how one culturally noteworthy Japanese island is imagined within creative practices and how island meaning is embodied in settings that are far removed from the island’s physical or lake environment. Expanding the scope of Island Studies, select creative works are discussed in terms of how they represent Chikubu Island through sound and symbol.
... The different methodological approaches share the foundational assumption that humans communicate and make meaning using a complex configuration of multiple modes (Jewitt 2017). The variety of methodological approaches differ mainly in units of analysis (Norris 2012). ...
The contexts and methods for communicating in healthcare and health professions education (HPE) profoundly affect how we understand information, relate to others, and construct our identities. Multimodal analysis provides a method for exploring how we communicate using multiple modes—e.g., language, gestures, images—in concert with each other and within specific contexts. In this paper, we demonstrate how multimodal analysis helps us investigate the ways our communication practices shape healthcare and HPE. We provide an overview of the theoretical underpinnings, traditions, and methodologies of multimodal analysis. Then, we illustrate how to design and conduct a study using one particular approach to multimodal analysis, multimodal (inter)action analysis, using examples from a study focused on clinical reasoning and patient documentation. Finally, we suggest how multimodal analysis can be used to address a variety of HPE topics and contexts, highlighting the unique contributions multimodal analysis can offer to our field.
... Begreberne 'multimodalitet' og 'interaktionsanalyse' daekker over en raekke markant forskellige videnskabelige og metodiske tilgange. Af sådanne tilgange kan fx naevnes 'interaction analysis' (Jordan og Henderson 1995), 'nexus analysis' (Norris 2004(Norris , 2011Scollon og Scollon 2003), 'multimodal communication analysis' (Paggio og Navarretta 2017) og ikke mindst den mest udbredte socialsemiotiske tradition (Böck og Pachler 2013;Hodge og Kress 2007;Kress 2001Kress , 2009). Ordet multimodalitet er en sammensaetning af morfemet multi-, der betyder 'mange', og modalitet, der lidt frit kan oversaettes til 'måde'. ...
Vi præsenterer i denne artikel en systematisk diskussion af, hvad multimodalitet er, samt hvordan multimodale interaktionsanalyser kan udføres ud fra et etnometodologisk og konversationsanalytisk (EMCA) perspektiv. Vi introducerer til de grundlæggende metodologiske antagelser og viser igennem tre eksemplariske analyser, hvilke temaer, problemstillinger og undersøgelsesområder analytikere kan fokusere på, hvordan de i praksis kan udføres, samt hvilke typer fund og resultater disse analyser kan generere. Et klassisk tema i metoden er intersubjektivitet; hvordan deltagere orienterer sig mod hinanden og skaber forståelse og mening. Det sker blandt andet ved kropsligt at søge den andens ansigt og øjne for at sikre modtagelse og respons. Artiklen giver både en anvendelsesorienteret introduktion til feltet og præsenterer helt ny forskning inden for tre udvalgte områder. De analyserede eksempler er fra forskellige interaktionssituationer mellem henholdsvis kursister og en service-robot, kolleger i et kontorlandskab samt mellem en borger og en bostøttemedarbejder. Hver især præsenterer eksemplerne, hvilke ressourcer deltagere anvender for at sikre den andens opmærksomhed. Dermed bidrager artiklen med ny viden om den multimodale semiotiske økologi, hvor talende kroppe interagerer i en materiel verden. Artiklen afsluttes med perspektiver på nye forskningsområder.
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... 5, we propose a theoretical model that refines and extends the grammars that have so far been proposed for description and analysis of gesture (cf. Martinec 2000Martinec , 2001; for a complementary activity theory-based perspective on multimodal discourse analysis, see Norris (2012)). ...
This book analyses the Youth Justice Conferencing Program in New South Wales, Australia. Exploring this form of diversionary justice from the perspectives of functional linguistics and performance studies, the authors combine close textual analysis with ethnographic research methodologies. They examine how participants use the discourse semantic resources available to them to achieve such outcomes as reparation for the victim, reintegration of the offender into the community, and reconciliation between the various parties. This uniquely-researched work is sure to be of interest to students and scholars of applied linguistics, sociolinguistics and discourse analysis.
Michele Zappavigna is a senior lecturer in the School of Arts and Media at the University of New South Wales, Australia. Her major research interest is the discourse of social media, and she has published widely on this topic in a range of books and journals.
JR Martin is Professor of Linguistics at the University of Sydney, Australia. His research interests include systemic theory, functional grammar, discourse semantics, register, genre, multimodality and critical discourse analysis, focusing on English and Tagalog. He was elected a Fellow of the Australian Academy of the Humanities in 1998, and awarded a Centenary Medal for his services to Linguistics and Philology in 2003.
... 5, we propose a theoretical model that refines and extends the grammars that have so far been proposed for description and analysis of gesture (cf. Martinec 2000Martinec , 2001; for a complementary activity theory-based perspective on multimodal discourse analysis, see Norris (2012)). ...
This chapter introduces the approach to restorative justice, which is the focus of this monograph—namely Youth Justice Conferencing in New South Wales, Australia. In Youth Justice Conferences, adolescent offenders meet with their victim and other relevant members of the community to discuss relatively minor offences and work out some form of community service by way of reparation (instead of going to court, getting a criminal record and possibly serving time in juvenile detention). The specific conferences which were the focus of this research are introduced, alongside the model of language and semiosis used for analysis, Systemic Functional Linguistics (SFL). The chapter ends with an outline of the topics to be considered: genre, exchange structure, appraisal, body language, identity and ceremony.
... L. O'Halloran & Smith, 4 2011). It highlights the involvement of semiotic resources which help the meaning-making process, the experiences and learning becomes sensible (Kress & Leeuwen, 2001;Jewitt & Kress, 2003;Norris, 2011). In practice, learning resources are barely unimodal, they are multimodal represented in texts along with color, audio-visual, hyperlink, interactive navigation, etc. (Kress & Leeuwen, 2001). ...
... The multimodal turn is also linked to the abovementioned shift (culminating in translanguaging theory) that has been going on for several decades in the way we view language as processes of meaning-making rather than as enclosed systems. Jewitt (2008Jewitt ( , 2009) and Norris (2012) have considered the similarities and differences between different approaches to multimodality, as well as the underlying theoretical backgrounds of each approach. The following three fields of application (Jewitt, 2008) can be recognised: (1). the semiotic approach linked to Halliday's social semiotic theory, (e.g. ...
This paper presents a critical examination of key concepts in the study of (signed and spoken) language and multimodality. It shows how shifts in conceptual understandings of language use, moving from bilingualism to multilingualism and (trans)languaging, have resulted in the revitalisation of the concept of language repertoires. We discuss key assumptions and analytical developments that have shaped the sociolinguistic study of signed and spoken language multilingualism as separate from different strands of multimodality studies. In most multimodality studies, researchers focus on participants using one named spoken language within broader embodied human action. Thus while attending to multimodal communication, they do not attend to multilingual communication. In translanguaging studies the opposite has happened: scholars have attended to multilingual communication without really paying attention to multimodality and simultaneity, and hierarchies within the simultaneous combination of resources. The (socio)linguistics of sign language has paid attention to multimodality but only very recently have started to focus on multilingual contexts where multiple sign and/or multiple spoken languages are used. There is currently little transaction between these areas of research. We argue that the lens of semiotic repertoires enables synergies to be identified and provides a holistic focus on action that is both multilingual and multimodal.
... La finalidad de las investigaciones sobre el discurso multimodal en Educación Superior tiene como meta la clarificación del uso y las diferentes fuentes en un texto multimodal, atendiendo a la singularidad del contexto en el que se desarrolla la comunicación. Las numerosas investigaciones realizadas sobre los textos multimodales han puesto de manifiesto prácticas alfabetizadoras en diversas tecnologías con un carácter general (Belfiore et al., 2003; Ferding & Pytash, 2014; Knobel & Lankshear, 2007; Lankshear & Knobel, 2013; Merchant et al., 2013; O'Halloran & Smith, 2011; Van Waes, Leijten & Neuwirth, 2006, entre otros), estudios sobre cómo interaccionan distintas fuentes (escritura e imagen, especialmente) en una página o pantalla de un libro, una página web, u otras fuentes digitales (Guo, 2004; Jewitt & Kress, 2008; Bezemer & Kress, 2008; Norris, 2012; O'Halloran & Smith, 2011; Unsworth, 2001 ), y cómo dichos textos multimodales caracterizan determinados géneros discursivos (Knox, 2007;). Esto último es uno de los objetivos fundamentales del análisis de los textos multimodales, pues en la Educación Superior se requiere el desarrollo de las competencias informacionales y digitales con el fin de acceder a una comunidad discursiva académica (Swales, 1990 ); es decir, el alumno debe desarrollar los conocimientos metadiscursivos que le permitan comprender y construir textos multimodales (Baldry, & Thibault, 2010) en un contexto cultural y de situación universitarios y en el entorno de una disciplina concreta (Fairclough, 1989; Lillis, 2001 ). ...
... With its emphasis on the complexity of social life, its methodical impartiality and acceptance of different types of data, an ethnography or, as Swales called it, a " textography " (1998, p. 1) of discourse-in-practice, should thus suggest itself as an analytical amplification (e.g. Macgilchrist and Van Hout, 2011; Norris, 2011; Scollon and Scollon, 2004 ). Such ethnographic analysis could especially help to untangle the trajectories between interactional patterns and types of signification by looking at what Giddens has named a structurating relation between the accomplishment of discursive practices and the coextensive enactment of discursive structures (1984, p. 25). ...
This article explores a way to reconstruct the verbally and visually constituted frames used in the coverage of the trial of John/Ivan Demjanjuk, a Ukraine-born U.S. citizen accused of holocaustrelated war crimes. The study looks at an exemplary case of current multimodal discourse, in which written messages and images from broadcasts and press, as well as the comments and visuals that spread through social media, can be seen to relate to each other in framing public issues. To establish a viable perspective that takes into account both the communicative organisation and the semiotic constitution of such discourses, this analysis combines approaches from frame semantics and social semiotics together with recursive sampling and coding. The article then explains the analytical procedures used to reconstruct the framing of the accused as either a responsible culprit or a victim of circumstances.
Visual politics is a fast-growing field and much of it is focused on images that inspire criticism. This tendency results in a lack of attention to oppressive visual practices. A political visual literacy approaches all visual practices as being layered with different “visual truths” that were developed in response to political commitments over time. These “visual truths” inflected visual practices in ways that may outlive the political settings in which they were first articulated. Most important of these is the desire to capture human difference that framed the development of visual technologies and is still embedded in a range of visual practices. The methodology I develop here links a conjunctural analysis of visual tools and practices and the visual truths implicated in them with their operationalisation by actors as cultural tools through the framework of mediated action. I develop this approach by interrogating two layered and harmful visual practices: the White-centrism of visual technologies and the racialised origins of transphobic visualities in automatic gender recognition technology.
Language textbooks mainly guide language learning and teaching activities. Most of the language textbooks used to comprise a range of visual texts, such as pictures, illustrations, and photos. The study adopts the Visual Grammar Theory by Kress and Leeuwen (2006) to elucidate the pedagogical functions of visual images and explore how such images can be exploited for learning tasks from a micro multimodal perspective. The data consist of 142 visual images in Indonesian senior high school EFL textbook grades ten (X) and eleven (XI). The findings pointed out that the textbook uses visual images' full potential to fulfill pedagogical aims. Many visual images or texts in language textbooks serve information and illustrations rather than as a decorative function. Visual images may assist students to engage effectively in learning tasks by emphasizing the meaning of information presented in images and text. This study suggests that learning activities should consider multimodal texts to contribute significantly. This research aims to enhance knowledge about the pedagogical function of visual images in textbooks and can be a reference for further exploration.
Military recruitment advertising, as a type of authoritative political video advertisement, demonstrates a country’s military culture. This paper will provide a brand-new perspective in researching Chinese military culture from the approach of multimodal metaphor. Currently, a multimodal analysis of military recruitment advertising in terms of short video clips does not exist, and most of the existing multimodal studies have focused on Western mainstream media, whereas media with Chinese characteristics have remained mostly untouched by research. With Forceville and Urios-Aparisi’s (2009) Multimodal Metaphor Theory (MMT) as a theoretical basis, and the newly-released recruitment advertising “The Power of China” as the research object, this paper utilizes the MIPVU and ELAN image tagging software to identify and analyze the multimodal metaphors in “The Power of China” under the framework of MMT.
Digital media tasks are increasingly used in university settings to assess students’ scientific knowledge. The use of such tasks has generated the need for principled analytical approaches to better understand how meaning-making operates in these artefacts. However, there has been a proliferation of approaches to multimodality in recent years and as a result deciding which tools to use is challenging for analysts and for subsequent translation into pedagogy. Systemic functional approaches to multimodal phenomena have proved useful as we study the digital artefacts produced in tertiary science classrooms. These approaches are grounded in Halliday’s (1994) social-semiotic theories of language and influenced by Kress and van Leeuwen’s (2006) visual grammar. Here, we present insights from theoretical and methodological developments in systemic functional approaches to multimodal phenomena. We illustrate these insights with a student-generated digital media artefact, revealing the affordances of a systemic functional approach and the perspectives it offers on semiosis.
Social semiotics is a social theory of meaning and communication modelled on Michael Halliday’s theories of language as social semiotic and Systemic Functional Grammar with a particular focus on the agency of social actors and social context. Chapter 1 provides a fully up-to-date introduction to the theory (while peculiar aspects of the theory will be dealt with in subsequent chapters when they become relevant for discussion and application to the data), and argues that social semiotics serves as a useful multidisciplinary framework for analysing text-image relations. With an aim to contributing to the existing monographs of empirical social semiotic analysis in the fields of education and humanities, this chapter argues for focussing the analytical lens on visual analysis in other major areas of impactful social research which could provide a nuanced view of semiotic resources and principles pertaining to visual imagery and text as the most prominent modes of communication in contemporary society before identifying and describing the resources and principles that operate within and across other modes.
While the linguistic model is necessary for examining the semiotic aspect of journalistic stance it has yet provided a rather partial view of journalistic stance taking practices.
This chapter suggests that conference theorists and advocates have been looking in the wrong place when trying to find a way of explaining the transformation that occurs in a conference. Instead of a theory based on personal internal emotional states, the chapter argues for a social semiotic perspective that accounts for the interactive power of the macro-genre—an approach that deals with the way conferences draw on shared cultural resources to realign the YP with the positive values of particular communities. Anthropological work on ceremony and Systemic Functional Linguistic work on iconization are brought together in an interpretation of conferencing as a powerful form of ceremonial redress.
This paper focuses on how signing students organise themselves spatially in social interactions in a university lecture hall. One may view space as a concrete location, a social construct, and a normative actor with historical trajectories. The study addresses the question, 'What are the mediated actions through which the students and teacher (re)configure space for participating in a class?' Following a methodological framework of Mediated Discourse Analysis and multimodal interaction analysis, I approach this question by examining the social actions occurring when entering a lecture hall. The primary data includes video recordings, photos, and participatory observations, documented by field notes. The analysis shows how the architectural specifications of a space pose restrictions on visual-embodied interactions. However, the participants configure and reconfigure the space to some extent to suit visual-embodied interaction through explicit and implicit negotiation.
We are interested in novel interactive uses of pressure sensors and vibration actuators that can augment the role of physicality for embodied human perception and experience. Specifically, we explore how wearable technology can be used to provide more realistic present-at-body self-awareness in equestrians. Self-awareness of a rider's own physical cues (output) and how a horse responds (input) requires practice to attain objective adjustment. In this paper we present a proof of concept prototype aimed at providing ways to bridge the gap between rider output perception and reality. Our prototype couples pressure data gathered at specific points of the body in real-time with non-audiovisual tactile vibration feedback that is also site-specific. Our design is intended to enable an effective way for riders to learn about asymmetries in seat-related pressure by providing a present-at-body self-awareness of pressure points.
The purpose of this essay is to address the multimodal nature of Matt Kish’s project Every Page of Moby-Dick, Illustrated, where Herman Melville’s 1851 masterpiece is set as paratext. Particular focus is set on the portrayals of Captain Achab, specifically “Page 153”, “Page 465” and “Page 469”. The basic theoretical framework has been offered by Alice Gibbons’ theorization of multimodal cognitive poetics and Sigrid Norris’ systematization of multimodal (inter)action. Useful insight has been lent by Sharon Cameron’s work on allegories of the body in Melville’s writing. The given analysis aims to pinpoint the elements of innovation in Kish’s work with respect to the canonical formal features of illustrated book and renewing of literary classics in multimodal terms.
Multimodality is an analytical framework created to examine complex discourses that draw upon or consist of various visual and textual modes; the framework, according to Iedema (2003), gained ground recently due to the salience of multimodal texts brought about mainly by the advent of new media technologies and a recent turn in linguistics toward viewing communication as multimodal.Keywords:agency;art and technology;communication;disability;multimodality;semotic modes
Multimodal interaction analysis (Norris, 2004) is a holistic methodological framework that allows the analyst to integrate the verbal with the nonverbal, and to integrate these with material objects and the environment as they are being used by individuals acting and interacting in the world.Keywords:discourse analysis;pragmatics;research methods in applied linguistics;sociolinguistics
This discussion challenges conventional marketing theory regarding posters and billboards in the twenty first century. It argues against the prevailing dogmatic statement that a poster or billboard must communicate its message in mere seconds (Drewniarny and Jewler 2011). It also points out that convergence of traditional media with new communication technologies is challenging the notion that posters and billboards are still
In contrast, it proposes that assessing how people interact with billboards and posters from a mediated and multimodal discourse perspective is more useful. This involves looking at posters and billboards when they are actually in use rather than considering them only as they are present in the environment. The modal density foreground-background continuum (Norris 2004), the concept of a site of attention (Jones 2005) and the idea of the communicative space (White 2012) are multimodal discourse methodology tools that prove particularly illuminating in this context. While no empirical data is analysed as part of this discussion, mediated and multimodal discourse analysis of two specific examples from previous studies serves to illustrate the points made here.
ResearchGate has not been able to resolve any references for this publication.