No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Modal density and modal configurations: Multimodal actions
Abstract
Multimodality is an innovative approach to representation, communication and interaction which looks beyond language to investigate the multitude of ways we communicate: through images, sound and music to gestures, body posture and the use of space.
The Routledge Handbook of Multimodal Analysis is the first comprehensive ‘research tool kit’ for multimodal analysis, with 22 chapters written by leading figures in the field on a wide range of theoretical and methodological issues. It clarifies terms and concepts, synthesizes the key literature with in-depth exploration and illustrative analysis, and tackles challenging methodological issues.
The Handbook includes chapters on key factors for Multimodality such as technology, culture, notions of identity and macro issues such as literacy policy. The handbook takes a broad look at multimodality and engages with how a variety of other theoretical approaches have looked at multimodal communication and representation, including visual studies, anthropology, conversation analysis, socio-cultural theory, socio-linguistics and new literacy studies.
Detailed multimodal analysis case studies are also included, along with an extensive glossary of key terms, to support those new to multimodality and allow those already engaged in multimodal research to explore the fundamentals further.
The Routledge Handbook of Multimodal Analysis is essential reading for undergraduate and postgraduate students as well as researchers involved in the study of multimodal communication.
... Furthermore, a discourse of science is multimodal in which various modes co-operate together to realize meaning by contextualizing the meaning deployed by each mode (Liu & Owyong, 2011;O'Halloran, 2007). However, the effects of multimodal texts and the density of modes (Norris, 2011) meaning-making and learning still need to be studied. This research focuses on the multisemiotic construction of scientific knowledge as part of meaning-making practices in the primary-level science classroom. ...
... The concept of mode density or modal density (Norris, 2011) is introduced as a parameter of the use of different modes, which have the best affordances to demonstrate meaning types and experiential meaning types in a science text. ...
Numerous studies demonstrated that the meaning-making of scientific knowledge is affected by the design of multimodal science texts. Various modes are co-operated together in certain inter-semiotic mechanisms to produce meaning in multimodal texts. Based on this perspective, this research seeks to investigate the effect of mode level in science texts and compositional arrangement on the meaning-making of science concepts and processes. In this context, four science texts with the same content (transformation of energy) at different mode densities and two science texts with the same content (covalent bonding) one of which is arranged in accordance with variation theory of learning are designed. By using the case study method, this research explored six experienced science teachers’ views about the effects of mode level and multimodal text composition on meaning-making. The data were collected with semi-structured interviews. The thematic analysis was employed for data analysis. The findings demonstrated that mode density may affect meaning-making and so learning since different modes have affordance to represent different meaning and meaning relationship types. Besides, multimodal text composition may foreground the critical aspects of content, and help to design a coherent multimodal science text.
... Furthermore, a discourse of science is multimodal in which various modes co-operate together to realize meaning by contextualizing the meaning deployed by each mode (Liu & Owyong, 2011;O'Halloran, 2007). However, the effects of multimodal texts and the density of modes (Norris, 2011) meaning-making and learning still need to be studied. This research focuses on the multisemiotic construction of scientific knowledge as part of meaning-making practices in the primary-level science classroom. ...
... The concept of mode density or modal density (Norris, 2011) is introduced as a parameter of the use of different modes, which have the best affordances to demonstrate meaning types and experiential meaning types in a science text. ...
Numerous studies demonstrated that the meaning-making of scientific knowledge is affected by the design of multimodal science texts. Various modes are cooperated together in certain inter-semiotic mechanisms to produce meaning in multimodal texts. Based on this perspective, this research seeks to investigate the effect of mode level in science texts and compositional arrangement on the meaning-making of science concepts and processes. In this context, four science texts with the same content (transformation of energy) at different mode densities and two science texts with the same content (covalent bonding) one of which is arranged in accordance with variation theory of learning are designed. By using the case study method, this research explored six experienced science teachers' views about the effects of mode level and multimodal text composition on meaning-making. The data were collected with semi-structured interviews. The thematic analysis was employed for data analysis. The findings demonstrated that mode density may affect meaning-making and so learning since different modes have affordance to represent different meaning and meaning relationship types. Besides, multimodal text composition may foreground the critical aspects of content, and help to design a coherent multimodal science text.
... que fazem parte de seu potencial semiótico(KRESS, 2010).Com relação à segunda vertente teórica (a análise multimodal interacional), retomamos, sob essa perspectiva, o conceito de modo, que consiste em um sistema de ação mediada, fruto de usos sociais, históricos e culturais, que se vincula fortemente ao contexto de produção em decorrência de nossa atuação no mundo(JEWITT (2011(JEWITT ( [2009NORRIS, 2013). Tal sistema prevê ações menores (nível mais baixo) e ações maiores (nível mais alto)(NORRIS, 2004(NORRIS, , 2006(NORRIS, [2009), que poderiam ser representados, por exemplo, por uma interação face a face (nível mais alto), em que estão presentes gestos, mudanças posturais, enunciados(NORRIS, [2009). Os conceitos de densidade modal (NORRIS, 2011[2009]) e de configuração modal (NORRIS, 2011 [2009], 2011) são basilares para essa vertente. ...
... que fazem parte de seu potencial semiótico(KRESS, 2010).Com relação à segunda vertente teórica (a análise multimodal interacional), retomamos, sob essa perspectiva, o conceito de modo, que consiste em um sistema de ação mediada, fruto de usos sociais, históricos e culturais, que se vincula fortemente ao contexto de produção em decorrência de nossa atuação no mundo(JEWITT (2011(JEWITT ( [2009NORRIS, 2013). Tal sistema prevê ações menores (nível mais baixo) e ações maiores (nível mais alto)(NORRIS, 2004(NORRIS, , 2006(NORRIS, [2009), que poderiam ser representados, por exemplo, por uma interação face a face (nível mais alto), em que estão presentes gestos, mudanças posturais, enunciados(NORRIS, [2009). Os conceitos de densidade modal (NORRIS, 2011[2009]) e de configuração modal (NORRIS, 2011 [2009], 2011) são basilares para essa vertente. ...
This production is another collective work resulting from
academic-scientific investigations of professors linked to the
International Scientific Research Project (PICNAB), network
which brings together researchers from the universities of
Nantes (France), Aveiro (Portugal) and Brasília (Brazil), and counts,
since 2016, with the support of the Research Support Foundation of the
Government of the Federal District (FAP-DF) and the CAPES-Print /
University of Brasilia. The Network's main objectives are to build artistic (literary and theatrical), linguistic research and pedagogical that envision new socio / intercultural practices before the challenges of contemporary reality, which has limits
tenuous, migratory movements and fluid identities. Thus,
researchers and researchers study issues concerning the
teacher training that will deal with the education of children
and young members of an educational system that needs to be
rethought and resized in their teaching practices, both in terms of
Brazil, as well as in Portugal and France, countries that experience the day
day of this new reality. In this sense, studies add not only
undergraduate students, but also graduate students (master's
and doctorate), linked to the programs of the three universities.
... Each frame of the visual transcription pictorially represents the lower-level action/s of analytical focus, coupled with the time signature of their unfolding. 1 While natural communicative activity appears propagated through multimodal configurations (Norris 2009(Norris , 2011(Norris , 2013a, there has been considerable contention regarding what exactly constitutes a communicative mode. Social semiotic perspectives champion the idea of semiosis whereby a mode is typically defined as a semiotic system with socio-culturally instantiated regularities attached to them. ...
... Thus, Sara addresses Isabel directly in a form of subordinate communicative sideplay (Goffman 1979) through the modes of gaze, physical touch and verbal language. Following Sara's alteration in attention exemplified through modal intensity and complexity (Norris 2009(Norris , 2011 in gaze, physical touch and direct verbal address, Sara undertakes another lowerlevel action through the mode of verbal language, requesting that her sister "show my tooth". ...
In this article, I detail incremental microgenetic alterations in the development of one particular socio-interactive aptitude: making a relevant interactive contribution. Taking heed of Clark’s (2014) call for the need to reorient our attention to investigate the pragmatics of interaction by accounting for the multiple communicative modes through which this is acccomplished I detail the ways in which parental facilitation and a flexible participatory configuration, made possible by video-conferencing technology, create conditions enabling the agentive re-introduction of a psycho-socially relevant topic. Paramount are the ways in which residual interactive specificities in introduction, co-production and multimodal configurations re-manifest suggesting a more symbiotic relationship between traditional notions of ‘message’ and ‘production’. During the microgenesis of interactive aptitudes, children are not just learning what constitutes psycho-socially relevant topoi, they also acquire an understanding of exactly how to make the contribution through multimodal ensembles.
... Because social action often involves more than one focused interaction, due to simultaneous engagements (e.g., carrying out a text message exchange while carrying out an oral conversation with someone else), modal density is another important and related subconcept of modality. The modal density denotes the level of complexity and intensity of a focused communicative action (Norris, 2011). For example, writing an email while talking to a colleague involves high-level actions characterized by high modal intensity. ...
... Another related key concept is modal configurations, which pertain to the hierarchical order among the modes at operation in a meaningful action. For example, in a dinnertime conversation, multiple modes of meaning operate simultaneously, but with superordinate, equal, or subordinate levels of meaning, such as superordinate spoken words combined with subordinate gestures, gaze, facial expressions, movements, tastes, spatial arrangements of the seating, food, and so on (Norris, 2011). Among listeners in a dinner conversation, the modal configurations take on differing meanings and the modes take on different levels of importance than they do for the speaker. ...
... Simultaneously, a methodological framework prioritizing mediation, given the inextricable effect of mediating technologies is directly congruent with the technological specificities of these new interaction types. While a complete explication of Multimodal Mediated Theory and Multimodal Interaction Analysis (Norris 2004(Norris , 2006(Norris , 2009(Norris , 2011(Norris , 2013Pirini 2014) is beyond the scope of a single article, below I detail some central methodological tools which are directly pertinent to the discussion which follows. ...
... Approaching the study of familial interaction through contemporary video-conferencing software and technologies from a Multimodal Mediated Theory perspective (Norris 2004(Norris , 2011(Norris , 2013(Norris , 2014Author 2013;Author et al. 2015); and employing Multimodal (Inter)action Analysis (MIA) (Norris 2004(Norris , 2006(Norris , 2009(Norris , 2011(Norris , 2013Pirini 2014) as a methodological framework has a number of distinct advantages. MIA provides an explicit framework for the analysis of verbal and non-verbal communication without overtly prioritising any single mode a priori (Norris 2004(Norris , 2011. ...
This article provides a preliminary answer to exactly why video-conferencing is evaluated as better than traditional telephony for long-distance familial interaction by allocating analytical attention to the showing of objects during interaction. While it is acknowledged that ‘showing’ constitutes an interactive move less contingent on linguistic maturation, more importantly, the showing of objects, artefacts or entities during video-conferencing interactions exemplifies an agentive and volitional production of identity elements on behalf of young children. Thus, while some have pointed to shortcomings of conversation-like activities mediated by video-conferencing in favour for more activity-driven tasks, I make a case for drawing upon pre-existing components of the material surround as a means to more comprehensively and longitudinally engage younger children in video-conferencing interaction.
... Starting with this framework, the design of the fine-grained video analyses has been generated on the acknowledgement that "transcription 2 and analysis go hand in hand" (Norris, 2019, p. 199) when it comes to working with multimodal data. The videos have been transcribed at the level of frame by employing multimodal transcription conventions (Baldry & Thibault, 2006;Norris, 2011;2019) in two phases. In the first phase, in the table that had been created for each video, each shotnamely each core multimodal data unit -was identified and annotated without prioritising any of the co-deployed semiotic modes. ...
... Once we have produced transcripts of pertinent excerpts from our video data, we engage with methodological tools that are relevant for the data pieces such as modal density (Norris 2004a), modal configuration (Norris 2009a), the foregroundbackground continuum of attention/awareness (Norris 2004a;2008), semantic/pragmatic means (Norris 2004a), levels of action (Norris 2009b), scales of action (Norris 2017b), agency (Norris 2005;Pirini, 2017), or the site of engagement (Scollon 1998(Scollon , 2001Norris, 2004aNorris, , 2019Norris, , 2020Norris and Jones 2005). Here again, we rely on audio-visual technology without, however, favouring any one kind. ...
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... As with lower-level actions, higher-level actions can be analysed for the meaning they produce. Using the notion of modal configurations (Norris, 2009), the relative contribution of lower-level actions to the meaning produced through the higher-level action can be determined. Modal configurations refer to the hierarchical configuration of modes within a particular higher-level action. ...
Multimodal (Inter)action Analysis was developed to study social interaction based upon the theoretical notion of mediated action. Building on this core concept, Multimodal (Inter)action Analysis includes several theoretical/methodological tools. These tools facilitate analysis which moves flexibly between micro-level moments of interaction and macro-level practices and discourses. In this chapter, the application of mediated action to multimodal analysis is discussed, before the central theoretical/methodological tools are introduced. Tight links are made between the tools used in Multimodal (Inter)action Analysis and the core theoretical tenets, to support robust multimodal interaction research.
... A central principle of multimodality is that of mediated action, which refers to the mediated nature of social action. The social actions should be construed as happening between social actors and/or between the actor and the surroundings, which means that they can't be separated from the context they appear in Jewitt (2009) and Norris (2009). This interaction can be defined as "any action that a social actor performs in which the actor communicates a message. ...
Even though construction tasks have a long history as an activity in the Swedish preschool, technology as a content matter (e.g., construction) is relatively new. Hence, preschool teachers are generally unsure of the content of technology and how to handle it from a teaching perspective. Thus, there is need for deeper understanding of how construction tasks in preschool can be enacted and what kind of premises are offered to the children. To investigate this, we took our stance in activity theory and the concepts of mediating artifacts, rules and division of labour. This helped us discern what type of instructional practices that were enacted by preschool teachers when working with construction tasks. Activity theory in combination with thematic analysis helped us distinguish four general didactic actions that the teachers used to bring about the construction task—to engage, to guide, to coordinate, to show. These four strategies were then formulated into specific technology didactic actions through the perspectives of technology as product, process and concepts.
... Multimodal interactional analysis was chosen since it focuses on the actions taken by a social actor through mediational means, i.e. how a variety of modes are added and become constitutive of social interaction, identities and relations (Jewitt, 2011). Modes can never exist without social actors utilizing them in some way or other (Norris, 2011). Interactional analysis also put an emphasis on the notion of context and mediational means used for action which also interact in the situations studied (Jewitt, 2011). ...
The aim of this study is to investigate children’s out-of-school learning in digital gaming communities. This was achieved by exploring girls’ participation in Minecraft communities. Data were generated through interviews, video-recorded play sessions and video-stimulated recall. Multimodal interactional analysis was applied in order to analyze children’s mediated actions. The components of Wenger’s Social Theory of Learning were used as a basis when exploring learning in children’s out-of-school digital gaming communities. Five significant themes of what characterizes learning in digital gaming communities were identified: learning through experiencing, learning through belonging, learning through performing, learning through struggling and learning through enacting participatory identities. The main findings are presented in a tentative conceptual framework that can support teachers, school leaders and policymakers who are interested in connecting children’s out-of-school learning experiences with their learning in school.
... That rhythm of a spoken language performed by one social actor could be well responded to by using hand-arm movements by another social actor. Thus, a spoken utterance is rhythmically responded through a set of gestures (Norris, 2009). ...
... Multimodality may be viewed as the vehicle for students to design and orchestrate their own modes of learning that are meaningful to them in the form of multimodal ensembles and semiotic resource [35] . Such multimodal ensembles may include images and language along with more static modalities resembling frozen actions [36] such as classroom objects and equipment incorporating desks, tables, displays, chairs, books and chalkboards. There is a plethora of evidence on how multimodality may be deployed in traditional learning environments [37][38][39] , in blended learning spaces [40][41][42] and online learning spaces [43][44][45] . ...
It is becoming increasingly prevalent in digital learning research to encompass an array of different meanings, spaces, processes, and teaching strategies for discerning a global perspective on constructing the student learning experience. Multimodality is an emergent phenomenon that may influence how digital learning is designed, especially when employed in highly interactive and immersive learning environments such as Virtual Reality (VR). VR environments may aid students' efforts to be active learners through consciously attending to, and reflecting on, critique leveraging reflexivity and novel meaning-making most likely to lead to a conceptual change. This paper employs eleven industrial case-studies to highlight the application of multimodal VR-based teaching and training as a pedagogically rich strategy that may be designed, mapped and visualized through distinct VR-design elements and features. The outcomes of the use cases contribute to discern in-VR multimodal teaching as an emerging discourse that couples system design-based paradigms with embodied, situated and reflective praxis in spatial, emotional and temporal VR learning environments.
... Through multiple semiotic resources such as gaze and gesture, people show each other the relevant material, artefacts, and other semantic content important in the interaction (Goodwin, 2000). A mutual orientation is crucial for creating a framework, a modal configuration, in which different semiotic resources can flourish (Goodwin, 2011;Norris, 2009). From the perspective of using pictures in communication -on paper or mobile devices -it is necessary that the physical place with its material objects and architectural layout offers possibilities for line-of-sight to the pictures and between participants (Tapio, 2018). ...
The paper reports the initial findings of the first phase of the research and development project PICCORE – Picture Communication in Reception Centres. The goal was to map the use of pictures and other visual modes of communication at reception centres in Finland using an ethnographic, multimodal research approach. The ethnographic data was collected at four reception centres in Finland. A multimodal viewpoint draws attention to how action and meanings are mediated through pictures. The initial findings mark established practices for enabling and coordinating mutual attention, supporting the use of visual and embodied resources in interactions and – as a consequence – supporting mutual understanding.
... The images on the other hand, may be used for "showing" the reader what is happening and make use of size, layout, shape, and colour. Modes do not stand alone within a text, the relationships between modes may themselves realise meanings through particular modal combinations, different weightings of modes (Martinec & Salway, 2005) or modal density in an ensemble (Norris, 2009). Colour, size and shape are not limited solely to images and may be employed through the use of font and typography to enhance and give greater meaning to the written content within an ensemble. ...
This study adopts a participatory visual mapping methodology to capture young children's interactions with, and comprehension of the visual mode in their familiar environments. Adopting a social semiotics theoretical framework, this study is designed around two main aims: firstly, to explore the ways young children conceptualise images in their environment and secondly to identify how visual mapping, as a methodological approach, can be used to capture young children's comprehension of, and interactions with the visual mode. The group of 4 to 5-year-old children involved in the study, are posed as social and active meaning makers with a variety of multimodal engagements, incorporating the visual mode. As communicative practices are argued to be understood in context with surrounding cultural and social practices, the data for this study were collected in three key locations seen as familiar to the children involved: the home environment, Early Years setting, and a community leisure centre. In each setting the children captured examples of the visual mode through digital photography, before creating two dimensional maps of their setting with the printed images. Conversations between the researcher and child were recorded and transcribed, to draw out the meanings expressed by the children. Thematic analysis was used to identify key themes. The mapping technique enabled the children to be positioned as both message creators, through the production of their visual maps, and message receivers as they sought to make meaning of the multimodal texts they encountered within their environment. The use of a mapping activity supported identification of the children's knowledge of the codes and conventions of the visual mode which may not so easily be put into words. The study revealed that, for children, the context and location of images are important, with the presence of images and artefacts enabling familiarity with a place. Furthermore, movement was identified as an intrinsic part of their multimodal engagements. This study contributes to the developing body of participatory visual research methods. The use of mapped representations allowed children's perceptions of the spatial and embodied aspects of meaning making to be foregrounded. It is now imperative that these perceptions are recognised and supported within the Early Years setting in order to create an environment which not only reflects children's prior experiences but the shifting communication practices of modern society.
... These studies foreground that children's out-ofschool activities seem to involve a lot of multimodal doing (Gee 2007) while school practices still revolve around linguistically dense printed-pages (Jewitt 2008;Serafini 2014). In this respect, traditional school practices-which became dominant during the industrial age and for industrialization purposes-are not able to account for the new ways of making meanings, the new multimodal ensembles and the higher multimodal density in text production in post-industrial societies (Matthiessen 2014;Norris 2011). In this chapter as well as in the remainder of this book I will refer to these traditional schooling practices as traditional ideologies of learning. ...
Canale introduces learning from a multimodal socio-semiotic perspective, arguing this has profound theoretical and ethical implications. In light of this, Canale explores some of the many connections between technology, meaning-making and multimodality in the understanding that new technology plays a fundamental role in current formal education and that the interaction, communication and learning that take place with such technology in schools needs to be approached socio-semiotically and multimodally. Drawing on classroom examples, he discusses the benefits of approaching learning from this multimodal socio-semiotic perspective as an alternative to long-ingrained ideologies of learning which permeate many school and classroom practices and which obscure much of the meaning-making and learning that takes place inside (and outside) the school walls.
... The last page (not shown) contains the "10 tips to stay ahead in the bedroom" and reminds the reader that a woman's sexual health is "a fundamental human right". The 'stories within a story' recalls Norris's (2011) concept of a 'higher-level' action, where actions such as conversations are embedded into other structures of activity such as a family dinner. High-level actions are complex and analysis can only be done by looking at the mediational means. ...
... Infographics are clusters with high 'modal density' (Norris, 2014), where visual resources, such as shape, color and symbols, interact with writing, and the relationship is complementary. Based on conventions, they convey specific information. ...
What role does multimodality play in assessment in the English subject in Norway? This article focuses on final written examinations from 2014 to 2018 and investigates the multimodal literacy skills that examinations invite lower secondary school students to demonstrate. Examinations in the English subject are digital and technically open to a rich multimodal practice. Analysis in this article finds that the texts to be read in examinations are carefully designed multimodal texts, with plentiful use of visual aspects of writing and with images
that add significantly to the creation of complex cohesive ensembles. When it comes to the examination tasks, however, the opportunity for the students’ multimodal output is limited and ambiguous. In sum, there is an imbalance between input and output.
... Seeing that Dương seems to be convinced by that reason (i.e., for the mother's sake), Kỉnh takes this opportunity to make another request, this time with higher verbal density and a complexity of actions (Norris, 2004(Norris, , 2009. The verbal density is realised in the different reasons he produces to ask for Dương's forgiveness. ...
This paper presents the major findings from a recent study conducted to explore how a Vietnamese woman refuses a high-stakes advice or request in everyday conversations. Data used in this study are conversations excerpted from a TV series entitled Những công dân tập thể (lit. the citizens living in the same apartment building). The analytical tool is a combination of Conversation Analysis (Hutchby & Wooffitt, 1998, 2008; Sacks, 1992a, 1992b) and Multimodal Interactional Analysis (Norris, 2004, 2009). The results show that (1) Vietnamese refusing is often performed concurrently by different modes of communication and language is only one of them; (2) refusing a high-stakes advice or request often takes a long time to negotiate in a conversation and through a series of conversations; and (3) Vietnamese women’s responsibility to obey their parents, a Confucian teaching, still has its role in contemporary Vietnamese society.
... The aim of multimodal analysis is to examine the ways actions are mediated, how semiotic resources work together in multimodal action and how those resources relate to each other in such events. According to Norris (2004Norris ( , 2009, the modes used, the way they are used and the hierarchy of modes in one situation or by one participant in a situation cannot be assumed to take place in other situations and with other participants. The assumption is, for example, that a gesture or gaze can play a superordinate or equal role to the mode of language in interaction. ...
This paper attends to languaging in the context of visually oriented communities of sign language users through the concept of chaining. I define chaining as the patterned, routine ways of interlinking different linguistic and multimodal elements. The goal of this paper is to discuss the concepts of chaining, languaging and remodalisation through practical examples. The material for the paper comes from two ethnographic research projects which examine the use of multiple signed and spoken languages via different modalities in educational contexts. Both projects take a point of departure in ethnography and multimodal approaches towards interaction stemming from mediated discourse analysis. By examining the multimodal and multilingual practices, this paper aims at providing teachers and language learners with tools with which to reflect upon complex and often unnoticed resources used in everyday interaction.
... In this paper, the aim is to examine how the participants coordinate courses of action together and create and manage sites of attention that enable joint communicative action in the context of an English language course where an online learning environment, with chatrooms and discussion forums were used during and between the classes. Following Norris (2004Norris ( , 2009, this is done by analysing classroom interaction from a multimodal viewpoint without pre-given definitions of modes or assumptions about the hierarchy of modes. In the following sections I will present the key concepts and terminology relevant for understanding complex, multimodal interaction across multiple spaces and the theoretical perspectives on mediated action and multimodality this study aligns with. ...
Through detailed multimodal analyses, this article shows how participants of an English language course create and manage sites of attention for achieving collaboration across multiple spaces. The ethnographic data for the study comes from an English language course ‘Academic reading’ offered for university students majoring in Finnish Sign Language. The paper examines a classroom situation where the participants interact simultaneously with each other in a chatroom environment and physical environment via several mediational means, such as Finnish Sign Language and typed text in a chatroom that appears on students’ tablets and on a large screen in the classroom. The emerging interactional patterns coordinate fragmented interaction into mutual sites of attention so that the participants are able to collaborate. Further, the paper reflects how gaze plays a crucial role in coordinating actions together. Gaze itself is seen as a mediated action which begins to display regularities, and ‘comes into being’ through long and short life trajectories. The longer trajectory of the mode of gaze is traced to the patterned ways of using gaze in signing communities of practice.
... We do not wish to create the impression, however, that research through a multimodal lens is completely novel to, or absent from, the social sciences per se . On the contrary, a vibrant community dedicated to this topic has emerged during the last two decades, with two dedicated journals, Multimodal Communication and Visual Communication , a bi-annual international conference, a dedicated series of monographs ( Routledge Studies of Multimodality , edited by Kay O'Halloran), and a range of edited books ( Bowcher, 2012 ;Djonov & Zhao, 2014 ;Jewitt, 2014 ;LeVine & Scollon, 2004 ;Norris, 2012 ;Norris & Jones, 2005 ;O'Halloran & Smith, 2006 ;Unsworth, 2008 ;Ventola, Charles, & Kaltenbacher, 2004 ), including a 4-volume anthology edited by Sigrid Norris (2016 ). ...
... They suggest that, if designed in a suitable way, video analysis can provide insights that go far beyond what traditional data sources such as surveys enable. In the area of multimodality, Baldry and Thibault (2006 ), Flewitt, Hampel, Hauck, and Lancaster (2009), Idedema (2001, 2003a, and Norris (2006Norris ( , 2009 have provided detailed methods for the analysis of video data and occasionally included analyses of organizational processes such as the planning of a new reception area in a mental hospital ( Idedema, 2001 ) and multiparty interaction in an accounting office ( Norris, 2006 ). ...
... Es multimodal (Jewitt, 2009;Kress, 2010) porque considera que la negociación de los significados interaccionales se produce a partir de la interrelación de recursos pertenecientes a diferentes modos (lenguaje verbal, imagen, gestualidad, música, color) que adquieren potenciales de representación propios de acuerdo al género y al registro en el que se utilicen. A diferencia de lo planteado por Tannen, que coloca al lenguaje verbal en un lugar central y le asigna un lugar secundario o subordinado a los otros modos de comunicación, en esta investigación, en cambio, sostenemos que los significados se realizan multimodalmente (Goodwin 1981(Goodwin , 2000Jewitt, 2009;Kress, 1997Kress, , 2009Norris, 2004Norris, , 2006Norris, , 2009), a partir de la combinación de diferentes recursos semióticos, dentro de los cuales se incluye el lenguaje verbal. Esta combinación no admite un desdoblamiento entre lo dicho verbalmente y lo expresado a través de otros recursos (imágenes, gestos, sonidos). ...
En este artículo nos proponemos analizar una selección de interacciones tomadas de diferentes grupos de WhatsApp para dar cuenta de las diferentes estrategias discursivas utilizadas por los participantes para hacer invitaciones o propuestas, utilizando los recursos que componen el paradigma de opciones de este medio de comunicación. Sostenemos que la indireccionalidad puede analizarse como un rasgo que se encuentra presente en todas las estrategias discursivas que emplean los participantes de una conversación con un propósito interaccional específico. Es decir, en toda estrategia discursiva analizada en el marco de una interacción es posible reconocer cierto grado de indireccionalidad. La metodología de trabajo que aplicamos es de tipo interpretativa y de análisis cualitativo de los datos. Caracterizaremos las estrategias con el propósito de establecer recurrencias y correlacionar las diferentes realizaciones con dos factores interaccionales: la preservación de la imagen propia y la reafirmación del vínculo con los otros participantes. A partir de los resultados obtenidos, concluimos que, aun cuando ambos factores están siempre presentes, a un mayor grado de indireccionalidad de la estrategia utilizada aumenta el propósito de reafirmar el vínculo y disminuye, en cierta medida, el de proteger la propia imagen.Palabras clave: interacción, multimodalidad, pragmática.
... Applied to the context of feedback around assessment, the typical typed feedback pro forma could be understood either as monomodal in that the communication of meaning depends entirely on the mode of language, or alternatively multimodal through the selection of font, color, spacing, and layout, all of which convey meaning in juxtaposition with the content of the words on the page or screen. We might navigate our way out of this conceptual conundrum through the concept of modal density advanced by Sigrid Norris [20]. Modal density, according to Norris, can be used to describe the complexity of the representational act, for instance, when it depends on a broad range of semiotic material. ...
This article is concerned with digital, multimodal feedback that supports learning and assessment within education. Drawing on the research literature alongside a case study from a postgraduate program in digital education, I argue that approaching feedback as an ongoing dialogue presented in richly multimodal and digital form can support opportunities for learning that are imaginative, critical, and in-tune with our increasingly digitally-mediated society. Using the examples of a reflective blogging exercise and an assignment built in the Second Life virtual world, I demonstrate how the tutor’s emphasis on providing feedback in multimodal form, alongside more conventional print-based approaches, inspired and emboldened students towards the creation of apt and sophisticated coursework. At the same time, the crafting of multimodal feedback carries resource implications and can sit uncomfortably with some deep-rooted assumptions around language-based representations of academic knowledge. This article should be seen in the context of a growing pedagogic and institutional interest in feedback around assessment, alongside the emergence of new ways of communicating and consuming academic content in richly multimodal ways. In this setting, multimodality, technology, and interaction refers to the digitally-mediated dialogue that takes place between the student and tutor around assessment.
... Restrictive relative clauses are the only forms to present both salience and relevance, i.e. focalisation cues at several levels as well as contributions which are more decisive to the construction of referential meaning and to the realisation of sequential discourse purposes than their co-text. These subordinate constructions also show the highest modal density (Norris 2009), in that the three modalities feature at least one cue participating to the presentation of high informational input. This suggests an even distribution of information, and a great variety in their presentational modes. ...
Based on a video recording of conversational British English, this paper tests within the framework of Multimodal Discourse Analysis whether several different subordinate structures all express background information. Subordinate constructions have been described in syntax as dependent structures elaborating on primary elements of discourse. Although their verbal characteristics have been deeply analysed, few studies have focused on the articulation of the different communicative modalities in their production or provided a qualified picture of their informational input. Beyond showing that subordinate constructions express different types of prominence, the results suggest that the creation of focalisation mainly relies on gestural cues in these constructions. Changes in the modal configuration throughout the sequence suggest modalities are dynamic and flexible resources for expressing background or foreground information in subordinate constructions relatively to their syntactic type.
In recent decades, there has been a growing interest in multimodal translation studies. However, there is currently no suitable framework available for analyzing how the original meaning is reconstructed through the interaction of various semiotic resources in the multimodal target text. Under this context, this article proposes a systemic functional framework that integrates the systems of the meaning potential of semiotic resources, including language, sound (human voice), objects, space, and semiotic ensembles of the human body, and a tripartite classification of the relationships between semiotic resources. To exemplify the analytical potential of the framework, this article presents an example of analysis, taking an excerpt from Fences as the object of analysis.
In this paper, we explore how a group of 10 and 11-year-old primary school children engage with a picture book about a refugee boy from Somalia. As we examine in some detail a video-recording of the children’s discussion, we suggest that the children’s emotional engagement with the story was pivotal to not only their making sense of the book, but to their critical discussion of the issues the story raised. The discussion we report on here was part of a wider project to examine the use of picture books for critical literacy in schools. Critical literacy is often discussed as a rational endeavour, where children are invited to ask analytical questions about the message a text seeks to communicate and the means by which this is achieved. Following others, for example Anwarrudin (2016), who have challenged this focus on rationalism, we explore the role of emotions in the session. Data show that the children’s critical-analytical discussions of the story were closely connected with their emotional engagement. We use Norris’ (2004) multimodal interaction analysis to examine the children’s emotional and embodied engagement with the book and its story. This analysis of the children’s words, gestures, posture, gaze and voice quality reveals the complexity of their reactions to the book and specifically the role of ‘emotional collisions’ (Kuby, 2012, p. 35) in provoking embodied and affective reactions but also intellectual curiosity and ‘critical engagement’ (Johnson & Vasudevan, 2012, p. 35). With regards to the role of picture books in critical literacy pedagogy, our paper offers teachers new insights into what processes of thinking, feeling and communicating they can expect to be part of critical literacy lessons.
Este artigo apresenta um processo de criação teatral performativa com pessoas afásicas participantes do Programa de Expressão Teatral do Centro de Convivência de Afásicos do IEL/UNICAMP. O objetivo foi verificar como se dão os processos interssemióticos interacionais durante o percurso criativo e observar a emergência de semioses coocorrentes multimodais decorrentes da própria via performativa. A emergência de tais semioses manifesta-se nas práticas expressivas, teatrais e performativas das atuadoras afásicas durante a ação teatral.
Meeting the pedagogical needs of gifted students in science classroom requires specific pedagogical strategies and the competencies of teachers. In gifted science classroom, this competency includes designing didactic texts which foster meaning making (internalized learning product) of content and creativity in externalized learning products. Informed by multimodality and social semiotics, this study aims to develop a professional development model (intervention) that will develop multimodal didactic science text design competencies of science teachers of gifted of gifted students. The multimodal didactic science texts are expected to support (1) meaning-making of science content and (2) creativity in learning products. The intervention model has been developed within educational design research methodology. Six experienced in-service science teachers who work in schools of gifted students (BILSEM). Data is continuously gathered by interview data and participant’s multimodal text designs. Data analysis was done both qualitatively and quantitatively. The developmental progress of participants’ text design competency was the monitor of the development of the training program. The development of intervention model involves a preliminary phase, development phase (prototyping), and evaluation phase. The developed training model consists of design principles and two hypothetical learning trajectories that have their own learning goals, content, learning activities, and assessment tools. Learning activities in a HLT involve awareness, recognize, overt instruction, design, feedback, and re-design activities. In the end of the study, it was observed that the training model for multimodal didactic science texts design significantly improved the design skills of the participant teachers in the expected direction and the program was developed for future implications.
Classroom lead-in is the initial stage for motivating students to become engaged in-class interaction. However, little research, to our knowledge, has analyzed the role of teachers’ multimodal competence reflected through their multimodal pedagogic discourse in the realization of the ultimate goals of classroom lead-ins. Based on the data collected from a teaching contest in China, this paper explores how two-winner teachers utilize their multimodal ensembles of communicative modes to engage students during classroom lead-ins. The analysis shows that different communicative modes construct the higher-level action of lead-in, and they are orchestrated into multimodal ensembles for the specific function of each lead-in move. The findings indicate that EFL teachers’ high multimodal competence plays a decisive role in performing classroom lead-ins, and different lead-ins strategies influence the different orchestration of communicative modes. In constructing multimodal pedagogic discourse, teachers build up their professional image and display their personal charm as well. Future research for multimodal discourse analysis and pedagogic research is suggested in the paper.
Using multimodal (inter)action/conversation analysis, the present contribution inventories the repertoire of higher-level actions that constitute musical instruction in orchestra rehearsals. The study describes the modal complexity of the instructional actions as built from a varied combination of speech, gesture, gaze, vocalizing and body posture/movement. A high modal intensity of speech and vocalizing is explained with recourse to their contextually useful modal reaches. While some modes, like vocalizing and body posture appear to be action-specific, others turn out to be pervasive default modes. Besides modal intensity, the study also attends to the transitioning between higher-level actions through gaze and the role of the score as frozen action. The analyses help demystify orchestra rehearsals as a special type of professional communicative interaction, which builds on a rich multimodal texture motivated by recurring instructional functions. The methodological rationale demonstrated will be suited to exploring the social variation of instructional interaction in orchestra rehearsals.
Many studies investigating the use and effectiveness of multimodal communication are now confronting the need to engage with larger bodies of data in order to achieve more empirically robust accounts, moving beyond the earlier prevalence of small-scale ‘case studies’. In this article, I briefly characterise how recent developments in the theory of multimodality can be drawn upon to encourage and support this change in both scale and breadth. In particular, the contribution will show how refinements in the degree of formality of definitions of the core multimodal constructs of ‘semiotic mode’ and ‘materiality’ can help bridge the gap between exploratory investigations of complex multimodal practices and larger-scale corpus studies.
As digital technologies proliferate throughout classrooms in the United States, iPads and other mobile tablets have been heralded as tools for enhanced peer collaboration. However, little research has examined exactly how students interact with and around iPads during collaborative learning tasks. This study employs multimodal interaction analysis to investigate how secondary Spanish immersion students leveraged the physicality of the iPads as multimodal affordances when completing a world history literacy-based task in a target language. In particular, I consider how the focal students employed touch-based communicative modes to complete higher-level actions, and the potential relationship between these modes and target language development. Findings have implications for materials use research in language teaching/learning classrooms as well as progressing our understanding of the impact of iPads on classroom discourse.
The communicative purpose of oral academic genres is frequently achieved through multimodal ensembles (i.e., a combination of meaning-making resources or modes). This is the case in research pitches. However, little has been written about this genre and its multimodal nature. Drawing from previous research on multimodality (Jewitt, 2009; Norris, 2009; Valeiras Jurado, 2019), I examine traces of modal density, complexity and cohesion in research pitches’ opening and closing moves, which often reflect researchers’ need to persuade their audience of the importance of their research. The aim of this study is twofold. First, I analyse the rhetorical structure of six well-rated 3-min research pitches following Swales’ move theory (2004). Second, adopting a Multimodal Discourse Analysis (MDA) perspective (Querol-Julián, 2011; Valeiras-Jurado & Ruiz-Madrid, 2019), I describe how the multimodal ensembles orchestrated by the speakers contributed to the communicative aim of the opening and closing moves. By way of conclusion, I derive pedagogical implications from the findings of this study.
Despite its obvious importance to learning and assessment across the academy, the undergraduate classroom presentation has received less research attention than other academic genres, and little is known about how multiple modes of communication are deployed within it. With the aim of exploring how the use of different modes varied between sections, and how these actions affected the speech of presenters, this research into student presentations given at a university in Turkey combined a move-step analysis of speech with a mixed-methods study of multimodality. The study’s main results were as follows: first, that presentation sections were distinctively configured by arrays of multimodal action; second, that the effectiveness of speech in performing specific moves in the genre was moderated in several specific ways by actions in other modes; and third, that some moves were performed in part by non-verbal actions. These findings are briefly discussed with reference to their theoretical and pedagogical implications.
The present study investigated students’ (N = 404) interpretations of the main message and use of modes in a persuasive multimodal video on vaccines. It also examined whether students’ topic knowledge, language arts grades, and self-identified gender were associated with their interpretations. Students analyzed a YouTube video in which two entertainers demonstrated the importance of vaccinating children. Students’ interpretations of the usefulness of vaccines varied in terms of quality of reasoning, which was associated with students’ topic knowledge. Notably, many students’ interpretations of the use of modes were incomplete, or they did not even mention certain modes in their response. The results suggest that students should be explicitly taught how to interpret different modes and their uses for argumentative purposes.
Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse . Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development . Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics . Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . Routledge, London. https://doi.org/10.4324/9780203379493 ; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis . de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280 ; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality . Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77 ) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.
The contexts and methods for communicating in healthcare and health professions education (HPE) profoundly affect how we understand information, relate to others, and construct our identities. Multimodal analysis provides a method for exploring how we communicate using multiple modes—e.g., language, gestures, images—in concert with each other and within specific contexts. In this paper, we demonstrate how multimodal analysis helps us investigate the ways our communication practices shape healthcare and HPE. We provide an overview of the theoretical underpinnings, traditions, and methodologies of multimodal analysis. Then, we illustrate how to design and conduct a study using one particular approach to multimodal analysis, multimodal (inter)action analysis, using examples from a study focused on clinical reasoning and patient documentation. Finally, we suggest how multimodal analysis can be used to address a variety of HPE topics and contexts, highlighting the unique contributions multimodal analysis can offer to our field.
In different cultures, colours are sometimes assigned different meanings. Understanding the origins of these cultural colour meanings has become increasingly important with the ongoing advances towards digitalization of business and communication, and the most recent phase of
globalization. However, academic research has largely neglected this phenomenon.
The purpose of this dissertation is to increase our understanding of the dynamics of crosscultural meaning-making for colours by examining the provenance of cultural colour meanings. This is achieved by employing empirical studies set in Chinese-Finnish business contexts, taking into account particular perspectives induced by globalization and online media, and their implications on the developments within the intersections of the research domains of business communication, culture, and colour.
Methodologically, this research applies an emic-etic cross-cultural approach, using within method triangulation of qualitative methods to explore the phenomenon. The empirical materials consist of narrative, visual, and observation materials generated within studies
conducted in China and in Finland between January 2007 and July 2009. The two countries were chosen due to the shift of interest of many Finland-based MNCs from China-as-a-factory to include China-as-a-market, and ensuing perception of previously unexplored cultural
differences in visual conventions, colour meanings in particular.
Engaging in the debate within management and communication studies this research suggests a potential synthesis between the systemic and the process views to Culture. It finds that the significance of some cultural forces is inherited and thus more persistent (systemic view),
while that of other cultural forces is more dynamic and transforming in nature (process view). Consequently, by framing Culture as dynamic heritage, this research proposes a conceptualization for Provenance of cultural colour meanings, contributing to the Multimodal theory of colour. Likewise, this research propositions the significance of the expressive
function of the mode of colour borrowing from the former art historical and psychoanalytic approaches as well as based on the present empirical studies. Consequently, engaging with the Interactionist theory of communication, this research suggests that Meaning-making for
colour can be conceived as an expression of cultural experience whereby the communicators become expressors of their respective cultural background and experience. Finally, this dissertation advances our understanding of meaning-making for colour in a multimodal
context by adding the cross-cultural dimension to previous models, and shows that different modes can elaborate, extend, enhance, and contradict each other.
This is a study of how children represent themselves when performing participatory identities in social media communities with relevance to constructing a learning self. Data was generated by filming eight children (6–11 years of age) talking about and showing their multimodal self-representations. On their out-of-school learning journeys, the children came into presences as ‘a someone’, in social media communities. The theoretical foundation informing the study is Wenger's theory on learning as social participation. Multimodal interactional analysis was applied to move the analysis beyond transcripts of texts to include actions children take with or through multimodal mediational means. The results display significant aspects of children's learning trajectories in self representation, presented as: Input from comments, understanding the other, preparing for a performing self and taking actions. Out of these acts of participation, three different participatory identities were constructed: the user, the producer and the designer. The main results show how children through participation, widen their learning repertoire and critically reflect on space and place. This research adds to the educational field by presenting children's experiences from navigating new worlds and enacting participatory identities, which is of relevance for their ongoing construction of a learning self.
Theories we accept and ‘think with’ open up possibilities and impose constraints on children’s musical engagement and on their understandings of their musical selves. In this chapter, I explore and combine theories based in research on children’s culture and suggest a possible theoretical framework for investigating and evaluating young children’s uses of recorded music (phonograms). I argue that Christopher Small’s (Musicking: The meanings of performing and listening. University Press of New England, Hanover, 1998) concept of musicking is a fruitful starting point, and from this I derive the notion of musickingship. With musickingship I mean a person’s capacity to participate in a musical performance in a broad sense in ways which are experienced as meaningful to her or him. I highlight the multimodal character of children’s participation in music, their experience of meaning and the relationships which form during the musical performances. I go on to elaborate on musickingship, drawing on the notion of affordance from Tia DeNora (Music in everyday life. Cambridge University Press, Cambridge, 2000) which stresses music as a resource for the self. Finally, my theoretical framework includes performativity, which here helps to focus on the constitutive musical moments of everyday life, and interpretative repertoires, which brings out the social and discursive system of meaning-making. In the last sections of the chapter, I employ this theoretical framework to discuss patterns of gendered musical engagement and raise critical questions about how we—parents, music educators and researchers—are regulated by discursive systems of meaning-making, and thus, how we might inadvertently constrain children’s musickingships when we mean to support them.
The present study belongs to an extensive project that explores how academic knowledge is mediated through new generic structures and publishing formats and provides data comprising research video articles from JoVE, the international Journal of Visualized Experiments. In order to deal with the implications of video formats in web contexts upon academic user engagement and knowledge building processes, we adopt a multimodal (inter)action approach in our analysis. We show how exploiting the complex hypermodal configurations contribute to changes in the balance of various types of knowledge and to the potential building of new ones. We also show that by embedding the video in a hypermodal context urges academic users to increase their engagement with the article in unprecedented ways.
Digital and electronic learning portfolios (e-portfolios) are playing a growing role in supporting admission to tertiary study and employment by visual creatives. Despite the growing importance of digital portfolios, we know very little about how professionals or students use theirs. This thesis contributes to knowledge by describing how South African high school students curated varied e-portfolio styles while developing disciplinary personas as visual artists. The study documents the technological and material inequalities between these students at two schools in Cape Town. By contrast to many celebratory accounts of contemporary new media literacies, it provides cautionary case studies of how young people’s privileged or marginalized circumstances shape their digital portfolios as well.
A four-year longitudinal action research project (2009-2013) enabled the recording and analysis of students’ development as visual artists via e-portfolios at an independent (2009-2012) and a government school (2012-2013). Each school represented one of the two types of secondary schooling recognised by the South African government. All student e-portfolios were analysed along with producers’ dissimilar contexts. Teachers often promoted highbrow cultural norms entrenched by white, English medium schooling. The predominance of such norms could disadvantage socially marginalized youths and those developing repertoires in creative industry, crafts or fan art.
Furthermore, major technological inequalities caused further exclusion. Differences in connectivity and infrastructure between the two research sites and individuals’ home environments were apparent. While the project supported the development of new literacies, the intervention nonetheless inadvertently reproduced the symbolic advantages of privileged youths.
Important distinctions existed between participants’ use of media technologies. Resource-intensive communications proved gatekeepers to under-resourced students and stopped them fully articulating their abilities in their e-portfolios. Non-connected students had the most limited exposure to developing a digital hexis while remediating artworks, presenting personas and benefiting from online affinity spaces.
By contrast, well-connected students created comprehensive showcases curating links to their productions in varied affinity groups. Male teens from affluent homes were better positioned to negotiate their classroom identities, as well as their entrepreneurial and other personas. Cultural capital acquired in their homes, such as media production skills, needed to resonate with the broader ethos of the school in its class and cultural dimensions. By contrast, certain creative industry, fan art and craft productions seemed precluded by assimilationist assumptions. At the same time, young women grappled with the risks and benefits of online visibility.
An important side effect of validating media produced outside school is that privileged teens may amplify their symbolic advantages by easily adding distinctive personas. Under-resourced students must contend with the dual challenges of media ecologies as gatekeepers and an exclusionary cultural environment. Black teens from working class homes were faced with many hidden infrastructural and cultural challenges that contributed to their individual achievements falling short of similarly motivated peers. Equitable digital portfolio education must address both infrastructural inequality and decolonisation.
This paper explores integrating a range of digital media into classroom practice to establish the effectiveness of the media and its encompassing modes as a pedagogical tool with a focus on assessment. Directing attention on a communication skills module, research indicated that bringing a range of digital media into the classroom motivated and aided further education students to achieve. The challenge of finding ways to converge everyday and college-assessed literacy practices is complex on many levels and demands an understanding of emerging digital literacy practices, particularly the adaptations students make at the level of mode. A concern was that the full nature of digital media and the semiotic resources students use is not recognised, and the potential digital literacy practices have may be underestimated. Taking an ethnographic perspective, one research goal was to gain insight into how students engage with digital media by developing multimodal transcription grids. The notions of ‘funds of knowledge’ and ‘social capital’ are drawn on to underpin an analysis of a range of digital media to establish how the benefits of digital media can be further understood and used within an educational setting, particularly for assessment.
Multimodality approaches representation and communication as something more than language. It attends to the complex repertoire of semiotic resources and organizational means through which people make meaning – image, speech, gesture, writing, three-dimensional forms, and so on. A social semiotic approach to multimodality sets out to reveal how processes of meaning making (i.e., signification and interpretation or what is called semiosis) shape individuals and societies. In this chapter, we use multimodality to refer to “multimodal social semiotics” (Kress, Multimodality: A social semiotic approach to contemporary communication. London: Routledge, 2010). Its basic assumption is that meanings derive from social action and interaction using semiotic resources as tools. A variety of disciplines and theoretical approaches can be used to explore different aspects of the multimodal landscape (Jewitt, The Routledge handbook of multimodal analysis. London: Routledge, 2014). Psychological theories can be applied to look at how people perceive different modes or to understand the impact of one mode over another on memory, for example. Sociological and anthropological theories and interests can be applied to examine how communities use multimodal conventions to mark and maintain identities. The term multimodality is, however, most strongly linked with theories rooted in linguistics, notably systemic functional linguistics, social semiotic theory, and conversation analysis (Jewitt et al., Introducing multimodality. London: Routledge, 2016a). Examining multimodal discourses across the classroom makes more visible the relationship between the use of semiotic resources by teachers and students and the production of curriculum knowledge, student subjectivity, and pedagogy.
Multimodal analysis of classroom music interactions, using the model of the ‘Space of Music Dialogue’ in video analysis of students' music improvisation, was useful to inform teachers of students' collaborative achievements in music invention. Research has affirmed that students' cognitive thinking skills were promoted by improvisation. Students purposefully selected from many modes such as movement, gaze and spatial relations as domains of learning. The students, for example, rearranged these modes to promote musical arrangements and a growing sensitivity to visual and rhythmic perception. Students selected and rearranged modes to solve problems. Over time, students realised cognitive relations of modes in music, for example, through a deeper understanding of the elements of music: pitch, rhythm, dynamics, structure, phrasing. Only some students reached the realm of transmodal redesign, made possible as students became familiar with the music mode, and the conceptual elements of music. Choices in problem solving in the arts, through multiple choices in multimodal redesign, granted all students the ability to build their self-esteem through transformational redesign. New challenges allowed students to develop conceptual understanding. Students succeeded at problem solving in music, and the model assisted in the analysis of events including improvisation.
ResearchGate has not been able to resolve any references for this publication.