Multimodal Theory and Methodology: For the Analysis of (Inter)action and Identity
Abstract
This concise guide outlines core theoretical and methodological developments of the growing field of Multimodal (Inter)action Analysis.
The volume unpacks the foundational relationship between multimodality and language and the key concepts which underpin the analysis of multimodal action and interaction and the study of multimodal identity. A focused overview of each concept charts its historical development, reviews the essential literature, and outlines its underlying theoretical frameworks and how it links to analytical tools. Norris illustrates the concept in practice via the inclusion of examples and an image-based transcript, table, or graph.
The book provides a succinct overview of the latest research developments in the field of Multimodal (Inter)action Analysis for early career scholars in the field as well as established researchers looking to stay up-to-date on core developments.
... There are many new developments and perspectives. The one that we work in at the AUT Multimodal Research Centre is multimodal (inter)action analysis (MIA) (Norris 2004a(Norris , b, 2011a(Norris , 2029(Norris , 2020Geenen et al. 2015), which has grown from theory and methodology to a fully-fledged framework. I will take a moment to set out the state of this framework and discuss the role that audio-visual technology plays in MIA. ...
... On the overarching philosophical strata, MIA posits that human action, interaction and identity come about through a primacy of perception and a primacy of embodiment (Norris, 2019). On the theoretical strata, MIA follows MDA (Scollon, 1998(Scollon, , 2001, insisting on the principles of social action (including communication) and history (Norris, 2020). On this theoretical level, MIA further follows MDA by arguing that all human actions are mediated, and that mediated actions with a history are practices. ...
... Then, MIA moves on from MDA to multimodal mediated theory, noting that mediated actions appear on different levels (lower-level, higher-level and frozen) (Norris, 2004a). Multimodal mediated theory further suggests that modes are systems of mediated actions (Norris, 2013) and discourses are practices (mediated actions with a history) with an institutional and/or ideological dimension (Norris, 2020). When looking at Figure 1, we see how the philosophical and the theoretical strata seemingly sit above method and methodology. ...
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... Since analyzing multimodal interaction means broadening the scope beyond writing and speech (Jewitt, 2009), a comprehensive framework offering researchers a tool to probe into a vast array of meaning-making resources is needed. Regarding this, the framework of multimodal interaction analysis proposed by Norris (2004Norris ( , 2020 is applicable in many educational research contexts. Norris (2004) explained that among the communicative modes observable through multimodal interaction analysis are "proxemics, posture, head movement, gesture, gaze, spoken language, layout, print, (and) musicwhich are essentially systems of representation." ...
... The framework is also applicable in science (Wilmes & Siry, 2021) and higher education (Bernad-Mechó, 2021). These studies are evidence that the framework established by Norris (2004Norris ( , 2020 is relevant and applicable in research across disciplines, classroom settings, and educational levels. Some scholars, however, adopted different frameworks in investigating semiotic resource orchestration in language teaching. ...
Teaching is an activity involving semiotic resources beyond verbal language. Therefore, more studies investigating what semiotic resources are employed and how they are utilized within EFL classroom settings are needed. This study sought to explore the types of semiotic resources employed by two EFL teachers in Indonesia and the roles of these semiotic resources in teaching activities. Through a qualitative case study, two EFL teachers in Bandung and Medan, Indonesia, were purposively selected. Data collection was through a semi-structured interview and document analysis. A six-phase of thematic analysis was employed to analyze the interview results, while a three-step of skimming, reading, and interpretation was used to analyze the documents. This study found that spoken language, pictures, videos, proxemics, gestures, head movements, gaze, and music were the semiotic resources used by both teachers in their teaching. The findings also demonstrated that these semiotic resources were used in the opening, delivering, and closing of the class, explaining learning content, performing interactions with students, checking the students' progress, managing classroom order, showing appreciation, and conducting evaluations. This study implies the crucial roles of all stakeholders in education to help EFL teachers in multimodal knowledge and practical strategy to create better teaching outcomes.
... MDA. Higher-level actions are the sum of chains of different single actions (Norris, 2004(Norris, , 2020, and higher-level actions make the actions interdependent (Zhang and Wang, 2016). Zhang (2009) proposes an outline of the multimodal discourse analysis model (MDAM; see Figure 1), which includes four levels-culture level, context level, content level, and expression level. ...
... Our findings corroborate the previous point of view that classroom teaching is a multimodal experience that happens through orchestration of spoken language and an array of other communicative modes, such as gesture, gaze, and facial expression (Kress et al., 2005;Jewitt, 2006;Peng, 2019;Lim, 2021;Qin and Wang, 2021). The ensemble of the actions, or the sum of chains of different single actions, belong to the higher-level actions (Norris, 2004(Norris, , 2020, which make the actions interdependent (Zhang and Wang, 2016). Our findings also reveal that AR is a useful method to improve teachers' professional development, just as Wallace (1998) mentioned an "empowering procedure, " or quality enhancement (Kember, 2000). ...
This study investigates the influences of action research on primary school English instruction from five dimensions in the classroom, viz. , types of questions, language errors, gestures, facial expressions, and interpersonal distance. Four English teachers’ 9 real classroom teaching videos before and after action research are collected and annotated by using ELAN software. The results show that primary school English teachers in Chinese rural areas prefer closed questions to open questions; They make some language errors; Deictic gestures are the most common gestures used, while metaphoric gestures, beat gestures and iconic gestures are rare; Teachers have the same preference for three types of facial expressions, and teachers’ serious expressions accounts for most of the time; They seldom keep an intimate distance or personal distance from their students. Second, Action research is effective to motivate teachers in rural areas, who make great progress in all five dimensions after AR: more open questions are asked; pragmatic errors and grammatical errors are reduced; deictic gestures increase; apathetic decrease, and more intimate distance is exhibited in the interpersonal distance dimension. Third, teacher’s English teaching competence is partly transferrable to her future professional development, and this is also the long-lasting effect of AR.
... For example, the objects present in a classroom construct the mode of layout, which gives off messages about the social actor and structure the interaction somehow. According to Norris (2004Norris ( , 2019Norris ( , 2020, the higher-level actions are also fluid and develop in real-time, and each higher-level action is bracketed by social openings and closings that are at least in part ritualized. Jewitt (2014, p. 36) holds that multimodal interaction focuses on the mediate interaction in a given context, that is, how a variety of modes are brought into and constitutive of social interaction. ...
... The researchers follow the procedures suggested by Norris (2004Norris ( , 2019Norris ( , 2020 to analyze two EFL teachers' multimodal pedagogic discourse during classroom lead-ins. According to Norris (2004, 2019, 2020), the first step to a multimodal interaction analysis is to understand an array of communicative modes. ...
Classroom lead-in is the initial stage for motivating students to become engaged in-class interaction. However, little research, to our knowledge, has analyzed the role of teachers’ multimodal competence reflected through their multimodal pedagogic discourse in the realization of the ultimate goals of classroom lead-ins. Based on the data collected from a teaching contest in China, this paper explores how two-winner teachers utilize their multimodal ensembles of communicative modes to engage students during classroom lead-ins. The analysis shows that different communicative modes construct the higher-level action of lead-in, and they are orchestrated into multimodal ensembles for the specific function of each lead-in move. The findings indicate that EFL teachers’ high multimodal competence plays a decisive role in performing classroom lead-ins, and different lead-ins strategies influence the different orchestration of communicative modes. In constructing multimodal pedagogic discourse, teachers build up their professional image and display their personal charm as well. Future research for multimodal discourse analysis and pedagogic research is suggested in the paper.
... Multimodal theory (e.g., Kress & van Leeuwen, 2006;Norris, 2020), social semiotics (e.g., Metz, 1974;van Leeuwen, 2004), and systemic functional linguistics (e.g., Bartlett & O'Grady, 2017;Halliday, 1978) provide useful conceptual guidelines for "reading" multimodal texts (including film) and are often utilized in cinema studies. The theories all address such topics as shared notions of the social dimension of communication, the grammatical (or structural, compositional) relations of semiotic elements (linguistic and visual grammar) in a text, the metafunctions of linguistic and non-linguistic texts (i.e., ideational, interpersonal, and textual), and contextually specific narratives, ideas, and meanings expressed in the interplay of different modes. ...
This paper describes one 4-week long online language learning module which utilizes the animated documentary film "My Favorite War" (Burkovska-Jakobsen, 2020) and analyzes four students’ final essays. The learning module was created considering ethnographic views of cultural inferences (Spradley, 2016), the framework of social space (Lefebvre, 1991), multimodal theory (Kress & van Leeuwen, 2006), and the model of learning activities, or knowledge processes (Cope & Kalantzis, 2015). It was developed for a Latvian language course at the University of Washington (Seattle, USA) in 2021. Its goals are to help students learn more about the target language and locals’ place-based experiences, and to promote students’ semiotic consciousness while developing their multiliteracies. However, the assignment’s assessment criteria related to the interaction of semiotic resources and of linguistic and non-linguistic information were not shared in order to find out if students would include these issues in their essays. Analysis of students’ essays show that after the learning module, students were able to comprehend (a) some individual compositional elements from the film; (b) symbolic and functional meanings of differently marked spaces; and (c) the main character’s feelings in various social spaces, internal conflicts, relationship models, and thoughts about family, work, war, historical truth, and independence. The described learning model can inspire educators worldwide to develop similar teaching practices for other less commonly taught languages.
... An adaptation of the multimodal (inter)action analysis (MIA) framework (Norris, 2004(Norris, , 2020) was used to explore how one EMI lecturer fostered and managed interaction during an online class. ...
... Another potential avenue to extend both research understandings and methods of representation that might do more justice to the multidimensional nature of emotions is multimodal analysis (e.g., Goodwin, 2000Goodwin, , 2010Goodwin, , 2018Norris, 2004Norris, , 2011Norris, , 2020. Like any area of inquiry, the breadth of this field is truly impressive, and I must admit that I am only just beginning to make my own way into exploring the landscape. ...
This short-form research monograph offers a socially situated view of the emergence of emotionality for additional language (L2) learners in classroom interaction in Japan.
Grounded in a complexity perspective, I argue that emotions need to be studied as they are dynamically experienced and understood in all of their multidimensional colors by individuals (in interaction). Via practitioner research, I apply a small-lens focus (Ushioda, 2016), interweaving experiential and discursive data, offering possibilities for exploring, interpreting and representing the lived experience of L2 study emotions in a more holistic yet detailed, social yet individual fashion. Amidst the currently expanding interest in L2 study emotions, the book presents a strong case for the benefits of locating interpretations of the emergence of L2 study emotions back into situated, dynamic, social context.
The electronic version of this book is available open access - free to view or download - from the following URL (Routledge):
https://www.taylorfrancis.com/books/oa-mono/10.4324/9781003306955/complexity-second-language-study-emotions-richard-sampson
... W ostatnich dwóch dekadach w szeroko pojętej lingwistyce można zauważyć rosnące zainteresowanie komunikacją niewerbalną -zjawisko określane w naukach humanistycznych mianem zwrotu wizualnego (Mitchell 1994) To zainteresowanie przekłada się na popularność takich nurtów, jak multimodalna analiza dyskursu (multimodal discourse analysis (np Kress, van Leeuwen 2001;O'Halloran 2004;Norris 2020)) czy gramatyka systemów wizualnych (np Kress, van Leeuwen 1996;Machin 2016) Ta ostatnia jest przykładem adaptacji modeli systemu językowego do opisu innych systemów semiotycznych na podstawie analogii między leksykonem i regułami gramatyki z jednej strony a zasobami niewerbalnymi danego systemu i zasadami ich użycia z drugiej W podobny sposób poszukiwanie funkcjonalnych analogii między językowymi środkami wyrazu a środkami niewerbalnymi 1 motywowało opracowanie kognitywnych modeli metafory wizualnej 1 Chodzi tu o analogie funkcjonalne na najbardziej ogólnym poziomie relacji, wskazujące na potencjał wyrażania metafor pojęciowych zarówno przez środki językowe, jak i wizualne bądź realizację struktury kognitywnej opartej na domenie źródłowej i domenie docelowej Specyfika obu systemów semiotycznych oraz właściwych im zasobów utrudnia, czy wręcz uniemożliwia, Celem niniejszego artykułu jest przedstawienie wybranych teorii, metod i analiz w dziedzinie kognitywnego ujęcia metafory wizualnej Na początku omawiam dwa źródła tego obszaru badań mieszczące się w tradycyjnym ujęciu metafory wizualnej jako środka artystycznej ekspresji i w kognitywnej teorii metafory werbalnej Następnie przedstawiam aspekt teoretyczny tej dziedziny, opisując wybrany model analizy (Forceville 1996) oraz identyfikacji (VISMIP) metafor wizualnych Przegląd analiz empirycznych pokazuje z kolei dalsze zróżnicowanie metod stosowanych przez autorów oraz badanych przez nich gatunków W ostatniej części artykułu omawiam aktualny stan badań nad metaforą wizualną w Polsce, zwracając uwagę na konieczność popularyzacji i systematyzacji tego obszaru ...
COGNITIVE MODELS OF VISUAL METAPHOR: THEORY, METHODOLOGY, ANALYSES
The aim of the article is to present cognitive studies on visual metaphor: selected theories, methods and analyses. An overview of literature shows a growing interest in this area of research worldwide, especially in the light of the “visual turn”. Compared to the scholarly output of foreign researchers, publications of Polish authors are scarce and their results are not systematized. The discussion of the current state of research on visual metaphor is therefore motivated, on the one hand, by knowledge gap, and on the other, by the need to popularize and systematize this area of study in Poland.
... The (inter)action multimodal framework (Norris 2004(Norris , 2019(Norris , 2020) was adopted to undertake the research. One of the main concerns of this approach is the study of social interaction. ...
This paper examines the evolving genre of university lectures. It focuses on synchronous online lectures. The aim of the study is to shed some light on how interaction between teacher and students unfolds in large English-medium instruction (EMI) lectures in the digital context. A qualitative multimodal microanalysis of an episode of interaction was performed from an (inter)action multimodal analysis framework. This preliminary exploratory study reveals the structural and multimodal complexity of interaction in live online lectures. EMI teacher's semiotic resources combine to make meaning comprehensible in a lingua franca and to engage learners in a virtual context where there is not eye-contact with them. Suggestions are made to undertake contractive studies on interaction in online and face-to-face lectures that may respond to the need of EMI teacher training. This paper aims to contribute to the literature of this still unexplored academic instructional digital genre.
... When investigating entries in the lab book, we have drawn on the notion of modal density, which is used to analyze when and how modes structure attention toward and away from higher level actions, including those "beyond the focus" (Norris, 2016(Norris, , p. 153, 2019(Norris, , 2020, for instance, how medical physicists engage in acceptable research practices of their community. ...
Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation.
... Multimodal (inter)action analysis (Norris, 2004(Norris, , 2019(Norris, , 2020 is "a holistic analytical framework that understands the multiple modes in (inter)action as all together building one system of communication" (Norris & Pirini, 2016: 24). Within this framework, all actions are considered interactions between social actors and other social actors, objects, or the environment. ...
Online language teaching is gaining momentum worldwide and an expanding body of research analyses online pedagogical interactions. However, few studies have explored experienced online teachers' practices in videoconferencing particularly while giving instructions, which are key to success in task-based language teaching (Markee, 2015). Adopting multimodal (inter)action analysis (Norris, 2004, 2019) to investigate the multimodal construction of instructions in a single case study, we examine instruction-giving as a social practice demonstrated in a specific site of engagement (a synchronous online lesson recorded for research purposes). Drawing on the higher-level actions (instruction-giving fragments) we have identified elsewhere (Satar & Wigham, 2020), in this paper we analyse the lower-level actions (modes) that comprise these higher-level actions, specifically focusing on the print mode (task resource sheets, URLs, text chat, and online collaborative writing spaces) wherein certain higher-level actions become frozen. Our findings are unique in depicting the modal complexity of sharing task resources in synchronous online teaching due to semiotic misalignment and semiotic lag that precludes the establishment of a completely shared interactional space. We observe gaze shifts as the sole indicator for learners that the teacher is multitasking between different higher-level actions. Further research is needed to fully understand the interactional features of online language teaching via videoconferencing to inform teacher training policy and practice.
This paper discusses the linguistic features of the production of football commentary within its macrosocial contexts. It contrasts the communication of the TV production team during a football game with the live football commentary aired on TV. Applying a mixed methods analysis, it reveals how football commentary is prepared and produced in the technical setting of the commentator’s booth in a stadium. This study reveals how a medial reality of the same events concerning a goal are broadcasted in the 11 min highlight video football commentary. The author and her co-principal investigator video recorded a total of 36 h in four work settings. Finally, retrospective interviews with the commentator and head of sports of the TV station place the study in the broader social context of football as a media production. In this paper, I show how live football reporting is prepared and conducted using two different theoretical and methodological frameworks: Conversation Analysis (CA) and Multimodal (Inter)action Analysis (MMIA). Both provide insight into the synthetic verbal, non-verbal and parallel actions for the successful production of the football commentary and they also reveal the sequential structure of naturally occurring talk (CA).
This visual essay examines the non-verbal actions and the production of identity of Serbian and English native speakers during a videoconferencing task in New Zealand. This cross-cultural study examines the modes of facial expression, posture, and hand/arm movements to show how these are telling of the identity of the Serbian and English native speakers. Although gesture and language have received significant attention in research, very few studies have addressed the cross-cultural non-verbal actions and no study has compared Serbian native speakers with New Zealand English speakers. This visual essay addresses this gap.
The present study centers on exploring how different modes such as speech, gesture, gaze, body posture, and head movement are employed by an EFL teacher and his students during a Critical Learning Episode (CLE) (Davis et al.,2009). CLEs are brief instances of classroom interaction where the instructor and the researcher believe that learning is being fostered or inhibited. This article is part of a larger qualitative multiple-case study that took place at a private Colombian University. The lesson was videotaped and then analyzed within a Multimodal (Inter)action Analysis framework (Norris, 2020). Findings indicate that CLEs are created in a highly embodied, multimodal, and ecological manner through different modal configurations. Besides speech and writing, modes such as gestures, posture, gaze, and head movement played not a marginal, but prominent role, in performing various pedagogical classroom activities such as enhancing shared/focused attention, strengthening alignment, helping teachers and learners to visually make meaning of morphological and syntactical units, and serving as devices to check for understanding.
The construction of Samoan identity in New Zealand is shaped by complex historical and contemporary social, economic, cultural and political factors. In addition, New Zealand-born Samoans are negotiating an ethnic and identity that incorporates their experience with the intergenerational stories and cultural knowledge of their ancestors. Such cultural and identity negotiations are occurring through the practice of Siva Samoa, Samoan dance. Using Multimodal (Inter)action Analysis we conduct a micro analysis of two excerpts of video data involving female Samoan dancers rehearsing for a dance showcase. Vertical identity production (Norris, S. (2011). Identity in (inter)action: introducing multimodal (inter)action analysis. Berlin and Boston: Mouton: 179, 2020:85) is used as a framework to analyze the multiple layers of discourse within each site of engagement that shape the construction of Samoan Identity for the participants involved.
In this paper, we explore how a group of 10 and 11-year-old primary school children engage with a picture book about a refugee boy from Somalia. As we examine in some detail a video-recording of the children’s discussion, we suggest that the children’s emotional engagement with the story was pivotal to not only their making sense of the book, but to their critical discussion of the issues the story raised. The discussion we report on here was part of a wider project to examine the use of picture books for critical literacy in schools. Critical literacy is often discussed as a rational endeavour, where children are invited to ask analytical questions about the message a text seeks to communicate and the means by which this is achieved. Following others, for example Anwarrudin (2016), who have challenged this focus on rationalism, we explore the role of emotions in the session. Data show that the children’s critical-analytical discussions of the story were closely connected with their emotional engagement. We use Norris’ (2004) multimodal interaction analysis to examine the children’s emotional and embodied engagement with the book and its story. This analysis of the children’s words, gestures, posture, gaze and voice quality reveals the complexity of their reactions to the book and specifically the role of ‘emotional collisions’ (Kuby, 2012, p. 35) in provoking embodied and affective reactions but also intellectual curiosity and ‘critical engagement’ (Johnson & Vasudevan, 2012, p. 35). With regards to the role of picture books in critical literacy pedagogy, our paper offers teachers new insights into what processes of thinking, feeling and communicating they can expect to be part of critical literacy lessons.
This chapter examines the miscommunication of an intercultural team working on a task via videoconferencing technology using English. We utilize multimodal (inter)action analysis (Norris, 2004, 2011, 2019, 2020) as our theoretical and methodological framework to shed new light on how participants appear to negotiate and co-construct common ground, while they in fact do not achieve conceptual convergence but, instead, produce their own actions. The data for this chapter comes from a corpus of twelve dyads working on tasks via videoconferencing technology in New Zealand. Data was collected from various English monolingual and Serbian multilinguals in various interactive constellations. In this chapter, we focus on a dyadic team with a Serbian native speaker and a monolingual New Zealand English speaker. Rajic and Norris (2018) show that Serbian native speakers’ nonverbal actions vastly differ from New Zealand English speakers’ nonverbal actions. While the difference in production of nonverbal actions is relevant in all interactions, they do not necessarily lead to intercultural miscommunications. However, as noted earlier, interactive alignment is not just a linguistic accomplishment (Pirini & Geenen 2018). Our results conflict somewhat with those of other scholars (House, 1999; Mauranen, 2006), who claim that few miscommunications occur in English as a Lingua Franca interactions. Owing to our analysis, we would like to claim that miscommunications in linguistically and culturally diverse communicative situations are more frequent than previously thought. However, many of the miscommunications that occur cannot be said to come about because of cultural differences.
Sports announcer talk is characteristic for its intonation and grammar. The language of sports commentary is unique in its syntactic style in that it uses grammatical omissions and dislocations (Callies and Levin 2019 Gerhard, 2008, Trouvain 2011). Research has shown that the dramatic phases in incidents in sports announcer talk are an pivotal aspect of speech (Trouvain 2011). The dramatic phases are created with para-verbal language such as prosody, pitch and loudness and they are typical of sports commentary and sports announcer talk. In this presentation, I compare multimodal communication, prosody and pitch during the real event of the goal and the goal as it is narrated on TV.
The purpose of the proposed paper – which places itself within the field of Postcolonial Critical Discourse Studies (see Esposito, E. (2021). Politics, ethnicity and the postcolonial nation – a critical analysis of political discourse in the Caribbean. John Benjamins Publishing Company) – is to analyse the multi-semiotic practices contributing to the characterisation of the protagonist of Trinidadian short-film “Doubles with Slight Pepper” as a cinematic portrayal of the hybrid Indo-Trinidadian identity. By way of a Multimodal Critical Discourse Analysis (Kress, G. and van Leeuwen, T. (1996). Reading images: the grammar of visual design. Routledge, London, Kress, G. and van Leeuwen, T. (2001). Multimodal discourse. The modes and media of contemporary communications discourse. Hodder Educations, London, Kress, G. and van Leeuwen, T. (2006). Reading images: the grammar of visual design, 2nd ed. London: Routledge; O’Halloran, K.L. (2004). Multimodal discourse analysis. Continuum, London), the study aims to cast light on the strategies exploited by the filmmaker (1) to depict the protagonist (Dhani) as an in-betweener whose ambitions are inhibited by his social status stemming from generations of subjugation and misuse by colonialists; and (2) to promote Indo-Trinidadian cultural specificities. Following an introduction to the key concept of individual and collective identity with a focus on Trinidad and Tobago, and an outline of diasporic cinema as applied to the Indo-Trinidadian community, the Multimodal Critical Discourse Analysis is carried out on the verbal, non-verbal, and visual sub-corpora gathered up in sequences according to the main three identitarian traits exhibited in the short-film: ‘religion/folklore’, ‘food’, and ‘lineage’.
(German:)
Sprache tritt immer in multimodalen Kontexten auf. Doch durch zunehmende gesellschaftliche Komplexität und dank technischer Erfindungen wird das Zusammenspiel von Sprache und anderen Modi, insbesondere technisch erzeugten stehenden und bewegten Bildern, immer enger und wirksamer. Eine Linguistik, die Sprache, Äußerungen und Texte nicht nur (wie früher meistens) an und für sich selbst, sondern in ihrem multimodalen Gebrauchskontext untersuchen möchte, steht vor erheblichen methodischen Herausforderungen. Angesichts der Fülle und Vielfalt multimodaler Daten sollte Sprachwissenschaft sich auf die Frage konzentrieren, ob und in welcher Weise Sprache eine kommunikative Arbeitsteilung mit anderen Modalitäten eingeht und wie sie dadurch in ihrer Funktion und Form beeinflusst wird. In diesem Aufsatz wird an zwei unterschiedlichen Beispielen (einem Musikvideo und einem wissenschaftlichen Erklärvideo) untersucht, in welcher Weise Sprache bei dynamischen Sehflächen andere Formen annimmt und andere Funk-tionen erfüllt als sonst üblich. Beide Beispiele sind (wie vermutlich alle dynamischen Sehflächen) mehr oder weniger intensiv ästhetisch komponiert. Dabei zeigen beide Bei-spiele, dass der Wortlaut sprachlicher Texte (Lexik und Grammatik) in Videos sehr stark auf ihre multimodale Umgebung abgestimmt ist.
(English:)
Language always occurs in multimodal contexts. However, due to increasing social complexity and thanks to technical inventions, the interplay between language and other modes, especially technically generated still and moving images, is becoming ever closer and more effective. A linguistics that wants to study language, utterances and texts not only (as was mostly the case in the past) in and of themselves, but in their multimodal context of use, faces considerable methodological challenges. In view of the abundance and diversity of multimodal data, linguistics should focus on the question of whether and how language enters into a communicative division of labor with other modalities and how this influences its function and form. This paper uses two different examples (a music video and a science explainer video) to examine the way in which language in dynamic visual surfaces takes on different forms and functions than usual elsewhere. Both examples (like presumably all dynamic visual surfaces) are more or less intensely aesthetically composed.
In the last decade, Virtual Reality (VR) has been increasingly studied and used for educational purposes (Mellet-d’Huart & Michel, 2006; Gobin-Mignot & Wolff, 2019). Whilst this technology is highly adapted for technical skills training such as flying an aircraft or performing surgery, simulations through VR with language learners seem less relevant at first sight. However, previous work has highlighted numerous potentials, especially for the development of language skills which requires a rich contextualisation to ensure the necessary input and intake processes for aural/oral skills acquisition (Pekarek Doehler & Berger, 2018). We have sought to identify professional social places virtualized by VR applications, the results of which are presented as a taxonomy for teachers in English Professional Purposes (EPP). As mainstream VR applications are mostly designed for leisure purposes, we will discuss their integration into formal learning through the predominant role of educational scenarios that some applications require if they are used in a language teaching/learning context. We will show the value of VR in supporting situated learning specific to professional situations, as well as their potential for soft skills development. The results open the floor for questions that discuss whether VR applications are indeed a complementary resource among other authentic resources used in the classroom and whether they lead to a renewal of language learning practices.
Universities in English speaking countries have been witnessing an increasing number of international faculty who speak English as an international language. However, the universities are mostly guided by the dominant monolingual ideologies and language policies that favor verbal repertoires in standard English even though instructional communication is characteristically multimodal. Against this backdrop, this article focuses on two faculty members in STEM and investigates their instructional interaction. Using multimodal interaction as a theoretical and methodological heuristic, the analysis pays attention to how modes, codes and material objects are ensembled in specific configurations as instructors and students engage in the process of meaning negotiation. The instructors’ instructional practices and perspectives demonstrate the importance of the entanglement of language with diverge semiotic, social and material elements of communication. The findings suggest a need for a broader definition and scope of instructional interaction in STEM.
In this paper, we analyse a group of 6 and 7 year olds’ interactions during a literacy event. We explore the complexities of their meaning-making following a read aloud of Where the Wild Things Are (Sendak 1963). Our focus is on discourses of gender/sex/uality, a term that acknowledges the complex relationship between gender, sex and sexuality, and how these discourses are enacted. Our guiding question was: How did discourses of gender/sex/uality circulate in this group of young children’s multimodal and playful responses to a literacy event? By considering the relationship between reader response, play and gender/sex/uality, we gained insight into how children’s responses to texts are connected to their own identities and lived experiences. We used critical multimodal discourse analysis to understand the children’s meaning-making processes. This revealed how the children were drawing from varying scripts to inform their play and creative processes. The children referenced gender/sex/uality to collaborate, to compete and to seek inclusion or status in the group. We discuss four children who drove this collective dialogue and who guided the group’s interactions. Another child’s responses pushed against and evolved in tandem with the emerging consensus. This study deepened and expanded our consciousness of children’s enactments of gender/sex/uality and how such enactments reinforced heteronormativity. The children’s artefacts, actions and talk are testimony of dominant discourses that guided and ultimately led them to adopt storylines that aligned with heteronormative scripts. Our analysis of how the children’s responses unfolded revealed how power asymmetries were reinforced and hegemonic ideologies persisted. Understanding the influences of social norms during interactive literacy events may help educators create opportunities for all learners to write themselves into these events and classroom interactions more broadly.
In this paper, I apply a metapragmatics-based approach to visual communication, combined with adapted concepts of Social Semiotics (“visual modality”) and CDA-oriented visual analysis (“canons of use”), to reconstruct two visual registers of authenticity which are prevailing within a social media photo sample of recent Austrian chancellor Sebastian Kurz (Facebook, Instagram; total of 84 photos), posted during the parliamentary elections in 2017. Triangulated with the discourse analysis of the marketing manager’s metapragmatic reflections on this social media campaign, the study shows how the partly intertwined registers of (1) “professional sensorism” (as a blended register comprising emblems of sensory modality and balanced composition, thereby drawing on the conceptualization of authenticity as sensory and affective experience of “now”) and (2) “voyeuristic fictionalization” (comprising indexicals associated with fiction genre, and based on the notion of authenticity as arising via “unnoticed observing”) are conceptualized and implemented as a—superior—visual stylization, acting as a social positioning, in mediatized political communication.
Speakers in research dissemination talks are challenged with the need to connect with an audience that does not necessarily share their knowledge and expertise. This communicative situation can be particularly challenging for speakers using English both as a foreign language and for academic purposes. This study combines multimodal and ethnographic methods to explore how speakers of dissemination talks engage with their public. It focuses on four presenters’ use and combination of language, paralanguage, kinesics, proxemics and gaze during intensive moments of engagement. The results show that these interpersonal rich points consist of dense multimodal ensembles that serve to shorten the distance between presenters and their audiences. The findings suggest that a skilful orchestration of modes can be greatly beneficial to achieve the desired level of audience engagement. Therefore, developing speakers' multimodal communicative competence should be a priority in English for specific and academic purposes (ESP/ EAP) training.
This article problematizes the notion of selecting micro-data pieces to shed light upon the focus of participants. The issue presented is two-fold: 1. The article shows that selecting micro-analytical data pieces does not allow a researcher to determine the focus of a participant; and 2. The article demonstrates that language use of a participant does not necessarily mean that the participant is focused upon a conversation. Both, purely working with micro-analytical data pieces and the presumption that language use indicates focus of the speaker, are problematized and it is shown with an example from a relatively large study of family Skype conversation that includes 82 participants that:
1. Focused attention can only be analyzed correctly when crossing micro-analytical boundaries; and
2. A participant can utilize language without paying focused attention to an interaction.
"Il y a eu une telle fertilisation réciproque des idées de la sémiotique de Saussure, centrée sur le code et le langage, et de la sémiotique inspirée par Peirce, qui est pragmatique et interprétative, qu'il est difficile de trouver aujourd'hui un sémioticien qui ne croit pas à la nécessité de développer une socio-sémiotique, interprétative et pragmatique ».
S'il fallait donner l'illustration de cette conception ouverte des avancées en sémiotique, l'ouvrage de Gunther Kress et Théo van Leeuwen : Reading Images - The Grammar of Visual Design, en serait
la meilleure preuve.
By testing a model for analysing identity in interaction, the present article explores how a history student teacher produces social identity in relation to his future profession as a teacher, with an important point of departure being the relationship between the academic and professional aspects of teacher education. This is addressed through an empirical analysis of a student teacher’s identity production in a specific academic setting: a bachelor thesis course. The main body of data consists of audio recordings and video recordings from a group of three student teachers giving feedback on each other’s theses. With respect to methodology, the article employs a model from multimodal (inter)action analysis that focuses on the concept of vertical identity – the notion that identity in interaction is produced in three layers of discourse simultaneously. The results show that the main participant produces the identity of history teacher in an academic setting where such identity production is not encouraged, e.g. by resemiotisising curricula: thus, policy documents can work as a tool when producing teacher identity. This production of identity is done by employing strong agency, which consequently points to the need of a more elaborated discussion on agency in the tested model.
Multimodality is a fast growing area of inquiry as the interest in modes beyond, but always also including language, such as layout, gesture, gaze, or body posture is increasing. A conglomeration of research from various fields interested in examining language as it is embedded in a vast array of other modes, rather than a coherent field of inquiry, this collection showcases the major achievements in this multidisciplinary field, through the inclusion of theoretically, methodologically and practically important works Topics range from language studies to education and classroom discourse, from sociolinguistic inquiries into work place practice to communication, from psycholinguistics and gesture studies to systemic functional grammatical inquiries.
article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Selection and peer-review under responsibility of the Organizing Committee of WCLTA 2013. Abstract This article examines tacit participation in an adult art class. Drawing on video excerpts from an extensive 4 month video ethnographic study of an art school, I elucidate how a new student tacitly learns to participate in the group dynamics of the art school. Through video analysis, and using a mediated discourse theoretical (Scollon, 1998, 2011) and multimodal (inter)action analytical lens (Norris, 2004, 2011), I illustrate how the learning of tacit practices is accomplished. I show how successful participation for a novice depends on the following three tenets: 1. the ability to gain focused attention (by the novice); 2. the ability to grant the novice access to shared focused attention (by expert participants); and 3. the ability and willingness of expert participants to relinquish their own focused interaction at times in order to allow the novice to learn successful participation. When these three abilities are present, a new student integrates successfully into a new classroom setting, even if the student is mediocre at art. While, if these abilities are missing, a new student will drop out of the class (in this art school), even if they are very good at art.
According to Nishida (1958) and Bourdieu (1977) social identity is embedded in cultural and social currents, constructed through social and societal histories, and internalized by the individual as habitus. Certainly, identity is constantly constructed on a micro level, where an individual’s identity is claimed, contested, and re-constructed in interaction and in relation to other participants (Scollon 1997). Such constructions are often, if not always expressed as elements through lower-level, higher-level and frozen actions (Norris, 2002, 2004). But social identity is also constructed in relation to others who may not be present at a given moment (Goffman 1959) and it is always constructed through schemes of perception.
In this chapter, I elucidate the interconnections between habitus, social identity construction, and two women’s construction of male domination. In order to demonstrate this complexity, I draw on nexus analysis and multimodal interaction analysis as my methodological frameworks (Scollon & Scollon 2004; Norris 2004); I take into consideration the performance of sequential and simultaneous actions and the (not necessarily intentional) re-production of practices on different timescales as constructed in the histories and cultures of the individuals and the societies that they belong to; and I discuss the two women’s identity and power construction, and thereby investigate what role – if any – agency plays in the process.
Visual attention can increase spatial resolution even when it leads to a decrease in performance. Whether this effect is mediated by reduction of external noise or by signal enhancement is an unsettled question. Although we previously demonstrated that attention can improve speed and accuracy in an acuity task, those experiments made use of a local postmask, which could be considered a source of external noise. In this work, a peripheral cue improved observers' abilities to indicate which side of a Landolt-square target had a gap whether or not a local postmask was used and with both central- and spread-neutral cues. In addition, we documented the presence of visual field inhomogeneities in a resolution task. Given that these experiments presented the target alone with no external noise added (i.e., without distracters or masks), our results indicate that transient attention enhanced the quality of the stimulus representation. Furthermore, because performance in the Landolt-square task indexes resolution, this attentional benefit indicates that transient attention can produce signal enhancement through finer spatial resolution.
Does attention alter appearance? This critical issue, debated for over a century, remains unsettled. From psychophysical evidence that covert attention affects early vision-it enhances contrast sensitivity and spatial resolution-and from neurophysiological evidence that attention increases the neuronal contrast sensitivity (contrast gain), one could infer that attention changes stimulus appearance. Surprisingly, few studies have directly investigated this issue. Here we developed a psychophysical method to directly assess the phenomenological correlates of attention in humans. We show that attention alters appearance; it boosts the apparent stimulus contrast. These behavioral results are consistent with neurophysiological findings suggesting that attention changes the strength of a stimulus by increasing its 'effective contrast' or salience.
Dieser Aufsatz ist in erster Linie ein theoretischer Beitrag, der mein Verständnis
von Rhythmus anhand einer internationalen Videokonferenz über eine sequenzielle
Rekonstruktion des von den Beteiligten im zeitlichen Verlauf konstituierten
Moments hinweg erlaubt. Basierend auf den soziokulturellen Theorien der mediatisierten
Diskursanalyse (Scollon 1998, 2001; Wertsch 1998; Norris/Jones
2005a) und der multimodalen (Inter-)Aktionsanalyse (Norris 2004, 2011a,
2013b; Pirini et al. 2014) beleuchtet dieser Aufsatz: (1) wie eine internationale
Videokonferenz durch rhythmische Zeitgeber (Scollon 2005) beeinflusst wird
und (2) wie die Videokonferenz durch große (Inter-)Aktionsrhythmen koordiniert
wird (Lemke 2000). Insbesondere befasst sich dieser Beitrag mit Rhythmen, die
den emergenten Rhythmen im zeitlichen Verlauf des konstituierten Moments
übergeordnet sind.
Identity construction is a widely covered topic in studies of discourse and a
topic that has interested me for some time (Norris 2002, 2004, forthcoming).
As in my other chapters, my focus in this chapter is a methodological
one that allows the investigation of identity construction from a slightly
new perspective. In this chapter, I take up the topic of personal identity
construction and illustrate what a multimodal approach can offer to grasp
such a complex, fluid and ever-changing notion. While these pages centre
around one social actor in particular, I would like to emphasize that the
reader needs to keep in mind the quote above, which alludes to the fact that
one social actor can never act alone or have a personal identity without the
collective.
My work is grounded in the methodological framework of multimodal
interaction analysis (Norris, 2004) and with this, my writing is first of all an
extension of Scollon’s (1998, 2001) mediated discourse analysis. Second,
this framework is strongly influenced by the work of Kress and van Leeuwen
in multimodality (1998, 2001; and van Leeuwen 1998). Besides these two
merging directions, the framework of multimodal interaction analysis draws
on and builds upon the micro analytical aspects found in interactional
sociolinguistics of Goffman (1959, 1961, 1974), Gumperz (1982) and
Tannen (1984); discourse analysis as in Hamilton (1996, 1998) or Schiffrin
(1994, 2005); and the macro analytical aspects of a historical approach of
Wodak et al. (2001).
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
Researchers seeking to analyse how intersubjectivity is established and maintained face significant challenges. The purpose of this article is to provide theoretical/methodological tools that begin to address these challenges. I develop these tools by applying several concepts from multimodal (inter)action analysis to an excerpt taken from the beginning of a tutoring session, drawn from a wider data set of nine one-to-one tutoring sessions. Focusing on co-produced higher-level actions as an analytic site of intersubjectivity, I show that lower-level actions that co-constitute a higher-level action can be delineated into tiers of materiality. I identify three tiers of materiality: durable, adjustable and fleeting. I introduce the theoretical/methodological tool tiers of material intersubjectivity to delineate these tiers analytically from empirical data, and show how these tiers identify a multimodal basis of material intersubjectivity. Building on this analysis I argue that the durable and adjustable tiers of material intersubjectivity produce the interactive substrate, which must be established in order for actions that display fleeting materiality to produce intersubjectivity. These theoretical/methodological tools extend the framework of multimodal (inter)action analysis, and I consider some potential applications beyond the example used here.
Multimodal discourse analysis is an emergent field that began around the verge of the millennium with books such as Reading Images: A Grammar of Visual Design (Kress & van Leeuwen, 1996/2006), Mediated Discourse as Social Interaction (Scollon, 1998), Multimodal Discourse (Kress & van Leeuwen, 2001), and Analyzing Multimodal Interaction: A Methodological Framework (Norris, 2004). Initially, multimodal discourse analysis was primarily the domain of mediated discourse analysts, social semioticians, and systemic functional linguists. While early developments were somewhat overlapping in time, these works resulted from, and aligned with, two separate major paradigm shifts stemming from previous work in discourse analysis...
The philosophical problem of an history of objects, and not only of the history of the « discovery” of an object is tackled in this theoretical article that uses an empirical example -an article by Pasteur- and Whithead’s philosophy. It explores on which conditions, according to Whithead, it would be possible to overcome the limits of « social” explanations of realism without falling back on the realism of the past.
In this article, we take a multimodal (inter)action analytical approach, showing how objects in everyday life are identity telling. As social actors surround themselves with objects, multiple actions from producing the objects to acquiring and placing them in the environment are embedded within. Here, we investigate examples from two different ethnographic studies, using the notion of frozen actions. One of our examples comes from a five-month long ethnographic study on identity production of three vegetarians in Thailand (Makboon, forthcoming); and the other example comes from a four-month long ethnographic study of three working parents on the East coast of North America (Norris, 2006, 2008). We illustrate the frozen actions embedded in particular objects and argue that the analysis of frozen actions allows us to partially understand how identity is produced and experienced by social actors in everyday life.
Pronoun comprehension is facilitated for referents that are focused in the discourse context. Discourse focus has been described as a function of attention, especially shared attention, but few studies have explicitly tested this idea. Two experiments used an exogenous capture cue paradigm to demonstrate that listeners' visual attention at the onset of a story influences their preferences during pronoun resolution later in the story. In both experiments trial-initial attention modulated listeners' transitory biases while considering referents for the pronoun, whether it was in response to the capture cue or not. These biases even had a small influence on listeners' final interpretation of the pronoun. These results provide independently motivated evidence that the listener's attention influences the online processes of pronoun comprehension. Trial-initial attentional shifts were made on the basis of non-shared, private information, demonstrating that attentional effects on pronoun comprehension are not restricted to shared attention among interlocutors.
Noam Chomsky, more than any other researcher, has radically restructured the study of human language over the past several decades. While the study of government and binding is an outgrowth of Chomsky's earlier work in transformational grammar, it represents a significant shift in focus and a new direction of investigation into the fundamentals of linguistic theory.This monograph consolidates and extends this new approach. It serves as a concise introduction to government-binding theory, applies it to several new domains of empirical data, and proposes some revisions to the principles of the theory that lead to greater unification, descriptive scope, and explanatory depth.Earlier work in the theory of grammar was concerned primarily with rule systems. The accent in government-binding theory, however, is on systems of principles of universal grammar. In the course of this book, Chomsky proposes and evaluates various general principles that limit and constrain the types of rules that are possible, and the ways they interact and function. In particular, he proposes that rule systems are in fact highly restricted in variety: only a finite number of grammars are attainable in principle, and these fall into a limited set of types.Another consequence of this shift in focus is the change of emphasis from derivations to representations. The major topic in the study of syntactic representations is the analysis of empty categories, which is a central theme of the book. After his introductory comments and a chapter on the variety of rule system, Chomsky takes up, in turn, the general properties of empty categories, the functional determination of empty categories, parasitic gaps, and binding theory and the typology of empty categories.Noam Chomsky is Institute Professor at MIT. The book is the sixth in the series Linguistic Inquiry Monographs, edited by Samuel Jay Keyser.
Earlier work suggested that "frame," "schema" and related terms could be understood as "structures of expectation." Suggesting now that there are two distinct types of structures of expectation to which such terms have been applied, we use "frames" to refer to the anthropological/sociological notion of interactive frames of interpretation, and "schema" to refer to the cognitive psychological/artificial intelligence notion of knowledge schemas. Drawing on and expanding earlier analysis of talk in a pediatric interaction, we show how frames and schemas interact. Balancing and shifting examination, consultation and management frames accounts for the burden on the pediatrician who examines a child in the mother's presence. Mismatches in the pediatrician's and mother's schemas for health and cerebral palsy account for the mother's discomfort and recalcitrant concerns, and consequently for her frequent questions which trigger the frame switches.
Building on the argument that practices between teacher and learners in classrooms may differ (Scollon and Scollon, 1981; Brice Heath, 1983 [1996]; Street, 1984; Gee, 1996; Barton and Hamilton, 1998), I look at how literacy focused school classroom teaching/learning practices instilled into an individual have a long-term effect. Using a multimodal (inter)action analytical approach (Norris, 2004, 2014) and the site of engagement as my analytical tool that brings together concrete actions, practices and discourses as a coherent whole, I examine actions, practices and discourses produced and reproduced by an art teacher and a new art student in a small private art school in Germany. While the art teacher draws on and re-produces the practice of painting, the new art student draws on and reproduces the practices and discourses that she learned in formal schooling, forcing her to produce and understand modal configurations that do not align with the creative practice that she is learning. This paper has potential educational and social ramifications as it illustrates that formal schooling may have a negative effect upon creativity by focusing the schooled individual upon results and on language/listening. These foci directly translate into modal behaviour which disadvantages the individual when trying to learn a creative practice, where the process and showing/seeing are emphasised. As the world becomes more multimodal and creative, we may want to engage in more research to rethink what and how children are taught.
Preface 2004 1. Introduction 2. Conversational Style: Theoretical Background 3. The Participants in Thanksgiving Dinner 4. Linguistics Devices in Conversational Style 5. Narrative Strategies 6. Irony and Joking 7. Summary of Style Features 8. The Study of Coherence in Discourse 9. Coda: Taking the Concepts into the Present Appendix 1: Key to Transcription Conventions Appendix 2: Steps in Analyzing Conversation Appendix 3: Participants in Thanksgiving Dinner Appendix 4: Flow of Topics in Thanksgiving Conversation REFERENCES AUTHOR INDEX SUBJECT INDEX
In this series of three lectures -shortened in one long paper- is presented the philosophical, sociological and mythical account of the links between humans and non-humans; the first section explores the notion of translation in order to give activity back to objects, the second follows empirical examples of technical systems and the third offers a mythical account of how humans and techniques co-evolved. The general purpose of the article is to show that there are many ways to escape the dualist paradigm separating humans and non-humans.
This paper describes a number of objective experiments on recognition, concerning particularly the relation between the messages received by the two ears. Rather than use steady tones or clicks (frequency or time‐point signals) continuous speech is used, and the results interpreted in the main statistically.
Two types of test are reported: (a) the behavior of a listener when presented with two speech signals simultaneously (statistical filtering problem) and (b) behavior when different speech signals are presented to his two ears.
Centering is a set of processes by which people can coordinate their attention during conversation. Speakers signal where their attention is via particular lexical, syntactic and prosodic choices. These choices make discourse entities salient to addressees and may also signal whether speakers expect addressees' attention to be already centered on the same entities. Two questions arise: What linguistic devices do speakers use to make an entity salient in a discourse? And how do speakers re-refer to discourse entities that move in and out of the center of attention? I manipulated speakers' center of attention using a videotaped basketball game. Speakers tended to refer to prominent entities as sentence subjects. When they referred to entities as sentence objects, they were more likely to re-refer next by repeating the full noun phrase verbatim rather than pronominalising. This happened even though both a recency-of-mention strategy and world knowledge would have provided enough information to uniquely identify the referent of a pronoun. When speakers pronominalised entities that had not been centered syntactically, they tended to make them salient prosodically by lengthening the duration of those pronouns.
Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences.
Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work.
Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
This article uncovers explicit simultaneous identity construction by applying Scollon and Scollon's (2001) notion of Discourse System and Multimodal Interaction Analysis (Norris, 2004a, 2004b). As a contribution to the theoretical discussion, this article investigates the micropolitics of personal national and ethnicity identity construction of Hispanic/Latino Americans in the Greater Washington DC area as a way of explicating a multimodal framework. This framework allows for the incorporation of multiple modes of communication into a discourse study, explicating how personal national and ethnicity identity can be misunderstood with far-reaching consequences. Turning towards a practical use of the theoretical knowledge, this article is relevant to societal discourses in which members from different cultural backgrounds interact, i.e. to any kind of intercultural scenario in the broadest sense. As such, the article suggests that educating diverse communities about simultaneous identity construction would result in a positive change and a possible solution to the discrepancies that can be found in communities, small groups and families.
This article identifies some limitations of discourse analysis by analyzing interactions between five boys in which the TV and the computer are featured as mediational means. The incorporation of several modalities into transcripts and a shift in focus from primarily language to human action facilitate a better understanding of the multi-modal interaction involved. The use of conventional transcripts with a focus on language demonstrates that movie- and computer-mediated interactions appear fragmented; by contrast, an inclusion of images into the transcripts, representing central interactions and/or images of a movie or computer screen, demonstrates the significant visual modes that are imperative to the ongoing talk. Just as written words correspond to the oral language, images can exemplify the global interaction among the participants, or they can represent the images on the screen. In addition, viewing an image is much faster than reading a description, so that these images also display the fast pace of the movie- and/or computer-mediated interaction.
This study develops an analysis of one site of engagement of public discourse in which identity is socially constructed. Through an analysis of the handing out of handbills in public places, the study argues that such sites of engagement are socially constructed through activities such as handing, a social situation frame in which there are expectations on appropriate behaviors, a regulatory frame of civic responsibility, and a generic frame in which the text itself implies a reader or receiver. The study argues that in such sites of engagement identities are imputed, claimed, ratified or contested and that the ascription of identity is, therefore, inherent in the activities at the sites of engagement in which this discourse takes place. Thus public discourse is argued to be inherently constitutive of social identity.
The dynamic emergence of new levels of organization in complex systems is related to the semiotic reorganization of discrete/continuous variety at the level below as continuous/discrete meaning for the level above. In this view both the semiotic and the dynamic closure of system levels is reopened to allow the development and evolution of greater complexity.
This book explores what speech, music and other sounds have in common. It gives a detailed description of the way perspective, rhythm, textual quality and other aspects of sound are used to communicate emotion and meaning. It draws on a wealth of examples from radio (disk jockey and newsreading speech, radio plays, advertising jingles, news signature tunes), film soundtracks (The Piano, The X-files, Disney animation films), music ranging from medieval plain chant to drum 'n' bass and everyday soundscapes.
Thesis (Ph. D.)--Georgetown University, 2002. Includes bibliographical references (leaves 340-349).
A new class of brightness illusions is introduced that cannot be entirely accounted for by bottom-up models of neuronal processing. In these new illusions, brightness can be modulated by the location of voluntary attention in the absence of eye movements. These effects may arise from top-down or mid-level mechanisms that determine how 3D surfaces and transparent layers are constructed, which in turn influence perceived brightness. Attention is not the only factor that influences perceived brightness in overlapping transparent surfaces. For example, grouping procedures may favor the minimal number of transparent layers necessary to account for the geometry of the stimulus, causing surfaces on a common layer to change brightness together. Attentional modulation of brightness places constraints on possible future models of filling-in, transparent surface formation, brightness perception, and attentional processing.
This chapter focuses on the effect of covert spatial attention on contrast sensitivity, a basic visual dimension where the best mechanistic understanding of attention has been achieved. I discuss how models of contrast sensitivity, as well as the confluence of psychophysical, single-unit recording, and neuroimaging studies, suggest that attention increases contrast sensitivity via contrast gain, an effect akin to a change in the physical contrast stimulus. I suggest possible research directions and ways to strengthen the interaction among different levels of analysis to further our understanding of visual attention.
If speakers articulate clearly enough to meet the perceptual needs of their listeners, clarity should depend on what listeners know about (listener-Given) rather than on what speakers know about (speaker-Given). For words excerpted from spontaneous speech, however, intelligibility to naive adult listeners showed only effects of the speaker's knowledge. Words introducing labeled map landmarks to two successive listeners were less clear on repetition even though the second listener had not heard the original mention (Experiment 1). Repeated mentions became less clear even after the listener reported inability to see the landmark (Experiment 2). Speakers were affected by what they had heard listeners mention: Intelligibility fell equally in coreferential repetitions across and within speakers (Experiment 3), whether or not the repeater could see the referent (Experiment 4). The results are explained via fast priming processes dependent on the speaker's knowledge and slow, optional processes drawing inferences about the listener's.
Chomsky's universal grammar: An introduction
- V J Cook
- M Newson
Cook, V.J. and Newson, M. 1996. Chomsky's universal grammar: An introduction. Cambridge, MA: Blackwell.
- W J M Levelt
Levelt, W.J.M. 1989. Speaking. Cambridge, MA: MIT Press.
Language, power and ideology
- R Wodak
Wodak, R. 1989. Language, power and ideology. Amsterdam: Benjamins.
- W Wu
Wu, W. 2014. Attention. London and New York: Routledge.
Timing and context in children's everyday discourse: Im plications for the study of referential and social meaning
- F Erickson
Erickson, F. 1980. Timing and context in children's everyday discourse: Im
plications for the study of referential and social meaning. Sociolinguistic
Working Paper 67(1-43). Austin, TX: Southwestern Educational Develop
ment Laboratory.