Book

Multimodal Theory and Methodology: For the Analysis of (Inter)action and Identity

Authors:

Abstract

This concise guide outlines core theoretical and methodological developments of the growing field of Multimodal (Inter)action Analysis. The volume unpacks the foundational relationship between multimodality and language and the key concepts which underpin the analysis of multimodal action and interaction and the study of multimodal identity. A focused overview of each concept charts its historical development, reviews the essential literature, and outlines its underlying theoretical frameworks and how it links to analytical tools. Norris illustrates the concept in practice via the inclusion of examples and an image-based transcript, table, or graph. The book provides a succinct overview of the latest research developments in the field of Multimodal (Inter)action Analysis for early career scholars in the field as well as established researchers looking to stay up-to-date on core developments.
... There are many new developments and perspectives. The one that we work in at the AUT Multimodal Research Centre is multimodal (inter)action analysis (MIA) (Norris 2004a(Norris , b, 2011a(Norris , 2029(Norris , 2020Geenen et al. 2015), which has grown from theory and methodology to a fully-fledged framework. I will take a moment to set out the state of this framework and discuss the role that audio-visual technology plays in MIA. ...
... On the overarching philosophical strata, MIA posits that human action, interaction and identity come about through a primacy of perception and a primacy of embodiment (Norris, 2019). On the theoretical strata, MIA follows MDA (Scollon, 1998(Scollon, , 2001, insisting on the principles of social action (including communication) and history (Norris, 2020). On this theoretical level, MIA further follows MDA by arguing that all human actions are mediated, and that mediated actions with a history are practices. ...
... Then, MIA moves on from MDA to multimodal mediated theory, noting that mediated actions appear on different levels (lower-level, higher-level and frozen) (Norris, 2004a). Multimodal mediated theory further suggests that modes are systems of mediated actions (Norris, 2013) and discourses are practices (mediated actions with a history) with an institutional and/or ideological dimension (Norris, 2020). When looking at Figure 1, we see how the philosophical and the theoretical strata seemingly sit above method and methodology. ...
Article
Full-text available
This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).
... For example, the objects present in a classroom construct the mode of layout, which gives off messages about the social actor and structure the interaction somehow. According to Norris (2004Norris ( , 2019Norris ( , 2020, the higher-level actions are also fluid and develop in real-time, and each higher-level action is bracketed by social openings and closings that are at least in part ritualized. Jewitt (2014, p. 36) holds that multimodal interaction focuses on the mediate interaction in a given context, that is, how a variety of modes are brought into and constitutive of social interaction. ...
... The researchers follow the procedures suggested by Norris (2004Norris ( , 2019Norris ( , 2020 to analyze two EFL teachers' multimodal pedagogic discourse during classroom lead-ins. According to Norris (2004, 2019, 2020), the first step to a multimodal interaction analysis is to understand an array of communicative modes. ...
Article
Full-text available
Classroom lead-in is the initial stage for motivating students to become engaged in-class interaction. However, little research, to our knowledge, has analyzed the role of teachers’ multimodal competence reflected through their multimodal pedagogic discourse in the realization of the ultimate goals of classroom lead-ins. Based on the data collected from a teaching contest in China, this paper explores how two-winner teachers utilize their multimodal ensembles of communicative modes to engage students during classroom lead-ins. The analysis shows that different communicative modes construct the higher-level action of lead-in, and they are orchestrated into multimodal ensembles for the specific function of each lead-in move. The findings indicate that EFL teachers’ high multimodal competence plays a decisive role in performing classroom lead-ins, and different lead-ins strategies influence the different orchestration of communicative modes. In constructing multimodal pedagogic discourse, teachers build up their professional image and display their personal charm as well. Future research for multimodal discourse analysis and pedagogic research is suggested in the paper.
... Another potential avenue to extend both research understandings and methods of representation that might do more justice to the multidimensional nature of emotions is multimodal analysis (e.g., Goodwin, 2000Goodwin, , 2010Goodwin, , 2018Norris, 2004Norris, , 2011Norris, , 2020. Like any area of inquiry, the breadth of this field is truly impressive, and I must admit that I am only just beginning to make my own way into exploring the landscape. ...
Book
This short-form research monograph offers a socially situated view of the emergence of emotionality for additional language (L2) learners in classroom interaction in Japan. Grounded in a complexity perspective, I argue that emotions need to be studied as they are dynamically experienced and understood in all of their multidimensional colors by individuals (in interaction). Via practitioner research, I apply a small-lens focus (Ushioda, 2016), interweaving experiential and discursive data, offering possibilities for exploring, interpreting and representing the lived experience of L2 study emotions in a more holistic yet detailed, social yet individual fashion. Amidst the currently expanding interest in L2 study emotions, the book presents a strong case for the benefits of locating interpretations of the emergence of L2 study emotions back into situated, dynamic, social context. The electronic version of this book is available open access - free to view or download - from the following URL (Routledge): https://www.routledge.com/Complexity-in-Second-Language-Study-Emotions-Emergent-Sense-making-in-Social/Sampson/p/book/9781032308449#sup
... The (inter)action multimodal framework (Norris 2004(Norris , 2019(Norris , 2020) was adopted to undertake the research. One of the main concerns of this approach is the study of social interaction. ...
Article
Full-text available
This paper examines the evolving genre of university lectures. It focuses on synchronous online lectures. The aim of the study is to shed some light on how interaction between teacher and students unfolds in large English-medium instruction (EMI) lectures in the digital context. A qualitative multimodal microanalysis of an episode of interaction was performed from an (inter)action multimodal analysis framework. This preliminary exploratory study reveals the structural and multimodal complexity of interaction in live online lectures. EMI teacher's semiotic resources combine to make meaning comprehensible in a lingua franca and to engage learners in a virtual context where there is not eye-contact with them. Suggestions are made to undertake contractive studies on interaction in online and face-to-face lectures that may respond to the need of EMI teacher training. This paper aims to contribute to the literature of this still unexplored academic instructional digital genre.
... When investigating entries in the lab book, we have drawn on the notion of modal density, which is used to analyze when and how modes structure attention toward and away from higher level actions, including those "beyond the focus" (Norris, 2016(Norris, , p. 153, 2019(Norris, , 2020, for instance, how medical physicists engage in acceptable research practices of their community. ...
Article
Full-text available
Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation.
... Multimodal (inter)action analysis (Norris, 2004(Norris, , 2019(Norris, , 2020 is "a holistic analytical framework that understands the multiple modes in (inter)action as all together building one system of communication" (Norris & Pirini, 2016: 24). Within this framework, all actions are considered interactions between social actors and other social actors, objects, or the environment. ...
Article
Online language teaching is gaining momentum worldwide and an expanding body of research analyses online pedagogical interactions. However, few studies have explored experienced online teachers' practices in videoconferencing particularly while giving instructions, which are key to success in task-based language teaching (Markee, 2015). Adopting multimodal (inter)action analysis (Norris, 2004, 2019) to investigate the multimodal construction of instructions in a single case study, we examine instruction-giving as a social practice demonstrated in a specific site of engagement (a synchronous online lesson recorded for research purposes). Drawing on the higher-level actions (instruction-giving fragments) we have identified elsewhere (Satar & Wigham, 2020), in this paper we analyse the lower-level actions (modes) that comprise these higher-level actions, specifically focusing on the print mode (task resource sheets, URLs, text chat, and online collaborative writing spaces) wherein certain higher-level actions become frozen. Our findings are unique in depicting the modal complexity of sharing task resources in synchronous online teaching due to semiotic misalignment and semiotic lag that precludes the establishment of a completely shared interactional space. We observe gaze shifts as the sole indicator for learners that the teacher is multitasking between different higher-level actions. Further research is needed to fully understand the interactional features of online language teaching via videoconferencing to inform teacher training policy and practice.
Article
The purpose of the proposed paper – which places itself within the field of Postcolonial Critical Discourse Studies (see Esposito, E. (2021). Politics, ethnicity and the postcolonial nation – a critical analysis of political discourse in the Caribbean. John Benjamins Publishing Company) – is to analyse the multi-semiotic practices contributing to the characterisation of the protagonist of Trinidadian short-film “Doubles with Slight Pepper” as a cinematic portrayal of the hybrid Indo-Trinidadian identity. By way of a Multimodal Critical Discourse Analysis (Kress, G. and van Leeuwen, T. (1996). Reading images: the grammar of visual design. Routledge, London, Kress, G. and van Leeuwen, T. (2001). Multimodal discourse. The modes and media of contemporary communications discourse. Hodder Educations, London, Kress, G. and van Leeuwen, T. (2006). Reading images: the grammar of visual design, 2nd ed. London: Routledge; O’Halloran, K.L. (2004). Multimodal discourse analysis. Continuum, London), the study aims to cast light on the strategies exploited by the filmmaker (1) to depict the protagonist (Dhani) as an in-betweener whose ambitions are inhibited by his social status stemming from generations of subjugation and misuse by colonialists; and (2) to promote Indo-Trinidadian cultural specificities. Following an introduction to the key concept of individual and collective identity with a focus on Trinidad and Tobago, and an outline of diasporic cinema as applied to the Indo-Trinidadian community, the Multimodal Critical Discourse Analysis is carried out on the verbal, non-verbal, and visual sub-corpora gathered up in sequences according to the main three identitarian traits exhibited in the short-film: ‘religion/folklore’, ‘food’, and ‘lineage’.
Article
Full-text available
(German:) Sprache tritt immer in multimodalen Kontexten auf. Doch durch zunehmende gesellschaftliche Komplexität und dank technischer Erfindungen wird das Zusammenspiel von Sprache und anderen Modi, insbesondere technisch erzeugten stehenden und bewegten Bildern, immer enger und wirksamer. Eine Linguistik, die Sprache, Äußerungen und Texte nicht nur (wie früher meistens) an und für sich selbst, sondern in ihrem multimodalen Gebrauchskontext untersuchen möchte, steht vor erheblichen methodischen Herausforderungen. Angesichts der Fülle und Vielfalt multimodaler Daten sollte Sprachwissenschaft sich auf die Frage konzentrieren, ob und in welcher Weise Sprache eine kommunikative Arbeitsteilung mit anderen Modalitäten eingeht und wie sie dadurch in ihrer Funktion und Form beeinflusst wird. In diesem Aufsatz wird an zwei unterschiedlichen Beispielen (einem Musikvideo und einem wissenschaftlichen Erklärvideo) untersucht, in welcher Weise Sprache bei dynamischen Sehflächen andere Formen annimmt und andere Funk-tionen erfüllt als sonst üblich. Beide Beispiele sind (wie vermutlich alle dynamischen Sehflächen) mehr oder weniger intensiv ästhetisch komponiert. Dabei zeigen beide Bei-spiele, dass der Wortlaut sprachlicher Texte (Lexik und Grammatik) in Videos sehr stark auf ihre multimodale Umgebung abgestimmt ist. (English:) Language always occurs in multimodal contexts. However, due to increasing social complexity and thanks to technical inventions, the interplay between language and other modes, especially technically generated still and moving images, is becoming ever closer and more effective. A linguistics that wants to study language, utterances and texts not only (as was mostly the case in the past) in and of themselves, but in their multimodal context of use, faces considerable methodological challenges. In view of the abundance and diversity of multimodal data, linguistics should focus on the question of whether and how language enters into a communicative division of labor with other modalities and how this influences its function and form. This paper uses two different examples (a music video and a science explainer video) to examine the way in which language in dynamic visual surfaces takes on different forms and functions than usual elsewhere. Both examples (like presumably all dynamic visual surfaces) are more or less intensely aesthetically composed.
Conference Paper
Full-text available
In the last decade, Virtual Reality (VR) has been increasingly studied and used for educational purposes (Mellet-d’Huart & Michel, 2006; Gobin-Mignot & Wolff, 2019). Whilst this technology is highly adapted for technical skills training such as flying an aircraft or performing surgery, simulations through VR with language learners seem less relevant at first sight. However, previous work has highlighted numerous potentials, especially for the development of language skills which requires a rich contextualisation to ensure the necessary input and intake processes for aural/oral skills acquisition (Pekarek Doehler & Berger, 2018). We have sought to identify professional social places virtualized by VR applications, the results of which are presented as a taxonomy for teachers in English Professional Purposes (EPP). As mainstream VR applications are mostly designed for leisure purposes, we will discuss their integration into formal learning through the predominant role of educational scenarios that some applications require if they are used in a language teaching/learning context. We will show the value of VR in supporting situated learning specific to professional situations, as well as their potential for soft skills development. The results open the floor for questions that discuss whether VR applications are indeed a complementary resource among other authentic resources used in the classroom and whether they lead to a renewal of language learning practices.
Article
Universities in English speaking countries have been witnessing an increasing number of international faculty who speak English as an international language. However, the universities are mostly guided by the dominant monolingual ideologies and language policies that favor verbal repertoires in standard English even though instructional communication is characteristically multimodal. Against this backdrop, this article focuses on two faculty members in STEM and investigates their instructional interaction. Using multimodal interaction as a theoretical and methodological heuristic, the analysis pays attention to how modes, codes and material objects are ensembled in specific configurations as instructors and students engage in the process of meaning negotiation. The instructors’ instructional practices and perspectives demonstrate the importance of the entanglement of language with diverge semiotic, social and material elements of communication. The findings suggest a need for a broader definition and scope of instructional interaction in STEM.
Article
In this paper, we analyse a group of 6 and 7 year olds’ interactions during a literacy event. We explore the complexities of their meaning-making following a read aloud of Where the Wild Things Are (Sendak 1963). Our focus is on discourses of gender/sex/uality, a term that acknowledges the complex relationship between gender, sex and sexuality, and how these discourses are enacted. Our guiding question was: How did discourses of gender/sex/uality circulate in this group of young children’s multimodal and playful responses to a literacy event? By considering the relationship between reader response, play and gender/sex/uality, we gained insight into how children’s responses to texts are connected to their own identities and lived experiences. We used critical multimodal discourse analysis to understand the children’s meaning-making processes. This revealed how the children were drawing from varying scripts to inform their play and creative processes. The children referenced gender/sex/uality to collaborate, to compete and to seek inclusion or status in the group. We discuss four children who drove this collective dialogue and who guided the group’s interactions. Another child’s responses pushed against and evolved in tandem with the emerging consensus. This study deepened and expanded our consciousness of children’s enactments of gender/sex/uality and how such enactments reinforced heteronormativity. The children’s artefacts, actions and talk are testimony of dominant discourses that guided and ultimately led them to adopt storylines that aligned with heteronormative scripts. Our analysis of how the children’s responses unfolded revealed how power asymmetries were reinforced and hegemonic ideologies persisted. Understanding the influences of social norms during interactive literacy events may help educators create opportunities for all learners to write themselves into these events and classroom interactions more broadly.
Article
Full-text available
In this paper, I apply a metapragmatics-based approach to visual communication, combined with adapted concepts of Social Semiotics (“visual modality”) and CDA-oriented visual analysis (“canons of use”), to reconstruct two visual registers of authenticity which are prevailing within a social media photo sample of recent Austrian chancellor Sebastian Kurz (Facebook, Instagram; total of 84 photos), posted during the parliamentary elections in 2017. Triangulated with the discourse analysis of the marketing manager’s metapragmatic reflections on this social media campaign, the study shows how the partly intertwined registers of (1) “professional sensorism” (as a blended register comprising emblems of sensory modality and balanced composition, thereby drawing on the conceptualization of authenticity as sensory and affective experience of “now”) and (2) “voyeuristic fictionalization” (comprising indexicals associated with fiction genre, and based on the notion of authenticity as arising via “unnoticed observing”) are conceptualized and implemented as a—superior—visual stylization, acting as a social positioning, in mediatized political communication.
Article
Speakers in research dissemination talks are challenged with the need to connect with an audience that does not necessarily share their knowledge and expertise. This communicative situation can be particularly challenging for speakers using English both as a foreign language and for academic purposes. This study combines multimodal and ethnographic methods to explore how speakers of dissemination talks engage with their public. It focuses on four presenters’ use and combination of language, paralanguage, kinesics, proxemics and gaze during intensive moments of engagement. The results show that these interpersonal rich points consist of dense multimodal ensembles that serve to shorten the distance between presenters and their audiences. The findings suggest that a skilful orchestration of modes can be greatly beneficial to achieve the desired level of audience engagement. Therefore, developing speakers' multimodal communicative competence should be a priority in English for specific and academic purposes (ESP/ EAP) training.
Chapter
Full-text available
This article problematizes the notion of selecting micro-data pieces to shed light upon the focus of participants. The issue presented is two-fold: 1. The article shows that selecting micro-analytical data pieces does not allow a researcher to determine the focus of a participant; and 2. The article demonstrates that language use of a participant does not necessarily mean that the participant is focused upon a conversation. Both, purely working with micro-analytical data pieces and the presumption that language use indicates focus of the speaker, are problematized and it is shown with an example from a relatively large study of family Skype conversation that includes 82 participants that: 1. Focused attention can only be analyzed correctly when crossing micro-analytical boundaries; and 2. A participant can utilize language without paying focused attention to an interaction.
Article
Full-text available
"Il y a eu une telle fertilisation réciproque des idées de la sémiotique de Saussure, centrée sur le code et le langage, et de la sémiotique inspirée par Peirce, qui est pragmatique et interprétative, qu'il est difficile de trouver aujourd'hui un sémioticien qui ne croit pas à la nécessité de développer une socio-sémiotique, interprétative et pragmatique ». S'il fallait donner l'illustration de cette conception ouverte des avancées en sémiotique, l'ouvrage de Gunther Kress et Théo van Leeuwen : Reading Images - The Grammar of Visual Design, en serait la meilleure preuve.
Article
Full-text available
By testing a model for analysing identity in interaction, the present article explores how a history student teacher produces social identity in relation to his future profession as a teacher, with an important point of departure being the relationship between the academic and professional aspects of teacher education. This is addressed through an empirical analysis of a student teacher’s identity production in a specific academic setting: a bachelor thesis course. The main body of data consists of audio recordings and video recordings from a group of three student teachers giving feedback on each other’s theses. With respect to methodology, the article employs a model from multimodal (inter)action analysis that focuses on the concept of vertical identity – the notion that identity in interaction is produced in three layers of discourse simultaneously. The results show that the main participant produces the identity of history teacher in an academic setting where such identity production is not encouraged, e.g. by resemiotisising curricula: thus, policy documents can work as a tool when producing teacher identity. This production of identity is done by employing strong agency, which consequently points to the need of a more elaborated discussion on agency in the tested model.
Book
Full-text available
Multimodality is a fast growing area of inquiry as the interest in modes beyond, but always also including language, such as layout, gesture, gaze, or body posture is increasing. A conglomeration of research from various fields interested in examining language as it is embedded in a vast array of other modes, rather than a coherent field of inquiry, this collection showcases the major achievements in this multidisciplinary field, through the inclusion of theoretically, methodologically and practically important works Topics range from language studies to education and classroom discourse, from sociolinguistic inquiries into work place practice to communication, from psycholinguistics and gesture studies to systemic functional grammatical inquiries.
Article
Full-text available
article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Selection and peer-review under responsibility of the Organizing Committee of WCLTA 2013. Abstract This article examines tacit participation in an adult art class. Drawing on video excerpts from an extensive 4 month video ethnographic study of an art school, I elucidate how a new student tacitly learns to participate in the group dynamics of the art school. Through video analysis, and using a mediated discourse theoretical (Scollon, 1998, 2011) and multimodal (inter)action analytical lens (Norris, 2004, 2011), I illustrate how the learning of tacit practices is accomplished. I show how successful participation for a novice depends on the following three tenets: 1. the ability to gain focused attention (by the novice); 2. the ability to grant the novice access to shared focused attention (by expert participants); and 3. the ability and willingness of expert participants to relinquish their own focused interaction at times in order to allow the novice to learn successful participation. When these three abilities are present, a new student integrates successfully into a new classroom setting, even if the student is mediocre at art. While, if these abilities are missing, a new student will drop out of the class (in this art school), even if they are very good at art.
Article
Full-text available
Visual attention can increase spatial resolution even when it leads to a decrease in performance. Whether this effect is mediated by reduction of external noise or by signal enhancement is an unsettled question. Although we previously demonstrated that attention can improve speed and accuracy in an acuity task, those experiments made use of a local postmask, which could be considered a source of external noise. In this work, a peripheral cue improved observers' abilities to indicate which side of a Landolt-square target had a gap whether or not a local postmask was used and with both central- and spread-neutral cues. In addition, we documented the presence of visual field inhomogeneities in a resolution task. Given that these experiments presented the target alone with no external noise added (i.e., without distracters or masks), our results indicate that transient attention enhanced the quality of the stimulus representation. Furthermore, because performance in the Landolt-square task indexes resolution, this attentional benefit indicates that transient attention can produce signal enhancement through finer spatial resolution.
Article
Full-text available
Does attention alter appearance? This critical issue, debated for over a century, remains unsettled. From psychophysical evidence that covert attention affects early vision-it enhances contrast sensitivity and spatial resolution-and from neurophysiological evidence that attention increases the neuronal contrast sensitivity (contrast gain), one could infer that attention changes stimulus appearance. Surprisingly, few studies have directly investigated this issue. Here we developed a psychophysical method to directly assess the phenomenological correlates of attention in humans. We show that attention alters appearance; it boosts the apparent stimulus contrast. These behavioral results are consistent with neurophysiological findings suggesting that attention changes the strength of a stimulus by increasing its 'effective contrast' or salience.
Chapter
Dieser Aufsatz ist in erster Linie ein theoretischer Beitrag, der mein Verständnis von Rhythmus anhand einer internationalen Videokonferenz über eine sequenzielle Rekonstruktion des von den Beteiligten im zeitlichen Verlauf konstituierten Moments hinweg erlaubt. Basierend auf den soziokulturellen Theorien der mediatisierten Diskursanalyse (Scollon 1998, 2001; Wertsch 1998; Norris/Jones 2005a) und der multimodalen (Inter-)Aktionsanalyse (Norris 2004, 2011a, 2013b; Pirini et al. 2014) beleuchtet dieser Aufsatz: (1) wie eine internationale Videokonferenz durch rhythmische Zeitgeber (Scollon 2005) beeinflusst wird und (2) wie die Videokonferenz durch große (Inter-)Aktionsrhythmen koordiniert wird (Lemke 2000). Insbesondere befasst sich dieser Beitrag mit Rhythmen, die den emergenten Rhythmen im zeitlichen Verlauf des konstituierten Moments übergeordnet sind.
Chapter
Identity construction is a widely covered topic in studies of discourse and a topic that has interested me for some time (Norris 2002, 2004, forthcoming). As in my other chapters, my focus in this chapter is a methodological one that allows the investigation of identity construction from a slightly new perspective. In this chapter, I take up the topic of personal identity construction and illustrate what a multimodal approach can offer to grasp such a complex, fluid and ever-changing notion. While these pages centre around one social actor in particular, I would like to emphasize that the reader needs to keep in mind the quote above, which alludes to the fact that one social actor can never act alone or have a personal identity without the collective. My work is grounded in the methodological framework of multimodal interaction analysis (Norris, 2004) and with this, my writing is first of all an extension of Scollon’s (1998, 2001) mediated discourse analysis. Second, this framework is strongly influenced by the work of Kress and van Leeuwen in multimodality (1998, 2001; and van Leeuwen 1998). Besides these two merging directions, the framework of multimodal interaction analysis draws on and builds upon the micro analytical aspects found in interactional sociolinguistics of Goffman (1959, 1961, 1974), Gumperz (1982) and Tannen (1984); discourse analysis as in Hamilton (1996, 1998) or Schiffrin (1994, 2005); and the macro analytical aspects of a historical approach of Wodak et al. (2001).
Article
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
Article
Researchers seeking to analyse how intersubjectivity is established and maintained face significant challenges. The purpose of this article is to provide theoretical/methodological tools that begin to address these challenges. I develop these tools by applying several concepts from multimodal (inter)action analysis to an excerpt taken from the beginning of a tutoring session, drawn from a wider data set of nine one-to-one tutoring sessions. Focusing on co-produced higher-level actions as an analytic site of intersubjectivity, I show that lower-level actions that co-constitute a higher-level action can be delineated into tiers of materiality. I identify three tiers of materiality: durable, adjustable and fleeting. I introduce the theoretical/methodological tool tiers of material intersubjectivity to delineate these tiers analytically from empirical data, and show how these tiers identify a multimodal basis of material intersubjectivity. Building on this analysis I argue that the durable and adjustable tiers of material intersubjectivity produce the interactive substrate, which must be established in order for actions that display fleeting materiality to produce intersubjectivity. These theoretical/methodological tools extend the framework of multimodal (inter)action analysis, and I consider some potential applications beyond the example used here.
Chapter
Multimodal discourse analysis is an emergent field that began around the verge of the millennium with books such as Reading Images: A Grammar of Visual Design (Kress & van Leeuwen, 1996/2006), Mediated Discourse as Social Interaction (Scollon, 1998), Multimodal Discourse (Kress & van Leeuwen, 2001), and Analyzing Multimodal Interaction: A Methodological Framework (Norris, 2004). Initially, multimodal discourse analysis was primarily the domain of mediated discourse analysts, social semioticians, and systemic functional linguists. While early developments were somewhat overlapping in time, these works resulted from, and aligned with, two separate major paradigm shifts stemming from previous work in discourse analysis...
Article
The philosophical problem of an history of objects, and not only of the history of the « discovery” of an object is tackled in this theoretical article that uses an empirical example -an article by Pasteur- and Whithead’s philosophy. It explores on which conditions, according to Whithead, it would be possible to overcome the limits of « social” explanations of realism without falling back on the realism of the past.
Article
In this article, we take a multimodal (inter)action analytical approach, showing how objects in everyday life are identity telling. As social actors surround themselves with objects, multiple actions from producing the objects to acquiring and placing them in the environment are embedded within. Here, we investigate examples from two different ethnographic studies, using the notion of frozen actions. One of our examples comes from a five-month long ethnographic study on identity production of three vegetarians in Thailand (Makboon, forthcoming); and the other example comes from a four-month long ethnographic study of three working parents on the East coast of North America (Norris, 2006, 2008). We illustrate the frozen actions embedded in particular objects and argue that the analysis of frozen actions allows us to partially understand how identity is produced and experienced by social actors in everyday life.
Article
Pronoun comprehension is facilitated for referents that are focused in the discourse context. Discourse focus has been described as a function of attention, especially shared attention, but few studies have explicitly tested this idea. Two experiments used an exogenous capture cue paradigm to demonstrate that listeners' visual attention at the onset of a story influences their preferences during pronoun resolution later in the story. In both experiments trial-initial attention modulated listeners' transitory biases while considering referents for the pronoun, whether it was in response to the capture cue or not. These biases even had a small influence on listeners' final interpretation of the pronoun. These results provide independently motivated evidence that the listener's attention influences the online processes of pronoun comprehension. Trial-initial attentional shifts were made on the basis of non-shared, private information, demonstrating that attentional effects on pronoun comprehension are not restricted to shared attention among interlocutors.
Article
Noam Chomsky, more than any other researcher, has radically restructured the study of human language over the past several decades. While the study of government and binding is an outgrowth of Chomsky's earlier work in transformational grammar, it represents a significant shift in focus and a new direction of investigation into the fundamentals of linguistic theory.This monograph consolidates and extends this new approach. It serves as a concise introduction to government-binding theory, applies it to several new domains of empirical data, and proposes some revisions to the principles of the theory that lead to greater unification, descriptive scope, and explanatory depth.Earlier work in the theory of grammar was concerned primarily with rule systems. The accent in government-binding theory, however, is on systems of principles of universal grammar. In the course of this book, Chomsky proposes and evaluates various general principles that limit and constrain the types of rules that are possible, and the ways they interact and function. In particular, he proposes that rule systems are in fact highly restricted in variety: only a finite number of grammars are attainable in principle, and these fall into a limited set of types.Another consequence of this shift in focus is the change of emphasis from derivations to representations. The major topic in the study of syntactic representations is the analysis of empty categories, which is a central theme of the book. After his introductory comments and a chapter on the variety of rule system, Chomsky takes up, in turn, the general properties of empty categories, the functional determination of empty categories, parasitic gaps, and binding theory and the typology of empty categories.Noam Chomsky is Institute Professor at MIT. The book is the sixth in the series Linguistic Inquiry Monographs, edited by Samuel Jay Keyser.
Article
Earlier work suggested that "frame," "schema" and related terms could be understood as "structures of expectation." Suggesting now that there are two distinct types of structures of expectation to which such terms have been applied, we use "frames" to refer to the anthropological/sociological notion of interactive frames of interpretation, and "schema" to refer to the cognitive psychological/artificial intelligence notion of knowledge schemas. Drawing on and expanding earlier analysis of talk in a pediatric interaction, we show how frames and schemas interact. Balancing and shifting examination, consultation and management frames accounts for the burden on the pediatrician who examines a child in the mother's presence. Mismatches in the pediatrician's and mother's schemas for health and cerebral palsy account for the mother's discomfort and recalcitrant concerns, and consequently for her frequent questions which trigger the frame switches.
Article
Building on the argument that practices between teacher and learners in classrooms may differ (Scollon and Scollon, 1981; Brice Heath, 1983 [1996]; Street, 1984; Gee, 1996; Barton and Hamilton, 1998), I look at how literacy focused school classroom teaching/learning practices instilled into an individual have a long-term effect. Using a multimodal (inter)action analytical approach (Norris, 2004, 2014) and the site of engagement as my analytical tool that brings together concrete actions, practices and discourses as a coherent whole, I examine actions, practices and discourses produced and reproduced by an art teacher and a new art student in a small private art school in Germany. While the art teacher draws on and re-produces the practice of painting, the new art student draws on and reproduces the practices and discourses that she learned in formal schooling, forcing her to produce and understand modal configurations that do not align with the creative practice that she is learning. This paper has potential educational and social ramifications as it illustrates that formal schooling may have a negative effect upon creativity by focusing the schooled individual upon results and on language/listening. These foci directly translate into modal behaviour which disadvantages the individual when trying to learn a creative practice, where the process and showing/seeing are emphasised. As the world becomes more multimodal and creative, we may want to engage in more research to rethink what and how children are taught.
Book
Preface 2004 1. Introduction 2. Conversational Style: Theoretical Background 3. The Participants in Thanksgiving Dinner 4. Linguistics Devices in Conversational Style 5. Narrative Strategies 6. Irony and Joking 7. Summary of Style Features 8. The Study of Coherence in Discourse 9. Coda: Taking the Concepts into the Present Appendix 1: Key to Transcription Conventions Appendix 2: Steps in Analyzing Conversation Appendix 3: Participants in Thanksgiving Dinner Appendix 4: Flow of Topics in Thanksgiving Conversation REFERENCES AUTHOR INDEX SUBJECT INDEX
Article
In this series of three lectures -shortened in one long paper- is presented the philosophical, sociological and mythical account of the links between humans and non-humans; the first section explores the notion of translation in order to give activity back to objects, the second follows empirical examples of technical systems and the third offers a mythical account of how humans and techniques co-evolved. The general purpose of the article is to show that there are many ways to escape the dualist paradigm separating humans and non-humans.
Article
This paper describes a number of objective experiments on recognition, concerning particularly the relation between the messages received by the two ears. Rather than use steady tones or clicks (frequency or time‐point signals) continuous speech is used, and the results interpreted in the main statistically. Two types of test are reported: (a) the behavior of a listener when presented with two speech signals simultaneously (statistical filtering problem) and (b) behavior when different speech signals are presented to his two ears.
Article
Centering is a set of processes by which people can coordinate their attention during conversation. Speakers signal where their attention is via particular lexical, syntactic and prosodic choices. These choices make discourse entities salient to addressees and may also signal whether speakers expect addressees' attention to be already centered on the same entities. Two questions arise: What linguistic devices do speakers use to make an entity salient in a discourse? And how do speakers re-refer to discourse entities that move in and out of the center of attention? I manipulated speakers' center of attention using a videotaped basketball game. Speakers tended to refer to prominent entities as sentence subjects. When they referred to entities as sentence objects, they were more likely to re-refer next by repeating the full noun phrase verbatim rather than pronominalising. This happened even though both a recency-of-mention strategy and world knowledge would have provided enough information to uniquely identify the referent of a pronoun. When speakers pronominalised entities that had not been centered syntactically, they tended to make them salient prosodically by lengthening the duration of those pronouns.
Chapter
According to Nishida (1958) and Bourdieu (1977) social identity is embedded in cultural and social currents, constructed through social and societal histories, and internalized by the individual as habitus. Certainly, identity is constantly constructed on a micro level, where an individual’s identity is claimed, contested, and re-constructed in interaction and in relation to other participants (Scollon 1997). Such constructions are often, if not always expressed as elements through lower-level, higher-level and frozen actions (Norris, 2002, 2004). But social identity is also constructed in relation to others who may not be present at a given moment (Goffman 1959) and it is always constructed through schemes of perception. In this chapter, I elucidate the interconnections between habitus, social identity construction, and two women’s construction of male domination. In order to demonstrate this complexity, I draw on nexus analysis and multimodal interaction analysis as my methodological frameworks (Scollon & Scollon 2004; Norris 2004); I take into consideration the performance of sequential and simultaneous actions and the (not necessarily intentional) re-production of practices on different timescales as constructed in the histories and cultures of the individuals and the societies that they belong to; and I discuss the two women’s identity and power construction, and thereby investigate what role – if any – agency plays in the process.
Book
Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences. Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work. Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
Article
This article uncovers explicit simultaneous identity construction by applying Scollon and Scollon's (2001) notion of Discourse System and Multimodal Interaction Analysis (Norris, 2004a, 2004b). As a contribution to the theoretical discussion, this article investigates the micropolitics of personal national and ethnicity identity construction of Hispanic/Latino Americans in the Greater Washington DC area as a way of explicating a multimodal framework. This framework allows for the incorporation of multiple modes of communication into a discourse study, explicating how personal national and ethnicity identity can be misunderstood with far-reaching consequences. Turning towards a practical use of the theoretical knowledge, this article is relevant to societal discourses in which members from different cultural backgrounds interact, i.e. to any kind of intercultural scenario in the broadest sense. As such, the article suggests that educating diverse communities about simultaneous identity construction would result in a positive change and a possible solution to the discrepancies that can be found in communities, small groups and families.
Article
This article identifies some limitations of discourse analysis by analyzing interactions between five boys in which the TV and the computer are featured as mediational means. The incorporation of several modalities into transcripts and a shift in focus from primarily language to human action facilitate a better understanding of the multi-modal interaction involved. The use of conventional transcripts with a focus on language demonstrates that movie- and computer-mediated interactions appear fragmented; by contrast, an inclusion of images into the transcripts, representing central interactions and/or images of a movie or computer screen, demonstrates the significant visual modes that are imperative to the ongoing talk. Just as written words correspond to the oral language, images can exemplify the global interaction among the participants, or they can represent the images on the screen. In addition, viewing an image is much faster than reading a description, so that these images also display the fast pace of the movie- and/or computer-mediated interaction.
Article
This study develops an analysis of one site of engagement of public discourse in which identity is socially constructed. Through an analysis of the handing out of handbills in public places, the study argues that such sites of engagement are socially constructed through activities such as handing, a social situation frame in which there are expectations on appropriate behaviors, a regulatory frame of civic responsibility, and a generic frame in which the text itself implies a reader or receiver. The study argues that in such sites of engagement identities are imputed, claimed, ratified or contested and that the ascription of identity is, therefore, inherent in the activities at the sites of engagement in which this discourse takes place. Thus public discourse is argued to be inherently constitutive of social identity.
Article
The dynamic emergence of new levels of organization in complex systems is related to the semiotic reorganization of discrete/continuous variety at the level below as continuous/discrete meaning for the level above. In this view both the semiotic and the dynamic closure of system levels is reopened to allow the development and evolution of greater complexity.
Article
Thesis (Ph. D.)--Georgetown University, 2002. Includes bibliographical references (leaves 340-349).
Article
A new class of brightness illusions is introduced that cannot be entirely accounted for by bottom-up models of neuronal processing. In these new illusions, brightness can be modulated by the location of voluntary attention in the absence of eye movements. These effects may arise from top-down or mid-level mechanisms that determine how 3D surfaces and transparent layers are constructed, which in turn influence perceived brightness. Attention is not the only factor that influences perceived brightness in overlapping transparent surfaces. For example, grouping procedures may favor the minimal number of transparent layers necessary to account for the geometry of the stimulus, causing surfaces on a common layer to change brightness together. Attentional modulation of brightness places constraints on possible future models of filling-in, transparent surface formation, brightness perception, and attentional processing.
Article
This chapter focuses on the effect of covert spatial attention on contrast sensitivity, a basic visual dimension where the best mechanistic understanding of attention has been achieved. I discuss how models of contrast sensitivity, as well as the confluence of psychophysical, single-unit recording, and neuroimaging studies, suggest that attention increases contrast sensitivity via contrast gain, an effect akin to a change in the physical contrast stimulus. I suggest possible research directions and ways to strengthen the interaction among different levels of analysis to further our understanding of visual attention.
Article
If speakers articulate clearly enough to meet the perceptual needs of their listeners, clarity should depend on what listeners know about (listener-Given) rather than on what speakers know about (speaker-Given). For words excerpted from spontaneous speech, however, intelligibility to naive adult listeners showed only effects of the speaker's knowledge. Words introducing labeled map landmarks to two successive listeners were less clear on repetition even though the second listener had not heard the original mention (Experiment 1). Repeated mentions became less clear even after the listener reported inability to see the landmark (Experiment 2). Speakers were affected by what they had heard listeners mention: Intelligibility fell equally in coreferential repetitions across and within speakers (Experiment 3), whether or not the repeater could see the referent (Experiment 4). The results are explained via fast priming processes dependent on the speaker's knowledge and slow, optional processes drawing inferences about the listener's.
Chomsky's universal grammar: An introduction
  • V J Cook
  • M Newson
Cook, V.J. and Newson, M. 1996. Chomsky's universal grammar: An introduction. Cambridge, MA: Blackwell.
  • W J M Levelt
Levelt, W.J.M. 1989. Speaking. Cambridge, MA: MIT Press.
Language, power and ideology
  • R Wodak
Wodak, R. 1989. Language, power and ideology. Amsterdam: Benjamins.
  • W Wu
Wu, W. 2014. Attention. London and New York: Routledge.
Timing and context in children's everyday discourse: Im plications for the study of referential and social meaning
  • F Erickson
Erickson, F. 1980. Timing and context in children's everyday discourse: Im plications for the study of referential and social meaning. Sociolinguistic Working Paper 67(1-43). Austin, TX: Southwestern Educational Develop ment Laboratory.