No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Three hierarchical positions of deictic gesture in relation to spoken language: A multimodal interaction analysis
Abstract
Taking the action, rather than the utterance or the text, as the unit of analysis, this article isolates different modes, investigating the interdependent relationships, illustrating that the visual mode of gestures can take up a hierarchically equal or a super-ordinate position in addition to the commonly understood sub-ordinate position in relation to the mode of spoken language. Building on McNeill, Birdwhistell, Eco, and Ekman and Friesen, and using a multimodal interaction analytical approach (Norris), I analyse in detail three separate everyday (inter)actions in which a deictic gesture is being performed and spoken language is used by the social actor performing the gesture. With these examples, I build on previous work in multimodal analysis of texts and multimodal interaction analysis, illustrating that the verbal is not necessarily more important than the visual (Kress and Van Leeuwen; Norris; Scollon), demonstrating that verbal and visual modes can be utilized together to (co)produce one message (Van Leeuwen), and showing that a mode utilized by a social actor producing a higher-level discourse structure hierarchically supersedes other modes in interaction (Norris).
... This can happen at two levels. The first is when gesture is sub-ordinate to spoken words (Norris, 2011). In this situation gesture builds up one system together with spoken words, but if the gesture is omitted it does not render the spoken words meaningless. ...
... The rest of the context often clarifies the intentions of the speaker. The second is when gesture and spoken words are on equal level (Norris, 2011). The omission of the gesture in this situation may affect communication. ...
... The globe gesture here was at a sub-ordinate level to spoken words (Norris, 2011). The team would still be able to understand the meaning of Jarrod's words without the globe gesture. ...
... For example, according to Norris (2004), a multimodality scholar in the Mediated Discourse Analysis strand, 'modes' (rather than 'modalities', see below) are semiotic resources such as speech, gestures, posture, head movements, dress and so on. Norris (2011) states that the hierarchical structures of modes during interactions cannot be assumed, indeed gesture can be equal, subordinate or superordinate to other modes, which means that spoken language is not always vital to the (inter)action as a whole ( Norris, 2011;also see Kendon, 2004). While I am interested in the interplay between body posture, head movements, object handling and so on, this article mostly focuses on the interplay between (features of) spoken languages and gestures. ...
... For example, according to Norris (2004), a multimodality scholar in the Mediated Discourse Analysis strand, 'modes' (rather than 'modalities', see below) are semiotic resources such as speech, gestures, posture, head movements, dress and so on. Norris (2011) states that the hierarchical structures of modes during interactions cannot be assumed, indeed gesture can be equal, subordinate or superordinate to other modes, which means that spoken language is not always vital to the (inter)action as a whole ( Norris, 2011;also see Kendon, 2004). While I am interested in the interplay between body posture, head movements, object handling and so on, this article mostly focuses on the interplay between (features of) spoken languages and gestures. ...
... These different modalities constitute 'language' and as such, sign language researchers have generally paid more attention to them, and their interplay, than to other modes (see Kusters et al., 2017). This approach of sign linguists is in line with Kendon's (2014), a prominent gesture studies scholar who regards gesture as part of language (together with speech), and contrasts with that of Norris and other multimodality scholars who equate language to speech and recognise an infinite number of other different modes ( Norris, 2004Norris, , 2011). The 'visual-gestural modality' includes both gesture and sign. ...
The article furthers the study of urban multilingual (i.e. metrolingual) practices, in particular the study of customer interactions, by a focus on the use of gestures in these practices. The article focuses on fluent deaf signers and hearing non-signers in Mumbai who use gestures to communicate with each other, often combined with mouthing, speaking and/or writing in different languages. The data were gathered through linguistic ethnography in markets, shops, food joints and public transport in Mumbai. Within gesture- based interactions, people with sensorial asymmetries (i.e. deaf vs. hearing) combined the visual-gestural modality and certain features of the auditory-oral modality, and/or switched between modalities. Interlocutors thus orient towards the ongoing interaction and negotiate the constraints and possibilities imposed not only by different modalities but also by different sensorial access to these modalities.
... In face-to-face conversation, participants negotiate meaning through multimodal contributions, in which the linguistic resources of speech interface with gesture. These modes do not work independently from one another, although a particular mode may weigh more than the others at some points (Norris 2004). Within speech itself, the vocal or the verbal mode may stand out as more prominent at particular points in time. ...
... projecting a space in which a semiotic entity makes meaning, providing unity, relation, and coherence) showing a multimodal distribution of labour between discourse organisation and information status. Modal density (Norris 2004) is suggested as a dynamic resource for constructing coherence. ...
... The highest modal density, i.e. the moment where all modes are the most active, drawn on and interwoven at the same time (Norris 2004;2011) lies in R, where most of the 'tying up' to the main sequence is achieved and where P is also indexed as 'particular' speech retrospectively. The negotiation of the communicative importance between the different modes fluctuates not only as different action sequences take place, but also within a particular sequence. ...
Based on a video recording of conversational British English, this paper aims at describing the relation between verbal and non-verbal signals in the production process of parentheticals within the framework of Multimodal Discourse Analysis. Parentheticals are described in linguistics as side sequences interrupting linear development. Although their syntactic, prosodic, and discursive characteristics have been deeply analysed, few studies have focused on the articulation of the different communicative modes in their production process. Beyond showing that gesture brings complementary information in regard to prosody, contributing to a composite collateral message, the results allow better delineation and understanding of skip-connecting phenomena as constructing coherence. Changes in the modal configuration throughout the parenthetical sequence suggest modes are dynamic and flexible resources for indexing parentheticals and their particular framing function.
... Gestos de apontar são usados quando interlocutores conectam o verbal ao visual, indicando objetos, locais ou imagens que estão presentes ou não no ambiente. Esses gestos não transmitem informações perceptivas ou de ação e podem ser produzidos independentemente de sua unidade de fala (Norris, 2011). A ação de apontar pode ser realizada de diferentes formas e usando diferentes tipos de materiais. ...
... Nos últimos anos, houve uma atenção crescente entre os gestos dêiticos em relação à sua linguagem falada, tanto fora das salas de aula (Norris, 2011) quanto dentro das salas de aula (Farsani, 2015b). Por exemplo, no campo do ensino e aprendizagem da matemática, Farsani (2015a) estudou dois professores de matemática trabalhando com a primeira e a segunda geração de britânicosiranianos no Reino Unido. ...
Este estudo objetiva examinar com que distância específica os alunos estão visualmente mais ou menos envolvidos com o professor nas aulas de matemática. Também busca analisar em que medida, as instruções feitas pelo professor, através de gestos de apontar, afetam o envolvimento visual dos alunos nas aulas dessa disciplina. Para isso foram selecionados, aleatoriamente, 50 alunos, 25 alunos do sexo masculino e 25 do sexo feminino, os quais, usavam uma minicâmara acoplada em óculos de lente que gravou 75 horas de videoaula. Os resultados mostram que os alunos estão mais envolvidos visualmente com as instruções do professor a uma proxêmica de 1,20 a 3,70 metros. Além disso, relata-se diferenças entre meninos e meninas e como eles estão visualmente envolvidos nas aulas de matemática. Por fim, conclui-se que os gestos de apontar realizados pelos professores podem servir como uma ferramenta para recuperar a atenção visual dos alunos.
... Pointing gestures are used when interlocutors connect the verbal to the visual, indicating objects, locations, and inscriptions that are present or not in the environment. These gestures do not convey perceptual or action information and can be produced independently of their speech unit (Norris, 2011). Pointing can be performed in different ways and use different types of materials. ...
... In recent years, there has been increasing attention to deictic gestures concerning spoken language, both outside (Norris, 2011) and inside the classrooms (Farsani, 2015a). For example, Farsani (2015a) studied two mathematics teachers working with first and second-generation British-Iranians in the UK. ...
Background: The interactions in the classroom are of particular interest to the teaching and learning processes. Objectives: This study examines nonverbal interaction in mathematics classrooms, and how different modes of nonverbal behaviour, contributed to the engagement in lessons. Design: A quantitative study. Setting and Participants: 30 randomly selected students wore mini camera-mounted eyeglasses in their mathematics and English lessons. Approximately 45 hours of video recording were made from these cameras (from a first-person's perspective) to analyse and compare the nonverbal interaction in mathematics and English lessons. Data collection and analysis: In Google Images, we objectively searched and statistically analysed frames in which the class teachers appeared within the students' visual field. Results: The results show that how students are visually engaged with the teacher depends on a set of proxemics. Differences were found related to visual attention both regarding the subject matter and the different proxemics of the student in relation to the teacher, pointing out that students are more visually involved with the teachers' instructions when at a proxemic of 1.20 to 3.70 meters. Furthermore, we report differences between boys and girls and how they are visually engaged in their mathematics classrooms. Conclusions: Finally, we report how teachers pointing gestures can serve as a tool to recapture students' visual attention in mathematics classrooms.
... Multimodal (inter)action analysis (Norris 2004(Norris , 2009(Norris , 2011a(Norris , 2011b(Norris , 2013) is a theory of human communication with an abundance of methodological tools to empirically * investigate interaction. Growing out of applied linguistics, anthropological linguistics, sociolinguistics, discourse analysis, and socio-cultural psychology (Goffman 1963(Goffman , 1974Gumperz 1982;Tannen 1984;Schiffrin 1987;Hamilton 1998;Scollon 1997;2001;van Lier 1996;Wertsch 1998;Wodak 1989) and strongly influenced by social semiotic thought (van Leeuwen 1999;Kress 2000;Kress andVan Leeuwen 1998, 2001), multimodal (inter)action analysis (Norris 2004(Norris , 2011a) is a multimodal discourse approach. ...
... Multimodal (inter)action analysis (Norris 2004(Norris , 2009(Norris , 2011a(Norris , 2011b(Norris , 2013(Norris , 2014(Norris , 2015 originating from mediated discourse analysis (Scollon 1998(Scollon , 2001 is based in the sociological interest of humans acting in the world that we find in the work of Goffman (1963); incorporates the interest in intercultural interaction that we find in the work of Gumperz (1982); includes an interest in power in interaction that we find in the work of Wodak (1989); delves into the microanalysis of interaction that we find in the work of Tannen (1984), Schiffrin (1987), or Hamilton (1998); has a strong interest in applied linguistics that we find in the work of van Lier (1996); is strongly influenced by socio-cultural psychology as we find in the work of Wertsch (1998); and is grounded in social semiotic thought that we find in the writings of van Leeuwen and Kress (van Leeuwen 1999;Kress 2000;Kress andvan Leeuwen 1998, 2001). With these foundations, multimodal (inter)action analysis (Norris 2004(Norris , 2011 has developed into a strong theoretical framework with an abundance of methodological tools (Norris 2004(Norris , 2009(Norris , 2011(Norris , 2013a(Norris , 2013b(Norris , 2014Geenen 2013;Makboon 2015;Pirini 2015Pirini , 2016) that make the analysis of (always) multimodal (inter)action possible, opening up research into new and promising directions. ...
This article presents theoretical concepts and methodological tools from multimodal (inter)action analysis that allow the reader to gain new insight into the study of discourse and interaction. The data for this article comes from a video ethnographic study (with emphasis on the video data) of 17 New Zealand families (inter)acting with family members via skype or facetime across the globe. In all, 84 social actors participated in the study, ranging in age from infant to 84 years old. The analysis part of the project, with data collected between December 2014 and December 2015, is ongoing. The data presented here was collected in December 2014 and has gone through various stages of analysis, ranging from general, intermediate to micro analysis.
Using the various methodological tools and emphasising the notion of mediation, the article demonstrates how a New Zealand participant first pays focused attention to his engagement in the research project. He then performs a semantic/pragmatic means, indicating a shift in his focused attention. Here, it is demonstrated that a new focus builds up incrementally: As the participant begins to focus on the skype (inter)action with his sister and nieces, modal density increases and he establishes an emotive closeness. At this point, the technology that mediates the interaction is only a mundane aspect, taken for granted by the participants.
... Gestures have been studied in many fields, including psychology (e.g., Ekman & Friesen, 1969;Iverson & Goldin-Meadow, 2005;McNeill, 2005), anthropology (e.g., Haviland, 2004;Kendon, 1997Kendon, , 2004, cognitive sciences and learning research (e.g., Congdon et al., 2017;Goldin-Meadow, 1999), visual communication (e.g., Lim, 2021Norris, 2011), and multimodal studies in linguistics (Crawford Camiciottoli & Bonsignori, 2015;Jewitt et al., 2016). Co-speech gestures help to enhance message comprehension and memorability (e.g., Dargue et al., 2019;Holler et al., 2014;Hostetter, 2011), foster teaching and learning capabilities (e.g., Congdon et al., 2017;Madan & Singhal, 2012;Schneider et al., 2022), and guide listeners to gauge the speaker's relationships with various actions and events (e.g., Chan & Kelly, 2021). ...
... Gesture-based communication is as essential as its verbal counterpart for in-person human-human interaction 1 . For exercise and other motor-learning tasks, physical demonstrations and gestures become an even more important part of the exchange. ...
Human instructors fluidly communicate with hand gestures, head and body movements, and facial expressions, but robots rarely leverage these complementary cues. A minimally supervised social robot with such skills could help people exercise and learn new activities. Thus, we investigated how nonverbal feedback from a humanoid robot affects human behavior. Inspired by the education literature, we evaluated formative feedback (real-time corrections) and summative feedback (post-task scores) for three distinct tasks: positioning in the room, mimicking the robot’s arm pose, and contacting the robot’s hands. Twenty-eight adults completed seventy-five 30-s-long trials with no explicit instructions or experimenter help. Motion-capture data analysis shows that both formative and summative feedback from the robot significantly aided user performance. Additionally, formative feedback improved task understanding. These results show the power of nonverbal cues based on human movement and the utility of viewing feedback through formative and summative lenses.
... But moreover, deictic reference, whether in the form of deictic language, gaze, or gesture, is a critical part of situated humanhuman communication [38,41]. Deixis is one of the earliest forms of communication both anthropologically and developmentally. ...
As robots are deployed into large-scale human environments, they will need to engage in task-oriented dialogues about objects and locations beyond those that can currently be seen. In these contexts , speakers use a wide range of referring gestures beyond those used in the small-scale interaction contexts that HRI research typically investigates. In this work, we thus seek to understand how robots can better generate gestures to accompany their referring language in large-scale interaction contexts. In service of this goal, we present the results of two human-subject studies: (1) a human-human study exploring how human gestures change in large-scale interaction contexts, and to identify human-like gestures suitable to such contexts yet readily implemented on robot hardware; and (2) a human-robot study conducted in a tightly controlled Virtual Reality environment, to evaluate robots' use of those identified gestures. Our results show that robot use of Precise Deictic and Abstract Pointing gestures afford different types of benefits when used to refer to visible vs. non-visible referents, leading us to formulate three concrete design guidelines. These results highlight both the opportunities for robot use of more humanlike gestures in large-scale interaction contexts, as well as the need for future work exploring their use as part of multi-modal communication. CCS CONCEPTS • Computer systems organization → Robotics; • Human-centered computing → Virtual reality; Empirical studies in interaction design.
... A mode is a unit of representation, which "is a semiotic system with rules and regularities attached to it". For example, gesture, gaze, posture, proximity and even technical equipment are modes that can be described systematically (Norris 2011). Gestures, for example, have their own grammar such as (onset, climax and retraction phase) which synthetically coincide with the mode language, gaze etc. (Stukenbrock 2015). ...
This paper discusses the linguistic features of the production of football commentary within its macrosocial contexts. It contrasts the communication of the TV production team during a football game with the live football commentary aired on TV. Applying a mixed methods analysis, it reveals how football commentary is prepared and produced in the technical setting of the commentator’s booth in a stadium. This study reveals how a medial reality of the same events concerning a goal are broadcasted in the 11 min highlight video football commentary. The author and her co-principal investigator video recorded a total of 36 h in four work settings. Finally, retrospective interviews with the commentator and head of sports of the TV station place the study in the broader social context of football as a media production. In this paper, I show how live football reporting is prepared and conducted using two different theoretical and methodological frameworks: Conversation Analysis (CA) and Multimodal (Inter)action Analysis (MMIA). Both provide insight into the synthetic verbal, non-verbal and parallel actions for the successful production of the football commentary and they also reveal the sequential structure of naturally occurring talk (CA).
... The study also aimed to consider how gestures made during outdoor field trips may serve as resources for conveying meaning during indoor geo-modeling activities. Multimodal transcription was used as a convention to represent, analyze, and compare students' multimodal interactions during these activities (Norris, 2011). ...
This study contributes to the literature by shedding light on students’ learning by examining and comparing students’ gestures in two educational contexts: geological field trips and classroom-based modeling activities. This study engaged 10 middle-school students in two different geological field trips and two classroom-based modeling activities and employed a hermeneutic approach to understand students’ use of gestures in both settings. Using multimodal interaction analysis ( MIA ) students’ gestures were explored to understand how students created and conveyed meaning using verbal and nonverbal interactions. The study found students’ gestures conveyed scientific content and differing social roles in the two settings. The results suggest the frequency and type of gestures used in each setting may be complementary to students’ development of accurate models. This study presents new classification criteria and characteristics to identify students’ gestures. This study demonstrates gestures can be an important resource for learning when students are engaging in modeling activities.
... Finally, within this broader area, our work examines robots' use of AR visualizations for the purposes of deictic gestures. Deictic gesture has been a topic of sustained and intense study both in humanhuman interaction [49,60] and human-robot interaction [77]. Deictic gesture is a key and natural communication modality, with humans starting to use deictic gesture around 9-12 months [7], and mastering it around age four [17]. ...
Augmented Reality (AR) technologies present an exciting new medium for human-robot interactions, enabling new opportunities for both implicit and explicit human-robot communication. For example, these technologies enable physically-limited robots to execute proven non-verbal interaction patterns such as deictic gestures despite not otherwise having the physical morphology necessary to do so. However, a wealth of HRI research has demonstrated that there are real benefits to physical embodiment (compared to, e.g., virtual robots on screens), suggesting AR augmentation of virtual robot parts could face similar challenges. In this work, we present empirical evidence comparing the use of virtual (AR) and physical arms to perform deictic gestures that identify virtual or physical referents. Our subjective and objective results demonstrate the success of mixed reality deictic gestures in overcoming these potential limitations, and their successful use regardless of differences in physicality between gesture and referent. These results help to motivate the further deployment of mixed reality robotic systems and provide nuanced insight into the role of mixed-reality technologies in HRI contexts.
... Deixis is a key component of human-human communication [41,48]. Humans begin pointing while speaking even from infancy, using deictic gestures around 9-12 months [4], and mastering deictic reference around age 4 [14]. ...
Mixed Reality provides a powerful medium for transparent and effective human-robot communication, especially for robots with significant physical limitations (e.g., those without arms). To enhance nonverbal capabilities for armless robots, this paper presents two studies that explore two different categories of mixed-reality deictic gestures for armless robots: a virtual arrow positioned over a target referent (a non-ego-sensitive allocentric gesture) and a virtual arm positioned over the gesturing robot (an ego-sensitive allocentric gesture). In Study 1, we explore the trade-offs between these two types of gestures with respect to both objective performance and subjective social perceptions. Our results show fundamentally different task-oriented versus social benefits, with non-ego-sensitive allocentric gestures enabling faster reaction time and higher accuracy, but ego-sensitive gestures enabling higher perceived social presence, anthropomorphism, and likability. In Study 2, we refine our design recommendations, by showing that in fact these different gestures should not be viewed as mutually exclusive alternatives, and that by using them together, robots can achieve both task-oriented and social benefits.
... In particular, the linking of science education and language (literacy)-based educational research has produced synergies that have brought value to researchers and teachers within the real-world contexts of schools (e.g., Authors). While writing, reading, and oral communication modalities are important (and traditional) ways to organize disciplinary discourse, our current research perspective views multimodal learning as meaning-making in science writ large (Danielsson & Selander, 2016;Kress, 2009;Norris, 2011). It has been known for some time that students learn more deeply from multimodal reading with texts and pictures than reading alone (Mayer, 2005). ...
This qualitative within-individual case design study involved six adolescents (age 10-14 years) engaging in a think-aloud observational protocol to read two texts on climate change from contrasting viewpoints. The participants completed a prior knowledge assessment and survey of technology used to assess potential mediating factors. Survey and observational data are presented as participant profiles. Results illustrated the effect of participants' background knowledge, emotional elicitation of text features, cognitive dissonance argument analysis due to the contrasting multimodal texts, and impact of visual images on participants' comprehension. Our data analyses revealed that there is an interconnected and nuanced relationship amongst many text and individual factors when adolescents engage in critical reading of SSI multimodal texts. This research provides direction for future science education research that support learners in critical reading of complex socioscientific topics as presented in multimodal texts with adolescent learners.
Supplementary information:
The online version contains supplementary material available at 10.1007/s10763-022-10280-8.
... Finally, within this broader area, our work specifically examines robots' use of AR visualizations for the purposes of deictic gesture. Deictic gesture has been a topic of sustained and intense study both in human-human interaction [40], [41] and human-robot interaction [5]. Deictic gesture is a key and natural communication modality, with humans starting to use deictic gesture around 9-12 months [42], and mastering it around age 4 [43]. ...
Augmented Reality (AR) or Mixed Reality (MR) enables innovative interactions by overlaying virtual imagery over the physical world. For roboticists, this creates new opportunities to apply proven non-verbal interaction patterns, like gesture, to physically-limited robots. However, a wealth of HRI research has demonstrated that there are real benefits to physical embodiment (compared, e.g., to virtual robots displayed on screens). This suggests that AR augmentation of virtual robot parts could lead to similar challenges. In this work, we present the design of an experiment to objectively and subjectively compare the use of AR and physical arms for deictic gesture, in AR and physical task environments. Our future results will inform robot designers choosing between the use of physical and virtual arms, and provide new nuanced understanding of the use of mixed-reality technologies in HRI contexts.
... In the five classes of gesture above, most can convey some visuospatial information, but Deictic gestures do so by definition. Deictics play a crucial role in human to human communication by supporting direct reference to visual and non-visual objects (Norris, 2011). It has also been shown that robotic deictic gestures can shift our attention in much the same way as human uses of these gestures (Brooks and Breazeal, 2006). ...
Acting, stand-up and dancing are creative, embodied performances that nonetheless follow a script. Unless experimental or improvised, the performers draw their movements from much the same stock of embodied schemas. A slavish following of the script leaves no room for creativity, but active interpretation of the script does. It is the choices one makes, of words and actions, that make a performance creative. In this theory and hypothesis article, we present a framework for performance and interpretation within robotic storytelling. The performance framework is built upon movement theory, and defines a taxonomy of basic schematic movements and the most important gesture types. For the interpretation framework, we hypothesise that emotionally-grounded choices can inform acts of metaphor and blending, to elevate a scripted performance into a creative one. Theory and hypothesis are each grounded in empirical research, and aim to provide resources for other robotic studies of the creative use of movement and gestures.
... In the five classes of gesture above, most can convey some visuospatial information, but Deictic gestures do so by definition. Deictics play a crucial role in human to human communication by supporting direct reference to visual and non-visual objects (Norris, 2011). It has also been shown that robotic deictic gestures can shift our attention in much the same way as human uses of these gestures (Brooks and Breazeal, 2006). ...
Acting, stand-up and dancing are creative, embodied performances that nonetheless follow a script. Unless experimental or improvised, the performers draw their movements from much the same stock of embodied schemas. A slavish following of the script leaves no room for creativity, but active interpretation of the script does. It is the choices one makes, of words and actions, that make a performance creative. In this theory and hypothesis article, we present a framework for performance and interpretation within robotic storytelling. The performance framework is built upon movement theory, and defines a taxonomy of basic schematic movements and the most important gesture types. For the interpretation framework, we hypothesise that emotionally-grounded choices can inform acts of metaphor and blending, to elevate a scripted performance into a creative one. Theory and hypothesis are each grounded in empirical research, and aim to provide resources for other robotic studies of the creative use of movement and gestures.
... But moreover, deictic reference, whether in the form of deictic language, deictic gaze, or deictic gesture, is a critical part of situated human-human communication [28,30]. Deixis is one of the earliest forms of communication, both anthropologically and developmentally. ...
To enable robots to select between different types of nonverbal behavior when accompanying spatial language, we must first understand the factors that guide human selection between such behaviors. In this work, we argue that to enable appropriate spatial gesture selection, HRI researchers must answer four questions: (1) What are the factors that determine the form of gesture used to accompany spatial language? (2) What parameters of these factors cause speakers to switch between these categories? (3) How do the parameterizations of these factors inform the performance of gestures within these categories? and (4) How does human generation of gestures differ from human expectations of how robots should generate such gestures? In this work, we consider the first three questions and make two key contributions: (1) a human-human interaction experiment investigating how human gestures transition between deictic and non-deictic under changes in contextual factors, and (2) a model of gesture category transition informed by the results of this experiment.
... Deixis is a key component of human-human communication [37,44]. Humans begin pointing while speaking even from infancy, using deictic gestures around 9-12 months [4], and mastering deictic reference around age 4 [14]. ...
Mixed Reality visualizations provide a powerful new approach for enabling gestural capabilities on non-humanoid robots. This paper explores two different categories of mixed-reality deictic gestures for armless robots: a virtual arrow positioned over a target ref-erent (a non-ego-sensitive allocentric gesture) and a virtual arm positioned over the gesturing robot (an ego-sensitive allocentric gesture). Specifically, we present the results of a within-subjects Mixed Reality HRI experiment (N=23) exploring the trade-offs between these two types of gestures with respect to both objective performance and subjective social perceptions. Our results show a clear trade-off between performance and social perception, with non-ego-sensitive allocentric gestures enabling faster reaction time and higher accuracy, but ego-sensitive gestures enabling higher perceived social presence, anthropomorphism, and likability.
... Deixis is one of the most crucial pieces of human-human communications [51], [59], as well as one of the oldest, both anthropologically and developmentally. Unlike many other aspects of human communication, there are clear analogues of deictic gesture in the animal kingdom (e.g., the signaling capabilities of animals in the presence of predators) [54], [62]. ...
Research has shown that robots that use physical deictic gestures such as pointing enable more effective and natural interaction. However, it is not yet clear whether these benefits hold true for new forms of deictic gesture that become available in mixed-reality environments. In previous work, we presented a human-subject study suggesting that these benefits may indeed translate in the case of allocentric mixed-reality gestures, in which target referents are picked out in users' fields of view using annotations such as circles and arrows, especially when those gestures are paired with complex referring expressions. In this paper we provide additional evidence for this hypothesis through a second experiment that addresses potential confounds from our original experiment. I. INTRODUCTION Robots are already being deployed in factories, hospitals, and search-and-rescue operations. Robots in these domains need to be able to communicate with their human users and teammates in a way that is effective and natural, while minimizing the need for special training. Accordingly, HRI researchers have sought to allow people to communicate with robots the same way they do with other people, through natural language and gesture. These two modes of communication are typically paired together, as gesture facilitates fluent speech, communication of abstract concepts, and deictic reference to nearby objects. Deictic gestures in particular are used by humans use to draw attention to the region of the environment containing their target referent. HRI researchers have studied how robots might generate types of deictic gestures beyond pointing, including presenting , exhibiting, touching, grouping, and sweeping, and how such robot-generated gestures are perceived by humans [74]. As robots' capabilities increase, it will be just as important to study any new forms of gesture available to robot (and how those new forms are perceived). One way in which robots' capabilities are currently increasing is through their integration with augmented and mixed reality technologies [93], [94], which serve to increase not only the flexibility of users' control over robots, but also the expressivity of users' view into those robots' internal states [92], through visualizations rendered onto users' Augmented Reality Head-Mounted Displays that reflect information from the robot's internal state. As a simple example, consider the case of a UAV communicating with human teammates about the location of a disaster
... Deixis is one of the most crucial forms of human-human communications [8,9], as well as one of the oldest, both anthropologically and developmentally. Humans point while speaking even from infancy, with deictic gesture beginning around 9-12 months [10], and general deictic reference mastered around age 4 [11]. ...
Mixed reality technologies offer interactive robots many new ways to communicate their beliefs, desires, and intentions to human teammates. In previous work, we identified several categories of visualizations that when displayed to users through mixed reality technologies serve the same role as traditional deictic gestures (e.g., pointing). In this work, we experimentally investigate the potential utility of one of these categories, allocentric gestures, in which circles or arrows are rendered to enable human teammates to pick out the robot’s target referents. Specifically, through two human subject experiments, we examine the objective and subjective performance of such gestures alone as compared to language alone and the combination of language and allocentric gesture. Our results suggest that allocentric gestures are more effective than language alone, but to maintain high robot likability allocentric gestures should be used to complement rather than replace complex referring expressions.
... Multimodal discourse analysis is a kind of perspective that uses various modalities composed of all meaning generating resources to analyze discourse and is an ideal method for describing and analyzing the relationship between speech and speech-associated gesture. Many foreign scholars (Roth & Lawless, 2002;Lazaraton, 2004;Norris, 2004Norris, , 2011Schnettler, 2006;Knoblauch, 2008) analyzed the spoken discourse in classroom and formal academic lecture from multimodal perspective, and discussed characteristics of the speaker's speech and speech-associated gesture; Roth & Lawless (2002: 285-304) analyzed the distribution frequency of modalities such as spoken language, charts and gestures in spoken discourse and their multimodal relations by taking the video of science lectures as an example and using quantitative and qualitative analysis method; taking interpretation of new vocabulary in a foreign language classroom as an example, Lazaraton (2004: 79-117) illustrated the auxiliary effects of four different gestures on expression of a meaning and proved that there was a dynamic hierarchical relation existing between spoken language and gesture in the process of construction of the meaning and the hierarchical relation might be changed in different stage of communication; for example, discourse modality was not always the major modality; sometimes gesture might play a role of major modality and spoken language was just a complement for construction of the meaning (Norris , 2004: 128-147;2011: 129-147). ...
... Deixis is one of the most crucial pieces of human-human communications [52], [61], as well as one of the oldest, both anthropologically and developmentally. Unlike many other aspects of human communication, there are clear analogues of deictic gesture in the animal kingdom (e.g., the signaling capabilities of animals in the presence of predators) [54], [64]. ...
In previous work, researchers have repeatedly demonstrated that robots' use of deictic gestures enables effective and natural human-robot interaction. However, new technologies such as augmented reality head mounted displays enable environments in which mixed-reality becomes possible, and in such environments, physical gestures become but one category among many different types of mixed reality deictic gestures. In this paper, we present the first experimental exploration of the effectiveness of mixed reality deictic gestures beyond physical gestures. Specifically, we investigate human perception of videos simulating the display of allocentric gestures, in which robots circle their targets in users' fields of view. Our results suggest that this is an effective communication strategy, both in terms of objective accuracy and subjective perception, especially when paired with complex natural language references.
... In DHA, "[e]mpirical data is systematically considered from four levels of context: i) the text; ii) other co-texts; iii) ethnographic observations of non-textual aspects; and iv) broader socio-political and historical issues" : 1; see also Clarke et al. 2012: 461 f.). In addition to questions such as why this model of context assumes that extra-linguistic aspects on a second level should constitute a more abstract level and thus context for the linguistic text on the first level rather than vice versa (Norris 2011), we should ask how the fourth level of the context is determined by the researcher. As suggested by Blommaert, good analysis should avoid using ethnographic observations as default evidence for researcher's postulations. ...
Interactions in guided tours, guestbooks, and the mass media characterise a variety of stages on which museums—in their interactions with the public—construct knowledge and the museums’ ‘brands.’ Journalists ask how autonomous the museums are from governments and about how they relate to current immigration politics. At the museums’ opening nights, curators and politicians contextualise the exhibition in relation to discourse about xenophobia and discrimination in “Fortress Europe.” In guided tours, systematic differences in the way participants select and contextualise exhibits characterise the specific institutional approach and spaces within the museum. Consequently, participants either position immigrant minorities in relation to a national public majority on a collective level, or engage in discourse about personal, multicultural identities in everyday life.
... Por ejemplo, el protocolo deja de lado el papel fundamental que juegan los gestos en la interacción social. La posición corporal, la mirada y el acto de señalar en general, funcionan como gestos deícticos que permiten a las personas que se encuentran en un mismo contexto, enfocar su atención en un objeto o evento compartidos, estando dichos gestos subordinados, superordinados o como acompañantes respecto a las expresiones habladas, tal como sucede en la narración de ocurrencias o cuando se señala algo al interlocutor (Norris, 2011;Tomasello & Carpenter, 2007) y nada de esto se evalúa mediante el protocolo. ...
Resumen Desde una perspectiva contextual-funcional, la Teoría de los Marcos Relacionales (RFT por su sigla en inglés) ha abordado los fenómenos psicológicos relacionados con la toma de perspectiva asumiéndolos como un tipo de responder relacional deíctico. Uno de sus propósitos ha sido abordar, a través de la investigación trasla-cional, problemáticas socialmente relevantes a partir del estudio sobre los procesos conductuales implicados en estos fenómenos. El objetivo de este estudio fue aplicar a dos participantes en edad escolar, un protocolo adaptado de evaluación y entre-namiento de marcos deícticos basado en la RFT, con el fin de atender a una de las amenazas más frecuentes a su validez ecológica (i.e., extenso número de ensayos) y valorar su suficiencia y eficiencia en la adquisición de dicho repertorio conductual, así como la conveniencia de su aplicabilidad en contextos de desarrollo típico y natural. Los resultados muestran que los dos participantes tuvieron desempeños del 100% de aciertos en entrenamiento y pruebas de relaciones deícticas simples. Mien
... For example spoken and written language may or may not be strongly implicated in an action. And, in those actions that do involve language as a relevant mediational means, analysis of the data does not establish a hierarchical relationship in which language always takes priority (Norris 2011b). Thus, actions are produced by social actors acting with and through mediational means, and these mediational means are always multiple and are structured in some relation to the action. ...
Building on multimodal (inter)action analysis as a theoretical and methodological framework, this article introduces and develops the theoretical/methodological tool called primary agency. Taking the mediated action as a unit of analysis, agency can be analysed as a feature of action. However, there is a lack of empirical approaches for the study of agency, and an overemphasis on language as the most important site for identifying agentive action. I develop primary agency through an analysis of three co-produced higher-level actions from a research project into high school tutoring. These are the higher-level actions of conducting research, tutoring and reading a text. Applying co-production and the modal density foreground/background continuum I explore how the researcher, the tutor and the student co-produce these higher-level actions. Through this analysis, I identify the most significant mediational means for each higher-level action, and the social actor with ownership or agency over these mediational means. I define this social actor as the one with primary agency over the co-produced higher-level action. Finally, my analysis outlines the implications of primary agency for co-produced higher-level actions, including the role of the researcher, the attention/awareness participants pay to overarching research projects, and links between primary agency and successful learning.
... Por ejemplo, el protocolo deja de lado el papel fundamental que juegan los gestos en la interacción social. La posición corporal, la mirada y el acto de señalar en general, funcionan como gestos deícticos que permiten a las personas que se encuentran en un mismo contexto, enfocar su atención en un objeto o evento compartidos, estando dichos gestos subordinados, superordinados o como acompañantes respecto a las expresiones habladas, tal como sucede en la narración de ocurrencias o cuando se señala algo al interlocutor (Norris, 2011;Tomasello & Carpenter, 2007) y nada de esto se evalúa mediante el protocolo. ...
La teoría de los marcos relacionales aborda los fenómenos psicológicos relacionados con la toma de perspectiva asumiéndolos como un tipo de responder relacional deíctico y a través de la investigación traslacional se abordan problemáticas socialmente relevantes partiendo del estudio de procesos conductuales posiblemente implicados. En este estudio participaron dos niños a quienes se les aplicó un protocolo adaptado de evaluación y entrenamiento de marcos deícticos basado en la RFT.La adaptación del protocolo intentó atender una de las amenazas más frecuentes a su validez ecológica (i.e., extenso número de ensayos), valorar la suficiencia y eficiencia en la adquisición de dicho repertorio conductual, así como la aplicabilidad en contextos naturales. Los resultados muestran que los participantes tuvieron desempeños del 100% de aciertos en entrenamiento y pruebas de relaciones deícticas simples.Mientras que en relaciones invertidas tuvieron desempeños efectivos en el entrenamiento de relaciones personales, temporales y espaciales, su ejecución fue deficiente en las fases de prueba de relaciones temporales y espaciales, lo cual impidió que pudieran evaluarse y entrenarse las relaciones doblemente invertidas.Estos hallazgos se discuten en términos de validez ecológica del protocolo y su aplicabilidad en contextos naturales para la solución de problemáticas relacionadas con la toma de perspectiva.
... These models, and much of earlier analyses of disciplinary discourse as such, are more or less biased towards verbals (writing and speech) rather than seeing all meaning making as multimodal (see Kress, 2009, or Norris, 2011. However, multimodality is ubiquitous in any discipline as well as in learning resources at all levels, and a number of scholars have pointed out the challenges for students regarding the ways that different meaning making resources are used and combined (Danielsson, 2013;Danielsson, accepted;Kress et al., 2001;Kress et al., 2004;Lemke, 1998;Tang & Moje, 2010;Tang et al., 2014, Unsworth;2001). ...
The re-conceptualisation of texts over the last 20 years, as well as the development of a multimodal understanding of communication and representation of knowledge, has profound consequences for the reading and understanding of multimodal texts, not least in educational contexts. However, if teachers and students are given tools to “unwrap” multimodal texts, they can develop a deeper understanding of texts, information structures, and the textual organisation of knowledge. This article presents a model for working with multimodal texts in education with the intention to highlight mutual multimodal text analysis in relation to the subject content. Examples are taken from a Singaporean science textbook as well as a Chilean science textbook, in order to demonstrate that the framework is versatile and applicable across different cultural contexts. The model takes into account the following aspects of texts: the general structure, how different semiotic resources operate, the ways in which different resources are combined (including coherence), the use of figurative language, and explicit/implicit values. Since learning operates on different dimensions – such as social and affective dimensions besides the cognitive ones – our inclusion of figurative language and values as components for textual analysis is a contribution to multimodal text analysis for learning.
Despite recent calls to more fully incorporate multimodal perspectives into literacies research, there is still limited scholarship examining how students critically engage in reading activities by drawing on embodied practices. Racially and linguistically minoritized students are particularly disadvantaged by dominant logocentric and developmentalist approaches, which privilege oral and written discourse and often position these students as less capable of performing complex literacy practices. Drawing from three independent ethnographic studies, our multimodal interactional analysis examines how students of a range of ages and raciolinguistic backgrounds use embodied actions and other semiotic resources to agentively navigate text, task, and ideological constraints in activities involving reading and analyzing texts. Our analysis demonstrates the crucial role of students' embodied practices in expanding upon and challenging the constraints of literacy activities, focusing particularly on how students leveraged epistemic stance‐taking, embodied affective responses, and embodied forms of argumentation to negotiate and co‐construct meaning. Through a focus on embodied agency, this paper presents and applies an interactional perspective on the embodied nature of literacy activities; shows how students' creative mobilizations of embodied and other semiotic resources contribute to their critical readings of texts; and offers pedagogical and methodological implications for ways educators and researchers can attend to the intricacies of students' embodied sense‐making in literacy activities.
This paper intends to bring the relevance of studies about classroom discourse from a multimodal perspective with focus on how gestures (nonverbal language) participate in making meaning processes and possibility, or not, engagement in classroom interaction. This paper presents a pseudo eye-tracking study to examine teachers’ unconscious, nonverbal cues when eliciting student answers. In particular, the study looks at how teachers address students by either pointing at them with their index finger or using an open-hand, palm-up gesture. The study was conducted in three schools in Chile and four schools in UK. Students not only follow the explicit nonverbal cues very well (and very subtly); they also display particular patterns of proxemics and nonverbal behavior. The results show that asking a student to answer a question by pointing at them with an index finger resulted in a sudden negative change, with the student distancing themselves from the teacher. In contrast, when a question was accompanied by an open-hand, palm-up gesture, the students moved close and turned their body toward the teacher. These findings bring important questions to think about processes of engagement in classroom discourse, focusing aspects that involve non-verbal signals that participate of meaning production in teacher-student interaction.
Keywords:
Turn-taking; Proxemics; Gesture; Multimodal communication; Engaging communication
A variety of semiotic resources makes the construction of scientific knowledge possible and meaning-making resources are conveyed by certain semiotic modes. Next, numerous studies have demonstrated the pedagogical importance of gestures in the demonstration of scientific knowledge in the classroom. Drawing on social semiotic systemic functional theory and legitimation code theory, this study explores the types and the role of gestures in the semiotic construction of scientific knowledge in pedagogic semiosis and their pedagogical values for the meaning-making of science content. With this aim, a gesture observation protocol (GOP) is developed to annotate and analyze data. Two science lectures, which are broadcast remotely for middle school students including gifted students, are observed. Systemic Functional Multimodal Discourse Analysis (SF-MDA) method is employed to analyze and evaluate data. Results demonstrated that the deployment of gestures that contributes to the construction of scientific knowledge is low and deployed gestures mostly do not have pedagogical value. It is recommended that teachers should be more conscious and educated about deploying gestures to present scientific knowledge efficiently.
Dooly (2017, p. 169) defines telecollaboration in education as “the use of computer and/or digital communication tools to promote learning through social interaction and collaboration, thus moving the learning process beyond the physical boundaries of classrooms.” As digital communication technologies advance, newer and more sophisticated cutting-edge ICT tools are being used for telecollaboration, including virtual reality (VR). Researchers have applied different models and approaches of multimodal analysis to understand the specific features of VR on students' language learning (Dubovi, 2022; Friend & Mills, 2021) and intercultural communication (Rustam, et al., 2020). Nevertheless, very little has been done to look into language teacher telecollaboration via VR technologies. This present study recruited student teachers of an additional language (LX) (Dewaele, 2017) from different geographical locations and cultural backgrounds to participate in a project aiming at cultivating the participants' critical views on LX teaching and intercultural communication skills. The participants interacted and discussed LX teaching/learning issues in VR environments. Their interactions were video recorded and analyzed. By applying Multimodal (inter)action Analysis (MIA) (Norris, 2014) as the analytical framework, this study systematically unpacked the thematical saliencies and significant moments of the participants' intercultural interaction in the three VR meetings. Based on the findings, suggestions and caveats for future designing and researching intercultural telecollaboration in VR environments are provided.
Rapid development of multimodal communication stimulates scholarly interest in elaborating methods and tools capable of investigating this phenomenon in its structural and functional complexity. One such approach is multimodal discourse analysis (MDA). Of relevance to the origins of MDA are social semiotics and systemic-functional linguistics, providing rationale to approach different modes as meaning-making systems on the one hand, and to investigate multimodal texts as communicative acts in context on the other. Having said that, there still seem to be unchartered areas in the vast universe of multimodal communication, all the more so as research can hardly keep pace with the constantly emerging and promptly popularized Web 2.0 technologies of communication. One important aspect that is still to receive a thorough scientific account is logico-semantic progression in multimodal texts, in particular the extent to which patterns of discourse organization, originally identified in written texts, are realized in non-textual modes. The study discusses how patterns of discourse organization are realized in simultaneously received modes in the multimodal genre of promotional video in crowdfunding project presentations. Three types of interaction between the patterns, their moves and the modes of expression are identified: (1) transfer of moves across modes; (2) move-and-pattern overlap; and (3) rupture of intermodal cohesion. The insights presented in this paper may be relevant not only to related studies of multimodal discourse but also for more effective content management in crowdfunding project presentations.KeywordsMultimodalityDiscourse organization patternDiscourse analysisCrowdfundingKickstarter
Using optical head-mounted display (HMD) devices, users can see both real world and Augmented Reality (AR) content simultaneously. AR content can be displayed to both eyes (binocular) or in one eye (monocular).
Medical trainees require sufficient practice to gain the experience and confidence needed to safely and reliably perform endotracheal intubations. While video laryngoscopy has been used to provide an advanced glottic view that can reduce intubation failure, prevent prolonged intubation time, and reduce repeated intubation attempts, most current devices require visualization on external monitors, disrupting the direct line-of-sight view. These devices also present a deep intra-oral view of the airway that may not be visible during a typical unassisted intubation attempt. As a result, these differences create new challenges to gaining competency in the standard, direct laryngoscopy technique when using video laryngoscopy as a learning tool.
This thesis considers child-initiated play from a multimodal social semiotic perspective, giving close attention to the ways in which children collaboratively make meaning in play in a multitude of ways. Such a perspective resists instrumental, developmental perspectives on play, and comes at a time when play-based approaches are in tension with increasingly formalised learning agendas and changes to early years assessment. In order to explore the multimodality of child-initiated play, apt theories and research methods are necessary for attending to the ways children make meaning in multiple modes. The study consists of video-based observations of child-initiated play collected through an ethnographic, teacher-research case study carried out in a nursery school in England. A particular challenge in multimodal research is developing forms of transcription which account for multiple modes in fine-grained detail, with the conventions developed for transcribing language proving insufficient. This thesis presents four multimodal transcript designs as analytic devices that bring multimodal aspects of play to the fore, and critically discusses the gains and losses of each multimodal transcript. The multimodal transcripts highlight the richness and complexity of child-initiated play as learning, making visible ways in which children’s play is complex, layered, transformative, creative and agentive meaning-making. This thesis proposes that multimodal transcription not only ‘visualises’ play by making it visible and sharable, but also offers a new lens through which we might understand the semiotic complexity of play. Through interwoven substantive and methodological strands, this thesis therefore offers a contribution towards the tools and dispositions necessary for recognising and valuing meaning-making in play, in early years research methodology, educational theory and practice.
This methodology assembles compatible tools from ethnomethodological Conversation Analysis, poststructural discourse analysis, and analytic ethnography. A focus on participants’ contextualisation cues in face-to-face conversation, written text, embodied ritual practices, and mass media communication enables an analysis of how certain practices instantiate what participants consider macro-societal phenomena. Conversation Analysis dissects the ephemeral details of social and institutional interaction and shows how participants’ referencing, pointing, orientation, and attributing generate territories of knowledge. Discourse analysis unravels polyphonic and hierarchical mappings of societal subject positions in single utterances in written texts. Theory-oriented, multi-sited ethnography allows the researcher to experience and compare longer sequences of institutional work and ritual practices, and to follow mass media communication in order to distinguish cases of goal-oriented practice in their social, material, and cultural contexts.
Classroom resources, iPad apps included, are very often designed with clear pedagogical goals in mind. Currently, a range of apps available to schools, such as Hairy Letters © and Pocket Phonics Letter Sounds and Writing ©, are specifically designed to support skills commonly deemed important for print literacy development. This chapter draws on four video recorded episodes and focuses attention on children’s communicative repertoires as they interact with and around iPad apps, therefore providing insights into the ways in which digital technologies mediate children’s early literacy learning experiences. What becomes clear is that apps in classrooms offer sites for collaborative engagement where novel communicative repertoires can emerge. These repertoires, in some part shaped by the operational dimension and the multimodal affordances of the touchscreen device, quickly become integrated into children’s playful activity. Such activity, as children’s interests emerge and are played out, stimulates participation and gives rise to children’s peer cultures. When seen in this way, it becomes apparent that while apps may support elements of print literacies, the role of apps in children’s early learning experiences goes far beyond specific pedagogical goals.
New touchscreen technologies have drawn attention to the materiality of literate behaviour. Reflecting on the concept of handiness, this chapter looks at meaning making as a kind of ‘engaged material consciousness’ (Sennett 2009) that depends upon but always exceeds the complex coordinations of hand and eye. Interrogating empirical materials gathered in a study of iPads in the early years, it shows young children and the adults that care for them engaged in complex negotiations and interactions that continually remake literacy, technology and learning.
This paper presents a critical examination of key concepts in the study of (signed and spoken) language and multimodality. It shows how shifts in conceptual understandings of language use, moving from bilingualism to multilingualism and (trans)languaging, have resulted in the revitalisation of the concept of language repertoires. We discuss key assumptions and analytical developments that have shaped the sociolinguistic study of signed and spoken language multilingualism as separate from different strands of multimodality studies. In most multimodality studies, researchers focus on participants using one named spoken language within broader embodied human action. Thus while attending to multimodal communication, they do not attend to multilingual communication. In translanguaging studies the opposite has happened: scholars have attended to multilingual communication without really paying attention to multimodality and simultaneity, and hierarchies within the simultaneous combination of resources. The (socio)linguistics of sign language has paid attention to multimodality but only very recently have started to focus on multilingual contexts where multiple sign and/or multiple spoken languages are used. There is currently little transaction between these areas of research. We argue that the lens of semiotic repertoires enables synergies to be identified and provides a holistic focus on action that is both multilingual and multimodal.
This chapter presents a participant observation case study of a child aged 5 1/2 years who used the interactive personalized app ‘Mr Glue Stories’ together with her father. The app encouraged the child to personalize a given narrative with text, audio recordings and her own drawings. Building on Bruner’s (1994, 2001) and Lemke’s (2000, 2002) theoretical foundations around self, narrative and text-making, we consider how the various personalization features of the app played a role in developing the child’s sense of self. We also examine the potentials of personalization for facilitating and shaping moments of particular attunement and closeness (‘moments of meeting’; Stern 2000, 2004) between the child and father. Our findings suggest that personalization features in some iPad story-making apps provides unique opportunities for children to explore their experiences of the world and share these with adults close to them.
Mediated discourse analysis (MDA), developed by , is a theory of human action.
This article takes a multimodal approach to examine how two young men communicate knowledge, shift attention, and negotiate a disagreement via videoconferencing technology. The data for the study comes from a larger ongoing project of participants engaging in various tasks together. Linking micro, intermediate and macro analyses through the various methodological tools employed, the article presents multimodal (inter)action analysis (Norris, 2004, 2011, 2013a, 2013b) as a methodology to gain new insight into the complexity of knowledge communication via videoconferencing technology, which is relevant to many settings from education to employment, from organizations to gaming.
This article explores how voice is expressed in a telecollaborative project using Skype to connect two groups of primary age English language learners across two countries. Voice is understood as the ways in which language and other semiotic means are used for communication (Blommaert, 2008). This theoretical view frames the qualitative study into how voice is expressed materially involving tools such as verbal language, body language, technology, and the spatial and temporal dimensions within which the children's conversation happens. A methodology for analysing the video recorded data was developed using Scollon and Scollon's concept of geosemiotics. This method of analysis investigates how language is materially assembled through interaction with others in the physical world. The study shows that telecollaborative conversations create particular conditions which affect the ways children express their voice. The implications discussed in the conclusion have the potential to initiate wider discussion in the context of early childhood education and language learning concerning the importance of a multimodal perspective on how children express voice to support their communication when using video conferencing.
This article develops a new methodological tool, called scales of action, which allows the empirical investigation of ubiquitous actions such as driving on the one hand, and the highly complex relationships between (for example) drives and other actions in everyday life on the other hand. Through empirical analysis of ethnographic data of drives performed by a German artist and an American IT specialist, the article illustrates how talk and driving are embedded differently in different cultural contexts. Examining the actions of the two drivers before, during, and after a drive further demonstrates that chronologically performed actions are not necessarily sequential in nature. Using a mediated discourse theoretical approach and building upon multimodal (inter)action analysis, the article provides analysts with a tool that captures the inherent complexities of everyday actions. Through the notion of scales of action and their composition, this article sheds new light upon the complexity and cultural differences of drives and car talk in middle class Germany and North America.
With digital texts being employed in classrooms, the construction and content of communication need to be examined to understand implications for classroom pedagogies and the development of new communicative practices. The study employs a multimodal interaction analysis (MIA) framework (Norris 2004) to carry out a microethnographic analysis of digital literacy events of first-year university students in a Japanese university explaining online stories. The data captured are analysed to reveal how meanings are multimodally constructed in relation to elements and features of the mediating digital texts. The analyses showed that specific “modal configurations” (Norris 2009) such as spoken language, gesture and gaze appeared to be shaped by the visual meanings in the mediating texts. The spoken meanings were also focussed directly on what was visually present in the texts, further demonstrating the impact of the visual nature of the texts in shaping meanings made in the digital literacy events.
Using data from more than ten years of research, David McNeill shows that gestures do not simply form a part of what is said and meant but have an impact on thought itself. Hand and Mind persuasively argues that because gestures directly transfer mental images to visible forms, conveying ideas that language cannot always express, we must examine language and gesture together to unveil the operations of the mind.
In this monograph, the author offers a new way of examining the much discussed notion of identity through the theoretical and methodological approach called multimodal interaction analysis. Moving beyond a traditional discourse analysis focus on spoken language, this book expands our understanding of identity construction by looking both at language and its intersection with such paralinguistic features as gesture, as well as how we use space in interaction. The author illustrates this new approach through an extended ethnographic study of two women living in Germany. Examples of their everyday interactions elucidate how multimodal interaction analysis can be used to extend our understanding of how identity is produced and negotiated in context from a more holistic point of view.
Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences.
Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work.
Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.
This article identifies some limitations of discourse analysis by analyzing interactions between five boys in which the TV and the computer are featured as mediational means. The incorporation of several modalities into transcripts and a shift in focus from primarily language to human action facilitate a better understanding of the multi-modal interaction involved. The use of conventional transcripts with a focus on language demonstrates that movie- and computer-mediated interactions appear fragmented; by contrast, an inclusion of images into the transcripts, representing central interactions and/or images of a movie or computer screen, demonstrates the significant visual modes that are imperative to the ongoing talk. Just as written words correspond to the oral language, images can exemplify the global interaction among the participants, or they can represent the images on the screen. In addition, viewing an image is much faster than reading a description, so that these images also display the fast pace of the movie- and/or computer-mediated interaction.
Some Body Motion Elements Accompanying Spoken American English Communication and Culture
- R L Birdwhistell
Birdwhistell, R.L (1966) 'Some Body Motion Elements Accompanying Spoken American English', in H.G. Smith (ed.) Communication and Culture. New York: Holt, Rinehart & Winston.
Multimodal Teaching and Learning: The Rhetorics of the Science Classroom
- G Kress
Kress, G. et al. (2001) Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. London: Continuum.
Modal Density and Modal Configurations: Multimodal Actions Routledge Handbook for Multimodal Discourse Analysis
- S Norris
Norris, S. (2009) 'Modal Density and Modal Configurations: Multimodal Actions', in C. Jewitt (ed.) Routledge Handbook for Multimodal Discourse Analysis. London: Routledge.
Mediated Discourse: The Nexus of PracticeMultimodality, Genre and Design
- R Scollon
Scollon, R. (2001) Mediated Discourse: The Nexus of Practice. London: Routledge. Van Leeuwen, T. (2005) 'Multimodality, Genre and Design', in S. Norris and R.H. Jones (eds) Discourse in Action: Introducing Mediated Discourse Analysis. London: Routledge.