Systematically Working with Multimodal Data: Research Methods in Multimodal Discourse Analysis.

Inside the commentator's booth: a Multimodal (Inter)action and Conversation Analysis on the production of first division football commentary on TV

Article

Full-text available

Aug 2023

Inke Du Bois

This paper discusses the linguistic features of the production of football commentary within its macrosocial contexts. It contrasts the communication of the TV production team during a football game with the live football commentary aired on TV. Applying a mixed methods analysis, it reveals how football commentary is prepared and produced in the technical setting of the commentator’s booth in a stadium. This study reveals how a medial reality of the same events concerning a goal are broadcasted in the 11 min highlight video football commentary. The author and her co-principal investigator video recorded a total of 36 h in four work settings. Finally, retrospective interviews with the commentator and head of sports of the TV station place the study in the broader social context of football as a media production. In this paper, I show how live football reporting is prepared and conducted using two different theoretical and methodological frameworks: Conversation Analysis (CA) and Multimodal (Inter)action Analysis (MMIA). Both provide insight into the synthetic verbal, non-verbal and parallel actions for the successful production of the football commentary and they also reveal the sequential structure of naturally occurring talk (CA).

Multimodal resemiotization: Examining one L2 writer's interactions with equity videos

Chapter

Sep 2022

In the era of rapidly advancing technology, multilingual writers’ semiotic landscapes are increasingly multimodal, such that semiotics other than the written and spoken word – images, lighting, movement – are implicated in their composing processes. While some multimodal scholarship focuses on the writer's process of converting traditional prose into multimodal forms, this chapter considers the multimodal pedagogical activities using technology that supported one South Asian writer in generating print-based prose and engaging in critical writing practices in the context of a university developmental English writing course in the Northeastern US. We consider the writer's interaction with an equity video introduced during class, which prompted her examination of linguistic hierarchization in India. The chapter features a multimodal analysis (Norris, 2019) of the video to deconstruct its interconnected semiotic nature and illuminate its potential resources for composing. Subsequently, our case analysis triangulates multiple data sources (writing samples, transcripts) which were generated with the focal learner (who participated in dissertation research by Author 1) to understand the “resemiotization” process which transpired through the learner's interactions with the video (Iedema, 2003). Ultimately, we offer a fine-grained perspective on multimodal idea development and a vision for multilingual pedagogies that integrate videos to support L2 writing development.

Multimodal (Inter)action Analysis in a Nutshell: Philosophy, Theory, Method and Methodology

Article

Full-text available

Nov 2021

Sigrid Norris

This paper presents a concise introduction to Multimodal (inter)action analysis (MIA), which began to be developed in the early 2000s in tandem with technological advances for visual qualitative research. By now, MIA has grown into a fully-fledged research framework, including multimodal philosophy, theory, method and methodology for the study of human action, interaction and identity. With systematic phases from data collection to transcription (including transcription conventions) and data analysis, this framework allows researchers to work in a data-driven and replicable manner moving past common interpretive paradigms (Norris 2019, 2020).

The Construction of Indigenous Identity Through Creativity: An Exploration of Māori and Pacific Identity Constructed Through Dance, Visual Arts, and Creative Writing

Thesis

Full-text available

Jan 2020

Tui Matelau

This thesis examines the construction of hybrid and fluid ethnic identity elements as produced by Māori and Pacific female artists. Ethnic identity studies within New Zealand reveal different types of ethnic identities, and although there is research being conducted into hybrid and fluid Māori and Pacific identity elements, no studies have been done examining the construction of these identities through mediated action. This thesis attempts to fill this gap. Using video ethnography and socio-linguistic interviews, data were collected and analysed utilising multimodal (inter)action analysis (MIA) as the theoretical and methodological framework. Vertical identity production and site of engagement are analytical tools within MIA that allow for the study of the intersection between discourses and mediated actions performed by social actors. These analytical tools were applied to interview and video transcripts selected from the data, following a systematic process of data cataloguing. Analysis of the data is presented in three chapters which show the ethnic and creative identity production of the participants as constructed through the central, intermediary and outer layers of discourse. The first analysis chapter demonstrates the way the participants create art by blending traditional and contemporary features and diverse knowledge, in turn constructing their immediate ethnic and creative identity elements. This analysis is compared to the way the participants verbalise these identity elements within their interviews. The second analysis chapter examines the way experiences of exclusion and inclusion from within their networks shape their continuous ethnic and creative identity elements. The third analysis chapter explores moments of exclusion and inclusion but within larger communities such as mainstream New Zealand, and their ethnic communities. It also illustrates the way in which the participants’ art creates inclusion and shapes the general ethnic and creative identity development of other social actors. Following this, wider discourses and practices are examined using the site of engagement as the analytical tool. This chapter demonstrates the way in which wider discourses such as colonial, superiority/inferiority and racism discourse intersect with practices such as superiority/inferiority, gratitude, and marginalisation and with the mediated actions performed by the participants. This analysis highlights the negative impact these discourses and practices can have on ethnic identity construction for Māori and Pacific social actors. To this end, numerous recommendations are made within the conclusion with the intention of changing these wider discourses and practices. This thesis contributes to knowledge in the area of Māori and Pacific identity studies by utilising multimodal (inter)action analysis to study identity production. It also contributes to the theoretical and methodological framework of multimodal (inter)action analysis.

“Everything Is in the Lab Book”: Multimodal Writing, Activity, and Genre Analysis of Symbolic Mediation in Medical Physics

Article

Full-text available

Jan 2022
WRIT COMMUN

Writing and genre scholarship has become increasingly attuned to how various nontextual features of written genres contribute to the kinds of social actions that the genres perform and to the activities that they mediate. Even though scholars have proposed different ways to account for nontextual features of genres, such attempts often remain undertheorized. By bringing together Writing, Activity, and Genre Research, and Multimodal Interaction Analysis, the authors propose a conceptual framework for multimodal activity-based analysis of genres, or Multimodal Writing, Activity, and Genre (MWAG) analysis. Furthermore, by drawing on previous studies of the laboratory notebook (lab book) genre, the article discusses the rhetorical action the genre performs and its role in mediating knowledge construction activities in science. The authors provide an illustrative example of the MWAG analysis of an emergent scientist’s lab book and discuss its contributions to his increasing participation in medical physics. The study contributes to the development of a theoretically informed analytical framework for integrative multimodal and rhetorical genre analysis, while illustrating how the proposed framework can lead to the insights into the sociorhetorical roles multimodal genres play in mediating such activities as knowledge construction and disciplinary enculturation.

Multimodal (inter)action analysis of task instructions in language teaching via videoconferencing: A case study

Article

May 2021
ReCALL

Online language teaching is gaining momentum worldwide and an expanding body of research analyses online pedagogical interactions. However, few studies have explored experienced online teachers' practices in videoconferencing particularly while giving instructions, which are key to success in task-based language teaching (Markee, 2015). Adopting multimodal (inter)action analysis (Norris, 2004, 2019) to investigate the multimodal construction of instructions in a single case study, we examine instruction-giving as a social practice demonstrated in a specific site of engagement (a synchronous online lesson recorded for research purposes). Drawing on the higher-level actions (instruction-giving fragments) we have identified elsewhere (Satar & Wigham, 2020), in this paper we analyse the lower-level actions (modes) that comprise these higher-level actions, specifically focusing on the print mode (task resource sheets, URLs, text chat, and online collaborative writing spaces) wherein certain higher-level actions become frozen. Our findings are unique in depicting the modal complexity of sharing task resources in synchronous online teaching due to semiotic misalignment and semiotic lag that precludes the establishment of a completely shared interactional space. We observe gaze shifts as the sole indicator for learners that the teacher is multitasking between different higher-level actions. Further research is needed to fully understand the interactional features of online language teaching via videoconferencing to inform teacher training policy and practice.

Fight for focus: attention and agency in sight-translated interaction

Article

Full-text available

Mar 2021

Randi Havnen

This paper presents an analysis of three roleplayed interpreted institutional meetings in which sight translation is part of the interaction. The analysis is based on multimodal (inter)action analysis and utilises the analytical tool of modal density as indication of attention/awareness. This analytical framework is novel in interpreting studies. The data include filmed material from an experimental setting and participants’ reflections about the situation. The findings show variations in sight translation practices and that the shift from interpreting to sight translation affects interactional patterns, particularly social actors’ attention and agency. In my discussion of agency in sight-translated interaction, I argue that interpreters, in addition to translating, need to pay attention to interactional issues related to attention and agency caused by the interpreting method.

Objects and materiality in pragmatic development: Here-and-now to then and-there

Article

Full-text available

Dec 2020

Jarret Geenen

Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse . Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development . Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics . Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . Routledge, London. https://doi.org/10.4324/9780203379493 ; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis . de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280 ; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality . Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77 ) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.

Constructing autism inside and outside the clinic: Exploring relationships between psychoanalytic psychotherapists' and activists' discourses

Thesis

Full-text available

Jul 2019

Georgiou Konstantinos

Negotiation of meaning via virtual exchange in immersive virtual reality environments

Article

Full-text available

Feb 2023

This study examines how English-as-lingua-franca (ELF) learners employ semiotic resources, including head movements, gestures, facial expression, body posture, and spatial juxtaposition, to negotiate for meaning in an immersive virtual reality (VR) environment. Ten ELF learners participated in a Taiwan-Spain VR virtual exchange project and completed two VR tasks on an immersive VR platform. Multiple datasets, including the recordings of VR sessions, pre-and post-task questionnaires, observation notes, and stimulated recall interviews, were analyzed quantitatively and qualitatively with triangulation. Built upon multimodal interaction analysis (Norris, 2004) and Varonis and Gass' (1985a) negotiation of meaning model, the findings indicate that ELF learners utilized different embodied semiotic resources in constructing and negotiating meaning at all primes to achieve effective communication in an immersive VR space. The avatar-mediated representations and semiotic modalities were shown to facilitate indication, comprehension, and explanation to signal and resolve non-understanding instances. The findings show that with space proxemics and object handling as the two distinct features of VR-supported environments, VR platforms transform learners' social interaction from plane to three-dimensional communication, and from verbal to embodied, which promotes embodied learning. VR thus serves as a powerful immersive interactive environment for ELF learners from distant locations to be engaged in situated languacultural practices that goes beyond physical space. Pedagogical implications are discussed.

How EFL Teachers Engage Students: A Multimodal Analysis of Pedagogic Discourse During Classroom Lead-Ins

Article

Full-text available

Dec 2021

Classroom lead-in is the initial stage for motivating students to become engaged in-class interaction. However, little research, to our knowledge, has analyzed the role of teachers’ multimodal competence reflected through their multimodal pedagogic discourse in the realization of the ultimate goals of classroom lead-ins. Based on the data collected from a teaching contest in China, this paper explores how two-winner teachers utilize their multimodal ensembles of communicative modes to engage students during classroom lead-ins. The analysis shows that different communicative modes construct the higher-level action of lead-in, and they are orchestrated into multimodal ensembles for the specific function of each lead-in move. The findings indicate that EFL teachers’ high multimodal competence plays a decisive role in performing classroom lead-ins, and different lead-ins strategies influence the different orchestration of communicative modes. In constructing multimodal pedagogic discourse, teachers build up their professional image and display their personal charm as well. Future research for multimodal discourse analysis and pedagogic research is suggested in the paper.

Multimodal positioning of artifacts in interaction in a collaborative elementary engineering club

Article

Full-text available

Dec 2021

This exploratory case study uses multimodal positioning analysis to determine and describe how a purposefully crafted emergent artifact comes to influence and/or manipulate social dynamics, structure, and positionings of one design team comprised of five third-graders in an afterschool elementary engineering and literacy club. In addition to social semiotic theories of multimodality (e.g., Kress, G. (2010). Multimodality: a social semiotic approach to contemporary communication . New York, NY: Routledge) and multimodal interactional analysis (Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . New York, NY: Routledge, Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Hoboken, NJ: Wiley-Blackwell), Positioning Theory (Harré, R. and Van Langenhove, L. (1991). Varieties of positioning. J. Theor. Soc. Behav. 21: 393–407) is used to examine group interactions with the artifact, with observational data collected from audio, video, researcher field notes, analytic memos, photographs, student artifacts (e.g., drawn designs, built designs), and transcriptions of audio and video data. Analysis of interactions of the artifact as it unfolds demonstrates multiple types of role-based positioning with students (e.g., builder, helper, idea-sharer). Foregrounding analysis of the artifact, rather than the student participants, exposed students’ alignment or opposition with their groupmates during the project. This study contributes to multimodal and artifactual scholarship through a close examination of positions emergent across time through multimodal communicative actions and illustrates how perspectives on multimodality may be analytically combined with Positioning Theory.

The Geosemiotics of a Thai University: The narratives embedded in schoolscapes

Article

Full-text available

Feb 2021
Ling Educ

Andrew Jocuns

The present study reports on the geosemiotics (Scollon & Scollon, 2003) of a Thai University. Walking interview tours (Lou, 2017; Stroud & Jegels, 2014a) of Thammasat University’s Tha Prachan campus were conducted. These interviews reveal how students narrate and take stances towards the geosemiotic artifacts that are found on their campus. The purpose of the study was twofold: 1) to gain an understanding of how students react to the geosemiotics on campus, and 2) to get a sense of their understanding of Thai history. For the latter the university has been the site of several historical events pertaining to Thailand’s spotty relationship with democracy, most notably the Thammasat massacre, and much of this history has been repressed (Huebner, 2017; Winichakul, 2002, 2020). Using the geosemiotic framework to discuss the multimodal make-up of this university’s signs and space, I illustrate how the narrations that emerged during walking interviews serve as lenses through which we are offered a glimpse of how students are socialized to think about the material environment of their campus. We can observe how students take different stances to the signs and places of their campus as well as Thailand’s history. Such narrations reveal traces of socialization on the one hand and the emergence of an affective regime of reverence on the other (Wee & Goh, 2019). The findings contribute to the growing body of literature on schoolscapes in multilingual educational settings (Gorter, 02018; Gorter & Cenoz, 2015).

More than words: how multimodal analysis can inform health professions education

Article

Full-text available

Dec 2020
ADV HEALTH SCI EDUC

The contexts and methods for communicating in healthcare and health professions education (HPE) profoundly affect how we understand information, relate to others, and construct our identities. Multimodal analysis provides a method for exploring how we communicate using multiple modes—e.g., language, gestures, images—in concert with each other and within specific contexts. In this paper, we demonstrate how multimodal analysis helps us investigate the ways our communication practices shape healthcare and HPE. We provide an overview of the theoretical underpinnings, traditions, and methodologies of multimodal analysis. Then, we illustrate how to design and conduct a study using one particular approach to multimodal analysis, multimodal (inter)action analysis, using examples from a study focused on clinical reasoning and patient documentation. Finally, we suggest how multimodal analysis can be used to address a variety of HPE topics and contexts, highlighting the unique contributions multimodal analysis can offer to our field.

BBC Public Radio Channels: Struggling for Survival in a Privatized Public Sphere

Article

Mar 2023

Lawrie Phillips

Movement as Literacy Learning in Elementary Classrooms

Chapter

Full-text available

Mar 2023

Courtney Shimek

Traditionally, literacy in elementary classrooms has been defined as reading, writing, and talk. Literature responses for assessment purposes, then, are often limited to what children say, draw, or write after experiencing a text. Although some work examining students’ dramatic responses to literature has been explored by literacy scholars, few studies have examined spontaneous moments of literacy learning that occur. This chapter examines the physical ways early elementary students respond to course content and provides examples of movement being an integral component of the meaning making process in whole group, small group, and individual settings. These examples illuminate the physical, embodied nature of literacy learning and provide alternative considerations for assessing student’s literacy development, especially in areas of vocabulary development, writing, and comprehension. These examples demonstrate the need for a reconceptualization of what counts as literacy in the elementary classroom and highlights the numerous ways students respond to texts that are often overlooked, ignored, or in some cases, discouraged in traditional classrooms. Instructional strategies that encourage alternative responses in the elementary classroom are offered, including a checklist to assist elementary teachers to consider the physical and embodied responses of students as a form of assessment in their classrooms.KeywordsLiteracyMultimodalityReader responsePicturebooks

Multimodal Discourse on BRICS Produced by Diverse Stakeholders: Identifying Attitudes, Cultures and Perspectives

Article

Full-text available

May 2020

The paper explores the conceptual vision of BRICS in the contemporary world. The study focuses on language and images that are used within BRICS-related institutional communication. We argue that the research is important because of the increasing impact of BRICS on the development of the multilateral and multipolar world. The research aims to offer preliminary considerations with regard to key topics, features and tools of multimodal discourse that comes from the BRICS nations and representatives of other international/regional organisations. This area has not been subject to academic analysis so far. This confirms the novelty of the present study. The research material includes 600 image-text correlated items from BRICS official sources of information and from organisation and institutions, which are not affiliated with the BRICS and refer to national or international actors. The research combined theoretical analysis of literature, empirical investigation of materials within qualitative paradigm, through content-based analysis and manual coding on thematic and pragmatic criteria. The findings reveal different approaches to BRICS that are introduced by different actors through specific coordination of verbal and visual tools, in explicit and implicit ways. The findings show that BRICS sources contain proportioned use of texts and photos of high-ranking official events, socio-cultural features of BRICS countries, and pictures of youth with regard to BRICS mission, values, goals, and policies. This strengthens the concept of equality and human rights provision in the modern world in general and leads to the understanding of the need to include the issues of youth rights and their equality on the BRICS agenda in an explicit way.

Reviving botany in the curriculum: the botanical journey of two Western Australian early childhood teachers

Article

Full-text available

Oct 2022

Environmental education across the early years has become increasingly important in Australia since the implementation of the Early Years Learning Framework and the Australian Curriculum. These documents promote a connection to nature for young children as well as environmental responsibility. In Western Australia, large areas of natural environments are bush spaces, accessible by young children, families and schools. There is no existing research investigating early childhood teacher’s knowledge of plants in these bush spaces and the utilisation of these spaces in teaching botany as part of their teaching practice. The discussion in this article examines part of a larger year-long multi-site case study of the changes in the botanical understanding of two early childhood teachers of children aged 5–8 years, in Western Australian schools both before and after the Mosaic Approach, botanical practices and Indigenous knowledges were incorporated into their teaching practice. This article focuses on the changes of botanical literacies of the early childhood teachers specifically. The findings suggest that using inquiry-based and place-based methods and including First Nations Peoples’ perspectives about plants whilst teaching in the bush can significantly increase the plant knowledge and understanding of teachers, as well their own scientific and botanical literacies.

Autism Multimodal Activism. Negotiating a Pallet of Identities on the YouTube Social Media

Article

Full-text available

Jan 2022

Experiences of children with disabilities during the COVID-19 pandemic in Sweden: a qualitative interview study

Article

Full-text available

Apr 2022

Background The impact of the COVID-19 pandemic on people with disabilities has been described as a ‘triple jeopardy’. Not only have they experienced the negative social impacts of disease control measures, but access to required health services has been affected, and, not least, they are at increased risk of severe outcomes from COVID-19. This study aimed to determine how children with disabilities have experienced the pandemic in Sweden and its impact on their lives. Methods Six children (5–13 years) were interviewed via video conferencing. An interview guide was adapted based on the children’s communicative abilities and included augmentative and alternative communication support. Reflective field notes were included in the analysis. The data were analysed using qualitative content analysis. Results Two themes were identified: The child’s knowledge of Corona raises anxiety and fear; and Boring Corona makes the child even lonelier. The children had knowledge about and were worried about COVID-19, primarily about illness and death of their grandparents. The children longed for their grandparents and other social contacts at school, and life was described as boring and lonely. Many families lacked adequate tools to communicate with their children about the pandemic. Conclusion Given adequate support, children with disabilities and communication difficulties can give insights to their unique life situations. The interviewed children reported significant impact on their life and school life. Children were worried about their grandparents based on their knowledge about the virus. The enthusiasm with which the children engaged in the interviews is testament to the need and right of all children, regardless of communicative competence, to voice their experiences

Multimodal (Inter)action Analysis

Chapter

Dec 2019

Multimodal (Inter)action Analysis was developed to study social interaction based upon the theoretical notion of mediated action. Building on this core concept, Multimodal (Inter)action Analysis includes several theoretical/methodological tools. These tools facilitate analysis which moves flexibly between micro-level moments of interaction and macro-level practices and discourses. In this chapter, the application of mediated action to multimodal analysis is discussed, before the central theoretical/methodological tools are introduced. Tight links are made between the tools used in Multimodal (Inter)action Analysis and the core theoretical tenets, to support robust multimodal interaction research.

Post-heroic heroism: Embedded masculinities in media framing of Australian business leadership

Article

Full-text available

Oct 2021
Leadership

Over time, the relevance of heroic leadership to contemporary corporate environments has been questioned, with media coverage arguing there is a need for alternate, post-heroic forms of leadership. Using a multimodal media analysis, we show how two leading Australian business magazines frame leadership in response to this debate, identifying three distinct frames of leadership. The first frame emphasizes masculinized heroic leadership as normative which reinforces gendered assumptions through differential framing of men and women’s leadership. We then argue media (re)frames post-heroic leadership as a variation of heroic leadership through two further frames; by subsuming feminized attributes into the repertoire of heroic leadership as ‘softer masculinities’ and through the construction of a masculinized post-heroic hero, both applied exclusively to men’s leadership. This (re)framing of heroic leadership has significant implications for perceptions of credible contemporary business leadership.

Semantic macro-structures and macro-rules in visual discourse processing

Article

Full-text available

Jul 2021
Vis Stud

Ahmed Abdel-Raheem

Semantic macrostructures, although strangely ignored or ruled outside most formal linguistics and even some methods of discourse analysis, define the general, overall meanings of discourse, informally called ‘topic,’ ‘theme,’ ‘gist,’ or ‘upshot’ and on the basis of the higher levels of the mental models of the social actors. Macro-structural (topical, crucial) information, according to Teun van Dijk, plays a fundamental role in discourse comprehension and recall― although other factors (such as remarkableness, vividness, etc.) may also affect attention, prominent representation, and hence recall. One can summarise a sequence of pictures by a title, like a summary of a movie. Similarly, a trailer serves to summarise the movie and to indicate what makes it different from other movies on the cinema circuit. But can we summarise a sequence of images by another (‘macro’) image or image-topic? This article discusses semantic macro-structures and processes of multimodal discourse comprehension, formulating the mapping rules underlying the global interpretation of political cartoons. The rules for multimodal discourse processing (involving group-specific knowledge) formulated in this study apply to Arabic-speaking audiences, which should be of interest for scholars in intercultural communication and cognition.

Mapping reflexivity in situ: A multimodal exploration of negotiated textbook discourses in Korean university EFL classrooms

Article

Jun 2021

Christopher A Smith

The present study identifies and maps the reflexive praxis of two experienced English as a foreign language (EFL) instructors as they reconstruct and negotiate textbook material in situ. An abundance of critical studies underscoring social injustices in the contents of globally published EFL textbooks do not sufficiently address the negotiation of their multimodal discourses during class time. Although reflexive teaching practice in language learning classrooms has a robust pool of research, limited scholarly attention has been given to the active negotiation of a textbook's multimodal discourse in Korean university classrooms. The present study asks: (1) How do two instructors at different Korean universities negotiate the contents of an EFL textbook with their students during class? (2) How do the students react to the multimodal discourse negotiated in their textbooks? (3) What pedagogical implications do the findings lend to EFL textbook instruction in Korean university contexts? Using Norris' (2004) framework for video transcription of multimodal interaction in two Korean university English communication classes, the findings reveal that reflexive negotiation between students and instructors is a kind of rhetorical accomplishment that lessens the potential for cultural marginalization in the multimodal discourse of EFL textbooks. Implications suggest that textbook reflexivity in situ raises the value of student EFL learning investments.

Teacher Identity as Discourse. A Case Study of Students in Swedish Teacher Education

Thesis

Full-text available

Feb 2021

Johan Christensson

This thesis comprises three separate studies that together explore how Swedish student teachers construct or produce professional identity in interaction while navigating different institutional and professional instances of teacher education. As a discourse analytical contribution to research on teacher identity, the main theoretical framework is mediated discourse theory (e.g. Scollon 2001a). For data construction and analysis in the studies, different parts of the two related methodologies of nexus analysis (Scollon & Scollon 2004) and multimodal (inter)action analysis (Norris 2011) are employed. Constructed through an ethnographic approach, the interactional data consist of audio and video recordings of interaction in instances from three different components of a Swedish teacher education program: a rhetoric course, a bachelor thesis course in history and teaching placement. Furthermore, the data include observational field notes and interviews, as well as resources used by the participants, primarily written texts. Taking place early on in teacher education, Study I focuses on student teachers performing oral presentations under the fictitious presumption that they are speaking as teachers. Employing the notion of communicative project (Linell 1998), the empirical aim of the study is to shed light on how student teachers manage institutional affordances and constraints affecting interactional role shifts from student teacher to teacher. In Study II, three student teachers are writing their bachelor theses in the subject of history, and the study focuses on the interactional production of teacher identity of one of the students during seminars. While partly being a methodological study, Study II empirically explores how student teachers interactionally relate to their future profession in an academic disciplinary setting, highlighting which actors and institutions are involved in the production of professional identity. Finally, Study III concentrates on a student teacher during his final teaching placement. Focusing on previous experiences resemiotized as stories, Study III highlights how discourse re-emerging from the historical body (Nishida 1958) can be used in interaction in producing identity. The results suggest that the production of teacher identity by the student teachers is a co-operative and communicative task, where previous experiences as well as an anticipatory perspective on the teaching profession are important features. The three studies identify different resources that can be used and adapted by students to suit different purposes in professional identity production, described as textual resources, embodied resources, and narrative resources. In turn, the different uses of such resources motivate the need for studying identity in interaction with an approach where ethnographic and sociocultural knowledge is part of the analysis. The creative use of resources in identity production highlights that students use knowledge and experience linked to academic and professional as well as everyday discourse in producing professional identity. Presuming an interest in opportunities for student teachers to develop professional identity during their education, it appears fruitful to reflect upon how potential resources are designed and implemented in teacher education, and how institutional affordances and constraints affect the possibilities of using them.

Establishing a zone of prioritized curricularivity: a critical approach to negotiating multimodal discourses in EFL textbooks

Article

Full-text available

Nov 2020

Christopher A Smith

English as a foreign language (EFL) university programs in expanding circle nations often pressure instructors and students to use globally published textbooks for reasons more socio-political than pedagogical. While some critical studies underscore multimodal discourse to be an under-appreciated source of dominant social narratives in EFL textbooks, few have investigated their live negotiation in classrooms. To address the challenges negotiating potentially harmful social narratives in EFL textbooks, the present study proposes a two-step model for achieving a zone of prioritized curricularivity (ZPC). The model informs reflexive teaching practice in EFL instruction because it necessitates an understanding of a) the curricular commonplaces of a particular EFL program and b) the power and ideologies in the multimodal discourse of their textbooks, to mitigate perceived social injustices in the textbook lessons as they are negotiated "in situ." Demonstrated in vignettes, featuring two EFL courses at Chung-Buk National University in Cheong Ju city, Korea, two instructors used the ZPC framework to inform their reconstruction of multimodal discourses in their EFL textbooks to inculcate student involvement and participation. A novel, multimodal interactional analysis of video recordings looked at proxemics, gaze, spoken language, head movement, auditory emphasis, and gesture and discovered that each instructor recontextualized, neutralized, or skipped much of the multimodal discourse in the lessons. The findings suggest that a ZPC is achieved when the efforts by instructors to recontextualize textbook lessons in situ is met with positive feedback from students in the classroom. The implications suggest a ZPC can help instructors and students and in EFL programs in any expanding circle culture because it can simultaneously improve student learning/acquisition in the classroom, diminish dominant, culturally marginalizing textbook content, while raising the value of student investment in EFL learning.

Title: Establishing a zone of prioritized curricularivity: a critical approach to negotiating multimodal discourses in EFL textbooks

Conference Paper

Nov 2020

Christopher A Smith

University EFL programs in expanding circle nations often pressure instructors and students to use globally published EFL textbooks for reasons more socio-political than pedagogical. While some critical studies underscore multimodal discourse to be an under-appreciated source of dominant social narratives in EFL textbooks, few have investigated their live negotiation in classrooms. The present study proposes that a two-step framework for achieving a zone of prioritized curricularivity (ZPC) can inform reflexive teaching practices by the instructor. The framework requires an understanding of the curricular commonplaces of a particular EFL program and an understanding of the power and ideologies in the multimodal discourse (Machin & Mayr, 2012) of their textbooks, to mitigate perceived social injustices in lesson contents textbook lessons as they are negotiated “in situ.” Demonstrated in vignettes featuring two EFL courses at Chung-Buk National University in Cheong Ju city, Korea, two instructors used the ZPC framework to inform their reconstruction of multimodal discourses in their EFL textbook lessons to inculcate student involvement and participation. A novel, multimodal interactional analysis (Norris, 2019) of video recordings in two EFL classrooms discovered that each instructor recontextualized, neutralized, or skipped much of the multimodal discourse in the lessons. The findings suggest that a ZPC is achieved when the efforts by instructors to recontextualizing textbook lessons in situ is met with positive feedback from students in the classroom – noted as heightened attentiveness, happy or cheerful participation, and enthusiastic discussion. The implications suggest a ZPC can help instructors and students and in EFL programs in any expanding circle culture because it can simultaneously improve student learning/acquisition in the classroom, diminish dominant, culturally marginalizing textbook content, while raising the value of student investment in EFL learning.

Digital intimacy and ambient embodied copresence in YouTube videos: construing visual and aural perspective in ASMR role play videos

Article

Jul 2020

Michele Zappavigna

This article explores how digital intimacy is construed through ambient embodied copresence in ‘personal attention’ role play videos, a type of Autonomous Sensory Meridian Response (ASMR) video that has become popular on YouTube. ASMR is the experience of positive sensations in response to visual and aural stimuli. Online video sharing platforms have provided a way for people who experience these ASMR sensations to watch, produce, and disseminate ASMR-invoking material. In ASMR role play videos the YouTuber constructs a conceit (e.g. a beautician visit) and uses visual and aural resources to encourage the feeling in the ambient viewers that they are there with the YouTuber experiencing the interaction. This article considers how these videos forge an immersive faux-interactional context, and invoke the visual and aural perspectives and embodiment of ambient viewers. The dataset explored is a playlist of 116 role play videos from the GentleWhispering ASMR YouTube channel, the most popular ASMR channel at the time of writing.

Crisis communications at the start of the Covid-19 pandemic: A case study of the Ghanaian president’s fourth update on coronavirus

Article

Jun 2023

Veronika Koller

The Covid-19 pandemic was a testbed for crisis communication, leading to recommendations on how to meet communicative goals and several individual case studies. This paper contributes to the latter by engaging in a detailed three-level analysis of an early, pivotal address to the nation by Ghana’s president Nana Akufo-Addo. In terms of infection rates and deaths, Ghana has been much less severely impacted by the pandemic than other countries, making it worthwhile to look at the role of official communications. This study investigates how the president addressed the public at the beginning of the Covid-19 pandemic, in what ways the linguistic features of his address reflected the specific political context, and what potential impact his language use had on the behaviour of the public. Findings show that linguistic and, to a lesser extent, visual elements represent the president as powerful, authoritative, but somewhat detached from the audience. However, this is balanced by direct appeals to the same audience, whose cooperation he seeks to win rather than enforce. This balance reflects the political and socio-cultural context of the text, as further evidenced by comments on the address on Akufo-Addo’s Facebook page.

Systemic Functional Linguistics: A Complete Guide

Book

Jul 2023

Organizational identity design: A multimodal discourse analysis of Australian university homepages

Thesis

Full-text available

May 2023

Nataliia Laba

This thesis studies web homepages to understand the complex social practice of organizational identity communication on a digital medium. It examines how designs of web homepages realize discourses of identity through the mobilization and orchestration of various semiotic resources into multimodal ensembles, addressing critical organizational visual identity elements (‘logo,’ ‘corporate name,’ ‘color,’ ‘typography,’ ‘graphic shapes,’ and ‘images’), communicative content of the page, and navigation structures. By examining these three ‘strata’ of organizational identity communication, it investigates how a homepage uses formal design elements and more abstract principles of composition, such as spatial positioning and content ordering, as resources for making meaning. The data consists of three complementary sets drawn from thirty-nine web homepages of Australian university websites in 2020. Data set #1 includes four homepages for an in-depth study of organizational identity designs; data set #2 consists of 400 images from the ‘above the fold’ web area as the most strategic space on four homepages between the years 2015 and 2021; data set #3 is comprised of eight historical versions of a selected web homepage between the years 2000 and 2021, with three most representative designs for an in-depth investigation. Grounded in the discourse-analytic approach informed by multimodal social semiotics, the thesis adopts a mixed-method approach to data analysis. It applies multimodal discourse analysis combining the Genre and Multimodality model (Bateman, 2008; Bateman et al., 2017) to document the structural design patterns and social semiotic (metafunctional) approach to address the meaning potentials of the identified patterns; (Kress & van Leeuwen, 2021); content analysis (Bell, 2001; Rose, 2016) and visual social actor framework (van Leeuwen, 2008) to identify key representational tropes and visual personae. The study reveals the role of design as a mediating tool between the participants of discourse – the rhetor-institution/designer and envisaged audiences – and offers systematic insights into the uses of semiotic resources, both material (e.g., formal design elements and navigation structures) and nonmaterial (e.g., spatial considerations and content structuring), all contributing to the production of meanings and fostering identification with such meanings in the form of association with the university’s identity. Addressing the subtle differences and shifts in the form and function of key layout structures and strategies of viewer engagement, the study concludes that is plural – each university constantly revises semiotic choices and their multimodal composition to achieve specific rhetorical purposes. Together with several visual design choices, five identified strategies of viewer engagement – proximation, alignment, equalization, objectivation, and subjectivation – promote the university as a place of opportunity, achievement, sociality, and intellectual growth for a student as an individual and as a member of the community. The current research contributes to the emerging collaboration between multimodality, organization studies, and branding, recognizing the complexities and importance of multimodal communication in web-mediated texts amidst the critically increased roles of marketization and social presence in the current higher education landscape.

Exploring Heroic Characters in Red Chinese Movies: A Multimodal Perspective

Article

Full-text available

May 2023

Red Chinese movies depict heroic individuals who make significant contributions to the nation, and these films play a vital role in constructing the national image of new China. This paper explores the images of heroic individuals in red Chinese movies using multimodal action theory. The findings indicate that these individuals are characterized by their selfless sacrifices for their country, fearless perseverance, embodiment of international communist ideals, and unwavering devotion and warmth. The study also reveals that facial expressions, body movements, and language serve as critical mediational means in shaping these characters, while other forms of mediation are underutilized. Recommendations include employing more mediational means to create more nuanced and complex characters in future red Chinese movies.

Multimodal Machine Learning in Prognostics and Health Management of Manufacturing Systems

Chapter

Jun 2023

Prognostics and health management (PHM) is a crucial enabler to reduce maintenance costs and enhance the availability and reliability of manufacturing systems. In the context of Industry 4.0, these systems become more complex and can be monitored by different types of sensors. The quality and completeness of data are crucial factors for the success of any PHM task in this paradigm. Here, we investigate the possibility of exploiting additional data sources in manufacturing besides monitoring sensors, e.g. production line cameras or maintenance reports. We first present the terminologies of multimodal learning and the potential it holds for industrial PHM. We then further explore the development and notable works in this field applied to other domains, look at the relevant works in PHM, and finally present a case study to demonstrate how multimodal learning can be performed to improve PHM processes.KeywordsMultimodal dataMultimodal learningPrognostics and health managementDeep learning

The Programmatic Dimension of Political Communication on Twitter: The Case of Costa Rica’s 2022 General Elections

Preprint

Full-text available

Mar 2023

This research seeks to analyze the use of the social network Twitter by presidential candidacies in the context of the first round of the 2022 electoral campaign in Costa Rica. This is a proposal that is articulated under an exploratory and descriptive scope, so that in the first place it tries to understand the characteristics under which politicians exercise a presence in digital environments and make general use of Twitter in the context of an electoral campaign, Secondly, it seeks to identify the programmatic dimension in the publications shared by the profiles of the candidacies with the highest voting intentions. The work makes a contribution based on the methodological background of the Manifesto Research on Political Representation (MARPOR), to inquire about electoral communication exercised through a social network, from a multimodal approach. The findings prove the existence of a clear programmatic dimension within the activity of the candidacies on the platform that is articulated not only with government plans, but also with conjunctural elements of the campaign such as debates. Likewise, the work explores some non-programmatic uses and offers new lines of research within the field of political communication in Costa Rica.

Göstergelerarası çeviri bağlamında sözlü çevirmenin rolüThe interpreter’s role in the context of intersemiotic translation

Article

Full-text available

Feb 2023

Burak Özsöz

Tarafların aynı dili ve ekinsel artalanı paylaşmadığı, bir sözlü çevirmen aracılığında gerçekleşen bildirişim ortamında, çevirmenin rolü katılımcılardan birinin kaynak sözcelerinin erek dile aktarılması ile sınırlandırılamaz. Bu sebepten, diller arası çeviri kadar diliçi ve göstergelerarası çeviri ediminde de bulunan çevirmene biçilen rol, durağan bir çerçevede değil, o bildirişim ortamının devingenliğinde ele alınmalıdır. Böylesi devingen bir yapıya yönelik araştırmalarda, dildışı olana kıyasla ekseriyetle dilsel olanın öncelendiği düşünüldüğünde, etkileşimin dildışı bileşenlerinin bildirişim ortamına verdiği katkıya yönelik çalışmalara ihtiyaç duyulmaktadır. Her iki gösterge dizgesini de bütüncül bir bakış açısıyla eşit ölçüde merkezine alan çalışmalarda, çoklu ortam yaklaşımının (multimodal approach) benimsenmesi önem arz eder. Çoklu ortam yaklaşımı, çevirmenin nasıl ve hangi ölçüde diğer eyleyenlerin davranışlarına müdahil olduğunun gözlenmesine imkân tanır. Bu çalışmada, çoklu ortam yaklaşımı çerçevesinde, sözlü çevirmenin muhataplarının dildışı eylemlerine verdiği tepkilerin etkileşim ortamına olan katkıları incelenecektir. Bu doğrultuda, Ekman ve Friesen’in (1969; 1981) dilsel iletişim öğelerinin dildışı iletişim öğeleri ile etkileşimine dair ortaya koydukları altı parçalı sınıflandırma, sözlü çevirmenin rolünün göstergelerarası çeviri bağlamında irdelenmesinde yol gösterici olacaktır. Çalışmanın derlemini görsel işitsel kayıt altına alınan iki ayrı diş ve saç estetiği özgün diyalog ortamının çevriyazısı oluşturmaktadır. Sözlü çevirmenin göstergelerarası aracılığının katılımcıların sözcelemsel varlığının tesis edilmesinde önemli bir işlevinin olduğu ortaya koyulmuştur.

Bold and impactful: a reappraisal of Gunther Kress’s (social) semiotic legacy in the light of current multimodality research

Article

Full-text available

Feb 2023

Hartmut Stöckl

This paper is a critical appreciation of some of Gunther Kress’s central (social) semiotic notions: i.e., motivation, materiality, rhetorical aptness and semiotic mode versus medium. These will be discussed in relation to four landmark models of sign-making and semiosis by Saussure, Peirce, Bühler and Jakobson. Based on these comments, the paper identifies the persistent difficulties current multimodality research faces in defining mode and in devising linguistically unbiased grammars of non-verbal modes. Finally, the argument is advanced that multimodal genre and discourse interpretation in particular deserve to be re-developed. The paper critiques Kress’s insistence on motivation as a universal principle of sign use and his overemphasis on materiality to the detriment of grammar, while praising his overall (social) semiotic legacy for multimodality research as far-sighted and lastingly influential.

Multimodality Analysis: Visual Representation in Bli-Bli 12.12 and Lazada 12.12 TV Commercials

Article

Full-text available

Jun 2022

This study examines the different meaning in the visual analysis of Bli-Bli 12.12 and Lazada 12.12 advertisements. This study aims to determine the difference in meaning in visual analysis using the theory of reading images from Kress and Leeuwen (2006). Visual data analysis on these two advertisements is divided into three levels, namely representational, interactional, and compositional. The data is in the form of screenshots taken from the ad videos of bli-bli 12.12 and lazada 12.12. This research method uses descriptive qualitative research. The results showed some similarities and differences between the two advertisements in the theory of Reading images by Kress and Leeuwen (2006). The similarities in the visual data of the two advertisements can be seen in the dominance of advertisements by primary announcements and the difference in the different amounts of data in each part of the analysis point.

Multimodality as civic participation: The case of Thailand’s rap against dictatorship

Article

Nov 2022

In October 2018, a collaboration between young rap artists in Thailand’s Indy rap scene, Rap Against Dictatorship (RAD), launched a video criticizing the ruling Junta that went viral within days of publication. The Junta soon after released its own video as a response to RAD. The production and publication of both videos are what Scollon (2001) calls social actions mediated by a distinct cultural toolkit. This study analyzed how modes such as music, text, color, camera angle, gestures, voice, image and iconicity emerged in both videos to realize scalar differences in civic participation. The Junta’s video represents a high sociolinguistic scale, whereas RAD realizes a lower scale. In a time of political unrest in Thailand, sociolinguistic scale and the semiotic resources that people employ to realize scales are a lens to analyze how different stakeholders address various perspectives of the political situation and appeal to different levels of civic participation.

Who counts? Legitimate solutions in construction activities in preschool

Article

Full-text available

Oct 2022
INT J TECHNOL DES ED

As has been pointed out in previous research, teacher-led learning plays an important role in developing preschool children's technological skills and technological self-esteem. What is missing in research are more detailed analysis of how the children’s and teachers’ actions and interactions shape the learning process. In order to study this within the field of construction, an action research project was conducted, where construction activities were developed, implemented and revised in an iterative procedure. Data from the second cycle were analyzed for this article using graphic transcriptions and multimodal analysis, with a focus on action, interaction and experience from a pragmatist perspective. Our findings show that children who quickly and decisively engage with the material, the teachers and their peers in suggesting which material to use and/or how the material can be used, end up in a central role in the design process. These children (or their actions) often get legitimized by the teachers. Thus, in order to give children access to equal opportunities in the construction activities, it is important for teachers to understand how the children’s construction-focused actions become constitutive and what their role in that process is.

AUTISMO E METÁFORAS MULTIMODAIS: IMPACTO DISCURSIVO DE AÇÕES E DE CONCEPÇÕES CAPACITISTAS

Article

Jul 2022

Alex Bezerra Leitão

Este artigo tem como objetivo investigar a (co)construção metafórico-multimodal que advém do impacto discursivo de ações e de concepções capacitistas em relação ao autismo. Para tanto, adoto uma aproximação entre a Análise de Discurso Crítica, orientado por Fairclough (2003, 2006), os estudos da metáfora conceptual, influenciado por Fauconnier e Turner (2003, 2008) e por Vereza (2010, 2017), a metáfora multimodal, encaminhada por Forceville (1988, 2009) e por Sperandio (2015), e a Sociolinguística Interacional, em consonância com Goffman (1998 [1979]) e com Tannen e Wallet (1998 [1987]). No âmbito da pesquisa qualitativa, o corpus desta pesquisa foi gerado a partir de discussões com pessoas autistas sobre um texto multimodal publicado na revista Saúde, em 2019, por meio da plataforma WhatsApp. Os resultados apontam que metáforas socioculturalmente situadas, instanciadas em práticas sociais que representam e que identificam pessoas autistas como ‘solitárias’, ‘seres de outro planeta’, ‘anjo azul’, subjazem a metáfora conceptual AUTISMO É ESTAR FORA, colaborando com a manutenção do capacitismo em estruturas sociais.

Multimodal Critical Discourse Analysis on the advertisement of Colgate

Article

Jul 2022

In semiotics there is an analysis called multimodal analysis. This study aimed to analyze the advertisement of Colgate in audiovisual formed, when Colgate advertisement is one of the Super Bowl commercial breaks in USA. This study applied on semantic approach focused on the multimodal discourse analysis. The method applied in this study was descriptive qualitative method. The result of this study was that Colgate Total advertisement covers five aspects in a multimodal semiotic system, namely linguistic, visual, audio, gestural, and location aspects. These five aspects come together to make the ad appearance very attractive and easy to remember. A product that is advertised, if many people are able to remember the words, visuals, accompanying music, and actors in the advertisement, it means that the message in the product has successfully reached the audience. It could be predicted that these products would be able to survive in the market. This ad for Colgate Total toothpaste products displays clear and interesting information about what people want from a product, namely its ingredients and the benefits of using it. Messages in the form of language carried out through spoken language and written language make the audience understand more about what is being promoted. This ad makes a unique impression on the audience because of its short duration but important overall message. The linguistic, visual, audio, gestural, and location aspects of a multimodal semiotic system were all addressed in this advertisement. These five elements combine to create a visually appealing and memorable advertisement. Keywords: multimodality, advertisement, analysis

Verbal Representation Analysis of Meaning in Gojek and Grab TV Commercials

Article

Full-text available

Aug 2020

This study aims to examine the difference in meaning in verbal analysis in Gojek and Grab advertisements. The purpose of this research is to find the difference in meaning in verbal analysis at the level of Systemic Functional Linguistics, namely ideational, interpersonal, and textual metafunction. The data were analyzed using the theory of Systemic Functional Linguistics Halliday (2006). The data is the utterances contained in the Gojek and Grab advertising videos. The method of this research uses descriptive qualitative research. The results showed that there were some differences in the two advertisements in using Systemic Functional Linguistics, namely mood analysis, attitude, modality, graduation, and theme. The five differences are dominated by Gojek advertisements.Key words: systemic functional linguistics, semiotics, advertisementAbstarkPenelitian ini bertujuan untuk mengkaji perbedaan makna dalam analisis verbal pada iklan Gojek dan Grab. Tujuan dari penelitian ini adalah untuk mengetahui perbedaan makna dalam analisis verbal pada tingkat Systemic Functional Linguistics yaitu ideational, interpersonal, dan textual metafunction. Data dianalisis dengan menggunakan theory Systemic Functional Linguistics Halliday (2006) dengan pedekatan semiotik. Datanya adalah ujaran-ujaran yang terdapat didalam vidio iklan Gojek dan Grab. Metode dari penelitian ini menggunakan descriptive qualitative research. Hasil penelitian menunjukkan bahwa terdapat beberapa perbedaan dalam kedua iklan dalam menggunakan Systemic Functional Linguistics, yaitu mood analysis, attitude, modality, graduation, dan theme. Lima perbedaan itu didominasi oleh iklan Gojek.

Toward Multimodal Pragmatics: A Study of Illocutionary Force in Chinese Situated Discourse

Book

Full-text available

Jan 2022

Lihe Huang

Classic pragmatic theories emphasize the linguistic aspect of illocutionary acts and forces. However, as multimodality has gained importance and popularity, multimodal pragmatics has quickly become a frontier of pragmatic studies. This book adds to this new research trend by offering a perspective of situated discourse in the Chinese context. Using the multimodal corpus approach, this study examines how speakers use multiple devices to perform illocutionary acts and express illocutionary forces. Not only does the author use qualitative analysis to study the types, characteristics, and emergence patterns of illocutionary forces, he also performs a quantitative, corpus-based analysis of the interaction of illocutionary forces, emotions, prosody, and gestures. The results show that illocutionary forces are multimodal in nature while meaning in discourse is created through an interplay of an array of modalities. Students and scholars of pragmatics, corpus linguistics, and Chinese linguistics will benefit from this title.

Алгоритм агрегації програмних метрик і її застосування при тестуванні програмного забезпечення.

Article

Oct 2021

Yu.S. Abharian

У статті розкрито алгоритм агрегації програмних метрик і її застосування при тестуванні програмного забезпечення. Визначено генезис формування наукової думки, щодо агрегації програмних метрик. Розкрито методологію тестування програмного забезпечення з відокремленням схеми процесу тестування програмного забезпечення. Наголошено, що в якості базису для визначення рівнів агрегації програмних метрик при тестуванні програмного забезпечення спочатку слід визначити процес і складові блоки на прикладі системи тестування. Підкреслено, що агрегація програмних метрик може проводитися на рівні винесення рішень, на рівні значень відповідності та на рівні ознак і зразків. Відзначено, що агрегація на першому та другому рівнях відбувається після залучення засобу порівняння, в той час як рівні третій та четвертий проводять операції до того, як пристрій порівняння видасть результуючі дані. Описано математичні властивості методів агрегації, а саме, домен, діапазон, інваріантність та розкладання. Представлено алгоритм агрегації програмних метрик до рейтингів, використовуючи порогові значення на основі еталонних показників. Покроково описано реалізацію алгоритму та визначено параметричні значення процесу агрегації. Наголошено, що зведення окремих вимірювань до рейтингів здійснюється за допомогою дворівневого процесу, заснованого на двох типах порогів, а окремі вимірювання об’єднуються в профілі ризиків за допомогою метричних порогів. При цьому, профілі ризику агрегуються за 5-бальною зірковою шкалою за допомогою порогових значень. Агрегація дворівнева, на першому рівні агрегація здійснюється шляхом обчислення відносного розміру системи, що підпадає під кожну категорію ризику, на другому рівні об’єднання профілів ризику в рейтинг здійснюється шляхом визначення мінімального рейтингу, для якого сукупний відносний розмір усіх категорій профілю ризику не перевищує набору порогів 2-го рівня. Здійснено тестування програмного забезпечення Dia. Профіль ризику для Dia містить 73,3% коду у низькому ризику, 8,2% помірного ризику, 7.9% високого ризику та 10.7% дуже високого ризику. Використання інтерпольованої функції дає рейтингове значення 2,99, рейтинг для Dia має три зірки.

Tam pam pam pam and mi – fa – sol: constituting musical instructions through multimodal interaction in orchestra rehearsals

Article

Full-text available

Oct 2021

Using multimodal (inter)action/conversation analysis, the present contribution inventories the repertoire of higher-level actions that constitute musical instruction in orchestra rehearsals. The study describes the modal complexity of the instructional actions as built from a varied combination of speech, gesture, gaze, vocalizing and body posture/movement. A high modal intensity of speech and vocalizing is explained with recourse to their contextually useful modal reaches. While some modes, like vocalizing and body posture appear to be action-specific, others turn out to be pervasive default modes. Besides modal intensity, the study also attends to the transitioning between higher-level actions through gaze and the role of the score as frozen action. The analyses help demystify orchestra rehearsals as a special type of professional communicative interaction, which builds on a rich multimodal texture motivated by recurring instructional functions. The methodological rationale demonstrated will be suited to exploring the social variation of instructional interaction in orchestra rehearsals.

One-school-for-all As Practice – A Nexus Analysis of Everyday Digitalization Practices

Thesis

Full-text available

Sep 2021

Lars Almén

Gesture studies and anthropological perspectives: An introduction

Article

Full-text available

Dec 2019

This contribution is the introduction for the special issue of Gesture entitled “Anthropology of Gesture”. As such, it raises two main questions: how do gestures contribute to the field of anthropology? And, inversely, how anthropology can improve our understanding of gesture and gestural behaviours? Of particular importance for this special issue, is the emphasis on what Lempert called “the anthropological sensibility” which aims at taking a more cultural and ethnographic approach to the study of gesture, especially but not only in cross-cultural contexts. The last part of this introduction presents all the contributions of this special issue.

Methodology of a phonetic experimental research of English video presentations prosody

Article

Sep 2020

N. Ya. Hrad

The Experiences of Refugee Youth from Burma in an American High School: Countering Deficit-Based Narratives through Student Voice

Book

Full-text available

Nov 2020

How do presenters engage with their audience? Speakers’ multimodal interpersonal behaviour in research dissemination talks

Article

Nov 2020

Speakers in research dissemination talks are challenged with the need to connect with an audience that does not necessarily share their knowledge and expertise. This communicative situation can be particularly challenging for speakers using English both as a foreign language and for academic purposes. This study combines multimodal and ethnographic methods to explore how speakers of dissemination talks engage with their public. It focuses on four presenters’ use and combination of language, paralanguage, kinesics, proxemics and gaze during intensive moments of engagement. The results show that these interpersonal rich points consist of dense multimodal ensembles that serve to shorten the distance between presenters and their audiences. The findings suggest that a skilful orchestration of modes can be greatly beneficial to achieve the desired level of audience engagement. Therefore, developing speakers' multimodal communicative competence should be a priority in English for specific and academic purposes (ESP/ EAP) training.

Systematically Working with Multimodal Data: Research Methods in Multimodal Discourse Analysis.

Abstract

Recommendations

Third Year Status of a Summer Faculty Immersion Program

Towards a Subjectively Devised Parametric User Model for Analysing and Influencing Behaviour Online...

Research methods for adult educators in Africa

Live operative workshops: A critique