Kay L. O’Halloran

International education is impacted by multiple discourses, in particular the discourse of university as an educational institution responsible for producing and curating knowledge for the public good, pursuing truth and transforming student life, and the neoliberal marketing discourse which portrays the university as a business organization provid...

Multimodal analysis traditionally involves conceptualising abstract frameworks for language, images, and other resources and their intersemiotic relations (e.g. text and image relations) and then demonstrating these frameworks with some examples. This scenario has changed with the recent move towards multimodal approaches to big data analytics whic...

This paper uses a Systemic Functional Multimodal Discourse Analysis (SF-MDA) approach to analyse tweets from the Twitter accounts of Presidents Barack Obama (@Barack Obama) and Donald Trump (@realDonaldTrump). The tweets were posted during the last nine months of President Obama’s effective presidency and the first nine months of President Trump’s...

This paper presents the findings from a case study which explores how students engage with learning materials mediated through 360-degree panoramic videos. The case study extends findings from a project which developed an online platform for viewing and annotating 360-degree videos, and for providing data analytics (heatmaps, viewpoint tracking and...

This article investigates the discursive construction of international students on Australian and Chinese university webpages that are directed at international students. The international student theme webpages of three Australian and three Chinese universities were examined in relation to how verbal and visual semiotic resources were co-deployed...

Visual thinking (VT) has potential for dealing with recurring problems in CLIL instruction. • Systemic Functional Multimodal Discourse Analysis facilitates identification of VT features. • Implications relevant for the design of printed and e-learning teaching materials

Multimodality, involving the study of meaning arising from the integration of language with images and other resources in multimodal texts, interactions and events, addresses the fundamental need to understand human communication in the current age of digital technology. However, multimodality is not considered to be a discipline per se at present....

In recent years there has been increased academic and professional interest and awareness in approaches to English language teaching (ELT) that take a plurilingual approach. This is often combined with a multimodal stance. The outcome of this combination is an approach to English language teaching that integrates multiple languages and multiple sem...

The Cambridge Handbook of Systemic Functional Linguistics - edited by Geoff Thompson May 2019

This study investigates an adult writer's multimodal composing and traversals across linguistic, symbolic and visual modes in a research article and PowerPoint slides. Adopting a process-oriented approach, and drawing upon analytical perspectives from ‘academic literacy’ and the concept of ‘the transmodal moment’ informed by social semiotics, this...

There is a growing interest in the use of visual thinking techniques for promoting conceptual thinking in problem solving tasks as well as for reducing the complexity of ideas expressed in scientific and technical formats. The products of visual thinking, such as sketchnotes, graphics and diagrams, consist of ‘multimodal complexes’ that combine lan...

This paper uses a social semiotic perspective to analyze Donald Trump’s domination of media coverage of the US presidential campaign from 16 June 2015, when he announced his candidacy for nomination as the Republican candidate until 8 November 2016, when he was elected as President of the United States. The paper argues that one of the keys to Dona...

This paper reports on a study that is part of a project which aims to develop a multimodal analytical approach for big data analytics, initially in the context of violent extremism. The findings reported here tested the application of natural language processing models to the text of a sample of articles from the online magazines Dabiq and Rumiyah,...

This paper adopts a multimodal approach to the latest generation of digital mathematics textbooks (print and online) to investigate how the design, content, and features facilitate the construction of mathematical knowledge for teaching and learning purposes. The sequential organization of the print version is compared to the interactive format of...

This study takes a systemic functional multimodal social semiotic approach to the analysis and discussion of image and text relations in two sets of data. First, patterns of contextualisation of images and text in the online magazines Dabiq and Rumiyah produced by the Islamic extremist organisation which refers to itself as Islamic State (referred...

This paper uses a multimodal mixed methods approach for exploring general recontextualisation patterns of violent extremist images in online media. Specifically, the paper reports on the preliminary findings of a preliminary study which investigates various patterns in the reuse of images which appear in ISIS’s official propaganda magazines Dabiq a...

In 2015, the Australian Council for Educational Research (ACER) was tasked with developing literacy and numeracy skills testing for pre-service teachers. All undergraduate and postgraduate trainee teachers are now required to pass these literacy and numeracy tests at some stage on their journey to becoming a teacher; for commencing students from 20...

Two notable features of the current conflict in Syria and Iraq are the number of foreign fighters from western countries fighting for Sunni militant organisations, and the use of the Internet and social media by some extremist groups to disseminate propaganda material. This article explores how the group which refers to itself as Islamic State and...

Aim We compared survival in out-of-hospital cardiac arrest (OHCA) cases recognised at initial dispatch (“primary recognition”) with those subsequently recognised as OHCA ?(“secondary recognition”) and those not recognised as OHCA (“non-recognition”). Methods We analysed cases of paramedic-confirmed OHCA in Perth, Western Australia (WA), from Janua...

Aim Clear and efficient communication between emergency caller and call-taker is crucial to timely ambulance dispatch. Within the Medical Priority Dispatch System,¹ the first opportunity that callers have of describing the situation is after the scripted prompt “okay, tell me exactly what happened”. However, in 60% cases, call-takers introduce a sl...

Intersemiotic translation is viewed as the basis of cultural communication through which ideas are circulated, translated and explained using language, images and other semiotic resources. From this perspective, intersemiotic translation is conceptualised through the principle of resemiotisation which is concerned with how semiotic choices are tran...

This book is a first attempt to map the broad context of performance studies from a multimodal perspective. It collects original research on traditional performing arts (theater, dance, opera), live (durational performance) and mediated/recorded performances (films, television shows), as well as perfor-mative discursive practices on social media by...

Traditional performing arts, such as theatre, dance, and opera, are multimodal meaning-making events, as they incorporate a wide range of semiotic resources, such as language, music, and body movements, for example, which are orchestrated to produce meanings in specific contexts and cultures. Other less easily classified performance-related events,...

This study employs a multimodal social semiotic approach to the analysis of text and image relations in material produced by the violent extremist organisation known as Islamic State in Iraq and al-Sham (ISIS). The study focuses on iconisation, where meanings are condensed and interpersonally charged through ‘bonding icons’ which embody the organis...

This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large dat...

Multimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of C...

This study examines the types of verbal, pictorial, and multimodal metaphors in the genre of sex education picture books for young children in Mainland China. Although being an educational discourse genre that is essentially concerned with transmitting scientific facts, sex education picture books employ a range of metaphors that categorize and con...

This paper introduces the Special Issue on the languages of performing arts and is therefore aimed at designing how the context of the latter can be illuminated by socio-semiotic and multimodal approaches to communication. In this Special Issue, performances and performing arts are described as multimodal semiotic acts that co-deploy a range of sem...

This paper adopts a multimodal social semiotic approach for exploring the semiotic changes involved in the transformation of a novel into stage and screen productions. It examines how semiotic resources are deployed in each medium through elements of mise-en-scène, such as speech, music, sound, lighting, props, staging, and cinematographic techniqu...

This study conducts a social semiotic analysis of six sex education picture books for preschool children in the People’s Republic of China. Following van Leeuwen’s framework for analysing the discursive construction of social actors, this paper examines two types of social actors directly associated with the education of human sexuality: the physic...

This paper proposes a pedagogical approach for teaching and learning critical thinking through multimodal analysis – that is, ‘multimodal analysis for critical thinking'. The approach builds on the conviction that students require competencies that move beyond traditional notions of literacy to meet the changing demands posed by media and technolog...

Semiotics has been making progressively inroads into marketing research over the past thirty years. Despite the amply demonstrated conceptual appeal and empirical pertinence of semiotic perspectives in various marketing research streams, spanning consumer research, brand communications, branding and consumer cultural studies, there has been a marke...

As leadership discourses in higher education are increasingly being mediated online, texts previously reserved for staff are now being made available in the public domain. As such, these texts become accessible for study, critique and evaluation. Additionally, discourses previously confined to the written domain are now increasingly multimodal. Thu...

This exploratory study introduces the systemic approach and the explicit teaching of a meta-language to provide conceptual tools for students for the analysis and interpretation of multimodal texts. Equipping students with a set of specialised vocabulary with conventionalised meanings associated with specific choices in multimodal texts empowers th...

Interactive digital technologies with facilities for developing conceptual frameworks and storing multidimensional data at different levels of description have the potential to advance research in semiotics (i.e., the study of sign systems and processes). Such an approach, called multimodal digital semiotics (O–Halloran et al., Text and Talk: Speci...

Multimodal learning, as an effective method for helping students to understand complex concepts, has attracted much research interest recently. Using more than one media in the learning process typically makes the study material easier to grasp. In the current study, students annotate linguistic and visual elements in multimodal texts by using geom...

In this chapter, mathematics is considered to be a multimodal semiotic process involving the use of language, images, and mathematical symbolism, each with their own systems of meaning which integrate to create mathematical knowledge. The specific relations between linguistic, visual, and symbolic systems have led to semantic expansions in mathemat...

The aim is to develop a multimodal (i.e. multisemiotic) approach to the mathematics register, where language is considered as one resource, often a secondary one, which operates in conjunction with mathematical symbolism and images to create meaning in mathematics. For this purpose, the linguistic features and grammatical difficulties of scientific...

Social photo sharing platforms on the Internet (e.g. Flickr) host billions of publicly accessible photos captured by millions of individual users from all over the world. These user-contributed and geo-tagged photo collections provide insights into human sociocultural life and provide important clues for understanding people’s engagement and reacti...

The analysis of social media data is an emerging research field that aims to study the dynamics of urban life. In this study, we adopt a multimodal digital humanities approach to combine the analysis of text-based social media data with visual social media data in an interactive map to investigate urban life in Singapore from a social semiotic pers...

Complementing cognitive theories which attribute the understanding of visual metaphors to situational and cultural contexts, this study adopts a social semiotic perspective to investigate how visual images themselves are constructed to cue conceptual metaphors. The visual realization of metaphors in representational, interactive and compositional m...

Using a critical ‘hypermodal approach’ informed by social semiotics, this paper investigates the changing discourses of marketization found on the website of the National University of Singapore over a 14-year period. Analysis of visual-spatial features and action potentials of progressive versions of the site reveals changes in the website functio...

This article presents a reading of David Byrne's Envisioning Emotional Epistemological Information, an art work created with MicroSoft's presentation software PowerPoint, as an instance of creative research on semiotics and semiotic technology. It reveals commonalities and differences between Byrne's ideas about PowerPoint and related ideas from li...

A "multimodal digital semiotics" approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News....

In what follows, the principal functionalities of Semiomix are firstdescribed (see also O’Halloran et al., 2012) and then illustrated via screenshots from Professor W. Gilbert Strang’s first lecture in Linear Algebra from Massachusetts Institute of Technology (MIT) Open Courseware (OCW).1 Following this, Professor Strang’s use of language and gestu...

There is a close historical association between multimodality and technology, in two senses. First, technologies have enabled a significant expansion of the range of media by which humans communicate, especially in terms of recording, replaying, and transmitting across time and space (that is, mediating) multimodal discourse.Keywords:computer media...

This study examines how emotive meaning is represented in visual images using the semiotic resources of facial expression, touch and body orientation. Complementing the cognitive metaphorical interpretation, the visually represented emotive behaviors are theorized as partial iconic representations of real life behaviors, which are in turn indexes o...

Paul J. Thibault (b. 1953) has a distinguished research background in general and applied linguistics in areas relating to the sociocultural and biological-physical dimensions of human meaning-making activity, which he has developed into a transdisciplinary field of knowledge involving social semiotics, cognitive science, psychology, biology, and c...

This article presents the design and functionalities of interactive software for multimodal analysis currently being developed in the Multimodal Analysis Lab, Interactive Digital Media Institute (IDMI) at the National University of Singapore. The software is being used for the annotation, analysis, search and retrieval of semantic patterns in unifi...

Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are well developed in the literature, and there is progr...

The positioning and movement of the teacher in the classroom are fundamental to the pedagogical process. Specific spaces in the classroom take on certain meanings because of the nature of pedagogic discourse that occurs on the site and the positioning and distance of the site relative to the students and the teaching resources. Spatial pedagogy is...

Building on research in systemic functional approaches to multimodal discourse analysis, this paper develops macro-analytical and micro-analytical techniques for transcription and analysis of a multimodal advertisement film text. This paper proposes the usefulness of the visual semantics stratum to investigate inter-frame relations in the multimoda...

This paper examines how linguistic and image elements in a text may work at cross-purposes, so that, despite an apparent overall visual-verbal coherence, one semiotic resource undermines the message of another. This effect has implications for the study of intersemiosis and the making of meaning that occurs when semiotic resources, such as language...

The present paper discusses issues arising from the use of an interactive digital software tool to analyse multimodal communication. The focus is on the ways in which such technical resources and associated techniques enable the analyst of social semiosis to apply different types of analysis, and provide the site for critical reflection upon the re...

Critical discourse analysts are increasingly required to account for multimodal phenomena constructed through language and other resources (e.g. images, sound and music) and to relate high-level critical insights on the social motivations of these texts to their realizations in low-level expressive phenomena, and vice versa. In this paper, we use i...

The overall research plan of this multiyear project is to apply methods and principles of dynamical systems theory (DST) to base data derived from systemic functional theory (SFT) analysis of text and multimedia resources, with the aim to identify and track evolving patterns, in particular those related to stability and instability. The goal of the...

The paper discusses the challenges faced by researchers in developing effective digital interfaces for analyzing the meaning-making processes of multimodal phenomena. The authors propose a social semiotic approach as the underlying theoretical foundation, because interactive digital technology is the embodiment of multimodal social semiotic communi...

In this paper we present a collaborative work between computer and social scientists resulting in the development of annotation software for conducting research and analysis in social semiotics in both multimodal and linguistic aspects. The paper describes the proposed software and discusses how this tool can contribute for development of social se...

The systemic functional (SF) approach to multimodal discourse analysis (MDA) is concerned with the theory and practice of analysing meaning arising from the use of multiple semiotic resources in discourses which range from written, printed and electronic texts to material lived-in reality. The SF-MDA approach developed in this article explores the...

Systemic frameworks for discourse analysis of visual/verbal and mathematics texts are used to examine the functionality of language? visual display and mathematical symbolism, and the meaning arising from interaction and interdependence between these semiotic codes in joint constructions. The nature of the interaction between options, classified as...

Selon Lemke (1998), les textes mathematiques sont necessairement multisemiotiques dans la mesure ou les systemes specifiques a chaque semiotique construisent la realite d'une maniere differente. Dans cet article, l'A. examine les fonctions du symbolisme mathematique, de la representation visuelle et du langage dans le contexte de l'evolution du sym...

The impact of the multisemiotic nature of mathematics on classroom discourse is examined from a systemic functional linguistic perspective. Mathematical discourse is multisemiotic because it involves the use of the semiotic resources of mathematical symbolism, visual display and language. There is constant movement between the three resources as th...