Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
Multimodal discourse
analysis in health
communication: sketching
out the field
QUALITATIVE HEALTH COMMUNICATION
VOLUME 4, ISSUE 1, 2025
ISSN: 2597-1417
Polina Mesinioti1
NAME OF DEPARTMENTS AND INSTITUTIONS:
1 Department of Health Sciences, University of York, UK
CORRESPONDING AUTHOR:
Polina Mesinioti. E-mail: polina.mesinioti@york.ac.uk
ABSTRACT
Background: Multimodal discourse approaches have only recently gained
consistent prominence in health communication research. The theoretical and
methodological underpinnings of each approach, and their contribution to the
health communication domain, require further articulation. Aim: This article aims
to sketch out the field, showcasing the methodological strengths and limitations
of multimodal discourse approaches, and their potential contribution to health
research. Methods: The article reviews four established and emerging multimodal
discourse approaches used in health communication research. A comparative lens
is taken, scrutinising each approach in terms of its theoretical underpinnings,
methodological implications, and analytical constraints. Findings: Key points of
convergence and divergence among the approaches are identified, with all
approaches sharing a commitment to investigating multiple modes and their
relationships in creating meaning within health research. The main point of
differentiation lies in what each approach considers the unit of analysis: Systemic
Functional Multimodal Discourse Analysis focuses on semiotic resources,
Mediated Discourse Analysis on action, Conversation Analysis on conversational
order, and Multimodal Critical Discourse Analysis on power and social structures.
Conclusions: Future directions include a focus on materiality, the integration of
emerging technologies, and the development of new analytical tools for
investigating crisis communication. All these can offer deeper insights into health
communication and enhance professional practices and patient outcomes.
KEYWORDS
Discourse analysis, health communication, multimodality, multimodal discourse
analysis
BIOGRAPHY
Polina Mesinioti is a sociolinguist in the Department of Health Sciences at the
University of York. She serves as Lead Researcher on the Response Study, a large-
scale mixed-methods evaluation of the National Health Service (NHS) patient
safety framework. Her research spans medical sociology, sociolinguistics,
interactional studies, and healthcare policy, reflecting a multidisciplinary
approach. Before joining York, she completed her PhD in Applied Linguistics at
the University of Warwick.
E-mail: polina.mesinioti@york.ac.uk. ORCiD: 0000-0002-2071-7303.
2
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
Introduction
Multimodal discourse analysis
Multimodal discourse analysis is an umbrella term for approaches examining how meaning is
constructed through multiple semiotic resources (or modes), including, amongst others,
verbal cues, gazes, facial expressions, hand gestures, images, spatial arrangements, and the
use of objects. The concept of multimodality is rooted in the late 1970s, when scholars in
social semiotics started paying attention to semiotic resources other than talk, such as
gestures and images, exploring how interaction extends beyond spoken and written language
(Halliday, 1978). This early work laid the groundwork for the systematic co-examination of
modes and the establishment of multimodal discourse analysis in the late 1990s/early 2000s,
when scholars increasingly recognised the complexity of meaning-making across different
semiotic modes and started developing analytical frameworks to address it (see, for instance,
Jewitt & Kress, 2003). A foundational – now classic – work is Kress and van Leeuwen’s Reading
Images: The Grammar of Visual Design, published in 1996, which articulated the need to
consider multiple semiotic modes in discourse analysis and provided one of the first
frameworks to do so. Multimodal discourse angles became more prominent with the rise of
video recordings and the emphasis on analysing ‘naturally occurring’ data, as these provided
the tools to capture the intricacy and nuances of real-life interactions and analyse them in a
way that more accurately reflects how people use multiple modes of communication in
everyday life.
There are various approaches to conducting multimodal discourse analysis, some of which I
detail in turn below. A key principle underpinning all of them, though, is the interplay of
multiple modes and the need to analyse them holistically. Well put by Bezemer and Jewitt
(2010, p. 184), “the meanings realized by any mode are always interwoven with the meanings
made with those other modes co-present and co-operating in the communicative event’’. The
idea that the different modes are interrelated and should be examined as a whole is also found
early on in Goodwin and Goodwin (1992), according to whom talk, intonation, and body
movements should be treated as elements that are integrated with one another rather than
as distinct separate channels.
Despite multimodal approaches being on the rise, persistent analytical constraints have been
noted, including systematising methods in multimodal research and creating ‘stable analytical
inventories’ of multimodal semiotic resources (Jewitt, 2013); bringing various semiotic modes
together under a cohesive analytical framework (Kress & Van Leeuwen, 2001); and integrating
different tools and techniques for analysing multimodal interactions (Norris, 2004). Against
this background, I consider the methodological affordances of each approach here, with a
particular focus on their application in the health communication domain.
Multimodal discourse analysis encompasses a broad scope, extending across various fields
where multimodal ‘texts’, in a broad sense, are prevalent, including the following:
• Education: classroom interactions, teaching materials, curriculum development,
assessment practices
3
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
• Media studies: advertising, websites/digital platforms, news media, film and
television
• Political discourse: campaigns, debates, protests
• Theatre and performance: stage interactions, stage design, audience interaction
and engagement
• Health communication: doctor-patient and inter-/intra-team interactions, public
health campaigns, crisis communication etc. – I elaborate on this strand in turn
below.
Health communication and multimodal discourse analysis
Health communication has evolved into a distinct field over the last three decades, with its
scope encompassing healthcare encounters, patient safety, health promotion, disease
prevention, the management of health and illness, and the improvement of patients’
experiences and quality of life (Jerome & Ting, 2022). Health communication scholars are
interested in how the ways in which health-related information is communicated influences
health behaviours, public health outcomes, and healthcare experience. Despite the focus on
how such information is communicated, though, paradoxically, discourse approaches remain
underemployed in the field of health communication, where monomodal approaches have
been traditionally the norm.
When it comes to multimodal discourse approaches, the gap is even greater, with Brookes
and Hunt’s (2021) edited volume being one of the very few ones focusing exclusively on this
matter, bringing together established and emerging discourse approaches on health
communication. Multimodal discourses have only recently attracted attention in
health/health communication research, including a range of contexts and topics, such as video
recorded clinical consultations, the affordances of public health campaigns, multisemioticity
in online health communities, health/illness social media discourses, and public health
discourses. In this context, multimodal approaches have the potential to advance our
knowledge and understanding of clinical practice (and thus patient safety), policy making, and
the patient experience.
The recent COVID-19 pandemic in 2019 has impacted the field, leading to a surge in
multimodal discourse analyses of public health campaigns. These analyses aim to understand
the effectiveness of such campaigns in reaching diverse audiences and promoting public
compliance with health guidance in crisis situations. Illustrative examples include Gill and
Lennon’s (2022) investigation of how the UK government has semiotically constructed and
utilised fear in COVID-19 adverts, Al-Subhi’s (2024) work on public health COVID-19 posters in
Saudi Arabia, and Ope-Davies and Shodipe’s (2023) work on COVID-19 online public health
campaigns in Nigeria. The work conducted on multimodal discourses of COVID-19 significantly
– and rapidly – advanced the field of multimodal discourse analysis, demonstrating how
multimodal resources can influence public opinion, mitigate the effects of the pandemic, and
contribute to public safety.
Despite this recent rise in multimodal discourse approaches, the methodological tools and
distinctive features of each approach warrant further articulation, as does their contribution
4
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
to the health communication domain specifically. Building on this agenda, this paper delves
into four established and emerging approaches, illustrating their theoretical underpinnings,
methodological affordances, and analytical constraints.
Approaches to multimodal discourse analysis
I introduce here four multimodal approaches to health discourse, including both established
and emerging ones. The list is not exhaustive – I return to this in the Discussion. These four
approaches have been selected on the basis that they offer complementary perspectives for
a comprehensive multimodal analysis of health communication. They cover a wide range of
communicative aspects, from the micro-level of interaction to the broader discursive and
social elements.
1. Systemic Functional Multimodal Discourse Analysis (SF-MDA)
2. Mediated Discourse Analysis (MDA)
3. Conversation Analysis (CA)
4. Multimodal Critical Discourse Analysis (MCDA)
A comparative lens is taken, scrutinising each approach in terms of its theoretical origins,
methodological underpinnings, and analytical constraints. Emphasis is placed on the
methodological implications of each approach and the practical application of theory,
addressing foundational aspects (frequently used datasets, relevant contexts, methodological
tools, key sources).
Systemic functional multimodal discourse analysis
SF-MDA is underpinned by a social semiotic approach, whereby the social interpretation of
language and its meanings is extended to the whole range of modes of representation and
communication employed in a culture (Kress, 2009; O'Halloran, 2008). Scholars working under
such an approach are interested in how meaning is embedded within images and artefacts,
including the study of speech, gestures, gazes, images, and writing (Bezemer & Jewitt, 2010).
The Systemic Functional (SF) approach originates in Kress and van Leeuwen’s (1996) and
O’Toole’s (1994) work and largely draws on Halliday’s (1978) theory. Halliday’s contribution
lies not only in making visible the role of multiple semiotic resources but, more importantly,
in foregrounding the interaction of these resources, viewing culture as ‘‘a set of semiotic
systems, a set of systems of meaning, all of which interrelate’’ (Halliday & Hasan, 1985, p. 4).
Methodologically, as SFL views language as a social semiotic system, it provides the analytical
tools to establish the ‘grammar’, or ‘systems’ underlying meaning making. Context-wise, SF-
MDA is distinguished for its focus on visual imagery and its interface with language for the
construction of meaning. Due to this emphasis on ‘artefacts’, this approach has been
prototypically associated with the analysis of advertisements, video campaigns, films, and
printed texts (textbooks, newspapers etc.). Kress and van Leeuwen extended Halliday’s theory
of SFL’s three metafunctions – ideational, interpersonal, and textual – to multimodal discourse
analysis:
5
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
• Representational Metafunction: Examines how different modes represent and
construct the world, including participants, actions, and circumstances.
• Interactive Metafunction: Examines how modes are used to enact social
interactions, express attitudes, and build relationships between the producer of a
text and the audience.
• Compositional Metafunction: Examines how modes are organised to create
coherent, meaningful texts, including aspects such as composition, layout, framing,
and the relationship between different modes.
Prevalent concepts in SF-MDA are intersemiosis and semiotic cohesion: the former refers to
the interplay between multiple semiotic resources, while the latter describes how system
choices come together to make the text cohesive (O’Halloran, 2008).
In health research, studies explicitly positioning themselves under an SF-MDA approach are
fewer, compared to some of the other approaches covered below (but many are informed by
a social semiotic approach; I return to this in the Discussion). An early example is Iedema’s
(2001) study of a documentary about waiting lists and budget management at a Melbourne
hospital. Drawing on the three metafunctions mentioned above, Iedema captured how certain
modes have been used to favour clinicians’ viewpoint (e.g., clinicians being level with the
camera, while administrators being filmed from lower angles, connotating different degrees
of power), illustrating how ‘‘organizational, orientational and representational patterns and
choices enhance and reinforce each other’’ (Iedema, 2001, p. 193). Turning to more recent
work, Yang (2017) drew on women's magazines in the US, examining the relationship between
the visual and verbal elements in the portrayal of skin cancer: in her findings, two competing
discourses were identified, with the verbal discourse highlighting the harmful effects of sun
exposure, while the images promoted the attractiveness of tanning. O’Halloran et al. (2019)
also focused on visual artefacts, demonstrating the affordances of SF-MDA via the analysis of
the World Health Organization (WHO) Ebola webpage. Through the co-examination of texts,
photographs, graphs, hyperlinks, and videos, they identified the textual, interpersonal, and
ideational meanings in the website’s subsections, underlining their intersemiotic connections.
Their work provides a detailed, step-by-step guide to conducting SF-MDA, highlighting its
methodological contribution to the field, which is the provision of a solid framework for
holistically exploring how the three metafunctions (representational, interactive,
compositional) work together to convey complex health-related meanings.
Turning to limitations, going back to intersemiosis, traditionally, scholars working under an SF-
MDA approach have primarily focused on the interaction of language and images. Studies
looking at the interface of other modes, particularly in health research, are sparser. This
constitutes a criticism of the uptake of the approach so far, though, rather than an inherent
limitation of its affordances. Another challenge is that, as it requires the identification and
analysis of the meta-functions of different semiotic resources (and their interface), the
analysis is multi-faceted and highly technical, rendering it a time-consuming and laborious
process (Iedema, 2001; O’Halloran et al., 2019). Finally, due to the strong focus on
grammatical systems, wider social aspects are sometimes neglected in SF-MDA analyses
compared to the three other approaches reviewed below. This issue was highlighted by Ledin
and Machin (2018), who critiqued SFL for its weak and somewhat superficial understanding of
context, particularly in its application to written texts.
6
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
Mediated discourse analysis
MDA (also called ‘nexus analysis’) was established by Scollon and Scollon (Scollon, 2001;
Scollon & Scollon, 2004), although many trace it back to Vygotsky (1981), who understood all
action as mediated by cultural tools. MDA aims to bring together discourse, agency, and
practice into what Scollon calls a ‘nexus of practice’, defined as ‘‘the intersection of multiple
practices (or mediated actions) that are recognisable to a group of social actors’’ (Lane, 2014,
p. 9). The unit of analysis in MDA approaches is social action and its complex relations with
discourse – mediated discourse analysts are concerned with what people do with discourse,
rather than just discourse itself. This social action, in turn, is always mediated by language,
technologies, visual elements etc., which shape and are shaped by the social context in which
they occur. In contrast to the social semiotic approach, in which visual imagery is often the
primary focus, in MDA the starting point is (inter)action, which is broadly viewed as
encompassing not only talk but also the use of artefacts, images, gestures and other semiotic
resources, with all these being considered significant to the extent that social actors interact
with them.
A key theoretical underpinning of MDA is that ‘‘the focus of mediated discourse analysis is not
discourse per se, but the whole intersection of social practices of which discourse is a part’’
(Jones & Norris, 2005, p. 4): this distinguishes MDA from turn-taking approaches, such as CA
(see following section). Compared to other multimodal discourse approaches outlined here,
MDA is the least centred on language per se, without, at the same time, denying that language
often plays the central role in interaction (Norris, 2004). Core concepts underpinning MDA are
sites of engagement and historical bodies: sites of engagement place emphasis on context and
are defined by Scollon (2001, p. 4) as a ‘‘window that is opened up through the intersection
of social practices and meditational means (cultural tools) that make that action the focal
point of attention of the relevant participants’’. Simply put, sites of engagement are all
surrounding factors/conditions (tools, actions, events, place, time, participants) that render
an action possible. Historical bodies, on the other hand, bring to the analytical fore social
actors themselves, their familiarity with certain social practices, and prior experiences and
knowledge.
In health communication research, MDA has not been employed as widely as other discourse
approaches. Notable examples, however, include Jones’ (2014) work on the concept of risk in
public representations of AIDS-HIV, illustrating the benefits of employing MDA compared to
more traditional approaches. Murdoch et al. (2015) applied Scollon and Scollon’s (2004)
framework to understand a particular social action – speaking about illness management – of
a patient who was considered nonadherent to asthma medication. Their analysis of MDA’s
three key elements (intersecting discourse, historical bodies, and interactional order) shed
light on how, in talking about asthma management, patients negotiate ‘‘complex discursive
spaces where they work to present themselves in ways in which they wish to be understood
and judged’’ (p. 290). More recently, Landqvist and Blåsjö (2024) employed an MDA approach
to explore communication professionals' experiences of COVID-19 and identify strategy
changes. In their analysis of interviews and textual material (i.e., internal institutional
documents), they illustrated how the individual (communication professionals), group
(professional teams), and discourse (health and risk discourses) interact to convey crucial
7
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
information during a global health crisis. Methodology-wise, their work makes a significant
contribution as it explicitly brings into the analysis the abstract concept of time – specifically,
how temporal aspects play a role in interactions and ‘‘how people relate to time’’ (p. 518).
Although time is central to MDA, it is often assumed and underdiscussed (for the
materialisation of time in MDA see Scollon and Scollon, 2004). Other than the emphasis on
temporal dimensions of health communication, a key methodological contribution of MDA is
that, with its shift from an exclusive focus on language to a more consistent consideration of
various tools (texts, technologies, artefacts), it allows researchers to understand how cultural,
social, and institutional factors shape health messages.
Turning to MDA’s constraints, as was the case with SF-MDA, capturing the various semiotic
resources and understanding their contribution to the performance of social actions can be
methodologically challenging. Social actions are always mediated by multiple layers of
discourse, tools, and technologies. The systematic analysis of the interplay between these
mediators can be intricate. Defining analytical boundaries is also difficult, given the
interconnectedness of social actions across different sites and times (Scollon & Scollon, 2004).
Further, methodological challenges are identified in pinpointing the role of temporal aspects,
which is inherent in the approach. Such challenges include the need to embed the temporal
sequencing in the analysis (i.e., a focus on how visual elements unfold over time) and
challenges in data representation, as researchers often rely on annotation software (like
ELAN) to code the timing and duration of multimodal elements accurately. Finally, I mentioned
earlier how a rich insight into the sociocultural context and interactants’ previous experiences
is a prerequisite for MDA work: access to this background knowledge, though, often goes hand
in hand with in-depth ethnography, which can pose significant challenges in terms of getting
access, ethical considerations, and time constraints, particularly in clinical contexts, where
ethical issues are – and should be – always at the forefront of research considerations.
Conceptually close to MDA, as it also centres on interaction, is CA, which is introduced in turn
below.
Conversation analysis
CA was first established as a sociological method in the 1960s, originating in
ethnomethodology (Garfinkel, 1967; ten Have, 2012) and the work of Sacks, Schegloff, and
Jefferson (e.g., Schegloff & Sacks, 1973), with its philosophical basis grounded in
phenomenology (Giorgi, 1985). It is concerned with the sequential organisation of ‘talk-in-
interaction’, with its guiding principle being that interaction exhibits ‘‘order at all points’’
(Sacks, 1992(I), p. 484). The preferred and more frequently used datasets in CA approaches
are audio and video recordings of ‘naturally occurring’ interactions, while the micro-analytic
approach has been marked by the detailed Jeffersonian transcription system (I expand on
multimodal CA transcripts below). As this is a well-established discourse approach, the rest of
the discussion focuses on CA’s contribution to multimodality.
Although MDA and CA share a commitment on multimodal interaction, CA is a turn-taking-
based approach, whereby talk is still considered the primary unit of analysis. An early interest
in other-than-talk aspects of interaction under turn-taking approaches is traced back at least
to the 1970s. Goodwin (1979), for instance, examined the role of gaze in interaction, analysing
gaze direction as indicative of the recipient’s attention to the speaker, and argued that an
8
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
utterance cannot be conceptualised as a unit apart from the situated occasion of its
production. Similarly, Schegloff (1984, p. 273) illustrated the ways hand gestures ‘‘are
organized, at least in part, by reference to the talk in the course of which they are produced’’,
acknowledging that word production is accompanied by aspects such as posture, gesture,
facial expression, preceding talk and voice quality.
The CA tradition has been influential in integrating various semiotic resources into the field of
discourse analysis, conceptualising early on spatiomaterial aspects and the interactional
context itself as interactively achieved. A more consistent interest on the role of the body in
interaction is traced around 2001 by Nevile (2015), who used the term ‘embodied turn’ to
refer to the point when the body attracted social scientists’ – and particularly conversation
analysts’ – interest in the study of language and social interaction, as well as the spatial and
material environment in which the interaction occurs. Similarly, Mondada (2016) introduced
the ‘visual turn’, within which action is conceptualised as ‘‘situated, indexically organized, and
specifically shaped by, as well as shaping, the social and material context in which it happens’’
(p. 339). More recent advancements in multimodal CA make a case for expanding the field to
encompass multisensorial practices, such as touch, smell, and taste: for Mondada (2019, p.
60), this interactional conceptualisation of multisensoriality ‘‘invites us to deepen our
understanding of what makes embodied details accountable, within their fine-grained
multiple temporalities, and how they contribute to the publicly intelligible shaping of actions’’.
Zooming in on the healthcare context, multimodal CA work has primarily focused on
healthcare teams’ interactions and doctor-patient interactions so far, unpacking diagnostic
processes, the delivery of care, epistemic claims, and the performance of leadership and
teamwork within teams. Illustrative work includes Heath et al. (2018), who examined the ways
in which materials are passed by the scrub nurse to the surgeon in the operating theatre,
viewing this process as a ‘‘collaborative production of complex tasks in and through bodily
action and interaction that reflexively reconstitutes the occasioned sense and significance of
material objects and artefacts’’ (p. 298). Their study is a valuable example not only because it
illustrates a detailed multimodal CA, but also foregrounds the role of objects in embodied
interaction and draws on real-life video recordings of surgical procedures – an increasingly
rare dataset nowadays. Remaining in the realm of operating theatres, Mondada (2014)
analysed a chief surgeon’s instructions addressed to his assistant during a surgical operation,
demonstrating surgical practice as a collaborative achievement which relies on finely tuned
embodied coordination between staff members. Mondada is a key advocate of multimodal
CA, with this (and her other) work sketching out what a systematic micro-analysis of embodied
interaction looks like. Note, also, that since the early 2000s, Mondada has been developing
her own multimodal CA transcription conventions, a highly technical system, which is now
used by many CA scholars (Mondada, 2018). Turning to doctor-patient interactions, Fatigante
et al. (2021) drew on video recorded oncological visits to examine the contribution of patients’
companions in the consultation. Their findings illustrated that the companions’ roles were
‘‘the contextualized results of complex temporal, sequential, multimodal and multiparty
arrangements of all participants' actions’’ (p. 19). CA’s focus on the micro-level is one of its
key methodological strengths: by thoroughly examining how participants take turns, manage
interruptions, and display responsiveness, researchers can identify effective communication
strategies, rendering CA a rich methodological approach, particularly valuable in
understanding the nuances of doctor-patient interactions.
9
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
Turning to key challenges, I already alluded to CA’s highly technical multimodal transcription
system. Although these systems allow the annotation of many semiotic resources, including
gestures, gazes, movement in the material space, use of objects etc., they also make the
transcription process extremely slow and tedious. The lack of standardisation in transcription
conventions for multimodal data further adds to the complexity. Another issue is that it can
become resource-intensive, which limits its accessibility: video recordings require advanced
recording equipment, while specialised training and sometimes software (e.g., ELAN) are
sometimes a prerequisite for the transcription. Moreover, it has already been noted that CA
mostly relies on audio and video recordings of naturally occurring interactions: in healthcare
settings, potential medicolegal implications (see, for instance, the National Health Service’s
rising litigation costs in the UK), new legal standards, and strict data protection principles (e.g.,
the UK General Data Protection Regulation/GDPR principles) have made such access incredibly
difficult (for a discussion see also Mesinioti, 2021).
Multimodal critical discourse analysis
The last approach reviewed here is MCDA, one of the emerging approaches. MCDA integrates
the principles of CDA and multimodal analysis and integrates principles from the SF-MDA
approach; I further elaborate on the porous boundaries between these approaches in the
Discussion. As is the case with all CDA approaches, it is underpinned by a strong focus on
circulating ideologies and social processes/structures along with the examination of linguistic
forms (cf. Fairclough’s framework, 1995). MCDA is interested in how the various semiotic
resources relate to their contexts of production and consumption, as well as the broader
sociocultural context in which they take place (Machin, 2013). What distinguishes it from the
previous approaches is its explicit focus on how the social context, namely, issues of power,
ideology, and social justice, is communicated and perpetuated through various semiotic
resources. Machin (2016) notes how, in MCDA, discourse is always present in the sign at all
levels, and the sign, in turn, shapes the ‘ideological consciousness’.
Methodologically, at the core of MCDA is the concept of recontextualisation, which refers to
the process of taking elements (such as participants, processes, or settings) from one context
and adapting or using them in a different context. This process often includes abstraction,
addition, substitution, and deletion to transform and repurpose these elements to fit new
contexts (Machin, 2013). In essence, MCDA examines how semiotic resources are utilised in
the recontexualisation of such elements, and why certain meanings are transformed when
resources are moved across different contexts.
In the health context, datasets employed so far are health campaigns and public health
announcements, health-related advertisements, newspaper and magazine articles, and health
information websites. Studies demonstrating MCDA include Gill and Lennon’s (2022) recent
work on COVID-19 information adverts, which articulated how the government attempted to
ensure compliance through implicit and explicit fear-evoking semiotic interactions. An MCDA
approach is also consistently taken in Brookes et al.’s ongoing work on discourses of dementia:
Brookes et al. (2018), for instance, examined representations of dementia in national
newspapers in the UK, with their detailed analysis revealing how various semiotic strategies
were employed for the discursive construction of dementia as a dreaded, devastating, and
agentive disease, while patients were represented via discourses of loss and victimhood. More
10
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
recently, Putland et al (2023) also took an MCDA approach for the examination of ideologies
circulating dementia in Artificial Intelligence (AI)-generated images, with their findings
illustrating a lack of visual diversity and the reproduction of prominent visual discourses, such
as a biomedical focus on dementia. Their innovative, for our field, focus on AI-generated
images could pave the way for future research: this is further unpacked in the Discussion.
MCDA’s methodological affordances allow the investigation of implicit power structures and
ideologies across multiple semiotic modes, which, in the field of health communication, has
the potential to ensure that communication materials promote equity and avoid reinforcing
stereotypes or biases.
What can be particularly challenging in MCDA is that it presupposes a solid understanding of
circulating ideologies and underpinning social structures: robust comprehension of the
sociocultural context is thus a requirement. Methodology-wise, another issue is that, given
that a) MCDA is still evolving, and b) there is significant methodological plurality within CDA
approaches, there is no single, standardised methodology for conducting analysis, leading to
significant variation within the field (cf. Wodak & Meyer, 2015). This is not necessarily a
weakness of the method, but it can be daunting. Moreover, it has already been mentioned
that a key aim of MCDA is to uncover how multimodal texts are embedded within broader
social practices. The relationship between these texts and social practices is never one-to-one,
though, making the task of analysing how each mode both reflects and shapes these practices
particularly challenging. There have also been some criticisms of MCDA, and, more broadly,
CDA, for being too ‘selective’ and ‘partial’ (for a discussion of this and other limitations, see
Machin & Mayr, 2012). Finally, issues of reflexivity and researchers’ beliefs are important
considerations in (M)CDA approaches: although this is always the case in discourse
approaches, CDA’s focus on issues of power and researchers’ interpretation of the
sociocultural context, which constitutes an integral part of the analysis, amplify this.
In the next and final section, I summarise these approaches, before outlining ongoing
developments in the fields of multimodal discourse analysis and healthcare communication,
suggesting avenues for future research.
Discussion
I presented above four key multimodal discourse approaches employed in the healthcare
context and health communication, focusing on their theoretical origins, methodological
implications, and analytical constraints and limitations. In reviewing these, I took a
comparative lens to illustrate key points of convergence and divergence. This information is
summarised below, in Table 1. It is worth noting that this was not intended to be an exhaustive
list of discourse approaches used in health communication: other, less established,
approaches include Visual Discourse Analysis (VDA), which is concerned with visual elements
of health campaigns, social media, etc. (Albers, 2013; Traue et al., 2019), and Multimodal
Narrative Discourse Analysis (MNDA), which has been so far used for the analysis of patient
narratives in digital health platforms, online forums, and blogs (Liang, 2019).
11
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
Multimodal
approach
Underpinning
approach
Focus
Key
advocates/
Illustrative
examples
Frequently used
datasets in
health
communication
research
Analytical constraints
Systemic
Functional
Multimodal
Discourse
Analysis
Social
semiotic
approach
Artefacts/visual
imagery
(and the ways
meaning is
constructed in
those)
Iedema
(2001);
O'Halloran,
(2008);
Jewitt et al.
(2016)
Health campaign
advertisement;
healthcare
websites and
online platforms;
patient
educational
videos and
applications
Currently limited to the
interaction of language
and image; requires a
multi-faceted and
highly technical
analysis; broader social
context is often
overlooked
Mediated
Discourse
Analysis
Action-based
approach
Social actions
(discourse is
considered part
of those)
Jones & Norris
(2005); Scollon
(2001); Scollon
& Scollon
(2004)
Video recordings
of clinical
encounters;
institutional
documents;
health policy
implementation
documents
Difficulty in defining
analytical boundaries;
requires deep
background knowledge
Multimodal
Conversation
Analysis
Turn-taking
approach
Talk
(and its situated
performance)
Mondada
(2016, 2019);
Heath et al.
(2018)
Video recordings
of face-to-face
and online
clinical
encounters,
consultations,
and healthcare
teams’
interactions
Resource-intensive;
highly technical
transcription system;
no consensus on
transcription
conventions
Multimodal
Critical
Discourse
Analysis
Power/social
justice-based
approach
Ideologies/social
structures
(and their
relationship to
semiotic
resources)
Brookes et al.
(2018);
Machin
(2013);
Machin &
Mayr (2012)
Health policy
documents;
advertisements;
public & social
media health
campaigns;
health
communication
materials in crisis
situations
Requires solid
understanding of
circulating
ideologies/sociocultural
context; significant
methodological
plurality
Table 1. Established and emerging multimodal approaches to discourse analysis.
Undoubtedly, all four approaches share common ground, particularly in their commitment to
the following two key principles: a) the consideration of multiple modes and the ways these
modes interact to create meaning, and b) the contextualisation of the modes – emphasising
the importance of analysing them within their situational context (considering what, who, and
how). As they are based on the same key principles and often share methodological tools,
datasets, and challenges, a clear-cut presentation of each approach is not straightforward:
this highlights the porous boundaries within the field as our thinking evolves, along with the
potential for combining various approaches. As an illustration, many scholars working in
MCDA, including Brookes and colleagues, as well as Gill and Lennon, situate their analyses
within the social semiotic tradition, particularly drawing on Kress and van Leeuwen’s seminal
Reading Images. A key difference, however, is their starting point: in introducing the
12
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
approaches, I drew attention to their units of analysis, which are the semiotic resource, in SF-
MDA, action, in MDA, conversational order, in CA, and power/social structure in MCDA.
Although challenges specific to each approach were outlined above, there are broader issues
persisting across all multimodal approaches. One of them is terminology, which is used
inconsistently, with authors often coming up with their own definitions: this can be partly
linked to the fact that multimodal approaches are used for the examination of a wide range
of contexts and topics. The various ‘turns’ used to mark the transition from a
conceptualisation of interaction as primarily verbal to one that encompasses a range of
semiotic resources – such as embodied, multimodal, spatial, visual, material, and mobile turn
– is a case in point (cf. Nevile, 2015). Another common denominator when considering
challenges in multimodal discourse approaches is the concept of ‘mode’ itself: although,
analytically, we try to isolate modes, in practice, this is unfeasible. Modes interact in elaborate
ways and the produced meaning is always more than the sum of the isolated modes (see
Machin, 2016, for a discussion). Finally, another criticism pertinent to all multimodal
approaches is the fact that, with semiotic resources being dynamic, fluid, and contextual, it is
difficult to build ‘stable analytical inventories’ of multimodal semiotic resources (Jewitt, 2013):
as such approaches gain ground, however, scholars increasingly develop and refine
methodologies to better capture and categorise these evolving resources, thereby enhancing
the robustness and applicability of multimodal analysis.
Future directions
In sketching out the field, it became evident that multimodal discourse approaches have, in
general, prioritised certain modes: gazes, gestures, and images have been more thoroughly
studied, while issues of materiality remain underrepresented, and are still viewed as
peripheral in the field of discourse studies, which remains a primarily logocentric field (De Fina
& Georgakopoulou, 2020). Recent work on intra-professional communication in health
emergencies, for instance, has started viewing the use of material zones of the emergency
room as a discursive strategy for doing teamwork and leadership (Mesinioti et al., 2023),
which is a prosperous field for further multimodal research. Going even further, I briefly
mentioned earlier how Mondada (2019) made a case for multimodal approaches to consider
other senses, such as smell, touch, and taste: if this will be picked up more widely by discourse
analysts, is yet to be seen.
The integration of emerging digital technologies in healthcare is set to propel our field
forward. The use of multimodal discourse analysis in digital contexts is a rapidly evolving field,
looking at how digital affordances (e.g., hyperlinks, interactive elements, multimedia)
contribute to meaning making. Advancements are already documented: with the rise of AI,
for instance, Putland et al. (2023) started examining the role of AI images in constructing
discourses of dementia. The role of AI in representations of health and illness, and the impact
this has on patients and the public, is a field with significant potential, in which multimodal
critical perspectives have valuable tools to offer. The study of multisemioticity in telehealth
and virtual consultations has also gained momentum, offering significant potential to enhance
our understanding of how these modes influence communication effectiveness and patient
experience. More, multimodal discourse approaches, particularly SF-MDA and MCDA, should
13
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
be more consistently employed for the study of patient portals and health apps, which are
being adopted more broadly by national healthcare providers, as such analyses can contribute
to the identification of effective communication patterns between patients and healthcare
providers and yield implications for patient satisfaction and adherence to treatment plans.
Finally, the COVID-19 pandemic and the in-depth multimodal analyses of healthcare
campaigns across the world that followed led to significant advancements in the field:
investigating how multimodal communication is employed during future health crises (e.g.,
pandemics, natural disasters) and the development of new analytical tools will be beneficial
for managing public response and disseminating critical information.
Overall, the future of multimodal discourse approaches in healthcare is possible to involve,
among other areas, a focus on materiality and the use of surrounding space, the integration
of emerging technologies such as AI, telehealth, and health apps, and the development of new
analytical tools for investigating crisis communication. By advancing these areas, multimodal
discourse analysis can offer deeper insights into health communication and enhance both
professional practices and patient outcomes.
14
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
References
Albers, P. (2013). Visual discourse analysis. In P. Albers, T. Holbrook, & A. Flint (Eds.), New methods of literacy
research (pp. 85–98). Routledge.
Al-Subhi, A. S. (2024). Interactional multimodal metadiscourse in public health posters during the COVID-19
pandemic. Pragmatics & Society. https://doi.org/10.1075/ps.22092.als
Bezemer, J., & Jewitt, C. (2010). Multimodal analysis: Key issues. In L. Litosseliti (Ed.), Research methods in
linguistics (pp. 180–197). Continuum.
Brookes, G., Harvey, K., Chadborn, N., & Dening, T. (2018). “Our biggest killer”: Multimodal discourse
representations of dementia in the British press. Social Semiotics, 28(3), 371–395.
https://doi.org/10.1080/10350330.2017.1345111
Brookes, G., & Hunt, D. (Eds.). (2021). Analysing health communication: Discourse approaches. Palgrave
Macmillan. https://doi.org/10.1007/978-3-030-68184-5
De Fina, A., & Georgakopoulou, A. (Eds.). (2020). Discourse materialities and embodiment: Introduction. In The
Cambridge handbook of discourse studies (pp. 255–370). Cambridge University Press.
Fairclough, N. (1995). Critical discourse analysis: The critical study of language. Longman.
Fatigante, M., Zucchermaglio, C., & Alby, F. (2021). Being in place: A multimodal analysis of the contribution of
the patient's companion to "first time" oncological visits. Frontiers in Psychology, 12, 664747.
https://doi.org/10.3389/fpsyg.2021.664747
Garfinkel, H. (1967). Studies in ethnomethodology. Prentice-Hall.
Gill, K. A., & Lennon, H. (2022). Conformity through fear: A multimodal critical discourse analysis of COVID-19
information adverts. Critical Approaches to Discourse Analysis across Disciplines Journal, 14(1), 22–44.
Giorgi, A. (1985). Phenomenology and psychological research. Duquesne University Press.
Goodwin, C. (1979). The interactive construction of a sentence in natural conversation. In G. Psathas (Ed.),
Everyday language: Studies in ethnomethodology (pp. 97–121). Irvington Publishers.
Goodwin, C., & Goodwin, M. (1992). Context, activity, and participation. In A. Luzio & P. Auer (Eds.), The
contextualization of language (pp. 77–99). John Benjamins. https://doi.org/10.1075/pbns.22
Halliday, M. (1978). Language as social semiotic: The social interpretation of language and meaning. Arnold.
Halliday, M., & Hasan, R. (1985). Language, context, and text: Aspects of language in a social-semiotic
perspective. Deakin University Press.
Heath, C., Luff, P., Sanchez-Svensson, M., & Nicholls, M. (2018). Exchanging implements: The micro-materialities
of multidisciplinary work in the operating theatre. Sociology of Health and Illness, 40, 297–313.
https://doi.org/10.1111/1467-9566.12594
Iedema, R. (2001). Analysing film and television: A social semiotic account of Hospital: An unhealthy business. In
T. van Leeuwen & C. Jewitt (Eds.), The handbook of visual analysis (pp. 183–204). Sage.
https://doi.org/10.4135/9780857020062
Jerome, C., & Ting, S. H. (2022). What's in a message: A systemic functional analysis of cancer prevention
messages. International Journal of Applied Linguistics, 32, 57–75. https://doi.org/10.1111/ijal.12380
Jewitt, C. (2013). Multimodal methods for researching digital technologies. In S. Price, C. Jewitt, & B. Brown (Eds.),
The SAGE handbook of digital technology research (pp. 250–265). SAGE.
https://doi.org/10.4135/9781446282229
Jewitt, C., Bezemer, J., & O’Halloran, K. (Eds.). (2016). Systemic functional linguistics. In Introducing multimodality
(pp. 30–57). Routledge.
Jewitt, C., & Kress, G. (2003). Multimodal literacy. Peter Lang Publishing.
15
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
Jones, R. (2014). Health risks and mediated discourse: A case study of 'AIDS in action'. In H. Hamilton & S. Chou
(Eds.), Routledge handbook of language and health communication (pp. 109–122). Routledge.
Jones, R. H., & Norris, S. (2005). Discourse in action: Introducing mediated discourse analysis. Routledge.
https://doi.org/10.4324/9780203018767
Kress, G. (2009). Multimodality: A social semiotic approach to communication. Routledge.
https://doi.org/10.4324/9780203970034
Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of visual design. Routledge.
https://doi.org/10.4324/9781003099857
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary
communication. Arnold.
Landqvist, M., & Blåsjö, M. (2024). Collaboration, reinvented tools, and specialist knowledge: Communication
professionals’ experiences of global health crisis management. Discourse & Communication, 18(4), 514–534.
https://doi.org/10.1177/17504813241227590
Lane, P. (2014). Nexus analysis. In J. Östman & J. Verschueren (Eds.), Handbook of pragmatics online (pp. 1–18).
John Benjamins. https://doi.org/10.1075/hop.18.nex1
Ledin, P., & Machin, D. (2018). Doing critical discourse studies with multimodality: From metafunctions to
materiality. Critical Discourse Studies, 16(5), 497–513. https://doi.org/10.1080/17405904.2018.1468789
Liang, M.-Y. (2019). Beyond elocution: Multimodal narrative discourse analysis of L2 storytelling. ReCALL, 31(1),
56–74. https://doi.org/10.1017/S0958344018000095
Machin, D. (2013). What is multimodal critical discourse studies? Critical Discourse Studies, 10(4), 347–355.
https://doi.org/10.1080/17405904.2013.813770
Machin, D. (2016). The need for a social and affordance-driven multimodal critical discourse studies. Discourse
& Society, 27(3), 322–334. https://doi.org/10.1177/0957926516630903
Machin, D., & Mayr, A. (2012). How to do critical discourse analysis: A multimodal introduction. Sage.
Mesinioti, P. (2021). ‘GET the oxygen get an airway’: Doing leadership in medical emergencies (Unpublished PhD
thesis). University of Warwick.
Mesinioti, P., Angouri, J., & Turner, C. (2023). ‘Do you want me to take over?’: Displaying epistemic primacy in
medical emergencies. Journal of Applied Linguistics and Professional Practice, 17(2), 161–187.
https://doi.org/10.1558/jalpp.21858
Mondada, L. (2014). Instructions in the operating room: How the surgeon directs their assistant’s hands.
Discourse Studies, 16(2), 131–161. https://doi.org/10.1177/1461445613515325
Mondada, L. (2016). Challenges of multimodality: Language and the body in social interaction. Journal of
Sociolinguistics, 20(3), 336–366. https://doi.org/10.1111/josl.1_12177
Mondada, L. (2018). Multiple temporalities of language and body in interaction: Challenges for transcribing
multimodality. Research on Language and Social Interaction, 51(1), 85–106.
https://doi.org/10.1080/08351813.2018.1413878
Mondada, L. (2019). Contemporary issues in conversation analysis: Embodiment and materiality, multimodality,
and multisensoriality in social interaction. Journal of Pragmatics, 145, 47–62.
https://doi.org/10.1016/j.pragma.2019.01.016
Murdoch, J., Salter, C., Poland, F., & Cross, J. (2015). Challenging social cognition models of adherence: Cycles of
discourse, historical bodies, and interactional order. Qualitative Health Research, 25(2), 283–294.
https://doi.org/10.1177/1049732314552074
Nevile, M. (2015). The embodied turn in research on language and social interaction. Research on Language and
Social Interaction, 48(2), 121–151. https://doi.org/10.1080/08351813.2015.1025499
Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. Routledge.
https://doi.org/10.4324/9780203379493
16
QUALITATIVE HEALTH COMMUNICATION · VOLUME 4, ISSUE 1, 2025
MULTIMODAL DISCOURSE ANALYSIS IN HEALTH COMMUNICATION
O’Halloran, K. L. (2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational
meaning using language and visual imagery. Visual Communication, 7(4), 443–475.
https://doi.org/10.1177/1470357208096210
O’Halloran, K. L., Tan, S., & Wignell, P. (2019). SFL and multimodal discourse analysis. In G. Thompson, W. L.
Bowcher, L. Fontaine, & D. Schönthal (Eds.), The Cambridge handbook of systemic functional linguistics (pp. 433–
461). Cambridge University Press. https://doi.org/10.1017/9781316337936.019
Ope-Davies, T., & Shodipe, M. (2023). A multimodal discourse study of selected COVID-19 online public health
campaign texts in Nigeria. Discourse & Society, 34(1), 96–119. https://doi.org/10.1177/09579265221145098
O’Toole, M. (1994). The language of displayed art. Routledge.
Putland, E., Chikodzore-Paterson, C., & Brookes, G. (2023). Artificial intelligence and visual discourse: A
multimodal critical discourse analysis of AI-generated images of “Dementia.” Social Semiotics, 1–26.
https://doi.org/10.1080/10350330.2023.2290555
Sacks, H. (1992). Lectures on conversation (Vols. 1 & 2). Blackwell.
Schegloff, E. (1984). On some gestures' relation to talk. In J. Atkinson & J. Heritage (Eds.), Structures of social
action: Studies in conversation analysis (pp. 266–295). Cambridge University Press.
Schegloff, E. A., & Sacks, H. (1973). Opening up closings. Semiotica, 8, 289–327.
https://doi.org/10.1515/semi.1973.8.4.289
Scollon, R. (2001). Mediated discourse: The nexus of practice. Routledge.
https://doi.org/10.4324/9780203420065
Scollon, R., & Scollon, S. (2004). Nexus analysis: Discourse and the emerging internet. Routledge.
https://doi.org/10.4324/9780203694343
ten Have, P. (2012). Ethnomethodology and conversation analysis. In H. Cooper, P. M. Camic, D. L. Long, et al.
(Eds.), APA handbook of research methods in psychology, Vol. 2: Research designs: Quantitative, qualitative,
neuropsychological, and biological (pp. 103–117). American Psychological Association.
https://doi.org/10.1037/13620-007
Traue, B., Blanc, M., & Cambre, C. (2019). Visibilities and visual discourses: Rethinking the social with the image.
Qualitative Inquiry, 25(4), 327–337. https://doi.org/10.1177/1077800418792946
Vygotsky, L. S. (1981). The instrumental method in psychology. In J. V. Wertsch (Ed.), The concept of activity in
Soviet psychology (pp. 134–143). M. E. Sharpe.
Wodak, R. E., & Meyer, M. (Eds.). (2015). Methods of critical discourse studies (3rd ed.). Sage.
Yang, Y. (2017). Living healthy versus looking attractive: The multimodal portrayal of skin cancer in women's
magazines in the United States. Journal of Magazine Media, 18(1). https://dx.doi.org/10.1353/jmm.2017.0005
QUALITATIVE HEALTH COMMUNICATION
VOLUME 4, ISSUE 1, 2025
