Experiences of Multimodal Teaching Through a Serious Game: Meanings, Practices and Discourses

Abstract

The aim of this chapter is to report on school teachers’ perceptions and approaches to multimodality using a serious game. STEAM is a game designed for helping school teachers to gain awareness of how multimodality may be enacted in the classroom for enhancing the student learning experience. The game embraces the notion of multimodal teaching and learning, as a way to present multiple representations of content such as text, images, video, audio and pervasive media, by augmenting modes with tools, teaching strategies and locations as means to create ideational, interpersonal and textual meanings. A questionnaire was employed to school teachers (n = 54) for understanding how multimodality was experienced through using the serious game as (1) stipulating diversity and increasing knowledge retention, (2) developing senses for attaining deeper understanding of the subject topic, (3) involving students into learning design and (4) supporting student’s autonomy and self-direction. The findings revealed an explicit connection between theory and practice as experienced through the game’s semiotic domain whilst contemplating on attempts to transcend experiences of in-game multimodality to lived classrooms.

Full text

Experiences of Multimodal Teaching Through
a Serious Game: Meanings, Practices and
Discourses
Petros Lameras and Vassiliki Papageorgiou
Accepted manuscript PDF deposited in Coventry University’s Repository
Original citation:
‘Experiences of Multimodal Teaching Through a Serious Game: Meanings, Practices
and Discourses’, in Technology Supported Innovations in School Education, ed. by
Pedro Isaias Isaias, Demetrios Sampson and Dirk Ifenthaler, pub 2020 (ISBN 978-3-
030-48193-3)
Publisher: Springer
Copyright © and Moral Rights are retained by the author(s) and/ or other
copyright owners. A copy can be downloaded for personal non-commercial
research or study, without prior permission or charge. This item cannot be
reproduced or quoted extensively from without first obtaining permission in
writing from the copyright holder(s). The content must not be changed in any
way or sold commercially in any format or medium without the formal
permission of the copyright holders.

Chapter # - will be assigend by editors
EXPERIENCES OF MULTIMODAL TEACHING
THROUGH A SERIOUS GAME: MEANINGS,
PRACTICES AND DISCOURCES
AUTHOR(s) – Petros Lameras, Vasiliki Papageorgiou
Abstract: The aim of this chapter is to report on school teachers’ perceptions of and
approaches to multimodality using a serious game. STEAM is a game designed
for helping schoolteachers to gain awareness of how multimodality may be
enacted in the classroom for enhancing the student learning experience. The
game embraces the notion of multimodal teaching and learning, as a way to
present multiple representations of content such as text, images, video, audio
and pervasive media, by augmenting modes with tools, teaching strategies and
locations as means to create ideational, interpersonal and textual meanings. A
semi-structured questionnaire was employed to school teachers (n=54) for
understanding how multimodality was experienced through using the game as:
(1) stipulating diversity and increasing knowledge retention, (2) developing
senses for attaining deeper understanding of the subject topic, (3) involving
students into learning design and (4) supporting student’s autonomy and self-
direction. The findings revealed an explicit connection between theory and
practice as experienced through the game’s semiotic domain whilst
contemplating on attempts to transcend experiences of in-game multimodality
to lived classrooms.
Key words: Multimodal teaching and learning, school teachers, serious games
1.
INTRODUCTION
This chapter elucidates on secondary school teachers’ experiences of
multimodal teaching through a serious game. We have developed a serious
game called STEAM as a digital medium to help teachers to understand the
concept of multimodality for teaching and learning. Serious games for
learning and teaching have been perceived as a medium for instigating playful

2
Chapter # - will be assigend by editors
learning aligned with rich-mediated content to achieve in-game learning goals.
The overarching aim of such games is to infuse learning content
amalgamated with teaching models, frameworks assessment and feedback in-
game representations for discerning a more constructive, reflective and
memorable learning experience (Bellotti et al., 2012; Boot et al, 2008; Del
Blanco et al., 2012).
There is an increasing body of evidence, which suggests that
multimodality is an activity-based and student-centred approach to teaching
in which a series of different tools, technologies, resources and environments
are deployed for helping students during the meaning-making process (Jewitt,
2008; Sun, 2015). The essence of multimodality therefore is to instantiate
teaching using an array of different resources and pedagogies for stimulating
learning in meaningful ways (Antonietti & Giorgetti, 2006) within and across
disciplines. This is directly relevant to the way teachers attempt to teach
and deliver subject-content knowledge in terms of understanding how the use
of a plethora of different learning resources, technology and tools would
likely influence students’ learning. Multiple modes of representation are
central for adopting a multimodal approach to teaching that combines print
with visual images and interactive resources. Multimodality is not a new
concept as it has been developed in early 2000s (Kress & Van Leeuwen,
2001) while connected concepts may embrace literacy-related terms such as
multi-literacy or multimodal literacy, which transcends the basic idea of
reading and writing to multiple-form of mixed-print representations (Miller &
McVee, 2013).
There is a widespread view from different commentators (e.g. Cope &
Kalantzis, 2009; Cowan & Cipriani, 2009) that school teachers seem to be
overwhelmed by the plethora of teaching representations such as digital tools,
resources and pedagogies that may be used for enacting teaching in more
activity-oriented ways. This, in principle, would allow to design activities that
encourage students to be actively involved in situated learning instances in or
out of the classroom. Multimodality is indeed an ill-defined concept
encompassing social and cultural shaped resources with an emphasis on the
inherently social negotiated character of meaning (Lave, 1991). To support
teachers’ efforts to understand a complex and ill-defined phenomenon such as
multimodality, we decided to design STEAM as a multimodal digital artefact
for discerning richer, context-orientated and visual-based representations of
multimodal teaching as experienced by teachers while interacting the game.
The following sections start by presenting the aims and research questions
underpinning this study and then we discuss the tenets of multimodality as a
teaching strategy and its variations in meaning making processes driven by
how meaning is conveyed from a multi-modal perspective. Then we present
the design of the STEAM game encompassing analysis of high-level goal,

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
3
scenarios and learning objectives. We then present our methodology and
findings. We conclude by providing a discussion on game design
considerations and implications for multimodal learning and teaching,
limitations and future research.
The aim of this study was to contemplate on how school teachers
understand multimodality through playing a serious game. The following
research questions were identified from the literature:
1. What are teachers’ experiences of multimodal teaching and learning
through interacting with a serious game?
2. What are the implications of teachers’ in-game experiences of
multimodality for practicing multimodal teaching in the classroom?
Despite increasing interest into how multimodality is conceived and
practiced for amplifying teaching and learning, there is no empirical evidence
that explores multimodal teaching through playing a multimodal serious game.
Current impetus is on how multimodal teaching is perceived and
developed through traditional teaching professional development programs or
more recently through Massive Open Online Courses (MOOCs). These
training programs typically focus on teacher development from a more
universal viewpoint encompassing broad aspects of teaching and not
specifically from a multimodal standpoint focusing on amalgamating and
aligning pedagogies, teaching strategies, technology and different learning
spaces. We hence take a different stance in developing teachers’
understandings of multimodality by perpetuating the development of a game
acting as the environment in which teachers’ views and beliefs of
multimodality are created, constructed and developed.
2.
VARIATION IN MULTIMODAL MEANING
MAKING PROCESSES
We argue that multimodality draws on the process of creating meaning through
connecting and combining teaching modes (e.g. in-class and out of class
teaching), semiotic resources (e.g. multimedia types such as video, audio,
images, animation, interactive content), teaching approaches (e.g.

4
Chapter # - will be assigend by editors
inquiry and problem-based learning) and immersive technologies (e.g. games,
simulations, online learning platforms). Meaning inferences may be perceived
as social actions triggered by experiences, beliefs and practices situated in
specific contexts. Jewitt (2013) identifies three meaning variations: (1)
ideational meaning, dealing with choices related to how people interpret
content meanings, (2) interpersonal meaning, employing the resources being
used to represent the social relations between communicators and (3) textual
and organisational meaning representing decisions on choice or resources as
means to understand a text’s structure or the nature of an interaction between
two people.
The proliferation of digital technologies (personal, mobile, networked)
may supplement or enhance conventional non-digital activities. From
administrative-level tasks such as organising and storing content (e.g.
Schoorman, Mayer, & Davis, 2007) or via transferring information to students
for content accommodation and assimilation (e.g. Mayer, 2001) to more
knowledge construction processes such as conducting research or for skills
development (Greenhow, Robelia, & Hughes, 2009). Multimedia tools in
these environments may include for example, interactive videos, recorded
lecture presentations, online quizzes, discussion forums, digital storytelling,
visual representations of student data to depict progress. This increasing use
of multimedia in teaching and learning may lead to presenting multiple
representations of content (e.g. text, images, video, audio, pervasive media) to
accommodate different teaching strategies, learning outcomes, assessment
methods and feedback mechanisms. It has also been argued that by
incorporating multimedia learning, students may be encouraged to develop a
more flexible and research-based approach to learning as activities are
designed towards encouraging discovery through information literacy skills,
critical thinking and inquiry-led investigations. Studies into multimodal
composition have revealed enhanced learning experiences and outcomes (e.g.
Crook & Crook, 2017; Miller & McVee, 2013) as well as enabling students to
identify and express their personal identity (e.g. Thibaut & Curwood, 2018).
Shah and Freedman (2003) identified a series of benefits of using
visualisations in learning such as (1) providing external representations of
information, (2) deeper learning, (3) triggering student’s attention and
concentration by making information more comprehensive, hence simplifying
ill-defined concepts and ideas.

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
5
2.1
Multimodality as a meaning for teacher-directed
teaching
In more traditional classroom settings, oral communication modes were
predominant and directly linked with a teacher-centred approach to teaching.
Reinforcement, memorization and repetition of desired actions, underpinned
teachers’ actions as ‘sage on the stage’. The main written communication
mode is the text-book covering the standard-based curriculum predefined by
the institution or country. The students are producing written texts (e.g. written
assignments or tests) and they are evaluated predominantly via a quantitative
score. Less emphasis is given to student’s own interests, needs and prior
knowledge because it is important for the teacher to cover the curriculum and
transmit the information prescribed. Student’s development of social skills
and the utilisation of peer-interaction for participatory and collaborative
learning is not encouraged hence transmissive teaching with uni-modal, non-
interactive tools for content transfer and basic skills acquisition are the focus
of attention.
We are not asserting that teacher-directed approaches to multimodality are
not appropriate for multimedia learning as it is has been shown that uni-modal
and non-interactive practices may be used in primary school learning,
particularly for helping students build a knowledge- rich subject domain or for
students with special needs or disadvantaged backgrounds, an information-
transmission mode may inform basic skill development (e.g. Lerkkanen et al.,
2016).
The STEAM game design employs different multimodalities and semiotic
resources to exemplify the inclusiveness and totality of different paradigms,
models and strategies needed for achieving the necessary outcomes as defined
by the teacher and external influences such as context, institutional or policy
educational developments.
2.2
Multimodality as a meaning for active and student-
centred learning
Multimodality is inherently connected to activity-led learning, in which
students are autonomous, self-directed and they construct knowledge through
processing prior learning experiences and knowledge as well as through
empathising, ideating, discovering and producing. Students may select
relevant words and images and organising them into coherent verbal and
visual models. In the teaching and learning spectrum, a different range of

6
Chapter # - will be assigend by editors
semiotic tools and resources are being used to facilitate students to apply,
analyse, evaluate and create new knowledge. The central focus is extracting
meaning though using visual representation techniques for understanding,
linking and negotiating ideas and concepts with the teacher and peers.
Arguably, we may observe a shift into students’ role from knowledge
consumers to knowledge producers (e.g. Cope, Kalantzis, & Abrams, 2017).
Technology and tools may involve among others serious games, virtual reality
and virtual learning environments. Teaching and learning can go beyond the
boundaries of the classroom and may be combined with visits to (open air)
museums, science centres, and library spaces as a way of extending the
learning process in other contexts where information is directly linked to
specific artefacts or tangible objects. Such modes and semiotic resources help
students to create, apply and transfer meaning within their own social context
and may signify the importance of stretching teaching and learning practice
beyond the potentialities and constraints of the normal classroom. In addition,
such technologies may enable to extend the standard forms of written and
spoken language to connect with the culturally and linguistically diverse
landscapes and the multimodal texts mobilized across these landscapes
(Jewitt, 2008).
The main assumption is that multimodality is the vehicle for students to
design, implement, share and re-use/repurpose semiotic resources and
multimodal ensembles (e.g. Greenhow et al., 2009). For example, students are
constructing learning through content and resources found online and offline,
in various places such as in the school, in field trips, in the library, in museums
and in science centres. This blend of learning spaces between formal and
informal enables the participation of actors with different experiences and
learning settings (e.g. museum curators, scientists, researchers and business
experts) having the role of the facilitator and thereby helping the students to
experience a multitude of opinions and ideas hence assimilating and
accommodating informed experiences meaningful to them. This more active
approach to learning where the students are constantly looking for new ways
of learning from diverse perspectives, spaces and educators from different
disciplines and sectors, may infer to the notion of multiple intelligences (e.g.
Picciano, 2009). Such diverse multimodal learning activities do not exist as a
linear equation of ‘yes’ or ‘no’ but within a continuum of naïve to more
coherent ways of learning influenced by the learning environment (formal and
informal), prior knowledge, design of learning and the subject- content
domain. It is perceived therefore that multimodality with active, student-
centred learning is aligned in terms of designing learning in multi-sensory
modes and delivered via an array of distinctive semiotic resources.

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
7
3.
DESIGN OF THE STEAM MULTIMODALITY
GAME
The aim of the STEAM serious game is to stipulate teachers’ awareness on
experiencing multimodality for teaching and learning as a process inherently
connected with rich-mediated pedagogies, a variety of tools and multiple
learning environments. The main impetus is to help teachers to understand
both the conceptual and practical manifestations of designing and
implementing learning and teaching activities with multiple media, instigating
different teaching strategies and exploit distinct modes of teaching.
STEAM is a simple point-and-click game that is played through a web
browser. The game play represents non-linear dialogues with a Non-Player
Character (NPC) visualising a set of choices for the player to choose from.
Complementing the dialogue game, there is a mini card game for players to
select card combinations for establishing a multimodal teaching environment
of their choice.
3.1
Goals and outcomes
The game’s narrative sets the player to have the role of a newly appointed
teacher interested to learn more about multimodality and how (e.g. resources,
pedagogies, modes, technology) it can be practiced. The main in-game goal
therefore is to create an awareness of what is multimodality and how it can be
enacted to enhance students’ leaning experience. The player commences the
game as having the fictional character of ‘Mary’ a newly appointed
mathematics teacher at Charles Darwin School that attempts to transcend
theory and practice as means to enhance teaching by using multimodal tools,
pedagogies and strategies. Interactions with the students and ways of
delivering content is influenced by the dialogue responses given by the player,
which in turn enable the collection of game cards for designing lesson plans.
3.2
In-game scenarios
To situate multimodality into a learning context, we have designed 3 chapters
encompassing four learning scenarios in which multimodal teaching
challenges are presented to the player. The flow of each scenario determines
the in -game sequence aligning the dialogue and cards game mechanics and
the actions of the player. Each chapter may be played as a way of introducing
the player to three multimodal in-game objectives following the subject
content and flow of each scenario. When the player loops out of the

8
Chapter # - will be assigend by editors
introduction sequence, then the scenario starts by introducing the narrative
sequence as a means for the in-game dialogue mechanic to engage the player
with the teacher NPC. This is for getting information about what multimodality
is and to highlight that it reflects a range of pedagogies, digital tools and
resources amalgamated together to form an interacting and multimodal
learning experience within and beyond the classroom context. Then, the
next core game mechanic, the game cards are combined together to form a
particular multimodal situation that the teacher would favor. A new dialogue
sequence follows, with students contemplating on and articulating about the
choices the player has made during the dialogues and possible applications
in the real-world teaching domain.
3.3
Core game mechanics
We have designed our core game mechanics to prompt for progressively
learning the objectives of each chapter whilst to rapidly comprehend the
multimodal aspects that the dialogues conveys to the player. The dialogue
mechanic drives the multimodality learning process and twins the pedagogical
objectives of each scenario with player’s chosen response. The dialogue
mechanic is part of the narrative sequence in which the player responds to
questions asked either by colleagues or by students. The player has three
options to choose from: one of the dialogue options is the correct, the other is
intermediate and the third is not correct. (see figure 1).
Figure 1: Left chart: The dialogue challenge is set for the player to respond.
Right chart: The player has 3 in-game options to choose from.
When the player selects a dialogue option it is highlighted with a green
frame as a visual representation to denote the choice is correct, with a yellow
frame to show that the choice is intermediate and with a red frame to signify
that the choice is not correct. The general consensus is to guide players

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
9
understandings on how multimodality may be viewed in multiple perspectives
that would likely increase student’s in-game learning and engagement.
As the player responds to questions, up to ten cards are available separated
into different categories such as strategies, activities and locations. The cards
need to be combined in order to form a particular multimodal situation.
When the initial engagement is low, the player should select a highly
engaging combination. The player selects the combination depending on the
objective of the chapter, determined by the initial engagement and learning
level of the students. As an example, when the initial engagement is low, the
player should select a highly engaging combination (see Figure 2).
Figure 2: Left-chart: Designing a multimodal situation comprised of strategies,
activities and location. Right-chart: Designing a multimodal situation that influences
students’ engagement levels.
Hints about the outcome of each card combination are provided to support
the player’s choice. Feedback is provided about the effects of designing an
activity with students from multiple backgrounds with the purpose of gaining
understanding of a multimodal activity involving oral, written, visual and
gestural modes of learning and communication. Each card includes
information about the type of strategy and modalities that are deployed, and
the locations that the activity can be implemented. Likewise, for the activity
cards, the properties and attributes are highlighted whereas the location cards
visualize the place in which a multimodal situation may be delivered. Based
on the card selection the player decides the engagement level bar increases or
decreases thereby affecting the average level’s grade.
3.4
In-game feedback
In-game feedback is augmented in multiple ways for helping players to
understand not only ways of conceiving and practicing multimodal teaching

10
Chapter # - will be assigend by editors
but also progress achieved in game-play. For example, including the
‘engagement bar’ and ‘the learning bar’ indicating engagement of students
and their learning level. This visual representation type of feedback is
displayed during the dialogue sequence with the students for the player to have
instant information on his/her progress for quickly adjusting performance. We
have integrated this type of visual feedback for the players to be able to
comprehend easily the meaning of the feedback received thus to get extrinsic
feedback on their performance.
The design of the multimodal card deck library had a dual feedback
purpose: Firstly, for players to reflect on the combinations already made, and
assimilate the multimodal features chosen. Secondly, to experience and
understand how new multimodal card combinations that have not been
selected during game-play, would help to understand different ways and
variations of using multimodality. Players may choose any of the card decks
from strategies, tools and locations categories and then suggested
combinations are provided by the game. For example, if the players select the
argumentation debate card from the strategy category, then a suggestion pops
up combining a blog tool and a home location (see Figure 3).
Figure 3: Top-left chart: Setting a multimodal strategy by using the
‘argumentation debate’. Top-right chart: Selecting the ‘audio’ tool. Bottom-left
chart: Selecting a ‘classroom’ location card. Bottom-right chart: Selecting a
museum location card.

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
11
4.
RESEARCH DESIGN
Fifty-four secondary education teachers were recruited for this study. We have
conducted an initial pilot study with 22 secondary school teachers for testing
our data collection methods, research instruments and processes of analysis
and themes. When we streamlined our methods and devised a set of
preliminary themes, we have carried out the main study with an additional
cohort of thirty-two teachers comprising a total sample of fifty-four (n=54)
teachers as means to achieve variation and commonality among experiences
from broad subject topics such as science and technology (e.g. mathematics,
physics, chemistry, computing), history, literature and languages. Twenty-six
were males and twenty-seven were females. All participants were working in
schools with varied teaching experience (1-30 years). Twenty participants
were playing games while none have interacted with games before as means
to develop their teaching practice. However, they all felt familiar with using
technology in designing and delivering learning activities and resources.
Twenty-two participants had a vague and fragmented view of multimodal
teaching whereas thirty-two had never came across the term multimodality
before commencing this study. Participants were drawn from a wide
geographical range from countries such as Germany (n=12), Denmark (n=18)
and Finland (n=24). This ensured that a wide range of contexts, practices,
beliefs and actions were part of our purposive sampling method for achieving
the required variation in the formulation of the experiences.
The data collection process started by involving participants to playtest the
game for forty minutes. Then, they had to complete a semi-structured online
questionnaire for approximately another forty minutes. Starting with the
playtesting process, we have introduced the game to participants, and we have
elaborated the objectives of the study. We have made explicit all ethical
procedures such as rights to withdraw at any time, data management, handling
and storage, their voluntary participation and the right to withdraw at any time.
Then, participants started to respond to the semi-structured questions
connecting their experiences of in-game multimodality with their own
experiences of teaching as a way of constituting a cohesive and structured
experience. By asking teachers to provide their own meanings of in-game
multimodal teaching would enable us to distil, capture and analyse what felt
important to them. Questions related to how the visual representations and
interactive nature of the game helped participants to gain an understanding of
the elements and properties of multimodal teaching as depicted in game-play.
The analysis commenced with the premise that a data-driven process will
be followed to constitute the themes following an iterative process of

12
Chapter # - will be assigend by editors
familiarising with the data, coding and associating relational inferences and
reflecting on the emerging themes. Dedoose, a digital analysis software for
organising, associating and reflecting on codings and for constituting the
themes was used in this study. As such, analysis employed an inductive
approach that delineated the themes from the data. To achieve consistency,
validity and reliability, the analysis was carried out in two stages. During the
first stage, the researchers individually identified and coded the data as means
to constitute preliminary themes. In the second stage, the researchers worked
together to compare, refine and reflect on the codings (consisted of themes
description along with interpretations of ‘strategies’, ‘tools’ and ‘learning
spaces’ for each theme) and associated quotations to constitute the final
datasets. Descriptive (e.g. illuminate themes) and interpretive (e.g.
comprehend subsequent meanings) analysis was therefore employed to ensure
optimal analysis procedures.
5.
RESULTS
We present the results of the analysis encompassing teachers’ experiences of
multimodal teaching through interacting with the STEAM serious game. The
themes may be used for rendering how multimodal tools, strategies and
locations were experienced by teachers to constitute their conceptions of in-
game multimodal teaching and learning and thereby investigating how the
game influenced the development of such in-game experiences of
multimodality. Table 1 presents the four themes emerged with associated
strategies, tools and locations and representative quotations. In presenting the
data below participant identifiers (P1-P54) were employed to indicate
responses from the participating teachers.
Table 1: Themes of in-game experiences of multimodal teaching with associated strategies,
tools and spaces along with representative quotations.
Themes Strategies, tools,
learning spaces
Representative quotations
Learning diversity
for engagement and
knowledge retention
Discussions knowledge
retention, lectures,
slideshows, videos, in-
class learning
“engaging students in different ways
of learning through discussing,
debating, and convincing” (P50).
“I was looking for choices that
increase retention of what was learnt”
(P54). [..] it is all about engaging
students through various oral and
written tools including presentations,

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
13
Developing senses
for attaining deeper
understandings of
the topic
Involving students
into learning design
Increasing attention and
focus, visuals, 3D
modelling, in-class
learning combined with
visits from professionals
Learning by design,
creativity, video creation
and editing, 3D printing,
question tool, inquiry,
making products, in-class
learning combined with
fieldtrips, learning in
public spaces
videos followed by assignments”
(P40). “students were concentrating
better in the school” (P12)
“multimodality sharpens the senses
and enables the student to think more
carefully when trying to explain
understandings” (P25). “tended to
combine a 3D application so students
can use their gestural ability to make
connections, links, shapes and overall
to use their hands with their minds
for increasing their understandings”
(P5). I wanted to combine tools
videos for visual learning followed
by a talk from a professional to gain
and share insights and observe how
they express and use specialist
vocabulary to describe a complex
topic” (P31).
“Combining cards that tried to
involve students into designing
learning activities based on their own
learning needs and interests could
help them to do and create rather than
just listen the topic” (P1). “Using
cards with a design-based approach
to teaching felt it was working well in
co-designing what they wanted to
learn” (P42). “I was looking in the
cards to find combinations such as a
3D printer or create a game where
students can make something as
being the designers of their own
learning and attempt to apply
practically what they know (P52). “I
combined the library and information
literacy cards for students to search
for resources they wanted to use by
visiting the library and then discuss

14
Chapter # - will be assigend by editors
Supporting student
autonomy and self-
direction
Establishing teams,
developing ideas,
gathering requirements,
designing, peer-
assessment, 3D-printing,
question-pool, blogging,
online surveys, out-of-
class teaching
how they did their searching in the
classroom” (P2).
“I selected a dialogue choice for
students to set-up their teams […], it
is just that they [students] need to feel
free to choose the peers that they will
like to work with… (P11). “I tried to
be bold enough to select a design-
based learning card, to get them
[students] sketch the main project
idea and figure out the relations with
other sub-ideas” (P18). “Wanted to
see if I could make students to start
doing some very basic research like
learn how to pose questions or collect
data from a survey so I combined
inquiry-based learning with opinion
polling” (P37). “Selecting a problem-
based learning card with a 3D
printing creation for using their skills
to see how they work in practice”
(P32). “I thought of multimodality as
a more of a self-directing, self-
assessment all-self -thing really, so I
combined interdisciplinarity and blog
cards for preparing collaboratively a
blog report for assessing work and
identify improvements based on their
needs, experiences and backgrounds”
(P41). “A game means going wild,
and this is what I did as an advocate
of context-specific learning though
research, problem-solving and
inquiry, I wanted to teach out of the
classroom, like set a problem-based
activity into groups, get them on
mobile games to start discover
museum -related content for their
projects and unleash their real
potential” (P41).

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
15
5.1
In-game multimodality as a way of stipulating
learning diversity for increasing engagement and
knowledge retention
In this theme teachers perceived in-game multimodality as helping students to
learn through employing multiple teaching strategies and using an array of
learning tools: “I will try to achieve a more intensive examination of the topic,
to increase motivation through the use of modern media (P29). Teaching
strategies such as initiating discussions and debates were prevalent as means
of helping students to externalise their interests and for practicing negotiation,
argumentation and oral skills. Retention of knowledge was a fundamental
purpose achieved via repetition and memorisation of learning content
provided by the teacher.
Tools used were predominantly in-class lectures, slideshows and videos
prepared are presented by the teacher, and writing assignments for recalling
subject-content. Combining in-game strategies such as choosing dialogues for
initiating reciprocal discussions with the teacher for encouraging debates and
argumentation was felt central for assimilating and understanding content.
This was also a way for students to engage with others via communicating
content already retained in memory. “I tend to think reciprocal discussions
about the topic I presented may improve student knowledge retention and
engagement” (P48). Teachers felt that students engagement may be further
facilitated by having a free choice in terms of the tools, resources and personal
learning strategies they deploy that would likely increase self-regulated
learning: “In the dialogues I tended to select a variety of tools and strategies
for enabling interaction and free choice for my students to choose what tools
and resources they may choose from to support their learning (P44).
The classroom is the predominant learning environment, in this theme,
perceived as the main space for delivering learning but also as a space for
introducing and contemplating on extended activities to be completed at home
such as quizzes, subject readings and solving exercises from textbook. This
was evident by the card game combinations selections where teachers selected
strategies, tools enacted in the class: “...in the card game I chose presenting
content in the classroom and then assigning an exercise at home to be presented
and discussed in class the next day...” (P8).

16
Chapter # - will be assigend by editors
5.2
In-game multimodality as developing senses for
attaining deeper understanding of the subject topic
In this theme in-game multimodality for teaching is viewed as developing and
applying students’ senses for gaining an in-depth understanding of the topic:
“[..] developing student’s senses such as seeing, hearing, feeling, touching and
experiencing in detail some of the topics presented...” (P49). Developing
senses therefore, was perceived as means to increase focus and concentration
for explaining ideas and understandings. Different types of senses were
interestingly correlated with types of learners, meaning that different learners
use different senses in order to enable learning: “my aim was to choose cards
with tools that would allow the use of different senses like for example oral,
written, visual or gestural to get to know how a learner learns best [..] I tested
different combinations every time” (P30).
Tools that have been used were mainly the ones that promoted different
senses for visual learning: “I chose slides, videos and diagrams for increasing
their visual skills” (P21), for auditory-oriented learning: “I wanted to combine
an audio track with blogging to augment audio with writing skills” (P43).
The classroom is the main space for teaching as in previous theme but in
conjunction to several informal learning events like invitations to people from
the industry to come and talk about a subject of interest in-class.
5.3
In-game multimodality as actively involving students
into learning design
In this theme in-game multimodality was perceived as collaborating with
students for designing multimodal learning activities: “I saw that some game
cards were describing students as the ones who were making choices on how
learning could be designed – I tried to play with those cards mostly” (P49).
The reasons participants explained approaching game-play towards allowing
students to participate into the design of learning as a pertinent multimodal
activity was for helping students to focus more on learning by doing. It was
felt that by giving a “designer” role to students, it may help them experience
different ways of learning by including aspects of learning mostly relevant to
them but also developing skills in the design of learning activities: “game
dialogue choices that encouraged students to have a say in the design of
activities - and hopefully teaching them the skill of designing learning and its
complexities” (P16). Design-based strategies were used in order to co-develop

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
17
learning activities with students. As such, design-based strategies were
correlated with more applied-based activities rather than focusing only on
retention and memorization as in the first theme: “we try to co-design the
creation of a learning video […] students had to practically engage with the
topic and develop something by themselves” (P33).
The prevailing digital tools that were exploited for in-game multimodal
teaching, were creative tools such as video creation and editing, 3D modelling
and 3D printing. This was perceived as a way to encourage students to develop
creative and design-like mindsets in terms of increasing possibilities to apply
theory into practice. ‘Hands-on’ and self-directed learning was the key and
the use of resources and tools that encouraged this design-like, more applied-
oriented learning was in the foreground of teachers’ ways of thinking about
multimodality during game-play.
Classroom was the primary space of learning. However, notably, in this
theme, participants increasingly started to select cards that resembled
activities in different learning spaces augmenting the classroom space with
different spaces. These included but were not limited to field-trips, visits to
other companies, learning in libraries and other public spaces perpetuating
learning as a process that is constantly enacted regardless of place and time:
“I looked in the library and I saw an interesting combination so I used this as
getting design-based learning, with a location-based play realised on the street
[…] What an excitement to realise that multimodality is applied regardless of
place or time” (P48).
5.4
In-game multimodality as supporting student’s
autonomy and self-direction
In this theme, teachers perceived in-game multimodality as a way to increase
student’s autonomy and self-direction: “the increasingly necessary
development towards independent, self-determined learning” (P32). This was
practiced through helping students to understand the iterative development
cycle of project development: (1) establishing teams for group-work, (2)
developing ideas, (3) gathering requirements, (4) designing and developing,
and (6) encouraging /activating peer-assessment. The first stage of the cycle
felt it was central for students to be guided through the importance of working
with peers. Developing an idea collaboratively was a process of debating,
visualising, comprehending and relating concepts together. Gathering user
requirements seemed to be as an in-game multimodal strategy increasingly

18
Chapter # - will be assigend by editors
correlated with research-related practices: “I chose this option about getting
students to start gathering data for their projects […] (P22). Designing and
developing was perceived as putting students’ creative skills into practice.
Peer-assessment was seen as an interdisciplinary process involving students
with different skills to assess project’s outcomes stemming from multiple
perspectives.
Tools that have been selected were mainly for helping students to design,
create and self-assess their projects: “3D printing was an interesting card for
encouraging students to sit down and discuss what went wrong and what went
well and then assess themselves” (P21). Learning how to formulate questions
as part of initiating a research inquiry was also a feature of in-game
multimodality. Participants increasingly used in-game cards representing tools
that helped students collaborate in developing their project ideas, creative
mind-sets and self-assess final project outputs: “I stretched my head to find a
card-combination that would quickly scribble down their initial ideas,
connected the dotted lines and evaluate final project outcomes as a first go”
(P15). Blogging was perceived as a tool for practicing reflection on the
collaborative project and thinking through both about the process of
collaborating on a project and the content of the outputs: “I saw blogs and I
thought why should I not use it for encouraging reflection on what students
done and how much they learned” (P24).
Participants were keen on experimenting with employing locations and
learning spaces (other than the classroom) to enact multimodal teaching as
means to promote a more favourable and context-specific environment for
learning: “I was pondering, if we change the classroom as the primary location
and select a ‘museum’ or a ‘nature’ card location, would this increase students’
creativity, autonomy and inquiry?”(P43). Being curious on how a learning
space other than a classroom could be combined with activities and tools for
enabling creativity and self-direction was central in this theme.
6.
DISCUSSION
Teachers’ conceptualisations of in-game multimodal teaching encompass the
use of teaching strategies, digital tools and learning environments in
developmental modes, and at different levels of progression. The four themes
that grew out of this study, may be seen as an extension to similar experiences
supporting the view that multimodality gives emphasis on multiple modes of
representation and communication for constituting meaning-making (e.g.
Hassett & Curwood, 2009; Jewitt, 2013). To this line, the identified themes

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
19
resemble a set of conceptualisations of in-game multimodality, that were
twinned with strategies, tools and learning spaces that represent a set of multi-
literacies transcending from monomodality, as a linear, framed and fragmented
principle to multimodality, as a non-linear, unframed and cohesive practice (e.g.
Kress and Leewen, 2001).
In an earlier version (Papageorgiou & Lameras, 2017), we revealed an
explicit and implicit connection between ‘espoused theories’ and ‘theories in
use’ in terms of observing contingency and alignment between conceptions of
in-game multimodality and the employment of certain strategies and tools for
practicing multimodality in specific learning contexts. For example, teachers
who perceived in-game multimodality as a process of knowledge
memorisation and retention, they described that their preferred strategies,
tools and learning environments supported the materialisation of this specific
conception of multimodality. In tandem, conceptions of in-game
multimodality that supported a design-based do-it-yourself culture were
consistent with strategies, tools and locations that allowed the instantiation of
such conceptions of multimodality. In a broader spectrum of analysis in the
first theme, teachers sought to employ more oral / written modes, although
there were instances of using visual tools such as slides, alluding to
monomodal practices with little emphasis on the semiotic sign (e.g. Kress,
2001).
In the second, third and fourth themes, multimodal conceptualisations and
associated strategies, tools and learning environments are employed as a set
of ‘modes’ that connect representational resources with how teachers in the
game are using them. Particularly in themes three and four, there is evidence
of representations or interactions of more than one mode (e.g. a question tool
with a 3D printer for inquiry and design-based learning) that distribute a
meaning across a number of modes and not individual modes. Each mode
therefore, has to accomplish a specific purpose, as in the case of deploying a
question tool to introduce a research-based endeavour and then designing a
model to be produced in the 3D printer. We assert therefore that ‘multimodal
ensembles’ are delimited creating an interplay between modes with explicit
goals that each mode is assigned from the teacher to pertain targeted meanings
(e.g. Bezemer & Kress, 2008; Jewitt, 2013).
We contextualised the experience of understanding and employing
multimodality through a serious game which by itself is a semiotic domain
involving symbolic and representational multimodal resources to discern
multiple conceptualisations of multimodality. Rich and meaningful
representations of multimodality have been communicated and represented to
teachers influenced by the rules, mechanics, game play, design and inner-logic

20
Chapter # - will be assigend by editors
of the game. Since the STEAM game is an instance of a semiotic domain, and
visualises multimodal representations as experienced by the teachers, we need
to consider the implications of embracing the orchestration of the themes and
associated multimodal strategies, tools and location choices in real
classrooms. Game design elements such as game and learning mechanics,
rules, goals, interfaces, symbols and images used to manifest a multimodal in-
game representation are specific to particular in-game situations. This may
imply that the way that multimodality was conceived and enacted in the game
could be manifested differently in a live classroom. The identified four themes
of in-game multimodality may be different to other semiotic domains and
specific learning situations. It is a matter of situating the themes and their
elements in embodied ways of adapted multimodal manifestations both in
terms of a particular learning situation and of a specific learning domain.
Building and translating an artificial in-game multimodal meaning to an actual
tangible multimodal activity in the classroom is a necessary acclimatisation
based on teacher’s individual experiences and conditions of a particular
classroom-domain. This will warrant that the range of meanings attributed to
the game’s semiotic domain will be consciously and systematically rendered
to the classroom’s semiotic domain.
Another implication to transferring in-game multimodal experiences to
practice is the internal and external structures such the game design’s
influences. The STEAM game was designed with specific content, rules,
mechanics, dynamics and aesthetics that may influenced how teachers
experienced and understood multimodal learning and teaching. For example,
feedback design through multiple visual, textual and numerical instances
could regulate alternative conceptions of multimodality due to positive or
negative feelings of the feedback received from the game. Another example
is core game mechanics influences of the developed multimodal experiences.
The dialogue and game-cards core mechanics of the game may have created
multimodal conceptions deemed as transferable, applicable and compatible to
be practiced to another semiotic domain with few adaptations or, arguably, as
not acceptable practice due to an incompatible internal design grammar (e.g.
Gee, 2003). Actively and consciously knowing therefore what counts as
multimodal teaching, multimodal strategies, tools and learning spaces and also
discerning the roles, goals and identities that are formed in a lived semiotic
domain we could make connections of espoused in-game experiences of
multimodality transcended to recognisable, compatible and applicable
practice.

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
21
7.
CONCLUSION
The findings of this exploratory study suggest a number of opportunities
for further research. This is important for building on the outcomes of this
study and extend interdisciplinary strands of research related to conceptions
of multimodality for teaching and learning but also on the use of serious games
as semiotic domains and artefacts for amplifying learning, teaching and
training. For example, it would be sensible to investigate conceptions of in-
game multimodality for university teaching or from the student perspective at
different levels of study with a different game genre. Another interesting study
would be to understand and compare game effects on learning about
multimodal teaching in comparison to more conventional forms of teacher
professional development programs.
Limitations arise from our study in terms of associating conceptions of in-
game multimodality with effects on learning and engagement. We did not aim
to develop metrics and criteria that would provide inferences between
particular ways of experiencing multimodal teaching with learning
performance. However, it would be interesting to investigate whether there is
correlation between teachers’ conceptions of in-game multimodality with
students’ learning performance. This would also pave the way in
understanding subtly the transferability and applicability of developing in-
game multimodal experiences to the classroom’s semiotic domain.
REFERENCES
Antonietti, A., & Giorgetti, M. (2006). Teachers’ beliefs about learning from multimedia.
Computers in Human Behavior, 22(2), 267–282. https://doi.org/10.1016/j.chb.2004.06.002
Bellotti, F., Berta, R., De Gloria, A., Lavagnino, E., Dagnino, F., Ott, M., Romero, M., Usart,
I. & Mayer, I. S. (2012). Designing a course for stimulating entrepreneurship in higher
education through serious games. Procedia Computer Science, 174–186.
https://doi.org/10.1016/j.procs.2012.10.069
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of
designs for learning. Written communication, 25(2), 166-195.
Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., & Gratton, G. (2008). The effects of
video game playing on attention, memory, and executive control. Acta Psychologica, 129(3),
387–398. https://doi.org/10.1016/j.actpsy.2008.09.005
Cope, B., & Kalantzis, M. (2009). "Multiliteracies": New literacies, new learning. Pedagogies:
An International Journal, 4(3), 164–195.
Cope, B., Kalantzis, M., & Abrams, S. S. (2017). Meaning-making and learning in the era of
digital text. In Remixing multiliteracies: Theory and practice from New London to new times
(pp. 35–49).
Cowan, K. W., & Cipriani, S. (2009). Of Water Troughs and the Sun: Developing Inquiry

22
Chapter # - will be assigend by editors
through Analogy. Young Children, 64(6), 62–67.
Crook, K. M., & Crook, C. K. (2017). Multimodal Opportunities with Digital Tools: The
Example of Narrated Photographs. In Handbook on Digital Learning for K-12 Schools (pp.
13–29). https://doi.org/10.1007/978-3-319-33808-8_2
Del Blanco, Á., Torrente, J., Marchiori, E. J., Martínez-Ortiz, I., Moreno-Ger, P., & Fernández-
Manjón, B. (2012). A framework for simplifying educator tasks related to the integration of
games in the learning flow. Educational Technology and Society, 15(4), 305–318.
Gee, J. P. (2003). What Video Games Have to Teach Us About Learning and Literacy.
Computers in Entertainment CIE, 1(1), 20–20. https://doi.org/10.1016/j.vetpar.2013.04.007
Greenhow, C., Robelia, B., & Hughes, J. E. (2009). Learning, teaching, and scholarship in a
digital age: Web 2.0 and classroom research: What path should we take now? Educational
Researcher, 38(4), 246–257. https://doi.org/10.3102/0013189X09336671
Hassett, D. D., & Curwood, J. S. (2009). Theories and practices of multimodal education: The
instructional dynamics of picture books and primary classrooms. The Reading Teacher,
63(4), 270–282.
Jewitt, C. (2008). Multimodality and literacy in school classrooms. Review of Research in
Education, 32(1), 241–267. https://doi.org/10.3102/0091732X07310586
Jewitt, C. (2013). Multimodal methods for researching digital technologies. In The SAGE
Handbook of Digital Technology Research (pp. 250–265).
https://doi.org/10.4135/9781446282229.n18
Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of
contemporary communication. Edward Arnold.
Lave, J. (1991). Situating learning in communities of practice. Perspectives on Socially Shared
Cognition, 2, 63–82.
Lerkkanen, M. K., Kiuru, N., Pakarinen, E., Poikkeus, A. M., Rasku-Puttonen, H., Siekkinen,
M., & Nurmi, J. E. (2016). Child-centered versus teacher-directed teaching practices:
Associations with the development of academic skills in the first grade at school. Early
Childhood Research Quarterly, 36, 145–156.
Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
Miller, S. M., & McVee, M. B. (2013). Multimodal composing in classrooms: Learning and
teaching for the digital world. In Multimodal Composing in Classrooms: Learning and
Teaching for the Digital World. https://doi.org/10.4324/9780203804032
Papageorgiou, V., & Lameras, P. (2017). Multimodal teaching and learning with the use of
technology: Meanings, practices and discourses. 14th International Conference on
Cognition and Exploratory Learning in the Digital Age, CELDA 2017.
Picciano, A. G. (2009). Blending with purpose: The multimodal model. Journal of
Asynchronous Learning Network, 13(1), 7–18.
Schoorman, F. D., Mayer, R. C., & Davis, J. H. (2007). An integrative model of organizational
trust: Past, present, and future. Academy of Management Review, 32(2), 344–354.
Sun, M. (2015). Application of multimodal learning in online English teaching. International
Journal of Emerging Technologies in Learning, 10(4), 54–58.
https://doi.org/10.3991/ijet.v10i4.4697
Thibaut, P., & Curwood, J. S. (2018). Multiliteracies in practice: Integrating multimodal
production across the curriculum. Theory Into Practice. Theory Into Practice, 57(1), 48–55.

# - will be assigend by editors. experiences of multimodal teaching
through a serious game: meanings, practices and discources
23
ACKNOWLEDGEMENTS
The authors would like to thank the school teachers and our end-user partners
that helped us with the data collection process and the EU Erasmus + project
with grant agreement n°2016-1-FR01-KA204-024178 which funded the
development, design and evaluation of the game.
AUTHOR INFORMATION
Include the following information for each contributing author:
Institutional affiliation:
Dr Petros Lameras: School of Computing, Electronics and Mathematics,
Coventry University
Ms Vasiliki Papageorgiou: Centre for Higher Education Research and
Scholarship, Imperial College London
Institutional address:
Dr Petros Lameras: Coventry University, Priory Str, CV1 5FB
Ms Vasiliki Papageorgiou: Imperial College London, South Kensington, SW7
2AZ
Complete mailing address:
Dr Petros Lameras: Chambers Building, IV8, Techno Park, Puma Way,
Coventry, CV1 2TT
Ms Vasiliki Papageorgiou: Level 5, Sherfield Building, Exhibition Road,
London, SW7 2AZ
Telephone number:
Dr Petros Lameras: (+44) (0)2476158253

24
Chapter # - will be assigend by editors
Ms Vasiliki Papageorgiou: (+44) (0) 2075943516, (+44) (0) 7542344722
Email address:
Dr Petros Lameras: ab3430@coventry.ac.uk,
Ms Vasiliki Papageorgiou: v.papageorgiou18@imperial.ac.uk
Short biographical sketch (200 words maximum):
Dr Petros Lameras is an Assistant Professor at the Faculty of Engineering,
Environment and Computing, Coventry University, an associate researcher at
the Post-Digital Cultures Research Centre and a Visiting Scientist at the
Massachusetts Institute of Technology – Media Lab for doing research on
games for STEM and games for science teacher training. He is currently the
principal investigator of research projects funded by the European
Commission such as the H2020 Beaconning the Erasmus + STEAM,
Rhythm4All and GOAL and a co-investigator on the NEWTON project
CreativeCulture. His interests span the areas of serious games, game-based
learning, deep learning and artificial intelligence in teaching and learning,
information science, STEM teaching, technology-enhanced learning in
general and within the research strand of mapping pedagogical models and
principles to game design. He has been awarded the prestigious Society for
Research into Higher Education (SRHE) award in UK for currying out
research on features of serious games design encompassing an attempt to
match game categories / mechanics with learning attributes.
Vasiliki (Vily) Papageorgiou is currently a Doctoral Researcher at Imperial
College London focusing on online pedagogy, course design in online
environments and decision-making in multidisciplinary design teams. She has
worked on FP7 and Erasmus+ European projects in the area of Technology-
Enhanced Learning as well as on educational projects with industry partners.
She has also been the principal investigator of individual research studies on
the use of technologies to enhance learning and teaching in K-12 Education
with a focus on collaborative and social learning. Vily is a qualified teacher
and has international experience in teaching nursery and primary school
students (including children with special needs).