Multimodal Learning

Abstract

The widespread use of technology in the digital age continually shapes how individuals consume knowledge and learn. In the digital age, ideas are shared and represented in multiple formats and through the integration of multiple modes. Technological advances, coupled with considerations of the changing needs of today’s learners, call for exploring new directions for multimodal teaching and learning. Yet, society’s increasing reliance on, and use of, technologies for communication and learning has introduced expanded forms of meaning-making. Information and communication technologies (ICTs) and the online networks that are facilitated by their use encourage educators to transform the way education is delivered. Learning environments are in need of becoming transformed so students are able to use immersive technologies to expand their learning opportunities. This chapter explores emerging trends and pedagogies in multimodal learning that seek to take advantage of the digital tools, texts, and learning approaches that are continually shaping the ways learning occurs inside and outside of higher education.
This chapter is outlined to highlight what is found in the literature on multimodal instruction, what findings were realized at eXploring the Future of Innovative Learning Environments (X-FILEs) workshops, and lastly how multimodal instruction can be used to transform the classroom of the future. Throughout this chapter, readers will get to know a student of the future, Juan Delgado. He attends a 4-year university in Dallas, Texas, and is majoring in Mechanical Engineering taking his Introduction to the Fundamentals of Science course. Each aspect of the learning process as it relates to multimodal instruction in 2023 is outlined through the experiences of Juan to situate the impact to learners.

Full text

35© The Author(s) 2021
J. Ryoo, K. Winkelmann (eds.), Innovative Learning Environments in STEM
Higher Education, SpringerBriefs in Statistics,
https://doi.org/10.1007/978-3-030-58948-6_3
Multimodal Learning
BettyjoBouchey, JillCastek, andJohnThygeson
1 Emerging Trends andPedagogies
The ubiquitous use of digital technologies is continually reshaping the ways indi-
viduals access information, share ideas, and communicate with one another. Doing
so requires the nimble use of skills, strategies, and mindsets to navigate, communi-
cate, and collaborate online and across multiple contexts (Leu etal. 2017). These
changes have profoundly affected instructional choices in education. In today’s edu-
cation landscape, three key motivations challenge traditional notions of teaching
and learning and set forth a strong case for multimodal learning as a critical
pedagogy:
1. There exists a proliferation of information in several modes: gestures, visuals,
haptics, auditory productions, text-based information, and multimedia.
Representing information through different modes, and/or using a combination
of modes, can create multiple access points for learning (Bezemer and Kress
2016; Matusiak 2013; Nouri 2018; Sankey etal. 2010).
2. There is an increased opportunity for individuals, including workers and learn-
ers, to represent and communicate information using different modes (Blum and
Barger 2018; Matusiak 2013).
B. Bouchey ()
National Louis University, Chicago, IL, USA
e-mail: bbouchey@nl.edu
J. Castek
University of Arizona, Tucson, AZ, USA
e-mail: jcastek@arizona.edu
J. Thygeson
Drexel University, Philadelphia, PA, USA
e-mail: jrt46@drexel.edu

36
3. Students are increasingly diverse (National Center for Education Statistics n.d.).
The academy has long relied upon instructional approaches that favor text-based
learning. These approaches are sometimes referred to as factory models of teaching
because they include top-down management and favor outcomes designed to meet
societal needs and rely on age-based classrooms (Bezemer and Kress 2016;
Matusiak 2013; Phuong etal. 2017; Sankey etal. 2010). Researchers posit that pre-
senting information in this one-size-ts-all way does not accommodate the widest
possible range of learners’ needs (Gee 1996; Phuong etal. 2017). As technologies
continue to shape the ways individuals engage in learning, exchange ideas in net-
works, and communicate in multiple forms, educators must expand their ideas of
how, where, under what conditions, and which tools people use to learn in a digital
age. The changing nature of student demographics and shifts in the technological
landscape call into question traditional ways of teaching and learning.
Technological advances, coupled with considerations of the changing needs of
today’s learners, call for exploring new directions for multimodal teaching and
learning. Multimodal learning (MML) can be dened as “learning environments
[that] allow instructional elements to be presented in more than one sensory mode
(visual, aural, written)” (Sankey etal. 2010: 853). Multimodality looks at the many
different modes that people use to communicate with each other and to express
themselves. Multimodal learning is relevant as increases in technological tools and
associated access to multimedia composing software have led to the ease of use of
many modes in presenting, representing, and responding to information.
MML calls for sensemaking where learners take in information, process, and
make personal sense of ideas to form deep learning patterns (Bezemer and Kress
2016; Moreno and Mayer 2007). MML expands the range of choices available in a
learning environment so that learners can co-construct learning through their pre-
ferred mode while also being challenged by integrating the use of other modes
(Nouri 2018; Phuong etal. 2017; Sankey etal. 2010). MML requires a high level of
agency (self-discipline) by learners, who must have the metacognition necessary to
understand how they learn and also when to challenge themselves to learn in ways
that lie outside their preferred modes. Such sophisticated awareness requires a stu-
dent to explore their metacognition and requires instructors to explore how assess-
ments are administered and leveraged (Bezemer and Kress 2016; Moreno and
Mayer 2007; Phuong et al. 2017; Sankey et al. 2010). MML promotes a greater
emphasis on encouraging student voices during the learning process and calls for
educators to listen to students’ preferences. As such, multiple modes should be used
to shape the content, response options, as well as the delivery of instruction (Nouri
2018; Phuong etal. 2017).
As a powerful means to customize and adjust learning strategies to reach diverse
learners, MML leverages our technology-rich world, digital tools, and networks and
can be used to address Universal Design for Learning (UDL) as well. UDL offers
three basic principles that encourage the design of supportive learning environ-
ments. These principles call for instructional designs that build in multiple means
and modes of (1) representation (offering exible ways to present what we teach
B. Bouchey et al.

37
and learn), (2) action and expression (offering exible options for how we learn and
express what we know), and (3) engagement (offering exible options for generat-
ing and sustaining motivation to learn) (Rose and Meyer 2002). Encouraging learn-
ers to compose ideas using different modes encourages connections to concepts and
illustrates ways to employ the principles of UDL.Moreover, deep learning happens
when students are given the opportunity to participate in a range of cognitive and
social learning activities that are responsive to their preferences and needs or modes
(Bezemer and Kress 2016; Moreno and Mayer 2007). Most importantly, MML
serves as well-researched pedagogy that sets a platform for educators needing to
shift their emphasis from the delivery of instruction to the engagement of their
learner (Bezemer and Kress 2016; Tonsing-Meyer 2013). In this way, MML can be
an equalizer for diverse students because the use of different modes encourages
diverse ways of communicating. MML can be inclusive of multilingual students,
English language learners, and those with learning differences and moreover can set
the stage for an engaging, rich, and creative learning experience.
1.1 Aspects ofTeaching andLearning Covered intheMML
Literature
An MML literature survey was conducted in conjunction with eXploring the Future
of Innovative Learning Environments (X-FILEs) project. Several areas of teaching
and learning were examined to create a framework to conceptualize new forms of
learning in the next veyears, in other words in 2026. The areas of teaching and
learning spanned (a) content presentation, (b) interactions and discussions, (c)
learner activities, (d) assessment and evaluation, and (e) co-curricular learning and
activities. The following is an overview of literature pertaining to each of these areas.
1.1.1 Content Presentation
Course content can be presented and delivered in multiple modes, such as visuals,
auditory productions, text, and multimedia (Bezemer and Kress 2016; Matusiak
2013; Nouri 2018; Sankey etal. 2010). Through MML, content can be curated,
either in advance or dynamically, to accommodate multiple access points. Multiple
modes are offered so that students can gravitate toward the modes that best align
with their individual styles and needs and to encourage the complementary nature
of multimodal content (Nouri 2018; Phuong etal. 2017; Sankey etal. 2010). This
section assumes that content is either contained in a learning management system
for purely online coursework or blended for hybrid and face-to-face instruction and
that multimodal content and delivery can augment traditional lectures.
Multimodal Learning

38
1.1.2 Interactions andCommunications
The use of MML in higher education creates a foundation for exible, open, and
multidirectional interaction and discussions with students, their peers, faculty, staff,
and the business and community at large. As students embark on sensemaking and
co-construction of knowledge in an MML-based classroom (Bezemer and Kress
2016; Moreno and Mayer 2007), they can use a variety of modes to engage with
other stakeholders to shape their understanding. Using different modes, students
can spontaneously engage in parallel and complementary dialog to deepen learning,
answer questions, and engage in sensemaking of the learning objectives in and
around a classroom.
1.1.3 Learner Activities
Jewitt (2008) suggests that how knowledge is represented, as well as the mode and
media selected, is a crucial aspect of knowledge construction that makes the form of
representation integral to sensemaking and learning, more generally. In other words,
the way something is represented shapes what is to be learned, the curriculum con-
tent, and how it is to be learned. In our digital world, our efforts to design instruction
must be responsive to the ways we intend learners to use the new knowledge they
acquire in the workplace and beyond. MML calls for educators to shift their role
from learning designer to learning facilitator (Bezemer and Kress 2016; Tonsing-
Meyer 2013). Because MML sets a foundation for a co-construction of knowledge
between faculty and students (Bezemer and Kress 2016), learner activities are avail-
able just-in-time, based on the student’s preferred mode and personalized, drawing
in the student’s preferences for learning. In addition, the use and integration of dif-
ferent modal representations can reinforce ideas and make learning more memorable.
1.1.4 Assessment
Course assessment strategies must also be multimodal in order to ensure objectivity.
Whether formative or summative, all course assessments are designed in a modally
agnostic way so that the nal product can be produced in the student’s preferred
mode and preferably in a way that demonstrates the intersectionality and comple-
mentary nature of different modes (Jewitt 2008; Nouri 2018; Phuong etal. 2017;
Sankey etal. 2010).
1.1.5 Co-curricular
Co-curricular learning refers to activities, programs, and learning experiences that
complement, in some way, what students are learning in formal classes, but also
serve to expand the relevancy and contextualization of that learning in the real
B. Bouchey et al.

39
world. Since MML is pedagogical, it does not explicitly reference co-curricular
activities or learning. Even so, opening up classroom interactions and discussions
into a multimodal environment provides a foundation for co-curricular learning
from stakeholders outside of the classroom, such as college and university staff,
business and community leaders, friends, and family. Learning from those outside
of the classroom should not be overlooked or undervalued.
1.2 Research Questions Answered
The literature on multimodal instruction also answered research questions around
design for multimodal instruction and how to use multiple modes to encourage deep
learning. Moreover, the literature helped to contextualize many of the early research
questions and provided a democratic, pedagogical framework within which to con-
ceptualize content presentation, interactions and discussions, learner activities,
assessments and evaluations, and co-curricular instruction in 2026.
1.3 Future Research
As noted in the 3-day X-FILEs workshop, there remain areas that call for future
research, specically in the areas of how MML can impact a student’s identity
development and self-initiative. Also discussed in the workshop, there remain some
practical areas that would benet from further formalization specic to how institu-
tions of higher education support the technology needed to implement multimodal
instruction. More inquiry into the efcacy of some modes over others in terms of
deep and sustained learning may also be needed, perhaps as a way of creating syn-
ergy across modes as well. Workshop participants also noted the importance of
developing a framework for the ethical use of MML that would suggest appropriate
use and rules of engagement that support students, faculty, and staff in setting
healthy boundaries for immersive learning in an information- and technology-rich
society. Further research is also needed to better understand the affordances that
multimodal instruction can create for students, especially as it pertains to students
with disabilities. Lastly, as co-curricular learning takes hold in the use of multi-
modal instruction, workshop dialog indicated external validation, perhaps by higher
education professionals or other community and business partners of learning cre-
dentials, will need to be investigated.
Multimodal Learning

40
2 Use ofMultimodal Learning in2026
This chapter brings MML into focus through the work of the eXploring the Future
of Innovative Learning Environments (X-FILEs) project spanning a series of focus
groups, 3-day in-person workshop, and further investigation through a team of
researchers, industry partners, and faculty brainstorming on how this democratic
pedagogy can shape (a) course content presentation, (b) interactions and discus-
sions within a course, (c) learner activities, (d) assessment and evaluation, and (e)
co-curricular aspects of education. Each section outlines opportunities for using
MML as a framework, implementation strategies, and key questions that have been
answered or left open in the literature.
2.1 Content Presentation
Content presentation has progressed from an educational format in which the
instructor is charged with passing on his/her information to students via lecture,
recitation, and associated activities into an experience in which information is pre-
sented to learners in many ways. While lecture is still used, it is frequently supple-
mented with additional educational content in additional formats, both in class and
away from the classroom.
2.1.1 Opportunities
Several affordances exist in providing multimodal course content, chief among
them being the self-directed nature of students selecting content delivered in the
most familiar or comfortable mode, which is when the deepest learning occurs
(Bezemer and Kress 2016; Moreno and Mayer 2007). Moreover, traditional class-
room lectures can be augmented by multimodal content and delivery in the learning
management system or can provide the opportunity for classroom time to be used
for group- or project-based learning. Augmenting content in a learning management
system or the traditional classroom lecture can create an active learning space where
students are focused on discovery, rather than consumption of information.
Discussed in the X-FILEs workshop, this content created by faculty could be com-
moditized, whereby students could subscribe to certain modes based on their indi-
vidual preferences for a small fee, rather than relying on traditional per-credit tuition
models. In an online course, MML course content could also spark faculty and
course designers to create and curate content in new and innovative ways.
Additionally, the multimodal nature of virtual environments such as lab simula-
tions, virtual reality, and augmented reality provide experiential opportunities that
would otherwise be very difcult to provide in “real life.”
B. Bouchey et al.

41
Another opportunity exists to deliver MML course content to those who require
assistive technologies. Assistive technology (AT) can be dened as any item, piece
of equipment, software program, or product that increases, maintains, or improves
the functional capabilities of people with disabilities (Assistive Technology Industry
Association 2019). The number of people who could benet from AT is not insig-
nicant. Currently, 15% of the world’s population has some form of disability, and
20% of those people experience signicant disabilities. As the world’s population
continues to grow, so too will the total population of those that experience disabili-
ties; furthermore, this population is aging, and there is an increase in chronic health
conditions (World Health Organization 2011).
In addition to reaching individuals who would normally have difculties access-
ing education, using these technologies can help educators stay compliant with the
law. The Rehabilitation Act of 1973 and the Americans with Disabilities Act
addressed workplace support for people with disabilities and were later adapted for
use in educational situations (U.S.Equal Employment Opportunity Commission
n.d.; U.S.General Services Administration 2018). Providing individuals with dis-
abilities with the opportunity to the same education and in the same time frame as
students without disabilities will benet the education provider by staying compli-
ant with laws and regulations.
2.1.2 Challenges
Workshop participants agreed that chief challenges in designing courses and deliv-
ering them to incorporate MML are cost and speed of design. It may be cost- and
time-prohibitive to create content in all learning modes. For example, one course
module on calculating averages in a statistics class could contain the following con-
tent items:
1. A word document with the problem completed by the professor (text-based)
2. A video of the problem being completed as students watch and observe on a
virtual whiteboard (gestures, video, and auditory)
3. An interactive, adaptive instructional module where a student answers a series of
prompts to solve the problem, with dynamic instruction provided if a wrong
answer is inputted or the student indicates they have a question
In addition to the time and cost of curating MML content, faculty may nd it
difcult to switch between modes while in the “live” classroom environment, sug-
gesting more of a need to augment the lecture through MML content in a learning
management system as well. Faculty and instructional designers may also nd it
difcult to nd or create content in several modes, especially if they fall outside of
their individually preferred modes.
Another challenge of using MML in course content is relying on a student’s self-
awareness and self-direction to explore and represent content in their preferred
mode or combination of modes. Accessing all course content in one mode, even if
preferred, might not be the most effective approach as combining modes may serve
Multimodal Learning

42
as reinforcement and encourage connections. There may also be implications with
respect to accessibility where some modes might be richer than others. For example,
a student who only accesses materials in a preferred mode may not benet from
deeper learning utilizing content presented in different modes or ones that incorpo-
rate many modes such as virtual reality.
2.1.3 Implementation Strategies
Incorporating MML in course content and delivery of this type of instruction in
“real time” by a faculty member requires a symbiotic relationship with the student
so that the faculty member can match instruction to the learning needs of their stu-
dents. In thinking through online courses, curating content requires examining
instructional design frameworks that guide the thoughtful assembly of course con-
tent. The seminal and informative ADDIE model of design, which includes the ele-
ments of analysis, design, development, implementation, and evaluation (Molenda
2003), can still be used, perhaps with iterations derived from rapid prototyping.
2.1.4 Research Questions
Literature on MML clearly answers the research questions set forth by the X-FILEs
team in this area of how to choose the best mode for a subject or student group.
Through its very nature, MML provides content that addresses all modes regardless
of subject matter or student group characteristics.
As Juan accesses his online class for his Introduction to the Fundamentals of
Science, he is faced with many choices that he will navigate throughout the
term. He has a selection of content available to him to support his rst week
of learning, including chapters from an online textbook, a series of videos and
podcasts, and a set of keywords he can use to locate his own or additional
content—he is also asked to register for his asynchronous virtual reality
classroom where he can access his book, the videos, and podcasts and per-
form and post his research. His university has obtained a grant to fund the
augmented course design, and Juan has a low-cost technology fee that he
pays each term to offset the cost of the virtual classroom licensing, hosting,
and support to the institution. The virtual classroom is where the majority of
his course will actually take place throughout the week, though he will have
several opportunities to participate in the learning “live” or through other
modes as well. To provide the students with a sense of familiarity, the virtual
classroom is designed just as a school is; each room corresponds to a different
subject or module. Juan navigates his avatar into each room to view content
and participate in class as though he was sitting in a physical classroom.
B. Bouchey et al.

43
2.2 Interactions and Communications
Interactions and discussions around academic coursework have evolved from epi-
sodic, in-class lectures and peer dialog to full-scale, immersive, and 24/7 opportuni-
ties for dialog and deep learning. While in-class lecture and dialog remain central
ways to build foundational learning, the co-curricular opportunities for interactions
and discussions serve as a platform for more immersive learning.
Using the principles of UDL (Gordon etal. 2014) and frameworks that encour-
age the wide use of MML, students, peers, faculty, staff, and community/business
stakeholders can engage in open and exible and dialogic interactions and discus-
sions. Dialogic interactions create space for shared leadership in learning in which
learners and instructors become co-inquirers; in this way, a more egalitarian class-
room environment can emerge, devoid of traditional teacher/student power dynam-
ics. Together, they collaboratively engage in creating and evaluating new
interpretations in order to gain insight into the world, themselves, and each other
(Reznitskaya 2012). Learners can and should be viewed as active drivers of their
own learning (Bezemer and Kress 2016; Sankey etal. 2010). In this way, students
can choose multiple modes within which to engage in learning and receive just-in-
time or real-time information to help contextualize their learning. Providing an
opportunity for multidirectional modes of communication enables students to be
resourceful and frees up faculty time as well.
2.2.1 Opportunities
As students shift from consuming knowledge to actively driving and constructing it,
they can enhance their understanding by using different modes to represent their
understanding. Interacting across modes frees students to think more creatively and
integrate knowledge faster. By opening up meaning-making (Postman and
Weingartner 1969), the process by which people interpret situations, events, objects,
or discourses, to expand communication, MML may also empower students to gen-
erate meaning and make sense of themselves, their experiences, and their relation-
ships through their learning. Moreover, MML affords students who may have
trouble communicating in a particular mode with the opportunity to express them-
selves in another. In this more exible and dynamic environment, students with
disabilities can freely operate within their most capable mode. MML also provides
new avenues for student collaboration, beyond the connes of the traditional class-
room or learning management system.
2.2.2 Challenges
Despite the benets and possibilities that MML can make possible, there are also
challenges introduced. For example, the use of images to convey meaning requires
faculty to develop visual literacies required to infer how others may interpret their
Multimodal Learning

44
creations. Not all viewers walk away with the same meaning from visuals, and thus
other modes may need to be integrated to convey a common meaning. Identied in
the workshop, another challenge may be introduced when initializing interactions
exclusively via technology among learners who come from different cultural back-
grounds. In the absence of context clues that come from facial expressions, body
language, and gestures, miscommunications may occur. Pairing MML strategies
with opportunities to connect through two-way videoconference technologies and to
openly dialog about different communication norms across cultural groups may
support overcoming these challenges. A nal challenge with MML identied in the
workshop centers around limiting opportunities for face-to-face communication due
to technological dependency. Relying solely on technologies for bridging distances
between learners may in fact create more distance and feelings of isolation. MML is
intended to provide options for learning and instruction and does not intentionally
aim to create distance-only options. Discussing ideas face-to-face, in combination
with the use of different modes of meaning-making, can provide exible options for
learning engagement. We aim to support educators in their implementation of MML
in support of designing learning interactions of all kinds, including face-to-face.
Another challenge in the use of MML in discussions and interactions is the level
of ability in technology use among faculty, staff, and students. The challenge stems
from the concept that today’s students are digital natives, while most of today’s
teachers are digital immigrants (Bezemer and Kress 2016; Blum and Barger 2018;
Kirschner and De Bruyckere 2017; Lambert and Cuper 2008; Moreno and Mayer
2007; Phuong etal. 2017; Picciano 2009; Prensky 2001; Sankey etal. 2010; Wang
et al. 2014). A less rigid denition of the digital natives/immigrant is now used in
which the technology exposure is more a factor of a variety of reasons, such as
demographics, opportunity, social inuence, cognitive knowledge, and socioeco-
nomic status (Kesharwani 2020). A key concept to take from these studies on digital
natives/immigrants is that a primary focus should be placed on reaching across the
technological divide between varying technology levels when designing course
content. Relatedly, it is important to think through access to technology among dif-
ferent members of the population and how this relates to the value of MML to dis-
cussions and interactions.
Lastly, as technology evolves, communication channels must be evaluated for
privacy and security. Workshop attendees stressed that it is critically important for
faculty and students to know what modes and channels may be open for public
viewing. In fact, faculty professional development on tool privacy and accessibility
will be important and ever-changing year to year or perhaps term to term as the
speed of technological advances and tool creation continues to increase. At the same
time, students need orientation to tool privacy and accessibility and experience with
the types of tools the university provides, as opposed to ones they are not. There
may be tools that do not meet privacy or accessibility standards that the university
would not support. The choice of tools and the benets and drawbacks, should be
communicated clearly to students.
B. Bouchey et al.

45
2.2.3 Implementation Strategies
To motivate students to interact beyond the traditional faculty-student relationship,
faculty can provide for multimodal communication either in advance, while a course
is in progress, or following learning activities to promote reection (Dewsbury
2019). For example, a classroom meeting or online discussion question could have
a companion virtual world classroom where students interact in real time through
avatars or a backchannel for a face-to-face meeting where students can contextual-
ize the lecture with peers or other educational stakeholders. Course participation
could also be linked to time-limited assignments in the workplace or community in
person, through videoconference, or virtual networking to encourage idea-sharing
and real-world application of learned content.
2.2.4 Research Questions
While research in MML is not entirely new, research that addresses challenges iden-
tied in the workshop concerning second language learners are an important direc-
tion to pursue. Offering multimodal texts is often benecial for second language
(L2) learners as multiple modes can scaffold understanding (Plass et al. 1998;
Royce 2002). This is because, as Farias and Abraham (2017) state:
multimodal texts more effectively support second language reading by providing input that
caters to different learning styles and that they are familiar, authentic, and contextualized to
the learners’ lives. Moreover, these texts facilitate learners’ meaningful interaction not only
intratextually, by exploring the text/image semiosis, but also intertextually, by allowing
readers to become literate in the different genres that are constructed multimodally.
Thus, while L2 learning is supported by the use of multimodal texts, it can also
develop learners’ broader understanding of concepts. Examining identity and liter-
acy development surrounding the creation of multimodal texts both within and
beyond course objectives is an fruitful area in need of additional research. These
directions can further support MML for L2 learners across elds. While MML
seems to indicate there are benets associated with the use of multiple modes of
communication for students, less is understood about the benets of students’
choices; research into this area is still needed.
In the virtual classroom, Juan can engage through phone, text, email, instant
messaging, or other means with his professor, a group of scientists in industry
and other colleges, as well as all his peers and advisors. Professor Aranda
also actively engages in the virtual classroom and spends time cultivating
personalized relationships with each student—in addition to maintaining the
relationship with the community of scientists and supporting the classroom.
During the Tuesday evening session, she carefully plans a short didactic lec-
ture that is then supplemented with a series of small group virtual breakout
rooms facilitated by other scientists where students and scientists work
through the theory and discuss ways to demonstrate mastery of the learning
objectives for the week.
Multimodal Learning

46
2.3 Learner Activities
Learner activities can take many shapes and forms and may connect resources avail-
able through coursework to online and community resources. The choices faculty
make include multimodal options that build toward applying what is learned within
and beyond the classroom. However, choices often depend on individual faculty and
the pedagogical disposition of a department, college, or university. In this way,
learner activities can be varied and may involve multiple formats and response
options that call for different ways to illustrate learning.
MML calls for educators to shift their role from lecturer to learning facilitator
and designer (Bezemer and Kress 2016; Tonsing-Meyer 2013). MML encourages
uidity between accessing information using multiple modes and representing
knowledge through the integration of multiple modes. This approach encourages
personalization as well as creativity and calls for a co-construction of knowledge
between faculty and students (Bezemer and Kress 2016). In an MML environment,
learner activities can be more collaborative, designed to require real-world problem-
solving, and created to apply knowledge based on a compelling purpose for learning.
2.3.1 Opportunities
In shifting the focus from lecturer to learning designer or facilitator, faculty are free
to work with individual students to understand their current state of learning and to
work collaboratively on learner activities that will enhance their mastery. Students
are thus empowered to nd the best ways to integrate the subject matter alongside
faculty. Faculty can also learn from student-designed learner activities when they
use modes and tools unfamiliar to the faculty.
2.3.2 Challenges
When students drive their own learning, they are able to benet from the multiple
modes that are available to them. Working between and across modes may serve to
support deeper learning and improve retention of key concepts. MML requires a
high degree of agency, and provides opportunities for learners to select a focus or
direction for themselves. This requires that learners develop the metacognition nec-
essary to understand how they learn and also when to challenge themselves (Bezemer
and Kress 2016; Moreno and Mayer 2007; Phuong etal. 2017; Sankey etal. 2010).
The time needed for faculty to engage with students on learner activity design may
be prohibitive in larger courses. It also may be difcult to objectively assess learn-
ing when activities are varied across a student group.
B. Bouchey et al.

47
2.3.3 Implementation Strategies
Learner activities in an MML environment are student-driven, based on preferred
modes. Still, faculty could develop a range of learner activities, mapped to course
learning objectives, that can be completed in several modes to enable students to
select an appropriate activity. These activities may be experiential and hands-on or
operate in a virtual environment. For example, an assignment either could be open-
ended in terms of how the learning is demonstrated such as a document, a presenta-
tion, or a video or could have a range of possible formats the students could
choose from.
2.3.4 Research Questions
Research questions in this area rest on the opportunity that exists to operationalize
MML-based learner activities to promote student engagement. The MML literature
clearly supports the shift from subject matter delivery to that of engagement, as well
as the positive benets of this. Workshop participants aptly pointed out that it will
also be important to compare MML-based learner activities to other types of peda-
gogical frameworks for efcacy.
2.4 Assessment
Traditional means of assessment and evaluation have often revolved around tests
and written assignments. Over time, educators have embraced new ways of assess-
ing competency through action projects, multimedia presentations, and other types
of practical application strategies. MML can be a helpful framework in the assess-
ment strategy in a course or classroom environment that frees up the faculty and
students for more creative ways to show mastery. No matter the type of assessment,
formative, summative, or performance-based, designing assessments that invite
All of Juan’s learning activities are “hands-on” in the virtual classroom and
include a series of science experiments that he can perform in the virtual
space, at home (where he can record and post in the virtual classroom), or at
a series of physical labs his university has partnered with. Throughout the
course, Professor Aranda is working to ensure that the course content, learn-
ing activities, and support from other scientists and peers all work together to
further his individual learning. Professor Aranda is aware that Juan’s pre-
ferred modes of learning are in visual and auditory realms so she takes time
to ensure that he is challenged to incorporate writing and presentation skills,
as well as to spend time in the virtual space to drive the intersectionality of
multimodal learning and to create more strength in other modes of learning
and information consumption.
Multimodal Learning

48
different response options may free students from a specic way of illustrating their
knowledge. In this way, through MML options, learners are encouraged to produce
demonstrations of their learning in modes that they have an afnity for. Moreover,
providing suggestions for multiple ways to illustrate learning may open the doors to
creativity and deeper levels of learning (William and Flora Hewlett Foundation 2016).
2.4.1 Opportunities
By designing assessments so that they can be completed in multiple modes, students
can draw from their modally-driven learning to produce artifacts in the mode that
they are most comfortable and familiar with. By reducing the cognitive load associ-
ated with using an unfamiliar or uncomfortable mode, learning and mastery can be
more easily conveyed. Students could avail themselves of technological advance-
ments to capture their learning as it evolves, ostensibly faster than higher education
curricular design cycles. In addition, by opening up the classroom co-curricularly
as discussed in section 2.5, assessments can be conducted by stakeholders outside
of the classroom walls, providing helpful feedback from employers, community
partners, and others that can help shape the student’s learning in ways that may not
be within the scope of what a faculty member knows or has experienced, personally.
Ultimately, the opportunity to have multiple stakeholders providing evaluation and
feedback affords faculty with more data to gauge learning and to plan to reduce gaps
and increase levels of mastery. With more and consistent data streams through mul-
timodal evaluation, the possibility for machine learning to aid in faster and more
efcient grading also presents itself.
2.4.2 Challenges
Students might not choose a mode that effectively demonstrates their learning, or
they may select a mode that the faculty member is not familiar with. Diverse out-
puts may challenge the use of rubrics or programmatic assessment plans that rely
upon standardized artifacts or other uniform, objective measures given to students.
Moreover, it may be very time-consuming for a faculty member to grade assess-
ments produced in many modes because mastery may “look” different based on the
mode in which it was produced. Students may also choose to demonstrate their
learning in a mode that lacks adequate privacy or accommodations for students with
disabilities—presenting concerns around assisting their selection of modal delivery
as part of the assessment directions.
2.4.3 Implementation Strategies
In higher education, a culminating project or nal assignment often requires learn-
ers to synthesize, analyze, and create content using a variety of modalities. Such
projects, when closely aligned with learning outcomes, support relevant and authen-
B. Bouchey et al.

49
tic assessment that leverages digital communication strategies. An emerging practi-
cal model designed specically for multimedia instructional strategies, CASPA,
contains ve components: consume, analyze, scaffold, produce, and assess. This
model can be used to promote curriculum-based integration of multimodal projects
for assessment in higher education courses. The CASPA framework for assignment
design asks students to iteratively attempt and create culminating assessments
throughout a course (Blum and Barger 2018). The framework allows students to
experiment with different modes as they complete a culminating assignment.
Faculty feedback and direction throughout the term could be used to enable learners
to choose the best modal demonstration of their learning by the time the end product
is due. Faculty will design assessment artifacts that are open-ended and driven by
course learning objectives. No formal description of the end products or artifacts is
provided to the student, though it may be appropriate to provide options to help
students choose effective outputs or ones that protect the student’s privacy and that
are accessible to all students, regardless of their abilities. Faculty may also consider
creating standardized metrics that would allow for cross-modal comparisons of
learning products illustrating that there are multiple ways to achieve course aims.
2.4.4 Research Questions
Research questions in this area center on the affordances of some modes over oth-
ers, and this remains an area for further research as identied by workshop partici-
pants; perhaps some modes are more effective than others in the demonstration of
learning. Furthermore, faculty modal preference is rarely discussed in the literature.
Faculty who are evaluating artifacts outside their modal preference may not be able
to objectively assess them due to the cognitive load required to adapt to new or
unfamiliar modes.
Juan and his peers are encouraged to locate and post content that aligns to
each week’s learning objective through a framework of information valida-
tion provided by the university. Students are encouraged to comment on these
new content postings as a way of sensemaking and co-construction of knowl-
edge. Experiments are the main source of assessment and evaluation in his
course, but Juan also has several opportunities to identify and capture scien-
tic theories in his everyday life through a system of digital storytelling. This
can include taking a picture and posting about an experiment, recording
audio or video about seeing science in nature, or providing a summary of a
scientic experience through video or a text-based summary. Professor
Aranda awards points to every learning capture that he is willing to submit in
the course; in fact, all of the assessments in the course are documented as
merely a list of learning objectives so that each student can demonstrate mas-
tery through modes that they have an afnity for or modes that lend them-
selves to the objective being assessed.
Multimodal Learning

50
2.5 Co-curricular Activities
While the idea that learning does not always take place in the traditional classroom
setting has been accepted for many years (Dewey 1997), educators and students are
now embracing the concept. Co-curricular learning refers to activities, programs,
and learning experiences that complement, in some way, what students are learning
in formal classes, but also serve to expand the relevancy and contextualization of
that learning in the real world. Co-curricular contexts enable the learner to interact
with material outside the classroom in such a way that they acquire new knowledge
or apply a skill from one area to a new area. Increasingly, institutions of higher
education are embracing co-curricular learning and its benets to their students
(Stirling and Kerr 2015; Turrentine etal. 2012).
2.5.1 Opportunities
By broadening learning to extend beyond the academy, students can build profes-
sional networks, enhance their learning through multiple perspectives, and more
quickly build workforce-ready skills. Through co-curricular access to other educa-
tional stakeholders, students can engage and consume new learning in the mode
they most prefer. Students could also participate and drive change in their commu-
nity by working directly with business and community leaders to couple their learn-
ing with practical applications.
These co-curricular activities could take many forms, and in one scenario, edu-
cational stakeholders such as a venerable scientist or engineer would be purpose-
fully contracted by the institution to participate in the virtual world associated with
the course to:
1. Host weekly informational sessions via web conferences, virtual world environ-
ments, or even via chat
2. Make himself/herself available for text- or video-based question/answer sessions
on the week’s subject matter from the classroom
3. Use virtualized laboratories for student observation
4. Provide feedback and insight into informal or formal assessment of the subject
matter
Each scenario further extends the student’s learning, infused with the relevancy
and currency of a working professional in their eld of study.
2.5.2 Challenges
Many institutions of higher education do not have formal connections with the pri-
vate sector that underrepresented students’ need to benet from co-curricular learn-
ing. Faculty may also be challenged with assessing which learning should happen
outside the classroom and what should be inside the classroom (face to face or
B. Bouchey et al.

51
online), may struggle with how to integrate or sequence this type of learning with
classroom-based learning. Federal credit hour denitions may also limit co-curric-
ular learning or make it difcult to “count” it, so it would be additive, perhaps con-
tributing to cognitive overload for students. Validation of co-curricular learning may
be difcult to assess as well.
2.5.3 Implementation Strategies
Faculty can design course learning objectives tied to business and community inter-
action, where the choice of modalities would be driven by a student’s individual
preferences. For example, a course learning objective could read “Students will
consult with a scientist to discuss what types of careers a science graduate can pur-
sue upon graduation.” Faculty could design course content and set the stage for
interactions and discussions that engage educational stakeholders from outside the
walls of the classroom.
2.5.4 Research Questions
Research questions in this area center on the benets and consequences of students’
learning outside of the classroom. MML literature is seemingly quiet in this area,
though it was a theme discussed in the workshop. There does remain a risk to the
delity, meaning the degree to which application of MML can be replicated, across
iterations of co-curricular learning programming. As co-curricular learning
increases, capturing the outcomes and related learning that stems from MML will
need to be developed and well understood by faculty and students as well.
3 Conclusion
While there are clear opportunities and affordances offered through the use of MML
in the classroom, there are also inherent limitations to MML, particularly related to
cognitive load. Too many modes and too much information can overwhelm learners
Each Tuesday night, Juan’s whole class, his professor, and a group of scien-
tists from all over the world gather to sit in the virtual classroom to discuss
vital scientic theory that will enable him to be more successful when he is
attempting his experiments. These same scientists are working with Juan and
Professor Aranda throughout the term to customize the learning and to be
sure Juan is engaged in the most up-to-date practice. Juan is developing a
strong network of future colleagues as he interacts with these external educa-
tional stakeholders.
Multimodal Learning

52
(Moreno and Mayer 2007; Sweller etal. 1998). Furthermore, fundamentally shifting
traditional teaching would require intentional, responsive, and on-going profes-
sional development so that teachers can develop multimodal literacies. One must
also consider the cost and complexity (Bezemer and Kress 2016) of curating learn-
ing environments so that learners can choose from all of the varied types of modes.
Though challenges do exist in the use of MML, the opportunities and affordances
that exist through incorporating facets of MML in classroom teaching and learning
far outweigh the challenges. Moreover, as student demographics continue to evolve
into more and more diversity and technology continues to reshape learning interac-
tions, instruction will inherently take on a multimodal nature.
References
Assistive Technology Industry Association (2019) What is AT? https://www.atia.org/at-resources/
what-is-at/
Bezemer J, Kress G (2016) Multimodality, learning and communication: a social semiotic frame.
Routledge, NewYork
Blum M, Barger A (2018) The CASPA model: an emerging approach to integrating multimodal
assignments. J Educ Multimedia Hypermedia 27(3):309–321
Dewey J (1997) Experience and education. Simon & Schuster, NewYork
Dewsbury B (2019) Teaching with technology in higher ed: start with relationship-building.
EdSurge. https://www.edsurge.com/news/2019-01-02-teaching-with-technology-in-higher-ed-
start-with-relationship-building
Farias M, Abraham P (2017) Reading with eyes wide open: reections on the impact of multi-
modal texts in second language reading. Íkala 22(1):57–70. http://dx.doi.org.ezproxy4.library.
arizona.edu/10.17533/udea.ikala.v22n01a04
Gee JP (1996) Discourses and literacies. In: Luke A (ed) Social linguistics and literacies: ideology
in discourses, 2nd edn. Taylor & Francis, London, pp122–148
Gordon D, Meyer A, Rose D (2014) Universal design for learning: theory and practice. CAST
Professional Publishing
Jewitt K (2008) Multimodality and literacy in school classrooms. Rev Res Educ 32:241–267
Kesharwani A (2020) Do (how) digital natives adopt a new technology differently than digi-
tal immigrants? A longitudinal study. Inf Manag 57(2):103170. https://doi.org/10.1016/j.
im.2019.103170
In the end, Juan nishes his Introduction to the Fundamentals of Science
course with a deep understanding of scientic theory and its application in
the real world. He also leaves the course with a newly formed group of men-
tors which includes Professor Aranda and the scientists who supported the
course. He has also formed deep bonds with his peers and plans to connect
with them in future courses. Lastly, he leaves the course with a new sense of
how improving his writing and presentation skills can provide value not only
in deepening his learning but in how he presents himself to the world outside
of the university.
B. Bouchey et al.

53
Kirschner PA, De Bruyckere P (2017) The myths of the digital native and the multitasker. Teach
Teach Educ 67:135–142. https://doi.org/10.1016/j.tate.2017.06.001
Lambert J, Cuper P (2008) Multimedia technologies and familiar spaces: 21st century
teaching for 21st-century learners. Contemporary issues in technology and teacher
education 8(3). https://www.citejournal.org/volume-8/issue-3-08/current-practice/
multimedia-technologies-and-familiar-spaces-21st-century-teaching-for-21st-century-learners/
Leu DJ, Coiro J, Kinzer C, Castek J, Henry LA (2017) New literacies: a dual level theory of the
changing nature of literacy, instruction, and assessment. Special issue teaching and learning in
the 21st century. J Educ 197(2):1–18
Matusiak KK (2013) Image and multimedia resources in an academic environment: a qualitative
study of students’ experiences and literacy practices. J Am Soc Inf Sci Technol 64(8):1577–
1589. https://doi.org/10.1002/asi.22870
Molenda M (2003) In search of the elusive ADDIE model. Perform Improv 42(5):34–36
Moreno R, Mayer R (2007) Interactive multimodal learning environments. Educ Psychol Rev
19(3):309–236
National Center for Education Statistics (n.d.) Table 203.60 in Digest of Education Statistics.
https://nces.ed.gov/programs/digest/d17/tables/dt17_203.60.asp?
Nouri J (2018) Students multimodal literacy and design of learning during self-studies in higher
education. Technol Knowl Learn 24:683. https://doi.org/10.1007/s10758-018-9360-5
Phuong AE, Nguyen J, Marie D (2017) Evaluating an adaptive equity-oriented pedagogy: a study
of its impacts in higher education. J Effect Teach 17(2):5–44
Picciano AG (2009) Blending with purpose: the multimodal model. JALN 13(1):7–18
Plass JL, Chun DM, Mayer RE, Leutner D (1998) Supporting visual and verbal learning prefer-
ences in a second language multimedia learning environment. J Educ Psychol 90:25–36
Postman N, Weingartner C (1969) Teaching as a subversive activity. Delacorte Press, NewYork
Prensky M (2001) Digital natives, digital immigrants. On the Horizon 9(5)
Reznitskaya A (2012) Dialogic teaching: rethinking language use during literature discussions.
Read Teach 65(7):446–456
Rose D, Meyer A (2002) Teaching every student in the digital age. ASCD, Alexandria. http://www.
cast.org/teachingeverystudent/ideas/tes/
Royce T (2002) Multimodality in the TESOL classroom: exploring visual-verbal synergy. TESOL
Q 36(2):191–205
Sankey M, Birch D, Gardiner M (2010) Engaging students through multimodal learning environ-
ments: the journey continues. In Proceedings of ASCILITE—Australian Society for Computers
in Learning in Tertiary Education Annual Conference 2010, pp852–863
Stirling AE, Kerr GA (2015) Creating meaningful co-curricular experiences in higher education. J
Educ Soc Policy 2(6). http://jespnet.com/journals/Vol_2_No_6_December_2015/1.pdf
Sweller J, van Merrienboer JG, Paas FC (1998) Cognitive architecture and instructional design.
Educ Psychol Rev 10(3):251–296
Tonsing-Meyer J (2013) An examination of online instructional practices based on the learning
styles of graduate education students. Q Rev Dist Educ 14(3):141–149
Turrentine C, Esposito T, Young MD, Ostroth DD (2012) Measuring educational gains from par-
ticipation in intensive co-curricular experiences at Bridgewater State University. J Assess Inst
Effect 2(1):30–54
U.S.Equal Employment Opportunity Commission (n.d.) The Rehabilitation Act of 1973. https://
www.eeoc.gov/laws/statutes/rehab.cfm
U.S. General Services Administration (2018, November) IT Accessibility Laws and Policies.
https://www.section508.gov/manage/laws-and-policies
Wang S, Hsu H, Campbell T, Coster D, Longhurst M (2014) An investigation of middle
school science teachers and student use of technology inside and outside of classrooms:
considering whether digital natives are more technology savvy than their teachers. Educ
Technol Res Dev 62(6):637–662. https://doi-org.ezproxy2.library.drexel.edu/10.1007/
s11423-014-9355-4
Multimodal Learning

54
William and Flora Hewlett Foundation (2016) What is deeper learning? http://www.hewlett.org/
programs/education/deeper-learning/what-deeper-learning
World Health Organization (2011) World report on disability. WHO Press, Switzerland. http://
whqlibdoc.who.int/publications/2011/9789240685215_eng.pdf?ua=1
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter's Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.
B. Bouchey et al.