Content uploaded by Peng Zhang
Author content
All content in this area was uploaded by Peng Zhang on Sep 25, 2024
Content may be subject to copyright.
Peng Zhang* and Yue Peng
Multimodal reading in reading-only versus
reading-while-listening modes: evidence
from Chinese language learners
https://doi.org/10.1515/caslar-2024-2003
Abstract: Language educators and researchers have shown keen interest in
enhancing second language (L2) learners’reading skills, and recently, more
emphasis has been placed on multimodal learning. Multimodal learning theories
posit that the confluence of auditory and visual stimuli facilitates learners to
establish robust referential connections, and thus integrating audio cues has
emerged as a strategy to enhance reading performance. While prior research has
affirmed the advantages of incorporating audio cues for fostering reading fluency
and comprehension in general, the findings are inconsistent. These investigations
have predominantly focused on learners of English as a second language. Consid-
ering the differences between English and Chinese, especially their relationship
between written scripts and oral language, investigating the reading experiences of
Chinese language learners can contribute to a fuller understanding of the effect of
audio inclusion on L2 learners. This study compared reading comprehension and
reading experiences in reading-only and reading-while-listening modes for
intermediate-low and intermediate-high-level Chinese language learners. After
learners read in two different modes, a T-test was conducted on their comprehension
performance followed by a qualitative analysis of their perspectives on their reading
experiences. The findings showed a significant influence of audio integration on
enhancing reading comprehension among intermediate-low-level Chinese language
learners engaged in multimodal Chinese reading, and these learners displayed a
preference for audio inclusion. On the other hand, a lack of significance in the effects
of supplementary audio cues on reading comprehension was identified for
intermediate-high-level learners, who manifested a mixed attitude towards audio
integration with reading. This study complements the existing understanding of
multimodal reading based on alphabetic language, particularly in English. Moreover,
the findings carry pertinent implications for Chinese language reading pedagogy.
*Corresponding author: Peng Zhang, Namseoul University, 10218, Humanities and Social Sciences
Building, 91 Daehak-ro, Seonghwan-eup, Seobuk-gu, Cheonan-si, Chungcheongnam-do, South Korea,
E-mail: nsuiben@nsu.ac.kr
Yue Peng, Room 404, Ho Tin Building The Chinese University of Hong Kong, Shatin, Hong Kong SAR,
China, E-mail: yuepeng@cuhk.edu.hk
CASLAR 2024; 13(2): 215–236
Keywords: audio integration; multimodal learning; reading comprehension;
reading-while-listening
摘要:近年来,语言教育者和研究人员对提升第二语言学习者的阅读技能表现出
浓厚的兴趣,尤其是多模态学习的研究备受关注。多模态学习理论认为,听觉和
视觉刺激的结合能够帮助学习者建立更强的参照联系,因此,整合音频提示已成
为提高阅读表现的一种策略。尽管已有研究证实音频提示在提升阅读流畅性和
理解方面的优势,但研究结果并不一致,且现有研究主要集中在以英语为第二语
言的学习者上。鉴于英语和汉语在书写系统与口语之间的关系不同,研究汉语
学习者的阅读体验有助于更全面地理解音频融入对二语学习者的影响。本研究
比较了中低水平和中高水平汉语学习者在纯阅读和边听边读两种模式下的阅读
理解和阅读体验。学习者在两种模式下进行阅读后,研究通过T检验分析他们的
阅读理解表现,并对其阅读体验进行了定性分析。研究结果表明,对于中低水平
的汉语学习者,音频融入显著提升了他们的阅读理解,这些学习者对音频融入表
现出明显的偏好。而对于中高水平学习者,补充音频线索对其阅读理解的影响
并不显著,这些学习者对边听边读表现出了混合态度。本研究丰富了关于基于
字母语言(特别是英语)的多模态阅读的现有理解。此外,研究结果对中文阅读
教学具有重要的启示。
关键词:多模态学习;阅读理解;边听边读;音频融入
1 Introduction
The development of reading abilities is a fundamental aspect of second language (L2)
acquisition since attaining fluent reading skills is crucial for achieving a higher level
of L2 proficiency in general (Kara 2019). In recent educational practices, there has
been a shift towards employing multimodal strategies to support the development
of L2 reading competencies (Zhang 2021). Multimodal learning, which involves
activating multiple sensory channels such as visual and auditory streams, is
increasingly recognized for its role in enhancing the educational experience and the
capacity for learning across different subject areas, including language acquisition
(Choi and Yi 2015; Zhang 2021).
Integrating multimodal elements into reading activities, specifically, has been
identified as a promising approach to improving reading fluency and comprehen-
sion. Engaging multiple senses by combining reading with auditory support, for
instance, has been shown to facilitate a deeper connection with the text and improve
retention, thereby enriching the learning process (Tragant et al. 2016). The added
value of multimodal reading practices, such as reading-while-listening, has been
substantiated in various studies, with findings indicating notable benefits in terms of
216 Zhang and Peng
literacy development and reading skill enhancement (Fälth et al. 2022; Nurviyani
et al. 2020).
While prior research has largely affirmed the different levels of advantages of
incorporating audio cues for fostering reading fluency and comprehension, these
investigations have predominantly focused on learners of English as a second lan-
guage (Bisson et al. 2015; Chang and Millett 2014, 2015; Conklin et al. 2020; Tragant
et al. 2016; Webb and Chang 2014, 2020). The unique characteristics of the Chinese
writing system, including its logographic nature and the incorporation of radicals
within a complex two-dimensional structure, present distinct challenges for L2
learners, particularly those with a background in alphabetic languages (Shen 2004).
Additionally, the phonetic and semantic nuances of the Chinese language add layers
of complexity for learners attempting to navigate this system (Ye 2011). Considering
the differences between English and Chinese, especially the relationship between
written scripts and oral language, investigating the reading experiences of Chinese
language learners can contribute to a fuller understanding of the effect of audio
inclusion on L2 learners. Addressing the above-noted research gap, this study
compared reading comprehension and reading experiences in reading-only and
reading-while-listening modes for intermediate-low and intermediate-high-level
Chinese language learners.
2 Literature review
2.1 Multimodal learning and multimodal reading
Multimodal learning, which underpins the concept of multimodal reading, is pred-
icated on the activation of various sensory modes during the learning process,
including visual, auditory, kinesthetic, text, and technological inputs (Choi and Yi
2015; Gilakjani et al. 2011; Moreno and Mayer 2007). This approach to learning
acknowledges the diversity of individual learning styles and preferences, positing
that the engagement of multiple senses can enhance information processing and
retention (Green and Abutalebi 2013). Evidence suggested that such an approach not
only enriched the educational experience for students (Choi and Yi 2015) but also
amplified their learning capacity and potential (Zhang 2021). Furthermore, Sacco
et al. (2021) contended that the efficacy of learning is bolstered by multimodal pre-
sentations, indicating a broader scope for improving educational outcomes through
this method.
Multimodal reading, which involves the interpretation of various modes such as
visual, audio, and spatial elements, has gained significant attention in educational
research. The use of multimodal texts in teaching and learning has been shown to
Multimodal reading in CSL learners 217
enhance students’literacy skills and comprehension (Damayanti and Febrianti 2020;
Fälth et al. 2022). Educators have recognized the importance of developing multi-
modal literacy in students, particularly in the context of digital reading (Lim and Toh
2020; Miki 2020). This is supported by the findings that a multimodal reading training
program had positive effects on students’literacy development, including decoding,
spelling, and reading comprehension (Fälth et al. 2022). Furthermore, the use of
multimodal texts in the classroom has been found to be effective in improving
students’reading skills significantly (Nurviyani et al. 2020).
The multimodal reading process entails integrating information from various
sources and establishing correlations between several modalities to construct
meaning (Pellicer-Sánchez et al. 2021). Leibrandt (2022) suggested that multimodal
reading competence encompasses several key components, including attributing
meaning to visual representations, comprehending the interplay between written
language and visual messages, and effectively combining semantically enriched
language with contextualized imagery. For reading literary text, for example,
Leibrandt noted that readers can enhance their understanding of a piece of litera-
ture by ascribing meaningful interpretations to visuals within the literary frame-
work, including visual messages that complement the written content and
combining language with contextualized imagery.
Given the advantages of multimodal reading, supporting learners with skills in
reading multimodal texts has grown in significance within educational contexts. It is
imperative for students to possess the skills necessary to navigate and comprehend
the various types of multimodal texts they come across effectively (Fälth et al. 2022;
Hung 2022; Lim and Toh 2020).
2.2 Reading-while-listening in L2 reading
The reading-while-listening approach is a method that integrates both the processes
of reading and listening in the context of second language acquisition. As a type of
multimodal reading, the role of audio inclusion in visual reading has been exten-
sively investigated in academic research. In general, studies provided evidence to
support the idea that engaging in simultaneous reading-while-listening activities
provides greater benefits for L2 acquisition compared to engaging in reading-only
activities (Bisson et al. 2015; Chang and Millett 2014, 2015; Conklin et al. 2020; Tragant
et al. 2016; Webb and Chang 2014, 2020; Van Vu 2023).
Among others, the advantage of reading-while-listening on reading compre-
hension is most pertinent to the focus of our study. Chang and Millett (2015) exam-
ined reading-only and reading-while-listening modes among a sample of English as a
second language (ESL) teenage learners in Taiwan. They found improvements in
218 Zhang and Peng
reading speed and comprehension for both groups. Notably, the reading-while-
listening group demonstrated superior performance in reading comprehension.
In their study, reading-while-listening led to higher levels of comprehension and
concentration, making the input more engaging for students. Conklin et al. (2020)
focused on eye movement during reading and found that reading-while-listening
influences where the eyes move on the page, showing a consistent pattern regardless
of whether reading in a first language (L1) or L2. This consistency has suggested
that reading-while-listening taps into similar mental processes active in separate
reading or listening, which might help explain why comprehension and retention
are improved with reading-while-listening. In addition to enhancing reading
comprehension, other advantages brought by reading-while-listening have been
acknowledged, such as fostering vocabulary acquisition. The study conducted by
Bisson et al. (2015) showed that incorporating audio with visual and textual com-
ponents appeared advantageous in facilitating incidental vocabulary acquisition.
Similarly, Webb and Chang (2014) also found evidence of more unintentional vo-
cabulary learning when reading-while-listening than in reading-only activities. In
sum, while the impact of reading-while-listening on L2 reading comprehension has
been extensively studied, generally in favor of incorporating auditory input for
improving reading comprehension, the current literature has primarily focused on
English language learners with other L2s neglected.
2.3 Reading in the Chinese language
Reading comprehension is vital in the acquisition of the Chinese language, particu-
larly for learners who are not native speakers. Research has illuminated various
factors that influence reading comprehension among Chinese learners. For example,
the significance of morphological awareness, especially among bilingual individuals,
was emphasized by studies such as those conducted by Pasquarella et al. (2011) and
Qiao et al. (2021), highlighting its substantial contribution to Chinese reading
comprehension. Specifically, Qiao et al. (2021) emphasized the contribution of
morphological awareness to reading comprehension in both English and Chinese,
indicating that the inclusion of morphological awareness in reading instruction
could enhance vocabulary, word reading, and reading comprehension. Additionally,
decoding, the ability to recognize characters and words, is another pivotal factor in
Chinese reading comprehension, as identified by Wong et al. (2023). Zhang et al.
(2021) drew attention to the intermediary role of lexical inference in linking
morphological awareness with reading comprehension in L2 Chinese learners.
These research findings supported the notion that audio inclusion, which would
Multimodal reading in CSL learners 219
potentially aid learners’morphological awareness through pronunciation and word
recognition, can enhance their Chinese reading comprehension.
In reading Chinese, the inclusion of audio in learning materials also emerged
as an intriguing area for exploration with its potential effects on reading compre-
hension. A recent eye-tracking-based research by Mo and Jiang (2024) provided
empirical evidence for the beneficial effects of reading-while-listening on Chinese L2
learners’reading processes. Their study found that reading-while-listening led to
more efficient reading, with learners exhibiting fewer fixations, shorter fixation
durations, and reduced regressions compared to reading-only. Elsewhere, Koh’s
(2024) study has shed light on how cross-linguistic orthographic distance impacted
the effectiveness of reading-while-listening for L2 reading comprehension. Exam-
ining intermediate Korean learners of Chinese, English, and Spanish, Koh found that
learners studying languages with greater orthographic differences from their L1
Korean, such as Chinese, benefitted more from reading-while-listening. Koh’s
research findings suggested that reading-while-listening can be particularly useful
when learners face significant orthographic disparities between their L1 and L2.
While these studies have provided intriguing insights into the impact of incor-
porating audio to reading, the research conducted on its effect on L2 Chinese reading
comprehension remains comparatively limited. Given the scarcity of empirical
evidence, despite some indications of the benefits of incorporating audio in Chinese
reading comprehension, further research is needed to gain a comprehensive
understanding of this issue.
Considering the advantages of multimodal reading strategies, especially the
simultaneous engagement in reading and listening, and recognizing the gap in
research regarding the impact of this approach on L2 Chinese learners of different
proficiency levels, particularly within the context of online multimodal reading
environments, this study seeks to investigate the following two research questions:
RQ1: To what extent does the reading-while-listening mode enhance reading comprehension for
Chinese language learners at intermediate-low and intermediate-high levels, respectively?
RQ2: How do learners in two proficiency groups perceive their experiences of reading in the two
different modes?
3 Methodology
The study adopted a mixed-methods approach. By including both quantitative and
qualitative data, this study aimed to gain a comprehensive understanding of the
impact of different reading modes on Chinese language learners’reading
220 Zhang and Peng
comprehension. Specifically, the quantitative data and analyses provided statistical
evidence regarding the influence of audio input on students’performance in
multimodal reading, addressing RQ1. The qualitative analysis, on the other hand,
allowed for an exploration of participants’perspectives of their reading experiences
in the two modes, addressing RQ2. Overall, qualitative data provided insights into
possible explanations of the patterns identified in RQ1.
3.1 Participants
The study sample comprised 66 secondary school students enrolled in Grades 7
through 10 at an international school in Hong Kong. Their participation was
entirely voluntary and was sanctioned by the necessary consent from the school’s
governing body and the student’s legal guardians. These individuals ranged in
age from 12 to 16 years and were all undertaking Mandarin Chinese courses as a
second language. The student demographic was characterized by a middle-class
upbringing, with English and Cantonese being the top two predominant home
languages, respectively, though not all students spoke both languages at home.
Notably, Mandarin was not commonly used in their home environments. The
educational context for these participants was an international school setting
where English was the primary language of instruction for all subjects other than
Chinese.
For the follow-up focus group interviews, eight students from the intermediate-
low group and eight students from the intermediate-high group volunteered to
participate. Within each group, half of the students experienced reading-only mode,
while the other half experienced reading-while-listening mode.
3.2 Materials
The study employed an online multimodal Chinese reading platform called iChine-
seReader. iChineseReader is a graded multimodal reading resource designed to meet
the reading needs of K-12 Chinese language programs, and is used in Chinese
language programs in immersion, bilingual, and international schools worldwide.
The platform offers over three thousand interactive mini e-books/articles catego-
rized into twelve distinct reading proficiency tiers, covering novice, intermediate,
and advanced levels, with reference to ACTFL’s scale. All e-books accessible through
this portal contain reading materials supplemented with colorful and dynamic
visuals related to the content, as shown in Figure 1, providing readers with
Multimodal reading in CSL learners 221
multimodal reading experiences. The visuals were included for both groups in this
study as part of the multimodal reading materials, and thus were not treated as a
variable between the groups.
In addition to visual input, teachers can set up various reading modes on iChi-
neseReader, such as reading-only and reading-while-listening, allowing for adapt-
ability to learners’individual preferences. The audio recordings of the texts were
narrated by professional voice talents in standard Mandarin at a speech rate of
130–165 Chinese characters per minute. It is noted that the reading materials that the
intermediate-high level group read were slightly faster than the intermediate-low
group. But the speed of the audio within each group remains the same. In general,
this feature-rich platform aligns well with the focus of this study.
3.3 Data collection
Assigning learners into two proficiency levels. The first step of the study was to gauge
the participants’Chinese language proficiency, particularly with regard to reading
skills. To achieve this purpose, a benchmark test was first administered on the
iChineseReader platform. This self-adaptive test gave participants passages and
articles to read, followed by questions of multiple choices, yes-no questions, and
Figure 1: Excerpt of multimodal reading materials in iChineseReader.
222 Zhang and Peng
short answers to assess their reading comprehension. Based on students’answers,
the test adjusted the difficulty of reading materials and questions automatically.
Participants completed the benchmark test in 20–35 min.
Against the 20-point-scale proficiency levels, learners were assigned by the
testing system to a specific level between 9 and 17. Students were then organized into
groups based on these levels: those at levels 9–12 were classified as intermediate-low,
while those at levels 14–17 were classified as intermediate-high. Notably, there were
three participants assigned to level 13 –directly in the center of the intermediate
range –and another three to level 19, which is considered advanced. These six
students were then excluded from the study. To ensure the accuracy of this classi-
fication, four classroom teachers, familiar with the students’reading skills, were
asked to review and validate the groupings. Their insights confirmed the initial
placement, necessitating minimal adjustments by the teachers. Following this
validation process, the final participant distribution consisted of 34 in the
intermediate-low group and 32 in the intermediate-high group.
Reading in two modes. Next, each group of readers was given four articles to
read: two in reading-only mode and two in reading-while-listening mode. The order
of the texts was randomized to control for order effects. All reading materials were
taken from iChineseReader’s collections, with materials for lower proficiency
levels chosen from levels 9–13, and material for higher proficiency levels taken from
levels 14–19. The articles were presented to students one at a time to avoid testing
memory in addition to comprehension. Upon completing the assigned reading,
participants were instructed to undertake a comprehension test to evaluate their
understanding of the reading materials before moving on to the next text. Each
comprehension test included 10 questions in total, with five multiple choice ques-
tions, three true-or-false questions, and two short answer questions. The reading
comprehension questions were evenly spread across the four texts, with five
multiple choice questions generated by iChineseReader, and another five questions
created by the researcher. The participants completed the reading followed by
reading comprehension questions within 45 min of class.
Focus group interviews. Upon completing the reading comprehension test, two
focus group interviews were conducted separately. The participants were invited to
reflect on their experiences of reading in the two distinct reading modes and how
the reading modes had an impact on their reading comprehension. The focus
groups were conducted in English, lasting for about 15 min on average. Open-ended
questions were asked to elicit students’reflections on their reading experiences in
the two modes and how the modes impacted their reading comprehension. The
interviews were audio-recorded and later transcribed for analysis.
Multimodal reading in CSL learners 223
3.4 Data analysis
The quantitative aspect of the data analysis involved the examination of reading
comprehension scores obtained from participants under two different conditions:
reading-while-listening and reading-only. The collected data were entered into the
Statistical Package for Social Sciences (SPSS) software for analysis. Two-sample t-tests
were employed to determine if there were statistically significant differences in
reading comprehension performance between the two instructional modes across
the two proficiency levels. The t-tests allowed us to compare the mean scores of the
intermediate-low proficiency group and the intermediate-high proficiency group,
providing a clear understanding of how each proficiency level responded to the
reading conditions.
For the qualitative segment, the participants’attitudes were explored along
with the reading conditions. The transcriptions of the focus group interview data
were incorporated to NVivo 14, a software for qualitative data analysis. The first step
in this analysis involved conducting an attitudinal analysis towards the reading
modes based on the participants’reflections by using NVivo’s sentiment analysis
function. The responses were coded using an attitudinal scale that included cate-
gories such as moderately positive, very positive, moderately negative, and very
negative. The categorization was based on the strength of the language used by
participants, with descriptors such as “very”,“greatly”,“significantly”,“a lot”, and
“super”classified as strongly positive or negative. Language such as “rather”,“some”,
and without any modifiers was classified as moderately positive or negative.
4 Findings
The results of the t-test and the qualitative analysis are presented in this section, with
a focus on the two research questions respectively.
4.1 RQ1: the effect of audio inclusion on reading
comprehension
Table 1 displays the results of the t-test conducted for the intermediate-low profi-
ciency level Chinese language learners. The results of the two-sample t-test show that
there is a statistically significant difference between the reading-while-listening
group (mean = 16.70, standard deviation = 1.99) and the reading-only group
(mean = 14, standard deviation = 2.92) when it comes to reading comprehension
224 Zhang and Peng
scores for Chinese language learners with an intermediate-low level of proficiency.
The t-test indicates that the reading-while-listening mode was associated with higher
reading comprehension scores compared to the reading-only mode. Furthermore,
the p-values for both one-tailed and two-tailed tests were well below the significance
level of 0.05, further supporting the observed difference.
In contrast, as depicted in Table 2, the results of the two-sample t-test demon-
strated that there was no statistically significant difference in the mean values
between the reading-while-listening group (mean = 16.75, standard deviation = 1.98)
and the reading-only group (mean = 16.38, standard deviation = 2.03) for the
Table :Two-sample assuming unequal variances T-test: intermediate-low.
RWL RO
Mean .
Variance .
SD . .
Observation
Hypothesized mean difference
Df
tStat .
P(T<=t) one-tail .
tCritical one-tail .
P(T<=t) two-tail .
tCritical two-tail .
Table :Two-sample assuming unequal variances T-test: intermediate-high.
RWL RO
Mean . .
Variance . .
SD . .
Observation
Hypothesized mean difference
Df
tStat .
P(T<=t) one-tail .
tCritical one-tail .
P(T<=t) two-tail .
tCritical two-tail .
Multimodal reading in CSL learners 225
intermediate-high proficiency group (p= 0.6). This finding implies that, at the
intermediate-high level of proficiency, both reading approaches have comparable
outcomes in improving reading comprehension. The results for the intermediate-
high proficiency level imply that there was no statistically significant difference in
the reading comprehension scores between the reading-while-listening and reading-
only modes.
4.2 RQ 2: learners’experiences on reading in the two modes
Learners’Attitudes. Derived from students’written reflections, the qualitative data
revealed different attitudes towards the two reading modes between the two pro-
ficiency groups. Those attitudes are exhibited in hierarchy charts, generated by
NVivo, and are used as a visual representation that organizes and displays data nodes
and themes in a structured manner. The size of each section in the hierarchy charts
represents the relative frequency coded references within each theme.
Figures 2 and 3 display the results of the attitudinal analysis for the
intermediate-high and intermediate-low groups respectively.
The analysis from the intermediate-high group revealed mixed attitudes
towards the audio incorporation in their reading process, as shown in Figure 2. There
were six very negative, five moderately negative, and five moderately positive
references.
Figure 2: Hierarchy chart of attitudinal codes: intermediate-high.
226 Zhang and Peng
In contrast to the intermediate-high group, the analysis from the intermediate-
low group revealed positive attitudes predominantly towards using multimodal
input in Chinese language learning, as shown in Figure 3. Specifically, there were 15
very positive and two moderately positive references.
Following the initial classification of readers’attitudes, the dataset was further
analyzed to identify specific themes related to the advantages and disadvantages of
including audio in the reading process. Figures 4 and 5 present the synthesized pros
and cons mentioned by the intermediate-high and intermediate-low groups
respectively. For the intermediate-high group as shown in Figure 4, the main
perceived advantages were improved understanding and concentration, while the
disadvantages included slower reading speed and over-reliance on audio. For the
intermediate-low group as shown in Figure 5, the primary benefits were enhanced
understanding, suitability for lower proficiency levels, and assistance with character
recognition. No disadvantages were mentioned by this group.
Perceived advantages and disadvantages. The advantages and disadvantages of
audio inclusion, as identified by intermediate-high-level learners, are summarized
in Figure 4. The results revealed that most of the students believed that audio
inclusion boosted their understanding of the reading material. One of the reasons
that the learners cited was that audio input helped with their concentration. For
example, S03 mentioned, “I like to read with the audio. I think it helps me under-
stand. I just like to listen to it to be more concentrated”. Similarly, students such as
S02 and S05 found the audio inclusion helpful, especially when they were tired or
needed to concentrate more on reading. S02 reflected:
Figure 3: Hierarchy chart of attitudinal codes (intermediate-low).
Multimodal reading in CSL learners 227
I might choose to listen to the audio when I am tired and don’t want to look at the texts. If I focus
on reading without sound, I’ll probably read faster and more efficiently. The audio is good for
me to understand, but it is not necessary.
At the same time, some students found disadvantages in the audio inclusion, stating
that it intervened with their reading speed. For instance, S06 reflected, “It stopped
me from reading fast, I can read better quietly.”Similarly, S8 mentioned,
Figure 4: Hierarchy chart of synthesized pros and cons of audio inclusion (intermediate-high).
Figure 5: Hierarchy chart of synthesized pros and cons of audio inclusion (intermediate-low).
228 Zhang and Peng
“iChineseReader reads too slowly; it even stopped to go to the next page because it
read too slowly. I want to read by myself without the reading voice.”The participants
also recognized that they sometimes over-relied on the audio and did not read
the texts properly, which further negatively influenced their Chinese character
recognition. S01 admitted:
I sometimes just listen to the audio without looking at the texts since listening is easier than
reading Chinese characters for me …this is not good, and I need to practice my text-reading, but
sometimes I just can’t help but reply to the listening.
In addition, the participants also perceived that the reading-while-listening mode
might be more suitable for lower grades and for those with lower proficiency levels.
S4 and S7 argued that they preferred reading with audio while using Chinese when
they were in lower grades and had lower Chinese skills before, and now they prefer
reading independently without listening.
As for the intermediate-low group, the perceived benefits and drawbacks of
audio inclusion are summarized in Figure 5. In general, the results uncovered that
the students found reading-while-listening to be highly beneficial. The majority
indicated that reading-while-listening mode boosted their understanding. For
example, S10 mentioned, “The audio in reading kind of helps me with understanding
because I can use both my ears and eyes while reading; it’s more efficient.”Some
learners believed that audio inclusion was suitable for those with lower proficiency
levels. S12, S13, and S09 stated that their Chinese is not as good as other classmates,
and they felt that the audio in iChineseReader is particularly helpful for them
because they could refer to two channels of information instead of just reading the
words. Notably, this group mentioned no disadvantages.
It is important to note that the benefit of understanding Chinese characters
provided by the audio input was a major reason that learners identified it as a
support to their reading comprehension. For example, S14 stated:
It helped me a lot because there are lots of words that I will not understand from just looking at
them, and having the sound of the words, in my reading, reminds me of the meaning of the
words. So, I can understand better.
In addition, S11 reflected, “I totally depend on the sound since it is difficult for me
to recognize all of the Chinese characters. The audio helps me read faster and
understand the things that I read better.”
At the same time, some students pointed out the relativity between their reading
ability and the reading materials when appreciating the audio input. S09 and S13
found the audio inclusion helpful when the reading materials were at or above their
proficiency level. For example, S13 stated:
Multimodal reading in CSL learners 229
If I read something very easy that I can totally understand without any problems or something
I learned, the audio might not be necessary. However, if I read iChineseReader that is at or
above my level, I always like to choose the audio reading version to help me understand better.
5 Discussion
In this section, quantitative and qualitative results are integrated. Specifically,
qualitative data on learners’reading experiences is incorporated to make sense of
the quantitative results on the two groups of learners’performance in multimodal
reading comprehension. Theoretical significance in relation to existing studies is
addressed, followed by offering pedagogical implications.
5.1 The factor of proficiency level to multimodal reading
In understanding the influence of multimodal reading strategies on Chinese
language learners, proficiency levels emerged as a pivotal factor. In this study,
proficiency significantly affected how multimodal reading was processed, and the
benefits arising from it. This finding is in line with previous research, which
underscored that learners at different stages of language acquisition might react to
multimodal inputs in varied ways (Taguchi et al. 2016).
The quantitative data disclosed a statistically significant difference between the
reading-while-listening and reading-only groups’reading comprehension scores at
the intermediate-low proficiency level. In particular, the reading-while-listening
group outperformed the reading-only group, indicating that intermediate-level
Chinese language learners better comprehend multimodal texts when reading with
auditory input. In contrast, this study did not find any statistically significant
differences between the reading-while-listening and reading-only groups at the
intermediate-high proficiency level. This implies that at higher proficiency levels,
where Chinese language learners are presumably more competent in the target
language, additional auditory stimulation may not necessarily result in improved
reading comprehension.
While the differences in multimodal reading were identified between the two
proficiency groups, it is important to note that such differences were not well-noted
in previous research focusing on English language learners. Previous studies
suggested that audio inclusion supports multimodal reading comprehension in
general (i.e., Chang and Millett 2015; Conklin et al. 2020; Kim 2021; Wang and
230 Zhang and Peng
Lee 2020), with no differences between learners at different proficiency levels being
reported.
Qualitative results in this study yielded insights to help understand the
above-identified difference in reading in different languages. In general, the effec-
tiveness of incorporating audio input for learners with lower proficiency aligned
with their dominantly positive attitudes towards reading-while-listening mode. They
did not mention any disadvantages, indicating that the positive aspects of audio
integration may outweigh any potential disadvantages among the participants at
lower proficiency levels.
Considering that intermediate-low-level Chinese language learners may have a
limited capacity for identifying Chinese characters, as mentioned by several learners
at the lower proficiency level, adding audio input may help them associate meaning
directly with sounds. This can be further explained by the fact that Chinese char-
acters rely more on form-to-meaning semantic processing than alphabetic writing
systems, whereas their form-to-phonology non-semantic processing is heavily
influenced by para-linguistic context (Wu 2022). Reading Chinese characters thus
requires the dynamic assimilation of multichannel information, such as audio and
visual, and multiple processing levels (ibid.). For intermediate-low proficiency
learners, integrating reading and listening may serve as a critical scaffold that aids in
decoding and comprehending more complex syntactical structures. The auditory
component can provide a model for pronunciation and intonation, which is partic-
ularly beneficial for learners of tonal languages like Chinese, where meaning is often
contingent on pitch patterns (Wang et al. 2018). Therefore, for these learners,
reading-while-listening can enhance immediate comprehension and facilitate
learning new words, ultimately contributing to their reading fluency. Mo and Jiang’s
(2024) findings also hold explanatory power to the findings of this study, highlighting
the potential of reading-while-listening to mitigate the challenges posed by Chinese
orthography for L2 learners, and reduce cognitive load to allow for greater atten-
tional resources to be allocated to comprehension processes.
Conversely, Chinese language learners at the intermediate-high level may not
require audio support because they have developed solid Chinese character recog-
nition skills to process the target texts. In this situation, audio integration does not
significantly aid comprehension. This is in line with the participants’mixed attitudes
towards adding audio input. While some learners found the audio helpful for reading
comprehension, others believed it slowed down their reading pace and caused them
to rely excessively on the audio. This indicates that including audio input could be a
double-edged sword for students with more advanced language proficiency.
Intermediate-high learners might leverage multimodal reading approaches to fine-
tune their comprehension skills and reinforce their grasp of subtle linguistic
nuances. Given their more robust vocabulary and a stronger grasp of grammar, these
Multimodal reading in CSL learners 231
learners are likely to benefit from multimodal reading by consolidating their existing
knowledge and applying it to infer meaning from more challenging texts. This can
reinforce the cognitive process of transitioning from learning to reading to reading
to learning, as the more advanced readers are more capable of engaging with the text
critically and reflectively. The convergence of findings from the present study, Koh
(2024) and Montero Perez (2022) underscores the value of reading-while-listening as
a pedagogical tool for supporting Chinese L2 learners’reading development. How-
ever, as indicated by the mixed attitudes among intermediate-high learners, the
effectiveness of this approach may vary depending on learners’proficiency levels
and individual differences. Future research can further explore these factors to
develop more nuanced guidelines for implementing reading-while-listening in
Chinese L2 instruction.
In sum, this study suggests that learners’proficiency levels play an important
role in processing multimodal texts in reading-only and reading-while-listening
modes, with lower-proficiency learners benefiting from the additional audio input.
Such a difference was not commonly noted in previous studies focusing on English,
and the uniqueness of processing Chinese characters in reading might play a key role
in this divergence in research.
5.2 Catering multimodal teaching to learners of different
proficiency levels
The viewpoints mentioned above highlight the need to consider learners’proficiency
levels while utilizing multimodal learning strategies. The disparity in perspectives
between the two proficiency groups further underscores the necessity for adapt-
ability and personalization in pedagogical approaches to reading instruction to
accommodate individual learners’particular needs and preferences.
Policymakers and educational leaders can use the findings from this research to
advocate for integrating technology and multimedia resources into language-
learning environments. Investing in such resources would enable a more inclusive
and effective language-learning experience that aligns with the diverse needs of
learners at various stages of their language-learning journey. On the other hand,
material developers are encouraged to develop a range of multimodal resources
that cater to different proficiency levels, ensuring that learners have access to
appropriate and challenging materials. This approach to curriculum development
can better support the journey of L2 learners as they advance in their studies.
A one-size-fits-all method is less likely to be effective; instead, pedagogical
strategies should be adaptive to the learners’proficiency stage. When it comes down
to classroom teachers, pedagogical strategies need to be adaptive to the learners’
232 Zhang and Peng
proficiency stages. For learners with lower proficiency, teaching strategies should
emphasize the systematic introduction of multimodal reading with auditory input.
This can involve the gradual introduction of audio support for text reading, starting
with texts that align closely with the learners’linguistic capabilities. The primary
goal is to build a strong foundational understanding of fundamental vocabulary and
sentence structures, using auditory reinforcement to enhance memory and reten-
tion. As learners progress to higher levels of proficiency, instructional strategies
should shift towards fostering analytical and critical reading skills. At this stage,
teachers can employ more complex and content-rich multimodal materials that
challenge learners to synthesize information from multiple sources, including text,
audio, and visual cues. This not only strengthens their multimodal literacy but also
prepares them for real-world scenarios where such skills are increasingly pertinent
(Lim and Toh 2020).
In sum, by acknowledging the nuanced influence of proficiency on the effec-
tiveness of multimodal reading strategies, educators can better support the diverse
needs of L2 Chinese learners. Such an approach has the potential to ultimately
enhance the overall efficacy of language instruction and foster a more inclusive and
adaptive educational environment.
6 Conclusions
The study examined the multimodal reading comprehension of intermediate-low
and intermediate-high-level L2 Chinese language learners in reading-while-listening
and reading-only scenarios. The empirical evidence indicates that using auditory
cues improves reading comprehension for intermediate-low-level Chinese language
learners, highlighting the lower significance of audio input among their
intermediate-high-level counterparts. The findings imply that Chinese language
educators should consider how to use these resources effectively for learners with
different proficiency levels. While agreeing with the current corpus of research
supporting the efficiency of audio inclusion in multimodal reading, this study
expands the scope of comprehension by exploring its application to Chinese lan-
guage learners –a relatively under-researched cohort. The findings further suggest
that learners’proficiency levels are important in leveraging audio inclusion when
reading multimodal texts. The findings imply that Chinese language educators need
to cater their teaching to learners’needs at different proficiency levels.
The study’s primary limitation is the relatively small sample size for the quan-
titative component –studies with a larger number of participants can yield better
validity. Another limitation is that learners’in this study’s proficiency levels fall into
intermediate levels. Considering the level-specific pattern identified in this study,
Multimodal reading in CSL learners 233
future research is encouraged to focus on learners with proficiencies at two
extremes, i.e., novice and advanced learners, to gauge this important issue better.
References
Bisson, Marie-Josée, Walter J. B. Van Heuven, Kathy Conklin & Richard J. Tunney. 2015. The role of verbal
and pictorial information in multimodal incidental acquisition of foreign language vocabulary.
Quarterly Journal of Experimental Psychology 68(7). 1306–1326.
Chang, Anna C.-S. & Sonia Millett. 2014. The effect of extensive listening on developing L2 listening fluency:
Some hard evidence. ELT Journal 68(1). 31–40.
Chang, Anna C.-S. & Sonia Millett. 2015. Improving reading rates and comprehension through audio-
assisted extensive reading for beginner learners. System 52. 91–102.
Choi, Jayoung & Youngjoo Yi. 2015. Teachers’integration of multimodality into classroom practices for
English language learners. TESOL Journal 7(2). 304–327.
Conklin, Kathy, Sara Alotaibi, Ana Pellicer-Sánchez & Laura Vilkaitė-Lozdienė. 2020. What eye-tracking tells
us about reading-only and reading-while-listening in a first and second language. Second Language
Research 36(3). 257–276.
Damayanti, Ika Lestari & Yusnita Febrianti. 2020. Multimodal literacy: Unfolding reading path in children’s
picture book and its potential for EFL classrooms. Indonesian Journal of Applied Linguistics 9(3).
616–627.
Fälth, Linda, Irma Brkovic, Gordana Kerestes, Idor Svensson, Erland Hjelmquist & Tomas Tjus. 2022. The
effects of a multimodal intervention on the reading skills of struggling students: An exploration
across countries. Reading Psychology 44(3). 225–241.
Gilakjani, Abbas Pourhosein, Ismail Hairul Nizam & Seyedeh Masoumeh Ahmadi. 2011. The effect of
multimodal learning models on language teaching and learning. Theory and Practice in Language
Studies 1(10). 1321–1327.
Green, David W. & Jubin Abutalebi. 2013. Language control in bilinguals: The adaptive control hypothesis.
Journal of Cognitive Psychology 25(5). 515–530.
Hung, The Nguyen. 2022. Building and using the system of competence development-oriented exercises
to develop the teaching reading comprehension multimodal texts competence for philology
pedagogical students. VNU Journal of Science: Education Research 38(4). https://doi.org/10.25073/
2588-1159/vnuer.4664.
iChineseReader. https://ichinesereader.com/about (Accessed 13 October 2023).
Kara, Bayram. 2019. The effect of graded readers on reading comprehension skill of EFL students.
Language Teaching and Educational Research 2(2). 160–172.
Kim, Na-Young. 2021. The more, the better? Effects of multiple modalities on EFL listening and reading
comprehension. STEM Journal 22(3). 29–45.
Koh, Joanne.. 2024. Deconstructing the benefits of reading-while-listening on L2 reading comprehension:
The influence of cross-orthographic distance. Foreign Language Annals 57(1). 163–183.
Leibrandt, Isabella. 2022. A multimodal approach to reading language-image-text combinations.
International Journal of New Trends in Social Sciences 6(2). 121–135.
Lim, Fei Victor & Weimin Toh. 2020. How to teach digital reading? Journal of Information Literacy 14(2).
https://doi.org/10.11645/14.2.2701.
234 Zhang and Peng
Miki, Mclean Hy. 2020. Investigating of pre-service teachers’perception and readiness to digital
multimodal literacy at Lakidende University. International Journal of Language Education and Culture
Review 6(1). 62–68.
Mo, Dan & Xin Jiang. 2024. 边听边读”输入对汉语二语学习者阅读的影响—来自眼动的证据[The
effect of auditory input on CSL learners’reading processing: Evidence from eye movements]. 世界汉
语教学[Chinese Teaching in the World] 38(1). 97–110.
Montero Perez, Maribel. 2022. Second or foreign language learning through watching audio-visual input
and the role of on-screen text. Language Teaching 55(2). 163–192.
Moreno, Roxana & Richard Mayer. 2007. Interactive multimodal learning environments: Special issue on
interactive learning environments: Contemporary issues and trends. Educational Psychology Review
19(3). 309–326.
Nurviyani, Vina, Didi Suherdi & Iwa Lukmana. 2020. Developing students’reading skill through making
multimodal inferences. English Review: Journal of English Education 8(2). 11.
Pasquarella, Adrian, Xi Chen, Katie Lam, Yang C. Luo & Gloria Ramirez. 2011. Cross-language transfer of
morphological awareness in Chinese-English bilinguals. Journal of Research in Reading 34(1). 23–42.
Pellicer-Sánchez, Ana, Kathy Conklin, Michael P. H. Rodgers & Fabio Parente. 2021. The effect of auditory
input on multimodal reading comprehension: An examination of adult readers’eye movements. The
Modern Language Journal 105(4). 936–956.
Qiao, Shen, Yingyi Liu & Susanna Siu-sze Yeung. 2021. Mechanisms by which morphological awareness
contributes to Chinese and English reading comprehension. Journal of Research in Reading 44(1).
189–209.
Sacco, Guillaume, Frédéric Noublanche, Frédéric Blazek, Catherine Hue, Loïc Carballido, Asfar Marine,
Philippe Allain & Annweiler Cédric. 2021. How to deal with the consent of adults with cognitive
impairment involved in European geriatric living labs? Philosophy, Ethics, and Humanities in Medicine
16(3). https://doi.org/10.1186/s13010-021-00101-1.
Shen, Helen H. 2004. Level of cognitive processing: Effects on character learning among non-native
learners of Chinese as a foreign language. Language and Education 18(2). 167–182.
Taguchi, Etsuo, Greta J. Gorsuch, Kristin Lems & Rory Rosszell. 2016. Scaffolding in L2 reading: How
repetition and an auditory model help readers. Reading in a Foreign Language 28. 101–117.
Tragant, Elsa, Carmen Muñoz & Nina Spada. 2016. Maximizing young learners’input: An intervention
program. The Canadian Modern Language Review 72(2). 234–257.
Van Vu, Duy. 2023. Incidental collocation learning from different modes of input and factors that affect
learning. Leuven, Belgium: KU Leuven dissertation.
Wang, Dingmin, Yan Song, Jing Li, Jialong Han & Haisong Zhang. 2018. A hybrid approach to automatic
corpus generation for Chinese spelling check. In Proceedings of the 2018 conference on empirical
methods in natural language processing, 2517–2527.
Wang, Shufang & Chang-In Lee. 2020. Exploring the effectiveness of hypermedia glosses for second
language reading. The Journal of Asia TEFL 17(4). 1284–1293.
Webb, Stuart & Anna C.-S. Chang. 2014. Second language vocabulary learning through extensive reading
with audio support: How do frequency and distribution of occurrence affect learning? Language
Teaching Research 19(6). 667–686.
Webb, Stuart & Anna C.-S. Chang. 2020. How does mode of input affect the incidental learning of
collocations? Studies in Second Language Acquisition 44(1). 35–56.
Wong, Yu Ka, Xuan Zang & Tomohiro Inoue. 2023. Promoting foundational linguistic skills for reading
development in young Chinese language learners: A 1-year intervention study. Journal of Research in
Reading 46(3). 247–277.
Multimodal reading in CSL learners 235
Wu, Junru. 2022. Neurocognition in the reading of Chinese characters. Journal of Chinese Writing Systems
6(3). 167–183.
Ye, Lijuan. 2011. Teaching and learning Chinese as a foreign language in the United States: To delay or not to
delay the character introduction. Atlanta, GA: Georgia State University dissertation.
Zhang, Peng. 2021. Learning strategies, motivation and learners’perspectives on online multimodal
Chinese learning. Chinese Language Teaching Methodology and Technology 4(1).
Zhang, Haomin, Xing Zhang, Mengjie Li & Yiming Zhang. 2021. Morphological awareness in L2 Chinese
reading comprehension: Testing of mediating routes. Frontiers in Psychology 12. 736933.
Bionotes
Peng Zhang
Namseoul University, 10218, Humanities and Social Sciences Building, 91 Daehak-ro, Seonghwan-eup,
Seobuk-gu, Cheonan-si, Chungcheongnam-do, South Korea
nsuiben@nsu.ac.kr
Peng Zhang (https://orcid.org/0000-0001-5164-1869) is an educator specializing in language education
and educational technology with a PhD in educational technology from the University of the Balearic
Islands and dual MAs from the University of Barcelona and Université d’Artois. A registered teacher in both
the US and UK, Peng leads workshops for the International Baccalaureate (IB) and serves as Assistant
Professor at NamSeoul University. He also works as an examiner for IBDP, IGCSE, and A-Level, and
contributes as a reviewer to several educational journals.
Yue Peng
Room 404, Ho Tin Building The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China
yuepeng@cuhk.edu.hk
Yue Peng (https://orcid.org/0000-0003-2935-688X, PhD, Queen’s University) is an Assistant Professor in
Chinese language education at the Chinese University of Hong Kong. Her research interests include
teaching and learning Chinese as a second language, particularly in teachers’professional learning,
multilingualism, and language assessment. Her recent publications appeared in journals such as The
Modern Language Journal, Language Teaching Research, Language Testing, Language, Culture and Curriculum,
and Journal of Multilingual and Multicultural Development.
236 Zhang and Peng