ArticlePDF Available

Multimodal reading in reading-only versus reading-while-listening modes: evidence from Chinese language learners

Authors:

Abstract and Figures

Language educators and researchers have shown keen interest in enhancing second language (L2) learners’ reading skills, and recently, more emphasis has been placed on multimodal learning. Multimodal learning theories posit that the confluence of auditory and visual stimuli facilitates learners to establish robust referential connections, and thus integrating audio cues has emerged as a strategy to enhance reading performance. While prior research has affirmed the advantages of incorporating audio cues for fostering reading fluency and comprehension in general, the findings are inconsistent. These investigations have predominantly focused on learners of English as a second language. Considering the differences between English and Chinese, especially their relationship between written scripts and oral language, investigating the reading experiences of Chinese language learners can contribute to a fuller understanding of the effect of audio inclusion on L2 learners. This study compared reading comprehension and reading experiences in reading-only and reading-while-listening modes for intermediate-low and intermediate-high-level Chinese language learners. After learners read in two different modes, a T-test was conducted on their comprehension performance followed by a qualitative analysis of their perspectives on their reading experiences. The findings showed a significant influence of audio integration on enhancing reading comprehension among intermediate-low-level Chinese language learners engaged in multimodal Chinese reading, and these learners displayed a preference for audio inclusion. On the other hand, a lack of significance in the effects of supplementary audio cues on reading comprehension was identified for intermediate-high-level learners, who manifested a mixed attitude towards audio integration with reading. This study complements the existing understanding of multimodal reading based on alphabetic language, particularly in English. Moreover, the findings carry pertinent implications for Chinese language reading pedagogy.
Content may be subject to copyright.
Peng Zhang* and Yue Peng
Multimodal reading in reading-only versus
reading-while-listening modes: evidence
from Chinese language learners
https://doi.org/10.1515/caslar-2024-2003
Abstract: Language educators and researchers have shown keen interest in
enhancing second language (L2) learnersreading skills, and recently, more
emphasis has been placed on multimodal learning. Multimodal learning theories
posit that the conuence of auditory and visual stimuli facilitates learners to
establish robust referential connections, and thus integrating audio cues has
emerged as a strategy to enhance reading performance. While prior research has
armed the advantages of incorporating audio cues for fostering reading uency
and comprehension in general, the ndings are inconsistent. These investigations
have predominantly focused on learners of English as a second language. Consid-
ering the dierences between English and Chinese, especially their relationship
between written scripts and oral language, investigating the reading experiences of
Chinese language learners can contribute to a fuller understanding of the eect of
audio inclusion on L2 learners. This study compared reading comprehension and
reading experiences in reading-only and reading-while-listening modes for
intermediate-low and intermediate-high-level Chinese language learners. After
learners read in two dierent modes, a T-test was conducted on their comprehension
performance followed by a qualitative analysis of their perspectives on their reading
experiences. The ndings showed a signicant inuence of audio integration on
enhancing reading comprehension among intermediate-low-level Chinese language
learners engaged in multimodal Chinese reading, and these learners displayed a
preference for audio inclusion. On the other hand, a lack of signicance in the eects
of supplementary audio cues on reading comprehension was identied for
intermediate-high-level learners, who manifested a mixed attitude towards audio
integration with reading. This study complements the existing understanding of
multimodal reading based on alphabetic language, particularly in English. Moreover,
the ndings carry pertinent implications for Chinese language reading pedagogy.
*Corresponding author: Peng Zhang, Namseoul University, 10218, Humanities and Social Sciences
Building, 91 Daehak-ro, Seonghwan-eup, Seobuk-gu, Cheonan-si, Chungcheongnam-do, South Korea,
E-mail: nsuiben@nsu.ac.kr
Yue Peng, Room 404, Ho Tin Building The Chinese University of Hong Kong, Shatin, Hong Kong SAR,
China, E-mail: yuepeng@cuhk.edu.hk
CASLAR 2024; 13(2): 215236
Keywords: audio integration; multimodal learning; reading comprehension;
reading-while-listening
:,育者
,论认,
视觉,,
,不一,
,
了中
,T他们
,,于中
,了他们,
明显于中,线
,
(),
:;;;
1 Introduction
The development of reading abilities is a fundamental aspect of second language (L2)
acquisition since attaining uent reading skills is crucial for achieving a higher level
of L2 prociency in general (Kara 2019). In recent educational practices, there has
been a shift towards employing multimodal strategies to support the development
of L2 reading competencies (Zhang 2021). Multimodal learning, which involves
activating multiple sensory channels such as visual and auditory streams, is
increasingly recognized for its role in enhancing the educational experience and the
capacity for learning across dierent subject areas, including language acquisition
(Choi and Yi 2015; Zhang 2021).
Integrating multimodal elements into reading activities, specically, has been
identied as a promising approach to improving reading uency and comprehen-
sion. Engaging multiple senses by combining reading with auditory support, for
instance, has been shown to facilitate a deeper connection with the text and improve
retention, thereby enriching the learning process (Tragant et al. 2016). The added
value of multimodal reading practices, such as reading-while-listening, has been
substantiated in various studies, with ndings indicating notable benets in terms of
216 Zhang and Peng
literacy development and reading skill enhancement (Fälth et al. 2022; Nurviyani
et al. 2020).
While prior research has largely armed the dierent levels of advantages of
incorporating audio cues for fostering reading uency and comprehension, these
investigations have predominantly focused on learners of English as a second lan-
guage (Bisson et al. 2015; Chang and Millett 2014, 2015; Conklin et al. 2020; Tragant
et al. 2016; Webb and Chang 2014, 2020). The unique characteristics of the Chinese
writing system, including its logographic nature and the incorporation of radicals
within a complex two-dimensional structure, present distinct challenges for L2
learners, particularly those with a background in alphabetic languages (Shen 2004).
Additionally, the phonetic and semantic nuances of the Chinese language add layers
of complexity for learners attempting to navigate this system (Ye 2011). Considering
the dierences between English and Chinese, especially the relationship between
written scripts and oral language, investigating the reading experiences of Chinese
language learners can contribute to a fuller understanding of the eect of audio
inclusion on L2 learners. Addressing the above-noted research gap, this study
compared reading comprehension and reading experiences in reading-only and
reading-while-listening modes for intermediate-low and intermediate-high-level
Chinese language learners.
2 Literature review
2.1 Multimodal learning and multimodal reading
Multimodal learning, which underpins the concept of multimodal reading, is pred-
icated on the activation of various sensory modes during the learning process,
including visual, auditory, kinesthetic, text, and technological inputs (Choi and Yi
2015; Gilakjani et al. 2011; Moreno and Mayer 2007). This approach to learning
acknowledges the diversity of individual learning styles and preferences, positing
that the engagement of multiple senses can enhance information processing and
retention (Green and Abutalebi 2013). Evidence suggested that such an approach not
only enriched the educational experience for students (Choi and Yi 2015) but also
amplied their learning capacity and potential (Zhang 2021). Furthermore, Sacco
et al. (2021) contended that the ecacy of learning is bolstered by multimodal pre-
sentations, indicating a broader scope for improving educational outcomes through
this method.
Multimodal reading, which involves the interpretation of various modes such as
visual, audio, and spatial elements, has gained signicant attention in educational
research. The use of multimodal texts in teaching and learning has been shown to
Multimodal reading in CSL learners 217
enhance studentsliteracy skills and comprehension (Damayanti and Febrianti 2020;
Fälth et al. 2022). Educators have recognized the importance of developing multi-
modal literacy in students, particularly in the context of digital reading (Lim and Toh
2020; Miki 2020). This is supported by the ndings that a multimodal reading training
program had positive eects on studentsliteracy development, including decoding,
spelling, and reading comprehension (Fälth et al. 2022). Furthermore, the use of
multimodal texts in the classroom has been found to be eective in improving
studentsreading skills signicantly (Nurviyani et al. 2020).
The multimodal reading process entails integrating information from various
sources and establishing correlations between several modalities to construct
meaning (Pellicer-Sánchez et al. 2021). Leibrandt (2022) suggested that multimodal
reading competence encompasses several key components, including attributing
meaning to visual representations, comprehending the interplay between written
language and visual messages, and eectively combining semantically enriched
language with contextualized imagery. For reading literary text, for example,
Leibrandt noted that readers can enhance their understanding of a piece of litera-
ture by ascribing meaningful interpretations to visuals within the literary frame-
work, including visual messages that complement the written content and
combining language with contextualized imagery.
Given the advantages of multimodal reading, supporting learners with skills in
reading multimodal texts has grown in signicance within educational contexts. It is
imperative for students to possess the skills necessary to navigate and comprehend
the various types of multimodal texts they come across eectively (Fälth et al. 2022;
Hung 2022; Lim and Toh 2020).
2.2 Reading-while-listening in L2 reading
The reading-while-listening approach is a method that integrates both the processes
of reading and listening in the context of second language acquisition. As a type of
multimodal reading, the role of audio inclusion in visual reading has been exten-
sively investigated in academic research. In general, studies provided evidence to
support the idea that engaging in simultaneous reading-while-listening activities
provides greater benets for L2 acquisition compared to engaging in reading-only
activities (Bisson et al. 2015; Chang and Millett 2014, 2015; Conklin et al. 2020; Tragant
et al. 2016; Webb and Chang 2014, 2020; Van Vu 2023).
Among others, the advantage of reading-while-listening on reading compre-
hension is most pertinent to the focus of our study. Chang and Millett (2015) exam-
ined reading-only and reading-while-listening modes among a sample of English as a
second language (ESL) teenage learners in Taiwan. They found improvements in
218 Zhang and Peng
reading speed and comprehension for both groups. Notably, the reading-while-
listening group demonstrated superior performance in reading comprehension.
In their study, reading-while-listening led to higher levels of comprehension and
concentration, making the input more engaging for students. Conklin et al. (2020)
focused on eye movement during reading and found that reading-while-listening
inuences where the eyes move on the page, showing a consistent pattern regardless
of whether reading in a rst language (L1) or L2. This consistency has suggested
that reading-while-listening taps into similar mental processes active in separate
reading or listening, which might help explain why comprehension and retention
are improved with reading-while-listening. In addition to enhancing reading
comprehension, other advantages brought by reading-while-listening have been
acknowledged, such as fostering vocabulary acquisition. The study conducted by
Bisson et al. (2015) showed that incorporating audio with visual and textual com-
ponents appeared advantageous in facilitating incidental vocabulary acquisition.
Similarly, Webb and Chang (2014) also found evidence of more unintentional vo-
cabulary learning when reading-while-listening than in reading-only activities. In
sum, while the impact of reading-while-listening on L2 reading comprehension has
been extensively studied, generally in favor of incorporating auditory input for
improving reading comprehension, the current literature has primarily focused on
English language learners with other L2s neglected.
2.3 Reading in the Chinese language
Reading comprehension is vital in the acquisition of the Chinese language, particu-
larly for learners who are not native speakers. Research has illuminated various
factors that inuence reading comprehension among Chinese learners. For example,
the signicance of morphological awareness, especially among bilingual individuals,
was emphasized by studies such as those conducted by Pasquarella et al. (2011) and
Qiao et al. (2021), highlighting its substantial contribution to Chinese reading
comprehension. Specically, Qiao et al. (2021) emphasized the contribution of
morphological awareness to reading comprehension in both English and Chinese,
indicating that the inclusion of morphological awareness in reading instruction
could enhance vocabulary, word reading, and reading comprehension. Additionally,
decoding, the ability to recognize characters and words, is another pivotal factor in
Chinese reading comprehension, as identied by Wong et al. (2023). Zhang et al.
(2021) drew attention to the intermediary role of lexical inference in linking
morphological awareness with reading comprehension in L2 Chinese learners.
These research ndings supported the notion that audio inclusion, which would
Multimodal reading in CSL learners 219
potentially aid learnersmorphological awareness through pronunciation and word
recognition, can enhance their Chinese reading comprehension.
In reading Chinese, the inclusion of audio in learning materials also emerged
as an intriguing area for exploration with its potential eects on reading compre-
hension. A recent eye-tracking-based research by Mo and Jiang (2024) provided
empirical evidence for the benecial eects of reading-while-listening on Chinese L2
learnersreading processes. Their study found that reading-while-listening led to
more ecient reading, with learners exhibiting fewer xations, shorter xation
durations, and reduced regressions compared to reading-only. Elsewhere, Kohs
(2024) study has shed light on how cross-linguistic orthographic distance impacted
the eectiveness of reading-while-listening for L2 reading comprehension. Exam-
ining intermediate Korean learners of Chinese, English, and Spanish, Koh found that
learners studying languages with greater orthographic dierences from their L1
Korean, such as Chinese, benetted more from reading-while-listening. Kohs
research ndings suggested that reading-while-listening can be particularly useful
when learners face signicant orthographic disparities between their L1 and L2.
While these studies have provided intriguing insights into the impact of incor-
porating audio to reading, the research conducted on its eect on L2 Chinese reading
comprehension remains comparatively limited. Given the scarcity of empirical
evidence, despite some indications of the benets of incorporating audio in Chinese
reading comprehension, further research is needed to gain a comprehensive
understanding of this issue.
Considering the advantages of multimodal reading strategies, especially the
simultaneous engagement in reading and listening, and recognizing the gap in
research regarding the impact of this approach on L2 Chinese learners of dierent
prociency levels, particularly within the context of online multimodal reading
environments, this study seeks to investigate the following two research questions:
RQ1: To what extent does the reading-while-listening mode enhance reading comprehension for
Chinese language learners at intermediate-low and intermediate-high levels, respectively?
RQ2: How do learners in two prociency groups perceive their experiences of reading in the two
dierent modes?
3 Methodology
The study adopted a mixed-methods approach. By including both quantitative and
qualitative data, this study aimed to gain a comprehensive understanding of the
impact of dierent reading modes on Chinese language learnersreading
220 Zhang and Peng
comprehension. Specically, the quantitative data and analyses provided statistical
evidence regarding the inuence of audio input on studentsperformance in
multimodal reading, addressing RQ1. The qualitative analysis, on the other hand,
allowed for an exploration of participantsperspectives of their reading experiences
in the two modes, addressing RQ2. Overall, qualitative data provided insights into
possible explanations of the patterns identied in RQ1.
3.1 Participants
The study sample comprised 66 secondary school students enrolled in Grades 7
through 10 at an international school in Hong Kong. Their participation was
entirely voluntary and was sanctioned by the necessary consent from the schools
governing body and the students legal guardians. These individuals ranged in
age from 12 to 16 years and were all undertaking Mandarin Chinese courses as a
second language. The student demographic was characterized by a middle-class
upbringing, with English and Cantonese being the top two predominant home
languages, respectively, though not all students spoke both languages at home.
Notably, Mandarin was not commonly used in their home environments. The
educational context for these participants was an international school setting
where English was the primary language of instruction for all subjects other than
Chinese.
For the follow-up focus group interviews, eight students from the intermediate-
low group and eight students from the intermediate-high group volunteered to
participate. Within each group, half of the students experienced reading-only mode,
while the other half experienced reading-while-listening mode.
3.2 Materials
The study employed an online multimodal Chinese reading platform called iChine-
seReader. iChineseReader is a graded multimodal reading resource designed to meet
the reading needs of K-12 Chinese language programs, and is used in Chinese
language programs in immersion, bilingual, and international schools worldwide.
The platform oers over three thousand interactive mini e-books/articles catego-
rized into twelve distinct reading prociency tiers, covering novice, intermediate,
and advanced levels, with reference to ACTFLs scale. All e-books accessible through
this portal contain reading materials supplemented with colorful and dynamic
visuals related to the content, as shown in Figure 1, providing readers with
Multimodal reading in CSL learners 221
multimodal reading experiences. The visuals were included for both groups in this
study as part of the multimodal reading materials, and thus were not treated as a
variable between the groups.
In addition to visual input, teachers can set up various reading modes on iChi-
neseReader, such as reading-only and reading-while-listening, allowing for adapt-
ability to learnersindividual preferences. The audio recordings of the texts were
narrated by professional voice talents in standard Mandarin at a speech rate of
130165 Chinese characters per minute. It is noted that the reading materials that the
intermediate-high level group read were slightly faster than the intermediate-low
group. But the speed of the audio within each group remains the same. In general,
this feature-rich platform aligns well with the focus of this study.
3.3 Data collection
Assigning learners into two prociency levels. The rst step of the study was to gauge
the participantsChinese language prociency, particularly with regard to reading
skills. To achieve this purpose, a benchmark test was rst administered on the
iChineseReader platform. This self-adaptive test gave participants passages and
articles to read, followed by questions of multiple choices, yes-no questions, and
Figure 1: Excerpt of multimodal reading materials in iChineseReader.
222 Zhang and Peng
short answers to assess their reading comprehension. Based on studentsanswers,
the test adjusted the diculty of reading materials and questions automatically.
Participants completed the benchmark test in 2035 min.
Against the 20-point-scale prociency levels, learners were assigned by the
testing system to a specic level between 9 and 17. Students were then organized into
groups based on these levels: those at levels 912 were classied as intermediate-low,
while those at levels 1417 were classied as intermediate-high. Notably, there were
three participants assigned to level 13 directly in the center of the intermediate
range and another three to level 19, which is considered advanced. These six
students were then excluded from the study. To ensure the accuracy of this classi-
cation, four classroom teachers, familiar with the studentsreading skills, were
asked to review and validate the groupings. Their insights conrmed the initial
placement, necessitating minimal adjustments by the teachers. Following this
validation process, the nal participant distribution consisted of 34 in the
intermediate-low group and 32 in the intermediate-high group.
Reading in two modes. Next, each group of readers was given four articles to
read: two in reading-only mode and two in reading-while-listening mode. The order
of the texts was randomized to control for order eects. All reading materials were
taken from iChineseReaders collections, with materials for lower prociency
levels chosen from levels 913, and material for higher prociency levels taken from
levels 1419. The articles were presented to students one at a time to avoid testing
memory in addition to comprehension. Upon completing the assigned reading,
participants were instructed to undertake a comprehension test to evaluate their
understanding of the reading materials before moving on to the next text. Each
comprehension test included 10 questions in total, with ve multiple choice ques-
tions, three true-or-false questions, and two short answer questions. The reading
comprehension questions were evenly spread across the four texts, with ve
multiple choice questions generated by iChineseReader, and another ve questions
created by the researcher. The participants completed the reading followed by
reading comprehension questions within 45 min of class.
Focus group interviews. Upon completing the reading comprehension test, two
focus group interviews were conducted separately. The participants were invited to
reect on their experiences of reading in the two distinct reading modes and how
the reading modes had an impact on their reading comprehension. The focus
groups were conducted in English, lasting for about 15 min on average. Open-ended
questions were asked to elicit studentsreections on their reading experiences in
the two modes and how the modes impacted their reading comprehension. The
interviews were audio-recorded and later transcribed for analysis.
Multimodal reading in CSL learners 223
3.4 Data analysis
The quantitative aspect of the data analysis involved the examination of reading
comprehension scores obtained from participants under two dierent conditions:
reading-while-listening and reading-only. The collected data were entered into the
Statistical Package for Social Sciences (SPSS) software for analysis. Two-sample t-tests
were employed to determine if there were statistically signicant dierences in
reading comprehension performance between the two instructional modes across
the two prociency levels. The t-tests allowed us to compare the mean scores of the
intermediate-low prociency group and the intermediate-high prociency group,
providing a clear understanding of how each prociency level responded to the
reading conditions.
For the qualitative segment, the participantsattitudes were explored along
with the reading conditions. The transcriptions of the focus group interview data
were incorporated to NVivo 14, a software for qualitative data analysis. The rst step
in this analysis involved conducting an attitudinal analysis towards the reading
modes based on the participantsreections by using NVivos sentiment analysis
function. The responses were coded using an attitudinal scale that included cate-
gories such as moderately positive, very positive, moderately negative, and very
negative. The categorization was based on the strength of the language used by
participants, with descriptors such as very,greatly,signicantly,a lot, and
superclassied as strongly positive or negative. Language such as rather,some,
and without any modiers was classied as moderately positive or negative.
4 Findings
The results of the t-test and the qualitative analysis are presented in this section, with
a focus on the two research questions respectively.
4.1 RQ1: the eect of audio inclusion on reading
comprehension
Table 1 displays the results of the t-test conducted for the intermediate-low pro-
ciency level Chinese language learners. The results of the two-sample t-test show that
there is a statistically signicant dierence between the reading-while-listening
group (mean = 16.70, standard deviation = 1.99) and the reading-only group
(mean = 14, standard deviation = 2.92) when it comes to reading comprehension
224 Zhang and Peng
scores for Chinese language learners with an intermediate-low level of prociency.
The t-test indicates that the reading-while-listening mode was associated with higher
reading comprehension scores compared to the reading-only mode. Furthermore,
the p-values for both one-tailed and two-tailed tests were well below the signicance
level of 0.05, further supporting the observed dierence.
In contrast, as depicted in Table 2, the results of the two-sample t-test demon-
strated that there was no statistically signicant dierence in the mean values
between the reading-while-listening group (mean = 16.75, standard deviation = 1.98)
and the reading-only group (mean = 16.38, standard deviation = 2.03) for the
Table :Two-sample assuming unequal variances T-test: intermediate-low.
RWL RO
Mean . 
Variance .
SD . .
Observation  
Hypothesized mean dierence
Df 
tStat .
P(T<=t) one-tail .
tCritical one-tail .
P(T<=t) two-tail .
tCritical two-tail .
Table :Two-sample assuming unequal variances T-test: intermediate-high.
RWL RO
Mean . .
Variance . .
SD . .
Observation  
Hypothesized mean dierence
Df 
tStat .
P(T<=t) one-tail .
tCritical one-tail .
P(T<=t) two-tail .
tCritical two-tail .
Multimodal reading in CSL learners 225
intermediate-high prociency group (p= 0.6). This nding implies that, at the
intermediate-high level of prociency, both reading approaches have comparable
outcomes in improving reading comprehension. The results for the intermediate-
high prociency level imply that there was no statistically signicant dierence in
the reading comprehension scores between the reading-while-listening and reading-
only modes.
4.2 RQ 2: learnersexperiences on reading in the two modes
LearnersAttitudes. Derived from studentswritten reections, the qualitative data
revealed dierent attitudes towards the two reading modes between the two pro-
ciency groups. Those attitudes are exhibited in hierarchy charts, generated by
NVivo, and are used as a visual representation that organizes and displays data nodes
and themes in a structured manner. The size of each section in the hierarchy charts
represents the relative frequency coded references within each theme.
Figures 2 and 3 display the results of the attitudinal analysis for the
intermediate-high and intermediate-low groups respectively.
The analysis from the intermediate-high group revealed mixed attitudes
towards the audio incorporation in their reading process, as shown in Figure 2. There
were six very negative, ve moderately negative, and ve moderately positive
references.
Figure 2: Hierarchy chart of attitudinal codes: intermediate-high.
226 Zhang and Peng
In contrast to the intermediate-high group, the analysis from the intermediate-
low group revealed positive attitudes predominantly towards using multimodal
input in Chinese language learning, as shown in Figure 3. Specically, there were 15
very positive and two moderately positive references.
Following the initial classication of readersattitudes, the dataset was further
analyzed to identify specic themes related to the advantages and disadvantages of
including audio in the reading process. Figures 4 and 5 present the synthesized pros
and cons mentioned by the intermediate-high and intermediate-low groups
respectively. For the intermediate-high group as shown in Figure 4, the main
perceived advantages were improved understanding and concentration, while the
disadvantages included slower reading speed and over-reliance on audio. For the
intermediate-low group as shown in Figure 5, the primary benets were enhanced
understanding, suitability for lower prociency levels, and assistance with character
recognition. No disadvantages were mentioned by this group.
Perceived advantages and disadvantages. The advantages and disadvantages of
audio inclusion, as identied by intermediate-high-level learners, are summarized
in Figure 4. The results revealed that most of the students believed that audio
inclusion boosted their understanding of the reading material. One of the reasons
that the learners cited was that audio input helped with their concentration. For
example, S03 mentioned, I like to read with the audio. I think it helps me under-
stand. I just like to listen to it to be more concentrated. Similarly, students such as
S02 and S05 found the audio inclusion helpful, especially when they were tired or
needed to concentrate more on reading. S02 reected:
Figure 3: Hierarchy chart of attitudinal codes (intermediate-low).
Multimodal reading in CSL learners 227
I might choose to listen to the audio when I am tired and dont want to look at the texts. If I focus
on reading without sound, Ill probably read faster and more eciently. The audio is good for
me to understand, but it is not necessary.
At the same time, some students found disadvantages in the audio inclusion, stating
that it intervened with their reading speed. For instance, S06 reected, It stopped
me from reading fast, I can read better quietly.Similarly, S8 mentioned,
Figure 4: Hierarchy chart of synthesized pros and cons of audio inclusion (intermediate-high).
Figure 5: Hierarchy chart of synthesized pros and cons of audio inclusion (intermediate-low).
228 Zhang and Peng
iChineseReader reads too slowly; it even stopped to go to the next page because it
read too slowly. I want to read by myself without the reading voice.The participants
also recognized that they sometimes over-relied on the audio and did not read
the texts properly, which further negatively inuenced their Chinese character
recognition. S01 admitted:
I sometimes just listen to the audio without looking at the texts since listening is easier than
reading Chinese characters for me this is not good, and I need to practice my text-reading, but
sometimes I just cant help but reply to the listening.
In addition, the participants also perceived that the reading-while-listening mode
might be more suitable for lower grades and for those with lower prociency levels.
S4 and S7 argued that they preferred reading with audio while using Chinese when
they were in lower grades and had lower Chinese skills before, and now they prefer
reading independently without listening.
As for the intermediate-low group, the perceived benets and drawbacks of
audio inclusion are summarized in Figure 5. In general, the results uncovered that
the students found reading-while-listening to be highly benecial. The majority
indicated that reading-while-listening mode boosted their understanding. For
example, S10 mentioned, The audio in reading kind of helps me with understanding
because I can use both my ears and eyes while reading; its more ecient.Some
learners believed that audio inclusion was suitable for those with lower prociency
levels. S12, S13, and S09 stated that their Chinese is not as good as other classmates,
and they felt that the audio in iChineseReader is particularly helpful for them
because they could refer to two channels of information instead of just reading the
words. Notably, this group mentioned no disadvantages.
It is important to note that the benet of understanding Chinese characters
provided by the audio input was a major reason that learners identied it as a
support to their reading comprehension. For example, S14 stated:
It helped me a lot because there are lots of words that I will not understand from just looking at
them, and having the sound of the words, in my reading, reminds me of the meaning of the
words. So, I can understand better.
In addition, S11 reected, I totally depend on the sound since it is dicult for me
to recognize all of the Chinese characters. The audio helps me read faster and
understand the things that I read better.
At the same time, some students pointed out the relativity between their reading
ability and the reading materials when appreciating the audio input. S09 and S13
found the audio inclusion helpful when the reading materials were at or above their
prociency level. For example, S13 stated:
Multimodal reading in CSL learners 229
If I read something very easy that I can totally understand without any problems or something
I learned, the audio might not be necessary. However, if I read iChineseReader that is at or
above my level, I always like to choose the audio reading version to help me understand better.
5 Discussion
In this section, quantitative and qualitative results are integrated. Specically,
qualitative data on learnersreading experiences is incorporated to make sense of
the quantitative results on the two groups of learnersperformance in multimodal
reading comprehension. Theoretical signicance in relation to existing studies is
addressed, followed by oering pedagogical implications.
5.1 The factor of prociency level to multimodal reading
In understanding the inuence of multimodal reading strategies on Chinese
language learners, prociency levels emerged as a pivotal factor. In this study,
prociency signicantly aected how multimodal reading was processed, and the
benets arising from it. This nding is in line with previous research, which
underscored that learners at dierent stages of language acquisition might react to
multimodal inputs in varied ways (Taguchi et al. 2016).
The quantitative data disclosed a statistically signicant dierence between the
reading-while-listening and reading-only groupsreading comprehension scores at
the intermediate-low prociency level. In particular, the reading-while-listening
group outperformed the reading-only group, indicating that intermediate-level
Chinese language learners better comprehend multimodal texts when reading with
auditory input. In contrast, this study did not nd any statistically signicant
dierences between the reading-while-listening and reading-only groups at the
intermediate-high prociency level. This implies that at higher prociency levels,
where Chinese language learners are presumably more competent in the target
language, additional auditory stimulation may not necessarily result in improved
reading comprehension.
While the dierences in multimodal reading were identied between the two
prociency groups, it is important to note that such dierences were not well-noted
in previous research focusing on English language learners. Previous studies
suggested that audio inclusion supports multimodal reading comprehension in
general (i.e., Chang and Millett 2015; Conklin et al. 2020; Kim 2021; Wang and
230 Zhang and Peng
Lee 2020), with no dierences between learners at dierent prociency levels being
reported.
Qualitative results in this study yielded insights to help understand the
above-identied dierence in reading in dierent languages. In general, the eec-
tiveness of incorporating audio input for learners with lower prociency aligned
with their dominantly positive attitudes towards reading-while-listening mode. They
did not mention any disadvantages, indicating that the positive aspects of audio
integration may outweigh any potential disadvantages among the participants at
lower prociency levels.
Considering that intermediate-low-level Chinese language learners may have a
limited capacity for identifying Chinese characters, as mentioned by several learners
at the lower prociency level, adding audio input may help them associate meaning
directly with sounds. This can be further explained by the fact that Chinese char-
acters rely more on form-to-meaning semantic processing than alphabetic writing
systems, whereas their form-to-phonology non-semantic processing is heavily
inuenced by para-linguistic context (Wu 2022). Reading Chinese characters thus
requires the dynamic assimilation of multichannel information, such as audio and
visual, and multiple processing levels (ibid.). For intermediate-low prociency
learners, integrating reading and listening may serve as a critical scaold that aids in
decoding and comprehending more complex syntactical structures. The auditory
component can provide a model for pronunciation and intonation, which is partic-
ularly benecial for learners of tonal languages like Chinese, where meaning is often
contingent on pitch patterns (Wang et al. 2018). Therefore, for these learners,
reading-while-listening can enhance immediate comprehension and facilitate
learning new words, ultimately contributing to their reading uency. Mo and Jiangs
(2024) ndings also hold explanatory power to the ndings of this study, highlighting
the potential of reading-while-listening to mitigate the challenges posed by Chinese
orthography for L2 learners, and reduce cognitive load to allow for greater atten-
tional resources to be allocated to comprehension processes.
Conversely, Chinese language learners at the intermediate-high level may not
require audio support because they have developed solid Chinese character recog-
nition skills to process the target texts. In this situation, audio integration does not
signicantly aid comprehension. This is in line with the participantsmixed attitudes
towards adding audio input. While some learners found the audio helpful for reading
comprehension, others believed it slowed down their reading pace and caused them
to rely excessively on the audio. This indicates that including audio input could be a
double-edged sword for students with more advanced language prociency.
Intermediate-high learners might leverage multimodal reading approaches to ne-
tune their comprehension skills and reinforce their grasp of subtle linguistic
nuances. Given their more robust vocabulary and a stronger grasp of grammar, these
Multimodal reading in CSL learners 231
learners are likely to benet from multimodal reading by consolidating their existing
knowledge and applying it to infer meaning from more challenging texts. This can
reinforce the cognitive process of transitioning from learning to reading to reading
to learning, as the more advanced readers are more capable of engaging with the text
critically and reectively. The convergence of ndings from the present study, Koh
(2024) and Montero Perez (2022) underscores the value of reading-while-listening as
a pedagogical tool for supporting Chinese L2 learnersreading development. How-
ever, as indicated by the mixed attitudes among intermediate-high learners, the
eectiveness of this approach may vary depending on learnersprociency levels
and individual dierences. Future research can further explore these factors to
develop more nuanced guidelines for implementing reading-while-listening in
Chinese L2 instruction.
In sum, this study suggests that learnersprociency levels play an important
role in processing multimodal texts in reading-only and reading-while-listening
modes, with lower-prociency learners beneting from the additional audio input.
Such a dierence was not commonly noted in previous studies focusing on English,
and the uniqueness of processing Chinese characters in reading might play a key role
in this divergence in research.
5.2 Catering multimodal teaching to learners of dierent
prociency levels
The viewpoints mentioned above highlight the need to consider learnersprociency
levels while utilizing multimodal learning strategies. The disparity in perspectives
between the two prociency groups further underscores the necessity for adapt-
ability and personalization in pedagogical approaches to reading instruction to
accommodate individual learnersparticular needs and preferences.
Policymakers and educational leaders can use the ndings from this research to
advocate for integrating technology and multimedia resources into language-
learning environments. Investing in such resources would enable a more inclusive
and eective language-learning experience that aligns with the diverse needs of
learners at various stages of their language-learning journey. On the other hand,
material developers are encouraged to develop a range of multimodal resources
that cater to dierent prociency levels, ensuring that learners have access to
appropriate and challenging materials. This approach to curriculum development
can better support the journey of L2 learners as they advance in their studies.
A one-size-ts-all method is less likely to be eective; instead, pedagogical
strategies should be adaptive to the learnersprociency stage. When it comes down
to classroom teachers, pedagogical strategies need to be adaptive to the learners
232 Zhang and Peng
prociency stages. For learners with lower prociency, teaching strategies should
emphasize the systematic introduction of multimodal reading with auditory input.
This can involve the gradual introduction of audio support for text reading, starting
with texts that align closely with the learnerslinguistic capabilities. The primary
goal is to build a strong foundational understanding of fundamental vocabulary and
sentence structures, using auditory reinforcement to enhance memory and reten-
tion. As learners progress to higher levels of prociency, instructional strategies
should shift towards fostering analytical and critical reading skills. At this stage,
teachers can employ more complex and content-rich multimodal materials that
challenge learners to synthesize information from multiple sources, including text,
audio, and visual cues. This not only strengthens their multimodal literacy but also
prepares them for real-world scenarios where such skills are increasingly pertinent
(Lim and Toh 2020).
In sum, by acknowledging the nuanced inuence of prociency on the eec-
tiveness of multimodal reading strategies, educators can better support the diverse
needs of L2 Chinese learners. Such an approach has the potential to ultimately
enhance the overall ecacy of language instruction and foster a more inclusive and
adaptive educational environment.
6 Conclusions
The study examined the multimodal reading comprehension of intermediate-low
and intermediate-high-level L2 Chinese language learners in reading-while-listening
and reading-only scenarios. The empirical evidence indicates that using auditory
cues improves reading comprehension for intermediate-low-level Chinese language
learners, highlighting the lower signicance of audio input among their
intermediate-high-level counterparts. The ndings imply that Chinese language
educators should consider how to use these resources eectively for learners with
dierent prociency levels. While agreeing with the current corpus of research
supporting the eciency of audio inclusion in multimodal reading, this study
expands the scope of comprehension by exploring its application to Chinese lan-
guage learners a relatively under-researched cohort. The ndings further suggest
that learnersprociency levels are important in leveraging audio inclusion when
reading multimodal texts. The ndings imply that Chinese language educators need
to cater their teaching to learnersneeds at dierent prociency levels.
The studys primary limitation is the relatively small sample size for the quan-
titative component studies with a larger number of participants can yield better
validity. Another limitation is that learnersin this studys prociency levels fall into
intermediate levels. Considering the level-specic pattern identied in this study,
Multimodal reading in CSL learners 233
future research is encouraged to focus on learners with prociencies at two
extremes, i.e., novice and advanced learners, to gauge this important issue better.
References
Bisson, Marie-Josée, Walter J. B. Van Heuven, Kathy Conklin & Richard J. Tunney. 2015. The role of verbal
and pictorial information in multimodal incidental acquisition of foreign language vocabulary.
Quarterly Journal of Experimental Psychology 68(7). 13061326.
Chang, Anna C.-S. & Sonia Millett. 2014. The eect of extensive listening on developing L2 listening uency:
Some hard evidence. ELT Journal 68(1). 3140.
Chang, Anna C.-S. & Sonia Millett. 2015. Improving reading rates and comprehension through audio-
assisted extensive reading for beginner learners. System 52. 91102.
Choi, Jayoung & Youngjoo Yi. 2015. Teachersintegration of multimodality into classroom practices for
English language learners. TESOL Journal 7(2). 304327.
Conklin, Kathy, Sara Alotaibi, Ana Pellicer-Sánchez & Laura Vilkaitė-Lozdienė. 2020. What eye-tracking tells
us about reading-only and reading-while-listening in a rst and second language. Second Language
Research 36(3). 257276.
Damayanti, Ika Lestari & Yusnita Febrianti. 2020. Multimodal literacy: Unfolding reading path in childrens
picture book and its potential for EFL classrooms. Indonesian Journal of Applied Linguistics 9(3).
616627.
Fälth, Linda, Irma Brkovic, Gordana Kerestes, Idor Svensson, Erland Hjelmquist & Tomas Tjus. 2022. The
eects of a multimodal intervention on the reading skills of struggling students: An exploration
across countries. Reading Psychology 44(3). 225241.
Gilakjani, Abbas Pourhosein, Ismail Hairul Nizam & Seyedeh Masoumeh Ahmadi. 2011. The eect of
multimodal learning models on language teaching and learning. Theory and Practice in Language
Studies 1(10). 13211327.
Green, David W. & Jubin Abutalebi. 2013. Language control in bilinguals: The adaptive control hypothesis.
Journal of Cognitive Psychology 25(5). 515530.
Hung, The Nguyen. 2022. Building and using the system of competence development-oriented exercises
to develop the teaching reading comprehension multimodal texts competence for philology
pedagogical students. VNU Journal of Science: Education Research 38(4). https://doi.org/10.25073/
2588-1159/vnuer.4664.
iChineseReader. https://ichinesereader.com/about (Accessed 13 October 2023).
Kara, Bayram. 2019. The eect of graded readers on reading comprehension skill of EFL students.
Language Teaching and Educational Research 2(2). 160172.
Kim, Na-Young. 2021. The more, the better? Eects of multiple modalities on EFL listening and reading
comprehension. STEM Journal 22(3). 2945.
Koh, Joanne.. 2024. Deconstructing the benets of reading-while-listening on L2 reading comprehension:
The inuence of cross-orthographic distance. Foreign Language Annals 57(1). 163183.
Leibrandt, Isabella. 2022. A multimodal approach to reading language-image-text combinations.
International Journal of New Trends in Social Sciences 6(2). 121135.
Lim, Fei Victor & Weimin Toh. 2020. How to teach digital reading? Journal of Information Literacy 14(2).
https://doi.org/10.11645/14.2.2701.
234 Zhang and Peng
Miki, Mclean Hy. 2020. Investigating of pre-service teachersperception and readiness to digital
multimodal literacy at Lakidende University. International Journal of Language Education and Culture
Review 6(1). 6268.
Mo, Dan & Xin Jiang. 2024. [The
eect of auditory input on CSL learnersreading processing: Evidence from eye movements].
[Chinese Teaching in the World] 38(1). 97110.
Montero Perez, Maribel. 2022. Second or foreign language learning through watching audio-visual input
and the role of on-screen text. Language Teaching 55(2). 163192.
Moreno, Roxana & Richard Mayer. 2007. Interactive multimodal learning environments: Special issue on
interactive learning environments: Contemporary issues and trends. Educational Psychology Review
19(3). 309326.
Nurviyani, Vina, Didi Suherdi & Iwa Lukmana. 2020. Developing studentsreading skill through making
multimodal inferences. English Review: Journal of English Education 8(2). 11.
Pasquarella, Adrian, Xi Chen, Katie Lam, Yang C. Luo & Gloria Ramirez. 2011. Cross-language transfer of
morphological awareness in Chinese-English bilinguals. Journal of Research in Reading 34(1). 2342.
Pellicer-Sánchez, Ana, Kathy Conklin, Michael P. H. Rodgers & Fabio Parente. 2021. The eect of auditory
input on multimodal reading comprehension: An examination of adult readerseye movements. The
Modern Language Journal 105(4). 936956.
Qiao, Shen, Yingyi Liu & Susanna Siu-sze Yeung. 2021. Mechanisms by which morphological awareness
contributes to Chinese and English reading comprehension. Journal of Research in Reading 44(1).
189209.
Sacco, Guillaume, Frédéric Noublanche, Frédéric Blazek, Catherine Hue, Loïc Carballido, Asfar Marine,
Philippe Allain & Annweiler Cédric. 2021. How to deal with the consent of adults with cognitive
impairment involved in European geriatric living labs? Philosophy, Ethics, and Humanities in Medicine
16(3). https://doi.org/10.1186/s13010-021-00101-1.
Shen, Helen H. 2004. Level of cognitive processing: Eects on character learning among non-native
learners of Chinese as a foreign language. Language and Education 18(2). 167182.
Taguchi, Etsuo, Greta J. Gorsuch, Kristin Lems & Rory Rosszell. 2016. Scaolding in L2 reading: How
repetition and an auditory model help readers. Reading in a Foreign Language 28. 101117.
Tragant, Elsa, Carmen Muñoz & Nina Spada. 2016. Maximizing young learnersinput: An intervention
program. The Canadian Modern Language Review 72(2). 234257.
Van Vu, Duy. 2023. Incidental collocation learning from dierent modes of input and factors that aect
learning. Leuven, Belgium: KU Leuven dissertation.
Wang, Dingmin, Yan Song, Jing Li, Jialong Han & Haisong Zhang. 2018. A hybrid approach to automatic
corpus generation for Chinese spelling check. In Proceedings of the 2018 conference on empirical
methods in natural language processing, 25172527.
Wang, Shufang & Chang-In Lee. 2020. Exploring the eectiveness of hypermedia glosses for second
language reading. The Journal of Asia TEFL 17(4). 12841293.
Webb, Stuart & Anna C.-S. Chang. 2014. Second language vocabulary learning through extensive reading
with audio support: How do frequency and distribution of occurrence aect learning? Language
Teaching Research 19(6). 667686.
Webb, Stuart & Anna C.-S. Chang. 2020. How does mode of input aect the incidental learning of
collocations? Studies in Second Language Acquisition 44(1). 3556.
Wong, Yu Ka, Xuan Zang & Tomohiro Inoue. 2023. Promoting foundational linguistic skills for reading
development in young Chinese language learners: A 1-year intervention study. Journal of Research in
Reading 46(3). 247277.
Multimodal reading in CSL learners 235
Wu, Junru. 2022. Neurocognition in the reading of Chinese characters. Journal of Chinese Writing Systems
6(3). 167183.
Ye, Lijuan. 2011. Teaching and learning Chinese as a foreign language in the United States: To delay or not to
delay the character introduction. Atlanta, GA: Georgia State University dissertation.
Zhang, Peng. 2021. Learning strategies, motivation and learnersperspectives on online multimodal
Chinese learning. Chinese Language Teaching Methodology and Technology 4(1).
Zhang, Haomin, Xing Zhang, Mengjie Li & Yiming Zhang. 2021. Morphological awareness in L2 Chinese
reading comprehension: Testing of mediating routes. Frontiers in Psychology 12. 736933.
Bionotes
Peng Zhang
Namseoul University, 10218, Humanities and Social Sciences Building, 91 Daehak-ro, Seonghwan-eup,
Seobuk-gu, Cheonan-si, Chungcheongnam-do, South Korea
nsuiben@nsu.ac.kr
Peng Zhang (https://orcid.org/0000-0001-5164-1869) is an educator specializing in language education
and educational technology with a PhD in educational technology from the University of the Balearic
Islands and dual MAs from the University of Barcelona and Université dArtois. A registered teacher in both
the US and UK, Peng leads workshops for the International Baccalaureate (IB) and serves as Assistant
Professor at NamSeoul University. He also works as an examiner for IBDP, IGCSE, and A-Level, and
contributes as a reviewer to several educational journals.
Yue Peng
Room 404, Ho Tin Building The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China
yuepeng@cuhk.edu.hk
Yue Peng (https://orcid.org/0000-0003-2935-688X, PhD, Queens University) is an Assistant Professor in
Chinese language education at the Chinese University of Hong Kong. Her research interests include
teaching and learning Chinese as a second language, particularly in teachersprofessional learning,
multilingualism, and language assessment. Her recent publications appeared in journals such as The
Modern Language Journal, Language Teaching Research, Language Testing, Language, Culture and Curriculum,
and Journal of Multilingual and Multicultural Development.
236 Zhang and Peng
... This flexibility in strategy contrasted with previous studies that suggested a stronger reliance on local linguistic cues in reading comprehension. Building on this, Zhang and Peng (2024) investigated the impact of multimodal reading strategies, such as integrating visual and auditory inputs, on comprehension and memory retention among CSL learners. Their study showed that combining modalities, like reading while listening, greatly enhanced learners' understanding and recall, highlighting the critical role of contextual and perceptual cues in facilitating effective language acquisition. ...
Article
Full-text available
This study examines how Chinese as a Second Language (CSL) learners acquire the Chinese stance marker haihao with a focus on type and saliency. A total of 56 participants took part in the research, including 28 English-speaking CSL learners and 28 native Chinese speakers. The study utilized two evaluation judgment tasks. Results showed that participants categorized haihao into two simplified groups, guided by the economy principle and a positivity bias. English-speaking learners, influenced by a stronger positivity bias, tended to select more positive options, while Chinese participants favored slightly negative ones. Saliency improved the accuracy of recognizing negative haihao among American learners and low positive haihao among Chinese participants, though it was less effective for ambiguous expressions. These findings highlight how cultural differences and language saliency impact the interpretation of stance markers, offering insights for improving CSL teaching strategies.
Article
Full-text available
Background Using a 1‐year longitudinal design, we evaluated the effectiveness of an intervention programme to improve reading attainments among Chinese‐as‐a‐second‐language (CSL) learners. Using the simple view of reading as a theoretical framework, the intervention focused on students' orthographic knowledge, word reading and listening comprehension to facilitate reading comprehension. Method A total of 186 Grade 4 CSL students (mean age = 9.22, SD = 0.56) participated in the study and were grouped into the treatment ( N = 96) and control ( N = 90) groups. A range of Chinese language and literacy skills were assessed before and after the intervention. Results The path analysis showed that the intervention significantly affected orthographic knowledge, word reading and listening comprehension directly and reading comprehension indirectly through the three componential skills after controlling for nonverbal reasoning, vocabulary and autoregressors. Conclusions The findings suggest that orthographic knowledge, word reading and listening comprehension may be promising targets for instructing young CSL learners.
Article
Full-text available
This mixed-method empirical study investigated the role of learning strategies and motivation in predicting L2 Chinese learning outcomes in an online multimodal learning environment. Both quantitative and qualitative approaches also examined the learners' perspectives on online multimodal Chinese learning. The participants in this study were fifteen pre-intermediate adult Chinese learners aged 18-26. They were originally from different countries (Spain, Italy, Argentina, Colombia, and Mexico) and lived in Barcelona. They were multilingual, speaking more than two European languages, without exposure to any other Asian languages apart from Chinese. The study's investigation was composed of Strategy Inventory for Language Learning (SILL), motivation questionnaire, learner perception questionnaire, and focus group interview. The whole trial period lasted three months; after the experiment, the statistics were analyzed via the Spearman correlation coefficient. The statistical analysis results showed that strategy use was highly correlated with online multimodal Chinese learning outcomes; this indicated that strategy use played a vital role in online multimodal Chinese learning. Motivation was also found to have a significant effect. The perception questionnaire uncovered that the students were overall satisfied and favoring the online multimodal learning experience design. The detailed insights from the participants were exhibited in the transcripted analysis of focus group interviews.
Article
Full-text available
The project presented here is an experience report carried out in a university C1 German course and based on the reading of F.C. Delius' novel The Walk from Rostock to Syracuse. This article aims to provide suggestions for the practical application and the linking types of language-image texts. The students re-live the protagonist's escape on a sailing boat across the Baltic Sea from Eastern Germany during Germany’s separation from the West to fulfill his dream of a journey through Italy. This text lends itself to be explored in a multimodal way, as it presupposes implicit knowledge that includes certain historical and cultural contextual references. Therefore, information and extra-textual images are necessary to visualise and understand the narrative. Both the reading process and the results of the multimodal project are recorded in a reading blog. This offers a variety of possibilities for linguistic expression and acquisition as the multimodal approach provides varied resources for activity-based reading and communicative learning among peers. Keywords: Images, intertextuality, literacy, multimodality, reading comprehension;
Article
Full-text available
The purpose of the present study was to investigate the effects of a multimodal program, designed for practicing reading, on reading development in struggling readers in two different countries. The research question was whether one specific training method will have a positive effect on pupils’ reading development in two different countries with different educational systems and as diverse orthographies as the shallow Croatian and the relatively deep Swedish orthography. It became clear that the Swedish teachers have a tradition of implementing interventions as opposed to in Croatia where there is no tradition of teachers conducting intervention studies in school. Comparing different school systems is difficult which is evident in the results that differ between the two countries. However, the results indicate that the multimodal reading training program used in the intervention had positive effects on pupils’ literacy development, including decoding, spelling, and reading comprehension in both countries.
Article
Full-text available
Chinese characters, which form an ideographic writing system, have shown distinctive features in the neurocognition of their reading. This paper gives a comprehensive review of five main issues of Chinese reading, which are (1) the relationship across written forms, sounds, and meanings during the neurocognitive processing of Chinese reading, (2) the involvement of structural and linguistic units, such as strokes (笔画), radicles (部件), components (偏旁), whole characters (整字), and words (词), in character reading, (3) the influence of attentional conditions on character reading, (4) the activation of Chinese characters from non-character input, and (5) the influences of readers’ linguistic experiences on character reading. We summarize that the reading of Chinese characters involves dynamic integration of multi-channel information and multiple levels of processing units, which, as compared with alphabetic writing systems, rely more on form-to-meaning semantic processing, while its form-to-phonology non-semantic processing is greatly influenced by para-linguistic context.
Article
Reading‐while‐listening (RWL) has been suggested to facilitate reading comprehension by establishing letter‐to‐sound correspondences in the word recognition process. However, previous findings are inconsistent regarding the effect of RWL on foreign language (L2) reading comprehension. To understand why such inconsistency occurs, the moderating role of L2 orthographic characteristics in the effects of RWL on L2 propositional comprehension was examined. The orthographic characteristics of a language are of particular relevance as languages differ in the time course and the degree to which visual and phonological coding strategies are involved in word recognition. L1 Korean (alphabetic, shallow orthography) intermediate undergraduate learners of Chinese (nonalphabetic), English (alphabetic, deep orthography), and Spanish (alphabetic, shallow orthography) read and read‐while‐listened to authentic reading texts in their target language, and solved a set of written, untimed multiple‐choice questions to test their propositional comprehension. Results indicated that learners of an L2 with a larger distance (i.e., Chinese) from the L1 benefitted more from RWL, indicating that L2 orthographic characteristics interacted with input modality. Pedagogical implications for the use of RWL are discussed coupled with qualitative findings of students' perceptions.
Thesis
Collocations, i.e., words that habitually co-occur in texts (e.g., strong coffee, heavy smoker), are ubiquitous in language and thus crucial for second/foreign language (L2) learners to master. However, previous research shows that L2 learners tend to have limited knowledge of collocations and underuse or misuse them. Given the limited classroom time for deliberately teaching all L2 collocations, incidental learning, i.e., learning while being engaged in meaningful input (e.g., listening, reading, viewing), should play an important role in broadening L2 learners’ collocational knowledge. Nonetheless, to date, there are relatively few studies on incidental collocation learning. In addition, little is known about the effects of different modes of input on incidental collocation learning as well as factors that affect learning. Therefore, the current PhD project was conducted to fill those research gaps. To that end, four empirical studies involving Vietnamese learners of English as a foreign language (EFL) were carried out. In each study, participants were exposed to target verb-noun and adjective-noun English collocations in different sets of modes of input (including reading-only, reading-while-listening, reading with textual input enhancement in the form of underlining, reading aloud, and captioned TV viewing) in a counterbalanced fashion. Learning gains in each study were measured with a form recall test on the target collocations administered two times: one week before the treatment (pretest) and one week after the treatment (delayed posttest). Different learner- and item-related factors that might affect learning gains were also examined. Study 1 (Chapter 3) investigated incidental collocation learning from reading-only, reading- while-listening (i.e., reading a book while listening to its audio version), and reading with textual input enhancement (i.e., reading with target items underlined). The results showed that reading with textual input enhancement resulted in significantly more incidental collocation learning than both reading-while-listening and reading-only. Reading-while-listening was more effective for incidental collocation learning than reading-only. Following Study 1, Study 2 (Chapter 4) examined incidental collocation learning from reading-while-listening, reading with textual input enhancement (i.e., underlining), and reading-while-listening plus textual input enhancement to explore if integrating textual input enhancement into reading-while-listening would make a significant difference. The findings revealed that reading with textual input enhancement and reading-while-listening plus textual input enhancement led to significantly more learning gains than reading-while-listening. Reading-while-listening plus textual input enhancement, however, did not differ significantly from reading with textual input enhancement. Study 3 (Chapter 5) looked into incidental collocation learning from reading-while-listening and captioned TV viewing, which differed in terms of the availability of imagery. The results showed that both reading-while-listening and captioned TV led to gains of collocations, but these two modes of input did not differ significantly. Study 4 (Chapter 6) explored incidental collocation learning from reading-only, reading-while-listening, and reading aloud, which were different concerning audio support and vocalization. The results showed that reading aloud and reading-while-listening resulted in significantly more learning gains than reading- only. There was no significant difference between reading aloud and reading-while-listening. With respect to factors that affect learning, all four studies showed that learners’ prior vocabulary knowledge significantly predicted the learning gains (Studies 1, 2, 3, and 4). Learners with more vocabulary knowledge were likely to pick up more collocations from reading. As for captioned TV viewing, Study 3 showed that participants with larger vocabularies picked up fewer collocations than those with smaller vocabularies. About collocational congruency (i.e., the availability of first language equivalents of L2 collocations), Studies 1, 2, and 3 showed that this factor was a significant predictor of learning gains. More congruent collocations were incidentally learned than incongruent collocations. Additionally, only Study 4 found that type of collocation affected learning gains, with more adjective-noun collocations being learned than verb-noun collocations, whereas Studies 1, 2, and 3 did not. Other factors, including frequency of exposure, corpus frequency, strength of association (Studies 1 and 2), and test modality (Study 4) did not affect learning gains. The project contributes to advancing our understanding of the effects of input modality on incidental collocation learning and factors that predict learning gains, which can also help relevant stakeholders make informed decisions about L2 vocabulary learning and teaching.
Article
Currently, multimodal texts are increasingly wide, requiring readers to have the competence to read. Besides, developing the teaching reading comprehension multimodal texts competence is also required as an inevitable need. From the undeniable role of competence development-oriented exercises as well as the need to develop the teaching reading comprehension texts competence of students in general, the teaching reading comprehension multimodal texts competence of Philology pedagogical students in particular, we propose to building and using exercises to develop the teaching reading comprehension multimodal texts competence for Philology pedagogical students. This article aims to provide some theoretical bases on exercises, competence development-oriented exercises, the process of building and using exercises and applying exercises to develop the teaching reading comprehension multimodal texts competence for Philology pedagogical students.
Article
This article discusses research into the role of audio-visual input for second language (L2) or foreign language learning. It also addresses questions related to the effectiveness of audio-visual input with different types of on-screen text such as subtitles (i.e., in learners’ first language) and captions (i.e., subtitles in the same language as the L2 audio) for L2 learning. The review discusses the following themes: (a) the characteristics of audio-visual input such as the multimodal nature of the input and vocabulary demands of video; (b) L2 learners’ comprehension of audio-visual input and the role of different types of on-screen text; (c) the effectiveness of audio-visual input and on-screen text for aspects of L2 learning including vocabulary, grammar, and listening; and (d) research into L2 learners’ use and perceptions of audio-visual input and on-screen text. The review ends with a consideration of implications for teaching practice and a conclusion that discusses the generalizability of current research in relation to suggestions for future research.
Article
Comprehension of many types of texts involves constructing meaning from text and pictures. However, research examining how second language (L2) learners process text and pictures and the relationship with comprehension is scarce. Thus, while verbal input is often presented in written and auditory modes simultaneously (i.e., audio of text with simultaneous reading of it), we do not know how the auditory input affects L2 adult learners’ processing of text and pictures and its relation to comprehension. In the current study, L2 adult learners and native (L1) adults read and read while listening to an illustrated story while their eye movements were recorded. Immediately after reading, they completed a comprehension test. Results showed that the presence of auditory input allowed learners to spend more time looking at pictures and supported a better integration of text and pictures. No differences were observed between L2 and L1 readers’ allocation of attention to text and pictures. Both reading conditions led to similar levels of comprehension. Processing time on the text was positively related to comprehension for L2 readers, while it was associated to lower comprehension for L1 readers. Processing time on images was positively related to comprehension only for L1 readers.