Content uploaded by Maja Gligora Markovic
Author content
All content in this area was uploaded by Maja Gligora Markovic on Jul 07, 2015
Content may be subject to copyright.
JIOS, VOL. 39, NO. 1 (2015) SUBMITTED 04/15; ACCEPTED 05/15
A Study on Knowledge Gain and Retention when Using
Multimedia Learning Materials of Different Quality
Maja Gligora Marković mgligoram@veleri.hr
Polytechnic of Rijeka
Rijeka, Croatia
Dijana Plantak Vukovac dijana.plantak@foi.hr
University of Zagreb, Faculty of Organization and Informatics
Varaždin, Croatia
Božidar Kliček bozidar.klicek@foi.hr
University of Zagreb, Faculty of Organization and Informatics
Varaždin, Croatia
Abstract
The usage of multimedia has proven to foster meaningful learning, but not every multimedia
resource will necessarily contribute to the teaching-learning process. Since for the
development of multimedia learning materials (MLMs) additional time and effort is required
for everyone involved in the development process to pay proper attention to its quality in order
to achieve desired learning effects. The paper deals with issues regarding the quality of
multimedia learning materials (MLMs) as well as the relationship between MLM quality and
knowledge retention.
The results of an experimental study that involves low-quality and high-quality MLMs for two
different topics and their effects on knowledge gain and retention of the polytechnic school
students are presented. For the purpose of the research the LORI assessment tool was used to
evaluate the quality of MLMs. The analysis of research data shows that MLMs developed
according to the principles of multimedia learning and principles for reducing cognitive load
were perceived as being of higher quality than those that were not developed using multimedia
principles. Furthermore, the students‟ usage of high-quality MLMs during treatment resulted
in better knowledge acquisition and retention indicated by significantly higher scores in three
knowledge assessments.
Keywords: multimedia learning materials, quality assessment tool, knowledge gain,
knowledge retention
1. Introduction
Representing knowledge in myriad ways by combining text, still graphics, animation, audio
and video, have promoted multimedia to be one of the most crucial teaching media [3].
Nowadays teachers can improve their teaching by using many forms of technologies (e.g. TV,
computers, tablets and smart phones) and media (books, audio, video, multimedia, social
media, educational games). Although a number of studies, e.g. [1], [2], [22], have confirmed a
beneficial influence of a combination of several digital media on learning goals, the
production of multimedia learning materials (MLMs) does not necessarily contribute to the
teaching-learning process. This is due to the lack of adequate consideration for the quality of
educational materials and pedagogical standards [6].
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
Design principles or guidelines established in relevant theories should be taken into
consideration when producing multimedia learning materials of high-quality. Also, these
materials have to accomplish their purpose, that is, they have to facilitate knowledge
acquisition.
Nonetheless, creators of multimedia learning materials are frequently not informed about
relevant research in psychology and education [27]. Furthermore, it appears that due to
insufficient focus placed on quality assurance of MLMs, materials are not evaluated by
appropriate assessment tools. Leacock and Nesbit [17] indicated that instruments used to
evaluate educational software packages may not be appropriate for evaluating small learning
objects. Therefore, the first issue that needs to be addressed is the problem of different
definitions and scope of MLMs. Firstly, they can be regarded as smaller digital learning
resources that combine different media and are intended for re-use [17], signifying that those
learning objects do not depend on technical and educational settings. Secondly, they can also
be considered a part of an e-learning setting and should thus be evaluated using a broader e-
learning quality framework.
Another issue with multimedia learning materials is their effectiveness regarding the
retention of acquired knowledge. Hermann Ebbinghaus, a psychologist and pioneer of
memory research, found out in a series of research experiments that forgetting is exponential
over time [10]. Similarly, Custers [9] in his research on medical students determined that after
one year approximately 25-35% of rudimentary science knowledge is forgotten and by the
next year more than 50%. There are many factors that influence the speed of forgetting, e.g.
difficulty of the learned material, relation to previous knowlegde, conditions during learning,
but also the form of the represented information [32]. With the introduction of multimedia in
education, numerous studies have been conducted to explore the effects of different media
forms on learning, but only a few have explored the influence of multimedia on knowledge
retention (e.g. [15], [29], [14]).
In this paper the study that focuses on two objectives is presented: 1) the assessment of
the quality of MLMs, and 2) the investigation of knowledge gain and retention over the
extended period of time when using MLMs of different quality. To achieve these objectives,
theoretical and empirical research on the influence of multimedia on learning and knowledge
retention were reviewed first, then the tools for quality assessment of MLMs were examined
and finally an experiment with low- and high-quality MLMs was conducted. In the final
section of this paper, research results and implications of MLM quality on the e-learning
systems design are discussed.
2. Review of Previous Research on Learning with Multimedia
2.1. The Influence of Multimedia on Learning
The usage of multimedia to induce purposeful learning has been researched extensively. One
of the most in-depth studies has been undertaken by Richard E. Mayer and his associates who
have conducted a series of experiments to corroborate Mayer‟s cognitive theory of
multimedia learning and display the benefits of using twelve principles of multimedia design
in the development of learning materials [20]. In one of his experiments, Mayer [21] proved
the importance of convergence of individual types of media such as auditory narration with
visual animation to promote meaningful learning. Accompanying that, Mayer and Moreno
suggested nine ways of reducing cognitive load in multimedia learning [23].
Other researchers explored the influence of digital media representation on learning, too.
For example, Alty et al. [1] showed that the sound-and-diagram media combination
significantly outperformed text-and-diagram and text-only presentations when learning
statistical topics within a university context domain. Lou and associates [18] have been
investigating multimedia learning effects in chemistry education and their experiments have
shown that students studying from video or animation learning materials achieve better
learning results than their peers who studied only from static pictures. Arguel and Jamet [2]
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
have also experimented with static pictures and video presentations to examine their influence
on learning outcomes. They found that undergraduate students who had learned from a
combination of video and static pictures performed better in the assessment than those
learning only from video presentations, pointing out that learners could benefit from the
presentation of static pictures in the vicinity of videos because these pictures emphasize the
key stages of an unfamiliar procedure.
The context of the video was also in the focus of the study conducted by Ljubojevic et al.
[19]. They examined the following types of multimedia learning materials with a
supplementary video introduced at different points of multimedia presentations: main
multimedia without a supplementary video; main multimedia with an educational
supplementary video showed at the beginning, in the middle, and at the end of the
presentation; main multimedia with an entertainment supplementary video introduced at the
beginning, in the middle, and at the end of the presentation. The experimental results showed
that integrating the video in the middle of a lecture is more efficient than showing it at other
points during the lecture. An entertainment video can be used for motivating students to learn,
however, it proved to be less efficient than an educational video.
On the other hand, there are studies in which no significant evidence on the impact of
different media modalities in the e-learning environment was obtained. Two experimental
studies performed by Jenkinson and associates [13] comprised respondents‟ time-limited
exposure to one of the two treatments: e-learning modules with static graphics versus
animated graphics, preceded by a pre-test and followed by a post-test. Although in their
studies no significant difference between treatments was found in the quantitative data,
crucial differences regarding students‟ perception of the effectiveness of the media were
identified in the qualitative data.
2.2. The Influence of Multimedia on Knowledge Retention
There are studies that explore students‟ knowledge retention when using two instructional
formats, e.g. lecture combined with discussion versus cooperative and active learning, such as
[24], [28]. Knowledge retention achievements gained by multimedia usage is much less
investigated.
One of the studies [29] explored long-term retention of knowledge gained by interactive
multimedia versus the conventional direct method of teaching English language. In
experimental research 154 students, aged 12-14 years, participated. They were divided at
random into two groups: a conventional teaching method group and multimedia teaching
method group. An experimental research method included a pre-test and a post-test of the
student‟s achievement. The post-test was administrated immediately after the experiment
finished. The retention test was administrated one month after the post-test. Mean
achievement scores on the post-test of multimedia group were higher than the mean
achievement scores on the post-test of the conventional group. The difference in mean
between these two retention scores was highly significant too. The author concluded that from
the above data it is clear that as far as the retention is concerned an interactive multimedia
method of learning is substantially better than the conventional direct method.
Another study [14] investigated knowledge gain and retention by using multimedia
learning materials among adult patients. Two experimental groups were assigned to learn
from a multimedia diabetes education program and take a pretest, post-test, and 2-week
follow-up knowledge test. The mean age of participants was between 56 and 52.8 years,
respectively. After the post-test one group was exposed to the teach-back technique. Research
results showed that all participants forgot approximately half of the new information they had
learned after 2 weeks. Additionally, the teach-back technique did not improve knowledge
retention at the 2-week follow-up. The authors suggested that besides MLMs appropriate
teaching strategies should be incorporated to improve long-term learning outcomes of the
targeted learners.
None of the prior studies have investigated the quality of multimedia learning materials
and its effects on knowledge acquisition and retention.
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
2.3. Evaluating the Quality of Multimedia Learning Materials
The importance of assessing the quality of digital multimedia learning resources is due to the
following issues:
a considerable amount of multimedia materials in repositories or on the Internet is of
dubious quality because their design is not grounded on relevant research in
psychology and education [6], [27],
inefficient or time-consuming search for MLMs of high quality [17],
the objective does not have to be choosing MLMs for re-use because search engines
implemented in some MLM repositories use certain quality metrics to get specific
(ordered) search results [22] and therefore it depends upon the validity of the
evaluation tool to generate quality ratings,
the design of multimedia learning materials can be upgraded by using criteria for
assessing the quality of MLM as guidelines [5], [26].
Numerous studies focus on the effects of multimedia learning, however, little attention
has been given to the quality of multimedia learning materials and this area has been
investigated scarcely.
Nonetheless, the relationship between students‟ cognitive styles, the quality of
multimedia service access by the Internet (QoS) and quality perceptions of participants (QoP)
of multimedia content were explored in one of the rare empirical studies by Chen at al. [8]
The quality of multimedia learning content was limited to the service of Internet access in
the research, which is only one of the attributes that should be considered within the quality of
multimedia learning materials in general. This technological approach to MLM quality is also
common in a number of other studies, e.g. [4], [6], [11], where technological standards were
used for MLM development and assessment of their quality, without focusing on its
pedagogical aspects.
The evaluation of multimedia learning materials is another issue and their quality is
frequently considered within the overall online course quality, as in the study by Sung et al.
[30]. In their investigation of e-learning quality assurance, sixty-seven e-learning courseware
applications were analyzed and evaluated for the purpose of e-learning courseware
certification.
Such an approach was defined as course providers' perspective by Bubaš et al. [5] in the
process of e-learning quality assurance, which emphasizes transparency, quality control and
standardization. Other side of quality assurance are students' standpoint, with a focus on the
effective presentation of content, promotion of interaction and greater achievement in learning
[5], or suppliers' aspect, where market-boosting and cost/benefit issues are considered [16].
The aforementioned examples illustrate different approaches to researching the quality of
multimedia teaching materials and their use in the teaching process. However, the relationship
between the MLM quality and achievement of learning process outcomes are scarce.
2.4. Tools for Measuring the Quality of Multimedia Learning Materials
Evaluation tools, such as questionnaires and checklists, are used to measure the quality of
multimedia learning materials. Reliability and validity of these evaluation tools is
questionable [27]. In the selection of a tool for assessing MLM quality, the scope of
multimedia learning materials should be taken into consideration as an additional criterion so
as to differentiate smaller independent digital learning resources („narrower scope‟) from
multimedia learning materials that are an integral part of an e-learning course („broader
scope‟).
In this paper, multimedia learning materials (MLMs) are considered as computer-based
learning materials intended to be used for teaching or learning that are not dependent on the
learning environment and can be presented online or offline. Therefore, for the purpose of this
research, two available instruments for measuring the quality of multimedia learning materials
with similar definitions of MLM were reviewed: LORI [17], [25], [27], and QAMLM [6], [7].
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
Leacock and Nesbit [17] define MLMs as digital learning resources that combine text,
images and other media intended to be reused in different learning contexts. They created an
instrument to measure the quality of multimedia learning resources named LORI (Learning
Object Review Instrument). LORI encompasses nine indicators or items for the evaluation of
multimedia learning resources. Every item is described with several dimensions referring to
that item rated on a 5-point scale, where 1 means the lowest grade and 5 means the highest
grade, and can be further described with reviewers‟ comments. The final score for MLM
quality is calculated by averaging the ratings.
The other instrument for measuring the quality of multimedia learning materials is the
assessment guide entitled Quality Assurance in Multimedia Learning Materials (QAMLM)
developed by the Commonwealth Educational Media Centre for Asia. There are two versions
of the guide exist: QAMLM 1.0 [6] and the revised QAMLM 1.5 [7]. In the latter version, the
guide provides quality assurance rating tools for the developers of MLM (e.g. designers,
technical specialists, subject matter experts), individuals who will perform the evaluation of
MLM quality (e.g. teachers, parents, administrators and librarians) and end-users of MLM
(learners), respectively. The revised version also incorporates some parameters that enable
evaluation of e-learning courses like reusability or information validity and updating, among
others [7]. The assessment guide for evaluators of MLMs (Part B) in the v1.0 version contains
19 quality indicators which are scored on a five-point scale. QAMLM is more suitable for
qualitative assessment of MLM quality [6] and should be adapted according to the context of
evaluation [7].
3. Empirical Research
3.1. Research Design
Identifying the quality of multimedia learning materials and analyzing the relationship
between MLM quality and knowledge gain and retention were the primary goals of our study.
Thus, in order to examine these issues, the following sub-goals were defined:
developing low-quality and high-quality multimedia learning materials,
selecting and adapting the MLM quality assesment tool,
conducting the experiment to measure the quality of MLMs and its effects on
knowledge gain and retention during a longer period of time.
The two research questions were:
(RQ1) What are the effects of high-quality multimedia learning material on knowledge
gain?
(RQ2) What are the effects of high-quality multimedia learning material on knowledge
retention?
3.1.1. Development of Multimedia Learning Materials (MLMs)
Multimedia learning materials can be implemented as part of a learning unit on a CD-ROM or
used in online learning that is not facilitated by a teacher. In this study, MLMs are considered
independent learning resources and were put into practice in a blended learning environment
in LMS Moodle v. 1.9. for the course “Graphics, text, multimedia”, which was taught to 1st
year students at the Polytechnic of Rijeka.
All multimedia learning materials were developed according to the guidelines for
developing interactive multimedia learning modules by Stanford University [12] and by
following the Quality Assurance Framework based on the ADDIE model of instructional
design [6]. MLMs were developed for the following two topics: „Colors and the Use of Colors
on the Internet‟ (MLM C) and „The Use of Graphics on the Internet‟ (MLM G).
The two versions of each topic were developed: the low-quality version (LQ) and the
high-quality version (HQ). Low-quality multimedia learning materials (MLM G-LQ and
MLM C-LQ) contained only texts and static images. In addition, in MLMs LQ learning goals
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
were not indicated, navigation was available only through the breadcrumbs determined by the
Moodle, pages did not have a title, important parts of the content were not emphasized with
formatted text or a different font, multimedia was not properly used for the presentation of the
content, and self-assessment was not available, to mention only some of their features (see
Figure 1). On the other hand, high-quality multimedia learning materials (MLM G-HQ and
MLM C-HQ) presented the same learning content as MLMs LQ, but with additional
multimedia types: audio, video, animations and interactive simulations (see Figure 2). Those
were developed by applying Mayer‟s principles of multimedia learning [20], as well as
principles for reducing cognitive load [23]. Processes and changing parameters, e.g. how a
change in the brightness parameter affects the picture, were presented with video or animation
as well as additional text. The latter was placed in the proximity of the animation to reduce
cognitive overload, according to the spatial contiguity principle [23].
Figure 1. The Lesson on "Color Models" in the MLM C-LQ
Figure 2. The Lesson on "Color Models" in MLM C-HQ
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
3.1.2. Selection and Adaptation of the MLM Quality Assessment Tool
Before conducting experimental research, the two available instruments for measuring the
quality of multimedia learning materials, LORI and QAMLM 1.0., were analyzed. While the
QAMLM assessment guide was still under revision when this research was performed, the
LORI tool had already been validated in several studies [25], [31] and proven to be easy to
use and time-effective [17]. The advantage of the latter is in quantitative measures, which
allows for easier interpretation of results. Furthermore, it had been used by many educational
institutions to teach learning resource evaluation as well as to perform evaluations within
different educational models (e.g. in a collaborative environment) [17]. Thus, it was decided
to use LORI for assessing the quality of the developed MLMs.
LORI‟s dimensions are described by the following nine items [17[, [25]: content quality
(veracity, accuracy, balanced presentation of ideas, appropriate level of detail), learning goal
alignment (alignment among learning goals, activities, assessments, learner characteristics),
feedback and adaptation (feedback or adaptive content driven by differential learner input or
learner modeling), motivation (ability to motivate and interest a population of learners),
presentation design (design of visual and auditory information for enhanced learning and
efficient cognitive processing), interaction usability (ease of navigation, predictability of the
user interface, quality of the interface help features), accessibility (design of controls and
presentation formats to accommodate disabled and mobile learners), reusability (ability to use
the tool in varying learning contexts and with learners from different backgrounds), and
standards compliance (adherence to international standards and specifications).
The original LORI was extended and adapted after the pilot research conducted among
fifteen teachers at the Polytechnic of Rijeka concerning their usage of MLMs in the classroom
and issues concerning MLM quality. Based on their answers in a semi-structured interview
three LORI items were extended with new statements. The reliability of the adapted LORI
was tested during the analysis of experimental data indicating high internal consistency, as
reported in Section 4.2.
3.1.3. Experimental Method Plan
The experiment was primarily aimed at collecting data to achieve the objective of the
research: to determine the quality of MLMs and the effect of using MLMs of different quality
on knowledge acquisition and retention. It consisted of: a) usage of multimedia learning
materials in a blended learning environment and repeated knowledge assessments on selected
topics and b) evaluation of the quality of multimedia learning materials.
In the initial part of research, multimedia learning materials were used by first year
students of a professional study program in Information Science during regular classes in a
computer laboratory
Since our intention was for every student to use low-quality and high-quality multimedia
learning materials, a within-subjects research design was chosen. In order to eliminate a
possible effect of the preferred learning topic on knowledge acquisition, students were
distributed randomly into two groups and assigned MLM C-LQ and MLM G-HQ (Group 1)
or MLM C-HQ and MLM G-LQ (Group 2). The procedure lasted for eight weeks, during
which three knowledge tests were administered for each of the two topics. In the first week a
diagnostic pre-test was performed with each group of students to identify their prior
knowledge of the topics from the course syllabus, including the two topics of our research
interest. Students were briefly introduced to the research study, but were not acquainted with
the objective of the experiment. They were also introduced to LMS Moodle. In the second
week MLM C-LQ (Group 1) or MLM C-HQ (Group 2) were used in class for 45 minutes,
with a knowledge test administered 10 minutes afterwards. The tests comprised 15 questions
(MLM C) and 21 questions (MLM G), respectively, and evaluated both recall and
comprehension of the given topic. Grades were assigned according to the following
percentages of points acquired in the test, as defined by the Assessment Regulations of the
Polytechnic of Rijeka: 1 – failing grade (less than 40% of acquired points), 2 – satisfactory
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
(40-59.9% of acquired points), 3 – good (60-69.9% of acquired points), 4 – very good (70-
79.9% of acquired points), and 5 – excellent (80-100% of acquired points). In the third week,
the knowledge assessment test was repeated. During the fourth week the groups studied the
other topic, using MLM G-HQ (Group 1) or MLM G-LQ (Group 2), with a knowledge test
administered as in the second week. In the fifth week they were re-tested on the second topic.
During the sixth and eight week, the third knowledge assessment was performed, on the first
and second topic, respectively. In the seventh week there was no class activity, but students
were able to interact with MLMs from their home (see Figure 3). They were also informed
about the weeks when retests are going to take place.
Figure 3. A workflow of the research experiment
The second part of the experiment comprised the evaluation of the multimedia learning
materials quality with the adapted LORI instrument that contained 50 statements. The
evaluation was performed by chosen Croatian computer science teachers who examined the
MLMs and assessed them in an online survey.
The usage of MLM C-LQ and
the first knowledge assessment
A pre-test of prior knowledge
and acquaintance with the LMS
Moodle interface
The second knowledge assess-
ment on topic C (MLM C-LQ)
1st week
2nd week
3rd week
The usage of MLM G-LQ and
the first knowledge assessment
Group 2: MLM C-HQ & MLM G-LQ
The usage of MLM C-HQ and
the first knowledge assessment
The second knowledge assess-
ment on topic C (MLM C-HQ)
1st week
2nd week
3rd week
Group 1: MLM C-LQ & MLM G-HQ
The usage of MLM G-HQ and
the first knowledge assessment
A pre-test of prior knowledge
and acquaintance with the LMS
Moodle interface
4th week
4th week
5th week
5th week
The second knowledge assess-
ment on topic G (MLM G-LQ)
The second knowledge assess-
ment on topic G (MLM G-HQ)
The third knowledge assess-
ment on topic C (MLM C-HQ)
The third knowledge assess-
ment on topic C (MLM C-LQ)
No activity
No activity
The third knowledge assess-
ment on topic G (MLM G-LQ)
The third knowledge assess-
ment on topic G (MLM G-HQ)
6th week
6th week
7th week
7th week
8th week
8th week
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
4. Research Findings
4.1. Participants’ Demographic Data
A total of 141 first-year students, 105 of them male (74.50%) and 36 female (25.50%)
participated in the usage of multimedia learning materials and knowledge testing. Among
them, 102 (72.30%) had previously used multimedia in learning. A vast majority of
respondents (133, or 94.30%) had attended a computer science course in high school.
In evaluating the quality of MLMs 19 teachers were involved, 13 (68.40%) of whom were
female and 6 (31.60%) male. Most respondents had between 3 and 5 years of teaching
experience, with 16 of them working at a higher education institution. All respondents had
been using multimedia in their teaching practice, 3 (15.80%) of them sometimes, 14 (73.70%)
of them often, and two (10.50%) of them constantly.
4.2. Quality of Multimedia Learning Materials
In the following paragraph research results were given focusing on MLM quality of the two
courses described in section 3.1.1.
Grades for each MLM were given on a scale of 1 (very low quality) to 5 (very high
quality). The analysis of data gathered in the evaluation of MLM quality with the adapted
LORI showed that teachers evaluated MLM LQ as low-quality and MLM HQ as high-quality
materials, which was evident in the average grade given to a particular MLM type. Figure 4
represents the average grades for MLMs of different quality.
Figure 4. Average grades for MLMs of different quality
The reliability of the adapted LORI was determined by running the Cronbach‟s Alpha test in
which all factors showed high internal consistency: 0.933 (MLM C-LQ), 0.947 (MLM C-HQ)
0.936 (MLM G-LQ) and 0.900 (MLM G-HQ).
Figure 5 represents the values for the nine items of the LORI assessment tool for the topic
“Colors and the Use of Colors on the Internet” (MLM C). The smallest difference in the score
(1.06) was identified for the Content Quality parameter, as both MLM C-LQ and MLM C-LQ
contained the same factual knowledge related to the aforementioned topic. The biggest
difference between scores (2.43) was found for the Learning Goal Alignment parameter,
which among other reasons, was due to the fact that in MLM C-LQ learning objectives were
not defined, whereas in MLM C-HQ they were stated clearly.
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
Figure 5. The LORI values for MLM C
Figure 6 represents values for the nine items of the LORI assessment tool for the topic „The
Use of Graphics on the Internet‟ (MLM G). The biggest difference between the average
scores (2.16) was identified for the Standards Compliance parameter, since MLM G-LQ did
not embed metadata and these were not developed by using web technologies such as CSS or
similar standards. The smallest difference between the average scores (0.87) was found for
the accessibility parameter since both types of MLMs were available via LMS Moodle, which
was accessed from computer labs equipped with a high-speed Internet connection.
Figure 6. LORI values for MLM G
4.3. Effects of MLM Quality on Knowledge Gain and Retention
In this paragraph the results of knowledge gain obtained were shown after the first knowledge
assessment and the results of the knowledge retention after the second and third knowledge
assessment.
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
To determine the results of knowledge testing, the average number of points and
associated grades (ranging from 1 to 5) acquired after the use of MLMs and each knowledge
assessments was calculated first. After that it was verified if there was a statistically
significant difference in those parameters for MLMs of different quality for each topic (MLM
C and MLM G).
Tables 1 and 2 show the results of the t-test for independent samples, where the number
of points generated as a commemorative shape represents the amount of acquired knowledge.
A statistically significant difference (p <0.01) was found between LQ and HQ for topics in the
first, second and third examination of acquired knowledge.
Variable
N
Mean
(max=32 points)
Std.
Deviation
t
Df
Sig. (2-
tailed)
D
First knowledge
assessment
-6.336
130.181
0.000
1.077
MLM C-LQ
70
21.43
4.292
MLM C-HQ
71
25.54
3.337
Second knowledge
assessment
-4.832
115
0.000
0.894
MLM C-LQ
57
19.23
3.727
MLM C-HQ
60
22.58
3.779
Third knowledge
assessment
-6.062
126
0.000
1.074
MLM C-LQ
65
15.94
4.387
MLM C-HQ
63
20.44
4.007
Table 1. The comparison of knowledge acquisition after the usage of MLMs of different quality and the
first, second and third knowledge assessment on topic C
Variable
n
Mean
(max=38 points)
Std.
Deviation
t
Df
Sig. (2-
tailed)
d
First knowledge
assessment
-7.475
139
0.000
1.262
MLM G-LQ
71
23.42
4.449
MLM G-HQ
70
25.54
5.183
Second knowledge
assessment
-5.590
117.852
0.000
0.981
MLM G-LQ
67
18.75
4.460
MLM G-HQ
67
24.07
6.401
Third knowledge
assessment
-6.006
122.225
0.000
1.040
MLM G-LQ
69
15.52
5.121
MLM G-HQ
68
21.85
7.050
Table 2. The comparison of knowledge acquisition after the usage of MLMs of different quality and the
first, second and third knowledge assessment on topic G
Furthermore, Table 3 shows knowledge acquisition expressed as the average scores achieved,
their percentages and the average rating for all MLMs and for all knowledge assessments. The
percentage of acquired knowledge upon the use of higher-quality MLMs (C-HQ and G-HQ)
was significantly higher (67.21% and 79.81%, respectively) than after the use of low-quality
MLMs (C-LQ: 66.96%, G-LQ: 61.63%) during the first knowledge acquisition.
After the second knowledge assessment, the average percentage of acquired knowledge
decreased approximately 6 to 12% for students who used low-quality MLMs, whereas for
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
students who used MLMs of higher-quality the percentage of acquired knowledge dropped
close to 4 to 9%. After the third knowledge assessment, the acquisition of knowledge in
comparison to the second knowledge assessment fell by around 10% for students who used
low-quality MLMs, and about 6 to 7% for those who used high-quality MLMs.
Variable
n
Score in points
(average/max)
Percentage of acquired
knowledge
Average grade
(max. 5)
First knowledge
acquisition
MLM C-LQ
70
21.43/32
66.96%
2.77
MLM C-HQ
71
25.54/32
79.81%
3.66
MLM G-LQ
71
23.42/38
61.63%
2.15
MLM G-HQ
70
25.54/38
67.21%
3.28
Second knowledge
acquisition
MLM C-LQ
57
19.23/32
60.09%
2.80
MLM C-HQ
60
22.58/32
70.56%
3.46
MLM G-LQ
67
18.75/38
49.34%
2.05
MLM G-HQ
67
24.07/38
63.34%
2.98
Third knowledge
acquisition
MLM C-LQ
65
15.94/32
49.81%
2.16
MLM C-HQ
63
20.44/32
63.87%
3.00
MLM G-LQ
69
15.52/38
40.84%
1.63
MLM G-HQ
68
21.85/38
57.5%
2.73
Table 3. Knowledge acquisition expressed by scores in points, percentages and average grades after the
usage of MLMs of different quality
5. Discussion
If the average number of the achieved points in Table 3 in all three knowledge assessments is
compared after using MLMs on the “Colors” topic, it is possible to notice that knowledge
acquisition between the first and second assessment when using MLMs of lower-quality drops
around 6.20% per week. The decline when it comes to knowledge acquisition after using
MLMs of higher-quality between the first and second assessment is approximately 9.30%.
(which is more than when using MLM of lower-quality), but that fall is 3,1% per week on
average, which argues in favor of better knowledge retention after using higher-quality
MLMa.
In case of using MLMs on the topic of “Graphics”, a difference between knowledge
acquisition in the first and second assessments, that is the second and third assessments can be
perceived when using MLM G-HQ decrease of 3.94% per week on average. The achieved
percentages after using MLM G-LQ from the first to the second assessment dropped by
13.10%, wheras the difference between the second and third assessment is approximately
7.80%, although there was a two-week period between the two assessments.
Besides, forgetting curves have been compared for all four MLMs (see Figure 7).
Forgetting curves expressed as exponential trendlines show proportional decline in knowledge
for both low-quality and high-quality MLMs regarding topic C. In case of topic G the
difference in forgetting curves between MLM G-LQ and MLM G-HQ is not proportional, so
additional research is needed to identify factors that influence this phenomenon. Given that
MLM G-HQ is evaluated as lower quality in comparison to MLM C-HQ, the only possibility
is to speculate why the forgetting curve of MLM G-HQ declines more slightly in comparison
to MLM C-HQ.
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
Figure 7. A comparison of percentages of acquired knowledge with low- and high-quality MLMs and the
forgetting curves
There is almost no research focused on the assessment of knowledge retention after using
MLMs and there is only one study [29] where knowledge was assessed after using MLM
(which is an equivalent to our first assessment) and after one month (in our case it is the third
assessment). Unfortunately, in that research the maximum number of points is not given,
therefore it is impossible to compare the results in this study with aforementioned research. In
the described study [29] the difference between the average achieved points in the first
assessment (mean 41.26) and the assessment after a month (mean 35.80) is 6 points
considering the usage of MLMs, and the difference between the first (mean 28,80) and the
second assessment (mean 21.02) after the employment of the direct teaching method is
approximately 7 points.
The author of the study [29] draws a conclusion based on these results that when it comes
to knowledge retention, the method of using MLMs is to some extent more successful than
the direct method. Although it is significantly more effective regarding knowledge acquisition
in general (knowledge acquisition by using MLMs after the first assessment is 41.26, whereas
by using direct teaching methods the average scored points are 28.80). The author is more
focused on proving the statistically significantly greater number of points when using MLM
in comparison to the direct method than on comparing percentages of knowledge acquisition,
which would be a much better criterion to compare it with. Unfortunately, due to the
unavailability of these data, a more detailed comparison of this and her study is not possible.
One of the limitations of our research regarding the assessment of MLM quality is a small
number of teachers who evaluated MLMs, so additional research with a larger number of
evaluators is needed in that respect as well. Also, evaluations performed by other stakeholders
(e.g. students) might yield different quality estimation results since different stakeholders may
emphasize different quality attributes, as suggested in [17]. Furthermore, there might be
various combinations of media types used in HQ and LQ MLMs with the same learning topic,
which can be another source of potential difficulties since those materials are not going to be
directly comparable. However, due to the fact that the learning content was the same in both
MLM LQ and MLM HQ, and that our goal was to evaluate the overall quality of multimedia
learning materials within the online course and determine the effects of their quality on
knowledge acquisition, this could be considered just as a minor obstacle.
As for knowledge retention, knowledge assessment in our experiment was not performed
after a longer period of time, that is after eight months as in studies [9] or [28]. Since those
two studies have showed contradictory results (no long-term retention effect in [9] and
significant retention effect in [28]), additional research that measures long-term retention in
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
general and using multimedia learning materials in particular would provide better insight into
this interesting topic. It would also be useful to know how many hours students spent studying
the topics between knowledge assessments and whether the introduction of another topic
affected in any way the test results of the first topic. Our future research will hopefully answer
some of these questions. Additional data analysis will be presented by using conventional
statistical methods and data mining methods.
6. Conclusion
The design and development of high-quality MLMs is a demanding endeavor, which explains
why most teachers are reluctant to invest their time and effort into their development. What is
more, outcomes regarding MLMs quality and knowledge gain are occasionally questionable.
Also, the majority of teachers are not sufficiently informed about recent research or guidelines
for creating learning materials that foster meaningful learning.
The results of this research confirm that the usage of Mayer's principles of multimedia
learning allow for creation of high-quality MLMs in a blended learning environment. In our
experiment multimedia learning materials of different quality were created for the topics
„Colors and the Use of Colors on the Internet‟ (MLM C) and „The Use of Graphics on the
Internet‟ (MLM G) and were subsequently implemented in a polytechnic school course via
LMS Moodle. The assessment of MLMs quality performed by teachers by means of the
adapted LORI evaluation tool showed significant difference between the high-quality (MLM
C-HQ, MLM G-HQ) and low-quality materials (MLM C-LQ, MLM G-LQ) for every LORI
item.
The adapted LORI assessment tool was validated as an effective evaluation tool with
clear benchmarks for different quality items. It can be used to evaluate not only small learning
resources, but also, as in this case, MLMs implemented in learning management systems.
Items like feedback and adaptation, interaction usability or accessibility evaluate dimensions
that are part of LMS's functionality and as such can be used for the evaluation of LMS as a
product. However, the above assumption needs to be confirmed by additional research.
From the description of the experiment reported in this paper, it is evident that students
who used MLMs of higher quality acquired the average grade that was approximately one
grade higher than that acquired by students who used lower-quality MLMs. This result was
obtained in the knowledge assessment test taken shortly after the usage of MLMs in class.
Besides, the last two assessments of knowledge acquisition showed that knowledge gain
is better among students who used higher-quality MLMs in comparison to those who used
lower-quality MLMs. However, the level of knowledge decreases in time almost
proportionally when using higher-quality MLMs and lower-quality MLMs. Thus, additional
research is needed to identify factors in MLMs that would cause slower decline of the
forgetting curve.
There is the need to invest in an appropriate design of high-quality multimedia learning
materials for the use in the e-learning environment due to beneficial effects of HQ MLMs,
which were identified in our research. These effects have impact on knowledge acquisition
and achievement of learning outcomes, but also result in better grades for learners.
References
[1] Alty, J.L; Al-Sharrah, A; Beacham, N. When humans form media and media form
humans: An experimental study examining the effects different digital media have
on the learning outcomes of students who have different learning styles.
Interacting with Computers, 18(5), 891–909, 2006.
[2] Arguel, A; Jamet, E. Using video and static pictures to improve learning of
procedural contents. Computers in Human Behaviour, 25(2), pages 354–359,
2009.
JOURNAL OF INFORMATION AND ORGANIZATIONAL SCIENCES
[3] Bates, A.W; Poole, G. Effective Teaching with Technology in Higher Education.
John Wiley & Sons, Inc., San Francisco, USA, 2003.
[4] Batteram, H; Damm, G; Mukhopadhyay, A; Philippart, L; Odysseos, R.; Urrutia-
Valdés, C. Delivering quality of experience in multimedia networks. Bell Labs
Techical Journal, 15(1), pages 175–193, 2010.
[5] Bubaš, G; Balaban, I; Begiĉević, N. Evaluation of Online Courses as an Element
of Instructional Design : The Case of Two Hybrid University Courses. In
Proceedings of the 30th MIPRO Jubilee International Convention on Computers
in Education - MIPRO 2007. pages 216–221. MIPRO, Rijeka, 2007.
[6] CEMCA (Commonwealth Educational Media Centre for Asia): Quality Assurance
of MultiMedia Learning Materials (Version 1.0), New Delhi, 2009.
http://cemca.org.in/ckfinder/userfiles/files/QAMLM 1_0.pdf, downloaded: May
11, 2013
[7] CEMCA (Commonwealth Educational Media Centre for Asia): Quality Assurance
of MultiMedia Learning Materials (Version 1.5), New Delhi, 2010.
http://cemca.org.in/ckfinder/userfiles/files/QAMLM 1-5.pdf, downloaded: May
11, 2013
[8] Chen, S.Y; Ghinea, G; Macredie, R.D. A cognitive approach to user perception of
multimedia quality: An empirical investigation. International Journal of Human-
Computer Studies, 64(12), pages 1200–1213, 2006.
[9] Custers, E.J.F.M; ten Cate, O.T.J. Very long-term retention of basic science
knowledge in doctors after graduation. Medical Education, 45(4), pages 422–430,
2011.
[10] Ebbinghaus, H. Uber das Gedachtniss. [Memory: A contribution to experimental
psychology. Transl. by Ruger HA & Bussenius CE, 1913.], 1885.
[11] Hede, T; Hede, A. Multimedia effects on learning: Design implications of an
integrated model. In Proceedings ASET Conference 2002, 2002.
[12] Huang, C. Designing high-quality interactive multimedia learning modules.
Comput. Med. Imaging Graph. 29(2-3), pages 223–33, 2005.
[13] Jenkinson, J. Measuring the Effectiveness of Educational Technology : what are
we Attempting to Measure ? Electronic Journal of e-Learning, 7(3), pages 273–
280, 2009.
[14] Kandula, N.R; Malli, T; Zei, C.P; Larsen, E; Baker, D.W. Literacy and Retention
of Information After a Multimedia Diabetes Education Program and Teach-Back.
Journal of Health Communication: International Perspectives, 16(sup3), pages
89–102, 2011.
[15] Karaman, S. Effects of audience response systems on student achievement and
long-term retention. Social Behavior and Personality, 39(10), pages 1431–1440,
2011.
[16] Kidney, G; Cummings, L; Boehm, A. Toward a Quality Assurance Approach to E-
Learning Courses. International Journal on E-Learning, 6, pages 17–30, 2007.
[17] Leacock, T.L; Nesbit, J.C. A Framework for Evaluating the Quality of Multimedia
Learning Resources. Educational Technology & Society, 10(2), pages 44–59,
2007.
[18] Lou, S; Lin, H; Shih, R.C; Tseng, K.H. Improving the Effectiveness of Organic
Chemistry Experiments Through Multimedia Teaching Materials for Junior High
School Students. The Turkish Online Journal of Educational Technology, 11(2),
pages 135–141, 2012.
GLIGORA MARKOVIĆ, PLANTAK VUKOVAC AND KLIĈEK KNOWLEDGE GAIN AND RETENTION WHEN USING...
[19] Ljubojevic, M; Vaskovic, V; Stankovic, S; Vaskovic, J. Using supplementary
video in multimedia instruction as a teaching tool to increase efficiency of learning
and quality of experience. International Review of Research in Open and Distance
Learning, 15(3), pages 275–291, 2014.
[20] Mayer, R.E. Cognitive Theory of Multimedia Learning. In: The Cambridge
Handbook of Multimedia Learning. Cambridge University Press, pages 31–48,
2005.
[21] Mayer, R.E. Multimedia learning: Are we asking the right questions?, Educational
Psychologist, 32(1), pages 1–19, 1997.
[22] Mayer, R.E; Moreno, R. Aids to computer-based multimedia learning, Learning
and Instruction, 12(1), pages 107-119, 2002.
[23] Mayer, R.E; Moreno, R. Nine Ways to Reduce Cognitive Load in Multimedia
Learning. Educational Psychologist, 38(1), pages 43–52, 2003.
[24] Morgan, R.L; Whorton, J.E; Gunsalus, C. A comparison of short term and long
term retention: lecture combined with discussion versus cooperative learning.
Journal of Instructional Psychology, 27(1), pages 53–58, 2000.
[25] Nesbit, J; Belfer, K; Leacock, T. Learning Object Review Instrument (LORI):
User Manual, 2004,
http://www.transplantedgoose.net/gradstudies/educ892/LORI1.5.pdf, downloaded:
February 3, 2010.
[26] Nesbit, J; Belfer, K; Vargo, J. A Convergent Participation Model for Evaluation of
Learning Objects. Canadian Journal of Learning and Technology, 28(3), 2002.
[27] Nesbit, J.C; Leacock, T.L. Web-Based Tools for Collaborative Evaluation of
Learning Resources. Systemics, Cybernetics and Informatics, 3(5), pages 102–
112, 2006.
[28] Pérez-Sabater, C; Montero-Fleta, B; Pérez-Sabater, M; Rising, B; De Valencia,
U.P. Active learning to improve long-term knowledge retention. In Proceedings of
the XII Simposio Internacional de Comunicación Social, pages 75–79, Santiago de
Cuba, Cuba, 2011.
[29] Sharma, P. Role of Interactive Multimedia for Enhancing Students ‟ Achievement
and Retention. International Women Online Journal of Distance Education, 2(3),
pages 12–22, 2013.
[30] Sung, Y.T; Chang, K.E; Yu, W.C. Evaluating the reliability and impact of a
quality assurance system for E-learning courseware. Computers and Education,
57(2), pages 1615–1627, 2011.
[31] Vargo, J; Nesbit, J.C; Belfer, K; Archambault, A. Learning object evaluation:
Computer mediated collaboration and inter-rater reliability. International Journal
of Computers and Applications, 25(3), pages 198–205, 2003.
[32] The forgeting curve, Syndeon Soft, http://www.flashcardlearner.com/articles/the-
forgetting-curve/, downloaded: February 3, 2015.
