Evaluation and Analysis of the Application on Interactive Digital Resources in a Blended-Learning Methodology for a Computer Networks Subject

Full text

Evaluation and analysis of the application of interactive digital resources in a
blended-learning methodology for a computer networks subject
Francisco A. Candelas, Pablo Gil, Carlos A. Jara, Juan A. Corrales
University of Alicante (SPAIN)
francisco.candelas@ua.es, pablo.gil@ua.es, carlos.jara@ua.es, jcorrales@ua.es
Abstract
This paper shows the successful application of a blended-learning methodology using a set of
interactive digital resources. This work has been developed for the subject Data Transport Systems,
which is about computer network systems and is taught in the Computer Science Engineering degree
at the University of Alicante, in Spain. These new resources are based on interactive user guides for
the practical exercises, created as SCORM objects, self-assessment and qualification questionnaires.
The set of digital resources has been included in a web learning platform based on Moodle with two
main objectives: to stimulate the autonomous learning by providing interactivity between students to
make flexible the way how the students study, and to facilitate the implantation of subjects in the new
European Space of Higher Education system by using tools which offer collaborative surroundings.
The results obtained show that this method has provided more flexibility and autonomy to the learning
process and it has improved students’ marks, in comparison with classical teaching and learning.
Keywords: blended-learning, digital resources, Moodle, SCORM packages, self-assessment.
1 INTRODUCTION
Blended-learning can be defined as a model for learning that combines face to face education with the
web technology applied to e-learning (‘electronic learning’) [1] for distance education. In contrast, the
concept of e-learning is used to define a learning model where the students rarely or never meet face
to face, that is, the students are only supported by online courses based on web technology. This
paper shows the application of this educational methodology completed by a set of interactive digital
resources. These resources are based on interactive user guides for the laboratory exercises, created
as SCORM objects [2], self-assessment and qualification questionnaires, managed by the web
learning platform Moodle [3].
In order to analyse the results that the new materials and the educational methodology employed have
had on students’ learning, a complete study based on the scores of all questionnaires (both self-
assessment and evaluation questionnaires) has been carried out. In addition, the student's opinion
about the new interactive digital resources developed for the subject has been also analysed.
As this paper describes, the results of the study indicate that this work has been helpful and useful to
improve students’ learning. Moreover, students’ opinion shows that they rated very well the new
materials used in the course and they also believe that the evaluation questionnaires are well suited to
the contents of the subject.
This paper is organized as follows: Section 2 describes the educational methodology employed.
Section 3 explains how the digital resources used in the course have been created and published by
the authors. Section 4 shows the results obtained
2 EDUCATIONAL METHODOLOGY
Data Transport Systems covers the study and experimentation of specific and current technologies of
data networks, as for example LANs (local area networks) like Ethernet or WiFi, remote connections
through VPNs (Virtual Private Networks), IP (Internet Protocol) routing or QoS (quality of service)
control. Previously to this subject which is taught in the second quarter of the academic year, students
have to study another subject about the basic concepts on networking in the first quarter. This other
subject is called Computer Networks and a similar study about it was shown in [4].
Due to its technical character, the practical experimentation is a very important part of the subject.
This experimentation is carried out in a laboratory which has specific equipment for computer
networks, located in the Polytechnic School at the University of Alicante so that the exercises
proposed to students are similar to common situations of the professional world. This also determines

that student must attend the lab to solve the majority of exercises. The student's homework focuses on
preparing the experiments before their implementation, and reviewing and analysing the results.
This section focuses first on describing how the laboratory work proposed to the student is organized
and evaluated using the interactive materials explained after in Section three. Then, the way in which
this methodology has been evaluated will be described.
2.1 ORGANIZATION AND EVALUATION OF LABORATORY CLASSES
In the laboratory, each student has to work with a computer where the necessary software tools are
available. It is worth to mention that all the used tools are free software. To guide students in the
experiments, the student is provided with practical manuals which, on the one hand, explain the basic
skills required and the tools used, and, on the other hand, describe the experiments to be performed in
a sequential and constructive way. In addition, the teacher briefly explains the main concepts and
organizes the work at the beginning of each session.
The experiments are organized as three main blocks, each one of which includes several experiments
and has four sessions of two hours. The content of these experiments will be detailed in Section 3.
The student must complete the experiments raised, individually or with help from his teammates, and
with the support of the teacher. It is intended that students learn by solving problems within the topics
addressed in accordance with the philosophy of constructive and collaborative learning.
The evaluation of the skills acquired by students is performed by a test at the end of each block of
experiments, which has a maximum duration of one hour. Each of these tests offers the same type of
issues that arise in practice manuals, and the student can use all the tools and equipment he used in
practice to resolve the tests. In addition, following the test session the teacher reviews the results of
students who requested them and the students who get a low score. For the final assessment of the
laboratory, the student's attendance and class participation are also taken into account.
The above process is completed in the academic year 2010-11 by manuals with interactive and static
formats, and especially through self-assessment activities, as described in Section 3.
In addition, along the course, the teacher also teaches the students how to resolve specific cases
about configuring data routing with IP, using exercises and examples that are solved in class, with the
participation of students. The skills acquired on this type of problems are evaluated with a fourth test,
which is a particular problem for data routing configuration. This test does not require the use of the
laboratory. For this part there are not self-assessment activities, although the student has a collection
of exercises and teacher support along the laboratory sessions.
As mentioned before, the realization of practical experiments requires the student's attendance at the
laboratory, but there are some questions that students can experience outside. In addition, for proper
monitoring of the practices, the student requires a work outside the laboratory on the preparation of
the experiments and subsequent analysis, in order to deal more successfully with the assessment
tests. This is where the addition of b-learning techniques can help students [5], especially through
interactive materials and self-assessment activities which are accessible through the Web.
2.2 ASSESSMENT OF THE PROPOSED METHODOLOGY
To analyse the degree of success of the proposed methodology, it is necessary to know how new
materials and teaching methods applied in the laboratory exercises affect student learning. This
knowledge has been extracted in two ways. On the one hand, the marks obtained by students who
have used new materials and methods have been compared to other students who have not. On the
other hand, the final qualifications of students have been compared with the results of self-assessment
tests. The results of both studies are detailed in Section 4.1.
For the first study, the authors of this work initially thought to define two groups of students for the
laboratory. But, soon it was concluded that it was difficult to plan the course with two different student
groups. Besides, that would cause inequality of opportunity in the evaluation of the subject. So, finally,
it was opted for all students in this academic year 2010-11 working with the new materials and
methods, and the comparison was made between the students of last course and the previous 2009-
10. This comparison is valid for several reasons: the students of the two courses have the same
profile, the content of the previous course is very similar to the last one, the same laboratory and
equipment has been used, and teachers are the same.

Finally, it has also been made a questionnaire to assess student opinion regarding new materials and
the self-assessment test, and, in this way, to know the degree of acceptance of the new methodology.
Section 4.2 shows the results drawn from this survey of opinion.
3 DEVELOPMENT OF THE DIGITAL RESOURCES
This section describes in detail the digital materials which have been created by the authors and
published in a web site based on the popular LCMS Moodle [3], which is available in the University of
Alicante as a corporative site: Moodle-UA.
3.1 CONTENTS AND ASSESSMENT
The authors have taken the contents of the laboratory classes of the previous year as a start point for
the new materials of the year 2010-2011 because the proposed experiments are very interesting for
the professional future of the students in the field of computer communications. Therefore, the
practical classes at the laboratory have been organized in the following three practices:
• Practice 1: Dynamic Routing with IPv4: This practice deals with the routing protocols and the
translation of IP directions.
• Practice 2: Tunnels and Virtual Private Networks (VPN): This practice presents the
configuration and operation of remote access to private networks.
• Practice 3: Quality of Service Control and Analysis of WiFi networks: This practice describes
the mechanisms that restrain or guarantee the bandwidths of a network and the analysis of
wireless network traffic.
The knowledge and skills achieved by the students are assessed by a questionnaire which is
performed at the end of each practice. These questionnaires propose experiments which are similar to
those experiments performed by the students during the development of the practice and they are also
completed in the laboratory. Three questionnaires have been implemented in Moodle-UA, one for
each practice and a final written exam have been proposed for the assessment of the routing
exercises. Each of these three tests represents a 25% of the final score of the practical classes.
3.2 CREATION OF THE TEACHING MATERIALS
In previous years, the materials for the practical classes of the subject “Data Transport Systems” were
based on static manuals of the three practical experiments for the laboratory sessions, a group of
exercises about routing, schemes of the network structure of the laboratory and slides with the
presentations of the teachers. The manuals of the practices have been improved this year by including
technical information about standards and technologies. This update is very important for the students
not only in order to teach them the last technologies but also to give them a general view of the
behaviour of the technologies that are taught during the practical classes and which are commonly
used in the professional environment.
Figure 1 depicts the snapshot of the Moodle-UA webpage where students can access to the materials
of the subject. The traditional materials mentioned above (i.e. manuals, schemes, slides and
exercises) are still provided but students can also access to new interactive manuals and self-
assessment questionnaires.

Figure 1: Snapshot of the webpage with the materials for the subject in Moodle-UA.
These interactive manuals complement the traditional static materials of previous years which can be
continue to be downloaded as pdf files. The interactive versions of the manuals allow the student to
easily navigate along the contents of the practice and answer interactively to questions which are
inserted in these manuals. These web manuals have been created as SCORM packages [2] by the
Wimba Create application [6] since they can be easily imported as activities in Moodle. Figure 2 shows
an example of several sections of these new manuals. Since all these materials and the assessment
questionnaires are available in the webpage of the subject in Moodle-UA, they can be easily accessed
by all the students through Internet anywhere so that b-learning methods can be applied.
Figure 2: Snapshots of several sections of the new interactive manuals based on SCORM.
3.3 ASSESSMENT AND SELF-ASSESSMENT QUESTIONNAIRES
Assessment and self-assessment questionnaires have been developed as questionnaires activities in
Moodle-UA. This method has many important advantages: the variety of different types of questions,
automatic correction, the possibility of reviewing the students’ answers and the easiness of
importing/exporting information about the results of the exams.
The questions developed for these questionnaires are based on two different types: multiple choice
and cloze. Multiple choice questions make students choose the best option in a list of several
alternatives (as shown in Figure 3). Cloze questions allow developing more general questions where
students have to fill textboxes (with strings and/or numbers) and/or choose an option in a list (as
shown in Figure 4).

Figure 3: Multiple choice question with feedback (in bold) in self-assessment questionnaire.
Figure 4: Cloze question with feedback (in bold) in self-assessment questionnaire.
These types of questions have not only been applied in the assessment questionnaires but also in the
self-assessment questionnaires. The self-assessment test is available during the previous week to the
exam (i.e. assessment questionnaire) and students have 4 attempts each one to complete it with a
minimum delay of 30 minutes between them. When students finish an attempt of the self-assessment
questionnaire, they get a general mark which identifies their knowledge of the practice and their level
of preparation for the exam. For each wrong answer, students get a feedback text in order to help
them find by themselves the right answer (as shown at the bottom of snapshots of Figures 3 and 4)
and thus a constructive learning process is made possible.
These types of questions have also been used for the implementation of the last survey which has
been answered by the students during the last week of the year in order to discover their opinions
about the new materials and methods. In particular, this survey is composed by 8 multiple choice
questions which are voluntary and are not considered in the mark of the subject.
3.4 COMPILATION OF RESULTS
In order to assess the effect of the new materials and the new self-assessment questionnaires over
the learning process of the students, a comparison between the marks of the students has been
developed. In particular, the authors have compiled the following data from the Moodle-UA platform:
the marks of the three assessment questionnaires (i.e. practice exams), the marks of the three self-
assessment questionnaires and the results of the final survey. In addition, the authors have also
compiled the marks of the students from the previous year (2009-2010) in order to compare them with
the marks of the current year. The analysis of all these data is explained in the following section of this
paper. Table 1 summarizes the number of students who have done each one of these questionnaires.
There were 84 students registered at the beginning of the year and 60 of them completed all the
compulsory exams (since the self-assessment questionnaires were optional). Finally, 71.4% of
students have completed the practical classes.

Table 1. ,umber of students who completed the different tests.
Practice 1 Practice 2 Practice 3 Routing
Exercise
Self-
Assessment 57 60 54 *
Assessment
(Exam) 80 75 73 60
* There was no self-assessment questionnaire for this test.
4 RESULTS OBTAINED
In previous works, some studies have been done to determine the degree of satisfaction of the
students in relation to the teaching methodology and the e-learning process with Moodle and SCORM
packets in comparison with traditional teaching methodology [4]. In addition, another study has shown
how easy is to generate simulations for courses of Computer Networks based on Java applets [7] from
tools as EJS [8] and KivaNS [9] and how this type of digital resources have improved the student
learning process. In this study, a teaching methodology based on self-assessments tests have been
designed and implemented. This way, the student is confronted the training test which measures
acquired knowledge level in a similar way that the assessment test. In general, the students have
welcomed this type of learning based on self-assessments.
4.1 IMPROVING THE LEARNING PROCESS WITH SELF-ASSESSMENTS
The students have performed three experiments (E1, E2, E3) about the topics of the Computer
Network course. The 71.25% of students (57 of 80 students), the 80% (60 of 75 students) and the
73.92% (54 of 73 students) have participated in the self-assessments, respectively. These self-
assessments can be answered one or more times. On the one hand, figure 5a shows the number of
participant students on the self-assessments process per number of attempts. The majority of the
students have spent two or less attempts. On the other hand, Figure 5b shows the achieved
qualification by the students according to the number of attempts used to pass the final test of each
experiment. In general, the majority of students have spent two attempts to achieve a qualification
between 7 and 10 according to Spanish Academic Grading System (70% of the answers are correct).
0
5
10
15
20
25
30
35
0 1 2 3 4
Self-assessments E1
Self-assessments E2
Self-assessments E3
a)
0,00
1,00
2,00
3,00
4,00
5,00
6,00
7,00
8,00
9,00
10,00
01234
Self-assessments E1
Self-assessments E2
Self-assessments E3
b)
Figure 5: a) Number of participant students on the self-assessment process per number of attempts.
b) Students’ marks according to the number of attempts to pass the test of each experiment.
Figure 6 shows a comparison between the marks achieved by students with regard to their
assessments and self-assessments. The difference is minimal when they have used 3 attempts to
answer correctly the questions of test (see images from Figure 6a-6c). The Spanish Academic
Grading System has been used to measure the reached level of representative knowledge. However,

this system has been transformed to the most popular grading system, the United States grade which
use discrete evaluation in the form of letter grades (Figure 6d). The Spanish Grade uses a scale
between 0 and 10 and the United States Grade uses a scale between 0 and 4. Furthermore, Figure 6a
shows as the non-participant students (number of attempts is 0) on the self-assessments have
obtained the lowest mark on the assessments.
In order to evaluate the teaching methods and how they influence the learning process, this paper
presents a comparison between the students’ marks of the two last academic courses: 2010 (92
students) and 2011 (84 students). In the two years, the students have the similar academic profile and
thus the comparison can be done. The only difference is the self-assessments system in 2011. They
did not self-assessments in 2010. This fact has allowed the students to improve their marks. These
results can be observed in Figure 7. Figure 7 shows that the percentage of students with ‘A’ marks
has grown respect to 2010 in all the experiments (E1-E3). In particular, the experiment 1 (Figure 7a)
shows 8.5% with mark ‘A’ and 4.5% with mark ‘B’ more students in 2011 than in 2010. In addition, the
number of non-participant students has decreased (5%). The student loses the fear of the
examination. Similarly, the 23% and 17% of students achieve a mark A in the experiment 2 (Figure 7b)
and experiment 3 (Figure 7c), respectively. At the same time, the number of students which do not
pass the final test have decreased, 7% and 18% for the experiments 2 and 3.
0
1
2
3
4
5
6
7
8
9
10
01234
Self-assessments E1
Qualification E1
a)
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4
Self-assessments E2
Qualification E2
b)
0
1
2
3
4
5
6
7
8
9
10
01234
Self-assessments E3
Qualification E3
c)
d)
Figure 6: a)b)c) Marks obtained by students in relation to the number of attempts to pass the test of
each experiment (E1-E3).d) Equivalence approximated among Spanish academic grading system and
USA grading system.

0%
5%
10%
15%
20%
25%
30%
35%
40%
45% Qualification 20 10
Qualification 20 11
a)
0%
5%
10%
15%
20%
25%
30%
35%
40%
45% Qualification 20 10
Qualification 20 11
b)
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Qualification 201 0
Qualification 201 1
c)
Figure 7: Final qualification obtained by the students to pass the test of the three experiments realized.
4.2 STUDENTS’ OPINION
An extension of the previous study was done to determine the degree of satisfaction of the students in
relation to the teaching methodology in 2011. A survey has been arranged in the educational platform
Moodle-UA. The survey consists of questions about the four different aspects such as assessments
(Figure 8a), self-assessments (Figure 8b), types of resources (Figure 9a) and quality of experiments
(Figure 9b). The survey is voluntary and it has been answered by 24 students (25%).
0 5 10 15 20
High
Medium
Low
Very low
Assessment
Dificulty leve l
Contents distribution level
a)
0 10 20 30
High
Medium
Low
Very low
Self
-
assessment
Similarity lev el
Dificulty leve l
b)
Figure 8: Results of students’ opinion about a): assessment exam: b) self-assessment questionnaires

Students’ opinion shows the degree of satisfaction about: a) the final test used to assess the subject.
b) Self-assessment tests to prepare the training to pass the final test. Seeing the Figure 8, two
important conclusions can be drawn:
- The students think that the level of requirements to pass the exam (final test) is according to the
contents and aspects studied in the four experiments. Figure 8a shows the contents distribution
level is good (medium or high). And the majority of students think that difficulty to pass the exam
is low. This means the exam is easy.
- The students think that the tests of self-assessment have been very useful to self-measure the
knowledge level and skills acquired before making the exam (final test). Figure 8b shows how the
students think that the self-assessment and assessments tests are quite similar though not too
(majority opinion is medium). In addition, they think that difficulty level of the self-assessment
tests is higher than the assessment test (majority opinion is high in Figure 8b in comparison with
low-medium in Figure 8a).
Respect to the students’ opinion about the educational platforms (Figure 9a) and the degree of utility
of the educational resources provided by teachers (Figure 9b) in order to understand and learn
concepts and topics, as well as acquire skills (Figure 9c) to pass the final test of the subject and to use
it in a future job, four affirmations can be highlighted.
- The 87% of students prefer Moodle versus Virtual Campus-UA (Figure 9a) independently of the
resources format (SCORM-web, PDF, etc.). Moreover, the 57% of students prefer PDF opposite
SCORM-web (Figure 9a). This result contrasts with other researches carried out previously in [4].
Perhaps, it can be due to low participation in the survey. The SCORM-web provides more
versatility and flexibility to add educational contents (videos, sounds, JAVA simulations, etc.).
- The students prefer assessment tests implemented by means of quiz module of Moodle (Figure
9b) versus classic test with short or long questions in paper.
- The 75% of students consider that the following of course is easy. Moreover, as final conclusion,
the majority of students consider that the skills achieved are useful or very useful for their first
future jobs (Figure 9c).
PDF-
Virtual
campus
UA
13%
PDF-
Moodle
57%
Web
SCORM-
Moodle
30%
Paper
0%
a)
Moodle
(online
form)
87%
Test-
Questions
(write)
4%
Reports
(write)
9%
b)
0 5 10 15 20
High
Medium
Low
Very low
Experime nts
Utility level for the future
job
Dificulty leve l to learn
c)
Figure 9: Students’ opinion that show their preferences about: a) the educational platform and
educational resources. b) Quiz module of Moodle to do the tests. c) The quality of the experiments.

5 CONCLUSIONS
The application of a blended-learning methodology using a set of interactive digital resources for the
subject Data Transport Systems in the Computer Science Engineering degree at the University of
Alicante, has been effective. The resources employed are based on interactive user guides for the
practical exercises, created as SCORM objects, self-assessment and qualification questionnaires, all
them included in the e-learning platform Moodle.
The results of the study show that students who used self-assessment questionnaires obtained higher
scores in the practical assessment test. It has also been verified that students' marks for this academic
year 2010-11 have improved and the percentage of students who passed the practical assessment
test is higher, both compared with the past academic year. This indicates that this work has been
helpful and useful to improve students’ learning. Moreover, it has been verified that qualifications were
not improved when students use more than three attempts for the self-assessment questionnaires.
With regard to the students’ opinion, they rated very well the new materials included in Moodle, and
especially the self-assessment questionnaires. Although students believe that the practices did not
have great difficulty, they are very useful for the professional world. They also believe that the
evaluation questionnaires are well suited to the contents of the subject and are very similar to the self-
assessment, although the distribution of the difficulty of the questions should be improved.
Finally, it is worth to mention that the blended-learning methodology using learning educational
resources based on digital documents, web packages and interaction tools, has been successfully
applied to provide more flexibility and autonomy to the learning process and it has improved students’
marks, in comparison with classical teaching and learning.
REFERENCES
[1] Falconer, I., Littlejohn A. (2007). “Designing for blended learning, sharing and reuse”. Journal of
Further and Higher Education, 31(1), pp. 41-52.
[2] Advanced Distributed Learning (2010). “SCORM Standard”. On-line:
http://www.adlnet.gov/Technologies/scorm.
[3] Moodle (2010). “LCMS Moodle”. On-line: http://moodle.org.
[4] Gil, P., Candelas F.A., Jara, C.A. (2011). Computer Networks E-Learning Based on Interative
Simulations and SCORM. International Journal of Online Education, 7(2) pp. 15-23,
doi:10.3991/ijoe.v7i2.1638.
[5] Graham, C. R. (2005). “Blended Learning Systems: Definition, Current Trends, and
FutureDirections”. Bonk, C. J. & Graham, C. R. (Eds). Handbook of Blended Learning: Global
Perspectives, Local Designs. San Francisco, CA, USA: Pfeiffer Publishing.
[6] Wimba (2010). Wimba Create. Web: http://www.wimba.com.
[7] Gil, P., Candelas, F.A., Pomares, J., Puente, S.T., Corrales, J.A. et al, (2010). Using Moodle for
an automatic individual evaluation of student’s learning. 2nd Int. Conf. on Computer Supported
Education, pp. 189-194.
[8] EJS web page: http://www.um.es/fem/EjsWiki/ (September 2011)
[9] Candelas, F.A., Gil, P., (2009). Practical experiments with KivaNS: a virtual laboratory for
simulating IP routing in Computer Networks subjects. Research Reflections and Innovations in
Integrating ICT in Education 3, pp. 1415-1418.