Content uploaded by Silvia Gaftandzhieva
Author content
All content in this area was uploaded by Silvia Gaftandzhieva on Mar 22, 2016
Content may be subject to copyright.
International Conference on e-Learning’15
150
Automated e-learning quality evaluation
Rositsa Doneva, Silvia Gaftandzhieva
Abstract: The quality assurance and evaluation of e-learning is of high priority for any up-to-date
higher educational institution. The paper is devoted on automation of related processes on the basis of a
modern approach for integration of heterogeneous software systems. It presents the context, substance and
objectives of a study related to the automated data retrieval in e-learning quality evaluation. It describes two
specific experiments on such integration for e-learning quality evaluation in terms of students’ satisfaction
and in terms of compliance with appropriate quality standards.
Key words: System Integration, Service Oriented Integration, e-Learning Quality Assurance,
Automated e-Learning Quality Evaluation.
INTRODUCTION
Quality(evaluation) being a key instrument of quality assurance and quality
enhancement in higher education (HE) is one of the most typical components of the
Bologna process.
Since e-learning technology is becoming an integral part of contemporary learning
activities in all learning modes offered by higher educational institutions, hence the
evaluation of e-learning quality is assumed as a top priority.
The monitoring, control and evaluation of the e-learning quality are key elements of
more and more institutional and national quality systems (i.e. internal and external
evaluation) of HE in Europe (e.g. Norway, Sweden, Great Britain, Bulgaria). The majority
of the academic community understands the importance of this issue – something
demonstrated by the large number of publications on the topic. Various experiments have
been conducted to evaluate the quality of e-learning in the terms of its various components
(like learning outcomes, learning process, academic staff, learning materials and activities,
infrastructure, students satisfactions, etc.), based on different relevant quality models and
standards (e.g. Context-Inputs-Process-Product of Stufflebeam, Reaction-Learning-
Behaviour-Results of Kirkpatrick, etc.).
nteresting examples in this field are the suggested models and approaches for
quality evaluation of e-learning: as a whole – in terms of planning, development, process
and product [8]; of the e-learning environments from the students’ perspective [5], of the
electronic educational resources [7], of the learners’ satisfaction (including learners with
special educational needs) [6] etc.
Another extremely discussed aspect of e-learning quality assurance and evaluation is
the need of automation of its related processes. The fact that the automated approach is
typical for this form of education logically implies the conclusion that it is appropriate to use
automated tools in evaluating its quality. This is the only possible manner to use effectively
and in full degree all the data that have been collected and stored in electronic format
during the organization and conduction of the e-learning.
The paper is dedicated namely on this important aspect of the subject. It presents the
context, essence and the purposes of a study on a modern approach for integration of
heterogeneous software systems and its application for automated data retrieval in e-
learning quality evaluation. Two specific experiments are described, carried out during this
study, with regard to such integration for e-learning quality evaluation in terms of students’
satisfaction and in terms of compliance with appropriate quality standards.
AUTOMATED DATA RETRIEVAL IN E-LEARNING QUALITY EVALUATION
(CONTEXT, ESSENCE AND PURPOSE OF THE STUDY)
Regardless of the needs for evaluating the e-learning quality, no matter whether for
internal or external evaluation, or for HE quality assurance and improvement, it is based
on the respective criteria and regulatory procedures. Usually such criteria systems (quality
International Conference on e-Learning’15 151
models) are very detailed and include a large number of evaluation criteria. For example,
the Swedish ELQ model for HE e-learning quality, consists of ten quality aspects, in
relation with which a total of 33 distinct criteria and 12 sub-criteria are evaluated while in
Bulgaria, the criteria system for evaluation of distance study programmes includes 46
quality indicators.
The quality evaluation, applying such standards, requires the collection, analysis and
interpretation of a huge amount of data in terms of learning materials used; infrastructure;
e-learning environment; tools and intensity of communication, cooperation and
interactivity; the application of a student assessment system; flexibility and adaptability of
the learning process; student and faculty support; team qualification and experience, etc.
On the one hand, the above is an argument for the automation of these activities. But
on the other hand, it poses a number of problems – it requires extraction and processing of
data from different information systems, which are often based on a different server,
operating system, communication platform, database etc. These problems are essentially
related to the need of integration of heterogeneous information systems.
Similar problems completely affect not only quality assurance, but also all aspects of
the automated information servicing of a typical university and of all various activities
necessary for the implementation of educational, management and administrative
functions of the institution as such.
The paper presents some of the results of a study, which aims to apply the
advantages of the modern approach for integration of heterogeneous software
applications, called Service Oriented Integration, in the field of HE.
The idea (fig. 1) is to apply the typical software architectural model Enterprise Service
Bus (ESB), based on Service-Oriented Architecture (SOA), to the integration of university
information systems. In accordance with the model, each university software system is
represented once by the relevant Web Services and is integrated into the so-called
“Integration Service Bus”. It allows the combination of these lower level services into
higher level business services by the Business Service Bus, so that they adequately meet
different new requirements, initiatives or changes in the information servicing of the
university.
In order to improve visibility, without any loss of generality, the scheme shown in fig.
1 reflects only the part of the studied architectural model which is of interest to the HE
aspect under discussion in this paper – e-learning quality evaluation. The scheme
illustrates:
Software applications: Learning Management System (LMS), Centralised system
for generating queries and reports and decision making (called University
Business Intelligence System – UBIS), other university systems and external
information systems;
Web Services: LMS Services, UBIS Services and other services;
Business services: Internal Evaluation of a Field of Study (FS), Internal Evaluation
of a University Unit, Internal Evaluation of the University and External Evaluation.
So far in the frame of this study a set of the web services necessary to represent the
different university systems to the Integration Service Bus has been specified (for some
examples see fig. 1). Various experiments have also been conducted on the software
implementation of web services, as well as on their usage for integration of different
university systems.
Two of these experiments, related to data retrieval from the LMS, aiming to automate
the quality evaluation of the conducted e-learning are presented in the following sections
of this paper. The results of such evaluation could be used within the framework of
different procedures for internal and external evaluation of a university and its educational
activity. The precise software systems, used in the experiments, are as follows:
one of the most widespread open source LMS – Moodle and
International Conference on e-Learning’15
152
specially developed UBIS, called UBIS-Jaspersoft, based on the popular software
Jaspersoft BI Suite, which allows the development of Business Intelligence
solutions for organizations of various type, including higher educational
institutions.
Figure 1. Integration of university information systems
EXPERIMENT: STUDENTS SATISFACTION
The first experiment is related to the integration of UBIS-Jaspersoft and LMS Moodle.
It aims to extract, analyse and interpret data about students’ satisfaction by the conducted
e-learning.
Figure 2. Electronic survey
For studying the students’ opinion the typical for each similar quality management
system approach is used, namely by conducting a survey. It is accomplished in an
electronic form. In order to fulfil this, the Moodle module Feedback is used as a tool for
creation of templates with questions that can be further used multiple times. The already
created survey template is included as part of the learning activities in each electronic
course (e-course) in order to be fulfilled by all the participating students after completion of
the training. The electronic questionnaire (see Fig. 2) contains around 50 questions for
quality evaluation of the conducted e-course in terms of its different characteristics, such
as course documentation and educational goals; learning materials and activities; team for
provision; communication; assessment methods; feedback; etc.
International Conference on e-Learning’15 153
Once the questionnaires are completed, the data is stored in the system’s database
and can be used for analysis of the results through the LMS Moodle module Feedback.
The problem that has to be solved within the scope of the experiment is that the built-in
module Feedback offers opportunities for generation of summarised and statistical reports
about the poll result only within a specific course and its participants.
In order for the obtained data from the survey to be used in accordance with the
purpose stated above (at internal and external evaluation in HE), it is necessary to provide
automated software tools for the synthesis and analysis of the results at a more general
level, e.g. for all e-courses in a field of study or for all e-courses in a scientific field (SF).
In solving this problem, the experiment is carried out in 4 (four) steps using the
system UBIS-Jaspersoft and the capabilities of its basic software Jaspersoft BI Suite for
creating reports and analyses by retrieving data from different sources, for storing and
organizing reports in a repository and for presenting them in the preferred by the user
form. Jaspersoft offers powerful tools for integration with various user software
applications through shared web services.
In Step 1 UBIS-Jaspersoft is integrated with Moodle database, which is set as a data
source for retrieving of the data from the students’ surveys and creation of reports,
reflecting student satisfaction. It should be noted that besides to relational databases (such
as the Moodle database) JasperSoft can be connected to just about any data source,
including JDBC, XML, CSV, Hibernate, POJO etc.
In Step 2 templates of analytical reports are developed, which can be later used to
generate the real reports containing summarised results from the conducted survey or
from other similar surveys, related to e-learning quality evaluation.
For templates description JasperSoft supports:
wide range of tools for visual design and good report layouts, representing data in
the form of tables, charts, or crosstabs-based reports;
a number of scripting languages (Java, Groovy, JavaScript, etc.) for construction
of expressions in order to declare report variables, to perform various calculations,
grouping data into the report, to specify report text field content, or to customize
further the appearance of report objects;
extensive set of query languages, including SQL, HQL, EJBQL, etc. for retrieving
data from different data sources.
In the present case 6 (six) templates are developed. They can be used in automated
surveys for the quality of e-courses in a concrete FS or SF in order to obtain
summarised information as follows:
the number of e-courses, in which conduction of the survey is planned (with added
questionnaires) by each FS offered in the university training;
a list of e-courses by the corresponding FSs, in which conduction of the survey is
planned;
the number of e-courses by FSs, where the survey is already conducted (with
completed questionnaires);
a list of e-courses by FSs, where the survey is conducted along with the
corresponding number of participants in the survey (number of completed
questionnaires);
summarised results of the survey by FSs;
summarised results of the survey by the evaluated characteristics of e-courses
and by SFs.
The templates include a presentation of the summarized results both in the forms of
tables and charts. Fig. 3 shows a screenshot from the template dedicated to generate the
summarised results of the survey by SFs during its design.
The templates include a presentation of the summarized results both in the forms of
tables and charts. Jaspersoft creates a source code in XML format for each designed
International Conference on e-Learning’15
154
template. The implemented templates are compiled in a special internal format and are
stored in the Jaspersoft repository, which is realized in Step 3. By this manner they can be
both used by the level of the very same UBIS-Jaspersoft system and by other external
application for the generation of the relevant reports, that are filled with data from the given
data source (Moodle database). The completed reports can be exported to a specified
document format (PDF, XLS, XLSX, XML, HTML, XHTML, CSV, DOC, etc.).
Figure 3. Template designing
Step 4. realizes the ultimate goal of the experiment to integrate LMS Moodle (as
external application) with UBIS-Jaspersoft through the shared UBIS-Jaspersoft web
services, represented in the integration service bus (fig. 1).
A new module is developed as a supplement to the LMS Moodle (as a Moodle report
on a system level), which adds to the system the already implemented reporting
functionality (Step1-3), using the corresponding web services. This allows the generation
of reports based on the above-mentioned templates by the LMS level.
This reporting functionality can be used in a similar way to expand the functionality of
other systems and for the development of higher level business services.
EXPERIMENT: CONFORMITY OF E-LEARNING QUALITY WITH STANDARDS
The purpose of the second experiment is to provide opportunities for LMS Moodle to
provide data to other specialized tools for automation of internal or external evaluation
procedures (other software systems or high level business services). That is why, the
extracted data must allow conformity verification of training conducted in LMS Moodle with
appropriate quality standards.
In accordance with the selected software architectural model, 4 (four) experimental
web services are developed to support corresponding Moodle integration. The services
and their functions are selected in such a way as to provide data (extracted from the
Moodle database), applicable in evaluation of some of the most common indicators of the
known e-learning quality standards. Table 1 describes web services, the results returned
by the functions implemented by them (from conceptual viewpoint) and respectively in the
evaluation of which quality indicator, they can be used.
International Conference on e-Learning’15 15
5
Table 1. Web-services for evaluation of e-learning quality indicators
Web
Service
Web Service Functions Results Quality Indicato
r
1. A list of team member names providing e-learning, the relevant e-courses and
their role (administrator, teacher, tutor, evaluator, educational content
designer, author, quality manager, etc.) in each of the courses
Team for provision of design,
implementation and
maintenance of e-learning
2. A list of various learning activities and resources used in the training (lesson,
page, book, quiz, assignment, chat, choice, database, forum, glossary)
and their number for each e-course
Provision of e-learning with
virtual learning activities and
resources
3. Teachers’ and students’ activity in usage of tools for synchronous communication
for each e-course
Teachers’ and students’ activity in usage of tools for asynchronous
communication for each e-course
Usage intensity of communication tools (synchronous and asynchronous) by
teachers and students
System for control and
stimulation of students’ and
teachers’ activity
4. Participation duration in e-learning (real use of activities and resources) by the e-
students (incl. in different courses)
Sustainability of the ensured
e-learning infrastructure
Web services are developed according to the documentation for the creation of web
services in Moodle [3, 4]. It is envisaged the functions of each service to provide
alternative access for client applications based on different protocols for client-server
communication (XML-RPC, SOAP, REST) and also to support the exchange of data via
the most popular for this purpose protocol (XML, JSON, AMF). The implemented web
services were tested through specially developed client applications.
CONCLUSION
The study and experiments, exposed in the current paper prove the feasibility of the
Service Oriented Integration approach for HE needs. It provides a common base for
integration of the full range of heterogeneous university information systems (providing
student admission, training, teaching, researches, management, educational marketing,
quality control, etc.) in order to improve the management, performance, reliability and
especially openness of a university to offer new services in response to specific needs.
The results described in the paper are achieved as part of a project for the
implementation of the studied architectural model carried out at the Plovdiv University [1].
The final stage of realizing an overall modern solution for automated quality evaluation of
e-learning is yet to take place. The building of the Integration Service Bus will be
completed by presenting through web services the student administration system (which
will provide data about students, curricula, etc.), academic staff development system
(providing data for teacher competences) etc. Based on this, it will be possible for different
processes related to assurance and evaluation of the quality of education to be
automatized (presented on the Business Service Bus level).
REFERENCES
[1] Gaftandzhieva S., Doneva R., Integrating Moodle with university information
systems, Proceedings of National Conference „Education and research in the Information
Society“ (G. Totkov and I.Koichev), Association for the Development of the Information
Society, Sofia, ISSN 1314-0752, 39-48.
[2] Kawachi Paul, Quality Assurance for OER: Current State of the Art and the TIPS
Framework, eLearning Papers, Issue 40. Assessment, certification, and quality assurance
in open learning, ISSN 1887-1542, pp. 3-13, January 2015.
[3] Moodle Docs, Adding a web service to a plugin, http://docs.moodle.org/
dev/Creating_a_web_service_and_a_web_service_function, last access 7.04.2015.
[4] Moodle Docs, Web services API, http://docs.moodle.org/dev/Web_services_API,
last access 7.04.2015.
International Conference on e-Learning’15
156
[5] Noaman A., A. Ragab, A. Fayoumi, A. Khedra, A. Madbouly, HEQAM: A Deve-
loped Higher Education Quality Assessment Model, Proceedings of the 2013 Federated
Conference on Computer Science and Information Systems, pp. 739–746, 2013.
[6] Rabai Latifa Ben Arfa, Neila Rjaibi, Assessing Quality in E-learning including
learner with Special Needs, Proceedings of The Fourth National Symposium on
Informatics, Technologies for Special Needs, April 23-25, 2013.
[7] Wetzler Ph., St. Bethard, K. Butcher, J. Martin, T. Sumner , Automatically
Assessing Resource Quality for Educational Digital Libraries, Proceedings of the 3rd
Workshop on Information credibility on the web table of contents, Madrid, Spain, 2009.
[8] Zhang W., Y L Cheng, Quality Assurance in E-Learning: PDPP Evaluation Model
and its Application, The International Review of Research in Open and Distance Learning,
Research Articles, Vol. 13, No3, pp. 66-82, June 2012.
ABOUT THE AUTHORS
Prof. Rositsa Doneva, PhD, Plovdiv University “Paisii Hilendarski”, Department of
Electronics, Communications and Information Technology, Е-mail: rosi@uni-plovdiv.bg.
Silvia Gaftandzhieva, PhD student, Plovdiv University “Paisii Hilendarski”,
Department of Computer Science, Е-mail: sissiy88@uni-plovdiv.bg.
The paper has been reviewed.