Content uploaded by Peter Mazohl
Author content
All content in this area was uploaded by Peter Mazohl on Dec 20, 2016
Content may be subject to copyright.
Chapter 2:
Quality Assurance
in Blended Learning -
a Quality Framework
Edited by: Peter Mazohl (European Initiative for Education, Austria)
Harald Makl (European Initiative for Education, Austria)
Data collection: Sophia Zolda, Kathrin Zehrfuchs
Final checks: Sylvia Mazohl
It is suggested that quality development is a constant negotiation process in which all
stakeholders should participate in a common effort to define and implement quality in a
continuous, improved way.1
1 Ehlers, Ulf (2007) p 97
Disclaimer: This project has been funded with support from the European Commission. This publication reflects the views only of the
BladEdu consortium, and the Commission cannot be held responsible for any use, which may be made of the information therein
Content of Chapter 2
2. Quality Assurance in Blended Learning – a Quality Framework ................................................................... 3
Description of a versatile quality framework for blended learning ..................................................... 3
2.1.1. Preconditions for quality frameworks .......................................................................................... 4
2.1.2. The quality framework developed in the project......................................................................... 5
Overview of the developed Quality fields ............................................................................................ 7
2.2.1. Quality of the Institution .............................................................................................................. 7
2.2.2. Enrolment ..................................................................................................................................... 8
2.2.3. Course .......................................................................................................................................... 9
2.2.4. Learning Environment and Learning Phase ................................................................................ 10
2.2.5. Assessment and Evaluation ........................................................................................................ 12
Access to quality development in educational courses...................................................................... 13
Sources ............................................................................................................................................... 15
List of Figures
Figure 2-1: Three fields involved in the quality framework .................................................................................. 4
Figure 2-2: Layer model (see: Varlamis, Apostolakis) ........................................................................................... 5
Figure 2-3: ISO process model (basic structure) ................................................................................................... 6
Figure 2-4: Defined quality fields as described in this paper ................................................................................ 6
Figure 2-5: The institution's quality ...................................................................................................................... 8
Figure 2-6: The enrolment .................................................................................................................................... 9
Figure 2-7: The course itself ................................................................................................................................ 10
Figure 2-8: Issues and elements of the learning environment ........................................................................... 12
Figure 2-9: Assessment and evaluation .............................................................................................................. 13
List of Tables
Table 2-1: Common features in an LMS .............................................................................................................. 12
1. Quality Assurance in Blended Learning – a Quality Framework
Blended learning enables the learner to learn and to study in a very special way. The teaching is split into on-
site learning and distance-learning phases. In spite of the on-site teaching, which is regulated very strictly by
time factors and the work in the group, the distance learning enables the learner to decide when, how and how
fast to learn.( Stein, Graham, 2014)
The system seems to provide a lot of freedom to the learner (Deschacht; Goeman, 2015) – on the other hand,
a learning success is expected from the learner. To evaluate the learning success an appropriate evaluation
system is necessary. Using well-defined indicators does not make it difficult to find out the learner´s success (if
the learner achieved a learning success).
The learner’s success is only one special part in the teaching and the learning process. An overall system must
exist to ensure the quality of the complete course. That makes it necessary to care for a certain level of quality
assurance during the course. To provide this quality assurance a well-defined and embracing quality framework
must exist.
Quality in teaching is an issue of increasing importance – for educational organisations as well as for learners.
Quality normally is defined using standards. These standards only can be developed by authorized institutions,
in Europe for example the ISO institution. These standards for blended learning are missing from the ISO norms.
There exists the ISO/IEC2 19796 since 2005, but it is not completed yet. The ISO/IEC19696 provides a process
model focusing on course providers including learners as well (Pawlowski 2007). The three main parts cover
1. The documentation of processes for the development and the implementation of a quality
management system
2. The analyses of an existing quality management system and the evaluation focusing on amendments
3. Re-structuring of processes and organisational units to provide a change management
These ISO/IEC norms are the first international standards for quality management with a focus on eLearning
(which is only one part of blended learning). These definitions provide a model that must be adapted to the
teaching conditions of a specific teaching institution or course provider. The missing standards for the on-site
teaching can be taken from other fitting ISO norms (for example from the ISO 900X family for educational
organisations).
That causes the definition of a specific quality framework based on the described process model. The current
project enhances the descriptions by a special versatile quality framework focusing on the learner’s needs.
Description of a versatile quality framework for blended learning
Developing a new quality system in an organisation means that quality objectives and instruments are
implemented for the core processes. For example, that covers analyses of learners´ needs, design of learning
systems, providing tutor support or performing assessments.
The process model serves as a guide to specify those objectives. An organisation should go through the
processes of the model and should answer the following questions for each process:
1. What is the main quality objective for a process?
2 International Organization for Standardization / International Electrotechnical Commission
2. Who are the responsible actors?
3. Which methods or instruments can be used to assure quality?
4. How can we measure the success of the quality objective?
Pawlowsky (2007) mentions the possibility of the development of quality profiles for organisations including
objectives, methods, relations and people involved. This matches to the proposal of the consortium.
The processes therefore just serve as a guideline to discuss quality and to set specific objectives in order to
reach the best outcome or results.
1.1.1. Preconditions for quality frameworks
Quality educational programs begin with the development of quality courses. Quality courses either need
standards for the quality assurance or a quality framework considering all necessary issues for an appropriate
quality assessment (Chao 2003).
Pawlowsky (2007) mentions that the needs of users and their organisations should be the main emphasis of
quality standards (and quality frameworks). He also mentions the awareness of teaching organisations, that
quality is important, but the adequate instruments are missing to fulfil the needs and to meet requirements.
Therefore, they cannot easily adopt quality approaches in their organisations.
The development of a quality framework for blended learning is extreme complex because there exist three
main fields, which must be connected:
Figure 2-1: Three fields involved in the quality framework
• On-site teaching quality covers the quality of the teaching environment as well as the
quality of organisation of the teaching and the other related typical on-site teaching
issues.
• Distance learning quality focuses on the virtual learning environment (VLE), the provided
material for students, the students’ support, and other issues typically related to distance
learning.
• Transfer and connection describes the methodology used to teach and assess the
competence oriented learning outcomes. That means the methods to split the defined
learning outcomes and dedicate partly to the distance learning or the on-site teaching.
Varlamis and Apostolakis (2010) define four layers for a typical blended learning course: the pedagogical layer,
a technical layer, a social layer and finally an organisational axis. This model is based on evaluation criteria for
learning systems.
Figure 2-2: Layer model (see: Varlamis, Apostolakis)
• The pedagogical aspect covers the quality of the learning process (this can be evaluated
by the reaching of the pedagogical targets). The pedagogical targets should be clear (and
appropriate communicated) for all learners. Tutors are responsible for performing
educational tasks (that implicates a well-defined tutorial support).
• The technical aspect means basically the infrastructure of the organisation.
• The social aspect addresses to a learning community. Learning (to reach the targets) is a
kind of communal effort.
• The centre of all learning and teaching activities is the organisation.
The mentioned model could be a solution to develop a quality framework; nevertheless, the access to the
quality framework in the current project is focusing on the learners´ needs and not on course evaluation.
Therefore, the project defines a quality framework based on the learner, the activities of the learner during the
course, the environment of the learner and other issues in direct context with the learner. The requirements
for the institutions are part of the framework as well as the necessary preconditions for teachers.
1.1.2. The quality framework developed in the project
The research work of the project defines a different model using the existing ISO/IEC 19796 to enhance the
definitions with a focus on the learner as the center of the learning and teaching process.
The process model of the ISO/IEC is a guide for the development of learning scenarios (Pawlowsky 2007). The
process itself is split into seven different parts. That are the
• Needs analysis, the
• Framework analysis, the
• Conception (or design), the
• Development (or production), the implementation, the
• Learning process, and finally the
• Evaluation (and optimization).
Mazohl (2015) gives a graphic overview of the process:
Figure 2-3: ISO process model (basic structure)
The goal of the ISO/IEC 19796-1 is to harmonize existing approaches to quality assurance. The description is on
an abstract level. There do not exist neither recommendations nor guidelines for quality management. The user
– in our case the course provider or providing institution – is responsible. These guidelines have to be developed
by the institution/course provider itself (Pawlowski 2007).
The consortium gives recommendations, how the guidelines – based on the description of the abstract model
– can be developed including the special focus on the learner’s needs.
Figure 2-4: Defined quality fields as described in this paper
The consortium proposes to acknowledge that quality of a learning process is not something that is delivered
to a learner by a course provider but rather constitutes a process of co-production between the learner and the
learning-environment. The organisational aspects are mainly researched and the currently used standards (like
ISO 900X) cover the quality fields of the course environment.
Ehlers (2008, p 21) gives some critical analyses of quality in the field of education and further education. He
explains that quality in education is a multilayer issue and that it is represented in various models.
Overview of the developed Quality fields
1.2.1. Quality of the Institution
The learner has to trust the institution and to feel sure that the teaching institution will undertake everything
to satisfy the learner’s needs. Here is a list of different aspects (mainly regulated by ISO or similar norms) which
are important for the learner.
• Administration
o Technical Administration
Students must be administrated well – that covers the procedure of the enrolment (including
appropriate privacy measures) as well as all the administrational stuff during the course
participation.
o Program Administration
The organisation cares for appropriate measures to announce the course, publish the content
and all issues related to the course.
• Documentation
The quality of documentation should cover the control of all documents, the change management,
course descriptions, produced and published materials, reports and other related issues.
• Resources of the institution / Course provider
o Technical Resources
Varlamis and Apostolakis refer to the technical quality aspect in their study and mention the
need of high quality of the used ICT as well as the used communication and learning platform
(Varlamis, Apostolakis 2015, p 27).
o Human Resources
The human resources are the available staff in the course and the additional people involved
in the course (for example administration).
o Financial Resources
The financial resources of the institution are necessary to ensure all learners to be able to
finish the course in an appropriate way.
• Teachers/Trainers
o ICT Skills
ICT and the use of ICT is a crucial quality criterion in modern teaching. Van Lakerfeld (2011, p
10) mentions ICT as a necessary tool in adult education – that must be expended to all kind of
education as well. Tilkin (2007, p 44 – 46) also mentions the need of ICT in teaching as an
important issue.
o Didactic Skills
Hénard and Roseveare (2012, p 17) explain in the report for the OECD that “there is evidence
that participation and engagement in professional development activities are related to the
quality of student learning.” Obviously is that relevant for the didactic skills.
• Instructional Design
Wright (2011, p 7) offers in his summary of quality criteria for evaluating the quality of online courses
a list of instructional strategies, which can be used as a checklist for quality in teaching.
Figure 2-5: The institution's quality
1.2.2. Enrolment
Athiyaman (1997) describes the context of student’s expectation and the student’s satisfaction. In literature,
the quality of enrolment is not described or mentioned. Therefore, the consortium developed guidelines for
quality assurance in context with the enrolment based on the learners needs.
The enrolment contains two different items that are crucial for learners: information about the course and the
practical handling of the enrolment.
• Course information
o Pre-Knowledge
A precise description of the necessary students´ pre-knowledge is an absolute quality criterion.
The course provider must care for a well-described list of requirements for students.
o ICT Skills
The necessary ICT skills must be published in an appropriate way to the students. High quality
institutions may offer special courses to take the students to the same (necessary minimum)
level.
o Structure of the course
The timetable, estimated workload, assessment rules, and other course related issues must be
published in a plain summary. Wright (2011) mentions that learners must be provided with
general information about the course structure.
• Enrolment procedure
o Registration
The registration procedure must be defined properly; also, the various steps for the enrolment
must be defined suitably. Students have to get all the information in time in a plain description.
Many big universities offer well-structured information and guide lines for their students and
may work as an example of good practice.3
o Handling
The teaching organisation provides o policy with well-defined and clear processes for the
learners during the enrolment.
o Access to software, materials, …
This information is necessary to inform the learner from the beginning about the necessary
tools and materials.
Figure 2-6: The enrolment
The literature research could not return any relevant results in context with quality and the course enrolment.
The consortium presents the results of the Wiener Neustadt workshop in the Quality in Blended Learning
Conference (2014) in this document. The findings need further investigation and should be topic of a broader
study in the future (Mazohl, Peter (Hg.) 2014).
1.2.3. Course
The course quality can be seen from the course organisers’ view as well as from the learners´. Jung and Latchem
(2007) found that most institutions apply the same quality criteria for eLearning (and Blended Learning) as for
the other modes of delivery. These criteria will satisfy the learner’s needs only partially.
The workshop results defined the quality criteria for the course itself as follows:
• Documentation
That covers the course documentation control, the description of the course, materials, and reports.
• Get to know the tutor(s)/teacher(s) and the other learners
3 See: Registration Guidelines (2015). Available online at
http://www.extension.harvard.edu/registration/registration-guidelines, updated on 3/6/2015, checked on
3/6/2015.
This special issue was mentioned by course participants of the EBI, but must be proved in further
studies.
• Well known course structure
Wright (2011, p 6) describes a well-defined course structure as a quality criterion.
Figure 2-7: The course itself
1.2.4. Learning Environment and Learning Phase
In blended learning courses, the learning environment describes the face-to-face phase as well as the distance-
learning phase. The UNESCO defines learning environment interestingly:
Learning takes place in multiple settings and the learning environment can be structured or
unstructured and the learning in different environments can complement each other
(Wright, Clayton R. 2011, p 6)
Graham (2013, pp 8) identifies technology, ownership, definitions and seat time, incentives and evaluation as
important issues in a blended learning environment.
The consortium identified various factors and issues of the teaching setting as an important quality criteria
(Mazohl, Peter (Hg.) 2014). The listed items below are all focusing on the learners´ needs.
• Teaching Activities
Teaching activities should be conducted targeting to the learning outcomes, which should be defined
as competence oriented learning outcomes as developed in the frame of LLP-Projects in Europe (van
Lakerveld & Zoete 2011).
• Distance Learning (e-Learning)
Ehlers (2007) defines quality in eLearning as an imperative for practitioners, and a huge demand for
learners.
• Media
The use of media is crucial in all learning environments, especially in the distance learning part of
blended learning courses. Holden and Westfall mention, “Media selection analysis must evaluate
general and specific criteria, including instructional, student, and cost aspects for each delivery
technology (or instructional medium) to ensure attainment of the instructional goal.” (Holden &
Westfall 2009, pp 13)
• Social Form and Contact
There are two important situations of interactions:
o Interaction Students/Students
o Interaction Students/Teachers
Besides the Learning Environment some other issues connected with the learner are to be considered.
• Motivation
Students’ motivation is a well-known success factor for learning and learning success. There exist many
different studies about motivation but well-fitting motivation strategies are missing. An interesting
access to motivate students is done by Chen and Jang (2010).
• Workload
During the planning of the course it’s necessary to estimate the students’ workload in context with the
learning and the necessary assignments. The strict planning of the course schedule must include
security time gaps and learner-centered time schedules.
• Communication
Mei-Jung Wang (2010) proves that – especially for students – the communication in the distance
learning becomes crucial.
• Technology
Blended learning is technology affected, especial ICT plays an important role. Amy Roche (2010, p 4)
mentions the importance of the role of technology and that both students and teachers have to be
comfortable with using technology (computers, software and internet).
• Equipment and software
The used equipment must be state of the art and a well-fitting software must be available in the
institution. If a special software is used the institution should offer the students special agreements to
use this software for their assignments in distance learning.
• Platform for the distance learning phase
To provide a high level of quality, the use of an eLearning platform is recommended (Aljawarneh,
Muhsin & Nsour 2011). The platform used for the distance learning must fulfil a list of criteria.
Stein and Graham (2014) give a simple definition of the common features in their book “Essentials for
Blended Learning: A Standards-Based Guide”:
Table 2-1: Common features in an LMS
This description covers the basic requirements to an eLearning platform that can be used for Blended
Learning.
Figure 2-8: Issues and elements of the learning environment
1.2.5. Assessment and Evaluation
Assessments must be planned and defined before the course; the information about the assessments should
be shared with the learner before the enrolment of the course (necessary pre-information). It could be useful
to define a Learning agreement where times, tools, goals and duties (of learners and the learners´ employer4 –
if existing - are well explained from the beginning
4 Not all learners are employed – there are freelancers or other people running their own business that are attending
Blended Learning courses.
The assessment should be the closing element of a blended learning course, followed by the evaluation of the
course (by the learner).
Two main criteria can be identified for the evaluation and assessment criteria fields:
• Planning and definition of assessments
The assessment criteria must be published to the students at the begin of the course (or be published
in the course description).
The definition of the assessment must be done in the design and definition of the course.
• Assessment execution
Assessments must be performed in a defined environment with properly defined assignments.
A properly defined assessment should cover
o The assessment’s testing definition (what is going to be assessed in which way when and how)
o The necessary description of the assessed competences or learning outcomes
o A qualified evaluation of the assessment’s results with fair feedback to the learners
• Evaluation
The evaluation should cover
o The evaluation of the course by the learners
o The evaluation of the teachers/trainers (by the learners)
o The evaluation of the course structure (in a defined quality management circle).
Figure 2-9: Assessment and evaluation
Access to quality development in educational courses
Ehlers (2008) mentions a list of different access methods to quality in teaching:
• Quality management (for example based on the ISO 9000:2000 standards)
• Evaluation techniques (for example self-evaluation) (Tilkin (Hg.) 2007, pp 8)
• Catalogues of criteria5 or checklists
• Benchmarking (Ubachs (Ed.) 2012)
5 The defined quality framework by the project’s consortium is an enhancement of a simple criteria catalogue and
covers more than simple checklists
• Accreditation and certification
• Seal of Quality
Sources
Aljawarneh, Shadi; Muhsin, Zahraa; Nsour, Ayman: E-learning Tools and Technologies in Education: A
Perspective 2011.
Athiyaman, Adee (1997): Linking student satisfaction and service quality perceptions. In: European Journal of
Marketing 31 (7), S. 528–540.
Bersin J. (2004). The Blended Learning Book. San Francisco; Pfeiffer.
Chao, Tracy (2003): Establishing a Quality Review for Online Courses. A formal review of online courses measures,
their quality in key areas and reveals changes needed for improvement. In: EDUCAUSE Quarterly (3), S. 32–39.
Chen, Kuan-Chung; Jang, Syh-Jong (2010): Motivation in online learning. Testing a model of self-determination
theory. In Computers in Human Behavior 26 (4), pp. 741–752. DOI: 10.1016/j.chb.2010.01.011.
Clark R.C. (2007): The New Virtual Classroom: Evidence-based Guidelines for Synchronous e-Learning, Pfeiffer.
Deschacht, Nick; Goeman, Katie (2015): The effect of blended learning on course persistence and performance
of adult learners: a difference-in-differences analysis. In: Computers & Education.
Ehlers, Ulf (2007): Quality Literacy — Competencies for Quality Development in Education and e-Learning. In:
Educational Technology & Society 10 (2), S. 96–108.
Ehlers, Ulf (2008): Qualität und Bildung. Bedingungen bildungsbezogener Qualitätsentwicklung in der Aus- und
Weiterbildung. Universität Duisburg-Essen, Duisburg-Essen.
Eulers, D. (2005): e-Learning in Hochschulen und Bildungszentren, Band 1, Buchreihe e-Learning in Wirtschaft
und Praxis, München, Wien, Oldenburg.
Graham, Charles R.; Woodfield, Wendy; Harrison, J. Buckley (2013): A framework for institutional adoption and
implementation of blended learning in higher education. In: The Internet and Higher Education 18, S. 4–14. DOI:
10.1016/j.iheduc.2012.09.003.
Graham, Charles R.; Woodfield, Wendy; Harrison, J. Buckley (2013): A framework for institutional adoption and
implementation of blended learning in higher education. In: The Internet and Higher Education 18, S. 4–14. DOI:
10.1016/j.iheduc.2012.09.003.
JISC (2006): Effective Assessment for the assurance of academic quality and standards in higher education;
Gloucester.
Jung, Insung; Latchem, Colin (2007): Assuring quality in Asian open and distance learning. In: Open Learning:
The Journal of Open, Distance and e-Learning 22 (3), S. 235–350.
Kirkpatrick D.L. & Kirkpatrick J.D. (2006): Evaluating Training Programs. The Four Levels. San Francisco; Berrett-
Koehler Publishers.
Mazohl, Peter (2015): Quality in Blended Learning, Wiener Neustadt, Mazohl.
Mazohl, Peter (Hg.) (2014): Quality issues for blended learning courses focusing on the learner. Quality in
Blended Learning. Wiener Neustadt, 20/02 - 22/02. EFQBL: BladEdu Consortium.
Ojstersek, N. (2009²): Betreuungskonzepte beim Blended Learning. Münster/New York/München/Berlin
Waxmann; 2009².
Pawlowski, Jan M. (2007): The Quality Adaptation Model: Adaptation and Adoption of the Quality Standard
ISO/IEC 19796-1 for Learning, Education, and Training. In: Educational Technology & Society 10 (2), S. 3–16.
Stein, Jared; Graham, Charles R. (2014): Essentials for blended learning. A standards-based guide. New York:
Routledge (Essentials of online learning series).
Tilkin, Guy (Hg.) (2007): Self evaluation in adult Life Long Learning. In coopperation with von Jaap Van Lakerveld,
Selma van der Haar
Ubachs, George (Ed.) (2012): Quality Assessment for E-learning: a Benchmarking Approach. European
Association of Distance Teaching Universities (EADTU). Heerlen.
Varlamis, S; Apostolakis, I (2010): A Framework for the Quality Assurance of Blended E-Learning Communities,
KES 2010, Part III, LNAI 6278, pp. 23–32, Springer-Verlag Berlin Heidelberg;
Wang, Mei-jung (2010): Online collaboration and offline interaction between students using asynchronous tools
in blended learning. In Australasian Journal of Educational Technology 26 (6). Available online at
http://www.ascilite.org.au/ajet/ajet26/wang.html.
Wright, Clayton R.: Criteria for Evaluating the Quality of Online Courses. Grant MacEwan College, Edmonton,
Canada. Instructional Media and Design.
