Content uploaded by Paul Bacsich
Author content
All content in this area was uploaded by Paul Bacsich on Jun 16, 2015
Content may be subject to copyright.
Bacsich 1 June 2015
The Showcases Reports
Quality Agencies in Europe and their approach to e -learning and open learning
(Deliverable 2.2 Appendices A, C and R)
Paul Bacsich
Sero Consulting Ltd
Consultant to EADTU
With assistance from
George Ubachs, Giles Pepler, Keith Williams
EADTU, March 2015
(republished separately June 2015)
Disclaimer: This research was conducted as part of the European Union-funded project SEQUENT –
Supporting Quality in E-learning European Networks. This project was supported by the European
Commission, DG EAC, under the Lifelong Learning Programme: ERASMUS Accompanying Measures.
However, sole responsibility for this report lies with the authors; and both the Commission and the
SEQUENT partners are not responsible for any use that may be made of the information contained
therein.
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 2 June 2015
Contents
0. Executive Summary .............................................................................................................. 4
1. Quality Agencies in Europe and their approach to e-learning and open learning ............... 5
1.1 Overview............................................................................................................................... 5
1.2 Specific countries with activity in QA in e-learning .............................................................. 8
1.3 Quality for OER and MOOCs ............................................................................................... 11
2. Exemplar university quality processes for e-Learning ....................................................... 13
2.1 TU Delft Education Quality Assurance ............................................................................... 13
2.2 University of Bristol ............................................................................................................ 14
2.3 Manchester Metropolitan University ................................................................................. 15
2.4 Northumbria University...................................................................................................... 16
2.5 University of Oxford ........................................................................................................... 17
2.6 University of Warwick ........................................................................................................ 18
2.7 University of Winchester .................................................................................................... 19
References ......................................................................................................................................... 21
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 3 June 2015
Acknowledgements / about SEQUENT project
This research was originally conducted as part of the European Union-funded project SEQUENT –
Supporting Quality in E-learning European Networks. This project, supported by the European
Commission, DG EAC, under the Lifelong Learning Programme, was funded from 1 October 2013 to
31 March 2015 (18 months).
The SEQUENT project aimed to promote excellence in the use of ICT in higher education, with a clear
goal to prepare European Universities in line with the European Modernisation Agenda and to make
higher education in Europe fit better to cross-border collaboration initiatives in the implementation
of innovative and ICT-enhanced partnerships. To this end, the project based itself on models that
had been developed by previous EU-funded projects and other internationally recognised models
that enhance the quality of ICT in higher education. The project raised awareness within the
European higher education community on the importance of a mainstreamed ICT uptake through
project events and the partners’ large memberships.
The consortium of the project was composed of the European Association of Distance Teaching
Universities (EADTU) as the coordinator and the European Association for Quality Assurance in
Higher Education (ENQA) as the partner. EFQUEL, the European Foundation for Quality in e-learning,
had been a partner until it closed down in December 2014, after which event the remaining work
they had to do was taken over by EADTU staff and consultants.
SEQUENT thanks the following partners, institutions and individuals for supporting and contributing
to this research:
EADTU, in particular George Ubachs
EFQUEL (until its dissolution in December 2014)
ENQA
ICDE and its advisors
QAA and its advisors
D-TRANSFORM consortium
QA-QE SIG, in particular Harvey Mellar
QQI
NVAO
Cyprus: University of Nicosia
Cyprus: Open University of Cyprus
England: University of Hertfordshire
Hungary: Budapest University of Technology and Economics
Italy: Polytechnic of Milan
Lithuania: Kaunas University of Technology
Netherlands: Technical University of Delft
Portugal: Universidade Aberta
Scotland: Glasgow Caledonian University
Spain: UNED – Universidad Nacional de Educación a Distancia
Sweden: Dalarna University
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 4 June 2015
0. Executive Summary
This report was originally Appendices A, C and R to Deliverable 2.2 within Workpackage 2 of the
SEQUENT project. The aim of WP2 was to gather examples of good implementation of QA in
educational institutions.
It begins with an overview of quality agencies in Europe with particular reference to e-learning and
OER aspects. (There seems no other public source of such information.)
It is followed by a chapter on quality procedures within universities, and concludes with a set of key
literature references.
It is hoped to update this document from time to time.
Bacsich 5 June 2015
1. Quality Agencies in Europe and their approach to e-learning and open
learning
1.1 Overview
Across the EU, quality in higher education is overseen by a set of quality agencies, typically (but not
always) nationally-oriented. They have some or all of the following four functions:
1. Accreditation of institutions (in other words, whether they can legally call themselves
‘university’ or similar title) – sometimes this is done by a separate government department
2. Accreditation of programmes (for example, a Masters in Computer Science; so that students
can be granted loans, institutions draw down government funding for these students, etc.) –
often some kinds of universities have power to self-accredit programmes
3. Ongoing quality assurance of programmes – this is a standard feature
4. Ongoing quality enhancement of programmes – some agencies do not do that, others (such
as QAA Scotland) do; in yet other cases another agency looks after quality enhancement (as
in the past in England and at present, to some extent, in Ireland).
All university-level quality agencies across the EU are, or are working towards being, members of
ENQA, the European Association for Quality Assurance in Higher Education. There are 20 Member
States with organisations who are members of ENQA – eight countries are still to achieve this status,
but projects to achieve this are usually under way (most recently in Malta and Cyprus). The table
below gives some of the complexity.
Table 1: Agencies and organisations in ENQA
EU
AGENCIES AND ORGANISATIONS
AUSTRIA
AQ Austria – Agency For Quality Assurance and Accreditation Austria
BELGIUM
AEQES – Agency for Quality Assurance in Higher Education, Brussels
VLUHR – QAU – Flemish Council of Universities and University Colleges – Quality Assurance Unit,
Brussels
BULGARIA
NEAA – National Evaluation and Accreditation Agency, Sofia (under review)
CROATIA
ASHE – Agency for Science and Higher Education, Zagreb
CZECH REPUBLIC
Accreditation Commission Czech Republic, Prague
DENMARK
EVA – Danish Evaluation Institute, Copenhagen
The Danish Accreditation Institution, Copenhagen
ESTONIA
EKKA – Estonian Higher Education Quality Agency, Tallinn
FINLAND
FINEEC – Finnish Education Evaluation Council, Helsinki
FRANCE
CTI – Commission des Titres d’Ingénieur, Paris
HCERES – High Council for the Evaluation of Research and Higher Education, Paris
GERMANY
ACQUIN – Accreditation, Certification and Quality Assurance Institute, Bayreuth
AHPGS – Accreditation Agency for Study Programmes in Health and Social Sciences, Freiburg
AQAS – Agency for Quality Assurance through Accreditation of Study Programmes, Cologne
ASIIN e.V – Accreditation Agency Specialised in Accrediting Degree Programmes in Engineering,
Düsseldorf
EVALAG – Evaluation Agency of Baden-Württemberg, Mannheim
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 6 June 2015
FIBAA – Foundation for International Business Administration Accreditation, Bonn
GAC – German Accreditation Council, Bonn
ZEvA – Central Agency for Evaluation and Accreditation, Hannover
HUNGARY
HAC – Hungarian Accreditation Committee, Budapest (under review)
IRELAND
QQI – Quality and Qualifications Ireland, Dublin
LITHUANIA
SKVC – Centre for Quality Assessment in Higher Education, Vilnius
NETHERLANDS
NVAO – Accreditation Organisation of the Netherlands and Flanders, The Hague
QANU – Quality Assurance Netherlands Universities, Utrecht
POLAND
PKA – Polish Accreditation Committee, Warsaw
PORTUGAL
A3ES – Agency for Evaluation and Accreditation of Higher Education, Lisbon
ROMANIA
ARACIS – Agency for Quality Assurance in Higher Education, Bucharest
SLOVENIA
SQAA-NAKVIS – Slovenian Quality Assurance Agency for Higher Education, Ljubljana
SPAIN
AAC-DEVA – Andalusian Agency of Knowledge, Department of Evaluation and Accreditation, Córdoba
ACSUCYL – Quality Assurance Agency for the University System in Castilla y León, Valladolid
ACSUG – Agency for Quality Assurance in the Galician University System, Santiago de Compostela
ANECA – National Agency for Quality Assessment and Accreditation of Spain, Madrid
AQU Catalunya – Catalan University Quality Assurance Agency, Barcelona
FCM – Fundación para el Conocimiento Madrimasd, Madrid
Unibasq – Agency for the Quality of the Basque University System, Vitoria-Gasteiz
UNITED
KINGDOM
BAC – British Accreditation Council, London
QAA – Quality Assurance Agency for Higher Education, Gloucester
EHEA not EU
AGENCIES AND ORGANISATIONS
HOLY SEE
AVEPRO – Agency for the Evaluation and Promotion of Quality in Ecclesiastical Faculties, Rome
KOSOVO
KAA – Kosovo Accreditation Agency, Pristina
NORWAY
NOKUT – Norwegian Agency for Quality Assurance in Education, Oslo
RUSSIA
AKKORK – Agency for Quality Assurance in Higher Education and Career Development, Moscow
NCPA – National Centre of Public Accreditation, Yoshkar-Ola
NAA – National Accreditation Agency of the Russian Federation, Moscow (under review)
SERBIA
CAQA – Commission for Accreditation and Quality Assurance, Belgrade
SWITZERLAND
AAQ – Swiss Agency of Accreditation and Quality Assurance, Bern
EUROPEAN
AGENCIES AND ORGANISATIONS
Europe
EUA – Institutional Evaluation Programme
Europe
ECCE – The European Council on Chiropractic Education
As said earlier, eight Member States – Cyprus, Greece, Italy, Latvia, Luxembourg, Malta, Slovakia and
Sweden – do not currently have quality agencies who are members of ENQA. While it might be
reasonable for small countries to partner with nearby countries for quality purposes (e.g. Flanders
with Netherlands – and the same theory might suggest that Luxembourg, Latvia and Slovakia might
partner with nearby countries), the omission of Greece and Sweden (10 million population each) and
Italy (61 million) are particularly notable. However, it is known that Malta has set up a quality agency
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 7 June 2015
working towards ENQA accreditation, and the omission of Sweden is a temporary phenomenon due
to special circumstances.
Although ENQA held an invitational workshop on quality in e-learning in 2009, and a small selection
of the papers presented was published in a workshop report Quality Assurance of E-learning (Grifoll
et al, 2009) progress since then has been very slow, and the recent revision (ENQA, 2014) of the
ENQA Standards and Guidelines (the version in current force, the 3
rd
edition, dates from 2009), which
is subject to approval by the Ministerial Conference in May 2015 in Yerevan (Armenia), does not
make explicit reference to e-learning, or indeed to OER or MOOCs.
There is an argument that can be made that quality guidelines for e-learning (and thence MOOCs,
OER etc.) do not require a reformulation of the ‘standard’ quality guidelines. However, the danger is
that without a reformulation, an implicit bias remains towards the ‘presential’ learning found in
classrooms and seminar rooms.
Experts in the field are usually rather cautious about making definitive pronouncements. In particular
in Quality Assurance of E-learning Grifoll et al (2009) notes (our italics):
We can start by formulating the following question: are programmes based on e-learning
methods requiring a reformulation of the ESG? If we understand the ESG as a frame for
quality assurance, the answer to that question is, in my opinion, no. Let’s say “no” for the
moment. Reasonably, what will be needed, sooner or later within the e-learning realm, is a
general reformulation of some current educational policies and practices.
Similarly, in his chapter in the same report, Mulder of NVAO notes, in response to earlier work on
quality of e-learning emanating from NAHE in Sweden (NAHE, 2008):
In its discussion about the integration of e-learning criteria in the national quality assurance
system, the NAHE report formulates this conclusion: “In order for quality assessment of e-
learning to become an integral part of national quality reviews, aspects and criteria need to
be incorporated into the general basis for assessment.”
NVAO agrees with this statement, although our interpretation of “incorporated” may not
conform completely to the one intended by the NAHE report. We are convinced that
incorporating e-learning specific aspects and criteria does not necessarily have to entail a
revision of our assessment framework. NVAO’s assessment framework is an “open” one, and
is very well capable of accommodating input on e-learning. For example, if teachers have
been trained to become versed in the art of on-line tutoring, it is possible – and relevant – to
elaborate on this under the assessment frameworks’ “Quality of Staff” standard. This is just
an example.
The issue is thus whether this “reformulation” has taken place, or is taking place fast enough. The
actual amount and percentages of tertiary-level online learning taking place currently across Europe
is unclear, with key reports in progress such as The Changing Pedagogical Landscape not available,
but figures for the number of distance learners in countries such as UK and Sweden and earlier
reports on virtual campuses such as the 2009 Handbook from the Re.ViCa project (Bacsich et al,
2009) make it clear that even five years ago online learning was widespread across Europe, even if
(then) only at a small percentage of institutions and a relatively small percentage of students.
However, with only a few exceptions (UK and Netherlands/Flanders in particular) this range of e-
learning activity, now including OER and MOOCs, is not reflected in quality agency reports.
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 8 June 2015
The soon-to-be approved revised Standards and Guidelines (ENQA, 2014) do start off well. On page 4
they state (our italics):
Broader access to higher education is an opportunity for higher education institutions to
make use of increasingly diverse individual experiences. Responding to diversity and growing
expectations for higher education requires a fundamental shift in its provision; it requires a
more student-centred approach to learning and teaching, embracing flexible learning paths
and recognising competences gained outside formal curricula. Higher education institutions
themselves also become more diverse in their missions, mode of educational provision and
cooperation, including growth of internationalisation, digital learning and new forms of
delivery. The role of quality assurance is crucial in supporting higher education systems and
institutions in responding to these changes...
However, the rest of the document does not follow this through. In our view, it should be possible to
avoid explicit remarks in the main text about new kinds of learning, provided that the range of
examples – with appropriate footnotes and references, establish the breadth of provision being
analysed.
1.2 Specific countries with activity in QA in e-learning
Most countries, in Europe and beyond, are in the process of improving their quality systems for
universities. In the international arena, quality is seen by governments as a key component of
reputation. In several countries – often in Asia – where there is a somewhat separate distance
education sector, there are also moves to define, implement and refine specific quality systems for
distance learning providers (Jung, 2007).
However, with some very rare exceptions, none of these quality systems take account of e-learning.
(In fact, none of them take account of any kind of non-standard learning – e.g. in the workplace or at
overseas campuses – taking place outside the classical paradigm of on-campus lectures and
associated activity.) While it can be argued that this is perfectly acceptable when e-learning is a
minor or non-strategic component of a course offering (for example when access to library books is
replaced by access to e-books) the case seems far less convincing when a whole programme is
offered by e-learning, as it is from many university providers across Europe – especially (in the EU) in
UK, France, Ireland, Sweden, Spain, Italy, and Germany – and (beyond the EU) especially in Norway,
Turkey and Russia.
Bacsich (2012) observed, rather depressingly:
There have been a number of attempts from experts to address this issue – but all have so
far failed. The reasons are varied but probably boil down to the following three: quality of e-
learning is not (yet) an issue for international agencies; quality of e-learning is never a high
enough priority for a national government; and that those involved in quality both within
universities and the national agencies are by and large conservative personalities.
Since then things have improved, slightly, as noted below.
United Kingdom
The United Kingdom has perhaps among EU Member States the longest recent history of national
funding for universities to engage in e-learning, with more or less continuous funding, from JISC and
the Higher Education Academy, for the last ten years, even if tailing off in the last three years.
Indeed, funding from JISC for infrastructural and technical aspects of e-learning goes back to the
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 9 June 2015
1990s. As an initially separate but increasingly integrated strand of development, distance learning
has been growing year on year since the 1990s, when the first “virtual campuses” at UK campus
universities joined the UK Open University in providing distance education to UK and beyond. Thus it
is not surprising that UK universities are among the most advanced in the breadth and depth of use
of e-learning.
Since quality agencies must in the main focus on the quality of offerings that exist, not those that
might exist, it is not surprising that there is usually a substantial time lag between the “flowering” of
e-learning in a country and its explicit recognition by its quality agencies.
So it was that as early (it may seem to other countries) as 1999 the UK quality agency, QAA,
produced a useful set of guidelines for distance learning, with many experts being consulted during
the drafting phase for these. The guidelines are no longer available on the QAA web site, but were
enormously influential, and were incorporated into procedures in several UK universities active in
distance learning (see Appendix B). The guidelines rapidly went out of date during the explosion of e-
learning in the early 2000s: but the guidelines were revised in 2004 (again the document is not now
available, but many universities took it into consideration). The process that QAA went through was
well described by Williams (2004).
In 2010 a Special Interest Group of experts spent considerable energy on a further revision,
producing an excellent report (QAQE SIG, 2010), only to find that the quality agency seemed to
ignore their views, instead producing generic guidelines. The current QAA set of quality guidelines is
expertly summarised in QAA (2015).
Sweden
After the UK, the most concerted attempt in recent years to define a regime for quality in e-learning
took place in Sweden when the National Agency produced a well-researched set of guidelines for
quality in e-learning (NAHE, 2008). Unfortunately for various reasons the guidelines were not
embedded into national quality processes. This was despite the fact that distance e-learning is a
significant fraction of total student learning in Sweden.
Netherlands
Like the UK, the Netherlands has a long history of state funding for e-learning, via the SURFNet
Foundation, but it has to be admitted, at a smaller scale, and there is still a substantial concentration
of distance learning at the Open University of the Netherlands, itself much smaller than the UK Open
University, rather than a wide spread across the sector, as is now the case in the UK. Thus perhaps it
is not surprising that the Accreditation Organisation of the Netherlands and Flanders (NVAO) has not,
it seems, spent much effort on guidelines for quality in e-learning: in fact some of its staff have
argued against the need for these (see earlier quotations). However, it has brought EADTU’s E-
xcellence system for benchmarking e-learning to the attention of institutions of higher education.
Estonia
In 2008 the e-Learning Development Centre formed a quality assurance task force to oversee the
process of awarding an “e-learning quality label”. Villems et al (2013) describe the process as follows:
In 1999, Estonian universities had only 14 elearning courses, but with thirteen years this
number has increased to more than 7000. The process of awarding the e-course quality label
was initiated in 2008. To run the awarding process, e-Learning Development Centre has
formed a quality assurance task force, consisting of experts from many different higher
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 10 June 2015
education organisations. The task force developed a manual with quality criteria, set up a 3-
tier process, beginning with self-evaluation, followed by organisation evaluation containing
also learners’ feedback and 3-member team expert evaluation. Applicants and experts are
encouraged to fill a feedback form about the application and evaluation process which helps
to improve the process from year to year. In 2011 the e-learning quality web which supports
all 3 tiers of the process was created. The quality web is also a good tool for collecting
statistics.
The paper contains full details of the process and the indicators used. It also establishes the need to
have a substantial amount of e-learning before such an adaptation of quality processes is needed.
Norway
As early as 1993, NADE (the Norwegian Association for Distance Education) developed quality
guidelines, using experts such as Erling Ljoså and Torstein Rekkedal (Bo, 2003). The standards were
revised twice, by NADE’s standing committee on quality in 1999 and 2001.
Little public trace of these guidelines remains now, but Bo (2003) reproduces a key table of the 16
guideline areas:
Conditions and
constraints
Implementation
Results
Follow up
Information
Counselling
External
Organisation
partners
Course
development
Cooperation
Guidance of authors
Choice of media
Evaluation during process
Course delivery
Students’
achievements
Course completion
Learning results
Documentation
Organisation
Evaluation
Reporting
Cyprus
In Cyprus, the Evaluation Committee Of Private Universities (ECPU)
(http://www.ecpu.ac.cy/en/index.html) is responsible for the quality approval of the University of
Nicosia’s programmes of study. ECPU is set up by the Council of Ministers of the Republic of Cyprus
following the provisions of the Private Universities (Establishment, Operation and Control) Law 109(I)
of 2005.
Lithuania
It should also be noted that LieDM (Lithuanian Association of Distance and e-Learning,
http://liedm.net/en/), a volunteer organization which unites all Lithuanian science, study, and
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 11 June 2015
education institutions which implement distance teaching and learning, has among its nine
Association Objectives:
To participate in preparation of distance and e-learning quality assurance procedures.
1.3 Quality for OER and MOOCs
The material in this section has been provided mainly by the POERUP network of OER experts in the
relevant countries. It focuses on what the quality agencies and related agencies are actually doing,
not what various researchers and lobbyists are trying to encourage them to do.
UK
Although there is as yet no mention of OER on the QAA web site, the QAA’s Statement on Massive
Open Online Courses (QAA, 2014) has been influential. The first UK credit-bearing MOOC has now
been developed by a UK university (Edge Hill University, a well-known flexible learning provider) and
the experience from that (Brabon, 2014) has been propagated by the QAA.
Despite ongoing concerns about issues of quality in both OER and online course materials in general,
no national rubrics have yet emerged against which to measure the quality of OER. However the OER
infoKit released under the UKOER Programme (2009-2012) includes a modicum of guidance for
those pursuing the use of OER. (Regrettably, a number of these documents have seemingly been
removed from the project wiki.)
Recent UK research into OER Quality has been undertaken by individual researchers, but results do
not indicate a systematic or comprehensive approach to the issue.
Netherlands (and Flanders)
Netherlands and Flanders share a joint quality agency for higher education, NVAO. It is a full member
of ENQA and is active in a number of projects in quality and benchmarking related to e-learning, but
not specifically in OER. However, NVAO has published a survey of MOOCs and online HE (NVAO,
2014).
France
There are no national quality procedures for OER in France – this is up to the individual provider.
However, to reinforce the quality of distance education, CHANED (n.d.) has set up a quality charter,
which 23 private institutions have signed and the quality charter covers resources, some of which are
OER.
Germany
The Framework Act for Higher Education (Hochschulrahmengesetz) sets out quality procedures for
universities. Federation and Länder have formulated general minimum requirements of a structural
and quantitative nature for institutions of continuing education and some Länder have adopted
specific quality assurance standards in their statutory provisions. However, none of these structures
mention OER and quality assurance of OER is largely ignored by the current systems – it remains up
to individual organisations to provide their own QA.
Ireland
The Irish Quality Agency for Higher Education, Quality and Qualifications Ireland (QQI), is a full
member of ENQA. It recently (QQI, 2013) published a Green Paper on the Recognition of Prior
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 12 June 2015
Learning, which raised key issues relevant to OER and MOOCs. It is believed to be working on
guidelines for online learning.
Latvia
Nationally, there are detailed quality assurance regulations for higher education (LIKUMI, 2013).
However, none of the government documents mention OER. Assuring OER quality remains in the
hands of individual institutions.
Norway
The Norwegian Agency for Quality Assurance in Education (NOKUT) is an independent government
agency that contributes towards quality assurance and enhancement in both higher education and
tertiary vocational education. It is a full member of ENQA. In its recent (NOKUT, 2013) self-reflection
on its activities there was no mention of OER or MOOCs and very little on e-learning, but the report
did note that certain aspects of flexible learning had been under study, for example flexible
professional education and other clues on the NOKUT site suggest that NOKUT is keeping such
aspects under review.
Poland
The Polish Quality Agency for Higher Education, Polish Accreditation Committee (PKA), is a full
member of ENQA. There is nothing non-trivial reported on any OER-related activities.
Portugal
A3ES, the Agency for Evaluation and Accreditation of Higher Education, is a full member of ENQA.
However there is no information on any activities in accreditation of OER.
Romania
The Romanian Agency for Quality Assurance in Higher Education (ARACIS) deals with QA in
universities and is a full member of ENQA. Universities are now less strictly controlled by the
Ministry. However, in the organisation there is no reference to OER, and quality assurance remains
the province of individual institutions and initiatives.
Spain
Quality procedures (in so far as they exist) are managed by providers and institutions. There are no
national procedures or systems for quality assurance of OER. However, and interestingly, Josep
Grifoll, Head of the Quality Assessment Department at AQU Catalunya, has written several articles on
quality in e-learning.
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 13 June 2015
2. Exemplar university quality processes for e-Learning
This is just a small sample of the university-specific quality guidelines publicly available in the UK and
Netherlands, the two most developed countries in such matters.
2.1 TU Delft Education Quality Assurance
The following is quoted from the introduction to the Education Quality Assurance Plan at TU Delft,
1
with emphasis by the editor of this Deliverable. The full plan is also available to the public.
2
A university that wishes to provide high quality teaching obviously must have a clear vision of teaching, and in particular of
education quality. TU Delft aspires to be such a university. A clear vision of education quality presupposes an institution-
wide system of quality assurance, in which the vision is described and with which quality can continuously be monitored
and improved. The system for institution-wide quality assurance described in this report aims to meet these requirements.
The aforementioned raises the question of TU Delft’s vision of education quality prior to the present system. It goes without
saying that this vision has long been entrenched in TU Delft. Testimony of this can be found in the Teaching Calendars, the
Plans of Action triggered by the inspections, the TU Delft instrument database for Faculty Education Quality Assurance
Cycles and the Logistics Quality Monitor. However, the various lines of approach were more or less disconnected prior to
this Quality Assurance System. In other words, TU Delft already had various programmes of quality assurance; what was
required was integration of these programmes into a single system.
In addition to the integration of the existing programmes in a new TU Delft framework, the Quality Assurance System also
needed to be expanded. To gain a clear picture of the quality of the degree programmes at the central level, the new
system incorporates programme-based annual reports. An important role in this system will be played by the impending
accreditation system, made up of Institutional Audits and programme accreditations. To acquire the classification ‘trusted
institution’ in an Institutional Audit, so that lighter programme accreditations can be applied, TU Delft has to demonstrate it
is ‘in control’ with regard to process related aspects such as quality assurance and personnel management. The annual
reports will be used as an instrument to achieve this.
Other new instruments are the Internal Programme Audits and the Internal Central Audit at TU Delft-wide level. These fulfil
a similar role to the midterm research review. A number of degree programmes already apply the programme audit
instrument. Using these audits, the university and the degree programmes can assess the quality of the teaching and
teaching-related aspects halfway through the Institutional Audit cycle of six years, so that there will be enough time to cope
with any weak points before the next Institutional Audit or programme accreditation.
The most important function of the TU Delft-wide quality system is to improve the quality of education. This can be
achieved if all involved parties are prepared to participate in an open discussion on the basis of the annual reports and are
willing to learn from each other’s best practices.
To recapitulate, both existing instruments and existing best practices have been used to the best possible effect in the
design of the new programme. The innovation is that these best practices have been combined with a number of new
instruments in an institution-wide framework. This new programme will demonstrate that TU Delft takes education quality
very seriously indeed!
1
https://intranet.tudelft.nl/en/targeted-info/education/onderwijsverbetering/education-quality-assurance/
2
https://intranet.tudelft.nl/fileadmin/UD/MenC/Support/Internet/TU_Website/TU_Delft_Medewerkers/Specifieke_ingange
n/Onderwijs/Onderwijsverbetering/Onderwijskwaliteit/doc/TU_Delft_Quality_Assurance_Plan.pdf
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 14 June 2015
2.2 University of Bristol
The Academic Quality and Partnerships Office, in a series of Annexes to the Regulations, states in
Annex 4, updated 26 January 2015:
3
7. Quality Assurance and Approval Processes
All new proposals for distance learning should be presented and considered in accordance with the agreed procedure for
new programmes, including consultation with relevant support services such as the Library and the Student Systems
Information Office. The proposal must be agreed by the School and Faculty before being presented for approval.
Distance learning programmes should conform to the University’s internal quality assurance mechanisms and the
University’s regulations. A sub-set of regulations for such programmes may need to be developed where this is not
possible.
8. Curriculum Design and Delivery
It is acknowledged that that the delivery of teaching via distance learning will vary by subject, depending the on the form of
teaching and content that is to be delivered. The following points should be followed wherever possible:
a) programme teams should consider how the curriculum is delivered, whether in a blended format or entirely
away from campus. Teams should consider whether students should be required to attend the University for an
induction session and/or other taught components as a mandatory part of the programme,
b) programme teams should consider how the learning outcomes from each unit, and those of the programme,
should be assessed. Consideration should also be given to whether the forms of assessment can be conducted
online (this will require that assurance can be provided that the assessment is completed by the student) or
whether students will need to attend the University or another designated venue to take summative
assessments;
c) students should receive feedback on their formative assessment, in accordance with University policy, in good
time to influence the next relevant activity or assessment, as set out in the Regulations and Code of Practice for
Taught Programmes,
d) the programme specification should make clear to students the academic, pastoral and other support that is
available to them in the School/Faculty, including outlining the appropriate forms and lines of communication
and expected timeframe for responses. Students should also be informed of how they can access central
services. Likewise staff need to be made fully aware of the time commitment that will be required of them (to be
available to respond to student requests). Good student support structures will be vital; a student away from
campus, must be able to interact with his or her peer group and communicate with the University (for academic,
administrative and pastoral reasons) and know they will receive a response in a set time. Cultural expectations
also need to be considered particularly for overseas students, e.g. the relationship between student and tutor
will need clear exposition,
e) programmes should provide opportunities for students on distance learning programmes to foster a community
of learners and for inter-learner discussions,
f) the experiences of students on a programme containing distance learning should be regularly monitored,
evaluated and updated, where necessary, at the Annual Programme Review meeting. Students should have
appropriate opportunities to provide formal feedback on their experiences of the teaching.
3
http://www.bristol.ac.uk/esu/assessment/annex/distancelearning.html
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 15 June 2015
2.3 Manchester Metropolitan University
This university has been active at scale in distance learning for many years. It has a thorough course
approval process with a 12-page form Developing Online Distance Learning Awards
4
of which the
form below is just a short extract:
Activity 1 Formal Proposal of a New Online Distance Learning Award or Revalidation of an Existing Award to include
Online Distance Delivery
Questions to Consider
Possible Actions
Is there a business case for delivering the award via
online distance learning?
Develop a sound and appropriately costed business case.
Complete market research to determine likely demand for the
award and proposed delivery model.
Is there a pedagogic argument for delivering the award
via online distance learning?
Develop a clear pedagogic rationale to support the proposal.
Complete market research to determine likely demand for the
award and proposed delivery model.
Is there an appropriate technical and administrative
infrastructure in place (centrally and locally) to support
the delivery of the award via online distance learning?
Engage with LRT, CELT, ITS, SAS and CASQE as early as possible to
assist in development of the course proposal.
Ensure that the appropriate ELSO (e-learning support officer) is a
full member of the course planning team.
Are all learning technologies being proposed within the
award supported by MMU?
Utilize MMU supported learning technologies within the proposal
unless these do not enable the model to be implemented. Discuss
with LRT as soon as possible in course planning.
Provide a technology specification as part of the proposal detailing
all learning technologies to be utilized along with details of access,
licensing, support models, etc.
Has a risk assessment been undertaken in relation to any
proposed learning technologies that are not controlled or
supported by MMU?
Undertake a risk assessment for any learning technologies not
supported by MMU providing full details of how these will be
supported and any technical issues will be mitigated.
Do licensing arrangements exist which will enable
distance learners to utilize all of the proposed learning
technologies?
Ensure appropriate licenses in place for all software, content and
tools used in the course. Consider how MMU-created content can
be controlled / licensed or made available on an open basis as
appropriate.
4
http://www.mmu.ac.uk/academic/casqe/event/docs/Online-DL-Guid-&-form.doc
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 16 June 2015
2.4 Northumbria University
The Guidelines on Approval of Distance Learning Programmes
5
require the supply of information on
the following topics:
3.1 Student Information
Procedures for preparing students to study the module
Were students provided with information about the:
Module aims
Learning outcomes
Learning and Teaching methods, e.g. delivery model, student workload, study schedule
Student support mechanisms
Assessment strategy, including formative and summative approaches
Induction into the technology e.g. eLearning Portal
How to be a distance learner, implications, approaches, commitment, just-in-time support, etc.
Broader induction approaches
3.2 Learning materials
Appropriateness of structure – fit for purpose
Relevance to module aims and learning outcomes – how are these flagged up within the learning materials, how
are learning packages structured – relevance, appropriateness, signposting for access
Appropriateness of materials to academic level – relationship to subject benchmarks, style, level of language,
intellectual rigour
Ease of navigation –avoidance of dead-ends, clear signposting, indication of where learner is an how to get back
to previous areas, ease of finding materials, adaptive release
Quality of materials – appropriateness to learner approach; copyright adherence
Accessibility – range and variety of means of access.
Adherence to Special Educational Needs and Disability Act (2001)
3.3 Assessment
Appropriateness of formative assessments used and provision of student feedback
Use of plagiarism detection measures
Summative assignment submission procedures
Provision of marks and feedback
Assignment return procedures
Tutorial arrangements
Electronic access to marks
Electronic submission and feedback approaches
3.4 Communication
Student support methods
Communication strategy
Provision of appropriate tools (only those which will actually be used
Service level agreement
Use of discussion room and other tools, e.g. blogs, WIKI
Student collaboration activity
Peer to peer & peer to tutor interaction and communication
3.5 Additional resources
Appropriateness of any other learning materials
Provision of reading list
Links to other resources
Collation of all links
Provision of glossary of key terms and acronyms
5
https://www.northumbria.ac.uk/static/worddocuments/ardocs/773481.doc
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 17 June 2015
Electronic reading lists
3.6 Other considerations
Copyright
Disability access, e.g. visual impairment, dyslexia
Download time
Software requirements – guidance on players/readers where specific software is used
It is worth noting that programme team wanting to develop distance deliver modules that have been approved for class-
based delivery can use the same module descriptor for the class delivery module, merely adding additional information
concerning the distance delivery in the sections concerning delivery location, learning and teaching strategy, notional
student workload, synopsis and resources for distance learning delivery.
2.5 University of Oxford
The Education Committee in 2008 produced extensive guidelines
6
correlated with the then-current
QAA guidelines. In particular it observed:
[introduction]
This document summarises the University’s guidance on Flexible and Distributed Learning (FDL) which at Oxford largely
covers programmes which are delivered using a large component of distance learning and/or e-learning. It draws on the
Quality Assurance Agency’s (QAA)1 guidance on FDL which is contained in their ‘Code of practice for the assurance of
academic quality and standards in higher education’. This is a statement of good practice which has been endorsed by the
higher education community, and is used by the QAA in its audit and review processes. This guidance has been drawn up in
consultation with the Technology-Assisted Lifelong Learning (TALL) group in the Department for Continuing Education, the
Learning Technologies Group (LTG) in OUCS and the Disability Office.
[and later]
5. Equivalence of standards
In the case of FDL, particular thought needs to be given to parallels and equivalences with face-to-face learning, e.g. in the
areas of provision of tutorial sessions, web-based conferencing and on-line contacts between tutors and student, and
between groups of students. Where some students study for the same award by FDL and some by traditional means, their
results and feedback on their experience should be compared and evaluated.
6
http://www.admin.ox.ac.uk/media/global/wwwadminoxacuk/localsites/educationcommittee/documents/notesofguidance
/Flexible_and_Distributed_Learning.pdf
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 18 June 2015
2.6 University of Warwick
The University of Warwick has, in its process for Approval of Distance Learning Courses,
7
a useful
Annex 3 which is a “Checklist of questions for departments to consider”:
Teaching, Assessment and Staffing
1. Has the department made realistic assessments of the staff time required to develop and update the course materials?
2. Will the department be able to offer a high quality course if student numbers are lower than expected or if key
members of staff leave the University?
3. Will the elements of teaching planned give all students a good chance of completing the course successfully? What
efforts are made to pace and structure students’ learning? Will the materials and contact with students make
departmental expectations of students sufficiently clear in terms of the volume and standard of work expected?
4. If electronic delivery methods are to be used, have all issues related to availability, standardisation and upgrades of
hardware and software both for students and staff been considered carefully? Have the training needs of staff and
students been assessed?
5. What lessons can be learned from experience of distance learning within the department, elsewhere in the University
or from other institutions? Are there any particularly relevant examples of good practice? Have efforts been made to
benchmark the proposed delivery against the experience of others in the sector?
6. Where staffs are using this mode of delivery for the first time, what training and support will they receive for
developing materials or providing guidance to students?
7. Does the course include an element of formative assessment, i.e. a non-credit bearing assessment, to enable staff to
gauge student progress and provide feedback at an early stage?
8. What proportion of assessment (formative and summative) should take place under controlled conditions and how will
the department verify the authenticity of assessed coursework?
9. Will there be any special arrangements for progression of distance-learning students (e.g. in relation to deferring
assessments, granting temporary withdrawal, the overall period of time allowed to complete the course)?
Costs and Resources
10. Has the department made a realistic assessment of all course costs including administrative support, production and
distribution of materials, copyright costs, IT requirements and training?
11. Is the department confident that all resources, including staff, IT and Library facilities and accommodation will be
available when required according to its proposed start date for the course?
12. What additional learning materials will be required to be successful on the course, e.g. library resources? Will there be
special requirements for some dissertation topics? If so how will these be addressed?
13. Has the department made all costs and requirements clear to students including those related to any attendance at
the University?
14. How will the department protect its intellectual property rights over teaching materials made available in hard copy or
an electronic format?
Communication
15. Does the department need to provide any special advice to students (UK, EU or overseas) beginning the course about
the learning methods, expectations, assessments etc? Are there University services outside the department which
should be drawn to students’ attention and arrangements made for them to be able to access remotely (e.g. library
and IT facilities, careers and welfare guidance?
16. Are the arrangements planned for student-student and student-staff interaction, including feedback on the quality of
modules, workable and how will their effectiveness be monitored?
7
http://www2.warwick.ac.uk/services/aro/dar/quality/categories/courseapproval/course/annex3.rtf.
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 19 June 2015
2.7 University of Winchester
The University, in its up to date Distance Learning Policy 2014/15,
8
observes that in general:
Programmes and modules, which are delivered by DL, are validated in the same way as any other programme or module. In
addition, the following key questions shall need consideration by the programme team and adequate documentation:
a) the management of the learning experience;
b) relevance, life expectancy, security and reliability of the learning platform and software applications;
c) staff skills and training;
d) student support arrangements; and
e) contingency plans in case of technological failure (to be discussed with appropriate University staff)
It then develops a comprehensive and paradigmatic set of detailed guidelines for staff, collaborative
partners, IT support and students.
3.4.1 Programmes shall ensure that students are provided with:
a) a delivery system of the programme or module that is secure, reliable, fit for its purpose, and has an appropriate
availability and life expectancy;
b) study materials that meet the expectations of the University in respect of the quality of teaching and learning-
support material for a programme or module leading to one of its awards;
c) an appropriate contingency plan, which would come into operation in the event of failure of the designed mode
of delivery;
d) from the outset of student’s study, at least one identified contact, either local or remote through email,
telephone and post, who can give constructive feedback on academic performance;
e) information about when and how they may contact staff, a clear schedule for the delivery of their study materials
and for assessment of their work;
f) a clear and realistic explanation of the expectations placed upon them for study of the programme or module,
and for the nature and extent of autonomous, collaborative and supported aspects of learning plus details of the
minimum technical requirements for the computer facilities required to access the online facilities;
g) information about the Library Services, detailed on the University’s Intranet > Information bank >Library
resources pages;
h) where appropriate, regular opportunities for inter-learner discussions about the programme, both to facilitate
collaborative learning and to provide a basis for facilitating their participation in the quality assurance and
enhancement of the programme;
i) appropriate opportunities to give formal feedback on their experience of the programme;
j) a copy of this Policy.
3.4.2 The University or Collaborative Partner shall ensure that:
a) staff who provide support to learners on their programmes have appropriate skills, and opportunities to receive
appropriate training and development;
b) the Programme Specification (for taught programmes) or Study Plan (for postgraduate research programmes)
details how the learner support that would normally be provided on site shall be delivered off-site (e.g. Student
Services, personal tutor and library resources,
c) support for learners, when normally provided by Student Services, is made available remotely wherever possible
and practicable;
d) staff with responsibility for assessment are capable of confirming that a student’s assessed work is the original
work of that student only, particularly in cases where the assessment is conducted through remote methods.
3.4.3 ITS,* who have responsibility for software for which it holds the licenses, shall ensure that:
a) the reliability of the delivery system within its remit is tested, and that contingency plans would come into
operation in the event of the system/network failing;
b) they provide support within normal working hours (currently Monday-Friday, 9-5).
* Where the Academic Resources Form has confirmed that the delivery system will not be supported by ITS, the
Programme shall be required to assume responsibility for the above.
8
http://www.winchester.ac.uk/Freedomofinformation/Publicdocuments/Documents/Distance%20Learning%20Policy%2020
14_15.pdf
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 20 June 2015
3.4.4 Students shall ensure that:
a) they have regular and reliable access to the internet with appropriate firewall protection and a computer that
meets the minimum technical requirements set by the programme;
b) they make their own arrangements for IT support to resolve technical failures relating to their ISP, firewall
protection and their computer hardware and software as the University can only answer queries or provide
support for University-owned equipment;
c) they ensure that they understand the basic terms and descriptions used in computing so that they can follow
instructions about how to use their computer to study and communicate;
d) they engage with the learning materials and mode of delivery;
e) they conform to the schedule for the programme delivery and assessment, monitor the receipt of materials and
alert the University if materials are corrupted or fail to arrive;
f) they take responsibility for developing their IT skills, where appropriate;
g) they regularly check their Unimail accounts to which essential information may be sent by the University.
Postgraduate research students only shall ensure that:
a) they are able to attend the University in person for Induction, Upgrade Viva, Thesis Proposal Viva, and a Viva Voce at the
end of their research study.
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 21 June 2015
References
Bacsich, P., Bastiaens, T., Frank Bristow, S., Op de Beeck, I., Reynolds, S. & Schreurs, B. (2009),
Reviewing the Virtual Campus Phenomenon: The Rise of Large-scale e-Learning Initiatives
Worldwide (the Re.ViCa Handbook),
http://revica.europace.org/Re.ViCa%20Online%20Handbook.pdf
Bacsich, P. (2012), “Impact of e-Learning in the 21st Century University”, in International Perspectives
on Higher Education, Bloomsbury.
Bo, I. (2003), E-learning, Flexible and Distance Learning – the Norwegian Approach and Experience
with Quality, presentation to EDEN conference, Rhodes, 2003, http://www.eden-
online.org/contents/conferences/annual/Rhodes/images/IngeborgBo.ppt
Brabon B. (2014), Talking about quality: Massive misalignment: the challenges of designing and
accrediting MOOCs, http://www.qaa.ac.uk/en/Publications/Documents/Talking-about-Quality-
MOOCs-Brabon.pdf
CHANED (Chambre Syndicale Nationale de l’Enseignement Privé à Distance) (n.d.), La charte qualité,
http://www.chaned.fr/charte.html
ENQA (2009), Standards and Guidelines for Quality Assurance in the European Higher Education Area,
3
rd
edition, http://www.enqa.eu/wp-content/uploads/2013/06/ESG_3edition-2.pdf
ENQA (2014), Standards and Guidelines for Quality Assurance in the European Higher Education Area
(ESG), Endorsed by the Bologna Follow-Up Group in September 2014,
https://revisionesg.files.wordpress.com/2014/10/esg-draft-endoresed-by-bfug.pdf
Grifoll, J., Huertas, E., Prades, A., Rodríguez, S., Rubin, Y., Mulder, F. & Ossiannilsson, E. (2009),
Quality Assurance of E-learning, ENQA Workshop report 14,
http://www.enqa.eu/indirme/papers-and-reports/workshop-and-seminar/ENQA_wr_14.pdf
Jung, I. (2007), “Editorial: Regional Focus on Asia Major: Changing Faces of Open and Distance
Learning in Asia, 2007”, International Review of Research in Open and Distance Learning, Volume
8, Number 1, http://www.irrodl.org/index.php/irrodl/article/download/1096/1915
LIKUMI (2013), Studiju programmu licencēšanas noteikumi, http://m.likumi.lv/doc.php?id=256418
Mayes, T., Adamson, V. & Plenderleith, J. (2008), Challenges and Realisations from the Higher
Education Academy/JISC Benchmarking and Pathfinder Programme: An End of Programme
Review by the Higher Education Academy, Evaluation and Dissemination Support Team, Higher
Education Academy, September 2008,
https://www.heacademy.ac.uk/sites/default/files/resources/Bench_and_PathFinalReview20080
926.pdf
NAHE (Swedish National Agency for Higher Education) (2008), E-learning quality: Aspects and criteria
for evaluation of e-learning in higher education, Report 2008:11 R,
http://www.hsv.se/download/18.8f0e4c9119e2b4a60c800028057/0811R.pdf
NOKUT (2013), ENQA Review of NOKUT: Self Evaluation Report – February 2013,
http://www.nokut.no/Documents/NOKUT/Artikkelbibliotek/Generell/Strategi/2013/ENQA%20R
eview%20of%20NOKUT%20selfevaluation%20Report_February%202013.pdf
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 22 June 2015
NVAO (2014), MOOCs and online HE: A survey,
http://www.nvao.net/page/downloads/NVAO_MOOCs_and_online_HE_A_survey_June_2014.pdf
QAA (2014), Statement on Massive Open Online Courses,
http://www.qaa.ac.uk/en/Publications/Documents/QAA-position-statement-MOOCs.pdf
QAA (2015), The Quality Code: A Brief Guide, February 2015,
http://www.qaa.ac.uk/publications/information-and-
guidance/publication/?PubID=180#.VUp1UPlVhBc
QAQE SIG (2010), QAA Code of Practice Section 2: Collaborative provision and flexible and distributed
learning (including e-learning): A Commentary & Critique, http://qaqe-sig.net/wp-
content/uploads/2010/09/QAQE_Report_final1.pdf
QQI (2013), Green Paper on the Recognition of Prior Learning,
http://www.qqi.ie/Publications/Green%20Paper%20-%20Section%204.7.pdf
Villems, A., Pilt, L., Plank, T., Varendi, M., Sutt, E. & Dremljuga-Telk. M. (2013), “Quality Assurance
Processes in E-Learning: An Estonian Case”, in InnoQual, The International Journal for Innovation
and Quality in e-Learning, 2013:1, http://papers.efquel.org/index.php/innoqual/article/view/16
Williams, P. (2004), Collaborative Provision, and Flexible and Distributed Learning (Including e-
Learning): QAA’s Revised Code Of Practice,
http://www.celt.mmu.ac.uk/ltia/issue8/williams.shtml
Quality Agencies in Europe and their approach to e -learning and open learning
Bacsich 23 June 2015
home.eadtu.eu
EADTU
P.O. Box 2960
6401 DL Heerlen
The Netherlands
Tel: +31 45 576 22 14
E-mail: secretariat@eadtu.eu
EADTU / www.eadtu.eu
Europe’s leading institutional association in online, open and flexible higher education
This work is licensed under a Creative Commons Attribution 4.0 International
License: https://creativecommons.org/licenses/by/4.0/ This license lets
others distribute, remix, tweak, and build upon this work (even
commercially) as long as credit is provided for the original creation. This is among the most
accommodating of CC licenses offered, and recommended for maximum dissemination and use of
licensed materials.
