Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
Human and Artificial Intelligence for the Society of the Future
European Distance and E-Learning Network (EDEN) Proceedings
2020 Annual Conference | Timisoara, 22-24 June, 2020
ISSN 2707-2819
doi: 10.38069/edenconf-2020-ac0027
289
DIGI-HE – A STRATEGIC REFLECTION TOOL ON DIGITALISATION
AT EUROPEAN HIGHER EDUCATION INSTITUTIONS
Ulf-Daniel Ehlers, Patricia Bonaudo, Baden-Württemberg Cooperative State University
Karlsruhe, Germany
Abstract
Digitalisation is an issue of growing importance at all higher education
institutions (HEIs). It is often developed and driven bottom-up. In this regard,
the intended self-assessment tool that the present paper aims to present “DIGI-
HE” will support higher education institutions in developing their individual
approaches to foster digitisation, methodological and conceptual approach. The
present paper will outline the methodological procedure of design and
subsequent validation of the tool. In a time when experimentation with, and
mainstreaming of digital technology use is progressing to develop holistic
strategies that encompass learning and teaching, research and innovation, as
well as cooperation and outreach DIGI-HE will represent a self-reflection tool
adapted to higher education to support the institutional efforts, to develop and
implement strategies, which purposeful and holistic in comprising both
missions, education and research. It will also furthermore attach particular
importance to the need for dialogue among all actors and stakeholders in
digitalisation, and address areas of activities relation to cooperation and
outreach, including internationalisation strategies and practices.
Introduction
Digitalisation is an issue of growing importance at all higher education institutions (HEIs).
It is often developed and driven bottom-up, i.e. by individuals and parts of the institution.
Recent studies report progress regarding the increased general acceptance of digital
learning and its strategic and more mainstreamed use. They show that a high number of
institutions report that they develop or enhance their strategies (Ehlers & Schneckenberg,
2020; Gaebel et al., 2018; p.60). Since digitalisation is often driven by individual actors
within institutions such as staff or departments, it develops bottom-up, often organically,
based on concrete needs, hence not particularly strategic (Gaebel et al., 2014; Haywood
et al., 2015; Ehlers & Schneckenberg, 2010; Ehlers, 2014).
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 290
ISSN 2707-2819
For institutional leadership, the development of an institutional strategy, or a general
holistic view on the progress of digital transformation and organisational development in
higher education institutions a valid concept and good measure is often missing. This is
hindering digital transformation in higher education from an institutional perspective. For
institutional leaders it is often difficult to assess the diversity of needs, to get an overview
on what is in place, and thus to support strategic planning in this area accordingly. In this
regard, the planned self-assessment tool that the present paper aims to present “DIGI-HE”
will support higher education institutions in developing their individual and strategic
approaches to foster digitisation. It is directed at leaders in higher education institutions.
Such a tool already exists for schools, the SELFI tool, but not for HEIs and above all not at
a European level (https://ec.europa.eu/education/schools-go-digital). Project funded by
the European Union; project partners: Dublin City University (DCU), Duale Hochschule
Baden-Württemberg (DHBW), Jÿvaskyla University (JYU), Vytautas Magnus University
(VMU).
In this paper, we present the conceptual approach of the instrument (section 2). In
section 3 we discuss the four methodological approaches, benchmarking, bench learning,
peer review and self-assessment, which underlie the methodological approach of DIGI-HE.
Finally, in section 4 we address the specific innovative aspects of DIGI-HE.
Conceptual Approach of DIGI-HE
Due to their identity as professionals, self-evaluation procedures enable the evaluators to
simultaneously assume the role of experts in the subject of evaluation and therefore to
contribute their intimate and detailed field and process knowledge of the subject of
evaluation. In contrast to evaluation procedures that are carried out by external experts,
internal evaluation procedures, due to their self-determination character, are also
associated with a high level of motivation and, as a result, a willingness to take up an active
part of those stakeholders involved. As a result, they hold the great potential of a higher
identification with the evaluation results. The evaluations and conclusions derived from
the evaluation process are considered valid by the stakeholders involved, since they are
assessed as adequate and validated in a dialogical process of communication. The
willingness to translate the results into concrete actions can thus be increased.
The approach of developing and implementing institutional strategies, strengthen
institutional leadership and enable participatory approaches has been tested successfully
also in other thematic areas. For instance, in 2018, EUA together with its partners in the
EU-funded EFFECT project, developed a self-reflection tool for the enhancement of
learning and teaching in higher education, which took the form of an institutional
strategies support package. As part of this self-reflection tool, guiding questions were
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 291
ISSN 2707-2819
prepared for institutional stakeholders (leadership, academic staff), in order to assist them
with a sustained reflection on how to improve learning and teaching practices.
The DIGI-HE will be built on existing tested, successful tools, the SELFIE tool for schools,
and the DigCompOrg Framework. It will transfer the successful approach of the SELFIE
and its lessons learnt from the school sector to the higher education sectors, thereby
adapting it to the specific needs of the higher education institutions, considering other
European and international instruments, which received sector recognition. This approach
is hitherto not existent for higher education and responds to higher education sector needs.
In order to ensure a broad and up-to-date knowledge of the needs of universities data on
the strategic development of digitalisation in higher education institutions are gathered
through a survey of higher education institutions. DIGI-HE will be based on new data on
the state of play of and strategic development goals and challenges in digitalisation. The
resulting knowledge will inform the development of the tool. This concept of approaching
digitalisation from the angle of current challenges ensures that the tool will reflect the
actual needs and demands of the sector and key stakeholders. As higher education
institutions are in the process of taking up digitalisation, the need to develop more strategic
and holistic approaches is commonly acknowledged, at institutions as well as at policy
level.
In order to help individual higher education institutions to develop strategies and build
capacity for digitalisation the tool will emphasize participatory approaches, in that it
demonstrates the need and shows ways for involving the different parts of the institutions,
staff and students, and external stakeholders into the process of digital strategy
development and implementation. Thus, it will support institutions to develop their own,
tailored approaches for digitalisation, in line with their mission goals and specific needs,
and result into enduring, sustainable fit-for-purpose approaches.
The DIGI-HE tool aims to reflect a European perspective on digitalisation of Higher
Education, accordingly the different systems in the individual European countries must be
considered. The project consortium and the Advisory Board, which will be composed of
experts, to bring together different complementary expertise, i.e. higher education
institutions from different European countries, a national rectors’ conference and major
European organisations representing universities and colleges (EUA, EURASHE and
EDEN). This will ensure the development of approaches and outcomes that will respond
to the needs of diverse types of institutions from different countries.
Methodological approach
DIGI-HE will be a self-assessment tool. Self-assessment in higher education is a well
introduced method. However, it may be coupled with a peer-review in which institutions
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 292
ISSN 2707-2819
are receiving reflections based on their self-assessment from peers outside the institutions.
Both processes can well be used in a benchmarking and bench-learning exercise which can
lead to inter-institutional learning processes.
This section reflects on the methods of benchmarking, bench-learning, peer-review and
self-assessment that are used as methodological approach of DIGI-HE. DIGI-HE is based
on four central methods for quality evaluation and validation that all have distinct
characteristics and potential advantages and disadvantages. These methods
benchmarking, bench-learning, peer-review and self-assessment need to be discussed
briefly as a foundation for further development of DIGI-HE.
Benchmarking
Kamiske and Brauer (2003; p.10) summarize benchmarking as the process of measuring
and comparing one’s products, services or processes with the best competitors or with
acknowledged market leaders (“best in class”). The target of benchmarking is to learn by
comparing with others, to identify best practice and to adapt these methods, processes etc.
for the own organisation to achieve improvement, and in the long-term, market leadership
or excellence. Benchmarking originates from the field of reverse engineering that is related
to physical products, but the concept has been transferred to services and processes. Camp
(1989; p.15) highlights that benchmarking leads to objectives when best practices are
transferred into targets that may in many cases be of a qualitative nature and indicate a
direction of development in the longer term rather than exact (quantifiable) short-term
operational targets.
According to Camp (1989; p.16), benchmarking is divided into four main stages, planning,
analysis, integration and implementation with a concluding fifth phase: maturity.
Planning includes the identification of the benchmarking object, organisations that should
be included in the comparison as well as a definition of methods and execution of data
collection. This phase includes a self-analysis or self-assessment that is already considered
to be helpful to identify areas for improvement (Lemmergaard, 2009; p.182). In a second
stage, gaps are identified and possible future performance levels identified. The third
phase; integration, includes communication of results and setting of targets for the next
phase; implementation. Within this phase, a plan for implementation is developed, the
implementation is executed and results are checked. The closing phase maturity includes
aiming for a leading position and integration of benchmarking into the organisation’s
processes (Kamiske & Brauer, 2003; p.15). There are a number of restrictions associated
with benchmarking. First, learning from benchmarking is mostly concerned with the past
as the “best in class” organisation or competitor has already achieved this level. Secondly,
it is argued that there is high uncertainty in identifying the “best in class” organisation or
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 293
ISSN 2707-2819
best practice. Only a step-by-step approach towards a relatively well performing
organisation is supposed to be possible (Kamiske & Brauer, 2003; p.18). Lemmergaard
(2009; p.182) suggests that benchmarking only focuses on current best practices and is not
a source of innovation and possible future best practice. Becker and Gerhard (1996; p.784)
also argue that one implicit assumption has to be made to consider benchmarking
successful; best practice cases are not specific to one organisation but need to be
generalizable to be transferable.
Bench-learning
Connected to benchmarking is the term bench-learning. According to Freytag and
Hollensen (2001; p.26) who define bench-learning as the “process of learning from the ‘best
in class’ with the purpose of integrating these best practices in all organisational levels of
the company.” Benchmarking is the foundation and bench-learning is the learning process
that follows it. Thought needs to be given to the issue of if, and how these identified best
practices could be transferred to the own organisation as well as how skills and processes
could be improved (2001; p.30). They also distinguish bench-action as the actual
implementation of all changes that have been set as targets (2001; p.31).
Peer-review
Peer-reviews have high practical relevance for external evaluation as well as quality
assurance and development according to Gutknecht-Gmeiner (2008; p.19). They are
central within research and publishing of scientific papers (Weingart, 2001; p.284); on
different levels within the field of education (educational systems, institutions, or levels of
individual learners or teachers); in the fields of medicine, nursing, social work- and
business-related professions, such as auditing (Gutknecht-Gmeiner, 2008; p.60).
Gutknecht-Gmeiner regards the classification of peer-review as a method of evaluation as
rather complex (2008; p.51). Normally, peer-review refers to an external evaluation by
experts that belong to a different organisation or, in some cases, may also be colleagues
within the same organisation (for instance many cases of peer-review in teaching). This
review is supposed to support an organisation or individual in its efforts on quality
assurance and development. In contrast to other external evaluators, peers act on the same
level, as they possess similar knowledge, experience and competencies as the evaluated
individuals/members of an organisation and originate from similar organisations or
contexts (Gutknecht-Gmeiner, 2008; p.51). An example is the evaluation of teachers by
teachers or the fundamental peer-review culture in scientific publishing. Gutknecht-
Gmeiner also mentions the synonym “critical friend” (2008; p.52) for a peer-reviewer that
demonstrates the special relationship. Peer-review is also clearly classified as a qualitative
method (that may also include quantitative data as a foundation for analysis). Gutknecht-
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 294
ISSN 2707-2819
Gmeiner summarises that a peer-review may be used for both formative and summative
evaluation purposes, depending on the specific review’s design (2008; p.51).
The design includes the question about what exactly is reviewed by the peer-reviewers, and
there are a number of options that differ in how close the review is to the subject of interest
(e.g. an institution). A review could focus on the self-assessment report/results to assess if
the report has been drafted well. The review could also be more detailed and comprise itself
of a review of the data that is provided to prove the conclusions in the self-assessment
report, and/or additional onsite visits and interviews with stakeholders could be performed
by the peer-reviewers to access more data. Finally, the peer-review could focus on the
subject of interest itself, for instance whether a teacher observes a colleague in a teaching
situation (Gutknecht-Gmeiner, 2008; p.51). Gutknecht-Gmeiner (2008; p.51) summarizes
that in practice, despite the fact that there are numerous possible designs for peer-reviews,
the following procedures for a peer-review are well established: The basis for the review is
formed by an extensive self-evaluation by the institution or individual, followed by the
external evaluation that is, in many cases, accompanied by an on-site visit and leads to a
final review report by the reviewers.
Peer-reviews are associated with a number of advantages. If conducted in a formative way,
peer-reviews include a (mutual) learning possibility as the work together with the external
expert may provide insights for improvement and development on both sides.
Furthermore, peer-reviews are considered to have a relatively good cost-benefit ratio
compared to an evaluation by potentially expensive and specially trained auditors. It is also
assumed that colleagues or experts from the same field of expertise are perceived as more
acceptable than external evaluators with possibly no expertise in the evaluated subject
matter (Gutknecht-Gmeiner, 2008; p.23).
There are also a number of disadvantages and open questions discussed. Srciven (1991;
p.255) considers peer-reviews to be “extremely shaky” and mentions (among others) halo-
effects, a possible secret-contract bias or the fear of possible retaliatory action as
problematic factors, but he sees a lot of potential for improvement. Furthermore, it is
questionable whether peers always possess the necessary qualifications in the field of
evaluation as well as communication, social and personal skills and whether they are in all
cases as objective as reviewers who are not subject matter experts (Gutknecht-Gmeiner,
2008; p.23).
Self-assessment
Both benchmarking as well as a peer-review require a prior self-assessment by the
organisation. Kamiske and Brauer (2003; p.18) consider (in a broader context) a self-
assessment to be a regular and systematic analysis of strengths and weaknesses of a
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 295
ISSN 2707-2819
company or organisation to determine one’s position, to identify areas for improvement
and to transfer these insights into implementation. The initiative for a self-assessment is
supposed to come from the organisation itself and the organisation that conducts the
assessment is also responsible for the process. Usually, the self-assessment is conducted
against a set of criteria as for example in the standards which are put forth by DIGI-HE.
Table 1: For DIGI-HE the following existing instruments have been analysed and assessed
against their usefulness for supporting digital transformation in higher education:
SELFIE
OLC Quality Scorecard Suite
DigiMirror
DigCompOrg Framework
Blended learning self-assessment tool
DigiComEdu Framework
Leibniz Digital benchmarking tool
Opeka
HEInnovate
Ropeka
Maturity Model for Blended education
Oppika
ACODE benchmarking
NSQ National Quality Standards for Online
Education
Jisc Digitally-capable Organisation
QQI Blended Learning Guidelines
ENQA - Considerations for quality assurance of
e-learning provision
Quality Matters
"E-xcellence - Quality Assessment for E-
learning: a Benchmarking Approach (3rd
edition)"
Technology Enhanced Learning Accreditation
Standards
Innovative Aspects of DIGI-HE
A European self-evaluation tool such as SELFIE does not exist for higher education
institutions. Tools that exist or are under development tend to focus on digital learning and
teaching or digital skills for individual member across the university such as staff or
students. Furthermore, they seem to resemble more external quality measures and thus
they usually require a team of external reviewers, and result into an assessment statement
or score of the university’s maturity and quality in digital or online education, based on
external criteria. Therefore, DIGI-HE will be the first of its kind.
With emphasize on self-review, self-reflection, and measures for strategic institutional
development a tool like the DIGI-HE will support the institutional efforts, to develop and
implement strategies, which are purposeful and holistic in comprising both missions,
education and research. It will also emphasise the need for dialogue among all actors and
stakeholders in digitalisation, and address areas of activities in relation to cooperation and
outreach, including internationalisation strategies and practices. The tool itself will require
the participation of a wider range of the members of the institution, including students,
teachers, researchers, administrators and technical staff, and leadership at different levels.
This will enable the institutional leadership to explore perceptions and perspectives of
different stakeholders across the institution, i.e. leadership, teachers, researchers,
administrative staff, technical and IT staff, and students. DIGI-HE will cover the main
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 296
ISSN 2707-2819
missions and areas of activity where digitisation plays a role, in a holistic way: learning and
teaching, research and innovation, governance and management, and cooperation and
outreach (including internationalisation).
Guidelines and exchanges with institutions will encourage and support intra-institutional
dialogue and cooperation. As one of the areas covered by the tool will be cooperation and
outreach, this will also consider the role of external parties (companies, NGOs, schools
etc.), and point to mutual benefits that this could render for the institution’s digitalisation.
Consequently, DIGI-HE will inform the intra-institutional strategic dialogue and
collaboration processes, thus contribute to mainstream approaches, improve support and
more transparent structures for digitalisation, and increase the proactive participation of
staff and students. Using the tool and join the community will also enable contacts,
exchanges, learning and networking among institutions. From other contexts, peer
learning has been confirmed as an invaluable means for innovative institutional
development – following lessons learnt from EUA’s annual European Learning and
Teaching Forum, and thematic peer learning groups bringing together leadership with
responsible for education.
The DIGI-HE tool will be useful for institutions at different level of maturity in their digital
approaches, which is important given significant country and institutional differences. For
instance, according to the Trends 2018, in Greece, Germany, Kazakhstan, Sweden,
Switzerland, the UK and Ukraine, all higher education institutions confirmed that digital
learning is becoming part of the institutional strategy – which has only been the case at a
third of the Polish, a quarter of French and Italian, and a fifth of Czech institutions.
Conclusion
This paper has provided a reflection on strengths and weaknesses of central methods for
quality evaluation and validation within the DIGI-HE. Benchmarking and the connected
concept of bench-learning are concerned with learning and improvement by comparing
with other “best in class” organisations or products and adapting these identified best
practices within the own organisation. Self-assessment is described as a (usually) regular
and systematic analysis of strengths and weaknesses of a higher education institution to
determine their potential, to identify areas for improvement that are usually conducted
with the help of criteria catalogues.
A peer-review is characterised as a qualitative method for external evaluation by an expert
on the same level that is, in many cases, based on a prior self-assessment and focuses on a
review of existing data. Peer-reviews are considered to offer a mutual learning opportunity
(“critical friend”) in addition to the review itself and are regarded as offering a relatively
good cost-benefit ratio.
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 297
ISSN 2707-2819
As the first of its kind, DIGI-HE will provide a self-assessment tool specifically designed
for HEIs, which will enable HEIs to self-reflect and self-review. It will promote the internal
dialogue and thus acknowledges the reality of the progress of digitalisation hitherto taking
place through a bottom-up process but enables to shape this process in a strategic sense.
Encouraging inter-institutional collaboration and support as well as densification in a
network will promote mainstream approaches, better support and more transparent
structures for digitisation.
References
Camp, R. (1989). Benchmarking: The Search for Industry Best Practices That Lead to
Superior Performance. Quality Resources.
Ehlers, U.-D. (2014). Open Learning Cultures. A Guide to Quality, Evaluation and
Assessment for Future Learning. New York, Heidelberg: Springer.
Ehlers, U.-D. (2019). Future Skills – Die Zukunft des Lernens, die Zukunft der Hochschule.
Wiesbaden: Springer.
Ehlers, U.-D., & Schneckenberg, D. (eds.) (2010). Changing Cultures in Higher Education
– Moving Ahead to Future Learning. A Handbook for Strategic Change. New York:
Springer International.
Freytag, P., & Hollensen, S. (2001). The process of benchmarking, benchlearning and
benchaction. The Tqm Magazine, 13, 25-34. doi: 10.1108/09544780110360624
Gaebel, M., Kupriyanova, V., Morais, R., & Colucci, E. (2014). E-Learning in European
Higher Education Institutions. Results of a Mapping Survey conducted in October-
December 2013. Retrieved from https://eua.eu/resources/publications/368:e-learning-
in-european-higher-education-institutions.html
Gaebel, M, Zang, T., Bunescu, L., & Stoeber, H. (2018). Trends 2018. Learning and
teaching in the European Higher Education. Retrieved from
https://www.eua.eu/downloads/publications/trends-2018-learning-and-teaching-in-
the-european-higher-education-area.pdf
Gutknecht-Gmeiner, M. (2008). Externe Evaluierung durch Peer Review. Wiesbaden:
Springer VS.
Hartz, S., & Meisel, K. (2004). Qualitätsmanagement. Studientexte für
Erwachsenenbildung. Gütersloh: Bertelsmann.
Haywood, J., Weller, M., Williams, K., Connelly, L., & Henderikx, P. (2015). The changing
pedagogical landscape. New ways of teaching and learning and their implications for
higher education policy – Study. Retrieved from https://op.europa.eu/en/publication-
detail/-/publication/f43a8447-7948-11e5-86db-01aa75ed71a1
Ehlers, U.-D., & Bonaudo, P.
DIGI-HE – A Strategic Reflection Tool on Digitalisation at European Higher Education Institutions
European Distance and E-Learning Network (EDEN) Proceedings 298
ISSN 2707-2819
Henderikx, P., & Jansen, D. (2018). The Changing Pedagogical Landscape: In search of
patterns in policies and practices of new modes of teaching and learning. Retrieved from
https://tinyurl.com/CPLreport2018
Kamiske, G., & Brauer, J. P. (2003). ABC des Qualitätsmanagements. Reihe Pocket Power.
München: Carl Hanser Verlag 2003.
Lemmergaard, J. (2009). From administrative expert to strategic partner. Employee
Relations, 31(2). 182-196. doi: 10.1108/01425450910925328
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Sage Publications, Inc.
Weingart, P. (2001). Die Stunde der Wahrheit. Zum Verhältnis der Wissenschaft zu Politik,
Wirtschaft und Medien in der Wissensgesellschaft. Weilerswist: Velbrück Verlag.
