ArticlePDF Available

OER Impact Assessment: A framework for higher education institutions and beyond. Approaches to assess the impact of Open Educational Resources

Authors:

Abstract and Figures

The purpose of this paper is to provide an overview of approaches to and insights for impact assessment on open educational resources (OER) in higher education and building on this to sketch a framework for university focussed OER impact assessment. The authors describe the literature on impact assessment in the OER context and the existing contributions to OER impact assessment in higher education. Findings of the analysis are that there are few contributions on the effects of OER in general or of specific OER initiatives. Four contributions are presented in more detail. From these examples and the literature analysis, derivations, and challenges for OER impact assessment are drawn, such as the large diversity in OER purposes and the invisibility of the re-usage of OER. The contribution sketches a framework model for describing OER-relevant results, outcomes, and impact, and more specifically demonstrates how this can be done for exemplary OER-related objectives. This contribution is thus of relevance to funding bodies and institutions working in the context of higher education that wish to systematically evaluate and monitor statements about the effectiveness of OER activities according to the UNESCO (2019) OER recommendation.
Content may be subject to copyright.
Open Education Studies, 2022; 4: 296309
Research Article
Martin Ebner, Dominic Orr, Sandra Schön*
OER Impact Assessment: A framework for higher
education institutions and beyond. Approaches to
assess the impact of Open Educational Resources
https://doi.org/10.1515/edu-2022-0018
received January 31, 2022; accepted May 31, 2022.
Abstract: The purpose of this paper is to provide an
overview of approaches to and insights for impact
assessment on open educational resources (OER) in higher
education and building on this to sketch a framework for
university focussed OER impact assessment. The authors
describe the literature on impact assessment in the OER
context and the existing contributions to OER impact
assessment in higher education. Findings of the analysis
are that there are few contributions on the effects of OER
in general or of specific OER initiatives. Four contributions
are presented in more detail. From these examples and the
literature analysis, derivations, and challenges for OER
impact assessment are drawn, such as the large diversity
in OER purposes and the invisibility of the re-usage of
OER. The contribution sketches a framework model for
describing OER-relevant results, outcomes, and impact,
and more specifically demonstrates how this can be done
for exemplary OER-related objectives. This contribution
is thus of relevance to funding bodies and institutions
working in the context of higher education that wish to
systematically evaluate and monitor statements about the
effectiveness of OER activities according to the UNESCO
(2019) OER recommendation.
Keywords: Open educational resources; impact
assessment; strategy; higher education; impact analysis.
1 The context: OER in Austria and
the TU Graz OER policy
Open educational resources are teaching and learning
materials that have explicit open licensing (UNESCO,
2019; Orr, Rimini, and van Damme, 2015). Among the
best-known open licences are the three licensing options
of Creative Commons CC BY, CC BY-SA and CC0. Since OER
are firstly (but not only) defined by the licence they use,
they can refer to many different educational artifacts, e. g.
they can be a printed book, a picture, a video, or any other
material, including software (Ebner and Schön, 2011).
The Massachusetts Institute of Technology (MIT)
is considered a pioneer in the field of OER with its
OpenCourseWare (OCW) initiative in 2001 (Lerman et
al, 2008). In Europe, a reference can be made to the
first EU-funded OER project OLCOS led by an Austrian
institution (Geser, 2007). An overview of the state of OER
in Austria shows that many initiatives in various education
sectors began in 2008 (Schön and Ebner, 2020).
Copyright issues are often seen as a key driver for the
development of OER in Europe (Ebner, Schön and Kumar,
2016), but there are also many other arguments for using
OER (Geser, 2011; Ebner et al., 2016; Ebner et al., 2017):
Possibility of updating (and adaptation) of teaching
materials: This is especially important to enable
adaptation to the requirements of specific teaching
and learning settings.
Re-use by third parties: This includes students and
learners in general.
Clear framework conditions for collaboration (and
exchange): The fact that OER can be revised and
repurposed makes it possible to use OER in a wide
variety of collaborative teaching situations.
Participation opportunities for learners. They can
directly contribute to the teaching materials and use
them in so-called “open education practices” such as
project work, e-portfolio, co-design.
*Corresponding author: Sandra Schön, Universitas Negeri Malang,
Malang, Indonesia, E-mail: sandra.schoen@tugraz.at
Martin Ebner, Graz University of Technology (TU Graz), Austria
Dominic Orr, University of Nova Gorica, Slovenia
Open Access. © 2022 Martin Ebner, Dominic Orr, Sandra Schön, published by De Gruyter. This work is licensed under the
Creative Commons Attribution 4.0 International License.
OER Impact Assessment: A framework for higher education institutions and beyond 297
Simple dissemination and adoption of materials by
others. This is of course an important and often
underestimated point. It can show that the quality
of one’s own teaching is worthy of attention if many
colleagues also use it.
Overall, OER contribute to making education
accessible to all, as OER enable free access to materials
and adaptation to specific student needs.
The Austrian Ministry of Education publicly recognised
the importance of OER in 2016, and together with the
Forum Neue Medien in der Lehre Austria (engl. “Forum
New Media in Higher Education Austria”) and other
stakeholders, issued a recommendation for the integration
of OER in Austrian higher education institutions (Ebner et
al., 2016; Ebner, Kopp et al., 2016). In 2017, a concept for the
certification of higher education institutions (HEIs) was
presented (Ebner et al., 2017; Ebner, 2018) and a feasibility
study on OER textbooks was published on behalf of the
Federal Chancellery of Austria and the Federal Ministry of
Education (Schön et al., 2017).
Furthermore, the contribution of OER was included
in the “National Strategy on the Social Dimension in
Higher Education” (Bundesministerium für Wissenschaft,
Forschung und Wirtschaft, 2017), where OER are mentioned
for their contribution to broader access and inclusion in
studies (p. 26). OER can also be found in the “All-Austrian
University Development Plan”, the technical-strategic
planning instrument on which the further development and
strategic orientation of the 22 public universities in Austria
is based and which forms the basis of the performance
agreements with each university (Bundesministerium für
Bildung, Wissenschaft und Forschung, 2020, p. 40).
In the project “Open Education Austria” and the
current successor project “Open Education Austria
Advanced” (2021-2024), the necessary infrastructure for
OER of the Austrian colleges and universities is being
further developed, co-financed by the Austrian Federal
Ministry. For example, university-owned repositories and
links to the nationwide OERhub.at for higher education
materials have already been developed (Ladurner et
al., 2020) and OER certification for individuals and
institutions in higher education will be introduced.
In all of this, Graz University of Technology (TU
Graz) has been a central actor for more than 10 years
now. In 2010, for example, a publication was released on
the use of OER at TU Graz in lifelong learning activities
(Ebner and Stöckler-Penz 2011). Since 2015, OER has
been anchored in the strategy of the organisational unit
“Educational Technology” and thus in the Vice-Rectorate
for Academic Affairs. Since 2017, OER have also been
mentioned in the development plan and in the university
performance agreement (TU Graz 2017, p. 60, TU Graz
and BMBWF, 2018). At TU Graz there are already several
OER activities that include the operation of the Austria-
wide MOOC platform iMooX.at, on which online courses
are offered exclusively as OER (Ebner, Lorenz et al., 2016),
OER training for teachers and a dedicated OER repository
(Ladurner et al., 2020). To further strengthen, expand
and strategically anchor the existing OER activities, the
performance agreement for 2019-2021 between the TU
Graz and the science ministry stated that an OER policy
would be developed (TU Graz and BMBWF 2018, p. 11).
In preparation for the development of the OER policy
for TU Graz, existing materials and resources for OER policy
development were first researched with the help of the
OER open education policy registry (Ebner et al, 2020). In
November 2020, the OER policy of TU Graz was published
as a guideline from the rectorate and recommends, among
other things, the use of the CC BY International 4.0 or
CC BY-SA International 4.0 licences. TU Graz is not the
first Austrian university with a dedicated OER policy; the
University of Graz (Universität Graz, 2020) published an
OER policy in March 2020. Also worldwide, countries and
first universities are positioning themselves in relation to
OER for strategic reasons and are developing and publishing
dedicated OER policies (Schaffert, 2010; Inamorato dos
Santos et al. 2017, e.g. University of Edinburgh 2016).
A key element of the TU Graz OER policy is
monitoring and evaluation: “TU Graz endeavours to
monitor the dissemination of its own OER and takes
corresponding indicators into account in the evaluation
and planning of teaching.” (TU Graz, 2020; translation
by the authors). This echoes part IV of the UNESCO (2019)
OER recommendation and reflects the recommendations
of the UNESCO publication on developing an OER
policy (Miao et al., 2019). This requirement leads to the
practical question: What are the suggestions for concrete
monitoring of teaching with OER, in the broader sense of
OER impact measurement? This paper will provide some
initial suggestions in relation to the context of TU Graz.
Since OER are social innovations (Schön, Ebner
and Hornung-Prähauser, 2017), we will first look at the
literature on social impact assessment. In the field of social
impact measurements, results are differentiated according
to immediacy. Stannard-Stockton (2010) describes the
different concepts with the help of a volunteer soup kitchen:
Outputs are the number of soups.
Outcomes are how many get a meal and are not
hungry at lunch.
Impact is how the soup influences the people and
community involved in the soup kitchen.
298 Martin Ebner, Dominic Orr, Sandra Schön
From the perspective of a funding agency, aiming to reach
a goal and check or assess the effectiveness of the funding,
the following requirements for the measurement of results
arise (Fritz, 2020):
Outputs are things that can be counted, measured
and listed.
Their quantitative and qualitative outcome is what
a project or a funder wants to achieve. Indicators
should be used that are measurable so their fulfilment
can be reviewed to assess the need for changes in the
activities as well.
Impact is long-term and may include some indirect
effects are hard to measure, as the intervention
contributes significantly to their achievement, but is
not the sole contributor to this impact.
Of interest as well is the funding agency’s requirement
to be able to measure the input for a certain activity
against output, as this is how indicators are commonly
constructed. This input can be money, time of staff or
volunteers, staff, expertise, methods, facilities as well as
expertise (Fritz, 2020).
Impact assessment can be done in parallel to an
activity or intervention - such as “formative evaluation”
or “ex-post” (summative evaluation) (see OECD, 2015).
The advantage of the former is that this enables feedback
loops for improvement, while the advantage of the latter is
that the assessment can usually be more comprehensive.
Impact assessment becomes more complicated and time-
consuming the more abstract the measures whose impact
is being examined are, for example the impact of political
measures (Stern et al., 2012).
There are quantitative approaches to “impact
analysis”, e. g. “regulatory impact analysis” but these
need to equate political or managerial decisions to
quantitative measurements of positive or negative effects
(see OECD, 2020).
Impact assessment is particularly difficult when
evaluating “innovations” and especially “social
innovations” like OER, since they are by their nature new
and the impacts uncertain. One way of approaching this
methodologically is to include the subjective assessment
of the users or benefactors. For instance, Antadze and
Westley (2012) describe ten conventional approaches
for measuring innovation impact, including asking how
much someone would pay for an innovation or asking
stakeholders to judge how the public value a service (p.
139ff).
In this paper, we build a framework for OER impact
assessment for organisations in the field of higher
education based, on the one hand, on insights after
collecting and learning about relevant practices in the
literature, and on the other, from other universities and
from related issues such as social impact assessment
. This is, however, not a purely academic task, as these
suggestions will also be implemented for the TU Graz.
2 Research questions, research and
development design
The reason for our research was that we could not
immediately find a suitable framework model for impact
assessment for a university.
It was necessary for us to find answers to the following
two questions:
How is the impact of OER measured?
Which are relevant components of an OER impact
assessment at a university?
To answer the first question, we researched examples in
a literature database and a project database. When using
“OER” and “impact”, 105 publications were found in
the ERIC database (January 2021), and all were reviewed
(abstract and/or full text) to see if they could provide
inputs to our research. Some of the papers we found in the
general search for OER and impact are not strictly about
OER impact. For example, Conrad and McGreal (2012)
looked at the ways in which prior learning is recognised
Figure 1: Monitoring possibilities of activities and their effects and Position of Impact Analysis.
OER Impact Assessment: A framework for higher education institutions and beyond 299
by the 31 member universities of the university network
called “OERu”. Or Okada and Sherbone (2018) investigated
the impact of a collection of activities which included
OER. We additionally used the OER World Map to search
for entries on “impact” and related papers or projects
(about 200 entries as of January 2021), albeit many of the
findings overlapped.
The selection of the contributions that are presented
here was done, on the one hand, based on the wish to find
different topics and approaches, while on the other hand,
wishing to uncover challenges that might be relevant to our
concern. The analysis was not carried out systematically in
the sense that all contributions were analyzed according
to uniform aspects, but regarding our practical questions.
The selection of the examples presented below is based on
the prominence of the examples and they were chosen in
such a way that as many different works and approaches
as possible are presented.
The paper is structured as follows: In chapter 3,
the findings of the literature and project research are
presented. Starting in section 3.1 with OER impact research
with a global or regional focus, the paper then presents
impact assessment examples where special aspects of
OER were in focus and concludes with a review of the OER
impact assessments of projects described. In section 3.2,
a synopsis is drawn for discussion, which highlights the
challenges and lessons learnt. Building upon this, chapter
4 presents a framework for OER impact assessment at
universities using the case of the TU Graz for a first OER
impact assessment sketch.
3 Findings from literature and
projects
3.1 OER impact assessment and its
approaches
In the following we present the results of the literature
and project research on OER impact assessment. We
provide an overview of existing contributions, as well
their approaches.
3.1.1 OER impact world-wide or in smaller regions
Some research studies deal with the impact of OER on a
general level: they want to find out who knows about it,
uses it and for what reasons, or what benefits and effects
are seen.
One, if not the first, OER impact assessment was
delivered by the William and Flora Hewlett Foundation, the
leading funding organisation for OER initiatives: Atkins,
Brown, and Hammond (2007) described and documented
the impact of their contribution to the early OER movement
and community and did this through interviews describing
certain OER initiatives, amongst them MIT OpenCourseWare.
Besides elaborating on the initiatives themselves, they align
them with their support offers or OER items and visits of their
websites, but not systematically.
To evaluate the effects of several OER projects in the
United Kingdom, Masterman et al. (2011) did a survey on
the benefits and practices around OER amongst educators
and learners in higher education in the UK. They asked
for and analysed pedagogic, attitudinal, logistic, and
strategic factors conducive to uptake and sustained
practice in the use of OER; and asked, conversely, what
the impediments were.
As shown in Table 1, the research approach of
Masterman et al. (2011) especially used a qualitative
approach with small groups and room to explore in-depth
data and understanding.
With a global approach, the main question for the OER
research hub of the Open University UK was to understand
the impact of OER by on 11 pre-formulated hypotheses
(see abstract de los Arcos et al., 2014; Farrow et al., 2015).
One result was the “OER impact map” (formerly available
under http://oermap.org). These hypotheses were applied
to several project and research collaborations, and
included “Performance: Use of OER leads to improvement
in student performance and satisfaction (OER improve
student performance/satisfaction); B - Openness: The
open aspect of OER creates different usage and adoption
patterns than other online resources (People use OER
differently from other online materials); C - Access:
Open Education models lead to more equitable access
to education, serving a broader base of learners than
traditional education (OER widen participation in
education)” (de los Arcos et al., 2014, p. 7). They were tested
using a cumulative collection of responses from surveys
conducted across the globe, consisting of more than 6,000
educators (de los Arcos et al., 2014; Farrow et al., 2015).
Later the same OER research hub group published their
results on teaching and learning practices (Weller et al.,
2015) using 21 surveys with a total of more than 7,400
participants. Overall, the approach of the OER research
hub highlights the potential to capture the development
of OER impact globally using the same or similar surveys
among multiple research projects in the field. In the last
few years, the OER research hub has published no such
follow-up OER impact study (see http://oerhub.net/).
300 Martin Ebner, Dominic Orr, Sandra Schön
3.1.2 Impact concerning special aspects or purposes of OER
Beside such general impact studies, we found several
publications that try to emphasise the (potential) OER
impact on special aspects and purposes.
For example, the book “Adoption and impact of OER in
the Global South” (Hodgkinson-Williams and Arinto, 2017)
includes several such studies, investigating OER effects
or impact. For example, Westermann and Venegas (2017)
investigated the effects of OER use on first-year higher
education students’ mathematical course performance
within a field experimental setting of control groups and
OER groups (see Figure 2). The study confirmed that OER
is useful, but it was concluded that openness alone does
not necessarily makes an impact.
White Baker and Siboba (2020) investigated if digital
OER impacts learning outcomes for social inclusion. Their
research supports the hypothesis that OER positively
influences students’ learning outcomes. Jenkins et al.
(2020) show that OER textbooks could improve social
justice in colleges, as underprivileged pupils, amongst
other deleterious aspects, very often have no textbooks at
all due to the costs.
These are only some of several studies and publications
that can be found when searching for OER and impact
which focus on special OER features and purposes, such
as the impact of OER on student’s performance in an
astronomy course (Mathew and Kashyap, 201). Such
studies have even been connected to a cost-benefit analysis.
This is done, for instance, by the “OER adoption impact
Table 1: Method and participants in the OER Impact Study by Masterman et al. (2011). Source: Masterman et al., 2011, Table 1, p. i, Source;
available under a CC BY NC-SA.
Method Participants (+ numbers) Purpose:
Interview and
focus groups
‘Strategists’: those
responsible for implementing
an institutional OER strategy
()
- Identify the drivers that prompt an institution to adopt a strategy for the use of OER
in teaching and learning.
- Collect evidence of models for engagement with, and uptake of, OER by individuals
and course
Interviews and
focus group
‘OER experts’: teaching staff
who were already using OER in
their courses ()
- Determine teachers’ conceptions of OER.
- Determine the relationship between their values and beliefs about teaching and
their disposition towards OER.
- Capture narratives of OER usage that is already embedded in teachers’ practice.
Online survey +
workshops
‘OER novices’: teaching
staff who had not yet used
OER in their courses ()
- Investigate the reality of searching for, locating and evaluating online resources.
- Gain insights into the issues faced by teachers engaging with OER for the first time.
Focus groups Students () - Determine the extent to which learners are aware of OER, the importance of
provenance and the relationship between their beliefs about learning and the use
of OER.
Figure 2: Overview of a scenario with OER treatment and proprietary textbook groups. Source: Westermann and Venegas (2017), part of
figure 4, p. 198. Available under a CC BY 4.0 International license.
OER Impact Assessment: A framework for higher education institutions and beyond 301
explorer” (Open Education Group and Lumen Learning,
2021). This online tool can be used to calculate the
potential cost savings and study successes of introducing
OER on campus. This is based, as noted on the webpage,
on the results of a study that has not yet been published,
and all parameters can also be set more conservatively or
optimistically. The issue with such studies is that they have
difficultly being comprehensive enough to really account
for the context in which the OER are being implemented 
what is the motivation of the students, what is the practice
of the teachers, what is the quality of the textbook vis-à-vis
the other materials. As one recent study has stated, the
effectiveness of any educational intervention will almost
always depend on the context in which it is implemented.”
(Grimaldi et al., 2019). This means that any evaluation
measures should ensure that many other factors are taken
into consideration too.
Regarding the Open University Sri Lanka, several
publications are available on OER impact assessment as
seen on teaching practices related to the use of OER – often
termed “open educational practices”. They focus on the
OER integration in teacher education (Karunanayaka and
Naidu, 2017) and develop an open educational practices
impact evaluation index.
The characteristic of re-use is also a special feature
of impact papers. White and Manton (2011), for example,
asked workshop participants about enablers and barriers
for re-use of existing OER. Pedagogic fit and relevance
were amongst the highlighted issues, others are source
and media format (p. 17). Currently, there are problems
tracking real re-use of OER (Thomas et al. 2012, p. 68ff), as
versioning, i.e., logging individual versions of an original
as practiced in open-source programming (e.g. via github),
is not a common practice in the OER community.
3.1.3 OER impact of defined OER projects and
repositories
In contrast to the research given above, some of the impact
studies within projects have been more inclusive and
more comprehensive. In this, they provide some impulses
for other methods which can be used.
3.1.3.1 MIT OpenCourseWare Program Evaluation (2006)
MIT OpenCourseWare (OCW) is one of the OER
lighthouses. One evaluation of the project documented
activities (“how we work”), outputs (“what we deliver”)
and outcomes/impact (“what differences it makes”) (MIT
OpenCourseWare, 2006). The evaluation includes the
process (effectiveness and efficiency) and the programme
(access, use, impact) (MIT OpenCourseWare, 2006, Table
128, p. 75).
The research questions for evaluation of impact
were: “What is the impact of OCW on individual teachers
and learners? What is the impact of OCW on learning
communities? What is the impact of OCW on the open
sharing of educational materials?” (MIT OCW, 2006, p.
77). To answer all evaluation aspects - access, use and
impact - the various data and approaches were used: web
analytics, online intercept survey (more than 100,000),
interviews (30), site feedback (via e-mail), and MIT
community surveys (e.g., as part of freshmen surveys).
3.1.3.2 OER Impact Assessment from the Bridge to
Success project (2013)
The OER impact assessment of the Bridge to Success
project used a mixed method approach to assess impact
(Pitt et al, 2013; Coughlan et al., 2013). For this purpose,
various existing documents and surveys were used and
supplemented by further surveys, analyses and interviews.
Figure 3 shows the model of OER impact evaluation,
which combines data from partner institutions and data
of the OER repository.
It analysed, for instance, how many learners
participated in the online courses and whether they had
better test results after participating in the course than in
the pre-test. In conclusion, the authors make the following
four recommendations for impact assessment (p. 12):
1. Extend an action research model to include other
educators in the research processes to provide
feedback on design, understanding of their ideas
for use in context, and their view of the impact on
learners.
2. Increase access to learners through surveys, focus
groups, and interviews to understand the effects on
motivation and the personal impact.
3. Highlight personal case studies captured in detail
through recorded interviews, preferably with
permission to share. These reveal the depth of factors
from individual perspectives. Such illustrations
apply to all project participants and are particularly
valuable in communicating the views of the end user,
in our case the learner.
4. Gather comparative performance data as available.
For formal learning such data may well be directly
available and the focus for evaluation. For instances
where learning takes place informally comparative
data may be found by looking for impact in parallel or
associated activities.”
302 Martin Ebner, Dominic Orr, Sandra Schön
3.1.3.3 OER Impact Assessment at the University of
Michigan (2011)
Rodgers (2011) describes the impact assessment of the
Open Michigan Initiative from the University of Michigan.
Existing data and reports were used, and new research
was done. It combines descriptions of their Wiki system
and description of their processes with web analytics,
surveys of students, faculty and staff and as well an “OER
Evaluation Environmental Scan”. The latter is the analysis
of existing approaches to the evaluation of OER activities.
Guiding questions were: “How and why is our OER being
used? Who is using our OER? What value does OER bring
to U-M [the University of Michigan]? In what contexts are
people using our OER?” (Rodgers, 2011, p. 4). Indicators
for the strategic goal “Provide comprehensive public
access to all scholarly output at U-M” are for example
published OER, workshops, training presentations, project
collaborations, papers published, collaborations, Do-It-
Yourself resources, software development, publishing
platform, policy development and guides, research (p.
6). Interestingly, the evaluation scheme is described as
having a more exploratory goal: “it will inventory other,
hitherto unanticipated uses of OCW” (p. 76) but does not
provide insights into such uses.
3.1.3.4 Impact Assessment of Open University
Bangladesh OER Repository (2019)
About 300 students took part at a survey on the impact
of the OER repository of the Open University Bangladesh
(Rahman, Panigrahi and Panda, 2019). The students
got 20 questions where they should give feedback on a
5-point Likert scale. The questions on impact of diverse
OER (e. g. video, textbook) were directly related to usage
and usability for the respondent, e.g., “I can use the
downloaded video in the smartphone” (p. 5).
3.2 Discussing OER impact assessments for
universities
In this article, we wanted to provide an overview of
current approaches to OER impact assessment for higher
education institutions. After reviewing them, we now
see some aspects as remarkable and challenging for OER
impact assessment at universities. Below we highlight
what we consider to be the most important new insights.
Diversity in OER purposes in universities:
There are different reasons and purposes for using
OER in universities, so the objective is not always
comparable. Case studies of OER implementation in
Figure 3: The Model of OER Impact Evaluation by Pitt et al. (2013). Source: Coughlan et al., 2013, Figure 1, p. 5; available under a CC BY 3.0
International license.
OER Impact Assessment: A framework for higher education institutions and beyond 303
13 higher education institutions (USA, Canada, five
other countries) by McGreal (2019) who asked them
about opportunity, innovation, benefits, challenges,
and potential of their OER implementation, found
no consensus besides cost-savings (p. 143). We
assume that this is also due, in part, to the completely
different conditions for OER in universities worldwide
(Marín et al., 2020), but also to the fact that there are
many purposes for which OER can be used. In German
speaking Europe with comparably low tuition fees
and textbooks available in libraries, the cost effects
for students are typically not relevant. Instead, as can
be seen later within the TU Graz OER policy, OER is a
possibility to use and modify existing materials which
is otherwise extremely limited by the copyright law
for proprietary resources.
Invisibility of the Re-Usage: OER re-usage is not
easy to document - especially when OER are not
republished but used by teachers offline or in closed
systems. Reasons for this usage are that there is no
need, or no opportunity, or perhaps no competency to
share for example modifications of OER adequately.
Wiley (2009) called this “dark use”. Others use
the metaphor of an iceberg; it is assumed that the
invisible usage, like the iceberg underwater, far
exceeds the visible mass of the iceberg above water
(White, Manton et al., 2011; Pitt et al., 2013, p. 19).
Overlooking innovations: As the “openness” of
OER make several uses possible, Coughlan et al.
(2013) stated: “With OER by its very ‘openness’ being
unpredictable in use, a model for research needs to
be flexible and attentive to diversity. It should focus
on achieving results based on definable quantities,
but with a detailed understanding of the distinctions
between contexts of use.” (p. 4). Practically, an
innovation is something new, and if we expect
OER to be a basis (or catalyst – Orr et al., 2015) for
innovation, we should design a framework to identify
such innovations.
Differences in the OER infrastructure and
maturity of the implementation: Additionally, we
also perceive clear differences in the extent to which
OER has been implemented at individual universities:
We have found impact assessments for single (and
primarily) OER projects, or of big and long-term
initiatives with a broad variety of measures.
Huge methodological variety: From maps that
visualise the dissemination of one’s own OER, to web
statistics, surveys, workshops, interviews, quasi-
experimental settings, and case studies, almost the
entire spectrum of social science methodology is used.
Visualisation and presentation of results:
Importantly, is not only about obtaining “objective”
data, but also about obtaining materials that show the
added value of OER, success stories and testimonials.
At least these are recommendations of social impact
consultants (Stannard-Stockton, 2010).
“Impact” is not used equally: We have seen a wide
variety of how researchers used the term within our
context – for instance influenced by the perspective
of a funder or an institution (exemplarily shown by
Coughlan et al. 2013, see Fig. 3).
Against this background, we would like to summarise in
the following the considerations that form a framework for
our efforts at the TU Graz and could shape similar projects
around OER impact assessment in other universities.
4 A framework for OER impact
assessment at universities
We now describe a framework for OER impact assessment
at universities based on the research and insights above.
4.1 Promoting OER as purpose: From
investment to impact
The TU Graz OER policy states that it “considers open
educational resources as a basis for high-quality teaching
and therefore supports the development and use of OER.”
That is to say, the policy assumes providing OER will
impact the quality of teaching (and learning) positively.
Therefore, it is important that the framework clearly
defines OER and distinguishes the ways OER will be
measured. We have seen that the various terms used in
impact research are not used consistently. We would
first like to make a proposal for the use of the terms,
which is in line with the literature review presented
above. As definition for OER we use the UNESCO (2019)
formulation: “Open Educational Resources (OER) are
learning, teaching and research materials in any format
and medium that reside in the public domain or are under
copyright that have been released under an open license,
that permit no-cost access, re-use, re-purpose, adaptation
and redistribution by others.”
Figure 4 illustrates key questions and indicators
derived from the OER definition, which are strongly
oriented towards this focus.
304 Martin Ebner, Dominic Orr, Sandra Schön
Furthermore, it is important to make a clear
connection in the framework to teaching practices.
According to Ehlers (2011) open educational practices are
“practices which support the (re)use and production of
OER through institutional policies, promote innovative
pedagogical models and respect and empower learners as
co-producers on their lifelong learning path” (Ehlers, 2011,
p. 3). We therefore develop a view of “open educational
practices” from investment to impact (Figure 5).
For the frameworks presented above, the question of
data and information remains open. The starting point
is to see how current OER activities are collected and
described. Part of this is to check how and where the
university, staff and students develop and publish OER.
Which diverse OER formats and platforms are used?
At the TU Graz we will be using the outcomes of key
instruments and measures (see below) to provide the
impact measurement on this level.
Figure 5: Overview from investment to OER impact on “open educational practices”.
Figure 4: Overview from investment to OER impact on OER at its core.
OER Impact Assessment: A framework for higher education institutions and beyond 305
4.2 Identifying the expected impact for the
impact assessment
From an empirical standpoint, this may at first glance
sound tautological, but it is important to start from the
expected outcome of the intervention (i.e., the so-called
“theory of change”) and to use this to construct the impact
assessment. We want to see how close to the expectations
we get.
Table 2: Sketch for objectives, output and outcome and impact indicators at TU Graz.
Expectation/
Objective
Output
Indicator
Outcome
indicator
Impact Indicator
OER creates the
opportunity for access
to free knowledge and
open exchange
Number/Uploads of OER at TU Graz OPEN
Repository according to the existing
metadata (format, target group)
OER views and downloads
External users, views and
downloads
Re-use of OER
views
There is evidence that
TU Graz OER makes a
relevant contribution to free
knowledge and is seen as a
relevant OER stakeholder.
Number of open licenced videos at TUbe
(TU Graz video platform)
Other TU Graz OER projects, such as the
Austria Forum and produced and hosted
OER
Number of open licenced offered MOOCs
Number of open licenced videos at iMooX.
at
Numbers of registered participants,
course participants, course
certificates,
views of videos
External usage and views, re-usage
TU Graz wide usage of open licenses Usage of open licenses for other
publications of the university (e.g.
the senate, policies)
OER supports and
enables open teaching
and learning scenarios
Number of qualified staff (OER certificates)
Materials and trainings about OER
developed and offered for lecturers
Percentage of staff and students
who knows, develops or re-use OER
Participation at courses and
downloads of materials about OER
External usage and views,
downloads and re-usage etc. of
materials and research about OER
There is evidence for
teaching innovations from
lectures and projects at
TU Graz that OER makes
a relevant contribution to
open teaching and learning
in the field
OER enables increased
collaboration between
companies and the
university
Number of partnerships and collaboration
in OER projects with companies and others
OER has a relevant role for projects
with companies
There is evidence that
OER is a relevant enabler
of partnerships with
companies with TU Graz
OER promotes inclusion Number of OER initiatives directly
addressing inclusion purposes
OER usage and production in
inclusion-related projects
There is evidence that OER
is a relevant measure for
inclusion at TU Graz
OER serves as a feature
for good teaching of
the university and its
teachers
Number of teacher trainings and their
participants where OER is addressed
Good teachers and teaching
practices are related with the
production and use of OER
There is evidence that
OER is a feature for good
teaching at TU Graz
OER is part of a
sustainable quality
assurance cycle for
teaching
Number of contributions in the field of
quality assurance of teaching addressing
OER
OER production and usage is part of
quality assurance considerations at
the university
There is evidence that
OER contributes to quality
assurance of teaching
OER serves as support
for copyright issues in
teaching
OER trainings includes copyright issues OER trained teachers are less
involved in copyright infringements
There is evidence that
TU Graz OER activities
contributed to less copyright
issues in teaching.
306 Martin Ebner, Dominic Orr, Sandra Schön
The TU Graz policy states expectations for impact in
seven areas, referring to a whitepaper from the national
OER working group (Ebner et al., 2016). These objectives
should be recognised in the evaluation framework:
1. OER supports and enables open teaching and learning
scenarios.
2. OER creates the opportunity for access to free
knowledge and open exchange.
3. OER enables increased collaboration with companies
and the university.
4. OER promotes inclusion.
5. OER serves as a feature for good teaching of the
university and its teachers.
6. OER is part of a sustainable quality assurance of
teaching.
7. OER serves as support for copyright issues in teaching.
In the case that this is not so clearly defined for other
universities, we suggest a pragmatic approach to get you
started: existing OER Policies or similar strategy papers
are checked concerning purposes, arguments or aims
related to OER. This should lead to the specification of
what exactly the institution’s OER should achieve and
who is the target group. Furthermore, this process should
be supported through consultation of key stakeholders.
4.3 Drafting indicators, methodology and
metrics at TU Graz
Our overview has shown a big variety in definitions of OER
impact, as well how to measure it. In the following, we
want to present a first sketch of what and how we suggest
quantifying and qualifying the TU Graz OER output,
outcomes and impact.
As described in Table 2, we use the objectives from
TU Graz policy to specify indicators for output, outcome
and impact. For example, the objective “OER creates
the opportunity for access to free knowledge and open
exchange” could best be supported by evidence - i.e.,
examples and proof - that TU Graz is making a relevant
contribution. Such evidence could be, for example,
reports from others in which TU Graz is mentioned as a
relevant player. We propose a similar impact indicator for
the objective “good teaching”: Evidence for the impact of
OER could be examples of good teaching practice at TU
Graz (for example on the Austria-wide collection for good
teaching) clearly related to the use of OER.
But the process does not end with drafting these
indicators. This first sketch needs to be discussed with
several stakeholder groups at the university. For example,
one of the reviewers of this paper remarked that we do not
address “learning outcomes”. Such suggestions should
be compared to the purpose documents used as a basis
for the monitoring framework. In this case, it was not
considered a clear objective of the TU Graz policy, so is not
(yet) included in our sketch.
Particularly in the case of outputs and outcomes,
the development of appropriate methods is also about
how well the indicators mentioned can be recorded and
monitored. For example, it should be considered if and
in which university-wide survey of teachers one could
integrate capturing their knowledge about and use of
OER. Particularly in the case of the impact indicators,
it will not be efficient to establish ad-hoc methods,
and corresponding routines must instead be planned
and carried out which enable insights to be captured
regularly and without too much effort. This will ensure
the sustainability of the impact assessment over time.
For an area of innovation such as OER this is particularly
important, as changes over time may be more meaningful
than quantitative benchmarks such as “50% of teachers
use OER”.
5 Coda
An impact assessment is about finding the right ways to
measure impact (i.e., making a link between objectives and
how to get there following the theory of change embedded
in the original strategy). But it also significantly relies on
finding the right means to gather inputs and data to make
this assessment. Some of the key questions that should be
raised are:
What are good indicators and what data and collection
options are available?
Which university institutions could also be
interested in the data provided or already collect
similar data (e.g. survey of first-year students, image
analysis, intellectual capital reporting, marketing,
digitalisation strategy, SDG/sustainability strategy)?
How can the different consequences and effects of
OER, i.e. outputs, outcomes and impact, be surveyed
in a particularly uncomplicated way, but still with
great significance?
How can the impact assessment be maintained
over time (e.g. a not become just one step in project
implementation), as only efficient and effective
assessments will ensure that implementers can learn
from their past actions and become better over time?
OER Impact Assessment: A framework for higher education institutions and beyond 307
The framework developed above has the goal of providing
some inspiration for this process, but also the authors
also wish to highlight that it is important to remain open.
The requirements of finding data and information that
can be used can indeed lead to limitations in the insights
provided by the evaluation.
The UNESCO Guidelines on the development of open
educational resources policies (2019) promote the concept
that any policy and, by implication, any evaluation and
monitoring framework, should be a “living document”.
Consultation and feedback will make it better. This
recognises the requirement, mentioned above, that the
innovative power of OER should be included into the
design of an evaluation framework. In our case, we are
using these insights and drafts to develop and introduce
our own OER impact assessment internally at TU Graz.
To this aim, we will not only use the research results
and drafts presented here, but also plan to cooperate
and exchange with stakeholders, building their ideas
and experiences and requirements into the final impact
assessment routines we will establish.
Acknowledgement: This contribution was also partly
developed in the context of the co-funded project “Open
Education Austria Advanced” (2021-2024 by the Austrian
Federal Ministry of Education, Science and Research).
Thereby, all project partners - i.e., TU Graz, University of
Vienna, University of Graz, University of Innsbruck, as
well as the Forum Neue Medien in der Lehre Austria and
the Austrian Institute for Vocational Training Research
(ÖIBF) - want to contribute to make more materials and
infrastructures available that support the systematic use
and publication of OER by Austrian universities.
Conflict of interest: Authors state no conflict of interest.
References
Antadze, N. and Westley, F. R. (2012). “Impact Metrics for Social
Innovation: Barriers or Bridges to Radical Change?”, in: Journal
of Social Entrepreneurship, Vol. 3, No. 2, 133–150, October
2012, URL: https://uwaterloo.ca/waterloo-institute-for-social-
innovation-and-resilience/sites/ca.waterloo-institute-for-
social-innovation-and-resilience/files/uploads/files/impact_
metrics_for_si_barriers_or_bridges.pdf
Atkins, D. E., Brown, J. S. and Hammond, A. L. (2007). “A review
of the Open Educational Resources (OER) movement:
Achievements, challenges, and new opportunities”,
pp. 1-84. URL: http://www.hewlett.org/uploads/files/
ReviewoftheOERMovement.pdf
Bundesministerium für Wissenschaft, Forschung und Wirtschaft
(2017). “Nationale Strategie zur sozialen Dimension in
der Hochschulbildung. Für einen integrativen Zugang und
eine breitere Teilhabe.“ Wien. URL: https://ph-tirol.ac.at/
sites/default/files/download/Nationale%20Strategie%20
%28PDF%29.pdf
Bundesministerium für Bildung, Wissenschaft und Forschung
(2020). „Gesamtösterreichische Universitätsentwicklungsplan
(GUEP).“, URL: https://www.bmbwf.gv.at/dam/jcr:b7701597-
4219-42f3-9499-264dec94506e/GUEP%202022-2027_
Aktualisiert_um_Statistik_final_bf.pdf
Coughlan, T.; Ebrahimi, N.; McAndrew, P. and Pitt, R. (2013).
“Assessing OER impact across varied organisations and
learners: experiences from the “Bridge to Success” initiative.”,
in: OER13: Creating a Virtuous Circle, 26-27 Mar 2013,
Nottingham, UK. URL: http://oro.open.ac.uk/37754/1/OER13-
Coughlanetal_Assessing_OER_Impact.pdf
Conrad, D. and McGreal, R. (2012). “Flexible paths to assessment
for OER learners: A comparative study”, Journal of Interactive
Media in Education, 2012(2), p. Art. 12. DOI: http://doi.
org/10.5334/2012-12
de los Arcos, B.; Farrow, R.; Perryman, L.-A.; Pitt, R. and Weller, M.
(2014). “OER Evidence Report 2013-2014”, OER Research Hub.
URL: http://oro.open.ac.uk/41866/
Ebner, M. (2018). “OER-Certification in Higher Education”, in
Proceedings of EdMedia: World Conference on Educational
Media and Technology. pp. 1-6 Amsterdam, Netherlands:
Association for the Advancement of Computing in
Education (AACE). URL: https://www.researchgate.net/
publication/326034702_OER-Certification_in_Higher_
Education
Ebner, M., Stöckler-Penz, C. (2011). “Open Educational Resources
als Lifelong- Learning Strategie am Beispiel der TU Graz.”, in
The Lifelong Learning University, pp. 53–60.
Ebner, M. and Schön, S. (2011). “Lernressourcen: Frei zugänglich
und einsetzbar.“, in: Handbuch E-Learning - Expertenwissen
aus Wissenschaft und Praxis - Strategie, Instrumente,
Fallstudien, pp. 1–14.
Ebner, M., Köpf, E., Muuß-Merholz, J., Schön, M., Schön, S.,
Weichert, N. (2015). “Ist-Analyse zu freien Bildungsmaterialien
(OER).“, Book on Demand, Norderstedt.
Ebner, M., Kopp, M., Freisleben-Teutscher, C., Gröblinger, O., Rieck,
K., Schön, S., Seitz, P, Seissl, M., Ofner, S., Zimmermann, C.,
Zwiauer, C. (2016). “Recommendations for OER Integration in
Austrian Higher Education.“, in: Conference Proceedings: The
Online, Open and Flexible Higher Education Conference, EADTU
2016, pp. 34-44.
Ebner, M., Lorenz, A., Lackner, E., Kopp, M., Kumar, S., Schön,
S. and Wittke, A. (2016). “How OER enhance MOOCs – A
Perspective from German-speaking Europe.”, in: Open
Education: from OERs to MOOCs. Jemni, M., Kinshuk, Khribi, M.
K. (Eds.). Springer. Lecture Notes in Educational Technology.
pp. 205-220.
Ebner, M., Freisleben-Teutscher, C., Gröblinger, O., Kopp, M., Rieck,
K., Schön, S., Seitz, P., Seissl, M., Ofner, S. and Zwiauer,
C. (2016). „Empfehlungen für die Integration von Open
Educational Resources an Hochschulen in Österreich“, Forum
Neue Medien in der Lehre Austria. URL: http://www.fnm-
austria.at/fileadmin/user_upload/documents/Buecher/2016_
fnma-OER- Empfehlungen_final.pdf
Ebner, M., Kopp, M., Hafner, R., Budroni, P., Buschbeck, V.,
Enkhbayar, A., Ferus, A., Freisleben-Teutscher, C. F., Gröblinger,
308 Martin Ebner, Dominic Orr, Sandra Schön
O., Matt, I., Ofner, S., Schmitt, F., Schön, S., Seissl, M.,
Seitz, P., Skokan, E., Vogt, E., Waller, D. and Zwiauer, C.
(2017). “Konzept OER-Zertifizierung an österreichischen
Hochschulen.“ Forum Neue Medien in der Lehre Austria,
URL: http://www.fnm- austria.at/fileadmin/user_upload/
documents/Buecher/OER_Labeling_2017.pdf
Ebner, M., Schön, S. and Kumar, S. (2016). “Guidelines for
leveraging university didactics centers to support OER
uptake in German-speaking Europe”, in Education Policy
Analysis Archives, 24 (39), URL: http://dx.doi.org/10.14507/
epaa.24.1856
Ebner, M.; Schön, S.; Atenas, J.; Havemann, L.; Nascimbeni, F. and
Neumann, J.L. (2020). “Ressourcen und Werkzeuge für die
Entwicklung einer OER-Policy an Hochschulen. Gemeinsamer
Report von TU Graz, Open Education Austria Advanced, OER
World Map und Open Education Policy Hub.” Graz: TU Graz.
DOI: 10.13140/RG.2.2.13705.47207/1
Ehlers, U.-D. (2011). “Extending the territory: From Open Educational
Resources to Open Educational Practices.”, in: Journal of Open,
Flexible and Distance Learning, 15(2), 1–10. URL: http://www.
jofdl.nz/index.php/JOFDL/article/view/6
Farrow, R.; Pitt, R.; De los Arcos, B.; Perryman, L.-A.; Weller, M. and
McAndrew, Patrick (2015). “Impact of OER use on teaching
and learning: Data from OER Research Hub (2013–2014).”, in:
British Journal of Educational Technology, Volume 46, Issue 5,
Special Issue: Open Data in Learning Technology, September
2015, pp. 972-976, URL: https://bera-journals.onlinelibrary.
wiley.com/doi/full/10.1111/bjet.12310
Fritz, J. (2020). “How to Talk About Nonprofit Impact from Inputs to
Outcomes. Inputs, Outputs, Outcomes, Impact - How are they
Different?”, in: small businesses, updated May 2020, URL:
https://www.thebalancesmb.com/inputs-outputs-outcomes-
impact-what-s-the-difference-2502227
Geser, G. (2007). “Open Educational Practices and Resources.
OLCOS Roadmap 2012.” Salzburg: Salzburg Research. URL:
http://www.olcos.org/ cms/upload/docs/olcos_roadmap.pdf
Grimaldi, P. J., Basu Mallick, D., Waters, A. E. and Baraniuk, R.
G. (2019). “Do open educational resources improve student
learning? Implications of the access hypothesis”, PloS One,
14(3), e0212508. DOI: https://doi.org/10.1371/journal.
pone.0212508
Hodgkinson-Williams, C. and Arinto P. B. (2017). “Adoption
and impact of OER in the Global South” DOI: https://doi.
org/10.5281/zenodo.600398
Inamorato dos Santos, A. I., Nascimbeni, F., Bacsich, P., Atenas, J.,
Aceto, S., Burgos, D. and Punie, Y. (2017). “Policy Approaches
to Open Education–Case Studies from 28 EU Member States
(OpenEdu Policies) (No. JRC107713)”, Joint Research Centre
(Seville site). DOI: https://doi.org/10.2760/283135
Jenkins, J. J.; Sánchez, L. A.; Schraedley, M. A. K.; Hannans, J.;
Navick, N. and Young, J. (2020). “Textbook Broke: Textbook
Affordability as a Social Justice Issue.”, in: Journal of Interactive
Media in Education, v2020 n1 Article 3 2020, URL: https://files.
eric.ed.gov/fulltext/EJ1253996.pdf
Karunanayaka, S. P. and Naidu, S. (2017). “Impact of integrating OER
in teacher education at the Open University of Sri Lanka”, in
C. Hodgkinson-Williams and P. B. Arinto (Eds.), Adoption and
impact of OER in the Global South, pp. 459–498. DOI: https://
doi.org/10.5281/zenodo.600398
Ladurner, C.; Ortner, C.; Lach, K.; Ebner, M., Haas, M.; Ebner,
M.; Ganguly; R. and Schön, S. (2020). “The Development
and Implementation of Missing Tools and Procedures at the
Interface of a University’s Learning Management System,
its OER Repository and the Austrian OER Referatory”, in:
International Journal of Open Educational Resources (IJOER),
Volume 3, No. 2 Fall 2020 Winter 2021, URL: https://www.ijoer.
org/the-development-and-implementation-of-missing-tools-
and-procedures-at-the-interface-of-a-universitys-learning-
management-system-its-oer-repository-and-the-austrian-oer-
referatory/
Lerman, S. R., Miyagawa, S., Margulies, A. H. (2008). “Open
Course Ware: Building a Culture of Sharing”, in: Opening Up
Education, Iiyoshi, T. and Kumar, M.S.V (Eds.), 2008, MIT Press,
pp. 213-227.
Marín, V. I., Bond, M., Zawacki-Richter, O., Aydin, C. H., Bedenlier,
S., Bozkurt, A., Conrad, D., Jung, I., Kondakci, Y., Prinsloo, P.,
Qayyum, A., Roberts, J., Sangrà, A., Slagter van Tryon, P. J.,
Veletsianos, G. and Xiao, J. (2020). “A comparative study of
national infrastructures for digital (open) educational resources
in higher education”, in: Open Praxis, 12(2), 241–256. DOI:
http://dx.doi.org/10.5944/openpraxis.12.2.1071
Masterman, L., Wild, J., White, D. and Manton, M. (2011). “JISC
Open Educational Resources Programme: OER Impact Study”,
Research Report. URL: https://process.arts.ac.uk/sites/
default/files/jiscoerimpactstudyresearchreportv1-0.pdf
Mathew, S. and Kashyap, U. (2019). “Impact of OER Materials
on Students’ Academic Performance in an Undergraduate
Astronomy Course.”, in: Journal of STEM Education: Innovations
and Research, v20 n1 p46-49 Apr-Sep 2019, URL: https://www.
jstem.org/jstem/index.php/JSTEM/article/view/2369/2123
McAuley, A.; Stewart, B.; Siemens, G. and Cormier, D. (2010).
“Massive Open Online Courses Digital ways of knowing and
learning, The MOOC model For Digital Practice”, URL: http://
www.elearnspace.org/Articles/MOOC_Final.pdf
McGreal, Rory (2019). “A Survey of OER Implementations in 13
Higher Education Institutions”, in: International Review of
Research in Open and Distributed Learning, 20, 5 pp. 141-145;
URL: https://files.eric.ed.gov/fulltext/EJ1237334.pdf
Miao, F., Mishra, S., Orr, D. and Janssen, B. (2019). “Guidelines
on the development of open educational resources policies.”
UNESCO. URL: unesdoc.unesco.org/in/rest/annotationSVC/
DownloadWatermarkedAttachment/attach_import_75b312a4-
e696-423c-b30d-f8573e3c819c?_=371129eng.
pdf&to=107&from=1
MIT OpenCourseWare (2006). “2005 Program Evaluations Finding
Report.” URL: http://ocw.mit.edu/ans7870/global/05_Prog_
Eval_Report_Final.pdf
Naidu, S. and Karunanayaka, S. P. (2017). “Development of the Open
Educational Practices Impact Evaluation Index.” Research
paper presented at OE Global 2017 Conference, 8–10 March
2017. Cape Town, South Africa. URL: https://idl-bnc-idrc.
dspacedirect.org/bitstream/handle/10625/57016/57028.
pdf?sequence=1&isAllowed=y
OECD (2015). “What is impact assessment?” OECD. URL: http://www.
oecd.org/sti/inno/What-is-impact-assessment-OECDImpact.
pdf
OECD (2020). “Tax Challenges Arising from Digitalisation –
Economic Impact Assessment.” OECD. URL: https://www.
OER Impact Assessment: A framework for higher education institutions and beyond 309
oecd.org/tax/beps/tax-challenges-arising-from-digitalisation-
economic-impact-assessment-0e3cc2d4-en.htm
Okada, A. and Sherbone, T. (2018). “Equipping the Next Generation
for Responsible Research and Innovation with Open
Educational Resources, Open Courses, Open Communities and
Open Schooling: An Impact Case Study in Brazil”, in: Journal of
interactive media in education, URL: https://jime.open.ac.uk/
articles/10.5334/jime.482/
Open Education Group and Lumen Learning (2021). “OER Adoption
Impact Explorer”, version 0.8. URL: https://opencontent.
shinyapps.io/OER-Impact-Calculator/
Orr, D.; Rimini, M. and van Damme, D. (2015). “Open Educational
Resources: A Catalyst for Innovation, Educational Research
and Innovation”, OECD Publishing, Paris. DOI: http://dx.doi.
org/10.1787/9789264247543-en
Pitt, R.; Ebrahimi, N.; McAndrew, P. and Coughlan, T. (2013).
“Assessing OER Impact across Organisations and Learners:
Experiences from the Bridge to Success Project.”, in: Journal
of Interactive Media in Education, Spec Iss 2013, URL: https://
files.eric.ed.gov/fulltext/EJ1034706.pdf
Rahman, M.; Panigrahi, M.R.; Panda, S. (2019). “Impact Assessment
of OER Repository of Bangladesh Open University.”, in: Pan-
Commonwealth Forum, 9-12 September 2019, Edinburgh,
Scotland. URL: http://oasis.col.org/handle/11599/3341
Rodgers, E. (2011). “Measuring Our Impact: the Open.Michigan
Initiative.InProceedings of OpenCourseWare Consortium Global
2011: Celebrating 10 Years of OpenCourseWare.” URL: http://
hdl.handle.net/2027.42/91308
Schaffert, S. (2010). “Strategic Integration of Open Educational
Resources in Higher Education. Objectives, Case Studies, and
the Impact of Web 2.0 on Universities.”, in: Ulf-Daniel Ehlers
and Dirk Schneckenberg (eds.), Changing Cultures in Higher
Education – Moving Ahead to Future Learning, New York:
Springer, pp. 119-131.
Schön S. and Ebner M. (2020). “Open Educational Resources in
Austria”, in: Huang R., Liu D., Tlili A., Gao Y., Koper R. (eds.).
Current State of Open Educational Resources in the “Belt and
Road” Countries. Lecture Notes in Educational Technology,
Springer, Singapore, pp. 17-33.
Schön, S., Ebner, M., Hornung-Prähauser, V. (2017). “Digital social
innovation within education: Five insights on the role of
digital tools in the field of Open Educational Resources (OER)
projects.” in: Nata, R. V. (ed). Progress in Education. Vol. 49.
Nova publisher, pp. 167-188.
Schön, S.; Kreissl, K.; Dobusch, L. and Ebner, M. (2017). “Mögliche
Wege zum Schulbuch als Open Educational Resources
(OER). Eine Machbarkeitsstudie zu OER-Schulbüchern in
Österreich.“ Band 7 der Reihe „InnovationLab Arbeitsberichte“,
herausgegeben vom Forschungsbereich InnovationLab der
Salzburg Research Forschungsgesellschaft mbH, gleichzeitig
erschienen als Band 15 der Reihe „Beiträge zu offenen
Bildungsressourcen“ (http://o3r.eu) Salzburg: Salzburg
Research. URL: http://l3t.eu/oer/images/band15.pdf
Stannard-Stockton, S. (2010). “Getting Results: Outputs, Outcomes
and Impact. Measuring Social Impact.”, in: Stanford Social
Innovation Review. URL: https://ssir.org/articles/entry/
getting_results_outputs_outcomes_impact#
Stern, E.; Stame, N.; Mayne, J.; Forss, K.; Davies, R. and Befani, B.
(2012). “Broadening the range of designs and methods for
impact evaluations.”, in: DFID Working Paper, volume 38, URL:
https://www.ids.ac.uk/publications/broadening-the-range-of-
designs-and-methods-for-impact-evaluations/
Thomas, A., Campbell, L. M., Barker, P. and Hawksey, M. (2012).
“Technology for Open Educational Resources - into the wild.”
University of Bolton, jISC. URL: http://publications.cetis.org.
uk/wp-content/uploads/2012/12/into_the_wild_screen.pdf
TU Graz and BMBWF - Technische Universität Graz und
Bundesministerium für Bildung, Wissenschaft und Forschung
(2018). “Leistungsvereinbarung 2019-2021.“ Graz: TU
Graz. URL: https://www.tugraz.at/fileadmin/user_upload/
tugrazInternal/TU_Graz/Universitaet/TU_Graz_kompakt/
Leistungsvereinbarung_2019-2021.pdf
TU Graz - Technische Universität Graz (2017). “Entwicklungsplan
2018plus zur Beschlussfassung am 14. Dezember 2017“. Graz:
TU Graz. URL: https://mibla-archiv.tugraz.at/17_18/Stk_6/
Entwicklungsplan2018plus_Beschluss_UR_20171214.pdf
Technische Universität Graz (2020). “Richtlinie zu offenen
Bildungsressourcen an der Technischen Universität Graz
(OER-Policy),“ November 2020 URL: https://www.tugraz.
at/fileadmin/user_upload/tugrazExternal/02bfe6da-df31-
4c20-9e9f-819251ecfd4b/2020_2021/Stk_5/RL_OER_
Policy_24112020.pdf
UNESCO (2019). “Recommendation on Open Educational Resources
(OER)”, 25 November 2019. URL: http://portal.unesco.org/
en/ev.php-URL_ID=49556&URL_DO=DO_TOPIC&URL_
SECTION=201.html
Universität Graz (2020). “Open Educational Resources Policy der
Universität Graz, Rektoratsbeschluss vom 4. März 2020, URL:
https://static.uni-graz.at/fileadmin/digitales-lehren-und-
lernen/Dokumente/OER_Policy.pdf (2020-12-10)
University of Edinburgh (2016). “Open Educational Resources
Policy.” URL: https://www.ed.ac.uk/files/atoms/files/
openeducationalresourcespolicy.pdf
Weller, M.; de los Arcos, B.; Farrow, R.; Pitt, B.; McAndrew, P. (2015).
“The Impact of OER on Teaching and Learning Practice.”, in:
Open Praxis, v7 n4 p351-361 Oct-Dec 2015. URL: https://files.
eric.ed.gov/fulltext/EJ1083261.pdf
Westermann Juárez, W. and Venegas Muggli, J. I. (2017).
“Effectiveness of OER use in first-year higher education
students’ mathematical course performance: A case study.”, in
C. Hodgkinson-Williams and P. B. Arinto (Eds.), Adoption and
impact of OER in the Global South (pp. 187–229). DOI: https://
doi.org/10.5281/zenodo.601203
White, D. and Manton, M., (2011). “Open Educational Resources:
The value of reuse in higher education”, University of Oxford,
URL: http://www.jisc.ac.uk/media/documents/programmes/
elearning/oer/OERTheValueOfReuseInHigherEducation.pdf
White Baker, E. and Sibona, C. J. (2020). “Digital OER Impact on
Learning Outcomes for Social Inclusion, Journal of Computer
Information Systems”, DOI: 10.1080/08874417.2020.1802789
Wiley, D. (2009). “Dark Matter, Dark Reuse, and the Irrational Zeal of
a Believer.” David Wiley Weblog, URL: http://opencontent.org/
blog/archives/905
Chapter
Full-text available
Based on the increasing demand for and promotion of Open Educational Resources (OER, see (UNESCO (2019), this chapter describes the objectives of Graz University of Technology (TU Graz) in Austria for good teaching. A description of how the impact of OER at TU Graz will be analysed and considerations around it is the central contribution. In addition, the effects, and potentials of selected OER initiatives of the university are described as examples and discussed as key potential for good teaching. For a better understanding of the role of OER at TU Graz, the national context of OER in the Austrian higher education landscape is described at the beginning of the chapter.
Article
Full-text available
To enable broad access to education and generous use of educational resources, Graz University of Technology (TU Graz) also relies on Open Educational Resources (OER). This article describes the technological developments and processes that enable teachers at TU Graz to use their own learning management system (LMS) for the publication of OER. The article describes how interfaces and processes have marked educational resources of TU Graz with metadata to offer them to a broad public via the university's own OER repository and via the Austrian OER portal of the University of Vienna. Only appropriately qualified lecturers at TU Graz are authorized to use the new OER plug-in. The article concludes with recommendations for projects in OER infrastructure implementations. **** Ladurner, Christoph; Ortner, Christian; Lach, Karin; Ebner, Martin Haas, Maria; Ebner, Markus; Ganguly; Raman & Schön, Sandra (2020). The Development and Implementation of Missing Tools and Procedures at the Interface of a University’s Learning Management System, its OER Repository and the Austrian OER Referatory. In: International Journal of Open Educational Resources (IJOER), Volume 3, No. 2 Fall 2020 Winter 2021, URL: https://www.ijoer.org/the-development-and-implementation-of-missing-tools-and-procedures-at-the-interface-of-a-universitys-learning-management-system-its-oer-repository-and-the-austrian-oer-referatory/
Technical Report
Full-text available
Hochschulen nutzen auf unterschiedliche Weise offene Bildungsressourcen (Schaffert 2010); seit etwa 15 Jahren positionieren sich Länder und erste Hochschulen aus strategischen Gründen für offene Bildungsressourcen und entwickeln und veröffentlichen dezidierte OER-Strategien (dos Santos et al. 2017, zum Beispiel University of Edinburgh 2016) . Wenn sich Hochschulen strategisch mit offenen Bildungsressourcen auseinandersetzen möchten, Richtlinien entwerfen und Prozesse etablieren möchten, die ihre Kompetenzen, Kapazitäten und Infrastrukturen rund um OER erhöhen, sind sie an Ressourcen, Methoden und Beispielen für die Entwicklung interessiert. In diesem Beitrag geben wir einen Überblick über vorhandene Materialien und Ressourcen zur Entwicklung einer OER-Policy im Hochschulsektor.
Article
Full-text available
This paper reports on the first stage of an international comparative study for the project "Digital educational architectures: Open learning resources in distributed learning infrastructures-EduArc", funded by the German Federal Ministry of Education and Research. This study reviews the situation of digital educational resources (or (O)ER) framed within the digital transformation of ten different Higher Education (HE) systems (Australia, Turkey and the United States). Following a comparative case study approach, we investigated issues related to the existence of policies, quality assurance mechanisms and measures for the promotion of change in supporting infrastructure development for (O)ER at the national level in HE in the different countries. The results of this mainly documentary research highlight differences and similarities, which are largely due to variations in these countries' political structure organisation. The discussion and conclusion point at the importance of understanding each country's context and culture, in order to understand the differences between them, as well as the challenges they face.
Chapter
Full-text available
Offene Bildungsressourcen (engl. Open Educational Resources, kurz OER) sind frei zugängliche, nutzbare und häufig auch modifizierbare Online-Ressourcen für das Lernen und Lehren. Seit Anfang des 21. Jahrhunderts begann das Thema mit einer zunehmenden Zahl an Projekten, Berichten und Mitwirkenden immer bekannter zu werden. Zahlreiche Argumente, unter anderem bildungspolitische, didaktische wie auch wirtschaftliche, spre- chen dafür, sich an der Erstellung von OER zu beteiligen. In diesem Beitrag werden ausgewählte OER-Initiativen und -Projekte vorgestellt, die Potenziale von OER diskutiert und Motive für die Einführung von OER-Strategien an Hochschulen beschrieben. Zudem werden auch praktische Tipps zur Recherche, Erstellung und zum Austausch von OER gegeben. Der Beitrag schließt mit einem Abschnitt, der darauf hinweist, dass bei offenen Bildungsressourcen sich nicht nur der Vertriebsweg deutlich von traditionellen Lernobjekten (z. B. gedruckte Lehrbücher und Arbeitsmaterialien) unterscheidet, sondern dass auch weitere Prozesse einfach anders sind, u. a. das Qualitätsmanagement.
Article
Full-text available
In light of rising textbook prices, open education resources (OER) have been shown to decrease non-tuition costs, while simultaneously increasing academic access, student performance, and time-to-graduation rates. Yet very little research to date has explored OER’s specific impact on those who are presumed to benefit most from this potential: historically underserved students. This reality has left a significant gap of understanding in the current body of literature, resulting in calls for more empirically-based examinations of OER through a social justice lens. For each of these reasons, this study explored the impact of OER and textbook pricing among racial/ethnic minority students, low-income students, and first-generation college students at a four-year Hispanic Serving Institution (HSI) in Southern California. Drawing upon more than 700 undergraduate surveys, our univariate, bivariate and multivariate results revealed textbook costs to be a substantial barrier for the vast majority of students. However, those barriers were even more significant among historically underserved college students; thus, confirming textbook affordability as a redistributive justice issue, and positing OER as a potential avenue for realizing a more socially just college experience.
Chapter
Full-text available
In Austria, located in the center of Europe, when compared earlier in relation to other German-speaking countries in Europe, individuals and groups started to develop and work on the idea of freely available and usable learning content on the Internet. A first Austrian milestone was the coordination of an international conference on open educational content in 2007 as the final activity of the first European project that was focused on OER (olcos.org). Within the contribution an overview of current state and developments of OER activities in Austria is given, also describing its infrastructure, policy, existing resources, curriculum and teaching methodologies, outcome, stakeholders and impact for education. The chapter gives a comprehensive overview of all OER activities in Austria and outlines the benefits for the educational system as well. It can be summarized that the Austrian way seems to be successful even though the steps forward are often small.
Book
Education in the Global South faces several key interrelated challenges, for which Open Educational Resources (OER) are seen to be part of the solution. These challenges include: unequal access to education; variable quality of educational resources, teaching, and student performance; and increasing cost and concern about the sustainability of education. The Research on Open Educational Resources for Development (ROER4D) project seeks to build on and contribute to the body of research on how OER can help to improve access, enhance quality and reduce the cost of education in the Global South. This volume examines aspects of educator and student adoption of OER and engagement in Open Educational Practices (OEP) in secondary and tertiary education as well as teacher professional development in 21 countries in South America, Sub-Saharan Africa and South and Southeast Asia. The ROER4D studies and syntheses presented here aim to help inform Open Education advocacy, policy, practice and research in developing countries.
Article
The Information Systems field has long been positioned globally as a particularly powerful instrument to promote social inclusion. While individual identity characteristics have been the primary research focus on social inclusion, our research study directly investigates an external environmental influence in social inclusion, course materials cost, and its impact on learning outcomes. General theories of student motivation and cognition in educational outcomes and social inclusion theory were used as a starting point to identify a significant positive difference in the learning outcomes of students who used open text materials contrasted with those who used proprietary text materials. Our research with 198 participants found support for student motivation and open instructional materials as positive influences on student learning outcomes. Implications for theory and future research on social inclusion in education at the environmental level are recommended based on the results.