Content uploaded by Clemens Striebing
Author content
All content in this area was uploaded by Clemens Striebing on May 27, 2019
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=yisr20
Interdisciplinary Science Reviews
ISSN: 0308-0188 (Print) 1743-2790 (Online) Journal homepage: https://www.tandfonline.com/loi/yisr20
Measuring gender in R&I – theories, methods, and
experience
Rachel Palmén, Evanthia Kalpazidou Schmidt, Clemens Striebing, Sybille
Reidl, Susanne Bührer & Dóra Groó
To cite this article: Rachel Palmén, Evanthia Kalpazidou Schmidt, Clemens Striebing,
Sybille Reidl, Susanne Bührer & Dóra Groó (2019) Measuring gender in R&I – theories,
methods, and experience, Interdisciplinary Science Reviews, 44:2, 154-165, DOI:
10.1080/03080188.2019.1603873
To link to this article: https://doi.org/10.1080/03080188.2019.1603873
Published online: 26 May 2019.
Submit your article to this journal
View Crossmark data
Measuring gender in R&I –theories, methods, and experience
Rachel Palmén
a
, Evanthia Kalpazidou Schmidt
b
, Clemens Striebing
c
,
Sybille Reidl
d
, Susanne Bührer
e
and Dóra Groó
f
a
Gender and ICT Research Program, Open University of Catalonia, Barcelona, Spain;
b
Department of Political
Science, Aarhus University, Aarhus, Denmark;
c
Fraunhofer Institute for Industrial Engineering, Stuttgart,
Germany;
d
Joanneum Research, Vienna, Austria;
e
Fraunhofer Institute for Systems and Innovation Research,
Karlsruhe, Germany;
f
The Association of Hungarian Women in Science (NATE), Budapest, Hungary
ABSTRACT
This article provides an overview of the theoretical assumptions,
methods, and key results from the Evaluation Framework for
Promoting Gender Equality in Research and Innovation (EFFORTI)
project, which was funded by the European Commission. The
purpose of EFFORTI was to analyse the impact of interventions to
promote gender equality in research and innovation (R&I), and to
establish criteria for more responsible and responsive research
and innovation (RRI) systems in Europe. This article provides an
overview of the project’s main results and the lessons learnt from
the empirical analysis of R&I systems in several European
countries and a comparison of 19 gender equality intervention
measures.
KEYWORDS
Gender equality; evaluation;
research and innovation;
evaluation framework;
EFFORTI; theory of change
Introduction and purpose
Previous studies of gender equality intervention programmes in research and
innovation (R&I) and their impacts have illustrated numerous implementation and
evaluation issues and a general lack of a systematic understanding of the underlying
mechanisms.
The aim of EFFORTI (Evaluation Framework for Promoting Gender Equality in
Research and Innovation), a project funded by the European Commission, has been to
clarify the interactions between the design and implementation of gender equality inter-
ventions and the expected and achieved results in research and innovation contexts.
The project conducted a comparative analysis of innovation systems, welfare, and
gender equality policy initiatives, as well as of programme evaluation cultures, in
Austria, Denmark, France, Germany, Hungary, Spain, and Sweden. It has also considered
19 case studies of gender equality interventions in R&I.
This paper describes the evaluation framework developed by EFFORTI to help pro-
gramme evaluators, science managers, policy makers, and programme owners to
conduct a sound analysis of the outputs, outcomes, and impacts of gender equality
measures in R&I.
© 2019 Institute of Materials, Minerals and Mining Published by Taylor & Francis on behalf of the Institute
CONTACT Clemens Striebing clemens.striebing@iao.fraunhofer.de Center for Responsible Research and
Innovation, Fraunhofer IAO, Hardenbergstraße 20, 10623 Berlin, Germany
INTERDISCIPLINARY SCIENCE REVIEWS
2019, VOL. 44, NO. 2, 154–165
https://doi.org/10.1080/03080188.2019.1603873
Theoretical background
The motivation behind EFFORTI
Gender equality is a core value of the European Union (EU) (Article 8 of the Treaty on the
functioning of the EU) and is currently one of the six top policy priorities for research and
innovation of the European Commission. As such, it is integrated into the EU Framework
Programmes for research and innovation and into the agenda for European Research
Area. In the current, 8th Framework Programme, Horizon 2020, gender equality is ident-
ified in terms of increasing the proportion of women in scientific roles, in decision-making
positions, as a dimension of research content, and also as a cross-cutting issue running
throughout the societal challenges. A variety of measures and toolkits have been produced
in the last decade to promote gender equality in R&I (GENDER-NET IGAR tool, released
2009, and GEAR Tool, EIGE 2016) and many gender equality-related policy initiatives
and interventions, including projects aiming to implement institutional change (e.g.
STAGES, n.d.) and guidelines for gender equality programmes in science (Cacace 2009;
PRAGES, n.d.).
The aim of EFFORTI was to clarify how the underlying mechanisms for implementing
intervention measures affect results. The result is an evaluation framework, which offers the-
ories on how progress on gender equality objectives affects the desired outcomes of research
and innovation, conventionally measured as number of publications, citations, and patents.
Basic theoretical assumptions of EFFORTI
The evaluation framework developed by EFFORTI is based on two basic assumptions. The
first is that it is not practical nor comprehensive to examine the complex societal effects of
gender equality interventions by using standard linear evaluation designs or experimental
designs with control groups. Evaluation must focus on the often unique context within
which the gender equality intervention is implemented.
The second assumption is that gender equality interventions in R&I institutions have
effects that go beyond gender equality itself by directly and indirectly affecting the
quality and quantity of the research outputs of the implementing R&I institution (Kalpa-
zidou Schmidt, Forbrig et al., 2017, 46).
The underlying analytical framework of EFFORTI is the I–O–O–I (Inputs, Outputs,
Outcomes, and Impact) model, which is used to guide the understanding of the interven-
tion logic in an easily communicable manner. This necessarily involves a simplification of
a complex process. In theory, establishing a causal link between a policy intervention and
an observed impact requires the attribution of the observed change to the policy interven-
tion. In practice, however, complex social contexts make such an approach problematic
(Cartwright and Hardie 2012; Dahler-Larsen 2012; Albæk 1989; Larsen and Lassen
2001). Kalpazidou Schmidt and Cacace (2017), for example, note how most approaches
to societal impact assessments focus on simple, linear models often embedded in a reduc-
tive causal-chain logic instead of taking a complexity stance. In line with the increasing
recognition of the importance of contextual conditions in complex frameworks, as the
ones studied in the EFFORTI project, notions of ‘attribution’have, as a general rule,
been replaced by ideas of ‘contribution’, which offer a more promising approach that
can factor in complexity as well as context (Mayne and Johnson 2015).
INTERDISCIPLINARY SCIENCE REVIEWS 155
Therefore, the EFFORTI method approaches effects, outcomes, and impact concepts by
means of evaluative analyses that follow ‘contributions’to achieve impact. As underlined
in the literature, the ability of complex intervention programmes to foster the right con-
ditions for change is central to success (Reale et al. 2014). Therefore, impact assessment
has to account for whether adequate ‘conditions for impact’are in place (Kalpazidou
Schmidt and Cacace 2017).
Methods and materials
Comparison of national contexts for gender equality programmes in R&I
In order to get a better overview of the differences and similarities between the insti-
tutional contexts for gender equality interventions in R&I of different EU countries, con-
crete analyses were carried out for Austria, Denmark, France, Germany, Hungary, Spain,
and Sweden.
The EFFORTI evaluation framework
In the development of the framework, several gender equality intervention examples and
evaluations within R&I were collected, representing a large variation in terms of geo-
graphical and organizational contexts, to facilitate the development of the list of progress
indicators to be used in EFFORTI. The evaluation framework was optimized through peer
reviews (Kalpazidou Schmidt, Bührer et al. 2017).
19 Case studies of gender equality interventions in R&I
The case study work followed concrete instructions for data collection, including research
questions, a detailed interview guide, and reporting templates.
The methods used in the case study work were documentary analysis and semi-
structured interviews with policy makers, programme managers, practitioners, and
beneficiaries.
Results
Comparison of national contexts for gender equality programmes in R&I
1
As already mentioned, gender equality interventions take place in complex environments
that can influence the design, implementation, and impact of these measures. The follow-
ing contextual framework conditions were considered as relevant:
.the structure and performance of the R&I system,
.gender equality policies in the labour market and welfare policies related to reproduc-
tive work and childcare,
.the governance and existing policies of gender equality in R&I, and
.the evaluation culture and policy especially in the field of gender equality in R&I.
1
Unless otherwise noted, this section is based on Reidl et al. (2018).
156 R. PALMÉN ET AL.
A main characteristic of the innovation system is the relevance of the constituent
sectors. All EFFORTI countries except Spain are more or less dominated by the business
enterprise sector (BES). The different relevance of the sectors has an impact on gender
equality in R&I. The share of female researchers working in the BES remains lower com-
pared to the higher education sector in all EFFORTI countries. An important reason for
this is that gender equality regulations in R&I mainly focus on the public sector and there
is a lack of laws or regulations for the private sector in all compared countries. Other
reasons for a low share of women in R&I might be the horizontal segregation, male organ-
izational cultures, and governance issues. The share of women among ISCED 6 graduates
increases, but the horizontal segregation in respect to fields of study is quite significant in
all countries; with the lowest shares of women in engineering, manufacturing, and con-
struction. Vertical segregation is also present in all countries; meaning the higher the
grade the lower is the proportion of women in EU28 and the EFFORTI countries. The
same can be said about female members on boards, especially in the economic sector,
where the share of women on boards in the largest companies, supervisory boards or
boards of directors hardly exceeds 25%.
Another aspect of gender equality in R&I concerns employment conditions as the
number of hours worked per week influences the work–life balance. In a culture of pre-
senteeism, the duration of working time can influence a career. Women professionals
tend to work fewer hours than men. But an alignment of working hours between male
and female professionals can be observed. The working hours for Austria and Germany
for full-time employees have so far been well above the EU average of 40 h, but have
been decreasing slowly since 2005. Both countries have hitherto been characterized by
an overtime culture. For the R&I sector, Austria and Germany still report a significantly
higher weekly working time of 44 h, compared to the other EFFORTI countries where
there is nearly no difference between general working time culture and that in R&I.
This working time culture does not make R&I attractive to women in Austria and
Germany because both countries are also characterized by comparatively high shares of
women working part-time, in the R&I sector as well as in the whole labour market –
often to accommodate work and childcare. In contrast, e.g. Denmark shows a low percen-
tage of both women and men in part-time employment in R&I. This might be due to a
combination of sufficient childcare facilities but also due to high demands for a stable
career path with high publication pressure.
Working conditions in R&I are influenced by the general working culture on the labour
market and the possibilities to reconcile work and private life. Concerning gender equality
in the labour market, the comparison shows that the labour market participation of
women is generally lower compared to men but differs widely between the EFFORTI
countries. A major reason for this is the impact of parenthood, which is influenced by
the division of labour in relation to domestic work and family care, labour market policies,
the availability of childcare facilities and gender roles. This can be seen for example in
Austria and Germany, where part time work is highly prevalent among women whereas
it is less common in Denmark and Sweden. As to the provision of childcare, high enrol-
ment rates of children under 3 years old can be seen in the Nordic countries, while they are
relatively low in the continental like Germany, Austria, or Hungary.
Lastly, a comparison of the evaluation cultures and policies among the EFFORTI
countries shows that evaluation culture development and capacity building vary greatly
INTERDISCIPLINARY SCIENCE REVIEWS 157
between the countries in terms of recognition of its importance and value, the methods
used, systematic adoption by institutions, the actors involved, and the trend towards stan-
dardization. The evaluation traditions vary across countries regarding the preferred types
of evaluations (e.g. ex-ante, interim, ex post, pluralistic approach), interest in the
approaches to quantify impacts, to evaluate implementation, or to measure causal mech-
anisms relating policy initiatives to their effects and impact. However, whilst interest in the
impacts and effectiveness have been found in all EFFORTI countries, the role of gender
equality in R&I is perceived very differently across the countries. On the one side,
Hungary and Spain do not consider gender equality in R&I evaluations at all, whilst
Sweden has a long tradition of monitoring gender equality in R&I. At the same time, in
Austria, Denmark, France, and Germany, gender equality in R&I has only recently
received growing attention in R&I evaluations.
The EFFORTI evaluation framework
The EFFORTI intervention logic
EFFORTI explored both theory and tools for analysing how gender equality-related
measures contribute to the achievement of three objectives: gender equality in careers,
gender balance in decision making, and integration of gender dimension in research
content. These objectives have been considered at three levels: team level (e.g. research
quality, productivity, innovative outputs,); organizational/institutional level (e.g. work-
place quality, recruitment capacity, efficiency); and country/system/policy level (pro-
ductivity), using theory-based impact evaluation approach (TBIE) (Fitz-Gibbon and
Morris 1996; Kalpazidou Schmidt and Cacace 2017). Theory-driven evaluations, especially
in the form of change models, focus on the following questions: (i) in which way and (ii)
under which conditions a programme intervention contributes to the intended and unin-
tended effects (Döring and Bortz 2016).
Impact story catalogue
A method of impact stories was developed to capture examples of good practices of gender
equality interventions in R&I and deliver stylized information on different types of inter-
ventions, suitable indicators to measure outputs, outcomes and impacts but also some
basic information on the organizational and policy context. This helped ensure confiden-
tiality of single case studies.
The impact stories represent ideal type impact chains, describing the outputs (immedi-
ate technical results), outcomes (direct social effects), and impacts (intended middle- or
long-term effects beyond the beneficiaries) of commonly used gender equality measures.
The impact stories are used to explain how the elements of the I–O–O–I chain (input,
output, outcome, and impact dimensions) sequentially interact with each other, as well
as which indicators can verify the expected positive effects, and which effects have to be
considered and how they can be fostered or, respectively, avoided.
First, the measure’s particular activities, outputs, outcomes and impacts are identified.
For example, a revision of internal promotion policies according to gender equality stan-
dards has the potential to lead to increased transparency in promotion processes which, in
turn, enables better career planning, improves work climate and job satisfaction (e.g. pro-
motions are perceived as more justified), and may also lead to higher performance
158 R. PALMÉN ET AL.
incentives for all team members (as performance requirements are clarified). Suitable
quantitative and qualitative indicators are attributed to all levels of the impact chain
(output, outcome, impact). All impact stories are constructed based on the same structure:
1. Definition of the gender equality measure
2. Output: Description of the measure’s immediate technical results and output indicators
3. Outcome: Description of the measure’s direct social effects and outcome indicators
4. Impact: Description of the measure’s intended middle- or long-term effects and impact
indicators
5. Description of the policy and organizational context.
The impact stories have identified the final set of intervention targets:
1. Increase the number of women in R&I positions
2. Improve working conditions/work–life balance
3. Boost professional capabilities of women to pursue promotion
4. Implement gender-fair organizational structures
5. Integrate the gender dimension in research and teaching
6. Foster ethics, public engagement, science education, open access and/or governance
7. Increase R&I outputs and impacts.
Indicator catalogue
From a dense list of more than 500 potential indicators, five main categories and a core set of
indicators have been selected for the evaluation framework. The five main categories are:
Personnel, Working conditions, Professional capabilities, Structural features, and R&I/RRI.
2
The developed indicator system covers all three levels of policy interventions, namely
micro (dealing with individuals or teams), meso (focusing on organizational issues such
as institutional rules, incentives, structures, and processes), and macro (system rules,
incentives, structures, and processes at regional, national or supranational level). In prac-
tice, the distinction between micro, meso, and macro levels may not be entirely clear-cut,
since the different levels are interrelated and a number of indicators can be applied at more
than one of these levels.
Theory of change-model
Both researchers and practitioners have increasingly looked to theories of change (ToC) as
a valid approach to evaluation in a variety of sectors and fields (Coryn et al. 2011, cited in
Ofek 2017, 175). A ToC can be used to shed light on the implementation of an interven-
tion, i.e. ‘what is required to translate objectives into ongoing service delivery and pro-
gramme operation’(Blamey and Mackenzie 2007, 444 cited in Ofek 2017, 175). It can
also be used to shed light on programme theory, i.e. the hypothesized sequential chain
and ‘mechanisms linking interventions to outcomes, and the reasons behind the develop-
ment of such mechanisms within specific contexts’.
The EFFORTI case study template was designed to help standardize the information
that was collected through the case study work to construct and validate the theories of
2
A full list of all indicators can be found in Kalpazidou Schmidt et al. (2018).
INTERDISCIPLINARY SCIENCE REVIEWS 159
change carried out in the 19 case studies throughout 6 countries in order to validate the
evaluation framework.
Lessons learned from case studies
3
For each case study, a theory of change was developed based on three main axes: concept/
design analysis, implementation analysis, and an impact assessment to identify the
strengths and weaknesses in the design of the interventions, factors affecting smooth
implementation and to draw contrasts between ‘expected’outcomes and impacts with
‘observed’outcomes and impacts.
Lessons for design
The analysis of the 19 case studies demonstrated various strengths and weaknesses of the
design of the interventions which was linked to their type and sub-field of action. Strengths
included: data-driven and evidence-based intervention design; mainstreaming of gender
equality throughout every step of assessment procedures; and tailoring a mix of measures,
i.e. combining those interventions aiming for a greater gender balance higher up the career
ladder with necessary structural change interventions.
Innovative intervention designs as for example ‘future potential analysis’–where a can-
didate for a leadership position is assessed for her/his ‘future potential’as opposed to past
achievement –was deemed successful in challenging the, often gender biased, assessment
procedures. How monitoring and transparency were embedded into intervention design
were also shown to be critical factors not only positively affecting outcomes but also for
demonstrating impact. Outcomes could be most readily detected in those interventions
that integrated the gender dimension into research content and in tertiary education
(i.e. which had both gender equality and R&I as main objectives).
Another factor that affected the impact of the intervention was the definition and
selection of the target group. In some cases, this was linked to whether or not it was
conceived as a positive action measure, i.e. specifically targeting women. In some
instances, funding targeted specifically at women was perceived negatively, yet in
other instances it provided a more concrete objective for the programme, which led
to a higher demand. In one case study, there was a real mismatch between the interven-
tion design and the target group, the programme was designed as a research funding call
but the target group was human resource managers who may not have had the necess-
ary training or skills to apply for the call. Also, how gender equality is conceived in the
intervention design, i.e. as equal participation of women and men in R&I or as organ-
izational and cultural change –was seen to contribute to its impact. In one case study,
gender equality was conceived as equal balance of women and men in R&I but lack of
focus on organizational and cultural change was identified as a design weakness, nega-
tively affecting outcomes.
Lessons for implementation
A variety of facilitating and hindering factors in the implementation of interventions were
identified through the case studies. Unlike the differences identified for intervention
3
Unless otherwise noted, this section is based on Palmén et al. (2018).
160 R. PALMÉN ET AL.
design –these tended to be cross-cutting across all types of interventions and sub-fields of
action. The governance framework was identified as a key driver contributing to impact.
This was particularly true where appropriate gender equality legislation was available and
actively implemented. In one case study, legislation requiring integration the gender
dimension into teaching in higher education was being implemented by an accreditation
agency, and an explanation had to be provided by the teaching institution if this was per-
ceived as not relevant for a given course. In addition to the general agreement on the
importance of top-level commitment to the implementation of gender equality interven-
tions, a bottom-up buy-in, and whether the intervention was promoted as a positive action
measure, also played a positive role. In some instances –funding targeted specifically at
women was perceived negatively, yet in other instances it provided a more concrete objec-
tive for the programme –which led to a higher demand.
Developing synergies with influential initiatives promoted by important institutions
was seen as significant for legitimizing interventions. For example, the German Research
Council’s (DFG) commitment to advance standards for research and research excellence
in Germany was highlighted as a trend setter, paving the way for more general acceptance
of gender equality interventions in R&I. Availability of resources for the implementation
of interventions was considered as crucial to creating an effective and long-term impact in
almost every case study. Gender competence, experience, and knowledge were also high-
lighted as key –and in those case studies where implementation was not optimal, it
could be linked to a lack of gender competence and experience. In one case study, where
external gender expertise could be brought into project design, this level of support and
external competence greatly facilitated the implementation process. It was also highlighted
that a positive evaluation of including gender dimension in research, or participating in
gender equality actions in research curriculum, might provide an incentive to boost compe-
tence. Formulating targets and standards followed up by monitoring of progress were seen
as necessary for successful implementation and conversely a lack of accessible data and
information had a negative impact on the smooth implementation of interventions. Positive
attitudes, interest, and motivation to participate were identified as crucial to successful
implementation, and to the contrary, resistance was identified as hindering implementation.
Some pro-active strategies to tackle and pre-empt resistance for a more effective implemen-
tation were identified in the case study work as aiding impact, for example integrating
gender equality issues into meetings with directors and managers thereby giving status to
gender equality as a strategic issue for the institution was identified as such a stratagem.
Lessons for outcomes
Gender equality and R&I outputs, outcomes and impacts were tracked throughout our
case studies and could be seen to be linked to the type of the intervention and the field
of action. In some case studies where the main objectives were linked to gender equality –
it was more difficult to discern R&I impacts and vice versa. In those programmes or inter-
ventions promoting scientific excellence or innovation –if they did not include an explicit
gender equality objective –gender equality impacts were more difficult to track. It was in
the field of integrating the gender dimension in research content and tertiary education –
where both gender equality and R&I impacts were most interlinked. In some cases, it was
difficult to substantiate with empirical evidence the achieved gender equality and R&I out-
comes and impacts because intervention monitoring was limited to tracking outputs. In
INTERDISCIPLINARY SCIENCE REVIEWS 161
these cases, our qualitative approach enabled us to pinpoint where future evaluations and
research might want to focus in order to provide evidence for potential ‘outcomes’and
‘impacts’.
General lessons
The case studies show that the intensity and quality of programme evaluations is highly
dependent on the national evaluation cultures. Some interventions were not monitored
and had no data gathering mechanisms built into the intervention. Others were monitored
but no evaluation or impact assessment had been carried out. In a few case studies, com-
prehensive monitoring had been carried out accompanied by evaluation which may or
may not include impact assessment. Thus, in cases where a strong evaluation cultures
exists (like Austria, Germany and Sweden), the programmes were more comprehensively
evaluated than for example in Spain or Hungary.
Conclusion
The results of the analyses carried out in EFFORTI lead to a number of recommendations
primarily addressed to policy makers, practitioners, and evaluators. The EU countries sur-
veyed are currently reluctant to impose bureaucratic gender equality obligations on the
business sector. However, the fact that regulation has an effect, and is manageable, is
demonstrated by the progress made in gender equality in the higher education and gov-
ernment sectors. With regard to work–life balance, the country level comparison
confirmed the ‘leaky pipeline’phenomenon as a key factor in the careers of women
researchers, which poses specific challenges for all countries studied (including the Scan-
dinavian countries).
The issue of ensuring an appropriate complexity of a gender equality intervention and
of their evaluation poses a challenge for practitioners to take sufficient account of the
influence of regional and organizational context factors, but at the same time to
develop a simple and ‘manageable’programme change theory according to which plan-
ning, implementation, and evaluation can be carried out.
1. Intervention design must be evidenced based, factor in complexity and be tailored to
the local context.
2. Data collection, monitoring, and evaluation should be embedded into the intervention
design from the outset and relevant information structures and systems developed for
effective implementation to demonstrate impact (or lack of).
3. For pragmatic reasons, complex programme change theories should be based on what
is feasible in the given context and not based exclusively on one-size fits all statistical
evidence.
The most effective case study interventions were those that were sufficiently tailored to
the local context and recognized the multi-faceted nature of effectively tackling gender
inequalities in R&I. They were also able to demonstrate the outcomes and impacts associ-
ated with the interventions in gender equality as well as in R&I –as sufficient data collec-
tion processes and procedures had been successfully embedded within the intervention.
162 R. PALMÉN ET AL.
This is key to the further development of the evidence base about the impacts of interven-
tions in the fields of gender equality and R&I.
4. Interventions should take into consideration the social impact of gender equality
interventions in R&I from the outset and embed relevant monitoring and evaluation
processes and procedures into programme design.
5. Gender equality interventions in R&I should not be limited to promoting a gender
balance of women and men but should aim for organizational and cultural change.
Although all case studies examined were identified as good practices in institutional,
regional, or national gender equality interventions, and in some instances recognized as
a‘good’practice on the international stage, it was almost impossible to draw conclusions
on potential R&I impacts from existing project reports and evaluations. Only in the Aus-
trian example it was really possible to observe that a gender equality intervention was
designed to take into consideration its possible R&I effects.
There are reservations about legitimizing gender equality interventions via their
R&I effects: what if these effects fail to materialize? Are subsidies also endangered
then? Is gender equality not a legitimate end in itself? Those responsible for gender
equality and subsequent bodies within an organization are often the only bodies in
an organization concerned with gender equality. Rolling out the desired effects of
gender equality programmes to the ‘R&I core business’can be perceived within an
organization as an attempt by the gender equality manager to expand his or her
own competencies while at the same time going beyond his or her original remit.
6. Funders of gender equality interventions in R&I should be willing to take risks pro-
moting innovative intervention design and embed monitoring and evaluation into
programme design and implementation to demonstrate impact (or lack of) in both
gender equality and R&I.
7. Gender competence must be promoted by providing resources (training, or experts)
and positively evaluated (i.e. for researchers this would mean recognizing the inte-
gration of the gender dimension in research content and gender expertise in organiz-
ational change in research curricula).
8. Pro-active strategies need to be embedded into intervention design to not only tackle
but also pre-empt resistance.
Gender equality interventions in R&I have tended to focus on ‘good’,‘best’, and ‘smart’
practices –this has resulted in a tendency to reproduce ‘successfully’demonstrated inter-
ventions and initiatives. Whilst on one level this makes sense, it may also stymie energies
that are invested into innovative intervention designs. Funders should be willing to invest
in innovative solutions to tackle the entrenched gender inequalities in R&I whilst setting
up the necessary data collection processes and procedures to track outcomes and impact
(or lack of). Innovative design has the potential to really challenge entrenched gender
biased processes and procedures. Whilst funders may be reluctant to risk channelling
scarce resources into untried and untested solutions, evidencing the lack of outcomes and
impact provides valuable information about what does not work in what context and why.
INTERDISCIPLINARY SCIENCE REVIEWS 163
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
This work was supported by the European Union’s Horizon 2020 research and innovation pro-
gramme under [grant number 710470].
Notes on contributors
Rachel Palmén is Senior Researcher at the Gender and ICT Research Program, Internet Interdisci-
plinary Institute, Open University of Catalonia and at Notus: Applied Social Research. She special-
izes in gender (in)equalities and science and interventions in this field focusing on the impact of
policy and other initiatives. Twitter handle @rachelpalmen.
Evanthia Kalpazidou Schmidt is Associate Professor and Research Director at the Department of
Political Science, Aarhus University. She specializes in science and innovation policy; Responsible
Research and Innovation (RRI); gender in knowledge production and research organizations; Euro-
pean gender policies and strategies; impact of knowledge governance and policy interventions;
research evaluation; European research policy and governance.
Clemens Striebing is Researcher at the Center for Responsible Research and Innovation (CeRRI) of
Fraunhofer IAO. At Fraunhofer CeRRI, he is concerned with gender equality in the innovation
system, organizational cultures and the transfer and exploitation of research outputs.
Sybille Reidl is Senior Researcher at the Institute for Economics and Innovation Research of JOAN-
NEUM RESEARCH, which is focused on applied research and consulting services in the fields of
science, technology, and innovation (STI). Her main research areas are gender/diversity and human
resources in STI and evaluation of gender equality policies and initiatives in R&I.
Susanne Bührer has been employed at Fraunhofer ISI in Karlsruhe since 1996, and has been the
Coordinator of the Business Unit ‘Policy Design and Evaluation’since 2010. Her specialized
research fields include program evaluations, monitoring the evaluation of institutional funding
measures, the topic of gender and innovation, and responsible research and innovation issues.
Twitter handle: @FraunhoferISI.
Dóra Groó is President of the Association of Hungarian Women in Science. A medical doctor by
training she made her PhD in experimental medicine and worked for 10 years in pharmaceutical
research. From 1994 to 2012, she was Director of the Hungarian Science and Technology Foun-
dation (TETALAP).
ORCID
Rachel Palmén http://orcid.org/0000-0003-3038-2807
Evanthia Kalpazidou Schmidt http://orcid.org/0000-0002-3204-0803
Clemens Striebing http://orcid.org/0000-0003-1410-3716
Sybille Reidl http://orcid.org/0000-0001-6453-5926
Susanne Bührer http://orcid.org/0000-0002-1923-1437
References
Albæk, Erik. 1989.“Policy Evaluation: Design and Utilization.”Knowledge in Society 2 (4): 6–19.
Blamey, Avril, and Mhairi Mackenzie. 2007.“Theories of Change and Realistic Evaluation.”
Evaluation 13 (4): 439–455. doi:10.1177/1356389007082129.
164 R. PALMÉN ET AL.
Cacace, Marina. 2009.Guidelines for Gender Equality Programmes in Science. PRAGES –Practising
Gender Equality in Science. Rome: Ministry of Economy and Finance.
Cartwright, Nancy, and Jeremy Hardie. 2012.Evidence-Based Policy: A Practical Guide to Doing It
Better. Oxford: Oxford University Press.
Coryn, Chris, Noakes Lindsay, Westine Carl, and Daniela Schroeter. 2011.“A Systematic Review of
Theory-Driven Evaluation Practice from 1990 to 2009.”American Journal of Evaluation 32 (2):
199–226. doi:10.1177/1098214010389321.
Dahler-Larsen, Peter. 2012.The Evaluation Society. Stanford: Stanford University Press.
Döring, Nicola, and Jürgen Bortz. 2016.Forschungsmethoden und Evaluation in den Sozial- und
Humanwissenschaften. Berlin: Springer.
EIGE. 2016. “Gender Equality in Academia and Research: GEAR Tool.”Unpublished manuscript,
last modified September 19, 2018.
Fitz-Gibbon, Carol T., and Lynn Lyons Morris. 1996.“Theory-based Evaluation.”Evaluation
Practice 17 (2): 177–184.
Horizon. 2020.“The Framework Programme for Research and Innovation: Communication from
the Commission to the European Parliament, the Council, the European Economic and Social
Committee and the Committee of the Regions.”2011. COM (2011) 808 final. Unpublished
manuscript, last modified November 19, 2015.
Kalpazidou Schmidt, Evanthia, and Marina Cacace. 2017.“Addressing Gender Inequality in
Science: The Multifaceted Challenge of Assessing Impact.”Research Evaluation 26 (2): 102–
114. doi:10.1093/reseval/rvx003.
Kalpazidou Schmidt, Evanthia, and Marina Cacace. 2018.“Setting up a Dynamic Framework to
Activate Gender Equality Structural Transformation in Research Organizations.”Science &
Public Policy 34 (2): 248. doi:10.1093/scipol/scy059.
Kalpazidou Schmidt, Evanthia, Susanne Bührer, Martina Schraudner, Sybille Reidl, Jörg Müller,
Rachel Palmén, Sanne Haase, Ebbe Krogh Graversen, Florian Holzinger, Clemens Striebing,
Dora Groó, Saskia Klein, Dorottya Rigler and Ea Høg Utof. 2017.“A Conceptual Evaluation
Framework for Promoting Gender Equality in Research and Innovation: Toolbox I –A synthesis
report.”https://www.efforti.eu/sites/default/files/2018-03/EFFORTI%20D3.3%20FINAL%20rep
ort%2027032018.pdf.
Kalpazidou Schmidt, Evanthia, Lea Forbrig, Susanne Bührer, and Cheng Fan. 2017.“EFFORTI –
Deliverable 3.1. Collection of quantitative and qualitative indicators for an Evaluation
Framework for Promoting Gender Equality in Research and Innovation.”
Larsen, Flemming, and Morten Lassen. 2001.“Evalueringen set i et Policy-Perspektiv.”In
Tendenser i Evaluering, edited by Peter Dahler-Larsen and Hanne Kathrine Krogstrup, 46–60.
Odense: Syddansk Universitetsforlag.
Mayne, John, and Nancy Johnson. 2015.“Using Theories of Change in the CGIAR Research
Program on Agriculture for Nutrition and Health.”Evaluation 21 (4): 407–428. doi:10.1177/
1356389015605198
Ofek, Yuval. 2017.“An Examination of Evaluation Users’Preferences for Program and Actor-
Orientated Theories of Change”,Evaluation, 23, (2): 172–191. doi:10.1177/1356389017699544
Palmén, Rachel, Sybille Reidl, Susanne Bührer, Evanthia, Kalpazidou Schmidt, Florian Holzinger,
Dora Groó, Dorottya Rigler, Clemens Striebing, Lidia Arroyo, Maria José Romano, Ebbe
Krogh Graversen, Ea Høg Utoft, Mathilde Cecchini, Sarah Beranek, Jürgen Streicher,
Maximilian Unger and Annika Koch. 2018.“Synthesis Report: EFFORTI –Deliverable 4.2
(Public Report).”
Reale, Emanuela, Maria Nedeva, Duncan Thomas, and Emilia Primeri. 2014.“Evaluation Through
Impact: A Different Viewpoint.”Fteval Journal 39: 36–41.
Reidl, Sybille, Florian Holzinger, Jürgen Streicher, Sarah Beranek, Maximilian Unger, and Silvia
Hafellner. 2018.“Comparative Background Report: EFFORTI - Deliverable 2.3 (Public
Report).”https://www.efforti.eu/sites/default/files/2018-03/EFFORTI%20WP2%20Comparative
%20Report%20Final.pdf.
PRAGES. n.d. PRAGES project. http://www.pragesdatabase.eu.Reale, Emanuela, Maria.
STAGES. n.d.STAGES project.http://www.stagesproject.eu.
INTERDISCIPLINARY SCIENCE REVIEWS 165
