A global call for action to include gender in research impact assessment

Abstract

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we – a group of scholars and practitioners from Africa, America, Asia and Europe – argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Full text

C O M M E N T A R Y Open Access
A global call for action to include gender in
research impact assessment
Pavel V. Ovseiko
1*
, Trisha Greenhalgh
2
, Paula Adam
3
, Jonathan Grant
4
, Saba Hinrichs-Krapels
4
, Kathryn E. Graham
5
,
Pamela A. Valentine
5
, Omar Sued
6
, Omar F. Boukhris
7
, Nada M. Al Olaqi
8
, Idrees S. Al Rahbi
9
, Anne-Maree Dowd
10
,
Sara Bice
11
, Tamika L. Heiden
12,13
, Michael D. Fischer
14,15
, Sue Dopson
15
, Robyn Norton
16,17
, Alexandra Pollitt
4
,
Steven Wooding
18
, Gert V. Balling
19
, Ulla Jakobsen
20
, Ellen Kuhlmann
21,22
, Ineke Klinge
23
, Linda H. Pololi
24
,
Reshma Jagsi
25
, Helen Lawton Smith
26
, Henry Etzkowitz
27,26
, Mathias W. Nielsen
28
, Carme Carrion
3,29
,
Maite Solans‐Domènech
3
, Esther Vizcaino
3
, Lin Naing
30
, Quentin H. N. Cheok
31
, Baerbel Eckelmann
32
,
Moses C. Simuyemba
33
, Temwa Msiska
34
, Giovanna Declich
35
, Laurel D. Edmunds
1
, Vasiliki Kiparoglou
36,37
,
Alison M. J. Buchan
38
, Catherine Williamson
39,40,41
, Graham M. Lord
40,41,42
, Keith M. Channon
1,36,37
,
Rebecca Surender
43,44
and Alastair M. Buchan
1,36,37
Abstract
Global investment in biomedical research has grown significantly over the last decades, reaching approximately a
quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows,
arguably, that scarce research resources may not be optimally invested (by either not supporting the best science
or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be
significantly underrepresented in research both as researchers and research participants, receive less research
funding, and appear less frequently than men as authors on research publications. There is also some evidence that
women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic
impacts. Historical gender biases may have created a path dependency that means that the research system and
the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult
(though not impossible) to eliminate gender bias. In this commentary, we –a group of scholars and practitioners
from Africa, America, Asia and Europe –argue that gender-sensitive research impact assessment could become a
force for good in moving science policy and practice towards gender equity. Research impact assessment is the
multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and
economic returns on investment in research. It encompasses many theoretical and methodological approaches that
can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer
a set of recommendations to research funders, research institutions and research evaluators who conduct impact
assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a
global call for action.
Keywords: Research impact assessment, Gender, Path dependency, Health research, Science policy, Athena SWAN,
Call for action
* Correspondence: pavel.ovseiko@medsci.ox.ac.uk
1
Medical Sciences Division, University of Oxford, John Radcliffe Hospital,
Oxford OX3 9DU, United Kingdom
Full list of author information is available at the end of the article
© 2016 The Author(s). Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Ovseiko et al. Health Research Policy and Systems (2016) 14:50
DOI 10.1186/s12961-016-0126-z
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Gender bias in health research
Global investment in biomedical research has grown sig-
nificantly over recent decades. In 2010, global invest-
ment reached US$240 billion (adjusted for purchasing
power parity), delivering important health dividends to
patients and citizens [1]. However, not all of this invest-
ment is distributed evenly by gender. It follows, argu-
ably, that scarce research resources may not be
optimally invested by either not supporting the best sci-
ence or by failing to investigate topics that benefit
women and men equitably. Gender bias in biomedical
and health research involves both biological sex differ-
ences and sociocultural differences in the way women
and men behave, and in the way they are treated [2].
There is evidence that gender bias in biomedical and
health research can occur at all stages of the research
process across the following four domains.
First, women tend to be significantly underrepresented
in research both as researchers and research partici-
pants. Although in 2013, women had reached 55% of ad-
missions to medical schools in the United Kingdom and
47% in the United States of America, they constituted
only 28% of faculty physician-scientists in the United
Kingdom and 38% in the United States [3]. Moreover, fe-
male faculty members tend to report less favourable ex-
periences and feel excluded [4, 5]. Women are also
underrepresented as research participants. Historical
analysis demonstrated a male bias in biomedical research
throughout the 20th century: it was evident in 8 of the
10 biomedical fields surveyed in 2009 [6]. For example,
while women represent nearly half of people living with
HIV, they are under-represented in clinical studies of
HIV antiretroviral drugs (19%), prophylactic vaccines
(38%) and curative strategies (11%) [7]. Women are also
under-represented in high-impact studies of non-sex-
specific cancers [8].
Second, female investigators tend to receive less re-
search funding than their male counterparts in absolute
and relative terms. This may occur because there are
fewer women investigators who apply for research fund-
ing [9], and those who apply receive smaller awards than
men [10]. There is some evidence that, in certain set-
tings, this is also amplified by reported nepotism and
sexism in peer-review [11]. Although an earlier meta-
analysis of empirical studies from different fields con-
cluded that women applying for grants have statistically
significant lower odds of receiving funding than men by
approximately 7% [12], a more recent and methodologic-
ally advanced meta-analysis of the same data [13] and a
recent empirical study [14] concluded the contrary.
United States research suggests that female early career
researchers receive significantly less start-up support
from their institutions [15] and are significantly less
likely than men to achieve independent funding awards
[16]. Further, United States research suggests that
women at particular career stages are less likely to apply
for the competitive grants for which they are eligible,
compared to their male counterparts [17]. Research
from the Netherlands demonstrated gender bias favour-
ing male grant funding applicants in the evaluation of
the ‘quality of researcher’, but not the ‘quality of pro-
posal’[18]. Even in the fields where there is no differ-
ence in funding rates between the genders, such as
radiology, women have less total grant funding than
men [19].
Third, women tend to appear less frequently than men
as winners of prestigious scientific awards and as authors
of research publications. Among 210 Nobel Laureates in
Physiology or Medicine awarded from 1901 to 2015, there
are only 12 (5.7%) women [20]. Despite significant pro-
gress in recent decades, women are still underrepresented
as authors of research articles in medical journals [21, 22],
especially as first and senior authors [23, 24]. For example,
the proportion of women first authors in six prominent
international medical journals increased from 6% in 1970
to 29% in 2004, and the proportion of women senior au-
thors increased during the same period from 4% to 19%
[23]. In radiology, the proportion of women first authors
increased from 8% in 1978 to 32% in 2013 [25]. The pro-
portion of women first authors in high impact general
medical journals increased from 27% in 1994 to 37% in
2014, but it has recently plateaued and even declined in
some journals [26].
Finally, women may be disadvantaged as the benefi-
ciaries of research in terms of its health, societal and
economic impacts [27–29]. There is evidence to suggest
that research that does not account for gender differ-
ences can result in inaccurate conclusions about how
women respond to disease and this, in turn, will influ-
ence the effectiveness of treatment choices [30, 31]. For
example, ‘Yentl syndrome’describes sex bias in the man-
agement of coronary heart disease due to the fact that
medical research had predominantly studied symptoms
of heart attacks in men [32]. Historically, those women
who presented with symptoms of heart attack similar to
those in men received the same diagnostic and treat-
ment procedures as men, but those whose symptoms
presented differently were not properly treated and may
have died unnecessarily. For such reasons, the League of
European Research Universities has recently stressed
that, without including gender analysis in research, the
impact of science may not be equally beneficial for both
men and women [33].
In what follows, we elaborate on the path dependent
nature of gender bias in science and why it is difficult,
but not impossible to address. Second, we outline key
characteristics of research impact assessment; we argue
that by investigating gender bias, research impact
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 2 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

assessment can become a force for good in moving sci-
ence policy and practice towards gender equity. Third,
we offer a set of recommendations about how research
funders, research institutions and research evaluators
can include and strengthen analysis of gender equity in
research impact assessment. We conclude by issuing a
global call for action.
Path dependency of gender bias
Historical gender biases have generated a form of path
dependency such that the research system, including re-
search impacts, is, to some extent, male oriented. In
most countries, women began entering higher education
institutions in substantial numbers only from the begin-
ning of the 20th century. However, over the last century,
remarkable social progress has been made, across much
of the globe, towards gender equity. Proponents of the
so-called ‘pipeline’argument believed that once numbers
of women entering universities reached sufficient num-
bers and they were not discriminated against for admis-
sion into the pipeline, gender bias in the representation
of women in science would gradually disappear [34–38].
Indeed, in many countries where there are strict non-
discrimination laws, women have reached 40% or more
in admissions to medicine and other university health
science degrees –a figure that is sometimes used as a
threshold for gender balance [39]. Nevertheless, gender
bias in health research still persists because the path
dependent nature of science makes it difficult to change
the status quo.
The notion of path dependency, which originates in
economics and political science, suggests that our
current actions depend on existing knowledge and past
decisions, and therefore strong conjunctural forces are
required to move policy and practice away from the
established path [40]. Research on path dependency in
political science suggests that such forces are associated
with rational values and a strong centralised authority
such as governments wishing to change current policy
and practice [40, 41]. Path dependent gender bias in re-
search is hard to address because it is often institutiona-
lised as policies, practices, beliefs and written or
unwritten rules of behaviour that structure modern sci-
ence and society.
The path dependent tendency of science is aptly illus-
trated by Isaac Newton’s metaphor of ‘standing on the
shoulders of giants’. Historically, these giants of science
have been overwhelmingly male [42]. The ‘Matthew
Matilda effect’highlights the tendency for women’s work
to be systematically omitted in the history of scientific
achievements or even to be misattributed to men [43].
This is part of a wider historical tendency within a range
of societies (regardless of culture, religion or political or-
ganisation) for an unequal distribution of scientific
resources and power between men and women. Both the
current body of scientific knowledge and current scien-
tific practices are consequently shaped predominantly by
male perceptions and norms, which are often influenced
by both conscious and unconscious gender bias.
As far as scientific knowledge is concerned, there has
been a propensity not to take into consideration possible
sex and gender differences in the research design and
analysis of clinical trials [44, 45]. However, given that
both the occurrence and outcomes of a range of medical
conditions differ for men and women, a large body of
clinical knowledge which does not take relevant sex and
gender differences into account may be flawed [44]. For
example, 50 years ago, few studies included women in
prospective cohort studies of all-cause or coronary heart
disease mortality [46]. Although a growing number of
peer-reviewed health research journals now have editor-
ial policies requiring sex- or gender-specific reporting of
results [29], there remain many journals without such
policies [47].
Scientific practices and structures can also be biased
against women. Male and female leaders may have dif-
ferent leadership traits: women are more likely to focus
on collaboration and may have less interest, and fare less
well, in hierarchical organisation [48, 49]. However, ap-
pointment criteria for scientific leadership are often
based on classical ‘male-gendered’traits [50–53]. Aca-
demic promotion and tenure criteria have also tradition-
ally been based on a male-gendered career trajectory
which collides with the biological clock of women wish-
ing to raise families [54], although much progress has
been made in some countries to protect against bias to-
wards those taking career breaks. Implicit bias affects
the advancement and promotion of women within the
research workforce [55], while recruitment materials
may be interpreted through a gender bias lens, with rec-
ommendation letters for men being read as stronger
than those for women with equal qualifications [56].
Likewise, women candidates for tenured research posi-
tions are sometimes put at a disadvantage due to ‘back-
door hiring’practices and mobilisations of informal,
potentially gendered, network ties, in academic recruit-
ment and selection [57, 58].
Nevertheless, there is an opportunity to move science
policy and practice towards greater gender equity, be-
cause scientific enterprise is founded on rational values.
In essence, scientific enterprise strives to produce object-
ive knowledge without bias, with a view to bringing soci-
etal and economic benefits to humanity. Greater gender
equity helps scientific enterprise achieve objective know-
ledge regardless of the ways objectivity is conceived of
by different epistemological communities. The trad-
itional view is that objective knowledge exists and can
be achieved by removing different biases, including the
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 3 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

gender bias. This view is questioned by feminist scholars
who contend that all knowledge is gender-biased in one
way or another and thus objectivity can be achieved by
increasing the pluralism of perspectives to balance biases
and by critically evaluating biases in both knowledge and
methods of enquiry [59–62].
Although the value of gender equity for increasing the
objectivity of science is acknowledged by different epis-
temological communities, the scope for centralised pol-
icy action to maximise research impact through the
gender equity pathway within scientific enterprise is lim-
ited. This is partly because many research funders and
institutions are decentralised (and cross multiple sectors
including private, public and third-sector), and partly be-
cause changes in policy may reduce but not eliminate
gender bias. Whilst national and local policies may be in
place, the reality of research enterprise is that decision-
making authority is devolved to the level of departments,
centres and, sometimes, individual principal investiga-
tors. Moreover, scientific enterprise has (encouragingly)
become globalised (there is increased mobility of re-
searchers, and many research funders and institutions
seek to become globally competitive in the knowledge
economy). It follows that gender equity in research re-
quires action at a global level to be maximally effective
(Fig. 1).
Research impact assessment
Research impact assessment is the multidisciplinary field
of scientific inquiry that examines the research process
to maximise scientific, societal and economic returns on
investment in research. Unlike the primary orientation
of basic science, which is concerned with the advance-
ment of knowledge for its own sake, research impact
assessment (a form of research on research) is predom-
inantly oriented towards applied research objectives of
influencing policy and practice. Research impact assess-
ment has developed its own methods of analysis, draw-
ing on other disciplines and fields of knowledge [63–68].
Its combination of applied research objectives and scien-
tific rigour should allow research impact assessment to
become a force for good in moving science policy and
practice towards gender equity by investigating gender
bias and recommending actions for change.
The driving forces behind research impact assessment
have been conceptualised into the Four “As”of Advo-
cacy, Accountability, Analysis and Allocation [69]. Given
the challenges of addressing gender bias in research,
each of the Four “As”of research impact assessment are
necessary to address gender bias in research (Fig. 2):
Advocacy is needed to ‘make the case’[69] for
science free from gender bias, by highlighting the
scientific, societal, and economic benefits of gender
equity in research.
Accountability to the public requires that funding
and staffing decisions by research funders and
research institutions are made fairly, and that an
account of such decision-making is captured and
reported transparently through appropriate metrics.
Analysis is necessary to challenge gender bias and
discover policies that eliminate it, as well as
conditions under which these policies could be
transferred to other organisations and countries.
Allocation of research funding to ensure equitable
participation of both genders in research is
imperative for the legitimisation of and public
support for science [38].
Various methodological approaches may be used
within research impact assessment to investigate and ad-
dress gender bias (Table 1). For example, logic modelling
and related theories of change [70] can be applied to sys-
tematically investigate gender equity at all stages of the
research process, as well as effective mechanisms for en-
suring the transparent and equitable distribution of re-
sources and power. Bibliometrics and other traditional
research output metrics (scientometrics) can be used to
Fig. 1 The gender equity pathway to maximise research impact. Shown are the forces that influence the key stakeholders in promoting gender
equity to maximise different possible types of research impact
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 4 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

measure gender-based differences in research outputs,
their academic impacts, including gender-based citation
behaviour, and non-academic impacts [23, 71–74]; how-
ever, unless supplemented by additional methodologies,
they will not illuminate the reasons for such differences.
Potentially, the more immediate academic and non-
academic impact of research can also be measured using
non-traditional web-based metrics (altmetrics) [75, 76].
Surveys and narrative case studies can be especially use-
ful in investigating these differences. Curriculum vitae
and survey data can be combined to measure variations
in researchers’societal outreach and orientation [77],
and text-mining algorithms can be employed to analyse
developments in the scholarly attention to gender and
sex issues in biomedical research [78]. Information and
management science approaches can be employed to de-
velop performance management systems (such as the re-
cently described ‘balanced scorecard’[79]) for assessing
and monitoring gender equity both in the awarding of
grants and selection of research topics. Finally, economic
modelling and cost-benefit analysis could be employed
to examine how gender equity can maximise economic
returns on investment in research.
Whichever methodological approaches are employed,
active engagement with users of impact assessment is
needed, in turn, to disseminate findings and influence
policy and practice. Communication of these findings is
likely to be strengthened by the use of infographics, dia-
grams, charts and other visual tools to help clearly con-
vey qualitative and quantitative data and trends. For
example, recent analysis of the United Kingdom’s Re-
search Excellence Framework 2014 impact case studies
has used innovative infographics, alluvial and chord dia-
grams, word clouds, heat maps and impact wheels, syn-
thesising complex data to reveal where research has had
a societal impact [80]. With increasing use of the web
and social media by researchers and research users, im-
pact assessment results can be rapidly communicated
through a variety of media, including research blogs, so-
cial networks and web feeds. Gender-sensitive visualisa-
tion and dissemination can further enhance engagement
with users of impact assessment.
Recommendations
For research impact assessment to become a force for
good in moving science policy and practice towards gen-
der equity, we propose that gender be routinely included
in research impact assessment. Based on our knowledge
Fig. 2 The Four “As”of research impact assessment with regard to
gender equity
Table 1 Selection of methods in research impact assessment and how they may be used to investigate and address gender bias
Methods Applications
Logic modelling describes and graphically represents the chains of
results associated with all stages of the research process; theories of
change are formulated on the basis of these chains of results to explain
how to achieve the desired results
To provide a framework to systematically investigate gender equity at all
stages of the research process, and investigate how gender equity can
help maximise desired research outputs and impacts
Scientometrics pertain to the statistical analysis of articles and citations in
academic journals (bibliometrics) and other research-based outputs, e.g.
patents, commercialisation
To measure gender-based differences in research outputs, their academic
impacts, including gender-biased citation behaviour, and non-academic
impacts
Altmetrics measure the online attention to academic articles on social
media, mainstream news websites, blogs and social bookmarking
websites
To measure gender-based differences in more immediate academic and
non-academic impact of research
Surveys, curriculum vitae data, narrative case studies and text-mining
algorithms collect quantitative and qualitative information that may
not already have been captured as part of the research process
To investigate gender-based differences in perceptions of and approaches
to the research process, outcomes and impacts
Information and management science approaches allow developing
balanced scorecards and other performance management systems
To assess and monitor gender equity in research organisations
Economic modelling categorises and cost-benefit analysis assesses
different types of benefits from research
To examine how gender equity can maximise economic returns on
investment in research
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 5 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

and experience of the field, we –a group of scholars and
practitioners representing research leaders from Africa,
America, Asia and Europe –offer recommendations to
research funders, research institutions and research eval-
uators (i.e. those who conduct research impact assess-
ment) on how to include and strengthen analysis of
gender equity in research impact assessment. Our rec-
ommendations are outlined below and summarised in
Table 2.
Research funders
Internationally, research funders have long conducted
research impact assessment in order to demonstrate ac-
countability to taxpayers and philanthropists as well as
to inform their resource allocation practices. In so doing,
many funders already have established systems for col-
lecting basic gender-based information on the research
they fund. Such information is often required to demon-
strate compliance with anti-discrimination legislation.
However, it is not routinely used for research impact as-
sessment. We recommend that research funders conduct
retrospective research impact assessment using gender-
based information already available. We suggest this
should inform the development and implementation of
gender equity policies, and lead to more gender-sensitive
prospective data collection. This would help identify re-
search programmes and practices that are most reflect-
ive of gender equity, areas which may need may need
further exploration or development, and policies to
support fairer application processes. For example, the
Australian Research Council has recently introduced a
‘gender equity action plan’, including policies to report
gender disaggregated data and monitor the gender out-
comes of selection rounds [81].
It has long been recognised that research results may
not equally apply to men and women if scientific evi-
dence fails to consider relevant sex and gender issues.
However, developing formal policies and approaches by
research funders to encourage researchers to include
considerations of sex and gender in their research de-
signs and analyses is a relatively recent phenomenon
[82]. For example, since 2009, the Canadian Institutes of
Health Research requires that all grant applicants con-
sider whether their research designs include sex and
gender when appropriate [83]. In 2014, the United States
National Institutes of Health announced new supple-
mental awards to explore the effects of sex and gender
in preclinical and clinical studies [84]. The European
Commission has integrated gender- and sex-based ana-
lysis into Horizon 2020 –its biggest research and
innovation programme for 2014–2020 [85]. Such pol-
icies encourage researchers to study both sexes to aid
scientific innovation and better health for all citizens.
We recommend that research funders adopt policies to
ensure that researchers address relevant sex and gender
issues in their research designs and analyses.
There are also research funders that now require re-
search institutions applying for research funding to dem-
onstrate how they support gender equality in research
careers. This is mainly achieved through various award
schemes designed to promote structural change in re-
search institutions [86]. For example, the United
Kingdom National Institute for Health Research made
designation and funding of its Biomedical Research
Centres conditional upon the achievement by academic
applicants of at least the Silver Award of the Athena
Project and the Scientific Women’s Academic Network
(SWAN) Charter for Women in Science, which recognises
work undertaken by research institutions to advance
women’s careers in science, technology, engineering,
mathematics and medicine (STEMM) [79]. Another ex-
ample of integrating gender equity into research funding
criteria is the report card developed by the New York
Stem Cell Foundation to assess institutions in grant appli-
cation processes [87]. We recommend that research
Table 2 Recommendations to include and strengthen analysis
of gender equity in research impact assessment
Stakeholders Recommendations
Research
funders
•Conduct retrospective and prospective observational
gender-based research impact assessment to inform
implementation of gender equity policies
•Adopt policies to ensure that researchers address
relevant sex and gender issues in their research
designs and analyses
•Identify and apply evidence-based approaches to
integrating gender equity into research funding
criteria
•Where appropriate, support theoretical and applied
research on the scientific, societal and economic
impact of gender equity in research
Research
institutions
•Establish information systems for collecting and
analysing gender-based information on the research
process
•Work towards improving and refining the quality of
gender-based data
•Train and support interested students and staff to
include gender in research impact assessment
•Establish gender equity activity as a criterion for
performance management and annual appraisal and
promotion
Research
evaluators
•Engage with the research impact assessment
literature with a view to identifying and applying
evidence-based tools and approaches to gender-
sensitive research impact assessment
•Identify, investigate and address gender differences in
research production and impact, including
commercialisation
•Increase the objectivity of research impact assessment
by conducting evaluations in gender-balanced teams
using an appropriate mix of methods
•Collaborate and share knowledge through gender-
balanced networks and communities of practice
•Engage impact assessment users and the public
in debate on gender bias and recommend
actions for change
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 6 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

funders identify and apply evidence-based approaches to
integrating gender equity into research funding criteria.
So far, relatively little scholarly attention has been
given to the scientific, societal and economic impact of
gender equity in research. Debate on gender equity in
research has been primarily driven by human rights and
equality imperatives and is based on evidence that is
sparse and of variable methodological quality [3]. Im-
portantly, experimental research shows gender differ-
ences in how the quality of evidence revealing gender
bias is evaluated by women and men [88]. There is a
need to systematise the current evidence to map more
precisely the state of knowledge, ignorance and uncer-
tainty in the field, and identify gaps in the evidence base
where new research is needed. Such a scoping exercise is
likely to highlight the need for further comparative ef-
fectiveness research to determine the most effective pol-
icy interventions, as well as the conditions under which
particular interventions can be effective in different
settings. Additionally, fields beyond medicine (such as
higher education studies and the sociology of science)
may provide important theoretical perspectives that will
help guide the design of empirical studies. We recom-
mend that, where appropriate, research funders support
theoretical and applied research on the scientific, societal
and economic impact of gender equity in research.
Research institutions
Increasingly, research institutions realise that “winning
the talent war for women”[89] has great potential to
contribute to their international competitiveness and fu-
ture growth. However, research institutions do not al-
ways have information systems that allow disaggregating
key research performance data by gender. Moreover,
progress towards gender equity and the enhanced ability
to compete for the best students, faculty and staff of
both genders are not routinely considered as key per-
formance indicators by universities themselves and as a
basis for competition with their peers globally, e.g. in
university rankings and league tables. We recommend
that universities and other research institutions establish
systems for collecting and analysing gender-based infor-
mation on the research process.
Gender equity in research institutions tends to vary by
level of seniority, department, and discipline. Women
tend to be better represented at entry and junior levels
and underrepresented at senior levels, especially in the
STEMM sciences and clinical trials. If gender analysis is
conducted on too coarse a unit of aggregation, different
biases can inadvertently cancel each other out (e.g.
women in nursing, men in surgery). We recommend
that research institutions work towards improving and
refining the quality of gender-based data, e.g. by ensur-
ing that it is possible to disaggregate gender-based
information vertically by the level of seniority and hori-
zontally by department and discipline.
Research institutions have a unique role to play in the
inclusion of gender in research impact assessment be-
cause they educate and employ today’s and tomorrow’s
research evaluators. Research institutions can create
awareness amongst their students and staff about the
importance of gender analysis and provide them with
the necessary knowledge and skills to analyse and report
gender-specific and gender-comparative results. For ex-
ample, they can include elements of gender analysis and
social science perspectives on gender biases in know-
ledge and methods of enquiry in the teaching curricula
of the relevant disciplines; organise multi-disciplinary
workshops and seminars to develop the competencies of
the students and staff interested in research impact as-
sessment; implement comprehensive diversity and inclu-
sion plans [90]; consider targets and quotas for women
in leadership [91]; actively develop expert knowledge
leadership in gender-sensitive research impact [92]; and
even evaluate and address the gender pay gap [93, 94].
We recommend that research institutions train and sup-
port interested students and staff to include gender in
research impact assessment.
Most research institutions across the globe now have
in place programmes and initiatives to support gender
equity, either as part of their own efforts or as a re-
sponse to the incentives provided by research funders.
For example, most research-intensive universities in the
United Kingdom participate in the Athena SWAN Char-
ter for Women in Science and its national and regional
equality networks [95]. This requires extensive data col-
lection and analysis, discussions in working groups and
committees, and preparation and implementation of ac-
tion plans. Female faculty bear much of the burden for
preparing Athena SWAN award applications [87], which
may even be to the detriment of their core scientific ac-
tivity and thus productivity and promotion. Therefore, it
is important to ensure that gender equity activity under-
taken by faculty is recognised within their own institu-
tions and scientific fields. We recommend that research
institutions include gender equity activity as a criterion
for performance management and annual appraisal and
promotion.
Research evaluators
The science of research impact assessment has already
developed a number of validated evidence-based ap-
proaches that can be used for gender analysis. Moreover,
there is an emerging evidence base of successful research
impact assessment studies focussing specifically on gen-
der [96, 97]. By using similar tools in new studies, re-
search evaluators can test their robustness and extend
the evidence base in ways that allow synthesis across
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 7 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

studies. Therefore, we recommend that research evalua-
tors engage with the research impact assessment literature
with a view to identifying and applying evidence-based
tools and approaches to gender-sensitive research impact
assessment. However, when new kinds of research ques-
tions are being asked about gender, it may be appropriate
to develop new tools and approaches.
Gender analysis can reveal important differences be-
tween men and women in research production and im-
pact that can inform policies to optimise performance of
the scientific enterprise. Whereas male scientists may be
more productive during the early stages of their careers,
the productivity rates of female scientists may equal or
surpass those of men later in their careers [98, 99].
Whereas male physician-scientists may be more inter-
ested in basic science with long-term impacts, female
physician-scientists may be more interested in clinical
work with more immediate patient benefits [100]. Like-
wise, there may be important gender differences in co-
authorship behaviour, interdisciplinary collaboration and
societal impact [99]. Whereas women leaders may be
more collaborative, more nurturing, and share power
and uncertainty more, male leaders may be more hier-
archical; and whereas male researchers may be geared
towards scientific rewards and recognition, female re-
searchers may have a greater societal orientation and
output [77]. However, on the basis of the evidence in re-
lation to the quality of women’s academic outputs and
citations, “there is no evidence that women do less im-
portant work than men”[52]. Thus, where gender differ-
ences exist, this not only requires further exploration,
but also highlights the need for the use of appropriate
metrics that do not lead to a systematic bias in assessing
the research production and impact of either men or
women. We recommend that research evaluators inves-
tigate gender differences in research production and im-
pact, especially in terms of productivity, collaboration,
impact time lags and types of impact, and carefully select
appropriate metrics to avoid bias.
It is important to ensure that the act of research im-
pact assessment itself does not become affected by gen-
der bias. For example, there may be a gender bias in the
selection of methods and models of inquiry. Whereas
quantitative methods and a positivist natural science
model of inquiry have been traditionally associated with
male researchers, qualitative methods and an interpreti-
vist social science model of inquiry have been tradition-
ally associated with female researchers [101]. Moreover,
there is a disagreement between different epistemo-
logical communities as to whether the objectivity of
knowledge can be achieved by removing gender bias or
by increasing the pluralism of perspectives to balance
different gender biases [60]. Regardless of the ways ob-
jectivity is conceived of by different epistemological
communities, increasing gender equity in research teams
and the pluralism of methods can increase the objectivity
of knowledge. We recommend that research evaluators in-
crease the objectivity of research impact assessment by
conducting evaluations in gender-balanced teams using an
appropriate mix of methods.
Research impact assessment is a rapidly growing field
of knowledge spanning disciplinary, organisational and
geographical boundaries. Research evaluators can stay
abreast of new knowledge and strategic developments in
the field through collaboration and knowledge sharing.
These can also help identify solutions to common prob-
lems and provide feedback without wasting time and re-
sources. Moreover, participation in boundary-spanning
networks and communities of practice can diffuse in-
novative ideas and unleash collective creativity to dis-
cover new approaches for the greater good. In doing so
it is important that networks and communities of prac-
tice strive to achieve gender balance among their mem-
bers and leaders, and routinely address gender issues.
We recommend that research evaluators collaborate and
share knowledge through gender-balanced networks and
communities of practice.
Research evaluators have a potentially transformative
role to play in the scientific enterprise because research
impact assessment is predominantly oriented towards
influencing policy and practice. As research evaluators
investigate research outputs and outcomes, they engage
with a wide range of stakeholders and communicate
their findings to decision-makers. This gives research
evaluators an opportunity not only to investigate gender
bias, but also to suggest a course of policy interventions
to promote gender equity. The knowledge of science
policy in various settings also enables research evaluators
to initiate public debate and advocate for science free
from gender bias. We recommend that research evalua-
tors engage impact assessment users and the public in
debate on gender bias and recommend actions for
change.
Conclusions
Growing global investment in biomedical research is un-
likely to result in outstanding science that benefits
women and men equitably if current levels of conscious
and unconscious gender bias in health research persist.
Gender bias is difficult to eliminate, in part because of
the historical late entry of women into higher education
and research, and a tendency of path dependency in sci-
ence. However, we argue that research impact assess-
ment can become a force for good in moving science
policy and practice towards gender equity by revealing
and challenging gender bias. Success in applying re-
search impact assessment to address gender bias will de-
pend on sustained action by multiple stakeholders at all
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 8 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

stages of the research process and internationally. We
call on research funders, research institutions and re-
search evaluators globally to include gender in research
impact assessment in order to maximise scientific, soci-
etal and economic returns on investment in research.
Abbreviations
Athena SWAN, Athena Project and the Scientific Women’s Academic Network;
STEMM, science, technology, engineering, mathematics and medicine
Acknowledgements
Early ideas for this paper were presented and discussed at the International
School on Research Impact Assessment, 8–12 November 2015, Doha,
organised by the Qatar National Research Fund, Alberta Innovates –Health
Solutions (AIHS), Agency for Health Quality and Assessment of Catalonia
(AQuAS), RAND Europe, and King’s College London. We would like to thank
Mrs. Safia Al Marhoobi and Mrs. Noora Al Balushi from The Research Council
(Oman) for their assistance with data collection, and Dr. Hasneen Karbalai
from The University of Oxford for his valuable comments and suggestions.
HLS is supported by European Commission & IGRUE FP7 award TRIGGER:
Transforming Institutions by Gendering contents and Gaining Equality in
Research (2014–2017) Grant Agreement n° 611034. LDE was supported by the
John Fell Fund and the Vice-Chancellor’s Diversity Fund, University of Oxford.
AMB, KMC and GML hold NIHR Senior Investigator Awards. The publication of
this article was supported by the NIHR Oxford Biomedical Research Centre.
Funding
NIHR Oxford Biomedical Research Centre, Oxford, United Kingdom.
Availability of data and materials
Not applicable.
Authors’contributions
PVO conceived of this call for action and led the drafting of the manuscript
with substantive input from TG. All other authors critically revised the
manuscript and provided important comments and suggestions. All of the
authors read and approved the final version of the manuscript.
Competing interests
TG represented Medicine on the Equality and Diversity Panel for the 2014
Research Excellence Framework (REF); in that role, she was involved in
assessing universities’procedures for assuring equality (by gender, ethnicity,
disability and sexual orientation) in their decisions on which individuals to
submit to the REF. RS is Advocate for Diversity and Pro Vice-Chancellor,
University of Oxford. RG has grant funding from the Doris Duke Charitable
Foundation, the National Institutes of Health, the Michigan Radiation Oncology
Quality Consortium, and the Greenwall Foundation for unrelated work. All other
authors declare that they have no competing interests.
Consent for publication
Not applicable.
Ethics approval and consent to participate
Not applicable.
Disclaimer
The views expressed are those of the authors and not necessarily of the
authors’respective organisations, funders or sponsors.
Author details
1
Medical Sciences Division, University of Oxford, John Radcliffe Hospital,
Oxford OX3 9DU, United Kingdom.
2
Nuffield Department of Primary Care
Health Sciences, University of Oxford, Radcliffe Primary Care Building,
Woodstock Road, Oxford OX2 6GG, United Kingdom.
3
Agency for Health
Quality and Assessment of Catalonia (AQuAS), Carrer de Roc Boronat, 81,
ES-08005 Barcelona, Spain.
4
The Policy Institute, King’s College London,
Strand Campus, London WC2R 2LS, United Kingdom.
5
Alberta Innovates –
Health Solutions, 10104-103 Avenue NW, Edmonton, AB T5J 4A7, Canada.
6
Fundación Huésped, Pasaje A. Peluffo 3932 (C1202ABB), Buenos Aires,
Argentina.
7
Qatar National Research Fund, P.O. Box 5825, Doha, Qatar.
8
Qatar
Foundation, P.O. Box 5825, Doha, Qatar.
9
Department of Studies and
Planning, The Research Council, P.O. Box 1422, Al Azaiba 130, Oman.
10
Commonwealth Scientific and Industrial Research Organisation, P.O. Box
883, Kenmore, Brisbane 4069, Australia.
11
Melbourne School of Government,
The University of Melbourne, Parkville, Victoria 3010, Australia.
12
School of
Population Health, University of Western Australia, Perth, WA 6009, Australia.
13
Knowledge Translation Australia Pty Ltd., Melbourne, Victoria, Australia.
14
Faculty of Business and Economics, University of Melbourne, 198 Berkeley
Street, Parkville, Victoria 3010, Australia.
15
Saïd Business School, University of
Oxford, Park End Street, Oxford OX1 1HR, United Kingdom.
16
The George
Institute for Global Health, University of Oxford, 34 Broad Street, Oxford OX1
3BD, United Kingdom.
17
The George Institute for Global Health, University of
Sydney, P.O. Box M201Missenden Road, Sydney NSW 2050, Australia.
18
RAND
Europe, Westbrook Centre, Milton Road, Cambridge CB4 1YG, United
Kingdom.
19
Novo Nordisk Foundation, Tuborg Havnevej 19, DK-2900
Hellerup, Denmark.
20
Lundbeck Foundation, Scherfigsvej 7, DK-2100
Copenhagen, Denmark.
21
Institute for Economics, Labour and Culture,
Goethe-University Frankfurt, Senckenberganlage 31, 60325 Frankfurt am
Main, Germany.
22
Medical Management Centre, Department of Learning,
Informatics, Management and Ethics (LIME), Karolinska Institutet,
Tomtebodavaegen 18a, 171 77 Stockholm, Sweden.
23
Horizon 2020 Advisory
Group for Gender, European Commission, Brussels, Belgium.
24
National
Initiative on Gender, Culture and Leadership in Medicine: C-Change, Brandeis
University Women’s Studies Research Center, 415 South Street, MS 079,
Waltham, MA 02454, United States of America.
25
Department of Radiation
Oncology, Center for Bioethics and Social Sciences in Medicine, University of
Michigan, Ann Arbor, MI 48109, United States of America.
26
Department of
Management, Birkbeck, University of London, Malet Street, London WC1E
7HX, United Kingdom.
27
International Triple Helix Institute, 1520 Sand Hill
Road, A210, Palo Alto, CA 94304, United States of America.
28
Gendered
Innovations, History Department, Stanford University, 450 Serra Mall, Stanford,
CA 94305, United States of America.
29
Health Sciences Department,
Universitat Oberta de Catalunya, Av. Tibidabo 39-43, ES-08035 Barcelona,
Spain.
30
PAPRSB Institute of Health Sciences, Universiti Brunei Darussalam,
Jalan Tungku Link, Gadong BE1410, Brunei Darussalam.
31
Faculty of
Integrated Technologies, Universiti Brunei Darussalam, Jalan Tungku Link,
Gadong BE1410, Brunei Darussalam.
32
QS Intelligence Unit, Quacquarelli
Symonds Ltd, 4 Heathgate, Agincourt Rd, London NW3 2NT, United
Kingdom.
33
Department of Public Health, School of Medicine, University of
Zambia, Nationalist Rd, Lusaka, Zambia.
34
Research Support Centre, College
of Medicine, University of Malawi, P.O. Box 360, Chichiri, Blantyre 3, Malawi.
35
Assembly of Women for Development and the Struggle against Social
Exclusion (ASDO), via Guido Reni 56, 00196 Rome, Italy.
36
NIHR Oxford
Biomedical Research Centre, Joint Research Office, Churchill Hospital, Oxford
OX3 7LE, United Kingdom.
37
Oxford University Hospitals NHS Foundation
Trust, John Radcliffe Hospital, Oxford OX3 9DU, United Kingdom.
38
Department of Physiology, University of Toronto, 1 King’s College Circle,
Toronto, Ontario M5S 1A8, Canada.
39
Women’s Health Academic Centre,
King’s College London, Guy’s Hospital, London SE1 1UL, United Kingdom.
40
NIHR Biomedical Research Centre at Guy’s and St Thomas’NHS Foundation
Trust and King’s College London, Guy’s Hospital, London SE1 9RT, United
Kingdom.
41
Guy’s and St Thomas’NHS Foundation Trust, Guy’s Hospital,
London SE1 9RT, United Kingdom.
42
MRC Centre for Transplantation, King’s
College London, Guys’Hospital, London SE1 9RT, United Kingdom.
43
Department of Social Policy and Intervention, University of Oxford, Barnett
House, 32-37 Wellington Square, Oxford OX1 2ER, United Kingdom.
44
Institute of Social and Economic Research, Rhodes University, P.O. Box 94,
Grahamstown 6140, South Africa.
Received: 5 April 2016 Accepted: 24 June 2016
References
1. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM,
et al. How to increase value and reduce waste when research priorities are
set. Lancet. 2014;383(9912):156–65.
2. Klinge I. Bringing gender expertise to biomedical and health-related
research. Gend Med. 2007;4 Suppl 2:S59–63.
3. Edmunds LD, Ovseiko PV, Shepperd S, Greenhalgh T, Frith P, Roberts NW, et al.
Why do women choose or reject careers in academic medicine? A narrative
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 9 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

review of empirical evidence. Lancet. 2016;pii: S0140-6736(15)01091-0.
doi:10.1016/S0140-6736(15)01091-0. Ahead of print.
4. Pololi LH, Jones SJ. Women faculty: an analysis of their experiences in
academic medicine and their coping strategies. Gend Med.
2010;7(5):438–50.
5. Pololi LH, Civian JT, Brennan RT, Dottolo AL, Krupat E. Experiencing the
culture of academic medicine: gender matters, a national study. J Gen
Intern Med. 2013;28(2):201–7.
6. Beery AK, Zucker I. Sex bias in neuroscience and biomedical research.
Neurosci Biobehav Rev. 2011;35(3):565–72.
7. Curno MJ, Rossi S, Hodges-Mameletzis I, Johnston R, Price MA, Heidari S. A
systematic review of the inclusion (or exclusion) of women in HIV research:
from clinical studies of antiretrovirals and vaccines to cure strategies. J
Acquir Immune Defic Syndr. 2016;71(2):181–8.
8. Jagsi R, Motomura AR, Amarnath S, Jankovic A, Sheets N, Ubel PA. Under-
representation of women in high-impact published clinical cancer research.
Cancer. 2009;115(14):3293–301.
9. Grant J, Burden S, Breen G. No evidence of sexism in peer review. Nature.
1997;390(6659):438.
10. Head MG, Fitchett JR, Cooke MK, Wurie FB, Atun R. Differences in research
funding for women scientists: a systematic comparison of UK investments
in global infectious disease research during 1997-2010. BMJ Open.
2013;3(12):e003362.
11. Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature.
1997;387(6631):341–3.
12. Bornmann L, Mutz R, Daniel HD. Gender differences in grant peer review: a
meta-analysis. J Informetr. 2007;1(3):226–38.
13. Marsh HW, Bornmann L, Mutz R, Daniel H-D, O’Mara A. Gender effects in the
peer reviews of grant proposals: a comprehensive meta-analysis comparing
traditional and multilevel approaches. Rev Educ Res. 2009;79(3):1290–326.
14. Turner S, Davidson P, Stanton L, Cawdeary V. Features of successful bids for
funding of applied health research: a cohort study. Health Res Policy Syst.
2014;12:54.
15. Sege R, Nykiel-Bub L, Selk S. Sex differences in institutional support for
junior biomedical researchers. JAMA. 2015;314(11):1175–7.
16. Jagsi R, Motomura AR, Griffith KA, Rangarajan S, Ubel PA. Sex differences in
attainment of independent funding by career development awardees. Ann
Intern Med. 2009;151(11):804–11.
17. Ley TJ, Hamilton BH. Sociology. The gender gap in NIH grant applications.
Science. 2008;322(5907):1472–4.
18. van der Lee R, Ellemers N. Gender contributes to personal research funding
success in The Netherlands. Proc Natl Acad Sci U S A. 2015;112(40):12349–53.
19. Vydareny KH, Waldrop SM, Jackson VP, Manaster BJ, Nazarian GK, Reich CA,
et al. Career advancement of men and women in academic radiology: is
the playing field level? Acad Radiol. 2000;7(7):493–501.
20. Noblelprize.org. All Nobel Prizes in Physiology or Medicine. http://www.
nobelprize.org/nobel_prizes/medicine/laureates/. Accessed 1 April 2016.
21. Bhattacharyya N, Shapiro NL. Increased female authorship in otolaryngology
over the past three decades. Laryngoscope. 2000;110(3 Pt 1):358–61.
22. Amering M, Schrank B, Sibitz I. The gender gap in high-impact psychiatry
journals. Acad Med. 2011;86(8):946–52.
23. Jagsi R, Guancial EA, Worobey CC, Henault LE, Chang Y, Starr R, et al. The
“gender gap”in authorship of academic medical literature–a 35-year
perspective. N Engl J Med. 2006;355(3):281–7.
24. Sidhu R, Rajashekhar P, Lavin VL, Parry J, Attwood J, Holdcroft A, et al. The
gender imbalance in academic medicine: a study of female authorship in
the United Kingdom. J R Soc Med. 2009;102(8):337–42.
25. Piper CL, Scheel JR, Lee CI, Forman HP. Gender trends in radiology
authorship: a 35-year analysis. AJR Am J Roentgenol. 2016;206(1):3–7.
26. Filardo G, da Graca B, Sass DM, Pollock BD, Smith EB, Martinez MA. Trends
and comparison of female first authorship in high impact medical journals:
observational study (1994-2014). BMJ. 2016;352:i847.
27. Sen G, Östlin P, George A. Unequal, unfair, ineffective and inefficient gender
inequity in health: why it exists and how we can change it. Final Report to
the WHO Commission on Social Determinants of Health. http://www.who.
int/social_determinants/resources/csdh_media/wgekn_final_report_07.pdf.
Accessed 1 April 2016.
28. Kuhlmann E, Annandale E. Gender and healthcare policy. In: Kuhlmann E,
Blank RH, Bourgeault IL, Wendt C, editors. The Palgrave international
handbook of healthcare policy and governance. Basingstoke: Palgrave
Macmillan; 2015. p. 578–96.
29. Schiebinger L, Klinge I, Sánchez de Madariaga I, Paik HY, Schraudner M,
Stefanick M. Gendered innovations in science, health & medicine,
engineering, and environment. http://genderedinnovations.stanford.edu/.
Accessed 1 April 2016.
30. Bartlett C, Doyal L, Ebrahim S, Davey P, Bachmann M, Egger M, et al. The
causes and effects of socio-demographic exclusions from clinical trials.
Health Technol Assess. 2005;9(38):iii–iv. ix-x, 1–152.
31. Johnson PA, Fitzgerald T, Salganicoff A, Wood SF, Goldstein JM. Sex-specific
medical research: why women’s health can’t wait. A Report of the Mary
Horrigan Connors Center for Women’s Health & Gender Biology at Brigham
and Women’s Hospital. http://www.brighamandwomens.org/Departments_
and_Services/womenshealth/ConnorsCenter/Policy/ConnorsReportFINAL.
pdf. Accessed 1 April 2016.
32. Healy B. The Yentl syndrome. N Engl J Med. 1991;325(4):274–6.
33. Buitendijk S, Maes K. Gendered research and innovation: integrating sex and
gender analysis into the research process. http://www.leru.org/files/
publications/LERU_AP18_Gendered_research_and_innovation_final.pdf.
Accessed 1 April 2016.
34. Etzkowitz H, Kemelgor C, Uzzi B. Athena unbound: the advancement of
women in science and technology. Cambridge: Cambridge University Press;
2000.
35. Ley TJ, Rosenberg LE. The physician-scientist career pipeline in 2005: build
it, and they will come. JAMA. 2005;294(11):1343–51.
36. Puljak L, Kojundzic SL, Sapunar D. Gender and academic medicine: a good
pipeline of women graduates is not advancing. Teach Learn Med.
2008;20(3):273–8.
37. Sexton KW, Hocking KM, Wise E, Osgood MJ, Cheung-Flynn J, Komalavilas P,
et al. Women in academic surgery: the pipeline is busted. J Surg Educ.
2012;69(1):84–90.
38. Etzkowitz H, Kemelgor C, Neuschatz M, Uzzi B, Alonzo J. The paradox of
critical mass for women in science. Science. 1994;266(5182):51–4.
39. Commission of the European Communities. Women in science: the gender
dimension as a leverage for reforming science. Brussels: Commission of the
European Communities; 2001.
40. Wilsford D. Path dependency, or why history makes it difficult but not
impossible to reform health care systems in a big way. J Public Policy.
1994;14(3):251–83.
41. Alexander AC, Welzel C. Empowering women: four theories tested on four
different aspects of gender equality. Chicago: Annual Meeting of Midwest
Political Science Association; 2007. http://www.democracy.uci.edu/files/
docs/conferences/grad/alexander.pdf.
42. Schiebinger L, editor. Women and gender in science and technology. London:
Routledge; 2014.
43. Rossiter MW. The Matthew Matilda effect in science. Soc Stud Sci.
1993;23(2):325–41.
44. Holdcroft A. Gender bias in research: how does it affect evidence based
medicine? J R Soc Med. 2007;100(1):2–3.
45. Lagro-Janssen T. Sex, gender and health: developments in medical research.
In: Kuhlmann E, Annandale E, editors. The Palgrave handbook of gender
and healthcare. Basingstoke: Palgrave Macmillan; 2012. p. 439–54.
46. Barrett-Connor E. Gender differences and disparities in all-cause and
coronary heart disease mortality: epidemiological aspects. Best Pract Res
Clin Endocrinol Metab. 2013;27(4):481–500.
47. Gahagan J, Gray K, Whynacht A. Sex and gender matter in health research:
addressing health inequities in health research reporting. Int J Equity Health.
2015;14:12.
48. Eagly AH, Johannesen-Schmidt MC, van Engen ML. Transformational,
transactional, and laissez-faire leadership styles: a meta-analysis comparing
women and men. Psychol Bull. 2003;129(4):569–91.
49. Smith-Doerr L. Women’s work: gender equality vs. hierarchy in the life
sciences. Boulder: Lynne Rienner; 2004.
50. Marchant A, Bhattacharya A, Carnes M. Can the language of tenure criteria
influence women’s academic advancement? J Womens Health (Larchmt).
2007;16(7):998–1003.
51. Van den Brink M. Scouting for talent: appointment practices of women
professors in academic medicine. Soc Sci Med. 2011;72(12):2033–40.
52. de Melo-Martin I. Patenting and the gender gap: should women be
encouraged to patent more? Sci Eng Ethics. 2013;19(2):491–504.
53. Lawton Smith H, Etzkowitz H, Meschitti V, Poulovassilis A. Female academic
entrepreneurship and commercialisation: reviewing the evidence and
identifying the challenges. In: Henry C, Nelson T, Lewis K, editors. The
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 10 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Routledge companion to global female entrepreneurship. London:
Routledge; 2017.
54. Fox G, Schwartz A, Hart KM. Work-family balance and academic
advancement in medical schools. Acad Psychiatry. 2006;30(3):227–34.
55. Committee on Maximizing the Potential of Women in Academic Science
and Engineering (U.S.). Beyond bias and barriers: fulfilling the potential of
women in academic science and engineering. Washington, D.C: National
Academies Press; 2007.
56. Schmader T, Whitehead J, Wysocki VH. A linguistic comparison of letters of
recommendation for male and female chemistry and biochemistry job
applicants. Sex Roles. 2007;57(7-8):509–14.
57. Nielsen MW. Limits to meritocracy? Gender in academic recruitment and
promotion processes. Sci Publ Policy. 2015:10.1093/scipol/scv1052.
58. Husu L. Gender discrimination in the promised land of gender equality.
High Educ Eur. 2000;25(2):221–8.
59. Harding SG. The science question in feminism. Ithaca: Cornell University
Press; 1986.
60. Longino HE. Science as social knowledge: values and objectivity in scientific
inquiry. Princeton: Princeton University Press; 1990.
61. Harding SG. Whose science? Whose knowledge? Thinking from women’s
lives. Ithaca: Cornell University Press; 1991.
62. Nelson LH, Nelson J, editors. Feminism, science and the philosophy of science.
Dordrecht: Kluwer Academic; 1996.
63. Kostoff RN. Research impact assessment. Principles and applications to
proposed, ongoing, and completed projects. Invest Radiol.
1994;29(9):864–9.
64. Fealing KH, Lane JI, Marburger JHI, Shipp SS, editors. The science of science
policy: a handbook. Stanford: Stanford University Press; 2011.
65. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual
frameworks and empirical approaches used to assess the impact of
health research: an overview of reviews. Health Res Policy Syst.
2011;9:26. doi:10.1186/1478-4505-9-26.
66. Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic
clinical medicine: a study using Research Excellence Framework pilot impact
indicators. BMC Health Serv Res. 2012;12:478.
67. Guthrie S, Wamae W, Diepeveen S, Wooding S, Grant J. Measuring research:
a guide to research evaluation frameworks and tools. http://www.rand.org/
pubs/monographs/MG1217.html. Accessed 1 April 2016.
68. Milat AJ, Bauman AE, Redman S. A narrative review of research impact
assessment models and methods. Health Res Policy Syst. 2015;13:18.
69. Morgan Jones M, Grant J. Making the grade: methodologies for assessing
and evidencing research impact. In: Dean A, Wykes M, Stevens H, editors. 7
essays on impact. DESCRIBE project report for JISC. Exeter: University of
Exeter; 2013. p. 25–43.
70. Coryn CLS, Noakes LA, Westine CD, Schröter DC. A systematic review of
theory-driven evaluation practice from 1990 to 2009. Am J Eval.
2011;32(2):199–226.
71. Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW. Sex differences in
application, success, and funding rates for NIH extramural programs. Acad
Med. 2011;86(6):759–67.
72. Eloy JA, Svider PF, Kovalerchik O, Baredes S, Kalyoussef E, Chandrasekhar SS.
Gender differences in successful NIH grant funding in otolaryngology.
Otolaryngol Head Neck Surg. 2013;149(1):77–83.
73. Lariviere V, Ni C, Gingras Y, Cronin B, Sugimoto CR. Bibliometrics: global
gender disparities in science. Nature. 2013;504(7479):211–3.
74. McMillan GS. Gender differences in patenting activity: An examination of
the US biotechnology industry. Scientometrics. 2009;80(3):683–91.
75. Altmetric LLP. About altmetrics: an introduction to altmetrics and how
they can be used. https://www.altmetric.com/about-altmetrics/.
Accessed 1 April 2016.
76. Bar-Ilan J, van der Weiden I. Altmetric gender bias? An exploratory study. Int
J Comput Sci. 2015;4(1):16–22.
77. Van der Weijden ICM, Zahedi Z, Must Ü, Meijer I. Gender differences
in societal orientation and output of individual scientists. In:
Proceedings of the 19th International Conference on Science and
Technology Indicators; Leiden: Universiteit Leiden; 3–5 September
2014. p. 680–6.
78. Oertelt-Prigione S, Parol R, Krohn S, Preissner R, Regitz-Zagrosek V.
Analysis of sex and gender-specific research reveals a common increase
in publications and marked differences between disciplines. BMC Med.
2010;8:70.
79. Ovseiko PV, Edmunds LD, Pololi LH, Greenhalgh T, Kiparoglou V, Henderson
LR, et al. Markers of achievement for assessing and monitoring gender
equity in translational research organisations: a rationale and study protocol.
BMJ Open. 2016;6(1):e009022.
80. King’s College London, Digital Science. The nature, scale and beneficiaries
of research impact: an initial analysis of Research Excellence Framework
(REF) 2014 impact case studies. http://www.hefce.ac.uk/media/HEFCE,2014/
Content/Pubs/Independentresearch/2015/Analysis,of,REF,impact/Analysis_
of_REF_impact.pdf. Accessed 1 April 2016.
81. Australian Research Council. Gender equality in research: ARC gender
equality action plan 2015–16. http://www.arc.gov.au/sites/default/files/
filedepot/Public/Policy%20&%20Strategy/Gender%20Equality/ARC_Gender_
Equality_Action%20Plan.pdf. Accessed 1 April 2016.
82. Johnson J, Sharman Z, Vissandjee B, Stewart DE. Does a change in health
research funding policy related to the integration of sex and gender have
an impact? PLoS One. 2014;9(6):e99900.
83. Canadian Institutes of Health Research. Sex, gender and health research
guide: a tool for CIHR applicants. http://www.cihr-irsc.gc.ca/e/32019.html.
Accessed 1 April 2016.
84. National Institutes of Health. New supplemental awards apply sex and
gender lens to NIH-funded research. http://www.nih.gov/news-events/
news-releases/new-supplemental-awards-apply-sex-gender-lens-nih-funded-
research. Accessed 1 April 2016.
85. European Commission. Gender equality in Horizon 2020. Version 1.0.
http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/hi/
gender/h2020-hi-guide-gender_en.pdf. Accessed 1 April 2016
86. Equality Challenge Unit. GENDER-NET analysis report: award schemes,
gender equality and structural change. London: Equality Challenge
Unit; 2015.
87. Smith KA, Arlotta P. Watt FM; Initiative on Women in Science and
Engineering Working Group, Solomon SL. Seven actionable strategies for
advancing women in science, engineering, and medicine. Cell Stem Cell.
2015;16(3):221–4.
88. Handley IM, Brown ER, Moss-Racusin CA, Smith JL. Quality of evidence
revealing subtle gender biases in science is in the eye of the beholder. Proc
Natl Acad Sci U S A. 2015;112(43):13201–6.
89. McCracken DM. Winning the talent war for women. Sometimes it takes a
revolution. Harv Bus Rev. 2000;78(6):159–60. 162, 164-157.
90. Davidson S. CSIRO diversity and inclusion plan 2012–15. Brisbane:
CSIRO; 2013.
91. Whelan J, Wood R. Targets and quotas for women in leadership: a
global review of policy, practice, and psychological research. https://cel.
edu.au/our-research/targets-and-quotas-for-women-in-leadership.
Accessed 1 April 2016.
92. Fischer MD, Dopson S, Fitzgerald L, Bennett C, Ferlie E, Ledger J, et al.
Knowledge leadership: mobilizing management research by becoming the
knowledge object. Hum Relat. 2016;69(7):1563–85. doi:10.1177/
0018726715619686.
93. Jagsi R, Griffith KA, Stewart A, Sambuco D, DeCastro R, Ubel PA. Gender
differences in the salaries of physician researchers. JAMA.
2012;307(22):2410–7.
94. Wright AL, Ryan K, St Germain P, Schwindt L, Sager R, Reed KL.
Compensation in academic medicine: progress toward gender equity. J Gen
Intern Med. 2007;22(10):1398–402.
95. Equality Challenge Unit. Athena SWAN members. http://www.ecu.ac.
uk/equality-charters/athena-swan/athena-swan-members/.
Accessed 1 April 2016.
96. European Commission. Gender impact assessment of the specific
programmes of the Fifth Framework Programme –an overview. ftp://ftp.
cordis.europa.eu/pub/science-society/docs/women_gender_impact_fp5_en.
pdf. Accessed 1 April 2016.
97. European Commission. Monitoring progress towards gender equality in the
6th Framework Programme –synthesis report. https://ec.europa.eu/
research/science-society/document_library/pdf_06/gender-monitoring-
studies-synthesis-report_en.pdf. Accessed 1 April 2016.
98. Eloy JA, Svider P, Chandrasekhar SS, Husain Q, Mauro KM, Setzen M, et al.
Gender disparities in scholarly productivity within academic otolaryngology
departments. Otolaryngol Head Neck Surg. 2013;148(2):215–22.
99. Elsevier Analytical Services. Mapping gender in the German research arena.
https://www.elsevier.com/research-intelligence/resource-library/gender-2015.
Accessed 1 April 2016.
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 11 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

100. Cydulka RK, D’Onofrio G, Schneider S, Emerman CL, Sullivan LM.
Women in academic emergency medicine. Acad Emerg Med.
2000;7(9):999–1007.
101. Oakley A. Gender, methodology and people’s ways of knowing: some
problems with feminism and the paradigm debate in social science.
Sociology. 1998;32(4):707–31.
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research
Submit your manuscript at
www.biomedcentral.com/submit
Submit your next manuscript to BioMed Central
and we will help you at every step:
Ovseiko et al. Health Research Policy and Systems (2016) 14:50 Page 12 of 12
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

1.
2.
3.
4.
5.
6.
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:
use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
use bots or other automated methods to access the content or redirect messages
override any security feature or exclusionary protocol; or
share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at
onlineservice@springernature.com