Available via license: CC BY
Content may be subject to copyright.
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
HANDLE: http://hdl.handle.net/10216/110854
ISSN 2183-0606
http://www.open-jim.org
http://creativecommons.org/licenses/by/3.0 18
Success Rates in Horizon 2020
Dr. Seán McCarthy
Hyperion Ltd, Watergrasshill, Co-Cork, Ireland
sean.mccarthy@hyperion.ie
Letter from Industry
Horizon 2020 is the name of the European Union’s research and innovation
programme. In the first three years of the programme (2014-2016) over 100,000
proposals were received. 10,456 proposals were selected for funding. This
success rate (11%) sounds very low and can discourage researchers from
applying to the programme. This article argues that this success rate is not a
reliable indicator for researchers. This article will show that a more realistic
success rate is 29%. In one sub-programme (Energy) a success rate of 47% was
measured. The message from this paper is that the chances of success in Horizon
2020 depends on two main criteria: a) having an excellent scientific idea; and b)
having a thorough understanding of the evaluation process.
Keywords
. Horizon 2020, Success Rate, Evaluation Process.
1 Introduction
Horizon 2020 has a budget of €77 billion and this will be allocated to successful
proposals approved in the period 2014 to 2020. The programme is divided into many
sub-programmes. The most famous of these is the European Research Council (ERC)
that supports fundamental research. Another well known programme is the Marie Curie
Actions that supports PhD and Post Doctoral training. Details of all the different
programmes can be found on the Horizon 2020 official website. (
http://ec.europa.eu/programmes/horizon2020/h2020-sections ). Other sub-programmes
include research into health, food, energy, transport, security, and social sciences and
humanities.
Every year ‘calls for proposals’ are published for each of these individual sub-
programmes. Researchers submit proposals (individually and as part of consortia). The
proposals are evaluated by independent evaluators. Successful proposers are invited to
prepare legal agreement with the European Commission.
In 2017 a ‘mid-term evaluation’ of Horizon 2020 was undertaken to report on the
progress of Horizon 2020 in the first three years (2014-2016). Arguments presented in
this paper are also based on data used from this official report.
Every year the European Commission published a report on the progress of the
programme entitled the ‘Annual Monitoring Report’. Data used in this paper is from
the most recent Annual Monitoring Report (2015).
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 19
2 The Evaluation Process in Horizon 2020
Before discussing success rates it is important to understand the evaluation processes
used in Horizon 2020. The evaluation process varies between the different
programmes. In this paper a general evaluation process is described. When a proposal
arrives in Brussels it is first checked by a Commission official against ‘eligibility’
criteria. ‘Eligible’ means that the forms were filled in properly. A proposal based on a
very weak scientific idea can be classified as ‘eligible’ if the forms are filled in properly
and the basic rules are met.
When the European Commission quotes ‘success rates’ they are based on these ‘eligible
proposals’. Clearly this is not a good denominator.
The ‘eligible’ proposals are then sent to independent scientific evaluators. The
evaluators read the proposals individually. In some progammes the evaluators meet
(Consensus meeting) and in other cases they simply send their evaluation scores to the
relevant administrative body in Brussels. A final score is agreed based on the evaluators
individual scores. A ‘threshold’ is set to define ‘low quality proposals’. For example,
in some programmes the maximum score is 15 and a threshold is set at 10. Any proposal
scoring below 10 is considered low quality and automatically elimated. Scoring below
10 is a very low score. These proposals should be classified as ‘low quality proposals’.
If research organisations had effective quality control procedures, these proposals
would never have been submitted.
In the case of the ERC Programme the maximum score is 8 and a threshold of 4 is set.
This threshold is sub-divided i.e. a threshold of 2/4 for the researcher and a threshold
of 2/4 for the idea.
Following this evaluation a list of ‘high quality proposals’ is compiled. The final
selection of successful proposals is based on these high quality proposals. This is a far
better denominator to use when calculating the success rates
3 Success Rates in Horizon 2020 (2014-2016)
Total number of proposals received (2014-2016) = >104,000
Total number of ‘eligible proposals’ = 102,076
(This means that over 2000 proposals did not fill in the forms properly)
Proposals below threshold (low quality proposals) = 56,444
(55.3% of eligible proposals)
High Quality Proposals (above threshold) = 45,632
Proposals selected for funding = 11,108
Reported Success rate (funded/eligible) = 10.88% (11,108/102,076)
Real Success rate(funded/high quality proposals) = 24.34% (11,108/45,632)
This overall success rate of 24.34% is a far more encouraging number for researchers
thinking about submitting proposals.
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 20
ERC Starting Grants (2014-2016)
Source: European Commission (EC) Mid-Term Rev iew of Horizon 2020 (Table 8 and 9)
The ERC Starting grant is one of the most prestigious grants for the career development
of young researchers. Proposals are submitted by individual researchers.
The maximum score that reviewers can award ERC proposals is 8 (4 for the researcher,
4 for the research idea). Proposals below the threshold (with a score < 4 ) in ERC are
given a Grade C. Any researcher receiving a grade C is not allowed to resubmit a
proposal to ERC for two years.
Total number of ERC Starting grant proposals received (2014-20160)
= 8947
Proposals below the threshold (Grade C) = 6120
(68.4% of eligible proposals)
High Quality Proposals (above threshold) = 2827
Proposals funded
= 950
Reported Success rate (funded/eligible)
= 10.6% (
950
/8947)
Real Success rate(funded/high quality proposals) = 33.6 % (950/2827)
This success rate (33.6%) is very encouraging for young researchers planning a career
in science.
4 Success Rates Horizon 2020 (2015)
The Annual Monitoring Report (2015) provides far more detailed data on individual
programmes. It is the most recent Annual Monitoring Report available at the time of
writing this article.
Total number of proposals received in 2015 = 42,535
Proposals below threshold (low quality proposals)
= 22,511 (53% of eligible proposals)
High Quality Proposals (above threshold) = 20,024
Proposals selected for funding = 4,565
Reported Success rate (funded/eligible) = 10.7% (4565/42535)
Real Success rate(funded/high quality proposals) = 22.8% (4565/20024)
ERC Statistics for 2015
Total number of ERC proposals received in 2015 = 10,019
Proposals below the threshold (Grade C) = 6083 (61% of eligible proposals!!)
High Quality Proposals (above threshold) = 3936
Proposals funded = 1327
Reported Success rate (funded/eligible) = 13.3% (1327/10019)
Real Success rate(funded/high quality proposals) = 33.7% (1327/3936)
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 21
SME Instrument (2015) (Page 122)
An SME Instrument is a special grant for Small and Medium Sized Enterprises (SME).
Companies submit proposals individually (no mandatory partners required).
Total number of proposals received in 2015 = 11008
Proposals below the threshold = 8378 (76% of eligible proposals)
(Here the threshold is < 13/15 for Phase I grants)
High Quality Proposals (above threshold) = 2630
Proposals funded = 714
Reported Success rate (funded/eligible) = 6.5% (714/11008)
Real Success rate (funded/high quality proposals) = 28.2% (714/2630)
For companies considering a proposal for an SME instrument a thorough understanding
of the evaluation process and the evaluation criteria is essential.
Table 1.
Summary of Real Success Rates in the individual programmes in 2015 (SME Instrument
Proposals are excluded from the different programmes)
Programme
Number of
Proposals
Below
Threshold
High Quality
Proposals
Funded
Proposals
Real Success
Rates
Health (page 132)
1212
75%
318
94
29.5%
Energy (page 145)
839
71%
243
114
47%
Food ( page 138)
358
36%
228
62
27%
Transport (page 153)
702
33%
467
167
35%
Climate (page 160)
648
48%
335
79
23.5%
Security (page 172)
463
45%
252
39
15.4%
5 Conclusions
5.1 Message for the European Commission
Why does the European Commission continue to quote success rates based on ‘eligible’
proposals? This number is meaningless. Success rates should be calculated as a
percentage of ‘High Quality Proposals’ – those that score above the threshold. This is
how 2015 results should be reported:
“In 2015 a total of 42,535 proposals were submitted to Horizon 2020. After evaluation
by independent experts, 22,511 (53%) were classified as ‘low quality proposals’. From
the remaining ‘High Quality Proposals’ (20,024) a total of 4,565 proposals were
accepted for funding. This represents a 22.8% success rate.”
5.2 Message for Research Organisations
Research Organisations must take a share of the responsibility for the large numbers of
low quality proposals submitted. Quality control procedures such as screening and
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 22
proposal clinics, should identify weak proposals – before any considerable effort can
be wasted on their preparation. This should be a core activity of Research Support
Offices.
5.3 Messages for Researchers
Researchers must bear in mind the words of the Greek statesman and orator Pericles
(450BC) “Having knowledge but lacking the power to express it clearly is no better
than never having any ideas at all.”
In a lecture you tell or express the idea. In a competitive proposal you have to sell the
idea to the evaluators.
It is essential to understand how different types of evaluators think and how decisions
are made in the evaluation process. There are two ways to learn this – attend training
courses or (better) become an evaluator.
6 References
Official Website of Horizon 2020. Access 10th January2018:
http://ec.europa.eu/programmes/horizon2020/
Projects funded in Horizon 2020. Accessed 10th January2018:
http://cordis.europa.eu/projects/home_en.html
Mid-Term Review of Horizon 2020. Accessed 10th January2018:
https://ec.europa.eu/research/evaluations/pdf/book_interim_evaluation_horizon_
2020.pdf
Annual Monitoring Report of Horizon 2020 (2015) Accessed 10th January2018:
https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-
monitoring-report-2015
