ArticlePDF Available

Abstract

Horizon 2020 is the name of the European Union’s research and innovation programme. In the first three years of the programme (2014-2016) over 100,000 proposals were received. 10,456 proposals were selected for funding. This success rate (11%) sounds very low and can discourage researchers from applying to the programme. This article argues that this success rate is not a reliable indicator for researchers. This article will show that a more realistic success rate is 29%. In one sub-programme (Energy) a success rate of 47% was measured. The message from this paper is that the chances of success in Horizon 2020 depends on two main criteria: a) having an excellent scientific idea; and b) having a thorough understanding of the evaluation process.
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
HANDLE: http://hdl.handle.net/10216/110854
ISSN 2183-0606
http://www.open-jim.org
http://creativecommons.org/licenses/by/3.0 18
Success Rates in Horizon 2020
Dr. Seán McCarthy
Hyperion Ltd, Watergrasshill, Co-Cork, Ireland
sean.mccarthy@hyperion.ie
Letter from Industry
Horizon 2020 is the name of the European Union’s research and innovation
programme. In the first three years of the programme (2014-2016) over 100,000
proposals were received. 10,456 proposals were selected for funding. This
success rate (11%) sounds very low and can discourage researchers from
applying to the programme. This article argues that this success rate is not a
reliable indicator for researchers. This article will show that a more realistic
success rate is 29%. In one sub-programme (Energy) a success rate of 47% was
measured. The message from this paper is that the chances of success in Horizon
2020 depends on two main criteria: a) having an excellent scientific idea; and b)
having a thorough understanding of the evaluation process.
Keywords
. Horizon 2020, Success Rate, Evaluation Process.
1 Introduction
Horizon 2020 has a budget of 77 billion and this will be allocated to successful
proposals approved in the period 2014 to 2020. The programme is divided into many
sub-programmes. The most famous of these is the European Research Council (ERC)
that supports fundamental research. Another well known programme is the Marie Curie
Actions that supports PhD and Post Doctoral training. Details of all the different
programmes can be found on the Horizon 2020 official website. (
http://ec.europa.eu/programmes/horizon2020/h2020-sections ). Other sub-programmes
include research into health, food, energy, transport, security, and social sciences and
humanities.
Every year ‘calls for proposals’ are published for each of these individual sub-
programmes. Researchers submit proposals (individually and as part of consortia). The
proposals are evaluated by independent evaluators. Successful proposers are invited to
prepare legal agreement with the European Commission.
In 2017 a ‘mid-term evaluation’ of Horizon 2020 was undertaken to report on the
progress of Horizon 2020 in the first three years (2014-2016). Arguments presented in
this paper are also based on data used from this official report.
Every year the European Commission published a report on the progress of the
programme entitled the ‘Annual Monitoring Report’. Data used in this paper is from
the most recent Annual Monitoring Report (2015).
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 19
2 The Evaluation Process in Horizon 2020
Before discussing success rates it is important to understand the evaluation processes
used in Horizon 2020. The evaluation process varies between the different
programmes. In this paper a general evaluation process is described. When a proposal
arrives in Brussels it is first checked by a Commission official against ‘eligibility’
criteria. ‘Eligible’ means that the forms were filled in properly. A proposal based on a
very weak scientific idea can be classified as ‘eligible’ if the forms are filled in properly
and the basic rules are met.
When the European Commission quotes ‘success rates’ they are based on these ‘eligible
proposals’. Clearly this is not a good denominator.
The ‘eligible’ proposals are then sent to independent scientific evaluators. The
evaluators read the proposals individually. In some progammes the evaluators meet
(Consensus meeting) and in other cases they simply send their evaluation scores to the
relevant administrative body in Brussels. A final score is agreed based on the evaluators
individual scores. A ‘threshold’ is set to define ‘low quality proposals’. For example,
in some programmes the maximum score is 15 and a threshold is set at 10. Any proposal
scoring below 10 is considered low quality and automatically elimated. Scoring below
10 is a very low score. These proposals should be classified as ‘low quality proposals’.
If research organisations had effective quality control procedures, these proposals
would never have been submitted.
In the case of the ERC Programme the maximum score is 8 and a threshold of 4 is set.
This threshold is sub-divided i.e. a threshold of 2/4 for the researcher and a threshold
of 2/4 for the idea.
Following this evaluation a list of ‘high quality proposals’ is compiled. The final
selection of successful proposals is based on these high quality proposals. This is a far
better denominator to use when calculating the success rates
3 Success Rates in Horizon 2020 (2014-2016)
Total number of proposals received (2014-2016) = >104,000
Total number of ‘eligible proposals’ = 102,076
(This means that over 2000 proposals did not fill in the forms properly)
Proposals below threshold (low quality proposals) = 56,444
(55.3% of eligible proposals)
High Quality Proposals (above threshold) = 45,632
Proposals selected for funding = 11,108
Reported Success rate (funded/eligible) = 10.88% (11,108/102,076)
Real Success rate(funded/high quality proposals) = 24.34% (11,108/45,632)
This overall success rate of 24.34% is a far more encouraging number for researchers
thinking about submitting proposals.
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 20
ERC Starting Grants (2014-2016)
Source: European Commission (EC) Mid-Term Rev iew of Horizon 2020 (Table 8 and 9)
The ERC Starting grant is one of the most prestigious grants for the career development
of young researchers. Proposals are submitted by individual researchers.
The maximum score that reviewers can award ERC proposals is 8 (4 for the researcher,
4 for the research idea). Proposals below the threshold (with a score < 4 ) in ERC are
given a Grade C. Any researcher receiving a grade C is not allowed to resubmit a
proposal to ERC for two years.
Total number of ERC Starting grant proposals received (2014-20160)
= 8947
Proposals below the threshold (Grade C) = 6120
(68.4% of eligible proposals)
High Quality Proposals (above threshold) = 2827
Proposals funded
= 950
Reported Success rate (funded/eligible)
= 10.6% (
950
/8947)
Real Success rate(funded/high quality proposals) = 33.6 % (950/2827)
This success rate (33.6%) is very encouraging for young researchers planning a career
in science.
4 Success Rates Horizon 2020 (2015)
The Annual Monitoring Report (2015) provides far more detailed data on individual
programmes. It is the most recent Annual Monitoring Report available at the time of
writing this article.
Total number of proposals received in 2015 = 42,535
Proposals below threshold (low quality proposals)
= 22,511 (53% of eligible proposals)
High Quality Proposals (above threshold) = 20,024
Proposals selected for funding = 4,565
Reported Success rate (funded/eligible) = 10.7% (4565/42535)
Real Success rate(funded/high quality proposals) = 22.8% (4565/20024)
ERC Statistics for 2015
Total number of ERC proposals received in 2015 = 10,019
Proposals below the threshold (Grade C) = 6083 (61% of eligible proposals!!)
High Quality Proposals (above threshold) = 3936
Proposals funded = 1327
Reported Success rate (funded/eligible) = 13.3% (1327/10019)
Real Success rate(funded/high quality proposals) = 33.7% (1327/3936)
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 21
SME Instrument (2015) (Page 122)
An SME Instrument is a special grant for Small and Medium Sized Enterprises (SME).
Companies submit proposals individually (no mandatory partners required).
Total number of proposals received in 2015 = 11008
Proposals below the threshold = 8378 (76% of eligible proposals)
(Here the threshold is < 13/15 for Phase I grants)
High Quality Proposals (above threshold) = 2630
Proposals funded = 714
Reported Success rate (funded/eligible) = 6.5% (714/11008)
Real Success rate (funded/high quality proposals) = 28.2% (714/2630)
For companies considering a proposal for an SME instrument a thorough understanding
of the evaluation process and the evaluation criteria is essential.
Table 1.
Summary of Real Success Rates in the individual programmes in 2015 (SME Instrument
Proposals are excluded from the different programmes)
Programme
Number of
Proposals
Below
Threshold
High Quality
Proposals
Funded
Proposals
Real Success
Rates
Health (page 132)
1212
75%
318
94
29.5%
Energy (page 145)
839
71%
243
114
47%
Food ( page 138)
358
36%
228
62
27%
Transport (page 153)
702
33%
467
167
35%
Climate (page 160)
648
48%
335
79
23.5%
Security (page 172)
463
45%
252
39
15.4%
5 Conclusions
5.1 Message for the European Commission
Why does the European Commission continue to quote success rates based on ‘eligible’
proposals? This number is meaningless. Success rates should be calculated as a
percentage of ‘High Quality Proposals’ – those that score above the threshold. This is
how 2015 results should be reported:
“In 2015 a total of 42,535 proposals were submitted to Horizon 2020. After evaluation
by independent experts, 22,511 (53%) were classified as ‘low quality proposals’. From
the remaining ‘High Quality Proposals’ (20,024) a total of 4,565 proposals were
accepted for funding. This represents a 22.8% success rate.”
5.2 Message for Research Organisations
Research Organisations must take a share of the responsibility for the large numbers of
low quality proposals submitted. Quality control procedures such as screening and
Journal of Innovation Management McCarthy
JIM 5, 4 (2017) 18-22
http://www.open-jim.org 22
proposal clinics, should identify weak proposals – before any considerable effort can
be wasted on their preparation. This should be a core activity of Research Support
Offices.
5.3 Messages for Researchers
Researchers must bear in mind the words of the Greek statesman and orator Pericles
(450BC) “Having knowledge but lacking the power to express it clearly is no better
than never having any ideas at all.”
In a lecture you tell or express the idea. In a competitive proposal you have to sell the
idea to the evaluators.
It is essential to understand how different types of evaluators think and how decisions
are made in the evaluation process. There are two ways to learn this – attend training
courses or (better) become an evaluator.
6 References
Official Website of Horizon 2020. Access 10th January2018:
http://ec.europa.eu/programmes/horizon2020/
Projects funded in Horizon 2020. Accessed 10th January2018:
http://cordis.europa.eu/projects/home_en.html
Mid-Term Review of Horizon 2020. Accessed 10th January2018:
https://ec.europa.eu/research/evaluations/pdf/book_interim_evaluation_horizon_
2020.pdf
Annual Monitoring Report of Horizon 2020 (2015) Accessed 10th January2018:
https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-
monitoring-report-2015
... The results of Polish higher education institutions in applying for European grants under the 7th Framework Program (FP) and the Horizon 2020 program are not satisfactory [20]. The participation of Polish institutions in the H2020 budget is only 1.21%. ...
... The topic of the success factors of research proposals has been tackled by numerous researchers, who investigated various aspects of key success determinants. McCarthy [20] noted that the success of the collaborative H2020 research proposals was determined by two main criteria: having an excellent scientific idea, and a thorough understanding of the evaluation process. Some research works analyse the networks of scientific institutions and their impact on the success of collaborative grant applications [1], [15]. ...
Conference Paper
Full-text available
The goal of the research is to understand the factors that determine the success of international research proposals. For this purpose, a multi-stage study will be carried out. The study will include a systematic literature review, semi-structured interviews with European Commission evaluators, and the development of the model of success determinants. Text analysis of historical proposals will enhance the knowledge of the success factors linked with applications' discourse and language. The proposed research will complement the existing literature as the study will be based on a comprehensive dataset covering all funded and rejected project applications under the European Union's Horizon 2020 Framework Program. The model will support the participation of Polish higher education institutions in European funds, whose acquisition is a key challenge for them, and provide an opportunity for the development of innovative research and international cooperation. It might also be useful for institutions in other countries, especially those that similarly to Polish institutions have a low share in acquiring European Union funds.
Conference Paper
Full-text available
Since the Industrial Revolution, industrialization has been spreading quickly. As the use of technology becomes more common, the need for producing increases. Because the improvement of technology determines a brand new product style for people. Developing technology leads many companies to be in a competition. Tüpraş, which has many successful initiatives around the world, has shown stability in terms of success and has made its name known to the world with many studies. Making the right decision in production is a prerequisite for most companies on the road to success. It is clear that there will be differences between the branches even if the same businesses. Because the branches differ from each other for many reasons such as the storage capacity, processing capacity, the number of staff and the number of sales. Despite these differences, high performance production is an important factor to show a successful appearance. In this study, data envelopment analysis was performed by determining the input and output data of Tüpraş Refineries which are located in Turkey. Gams optimization program was used for this analysis. With the results obtained, it has been concluded that among the refineries in different cities which ones had low or high-performance. In addition, Topsis method which is a multi-criteria decision-making system was used to analyze and compare the results. The aim of the study is to help to make comparisons between performance situations without risk if there is an increase in the number of production areas in the existing businesses.
Official Website of Horizon 2020
Official Website of Horizon 2020. Access 10th January2018: http://ec.europa.eu/programmes/horizon2020/ Projects funded in Horizon 2020. Accessed 10th January2018: http://cordis.europa.eu/projects/home_en.html
Mid-Term Review of Horizon 2020
Official Website of Horizon 2020. Access 10th January2018: http://ec.europa.eu/programmes/horizon2020/ Projects funded in Horizon 2020. Accessed 10th January2018: http://cordis.europa.eu/projects/home_en.html Mid-Term Review of Horizon 2020. Accessed 10th January2018: https://ec.europa.eu/research/evaluations/pdf/book_interim_evaluation_horizon_ 2020.pdf