Technical ReportPDF Available

Abstract and Figures

The aim of this guidance document is to disentangle and characterise the concrete benefits of research infrastructure investment for different stakeholders and to build a schematic impact assessment framework that can be used in evaluations to trace the core impact pathways.
Content may be subject to copyright.
www.technopolis-group.com
2015
Evaluating and Monitoring the Socio-
Economic Impact of Investment in
Research Infrastructures
www.technopolis-group.com
Evaluating and Monitoring the Socio-
Economic Impact of Investment in Research
Infrastructures
technopolis |group|, 2015
Elina Griniece, Alasdair Reid and Jelena Angelis
Table of Contents
1. Introduction 1!
1.1 Why evaluate the socio-economic impact of research infrastructures? 1!
1.2 Previous evaluations of the impact of research infrastructure investment 2!
1.3 The aim of this guidance document 3!
2. Logical framework for socio-economic impacts of investment in research
infrastructure 4!
2.1 Design and construction phase 4!
2.2 Operational phase 7!
3. Impact assessment methods and tools 13!
3.1 Impacts on economy 13!
3.2 Impacts on human resource capacity 14!
3.3 Impacts on innovation 14!
3.4 Impacts on scientific activity 15!
3.5 Impacts on society 15!
3.6 IA implementation process, its challenges and limitations 17!
Bibliography 18!
Table of Figures
Figure 1: Typology of research infrastructures ................................................................ 1!
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure ....... 2!
Figure 3: Logical framework for socio-economic impact assessment of investment in
research infrastructure ..................................................................................................... 5!
Figure 4: Indicators for economic impacts of RI construction phase ............................. 6!
Figure 5: Indicators of innovation impact of RI construction phase .............................. 7!
Figure 6: Indicators for economic impact of RI operational phase ................................. 8!
Figure 7: Indicators for impact of RI operation on human resource capacity ................ 9!
Figure 8: Indicators for impacts on innovation of RI operational phase ...................... 10!
Figure 9: Indicators for impact on scientific activity ..................................................... 12!
Figure 10: Indicators for impacts on society .................................................................. 13!
Figure 11: Implementation of an impact assessment of research infrastructure .......... 16!
Evaluating the socio-economic impact of research infrastructures
© Technopolis Group, 2015 1
1. Introduction
1.1 Why evaluate the socio-economic impact of research infrastructures?
Research infrastructures (RI) refer to facilities, resources (including human) and
related services needed by the research community to conduct research in any
scientific or technological field. Research infrastructures include:
Major equipment or group(s) of instruments used for research purposes;
Permanently attached instruments, managed by the facility operator for the
benefit of researchers, industrial partners and society in general;
Knowledge-based resources such as collections, archives, structured
information or systems related to data management, used in scientific
research;
Enabling information and communication technology-based (ICT) or ‘e-
infrastructures’ such as grid, computing, and software communications;
Any other entity of a unique nature that is used for scientific research.1
Due to the large number of research communities and complex research needs, there
are very different types of research infrastructures with specific characteristics. Figure
1 provides an overarching typology of RIs.
Figure 1: Typology of research infrastructures
Type of research
infrastructure
Description
Examples
Single-site facility
Unified body of
equipment at one physical
location
High-performance laser system,
clean room, coastal observatory,
centre of competence
Distributed facility
Network of distributed
instrumentation or
collections, archives and
scientific libraries
ELI: European Light
Infrastructure; Council of
European Social Science Data
Archives
Mobile facility
Mobile vehicles specially
designed for scientific
research
Research vessels, satellite and
aircraft observation facilities
Virtual facility
ICT-based system for
scientific research,
including high-capacity
communication networks
and computing facilities
European Grid Computing
Infrastructure; Digital Research
Infrastructure for the Arts and
Humanities (DARIAH)
Setting-up, or renovating, RI usually requires a considerable level of financial
investment and a long-term operation strategy. For example, in the environmental
sciences, many research infrastructures bring their scientific return only after decades
of sequential data recording. The investment strategy requires careful planning of the
operation phase and possible future reinvestments. This entails the purchase of
technologically advanced equipment, clustering of specific skills to enable its
deployment and devising appropriate governance structures. RIs can also be closely
linked to other research and innovation establishments, such as vocational schools,
universities, private and public research centres, research hospitals, business
incubators and science and technology parks.
1 Definition used by the European Commission
Evaluating the socio-economic impact of research infrastructures
© Technopolis Group, 2015 2
While RIs are designed for research needs, the impacts of these facilities reach beyond
fuelling scientific excellence. The advanced technical opportunities and the
concentration of skilled human capital and know-how can foster innovation, create
new or expand the existing markets, attract inward investment, increase economic
activity and potentially have an impact on the social and cultural life in a particular
region. In this regard RIs can be viewed as focal points for continuous interaction
between scientific, technological and socio-economic development.2
The construction and operational phase of RIs are largely funded from public budgets
either through national funding from one or several countries, or a mix of national and
EU funding. For this reason it is crucial to understand the return on investment in RI
to support informed decision-making. Yet it is difficult to quantify and understand this
return in conventional commercial terms. Investment in RI brings a broad range of
benefits that spreads across wider society rather than serving merely the direct
stakeholders (owners and users of RI). Official statistics do not sufficiently describe
the variety of benefits associated with the development and, more importantly,
exploitation of RI. It calls for more elaborate and fine-tuned approaches to account for
the impacts that the RI investment brings on science, economy and society.
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure
1.2 Previous evaluations of the impact of research infrastructure investment
Currently there is no unified framework for the impact assessment (IA) of investment
in research infrastructures. Various conceptual frameworks exist in parallel
comprising a range of observable direct and indirect effects and longer-term impacts.3
The existing studies focus on three types of impacts: 1) the direct and indirect
economic benefits of spending large amounts of public money in a single location; 2)
the industrial knowledge spillovers realised by contractors that design, build and
equip research facilities and spin-offs that provide specialist technical services back to
the facility (or other facilities); and 3) the local economic effects and high-technology
2 Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop
“The Socio-Economic Relevance of Research Infrastructures”, 31 May-1 June 2012, Hamburg
3 See, for example, project Research Infrastructures: Foresight and Impact (RIFI), project on Evaluation of
Research Infrastructures in Open innovation and research systems (EVARIO), Research Infrastructure
Group of UK Science & Technology Facilities Council, Czech Metodika: Evaluation Methodology for
Research Infrastructures
Evaluating the socio-economic impact of research infrastructures
© Technopolis Group, 2015 3
clusters that grow up around some of the larger facilities.4 A heterogenous set of
methods is applied to capture these effects of RI. Most of them address standard
economic impacts (direct effects) and to some extent economic multipliers.5
Comprehensive and methodologically demanding studies are still rare. Core aspects of
RI benefits, such as their impact on human and social capital formation and
innovation, are not extensively explored. Up to date, the existing literature provides
insufficient evidence to support claims that investment in RI (even large-scale) attract
and retain talent and promote innovation.6 Therefore, important socio-economic
contributions remain poorly understood. In this respect, significant work is
undertaken by FP7 project “Research Infrastructures: Foresight and Impact (RIFI). A
Foresight Enriched Research Infrastructure Impact Assessment Methodology
(FenRIAM)7 was developed during the project, which significantly contributed to the
understanding of the impact of RI on learning and capacity of RI operators, suppliers
and users. A key lesson is that there is a need to treat the impact pathways in greater
detail and substantiate them with empirical evidence.
Most previous concept papers and evaluations have been devoted to studying large-
scale research infrastructures, or so-called ‘big science’ facilities that are of European
and international significance. Empirical work on investment in mid-size
instrumentation of national and macro-regional significance barely exists. This is
mainly the result of the fact that RI investments have not yet been subject to regular
evaluations. There is a scope for further methodological developments and their
applications in evaluations to evolve from narratives of socio-economic effect of RIs to
more empirical models and measurement techniques. This knowledge would help to
move from simple ex-post detection of intended and unintended returns of RIs to a
better understanding and planning of future investment.
1.3 The aim of this guidance document
The aim of this guidance document is to disentangle and characterise the concrete
benefits of RI investment for different stakeholders and to build a schematic impact
assessment framework that can be used in evaluations to trace the core impact
pathways. Section two introduces a logical framework for the impact assessment of RI.
Recognising that each RI is embedded in specific socio-economic conditions, there will
never be a ‘one-size fits all’ approach to mapping all socio-economic impacts. This
work should be regarded as a step towards defining a typology of possible effects and
accounting for the conditions and patterns that enable their creation and diffusion.
The third section elaborates on the methods and tools that are suitable for the
measurement of respective impacts. The identified impact pathways involve complex
and multidimensional phenomena, which in most cases cannot be captured with
standard quantitative indicators, but require a thorough triangulation of quantitative
and qualitative evidence. The section elaborates on the appropriate indicators for each
type of impact and provides additional guidance on the respective data requirements,
their potential sources and advice on collection routines. Reflections on the limitation
and challenges of the application of this IA framework are outlined in conclusion.
4 Simmonds, P. et.al (2013) Big Science and Innovation. Report for the UK Department for Business,
Innovation & Skills
5 Multipliers - Further economic activity (e.g. jobs, expenditure or income) associated with the inputs to or
outputs from the project. According the European Commission (DG REGIO) an income multiplier is a
secondary effect resulting from increased income and consumption generated by the public intervention
(investment). Multiplier effects are cumulative and take into account the fact that part of the income
generated is spent again and generates other income, and so on in several successive cycles. In each cycle,
the multiplier effect diminishes due to purchases outside the territory. The effect decreases much faster
when the territory is small and when its economy is open.
6 Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau Instituut, the Hague,
the Netherlands
7 http://www.fenriam.eu/
Evaluating the socio-economic impact of research infrastructures
© Technopolis Group, 2015 4
2. Logical framework for socio-economic impacts of investment in
research infrastructure
Figure 3 provides a schematic overview of the various direct and indirect benefits that
arise during two main periods of a research facility’s lifecycle:
1) design and construction phase; and
2) operational phase of research facilities.
The specificities of each of the pathways and their contributions to other impacts that
are singled-out in this two-dimensional graph are described in detail in the following
sub-sections of this chapter. The types of impacts are positioned in the graph to
visually reflect better their relation to each other, not an order of their importance.
Societal impacts can be of very diverse nature and lead to broad, overarching
outcomes. In order to provide a concise graphical representation of IA framework
these impacts are not schematically reflected in this graph.
This IA framework is predominantly designed for single-sited and geographically
confined national or macro-regional distributed research infrastructures. With regards
to highly distributed and virtual RI case-specific a range of additional factors have to
be considered. These types of RI usually are decentralised administratively and
financially and include much larger geographical areas of impact. They can have
mainly or only virtual access without the requirement to cover physical support to
individual users. Advanced ICT-based technical solutions and well-designed access
rules may ensure much larger outreach with dispersed impact on user and stakeholder
groups. These characteristics of highly distributed and virtual RI require more case-
specific approach to impact assessment.
2.1 Design and construction phase
2.1.1 Economic impacts
During the design and construction phase a wide range of technical and scientific
knowledge is applied to set-up the required facilities. The physical construction and
refurbishment of buildings and their accompanying networks and utilities engages
many companies ranging from construction material suppliers to technical and legal
service providers. The procurement of core construction services typically involves
local companies8, which creates additional revenue and increases temporary
employment opportunities in the region. Despite the fact that the impact of many of
these expenditures is usually confined and short-lived for smaller-scale RI, there can
be a significant further multiplier effect on the local economy, for example, due
improved reputation credentials for firms that supply high-tech facilities and an
increase in the local tax base. In-depth analysis of the local industrial base and
mapping of relevant global supply chains is required to scope the total economic effect
of the RI investment.
8 Locally-owned businesses, franchise or corporate branch operating within a local area
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 5
Figure 3: Logical framework for socio-economic impact assessment of investment in research infrastructure
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 6
Figure 4: Indicators for economic impacts of RI construction phase
No
Indicators
Explanatory remarks and examples
1.
Number of commercial suppliers
for RI design and construction
phase
Suppliers can be mapped per sector, field
of activity, size, technology group (low-
tech/high-tech), geographic scope
(national/international/multinational)
2.
Scale of commercial suppliers’
turnover increase due to RI
For applied method to estimate utilities in
monetary terms see the study “Economic
utility resulting from CERN contracts”9
3.
Scale of commercial suppliers’
employment increase due to RI
Measured in FTE and grouped per
category of employees (e.g. technical staff,
R&D staff)
2.1.2 Impacts on innovation
Research facilities under construction may require specific design and building
solutions to serve the purposes of a particular research field (e.g. laboratories, clean
rooms, etc.). Such requirements foster collaboration among the involved suppliers,
facility mangers and scientists in developing innovative design solutions and building
functionalities. Local companies can benefit from accumulated know-how and
complementary skills that expand their competitive advantage in other markets (e.g.
enable winning international procurements of similar research facilities, secure service
contracts as external consultants to other developers). In cases where foreign
companies demonstrate a superior skills base and are selected to address particular
design and construction tasks, the benefits of the investment flow out of the national
economy. Moreover, in the RI construction phase, the universities and responsible
government agencies may acquire additional and more sophisticated project
management skills that can have knock-on effects on the potential for public sector
innovation.
The purchase of the necessary scientific instruments and supporting equipment is
generally open to international bidders, as the supplier base for such specific
commodities and accompanying services is often very specialised and, in certain cases,
the specific equipment may not exist ‘off-the-shelf’ on the market. Hence, the
procurement of equipment can have a considerable impact on extending the
technological frontier and fostering innovation10. As a result, the impact from this
procurement will be distributed across specific global supply chains for scientific
instrumentation.
9 Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second study). Report of
the European Organisation for Nuclear Research (CERN)
http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
10 For the theoretical background of this nexus see work of von Hippel, E. (1976) The Dominant Role of
Users in the Scientific Instrument Innovation Process, Research Policy 5 (3)
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 7
Figure 5: Indicators of innovation impact of RI construction phase
No
Indicators
Explanatory remarks and examples
1.
Number of joint
development activities
with suppliers
The novel technical requirements for the
construction (design or building techniques) of
scientific facilities or the equipment require
close interaction between the recipient
organisation and one or more suppliers. This
can be realised through a process of pre-
commercial public procurement11, design
contest, a forward commitment procurement or
a competitive dialogue. In these cases, the
recipient organisation defines specific technical
requirements or criteria to be met but not the
precise solution (in the form of a specific piece
of equipment, etc.).
2.
Number of contracts
concluded for high-tech or
specialist services that
require development, or
calibration of
designs/equipment to
meet specific requirements
The focus should be on the extent to which
specific design or technical requirements for
scientific buildings or equipment induce an
impact on learning and skill development
amongst supplier firms, notably local (national)
firms.
2.2 Operational phase
Two main categories of impacts of the operational phase of RI exist: 1) impacts from
the routine operation, maintenance and upgrading of RI; and 2) impacts from the use
of research facilities.
2.2.1 Economic impacts
Benefits from RI operation include the assessment of the performance characteristics
of the research facilities. This includes a review of RI performance addressing aspects
of quality, access, reliability of the infrastructure, account of all the services provided
and assessment of the optimal usage of equipment. Among the criteria to review is the
flexibility of staff to support external users, feasibility of contractual arrangements for
granting access, quality of accompanying services (e.g. tech transfer, incubation
services). Performance aspects have a direct impact on the efficiency and effectiveness
of RI operations, user satisfaction, and consequently on the generated revenues to RI
and its users.
Maintenance and operation of RI involves longer-term effects on employment in
higher education establishments, science parks and companies, such as additional jobs
for scientists, technicians, administrative and support personnel working on the RI.
The operation of facilities also includes expenditure on goods and services and routine
upgrades with additional procurement design and associated equipment. These
activities have a further multiplier effect on the local economy and relevant global
supply chains.
11 http://www.innovatiefaanbesteden.be/files/409
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 8
Figure 6: Indicators for economic impact of RI operational phase
No
Indicators
Explanatory remarks and examples
1.
Number of scientists,
students, state-owned or
private enterprises that
benefitted from RI services
Service can be categorised into broad types,
such as: free or for a fee access to equipment
(including technical support from qualified
technicians), testing or metrology (carried out
by the RI staff in laboratories or using mobile
equipment), contract research, consultancy,
hosting doctoral students, etc.
2.
Total amount of funding
generated from services,
grants and joint projects
Per service type, source of research grant and
type of joint project
3.
Number of new directly
and indirectly created jobs
Per category (scientific/ technical/
administrative) and wage level; the aim is to
scope how RI impacts on an increased
employability of human resources.
4.
Total amount of
expenditure on personnel,
operations, maintenance
For centralised large-scale facilities the rule of
thumb is to calculate around 10% of investment
costs for annual operation expenditure. Yet for
most other types of RI this percentage is
significantly higher.12 For distributed RI there
are higher costs due to coordination,
management and consultation tasks requiring
qualified personnel. Operating costs for RI in
the humanities and social sciences are usually
the most significant expenditure.13
5.
Total RI capacity
utilisation
RI access hours used as % of the total available
access time
6.
RI capacity utilisation
external business users
RI access hours used by business entities as % of
the total available access time
7.
Financial sustainability of
RI
Measured as % of the total costs funded from
the provided services, received grants and
realised joint projects
2.2.2 Impacts on human resource capacity
The build up of human resource capital is a major benefit of RI. Modern research
facilities can serve as magnets to retain and attract talent. This includes local
researchers, technicians, students and global talent through recruitment of permanent
staff and the promotion of the flow of visiting researchers and foreign students.
RI has a crucial role to play in training and skills development. Research facilities
concentrate skilled staff that hold tacit knowledge how to operate them. Only with the
help of skilled technicians and experienced researchers that students and other
interested stakeholders can learn to set up experiments and interpret their results.
This tacit dimension of knowledge transfer makes RIs entry points into networks of
knowledge, expertise and practice. To determine the impact of RI on skills
development it is important to explore how the exchange between the younger staff
12 Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research Infrastructure
Projects for the National Roadmap (Pilot Phase)
13 Wissenschaftsrat (2011) Recommendations on Research Infrastructures in Humanities and Social
Sciences
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 9
(MAs, PhDs, post-docs) and experienced researchers and technicians is structured and
promoted. Equally important is to appraise the ways tacit knowledge is transferred to
private sector users.
The diffusion of knowledge gained by the RI management, operators and users may
have a significant societal value. For example, the research commercialisation skills
acquired by the management and scientists at a RI is an important benefit with far-
reaching implications on the local innovation system. Closer formal and informal
social networks increase interpersonal trust and knowledge sharing. This consequently
augments the quantity and diversity of knowledge that is available to RI users. Social
networks create new forms of interaction among the actors of the innovation system,
stimulate learning environments and increase the awareness of users (potential and
existing) about the scope of knowledge that is available on RI. In short, social capital
enables the full exploitation of accumulated human capital.
The mobility of people and emergence of new cooperation networks is a major
mechanism through which knowledge circulates and diffuses the benefits to wider
economy. Graduates leave RI with the knowledge of the most recent scientific results,
skills using advanced instrumentation and ability to apply them to complex problem
solving. Internships and recruitment of graduates mean that students act as a form of
‘social glue’ and they often create long-term links between scientists and businesses.14
However, if the local academic and industrial environment does not provide sufficient
absorption capacity for these advanced skills, the accumulated human capital may
remain underutilised or emigrate.
Figure 7: Indicators for impact of RI operation on human resource capacity
No
Indicators
Explanatory remarks and examples
1.
Number of new jobs for
research and technical staff
attracted from abroad as %
of the total number of staff
employed on RI
Per category (permanent/visiting) and per year;
the aim is to understand the influx of new
knowledge that is directly linked with RI
2.
Number of Master thesis
defended, where
knowledge and skills
gained on RI were
exploited
Per year of completion and per scientific field;
Master theses are considered as a
training/education impact on skills profile of
population; the number of PhD thesis
completed is counted in the impacts on
scientific activity
3.
Number of graduates
trained on RI
Per stage (MA, PhD) and per year; distinction
should be made on skills gained by PhD
students that improve their general academic
and employability profile and skills directly
applied in PhD thesis work; the latter should
account for impacts on scientific activity
4.
Number of foreign
students as % of all
students trained on RI
Per stage (MA, PhD) and per year
5.
Data on the post-diploma
employment path of those
graduates trained on RI
Descriptive data should include at least
indication on the share of graduates finding
employment in academia/public sector/private
sector; further statistics on types of graduate
employment (part time/full time) and sector of
activity are desirable
14 Bozeman, B. (2000) Technology transfer and public policy: a review of research and theory. Research
Policy, 29(45)
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 10
2.2.3 Impacts on innovation
Innovation can result from RI through industry collaborations with academic research
groups and contract research that result in knowledge transfer. Research facilities
generate knowledge that businesses cannot acquire through their networks. More
exposure to RI knowledge networks through formal and informal social interaction
increases the possibility that private sector discerns new opportunities for exploiting
intellectual and technical capital available on RI.
Spin-offs are one way in which knowledge generated using RI can diffuse into the
market in the form of innovative product, process or service. Other avenues for
innovation impacts arise from the commercialisation of research through licences or
joint ventures with existing companies.
There are important barriers for private sector use of RI due to different objectives and
‘cultural factors’. The industrial community is often less interested in using RI directly
(e.g. placing industrial researchers in RIs to carry out R&D). There is more interest in
acquiring technical training and co-developing new IPR.15 In some sectors (like food or
agriculture) the industrial landscape can be dominated by a large number of SMEs
that, individually, lack resources to engage in research collaboration.
The table below provides a number of indicators that can be used to track the impact
of RIs on knowledge transfer and innovation. One example of a framework for
measuring the quantity of knowledge transfer from the research base is the Scottish
Funding Council's (SFC) Knowledge Transfer Metrics Return16.
Figure 8: Indicators for impacts on innovation of RI operational phase
No
Indicators
Explanatory remarks and examples
1.
Number of collaborative
research projects and
volume of funding
Collaborative research carried out at that the RI
might involve both regional and international
researchers using the equipment for shorter-or-
longer periods. The funding generated from
service fees charged to research team will enable
reinvestment in equipment. Qualitative research
on the research teams working in or with the RI
can be used to map (e.g. using social network
analysis) the place of the RI in global research
networks.
2.
Number of R&D projects
commissioned by
companies and volume of
their funding
Contract research is one step closer to
generating a socio-economic impact since firms
purchasing research services will only do so in
order to improve production processes
(enhancing productivity) or develop prototypes
of new products or services. Tracking trends in
volume of funding provides an indicator of the
longer-term financial sustainability of the RIs.
Similarly, tracking the renewal of the ‘client
base’ (contracts with new clients) rather than
just number of companies contracting research
is useful.
15 ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31 May 1 June 2012,
Hamburg, Germany
16 The metric is used by the Scottish Government to measure as a proxy measure of the quantity, but not the
quality, of knowledge exchange activities undertaken by Scottish universities.
http://www.gov.scot/About/Performance/scotPerforms/TechNotes/knowledge
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 11
No
Indicators
Explanatory remarks and examples
3.
Number of technologies,
prototypes, industrial
designs developed and
transferred
Depending on the RI’s intellectual policy policy,
the results of collaborative and contract
research may be potentially exploited through
proof of concept and prototyping either by the
RI itself or in co-operation with academic or
industrial partners. Ideally, the monitoring of
outcomes should go beyond ‘number counting’
and aim to track whether prototypes have been
scaled up or further developed commercially.
4.
Number of start-ups and
spin-offs created with
support from RI services
growth in turnover/value
added and employment
The number of start-ups/spin-offs is an
indicator commonly used to assess economic
impact. However, there is a need to avoid
unrealistic expectations when setting targets for
numbers of spin-offs as this involves IP transfer
to the new companies potentially closing future
avenues of academic research. Moreover, many
spin-offs fail to grow substantially and there is
an need to track growth in turnover and
employment on a longitudinal basis (over 5-10
years) to fully understand impact.
5.
Number of feasibility or
market studies for
industrial investment and
application of technologies
Commercialisation of research results will
involve, traditionally, market research or
industrial scale feasibility study prior to
investment. The number of (e.g. proof of
concept/prototyping) projects that move to the
stage of industrial investment is a strong
indication of the economic relevance of the
research. Tracking over time (5-10 years) the
actual investment in the application of new
technologies by businesses provides a stronger
indication of economic impact.
6.
Procurement contracts
signed for development
and upgrade of research
equipment
Tracking the number and type of procurement
contract for the development of innovative
instruments and products (rather than simply
‘off-the-shelf purchases from equipment
suppliers provides an indication of the impact of
the RI on the scientific equipment market this
is particularly pertinent if suppliers are
regionally based.
2.2.4 Impacts on scientific activity
The impact of RI on scientific activities can result in accumulation of new knowledge
and methodologies to push the boundaries of fundamental science. There is an
inherent tension between the development of non-proprietary research, which is
published in relevant scientific journals and can be accessed by other interested
parties, and propriety research where knowledge and technical data, which can be
basis for specific inventions, are not disclosed openly.
Another linkage to scrutinise is the impact of the use of RI on scientific productivity,
international recognition and reputation. The availability of advanced equipment has
an effect on shaping how scientific communities organise themselves. It can
significantly increase the productivity of research teams, as they do not need to seek
and arrange experimentation and testing opportunities on limited-availability facilities
or abroad (e.g. access to Grid computing facilities substitutes the need to queue for
supercomputer service time).
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 12
Open access to RI also induces more regular inward flow of researchers promoting
closer involvement of local teams in international research networks (brain exchange).
Such interaction can generate important learning effects for experienced and early-
stage researchers alike. Increase in international recognition of leading scientists and
the capacity of research teams can attract further international competitive funding to
the research system. In addition, researcher experience gained during the set-up and
operational phase of RI can have further policy impact through contributions to RI
roadmaps.
Figure 9: Indicators for impact on scientific activity
No
Indicators
Explanatory remarks and examples
1.
Number of articles
published in the ISI level
international scientific
journals as a direct result
of research using RI
The aim should be to track the dynamics of
publishing behaviour due to RI; peer-reviewed
articles applies as an indicator of scientific
activity to most scientific fields; in humanities
and some sub-fields of social sciences books,
book chapters and monographs are more
relevant indicator
2.
Number of
methodologies/designs
developed
Applies to those scientific fields where this
output is relevant to measure scientific
productivity
3.
International patents
granted and published
patent applications (all
types)
Indicator is relevant for most scientific fields in
natural sciences, engineering, life sciences and
medicine; in humanities and social sciences
other types of IPR (e.g. copyrights) should be
considered
4.
Number of PhD
dissertations completed
Per year of completion and per scientific field;
includes those PhD dissertations which have
been predominantly or partly-based on the use
of RI services or equipment
5.
Number of scientific events
organised on research
topics directly relating to
RI services
Data on the frequency, types of events,
geographic origin and institutional affiliation of
participants can provide some proxy on the RI
impact on mutual learning, knowledge exchange
and researcher involvement in major scientific
networks; some conclusions can be drawn on
the critical mass and synergy effects with other
RI facilities in the region
2.2.5 Broader impacts on society
Regarding wider social impacts, it is important to outline that research infrastructures
can play an important role in scientific communication and scientific education. RI
can be used to inspire school students to learn STEM subjects. For example, large-
scale RI may organise open days for the general public or for schools. The policy-
decision on investment in RI can also be widely reflected in the press leading to
increased public awareness of science.
Research outcomes from using RI services can lead to concrete innovative products,
and services that are taken up and diffused in society (for example, new medical
instruments, diagnostics, treatments). While such impacts on society are broad,
indirect and very difficult to attribute and quantify, it is nevertheless important to
scope the potential contribution of RI to solving various societal problems.
There can also be concrete benefits that result from improving local infrastructure,
urban planning and community services as the RI investment may revitalise certain
areas with important indirect societal benefits. These developments can further shape
local cultural activities and citizen lifestyles.
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 13
Figure 10: Indicators for impacts on society
No
Indicators
Explanatory remarks and examples
1.
Number of organised RI open
days for wider public and any
available data on participant
satisfaction with the events
Per year and per target group (e.g.
schools/general public); aim to provide an
indication on the impact of public
awareness of science
2.
Number of press articles on the
investment in research
infrastructure
Per year and per type of publication (print
media/online articles/specialist
publications)
3.
Number of new or improved
products, services, solutions as a
result of research using RI that
are diffused in society
This indicator involves especially
significant time-lag; the aim to scope the
possible contribution of RI to specific
societal outcomes
4.
Account of improved local
infrastructure, community
services, increase in local
cultural/recreational activities
due to RI
Aim to scope the possible contribution of
RI to the outcomes
3. Impact assessment methods and tools
3.1 Impacts on economy
To measure the direct economic effect of the RI design and construction phase
requires an analysis of all suppliers that have won the procurement tenders to scope
the contract values per sector of activity and service provider affiliations
(local/foreign/multinational). Relevant supply chains should be mapped to determine
the scope of the output, value added and additional employment that has been created
locally and the benefits that have been diffused internationally. Desk analysis of the
proposal and contractual documentation and survey among suppliers can be used to
collect this information.
The operational phase requires a comprehensive look at the activities undertaken by
the managers responsible for establishing and running the research facility. This
involves a detailed account of facilities expenditure on salaries, purchase of goods and
services from financial records and direct output generated. Desk research and
interviews with RI managers/coordinators can be used to collate and analyse financial
and activity data.
The multiplier effect on the economy is addressed through an input-output analysis.
This ‘ripple effect’ includes the indirect economic effects that occur from all secondary
industries supporting the procured activities and induced effect representing the
changes in consumption of respective households. The input-output model is a
quantitative econometric technique that represents the interdependencies between
different sectors of a national/ regional economy. An input-output analysis estimates
the value of the purchases that flow between the identified supply chains in a given
period of time. Building IO model it is possible to derive economic multipliers.
Methodological approaches have been developed on applying cost-benefit analysis to
RI projects17, yet these are suited more for ex-ante project evaluation as it takes a long
time period for the effects of RI investment costs to materialise. Nonetheless, the
applicability of CBA type approaches can be explored. For instance, the effects on RI
17 See e.g. Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects. JASPERS
Staff Working Paper
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 14
users can be scoped through application of contingent valuation techniques.
Contingent valuation is a method that uses surveys to draw out the willingness of
actual and potential users to pay for certain services. The method involves assigning of
monetary values to non-market goods and services based on preferences.
Data requirements
Overview of contractual information with RI suppliers (design, construction,
maintenance, operations, renewal)
Data on the affiliation of suppliers (local/foreign/multinational)
Data on employment, wages
Facilities annual expenditure broken down by function in time series
Strategic plans and annual activity reports
3.2 Impacts on human resource capacity
To account for the impacts on human resource capacity it is necessary to capture the
structural changes in organisational behaviour that arise as a result of the research
infrastructure. Monitoring statistics on the number of staff and students attracted, in
particular from abroad, can shed a light on the dynamics of knowledge exchange that
RI is facilitating. It is necessary to understand what are the formal organisational
aspects, how training on RI is carried out and what are the practices for informal
know-how transfer (interaction with technicians, networks). Surveys among students,
vising and permanent researchers, technicians and RI managers can be employed for
necessary data gathering. Particularly important is to explore the spaces of student
and researcher interaction with industry. Longitudinal analysis of graduate career
paths is a valuable approach.
In addition to the available monitoring statistics and information from surveys, data
collection strategies for tracing impacts on human resource capacity should involve
extensive qualitative research techniques. These can include semi-structured
interviews with facility owners and a representative population of researchers and
students. A case study approach is an appropriate way to capture the dominant
patterns of knowledge diffusion and the degree of social capital formation that the
establishment of RI has promoted.
Data requirements
Relevant monitoring data, including e.g. statistics on the numbers of attracted staff and
students, (locally and from abroad) and career paths of graduates that have used RI for
training or MA/PhD thesis
Database of concluded service contracts
Documentation of human resource development strategies, user guides and other
documented evidence
Annual activity reports
3.3 Impacts on innovation
Impacts on innovations arise via multiple impact pathways. The most feasible method
is to apply quantitative analysis to scope economic effects of innovation (e.g. value of
IP portfolio) and use qualitative investigation to understand in detail the knowledge
spillovers. Monitoring data on knowledge transfer activities can be supplemented with
supplier surveys (for tracing procurement-led innovation and skills formation) and in-
depth interviews (with research teams, spin-offs, relevant industry representatives).
Narrative descriptions of individual case studies that look in detail into specific aspects
of innovation impact (e.g. impact via prototyping, industrial demonstrators) would be
supportive for the overall assessment. If any routine user surveys are executed by RI
management this would be very useful source of material to better understand user
experiences, motivations and wider effects of RI use.
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 15
Data requirements
Relevant official statistics and structured monitoring data (i.e. regular progress reports
and data for measure project level key performance indicators)
Database of users (academic, industrial) over the lifetime of the facility
Database of supplier contracts (held either by the funding agency or by beneficiary
higher education or public research institutes)
3.4 Impacts on scientific activity
Impacts on scientific activity can be analysed through a bibliometric analysis of the
research teams working on the RI. Especially valuable would be to see the evolution of
co-publications over a period of time. A time-series/network analysis of the
participation in international research project consortia (FP7, H2020) may shed some
light on the visibility of research groups on the international research landscape.
All quantitative techniques should be supplemented with in-depth interviews with key
researchers to deepen, test and validate the claims. There are many limitations, as the
knowledge spillovers that arise as a result of the intellectual advances may be largely
invisible to facility owners and stakeholders.
Data requirements
Database of researchers working on the RI
Relevant official statistics and structured monitoring data
Annual activity reports
3.5 Impacts on society
The broader impacts that research infrastructure investment have on society is the
least scrutinised area due to the fact that such impacts are difficult to trace and
quantify. While direct causal chains are almost impossible to establish, social science
methods can be employed to estimate the potential contribution of RI. Surveys among
schools, public sector, relevant stakeholder groups can be used for this purpose.
Further interviews with relevant target groups for RI communication and PR activities
and the involved researchers can be undertaken. Impact on STEM education and
public understanding of science can be best explained through analytical narrative in
the form of case studies.
Data requirements
Documented materials on RI public relations activities (open days, public lectures,
seminars)
Available data from satisfaction and feedback surveys of participants to RI
communication events
Track of media publications/online content regarding RI
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 16
Figure 11: Implementation of an impact assessment of research infrastructure
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 17
3.6 IA implementation process, its challenges and limitations
Figure 11 provides an overview the IA implementation steps and use of analytical
techniques. During the inception phase desk research is performed on all available
documented evidence to determine the information gaps that should be covered
through additional data collection methods. In the data collection phase surveys are
launched beforehand. In-depth interviews and the application of quantitative
techniques can be run in parallel. Case studies are usually chosen after the first
preliminary findings to ensure that the most relevant aspects and impact areas are
scrutinised in detail. The synthesis phase of findings ideally should involve a peer
review and an international expert team to ensure appropriate benchmarking of RI
performance. A full impact assessment takes usually around six to nine month period.
Data shortages are likely to be the most important challenge due to the limited data
caption routines during the lifetime of research facility. Database development with
appropriate quantitative data is a foundation for carrying out an in-depth impact
assessment. This calls for anticipatory collection of relevant data at an early stage of RI
project implementation. The principle should be that the amount if work for
researchers is kept to a minimum. RI management should be encouraged to regularly
assess key sources of information and collect relevant data on an on-going basis. These
efforts should include as a minimum data which is readily available or which can be
collected automatically, e.g. CVs, completed doctorates, concluded contracts, etc.
Bearing in mind the challenge of data availablity and quality, the expectations have to
be realistic in terms of the quantifiable findings that can result from the evaluation.
Further challenges and limiting factors to IA of RI include:
The difficulty of capturing multidimensional and complex research outcomes and
impacts with simple metrics that are used as core indicators. For instance, the
accuracy of input-output models to determine multipliers is questioned. Findings
from the application of quantitative methods should be well-balanced with
qualitative insights and domain expert validation.
Issue of displacement or the extent to which economic benefits occur at the
expense of other economic activities is hard to address in the analysis.
Delineation between short-term, medium and long-term effects will be very
difficult due to restraints in data availability in time series.
It is hard to separate between RI induced benefits and those fostered by other
determinants. The impacts of RI should be accompanied by analysis of parallel
initiatives in higher education and support measures for enterprises. National
legal frameworks (e.g. rules for skilled migration, IPR) are all important
determinants.
The establishment of systematic evaluation practice for investment in research
infrastructures is key to expand reference material and foster learning for more
comprehensive analyses of RI impacts.
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 18
Bibliography
Autio, E. (2014) Innovation from Big Science: Enhancing Big Science Impact Agenda.
Report to UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/28
8481/bis-14-618-innovation-from-big-science-enhancing-big-science-impact-
agenda.pdf
Autio, E., Streit-Bianchi, M., and Hameri, A.-P. (2003) Technology Transfer and
Technological Learning Through CERN’s Procurement Activity, CERN Scientific
Information Service. Geneva
Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second
study). Report of the European Organisation for Nuclear Research (CERN). Available
at: http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
Bozeman, B. (2000) Technology transfer and public policy: a review of research and
theory. Research Policy, 29(45)
Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects.
JASPERS Staff Working Paper. Available at:
http://www.jaspersnetwork.org/display/for/Project+Preparation+and+CBA+of+RDI
+Infrastructure+Projects
European Strategy Forum on Research Infrastructures (2011) Evaluation Report 2011.
Available at:
http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011.pdf
Fotakis, C. (2010) Analyses of FP7 supported Research Infrastructures initatives in the
context of ERA. Available at:
http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence-
base/experts_analysis/c.%20fotakis_-_research_infrastructure.pdf
Hallonsten, O., Benner, M. and Holmberg. G (2004) Impacts of Large-Scale Research
Facilities- A Socio-Economic Analysis. A study done at the Research Policy Institute,
Lund University. Available at:
http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R
I.pdf
von Hippel, E. (1976) The Dominant Role of Users in the Scientific Instrument
Innovation Process, Research Policy 5 (3)
Hickling Arthurs Low Corporation (2013) Return on Investment in Large Scale
Research Infrastructure. Report for National Research Council Canada. Available at:
http://www.triumf.ca/sites/default/files/HAL-ReturnOnInvestmentStudy-May-
2013.pdf
Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau
Instituut. Available at: http://www.rathenau.nl/en/publications/publication/the-
societal-footprint-of-big-science.html
National Research Council Canada (2013) Return on Investment in Large Research
Infrastructure. Available at: http://www.triumf.ca/sites/default/files/HAL-
ReturnOnInvestmentStudy-May-2013.pdf
OECD (2014) International Distributed Research Infrastructures: Issues and Options.
Available at: http://www.oecd.org/sti/sci-tech/international-distributed-research-
infrastructures.pdf
OECD (2014) Report on the Impacts of Large Research Infrastructure on Economic
Innovation and on Society: Case Studies at CERN. Available at:
http://www.oecd.org/sti/sci-tech/CERN-case-studies.pdf
Evaluating the socio-economic impact of research infrastructure
© Technopolis Group, 2015 19
Rizutto, C. and Wood, J. (eds.) (2013) RAMIRI Handbook. Deliverable of FP7 project
Realising and Managing International Research Infrastructures. Available at:
http://www.ramiri-blog.eu/index.php?n=Main.HomePage
Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation
at ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31
May-1 June 2012, Hambourg
Sallee, C.M., Watkins, S.D and Rosaen, A.L. (2011) The Economic Impact of Fermi
National Accelerator Laboratory. Anderson Economic Group report to the University
of Chicago. Available at:
https://ovprnl.uchicago.edu/sites/research.uchicago.edu/files/Fermilab_Economic_I
mpact_Full_Study.pdf
Simmonds, P. et.al (2013) Big Science and Innovation. Technopolis Group report for
the UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/24
9715/bis-13-861-big-science-and-innovation.pdf
Wissenschaftsrat (2011) Recommendations on Research Infrastructures in
Humanities and Social Sciences. Available at:
http://www.wissenschaftsrat.de/download/archiv/10465-11_engl.pdf
Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research
Infrastructure Projects for the National Roadmap (Pilot Phase). Available at:
http://www.wissenschaftsrat.de/download/archiv/2841-13_engl.pdf
Zuijdam, F. et. al (2011) The role and added value of large-scale research facilities.
Technopolis Group report. Available at: http://www.technopolis-group.com/wp-
content/uploads/2011/02/1379_Report_Large-scale_Research_Facilities_EN1.pdf
Please cite this publication as:
Griniece E., A Reid and J. Angelis (2015): Guide to Evaluating and Monitoring Socio-
Economic Impact of Investment in Research Infrastructures. Technopolis Group.
Tallinn. Estonia
Front cover photo: Centre for New Pharmaceutical and Health Technologies of the
Santaka Valley, Kaunas, Lithuania. Source: http://www.mitnija.lt/
... Impacts are traced in terms of their distance from the funded activities. The model is depicting impacts that arise predominantly from single-sited RIs and covers impacts on economy, innovation, human resources and 87 See Meijer et al (2010) for details 88 89 Griniece et al. (2016). The basis for constructing impact pathways has been documentary evidence from available case studies and analytical reports, as well as hands-on experience interacting with and advising managers of ESIF funded RI projects in Lithuania in the period 2010-2014. ...
... While the logic model does not reflect in detail wider contextual factors that can influence the outcomes of investments in a RI, it does acknowledge the interlinked and cumulative effects that arise in the areas of human resources, skills formation and networking, as well as intricate mechanisms behind and between impacts on innovation and scientific impacts. Source: Griniece, Reid and Angelis (2016) The European FP7 RIFI (Research infrastructures: Foresight and impact) project, 90 can be considered an additional logic model that can be placed under the heading of the theory-based approaches. It started in 2009, ran for two years, and was motivated by the fact that Central-East, and South-East Europe did not play a significant role as host countries for RI. ...
... In this context, the RI-PATHS project 8 aims to provide policy makers, funders and RI managers the tools to assess RI impact on the economy and their contribution to resolving societal challenges, etc. The goal is to improve the understanding of the long-term impact pathways [3] of the various types of RIs operating in Europe, and, indeed, internationally. The project is being implemented over a 30-month period beginning in January 2018 and ending in June 2020. ...
Chapter
Full-text available
The Horizon 2020 funded RI-PATHS project aims at providing policy makers, funders and Research Infrastructure (RI) managers the tools to assess the impact on the RI on the economy and their contribution to resolving societal challenges, etc. The goal is to improve the understanding of the long-term impact pathways of the various types of RIs operating in Europe, and, indeed, internationally. In the following we briefly summarise the work carried out by the RI-PATHS consortium partners to mid-2019 and derive useful lessons for future work.
... Much of the select literature analysed made insightful observations about large-scale, technology-driven global infrastructure, such as those encompassed by the ESFRI programme, its distinct phases, and the varied evaluation needs associated with each phase [15][16][17]. In addition, there has been much discussion of how the context and type of RI affects impact assessments-for instance, whether RI is virtual or a single site, or for basic science or applied research [18]. ...
Article
Full-text available
Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a “pipeline” approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to “take up the mantle” and tackle the complexity of real-world research translation and evaluation.
... Besides, Griniece et al. (2015) conducted an assessment of the socio-economic impact of investment in research infrastructure. The paper assessed the research infrastructure in the operational phase based on aspects such as the impact on the economy, on the capacity of human resources, on the research activities, and society. ...
Article
Full-text available
Evaluating the socio-economic efficiency of scientific and technological topics/projects after acceptance is very difficult because it is not easy to find common ground on the criteria for evaluating the socio-economic efficiency of topics/ projects of different research fields. Hence, depending on researchers’ views, the evaluation criteria or the weight of evaluating the socio-economic efficiency of each topic/project should be different. This study uses the expert method and the Analytic Hierarchy Process (AHP) method to determine the criteria and to calculate weights for criteria on five scientific fields according to scientific sub-disciplines in Binh Dinh province, including agricultural science, natural science, social sciences and humanities, medical science, engineering and technological sciences. Our research shows that the evaluation of the socio-economic efficiency of the provincial topics/projects should be based on eight aspects, including science, technology, economics, environment, culture - society, information - management, and education. At the same time, the results also identified a set of common criteria for evaluating the socio-economic efficiency of topics/projects and suggested how to calculate points and to rank the effectiveness of topics/projects. However, the weight of the criteria and the composition of each criterion is different.
Article
Sustainability of research infrastructures (RIs) is a big challenge for funders, stakeholders and operators, and the development and adoption of adequate management tools is a major concern, namely tools for monitoring and evaluating their performance and impact. BioData.pt is the Portuguese Infrastructure of Biological and Portuguese node of the European Strategy Forum on Research Infrastructures "Landmark" ELIXIR. The foundations of this national research infrastructure were laid under the “Building BioData.pt” project, for four years. During this period, performance and impact indicators were collected and analysed under the light of international guidelines for assessing the performance and impact of European research infrastructures produced by the European Strategy Forum on Research Infrastructures, the Organisation for Economic Co-operation and Development and the EU-funded RI-PATHS project. The exercise shared herein showed that these frameworks can be adopted by national RIs, with the necessary adaptations, namely to reflect the national landscape and specificity of activities, and can be powerful tools in supporting the management of RIs. “Not everything that counts can be counted, and not everything that can be counted, counts”. (Attributed to William Bruce Cameron)
Article
Sustainability of research infrastructures (RIs) is a big challenge for funders, stakeholders and operators, and the development and adoption of adequate management tools is a major concern, namely tools for monitoring and evaluating their performance and impact. BioData.pt is the Portuguese Infrastructure of Biological and Portuguese node of the European Strategy Forum on Research Infrastructures "Landmark" ELIXIR. The foundations of this national research infrastructure were laid under the “Building BioData.pt” project, for four years. During this period, performance and impact indicators were collected and analysed under the light of international guidelines for assessing the performance and impact of European research infrastructures produced by the European Strategy Forum on Research Infrastructures, the Organisation for Economic Co-operation and Development and the EU-funded RI-PATHS project. The exercise shared herein showed that these frameworks can be adopted by national RIs, with the necessary adaptations, namely to reflect the national landscape and specificity of activities, and can be powerful tools in supporting the management of RIs. “Not everything that counts can be counted, and not everything that can be counted, counts”. Albert Einstein, Theoretical physicist and Nobel Prize winner
Book
Full-text available
The essays in this open access volume identify the key ingredients for success in capitalizing on public investments in scientific projects and the development of large-scale research infrastructures. Investment in science – whether in education and training or through public funding for developing new research tools and technologies – is a crucial priority. Authors from big research laboratories/organizations, funding agencies and academia discuss how investing in science can produce societal benefits as well as identifying future challenges for scientists and policy makers. The volume cites different ways to assess the socio-economic impact of Research Infrastructures and their role as hubs of global collaboration, creativity and innovation. It highlights the different benefits stemming from fundamental research at the local, national and global level, while also inviting us to rethink the notion of “benefit” in the 21st century. Public investment is required to maintain the pace of technological and scientific advancements over the next decades. Far from advocating a radical transformation and massive expansion in funding, the authors suggest ways for maintaining a strong foundation of science and research to ensure that we continue to benefit from the outputs. The volume draws inspiration from the first “Economics of Big Science” workshop, held in Brussels in 2019 with the aim of creating a new space for dialogue and interaction between representatives of Big Science organizations, policy makers and academia. It aspires to provide useful reading for policy makers, scientists and students of science, who are increasingly called upon to explain the value of fundamental research and adopt the language and logic of economics when engaging in policy discussions.
Chapter
Full-text available
Scientific discoveries can be classified as public goods. Arrow [1] discussed properties of knowledge that make it a public good; highlighting in particular, the fact that it cannot be depleted when shared, and once it is made public others cannot easily be excluded from its use. So, public good is a commodity or service that is provided without profit to all members of a society, either by the government or by a private individual or organization. Thus, a global public good is a public good that goes beyond borders, and CERN scientific output is the perfect example of a global public good.
Technical Report
Full-text available
This report presents the findings of the comparative impact study of the European Social Survey (ESS). It forms a work package of the Horizon 2020 project ‘ESS-SUSTAIN’, and was carried out by Technopolis, with bibliometric analysis undertaken by the Centre for Science and Technology Studies (CWTS). The study assesses the academic, non-academic and teaching impacts that have been achieved through the ESS, by all different user groups and in all current member/observer countries. It also assesses how these impacts came about (‘pathways’ to impact), identifies best practice, and makes recommendations to ensure the long-term sustainability of the ESS. The ESS is an international, comparative survey of social and political values and attitudes, which was launched in 2002 and is now in its 9th round of data collection. In 2013, it was given the status of a European Research Infrastructure Consortium (ERIC). The ESS ERIC currently has 16 member countries and one observer country. In total, 24 countries (including ‘guest’ countries) participated in the eighth round of data collection.
Article
Full-text available
Large-scale research facilities are among the most expensive items of scientific expenditure. Any decision to invest in the knowledge infrastructure should be based on sound evidence. There is, however, a lack of evidence on the nature and extent of the societal impacts of large-scale research facilities and on the mechanisms that generate such effects. We review the literature to determine whether or not and, if so, through which mechanisms large-scale research facilities produce economic and social impacts. The literature provides no direct, empirical evidence to show that such impacts actually occur around large-scale research facilities. There is insufficient evidence to support the claim that they attract and retain talent and promote innovation. More evidence exists that large infrastructures forge new networks and communities. Empirical research on large-scale research facilities and their effects on science, economy and society is sorely needed.
Article
My purpose is to review, synthesize and criticize the voluminous, multidisciplinary literature on technology transfer. To reduce the literature to manageable proportions, I focus chiefly (not exclusively) on recent literature on domestic technology transfer from universities and government laboratories. I begin by examining a set of fundamental conceptual issues, especially the ways in which the analytical ambiguities surrounding technology transfer concepts affect research and theory. My literature review follows and I emphasize technology transfer's impact and effectiveness. I employ a “Contingent Effectiveness Model of Technology Transfer” to organize the literature. As the model's name implies, it assumes that technology effectiveness can take a variety of forms. In addition to examining the more traditional effectiveness criteria- those rooted in market impacts- the model considers a number of alternative effectiveness criteria, including political effectiveness, capacity-building.
Innovation from Big Science: Enhancing Big Science Impact Agenda
  • E Autio
Autio, E. (2014) Innovation from Big Science: Enhancing Big Science Impact Agenda. Report to UK Department for Business, Innovation & Skills. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/28
Economic utility resulting from CERN contracts (second study) Report of the European Organisation for Nuclear Research (CERN) Available at: http://indico.cern.ch/event
  • M Bianchi-Streit
Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second study). Report of the European Organisation for Nuclear Research (CERN). Available at: http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
Project Preparation and CBA of RDI Infrastructure Projects
  • S Clarke
Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects. JASPERS Staff Working Paper. Available at: http://www.jaspersnetwork.org/display/for/Project+Preparation+and+CBA+of+RDI +Infrastructure+Projects
Available at: http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011 Analyses of FP7 supported Research Infrastructures initatives in the context of ERA. Available at
European Strategy Forum on Research Infrastructures (2011) Evaluation Report 2011. Available at: http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011.pdf Fotakis, C. (2010) Analyses of FP7 supported Research Infrastructures initatives in the context of ERA. Available at: http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence- base/experts_analysis/c.%20fotakis_-_research_infrastructure.pdf
Impacts of Large-Scale Research Facilities-A Socio-Economic Analysis. A study done at the Research Policy Institute Available at: http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R I
  • O Hallonsten
  • M Benner
  • E Holmberg
Hallonsten, O., Benner, M. and Holmberg. G (2004) Impacts of Large-Scale Research Facilities-A Socio-Economic Analysis. A study done at the Research Policy Institute, Lund University. Available at: http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R I.pdf von Hippel, E. (1976) The Dominant Role of Users in the Scientific Instrument Innovation Process, Research Policy 5 (3)
2013) RAMIRI Handbook. Deliverable of FP7 project Realising and Managing International Research Infrastructures
  • C Rizutto
  • J Wood
Rizutto, C. and Wood, J. (eds.) (2013) RAMIRI Handbook. Deliverable of FP7 project Realising and Managing International Research Infrastructures. Available at: http://www.ramiri-blog.eu/index.php?n=Main.HomePage
Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop "The Socio-Economic Relevance of Research Infrastructures
  • C Rizzuto
Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop "The Socio-Economic Relevance of Research Infrastructures", 31