Research Evaluation

Published by Oxford University Press (OUP)
Online ISSN: 1471-5449
Print ISSN: 0958-2029
Publications
SciTS cluster map Note: Two-dimensional map of the 95 final synthesized SciTS topic statements, grouped into seven clusters. Each numbered point represents one synthesized statement (a list of all statements organized by cluster is in Table 1). Statements closer to each other are considered to be more similar in meaning than statements further away from one another. The grouping (as defined by polygon-shaped boundaries) displays the statements into related clusters
SciTS concept map Note : A comprehensive SciTS issues map showing labeled clusters and regions. Synthesized SciTS topic statements (refer to Figure 1) are no longer shown as individual points; rather, they are now grouped and represented by clusters (7), and then by regions (4). The average importance rating for each cluster is displayed inside the clusters 
An increase in cross-disciplinary, collaborative team science initiatives over the last few decades has spurred interest by multiple stakeholder groups in empirical research on scientific teams, giving rise to an emergent field referred to as the science of team science (SciTS). This study employed a collaborative team science concept-mapping evaluation methodology to develop a comprehensive research agenda for the SciTS field. Its integrative mixed-methods approach combined group process with statistical analysis to derive a conceptual framework that identifies research areas of team science and their relative importance to the emerging SciTS field. The findings from this concept-mapping project constitute a lever for moving SciTS forward at theoretical, empirical, and translational levels.
 
Funders of biomedical research are often challenged to understand how a new funding initiative fits within the agency's portfolio and the larger research community. While traditional assessment relies on retrospective review by subject matter experts, it is now feasible to design portfolio assessment and gap analysis tools leveraging administrative and grant application data that can be used for early and continued analysis. We piloted such methods on the National Cancer Institute's Provocative Questions (PQ) initiative to address key questions regarding diversity of applicants; whether applicants were proposing new avenues of research; and whether grant applications were filling portfolio gaps. For the latter two questions, we defined measurements called focus shift and relevance, respectively, based on text similarity scoring. We demonstrate that two types of applicants were attracted by the PQs at rates greater than or on par with the general National Cancer Institute applicant pool: those with clinical degrees and new investigators. Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing. Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements. We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.
 
A conceptual model was developed to guide evaluation of the long-term impacts of research grant programs at the National Institutes of Health, National Institute of Environmental Health Sciences. The model was then applied to the extramural asthma research portfolio in two stages: (1) using extant data sources, (2) involving primary data collection with asthma researchers and individuals in positions to use asthma research in development of programs, policies and practices. Reporting on the second stage, this article describes how we sought to broaden the perspectives included in the assessment and obtain a more nuanced picture of research impacts by engaging those involved in conducting or using the research.
 
Collaboratively authored concept map of HIV/AIDS clinical trials network success factors Source: Kagan et al (2009)  
Days for protocol registration, non-US clinical research sites
Kaplan-Meier plots for protocols in US and non-US sites: days for protocol registration
New discoveries in basic science are creating extraordinary opportunities to design novel biomedical preventions and therapeutics for human disease. But the clinical evaluation of these new interventions is, in many instances, being hindered by a variety of legal, regulatory, policy and operational factors, few of which enhance research quality, the safety of study participants or research ethics. With the goal of helping increase the efficiency and effectiveness of clinical research, we have examined how the integration of utilization-focused evaluation with elements of business process modeling can reveal opportunities for systematic improvements in clinical research. Using data from the NIH global HIV/AIDS clinical trials networks, we analyzed the absolute and relative times required to traverse defined phases associated with specific activities within the clinical protocol lifecycle. Using simple median duration and Kaplan-Meyer survival analysis, we show how such time-based analyses can provide a rationale for the prioritization of research process analysis and re-engineering, as well as a means for statistically assessing the impact of policy modifications, resource utilization, re-engineered processes and best practices. Successfully applied, this approach can help researchers be more efficient in capitalizing on new science to speed the development of improved interventions for human disease.
 
Development of effective quantitative indicators and methodologies to assess the outcomes of cross-disciplinary collaborative initiatives has the potential to improve scientific program management and scientific output. This article highlights an example of a prospective evaluation that has been developed to monitor and improve progress of the National Cancer Institute Physical Sciences-Oncology Centers (PS-OC) program. Study data, including collaboration information, was captured through progress reports and compiled using the web-based analytic database: Interdisciplinary Team Reporting, Analysis, and Query Resource. Analysis of collaborations was further supported by data from the Thomson Reuters Web of Science database, MEDLINE database, and a web-based survey. Integration of novel and standard data sources was augmented by the development of automated methods to mine investigator pre-award publications, assign investigator disciplines, and distinguish cross-disciplinary publication content. The results highlight increases in cross-disciplinary authorship collaborations from pre- to post-award years among the primary investigators and confirm that a majority of cross-disciplinary collaborations have resulted in publications with cross-disciplinary content that rank in the top third of their field. With these evaluation data, PS-OC Program officials have provided ongoing feedback to participating investigators to improve center productivity and thereby facilitate a more successful initiative. Future analysis will continue to expand these methods and metrics to adapt to new advances in research evaluation and changes in the program.
 
Milestone timing (in months) for the 22 primary studies publications 
Distribution of all 1,429 citations from 2006–10 for the 22 primary studies publications.  
Publication dissemination landscape for the 22 primary studies publications.  
Differences in pace of citation for high–low AIS groups of the 22 primary studies publications.  
A select set of highly cited publications from the National Institutes of Health (NIH) HIV/AIDS Clinical Trials Networks was used to illustrate the integration of time interval and citation data, modeling the progression, dissemination, and uptake of primary research findings. Following a process marker approach, the pace of initial utilization of this research was measured as the time from trial conceptualization, development and implementation, through results dissemination and uptake. Compared to earlier studies of clinical research, findings suggest that select HIV/AIDS trial results are disseminated and utilized relatively rapidly. Time-based modeling of publication results as they meet specific citation milestones enabled the observation of points at which study results were present in the literature summarizing the evidence in the field. Evaluating the pace of clinical research, results dissemination, and knowledge uptake in synthesized literature can help establish realistic expectations for the time course of clinical trials research and their relative impact toward influencing clinical practice.
 
Evaluators of scientific research programs have several tools to document and analyze products of scientific research, but few tools exist for exploring and capturing the impacts of such research. Understanding impacts is beneficial because it fosters a greater sense of accountability and stewardship for federal research dollars. This article presents the High Impacts Tracking System (HITS), a new approach to documenting research impacts that is in development at the National Institute of Environmental Health Sciences (NIEHS). HITS is designed to help identify scientific advances in the NIEHS research portfolio as they emerge, and provide a robust data structure to capture those advances. We have downloaded previously un-searchable data from the central NIH grants database and developed a robust coding schema to help us track research products (going beyond publication counts to the content of publications) as well as research impacts. We describe the coding schema and key system features as well as several development challenges, including data integration, development of a final data structure from three separate ontologies, and ways to develop consensus about codes among program staff.
 
Peer-evaluation-based measures of group research quality such as the UK's Research Assessment Exercise (RAE), which do not employ bibliometric analyses, cannot directly avail of such methods to normalize research impact across disciplines. This is seen as a conspicuous flaw of such exercises and calls have been made to find a remedy. Here a simple, systematic solution is proposed based upon a mathematical model for the relationship between research quality and group quantity. This model manifests both the Matthew effect and a phenomenon akin to the Ringelmann effect and reveals the existence of two critical masses for each academic discipline: a lower value, below which groups are vulnerable, and an upper value beyond which the dependency of quality on quantity reduces and plateaus appear when the critical masses are large. A possible normalization procedure is then to pitch these plateaus at similar levels. We examine the consequences of this procedure at RAE for a multitude of academic disciplines, corresponding to a range of critical masses.
 
We introduce a measure of interdisciplinarity as the diversity of academic research production across scientific domains. Our dataset concerns more than 900 permanent researchers employed by a large French university which is ranked first among French universities in terms of impact. As expected we find that the traditional academic career incentives do not stimulate interdisciplinary research, while having connections with industry does. The context of work in the laboratory (size, colleagues' status, age and affiliations) strongly affects the propensity to undertake interdisciplinary research.
 
The paper provides summary statistics from the KEINS database on academic patenting in France, Italy, and Sweden. It shows that academic scientists in those countries have signed many more patents than previously estimated. This re-evaluation of academic patenting comes by considering all patents signed by academic scientists active in 2004, both those assigned to universities and the many more held by business companies, governmental organizations, and public laboratories. Specific institutional features of the university and research systems in the three countries contribute to explaining these ownership patterns, which are remarkably different from those observed in the USA. In the light of these new data, European universities' contribution to domestic patenting appears not to be much less intense than that of their US counterparts.
 
Different web entities (site, directory and page) in an open accessible web space.  
A log file sample showing the proposed access types search engine, backlink and direct access which can be differentiated by a heuristic.  
The ongoing paradigm change in the scholarly publication system makes it necessary to construct alternative evaluation criteria/metrics which appropriately take into account the unique characteristics of electronic publications and other research output in digital formats. Today, major parts of scholarly open access (OA) publications and the self-archiving area are not well covered in traditional citation and indexing databases. The growing share and importance of freely accessible research output demands new approaches/metrics for measuring and evaluating these new types of scientific publication. We propose a simple quantitative method which establishes indicators by measuring the access/download pattern of OA documents and other web entities of a single web server. The experimental indicators are constructed, based on standard local web usage data. This new type of web-based indicator is developed to model the specific demand for better study/evaluation of the accessibility, visibility and interlinking of open accessible documents. We conclude that escience will need new stable e-indicators.
 
This article introduces an agent-based simulation model representing the dynamic processes of cooperative R&D in the manufacturing sector of South Korea. Firms’ behaviors were defined according to empirical findings on a data set from the internationally standardized Korean Innovation Survey in 2005. Simulation algorithms and parameters were defined based on the determinants of firms’ likelihood to participate in cooperation with other firms when conducting innovation activities. The calibration process was conducted to the point where artificially generated scenarios were equivalent to the one observed in the real world. The aim of this simulation game was to create a basic implementation that could be extended to test different policies strategies in order to observe sector responses (including cross-sector spillovers) when promoting cooperative innovation. Based on the evaluation of simulated research collaboration data, sector responses to strategies concerning government intervention in R&D of the firms can now be assessed.
 
Does past performance influence success in grant applications? We tested whether the decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of applicants in publications and citations, and with the results of the Council's peer reviews. The Council proves successful in distinguishing grant applicants with above-average from below-average performance, but within the former group there was no correlation between past performance and receiving a grant. When comparing the best-performing researchers who were denied funding with those who received it, the rejected researchers significantly outperformed the funded ones. The best rejected proposals score on average as high on the outcomes of the peer-review process as the accepted proposals. The Council successfully corrected for gender effects during the selection process. We explain why these findings may apply beyond this case. However, if research councils are not able to select the ‘best’ researchers, perhaps they should reconsider their mission. We discuss the role of research councils in the science system in terms of variation, innovation and quality control.
 
How were excellent papers defined in the study (in percent)? 
As the subject of research excellence has received increasing attention (in science policy) over the last few decades, increasing numbers of bibliometric studies have been published dealing with excellent papers. However, many different methods have been used in these studies to identify excellent papers. The present quantitative analysis of the literature has been carried out in order to acquire an overview of these methods and an indication of an "average" or "most frequent" bibliometric practice. The search in the Web of Science yielded 321 papers dealing with "highly cited", "most cited", "top cited" and "most frequently cited". Of the 321 papers, 16 could not be used in this study. In around 80% of the papers analyzed in this study, a quantitative definition has been provided with which to identify excellent papers. With definitions which relate to an absolute number, either a certain number of top cited papers (58%) or papers with a minimum number of citations are selected (17%). Around 23% worked with percentile rank classes. Over these papers, there is an arithmetic average of the top 7.6% (arithmetic average) or of the top 3% (median). The top 1% is used most frequently in the papers, followed by the top 10%. With the thresholds presented in this study, in future, it will be possible to identify excellent papers based on an "average" or "most frequent" practice among bibliometricians.
 
A new approach to producing multidisciplinary lists of highly cited researchers is described and used for compiling the first multidisciplinary list of highly cited researchers. This approach is essentially related to the recently discovered law of the constant ratios, and gives a better-balanced representation of different scientific fields.
 
This paper contributes to the quest for an operational definition of ‘research excellence’ and proposes a translation of the excellence concept into a bibliometric indicator. Starting from a textual analysis of funding program calls aimed at individual researchers and from the challenges for an indicator at this level in particular, a new type of indicator is proposed. The impact vitality indicator (Rons and Amez, 2008) reflects the vitality of the impact of a researcher's publication output, based on the change in volume over time of the citing publications. The introduced metric is shown to possess attractive operational characteristics and meets a number of criteria which are desirable when comparing individual researchers. The validity of one of the possible indicator variants is tested using a small dataset of applicants for a senior full-time research fellowship. Options for further research involve testing various indicator variants on larger samples linked to different kinds of evaluations.
 
Percentage of publications with local focus over time 
Percentage of local and non-local papers with a Colombian address by degree of interdisciplinarity (Rao-Stirling diversity) 
This paper examines the role of interdisciplinarity on research pertaining to local issues. Using Colombian publications from 1991 until 2011 in the Web of Science, we investigate the relationship between the degree of interdisciplinarity and the local orientation of the articles. We find that a higher degree of interdisciplinarity in a publication is associated with a greater emphasis on local issues. In particular, our results support the view that research that combines cognitively disparate disciplines, what we refer to as distal interdisciplinarity, is associated with more local focus of research. We discuss the policy implications of these results in the context of national research assessments targeting excellence and socio-economic impact.
 
Allocation of research funding, as well as promotion and tenure decisions, are increasingly made using indicators and impact factors drawn from citations to published work. A debate among scientometricians about proper normalization of citation counts has resolved with the creation of an Integrated Impact Indicator (I3) that solves a number of problems found among previously used indicators. The I3 applies non-parametric statistics using percentiles, allowing highly-cited papers to be weighted more than less-cited ones. It further allows unbundling of venues (i.e., journals or databases) at the article level. Measures at the article level can be re-aggregated in terms of units of evaluation. At the venue level, the I3 creates a properly weighted alternative to the journal impact factor. I3 has the added advantage of enabling and quantifying classifications such as the six percentile rank classes used by the National Science Board's Science & Engineering Indicators.
 
This paper describes the method for ex-post peer-review evaluation per research discipline used at the Vrije Universiteit Brussel, and the outcomes obtained. Pertinent advice and responses at different levels benefit research quality, competitivity and visibility. Imposed reflection and contacts modify the researcher's attitude and improve team strategies. Deeper insights and data sets on research disciplines and extracted general recommendations support the university management's policy decisions, instruments and guidelines. Comparisons with other assessments lead to a better understanding of possibilities and limitations of different evaluation processes. The described peer-review method can be applied systematically, yielding a complete overview, or on an ad hoc basis for a particular discipline, based on demands from research teams or on strategic or policy arguments.
 
Canada has a service economy and R&D in Canada is mainly a service sector activity. This paper examines the sectoral distribution of expenditure on R&D performance, with emphasis on the business sector in Canada and with international comparisons. Human resources are a key component in the performance of R&D, and comparisons are made, overtime, of the number of research workers in service and non-service industries, of the ratio of professional to technical and other personnel, and of the changes in educational levels of R&D personnel. Using Canadian experience as a guide, some conclusions are drawn about the measurement challenges in producing indicators of the transition to a service economy.
 
Distribution of academic researchers by network size  
Participation in collaborative grants by type of partner and discipline
Average network size by discipline
Summary statistics
We study the patterns of engagement in collaborative research among university researchers. We investigate how frequently researchers engage in collaborative research with third parties, as well as the factors that affect the probability of interacting with industry as compared to other types of partners. By focusing on a large sample of recipients of collaborative research grants, we examine how environmental and individual characteristics impact on both the size of the network set up by academic researchers, and the type of partner chosen. Our results demonstrate that individual characteristics matter both for the size and the type of the network, while department-level characteristics are particularly important for the type of the network set up by the researcher.
 
Effects of patenting on research performance: results from random effects panel tobit regression
The growing importance of technology-relevant non-publication output of university research has come into the focus of policy-makers' interest. A fierce debate arose on possible negative consequences of the increasing commercialization of science, as it may come with a reduction in research performance. This paper investigates the relationship between publishing as a measure of scientific output and patenting for German professors active in a range of science fields. We combine bibliometric/technometric indicators and econometric techniques to show that patenting positively correlates with the publication output and quality of patenting researchers.
 
This paper describes the development of research evaluation in Spain. It assumes that research evaluation, R&D policy and programme evaluation are embedded in the development of and R&D system and are characterised by general Spanish policy-making. Research evaluation in a context of delegation and as a self-organising system for research actors guaranteed by the state, has been strongly developed in the last few years: R&D policy and programmme evaluation in less institutionalised. The explanation is linked to the sequence of reforms of the R&D system and to the set up of the first Spanish science and technology policy. The support of the European Commission is acknowledged (HCM contract CHRX-CT93-0240) and the Spanish National R&D Plan (projects SEC 93-0688 and SEC 94-0796).
 
The overarching aim of this paper is to contribute to a better understanding of how the main target variable of innovation policy – change in behaviour – can be better conceptualized and put into practice in evaluation and policy making. The paper first develops a theoretical framework of the concept of behavioural additionality. On that basis it looks in detail at the way behavioural additionality is operationalized in evaluation practice and how the concept is discussed and applied in the interaction between policy-makers and evaluators. The paper utilizes a statistical analysis of 171 innovation policy evaluations, a text analysis of selected behavioural additionality evaluation reports and finally a number of in-depth case studies of evaluations. Based on the theoretical considerations and the empirical findings, the paper identifies three different uses of behavioural additionality in innovation policy evaluations. It concludes that despite the widespread use of the behavioural additionality concept, an improved theoretical basis and serious methodological improvements are needed to realize the full potential of this concept for evaluation and policy practice.Those who are interested in this article or the underlying study are welcome to contact the authors.
 
Annual growth in labour productivity in 1990-97 and manufacturing share of GDP.
Estimated elasticity of productivity growth with respect to innovation output.
A broad definition of innovation input is used, in which R&D is one of several sources of innovation. A quantitative innovation output measure is used in the analysis, which is based on a large representative sample of firms, including small firms. An econometric framework based on the knowledge-production function accounting for both selectivity and simultaneity bias is employed. The results from Nordic countries show that, given difficulties in pooling the data, it is important to identify country-specific models to account for country-specific effects and differences in countries' national innovation systems.
 
This paper borrows analytical tools from the study of public policy, applying them to issues surrounding the development of universities' ‘Third Mission’ indicators. The ‘Third Mission’ refers to all activities concerned with the generation, use, application and exploitation of knowledge and other university capabilities outside academic environments. The paper analyses the generation and management of Third Mission indicators as an example of policy implementation. Using the cases of the UK and Spain, it finds a high level of ambiguity in the definition of policy goals accompanied by conflict among policy stakeholders. In these cases, policy analysts have suggested that policy implementation becomes ‘symbolic’, and depends on the strength of local stakeholder coalitions. We find that the development of Third Mission indicators fits this model, and conclude by discussing the policy implications of this finding.
 
‘Impact factor (IF)’ has been practically the only indicator to assess journal quality. However, it has various problems associated with citation analysis, such as the effects of ‘different sizes of audience’ and ‘biased citation’. To overcome this, we here propose a new objective index, ‘perspective factor (PF)’, which estimates the journal quality independently of citation analysis. The relationship between IF and PF of life science journals published in 1997, for example, gives a positive correlation when we excluded review journals and extremely high-IF journals, which could not gain comparatively high PF values.
 
CHANGES IN CAPABILITIES AND COMPETENCIES FOR TECHNOLOGICAL DEVELOPMENT AND INNOVATION
CHANGES IN THE MARKET FOR SERVICES
CLUSTERS LOCATED IN PRODUCER SUB-SAMPLE
This article presents a methodological proposal and an empirical validation for the assessment of ST&I programs, particularly for choosing indicators and metrics from a multidimensional perspective. Its key feature is the construction of indicators designed to apprehend the wide array of characteristics displayed by the many different types of innovation program. Thus it combines deductive procedures (decomposition of goals) and inductive procedures (discussion with the players involved) with evaluation results and impacts in specific dimensions linked to the nature of a given program. We illustrate the approach using a World Bank Program called INCAGRO. The proposed method is a contribution to systematizing the definition of indicators and metrics for the assessment of ST&I programs. It helps define ‘what should be evaluated’ in a given program. Many different (unforeseen) results and impacts can emerge simultaneously with the main subject of the program. Thus multidimensional issues and multiple actors should be considered in the impact evaluation process.
 
This paper considers the role that the journals system plays in supporting scientific productivity: a critical issue for publishers, librarians and research managers at a time of considerable upheaval in scholarly communication, notably the rise of open access models. The survey findings presented here support the unfashionable view that reader open access is something of a non-issue for the two author communities studied: corresponding authors in immunology and microbiology. Other issues, notably how financial and human resources are organised and deployed appear to be much more serious issues for biomedical researchers. The key policy implication of this work is that Europe could improve its research performance by listening more carefully to its scientists and harnessing those resources more effectively.
 
The paper argues that studies of the social role of basic research and its contribution to innovation and technological development in the commercial sector tend to overemphasise the propositional knowledge it produces. Differentiating this propositional knowledge from underlying, generic problem-solving procedures, it is suggested that the latter are socially useful beyond the scientific process. Their universal nature makes them important inputs for the R&D activities of innovative firms. Due to the natural and institutional limits on the transferability of instrumental knowledge, the paper suggests that an evaluation of the social value of basic research must focus on the career trajectories of scientists who migrated from basic research institutions into the commercial sector.
 
,°' there are several ways in which
Technology foresight is a process for bringing together scientists, industrialists, government officials and others to identify the areas of strategic research and the emerging technologies likely to yield the greatest economic and social benefits. The experiences with technology foresight in six countries are summarised. A conceptual model of the foresight process is proposed, and some of the factors structuring success and failure in foresight are analysed.
 
Disciplinary composition among interviewees per interdisciplinary fellowship competition
This paper examines how panelists serving on interdisciplinary funding panels produce evaluations they perceive as fair, drawing on 81 interviews with panelists serving on multidisciplinary fellowship competitions. We identify how peer reviewers define “good” interdisciplinary proposals and the rules they follow: respect for disciplinary sovereignty, deference to expertise and methodological pluralism. These rules ensure the preponderance of the voices of experts over non-experts. Panelists also adopt strategies to make other reviewers who lack expertise trust that their judgments are disinterested and unbiased, while reviewers who lack expertise are not afraid to make decisions based on idiosyncratic tastes rather than substantive quality.
 
This paper examines the determinants of researcher contribution-spans. The contributionspan is the number of years spanning a researcher's first and last known publications in a field. It serves as a unique measure of researchers' persistence. Based on co-authorship data, several sociometric indices are created and their impact on researchers' persistence in their efforts to develop a technology is examined. Evidence is provided from 2,876 researchers active in the field of transgene plants over 11 years. The findings lend support to the proposition that an individual researcher's network position is an important determinant of persistence.
 
Papers co-authored by Swedish and Italian scientists in 1996-97 and 1998-99 respectively in the CC editions "Life Sciences", "Physical, Chemical & Earth Sciences", "Engineering, Computing & Technology" and "Agriculture, Biology & Environmental Sciences" . The figure under Sweden (or Italy) indicates the total number of papers in the four years with at least one Author from that nation, and the figure under Coll the number of co-authored papers.  
Homogeneous and easy to update scientometric indicators for a wide spectrum of problems were obtained by making use of the bibliographic database Current Contents. The procedures adopted and the major factors which can affect the accuracy of the results are reported and discussed. Case studies are presented concerning: international co-operation, foreign policy, and vocation for industrial settlements, all of which provide examples of how these indicators can comply with a variety of needs.
 
Lists of books about the history of medicine were compiled from book reviews and from the references in papers in the subject taken from the Social Sciences Citation Index. The authors' addresses were sought from papers in this database and thereby some of the books could be attributed to individual countries, such as the UK and the USA, which were to be compared. Counts of citations and of reviews to individual books both showed that the UK was a very strong performer in this subject, and that its standing relatively had increased over the last few decades. This judgement reinforced the results of interviews carried out overseas with senior historians of medicine and gave them extra credibility.
 
Basic schema of decomposition method Source : Salles-Filho et al (2008) 
Types of result from PIPE Phase I projects Source : field survey (2007) Note : n = 50 
Types of result from PIPE Phase II Source : field survey (2007) Note : n = 50 
Differences between PIPE and SBIR
Were PIPE project results converted into innovations? Source : field survey (2007) Note : n = 145 
Evaluation of research results and impacts is a topic of growing interest to public and private organizations worldwide. Indeed, it can be said that evaluation initiatives are a top priority for many R&D institutions if they are to assure social legitimation. Assuming that the innovation process is uncertain, complex, linked to social and technical imponderables, and capable of taking a different course from what is planned, the goals of evaluation almost always become moving targets. The objective of this article is twofold: a) to present a methodological proposal for the evaluation of STR&I programs, particularly to allow multidimensional approaches in order to evaluate the efficiency and effectiveness of STR&I programs; and b) to present an application of this methodology to a Brazilian public program that aims to foster the emergence of innovative small firms, and compare it with some findings for a similar US program.
 
Mean Normalized Citation Score indicators at the field level for the three geographical areas (MNCS k f ). Articles published in 1998–2002 with a  
U.S./EU and RW/EU gaps according to the Mean Normalized Citation Score indicators at the field level. Articles published in 1998–2002 with a five-year citation window.  
Comparison between geographical areas at the discipline level according to the MNCS indicator. U.S. versus EU
This paper uses high- and low-impact citation indicators for the evaluation of the citation performance of research units at different aggregate levels. To solve the problem of the assignment of individual articles to multiple sub-fields, it follows a multiplicative strategy according to which each paper is wholly counted as many times as necessary in the several categories to which it is assigned at each aggregation level. To control for wide differences in citation practices at the lowest level of aggregation, we apply a novel sub-field normalization procedure in the multiplicative case. The methodology is applied to a partition of the world into three geographical areas: the U.S., the European Union (EU), and the Rest of the World. The main findings are the following two. (i) Although normalization does not systematically bias the results against any area at lower aggregate levels, it reduces the U.S./EU high-impact gap in the all-sciences case by a non-negligible 14.4%. (ii) The dominance of the U.S. over the EU in the basic and applied research published in the periodical literature is almost universal at all aggregation levels. From the high-impact perspective, for example, the U.S. is ahead of the EU in 77 out of 80 disciplines, and all of 20 fields. For all sciences as a whole, the U.S. high-impact indicator is 61% greater than that of the EU.
 
The pace of evolution of a technology cluster is determined by a variety of endogenous and exogenous factors related to its local and global environment. Extensively studied over the years, the factors that influence the growth of technology-based clusters can provide valuable indications about the capacity of clusters to achieve their objectives, attract new firms and become internationally competitive. Following an extensive literature review, the most prevalent factors that influence technology-based cluster growth were identified. These factors are those that must normally be present in order for a cluster to exist and to progress over time. This article outlines the method used to identify 12 recurring growth factors and discusses the use of these in the evaluation of technology-based clusters.
 
Interdisciplinarity is analyzed in three different research areas: pharmacology and pharmacy, cardiovascular system and materials science, using data from a survey to Spanish scientists. The study focuses on three complementary dimensions: diversity in personal training and research specialization of scientists; research practices and behavior of the groups; and the cognitive inputs and outputs of the research activity. Interdisciplinarity emerges as a double-edged process: of jumping into a new area with people of different disciplines, in coherence with the traditional disciplinary research teams; and of specialization in a field traditionally dominated by a single group of disciplinary backgrounds, in which researchers from different areas join the teams. Thus specialization-fragmentation-hybridization all come together.
 
In a growing number of countries, governments and public agencies seek to systematically assess the scientific outputs of their universities and research institutions. Bibliometrics indicators and peer review are regularly used for this purpose, and their advantages and biases are discussed in a wide range of literature. This article examines how three different national organisations produce journal ratings as an alternative assessment tool, which is particularly targeted for social sciences and humanities. After setting out the organisational context in which these journal ratings emerged, the analysis highlights the main steps of their production, the criticism they received after publication, especially from journals, and the changes made during the ensuing revision process. The particular tensions of a tool designed as both a political instrument and a scientific apparatus are also discussed.
 
The road between a discovery generated from basic research to a commercial product or process is long and, according to some, rife with significant roadblocks. Innovators and investors alike routinely claim that a 'funding gap' or 'Valley of Death' exists between basic research and commercialization of a new product. We show that the standard explanations for underinvestment in R&D are not the cause of this phenomenon. Rather, the Valley of Death occurs only in the presence of 'non-economic' investments (such as government expenditures on basic research) that are made in very early stage research without sufficient attention to the likely investment decisions at later stages of the innovation process. Other implications for the Valley of Death of government funding of R&D are also considered. Some policy implications of these findings are provided.
 
Some diverse indicators used to measure the innovation process are considered. They include those with an aggregate, and often national, focus, and rely on data from scientific publications, patents and R&D expenditures, etc. Others have a firm-level perspective, relying primarily on surveys or case studies. Also included are indicators derived from specialized databases, or consensual agreements reached through foresight exercises. There is an obvious need for greater integration of the various approaches to capture more effectively the richness of available data and better reflect the reality of innovation. The focus for such integration could be in the area of technology strategy, which integrates the diverse scientific, technological, and innovation activities of firms within their operating environments; improved capacity to measure it has implications for policy-makers, managers and researchers.
 
CSIC human and financial resources
CSIC research units by region and integration into a service centre
Length of CSIC Presidencies by party in government
Determinants of the number of CSIC patents by research unit and year
While some studies on patenting by public research organisations (PROs) and universities together tend to be positive and descriptive, normative concerns have risen mainly on the side of university patenting. The specific dynamics of PROs allow them to make strategic considerations which are less present in universities but which may have an impact on patenting. However, PROs are often subject to similar requirements and legal frameworks to increase patenting, so the question of which factors have a determining influence becomes relevant. Without aiming to build a typology, we focus on large PROs that face singular decisions on the priority of scientific areas, decentralisation to regions and joint management of their research units and we take the Spanish National Research Council (CSIC), the largest PRO in Spain, as a case study.
 
Summarising observations gathered in earlier studies, the effects of EU Structural Funds on regional research, technological development and innovation (RTDI) activities as well as on regional innovation policies are investigated. It is by no means clear that the subsidies granted for RTDI through the Structural Funds have resulted in the emergence of innovations. In order to do so, more effort should be devoted to the upgrading and creation of RTDI capacities, particularly among SMEs, and to fostering technology transfer. Nevertheless, one of the major benefits attributable to the Funds is strategic thinking among regional players. Strategic discussions have helped to optimise technological assets and to avoid spreading resources either too thinly or too broadly.
 
Labour markets for research are changing and the traditional segmentation model of the research labour market where the doctorate was mainly valuable in the academic sector is loosing ground. The paper studies a sample of PhDs and their corresponding employing firms to analyse patterns of mobility, economic returns, and innovation outputs. Qualitative and quantitative indicators are combined to tackle two sets of general questions: The first relates to the incentives for doctorate holders to pursue a company career versus an academic career. The second concerns the flexibility and/or reversibility of career options for young PhDs and the relative value of a doctorate outside academia. The results question the idea that the labour market for PhDs is tightly segmented and highlights the complementarity of PhDs' individual competencies and collective capabilities in the assessment of innovation outputs. They also demonstrate that economic returns are significantly different by gender
 
Using an Italian questionnaire not specifically designed to study geographic phenomena, the potential and limitations of the régionalisation of innovation information is analysed. There were difficulties involved in introducing the territorial dimension into the existing questionnaire and proposals are made for incremental amendments to the survey method for collecting and presenting the data, aimed at capturing the regional interdependencies. These include a request for the location of the main source of information, the broad geographic area from which technology has been acquired or transferred, and the sector and region of origin of the innovation.
 
This paper characterizes and analyses evaluation practice in national innovation policy across Europe. It is the first study that examines and interprets the characteristics, quality, usefulness, and consequences of evaluations in a systematic way. The analysis is based on the comprehensive INNO-Appraisal repository of 171 evaluation reports of national innovation policies of EU25 countries, conducted between 2002 and 2007. The paper seeks to assess the state of the art of evaluation in innovation policy at national level, to understand how different key dimensions of evaluation (timing, purpose, methods, tendering process, etc.) relate to each other, and to explore types of evaluations. On that basis, we draw lessons as to what constitutes good practice in evaluation, as the results of the survey have been exchanged and discussed with a number of policy makers of the sample responsible for the evaluation. The paper thus both contributes to the academic understanding of policy evaluation and supports use in policy practice.Those who are interested in this article or the underlying study are welcome to contact the authors.
 
Policy-related functions of foresight Source: adapted from Eriksson and Weber (2008)
A framework to classify the impacts of foresight activities
International foresight activities and their evaluations
Foresight has evolved as a distinct prospective analytical tool: it considers alternative futures of various S&T fields or socio-economic systems by bringing together the perspectives of various stakeholder groups, and thus assists the decision-making processes at different levels. However, in order to avoid hypes — and subsequent disappointments — about what foresight can deliver, the potential contributions to decision-making processes by foresight should be clearly understood. The article puts foresight into this broader context of policy-making processes, with a particular emphasis on innovation policy. It describes the evolution of different policy rationales since the 1960s, develops a framework to classify the impacts of various types of prospective analyses, and reviews the evaluation results of several national foresight programmes by using this framework. On that basis, future directions of how foresight might evolve are considered to spur discussions.
 
In recent years, there has been a growing view that universities could and should play a larger and more direct role in assisting industry and promoting national competitiveness. This review of the literature on university patenting and licensing activity is based on 125 papers published between 1980 and 2004, which were obtained by querying ABI/INFORM and EconLit using as keywords “university”, “patent”, “license”, “Bayh-Dole”, “triple helix” and by an abstract-by-abstract reading of all issues of 15 scientific journals and the NBER working papers database since 1995. Major findings include that the surge of university patents did not happen at the expense of their quality, nor of the quality of research. Moreover, scientific excellence and technology transfer activities mutually reinforce. Finally, university patenting and related activities need a fertile context to develop both inside and outside the campus. The US success story cannot be imitated by simply changing IP laws and by transferring ownership of IPRs from the inventors to the performing institutions.
 
Top-cited authors
Steven E. Stemler
  • Wesleyan University
Donald Sharpe
  • University of Regina
Teresa Penfield
  • University of Exeter
Rosa Scoble
  • Brunel University London
Michael C. Wykes
  • University of Exeter