Research Evaluation

An increase in cross-disciplinary, collaborative team science initiatives over the last few decades has spurred interest by multiple stakeholder groups in empirical research on scientific teams, giving rise to an emergent field referred to as the science of team science (SciTS). This study employed a collaborative team science concept-mapping evaluation methodology to develop a comprehensive research agenda for the SciTS field. Its integrative mixed-methods approach combined group process with statistical analysis to derive a conceptual framework that identifies research areas of team science and their relative importance to the emerging SciTS field. The findings from this concept-mapping project constitute a lever for moving SciTS forward at theoretical, empirical, and translational levels.

Funders of biomedical research are often challenged to understand how a new funding initiative fits within the agency's portfolio and the larger research community. While traditional assessment relies on retrospective review by subject matter experts, it is now feasible to design portfolio assessment and gap analysis tools leveraging administrative and grant application data that can be used for early and continued analysis. We piloted such methods on the National Cancer Institute's Provocative Questions (PQ) initiative to address key questions regarding diversity of applicants; whether applicants were proposing new avenues of research; and whether grant applications were filling portfolio gaps. For the latter two questions, we defined measurements called focus shift and relevance, respectively, based on text similarity scoring. We demonstrate that two types of applicants were attracted by the PQs at rates greater than or on par with the general National Cancer Institute applicant pool: those with clinical degrees and new investigators. Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing. Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements. We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.

A conceptual model was developed to guide evaluation of the long-term impacts of research grant programs at the National Institutes of Health, National Institute of Environmental Health Sciences. The model was then applied to the extramural asthma research portfolio in two stages: (1) using extant data sources, (2) involving primary data collection with asthma researchers and individuals in positions to use asthma research in development of programs, policies and practices. Reporting on the second stage, this article describes how we sought to broaden the perspectives included in the assessment and obtain a more nuanced picture of research impacts by engaging those involved in conducting or using the research.

New discoveries in basic science are creating extraordinary opportunities to design novel biomedical preventions and therapeutics for human disease. But the clinical evaluation of these new interventions is, in many instances, being hindered by a variety of legal, regulatory, policy and operational factors, few of which enhance research quality, the safety of study participants or research ethics. With the goal of helping increase the efficiency and effectiveness of clinical research, we have examined how the integration of utilization-focused evaluation with elements of business process modeling can reveal opportunities for systematic improvements in clinical research. Using data from the NIH global HIV/AIDS clinical trials networks, we analyzed the absolute and relative times required to traverse defined phases associated with specific activities within the clinical protocol lifecycle. Using simple median duration and Kaplan-Meyer survival analysis, we show how such time-based analyses can provide a rationale for the prioritization of research process analysis and re-engineering, as well as a means for statistically assessing the impact of policy modifications, resource utilization, re-engineered processes and best practices. Successfully applied, this approach can help researchers be more efficient in capitalizing on new science to speed the development of improved interventions for human disease.

Development of effective quantitative indicators and methodologies to assess the outcomes of cross-disciplinary collaborative initiatives has the potential to improve scientific program management and scientific output. This article highlights an example of a prospective evaluation that has been developed to monitor and improve progress of the National Cancer Institute Physical Sciences-Oncology Centers (PS-OC) program. Study data, including collaboration information, was captured through progress reports and compiled using the web-based analytic database: Interdisciplinary Team Reporting, Analysis, and Query Resource. Analysis of collaborations was further supported by data from the Thomson Reuters Web of Science database, MEDLINE database, and a web-based survey. Integration of novel and standard data sources was augmented by the development of automated methods to mine investigator pre-award publications, assign investigator disciplines, and distinguish cross-disciplinary publication content. The results highlight increases in cross-disciplinary authorship collaborations from pre- to post-award years among the primary investigators and confirm that a majority of cross-disciplinary collaborations have resulted in publications with cross-disciplinary content that rank in the top third of their field. With these evaluation data, PS-OC Program officials have provided ongoing feedback to participating investigators to improve center productivity and thereby facilitate a more successful initiative. Future analysis will continue to expand these methods and metrics to adapt to new advances in research evaluation and changes in the program.

A select set of highly cited publications from the National Institutes of Health (NIH) HIV/AIDS Clinical Trials Networks was used to illustrate the integration of time interval and citation data, modeling the progression, dissemination, and uptake of primary research findings. Following a process marker approach, the pace of initial utilization of this research was measured as the time from trial conceptualization, development and implementation, through results dissemination and uptake. Compared to earlier studies of clinical research, findings suggest that select HIV/AIDS trial results are disseminated and utilized relatively rapidly. Time-based modeling of publication results as they meet specific citation milestones enabled the observation of points at which study results were present in the literature summarizing the evidence in the field. Evaluating the pace of clinical research, results dissemination, and knowledge uptake in synthesized literature can help establish realistic expectations for the time course of clinical trials research and their relative impact toward influencing clinical practice.

Evaluators of scientific research programs have several tools to document and analyze products of scientific research, but few tools exist for exploring and capturing the impacts of such research. Understanding impacts is beneficial because it fosters a greater sense of accountability and stewardship for federal research dollars. This article presents the High Impacts Tracking System (HITS), a new approach to documenting research impacts that is in development at the National Institute of Environmental Health Sciences (NIEHS). HITS is designed to help identify scientific advances in the NIEHS research portfolio as they emerge, and provide a robust data structure to capture those advances. We have downloaded previously un-searchable data from the central NIH grants database and developed a robust coding schema to help us track research products (going beyond publication counts to the content of publications) as well as research impacts. We describe the coding schema and key system features as well as several development challenges, including data integration, development of a final data structure from three separate ontologies, and ways to develop consensus about codes among program staff.

Peer-evaluation-based measures of group research quality such as the UK's Research Assessment Exercise (RAE), which do not employ bibliometric analyses, cannot directly avail of such methods to normalize research impact across disciplines. This is seen as a conspicuous flaw of such exercises and calls have been made to find a remedy. Here a simple, systematic solution is proposed based upon a mathematical model for the relationship between research quality and group quantity. This model manifests both the Matthew effect and a phenomenon akin to the Ringelmann effect and reveals the existence of two critical masses for each academic discipline: a lower value, below which groups are vulnerable, and an upper value beyond which the dependency of quality on quantity reduces and plateaus appear when the critical masses are large. A possible normalization procedure is then to pitch these plateaus at similar levels. We examine the consequences of this procedure at RAE for a multitude of academic disciplines, corresponding to a range of critical masses.

We introduce a measure of interdisciplinarity as the diversity of academic research production across scientific domains. Our dataset concerns more than 900 permanent researchers employed by a large French university which is ranked first among French universities in terms of impact. As expected we find that the traditional academic career incentives do not stimulate interdisciplinary research, while having connections with industry does. The context of work in the laboratory (size, colleagues' status, age and affiliations) strongly affects the propensity to undertake interdisciplinary research.

The paper provides summary statistics from the KEINS database on academic patenting in France, Italy, and Sweden. It shows that academic scientists in those countries have signed many more patents than previously estimated. This re-evaluation of academic patenting comes by considering all patents signed by academic scientists active in 2004, both those assigned to universities and the many more held by business companies, governmental organizations, and public laboratories. Specific institutional features of the university and research systems in the three countries contribute to explaining these ownership patterns, which are remarkably different from those observed in the USA. In the light of these new data, European universities' contribution to domestic patenting appears not to be much less intense than that of their US counterparts.

The ongoing paradigm change in the scholarly publication system makes it necessary to construct alternative evaluation criteria/metrics which appropriately take into account the unique characteristics of electronic publications and other research output in digital formats. Today, major parts of scholarly open access (OA) publications and the self-archiving area are not well covered in traditional citation and indexing databases. The growing share and importance of freely accessible research output demands new approaches/metrics for measuring and evaluating these new types of scientific publication. We propose a simple quantitative method which establishes indicators by measuring the access/download pattern of OA documents and other web entities of a single web server. The experimental indicators are constructed, based on standard local web usage data. This new type of web-based indicator is developed to model the specific demand for better study/evaluation of the accessibility, visibility and interlinking of open accessible documents. We conclude that escience will need new stable e-indicators.

This article introduces an agent-based simulation model representing the dynamic processes of cooperative R&D in the manufacturing sector of South Korea. Firms’ behaviors were defined according to empirical findings on a data set from the internationally standardized Korean Innovation Survey in 2005. Simulation algorithms and parameters were defined based on the determinants of firms’ likelihood to participate in cooperation with other firms when conducting innovation activities. The calibration process was conducted to the point where artificially generated scenarios were equivalent to the one observed in the real world. The aim of this simulation game was to create a basic implementation that could be extended to test different policies strategies in order to observe sector responses (including cross-sector spillovers) when promoting cooperative innovation. Based on the evaluation of simulated research collaboration data, sector responses to strategies concerning government intervention in R&D of the firms can now be assessed.

Does past performance influence success in grant applications? We tested whether the decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of applicants in publications and citations, and with the results of the Council's peer reviews. The Council proves successful in distinguishing grant applicants with above-average from below-average performance, but within the former group there was no correlation between past performance and receiving a grant. When comparing the best-performing researchers who were denied funding with those who received it, the rejected researchers significantly outperformed the funded ones. The best rejected proposals score on average as high on the outcomes of the peer-review process as the accepted proposals. The Council successfully corrected for gender effects during the selection process. We explain why these findings may apply beyond this case. However, if research councils are not able to select the ‘best’ researchers, perhaps they should reconsider their mission. We discuss the role of research councils in the science system in terms of variation, innovation and quality control.

As the subject of research excellence has received increasing attention (in science policy) over the last few decades, increasing numbers of bibliometric studies have been published dealing with excellent papers. However, many different methods have been used in these studies to identify excellent papers. The present quantitative analysis of the literature has been carried out in order to acquire an overview of these methods and an indication of an "average" or "most frequent" bibliometric practice. The search in the Web of Science yielded 321 papers dealing with "highly cited", "most cited", "top cited" and "most frequently cited". Of the 321 papers, 16 could not be used in this study. In around 80% of the papers analyzed in this study, a quantitative definition has been provided with which to identify excellent papers. With definitions which relate to an absolute number, either a certain number of top cited papers (58%) or papers with a minimum number of citations are selected (17%). Around 23% worked with percentile rank classes. Over these papers, there is an arithmetic average of the top 7.6% (arithmetic average) or of the top 3% (median). The top 1% is used most frequently in the papers, followed by the top 10%. With the thresholds presented in this study, in future, it will be possible to identify excellent papers based on an "average" or "most frequent" practice among bibliometricians.

A new approach to producing multidisciplinary lists of highly cited researchers is described and used for compiling the first multidisciplinary list of highly cited researchers. This approach is essentially related to the recently discovered law of the constant ratios, and gives a better-balanced representation of different scientific fields.

This paper contributes to the quest for an operational definition of ‘research excellence’ and proposes a translation of the excellence concept into a bibliometric indicator. Starting from a textual analysis of funding program calls aimed at individual researchers and from the challenges for an indicator at this level in particular, a new type of indicator is proposed. The impact vitality indicator (Rons and Amez, 2008) reflects the vitality of the impact of a researcher's publication output, based on the change in volume over time of the citing publications. The introduced metric is shown to possess attractive operational characteristics and meets a number of criteria which are desirable when comparing individual researchers. The validity of one of the possible indicator variants is tested using a small dataset of applicants for a senior full-time research fellowship. Options for further research involve testing various indicator variants on larger samples linked to different kinds of evaluations.

This paper examines the role of interdisciplinarity on research pertaining to local issues. Using Colombian publications from 1991 until 2011 in the Web of Science, we investigate the relationship between the degree of interdisciplinarity and the local orientation of the articles. We find that a higher degree of interdisciplinarity in a publication is associated with a greater emphasis on local issues. In particular, our results support the view that research that combines cognitively disparate disciplines, what we refer to as distal interdisciplinarity, is associated with more local focus of research. We discuss the policy implications of these results in the context of national research assessments targeting excellence and socio-economic impact.

Allocation of research funding, as well as promotion and tenure decisions, are increasingly made using indicators and impact factors drawn from citations to published work. A debate among scientometricians about proper normalization of citation counts has resolved with the creation of an Integrated Impact Indicator (I3) that solves a number of problems found among previously used indicators. The I3 applies non-parametric statistics using percentiles, allowing highly-cited papers to be weighted more than less-cited ones. It further allows unbundling of venues (i.e., journals or databases) at the article level. Measures at the article level can be re-aggregated in terms of units of evaluation. At the venue level, the I3 creates a properly weighted alternative to the journal impact factor. I3 has the added advantage of enabling and quantifying classifications such as the six percentile rank classes used by the National Science Board's Science & Engineering Indicators.

This paper describes the method for ex-post peer-review evaluation per research discipline used at the Vrije Universiteit Brussel, and the outcomes obtained. Pertinent advice and responses at different levels benefit research quality, competitivity and visibility. Imposed reflection and contacts modify the researcher's attitude and improve team strategies. Deeper insights and data sets on research disciplines and extracted general recommendations support the university management's policy decisions, instruments and guidelines. Comparisons with other assessments lead to a better understanding of possibilities and limitations of different evaluation processes. The described peer-review method can be applied systematically, yielding a complete overview, or on an ad hoc basis for a particular discipline, based on demands from research teams or on strategic or policy arguments.

Canada has a service economy and R&D in Canada is mainly a service sector activity. This paper examines the sectoral distribution of expenditure on R&D performance, with emphasis on the business sector in Canada and with international comparisons. Human resources are a key component in the performance of R&D, and comparisons are made, overtime, of the number of research workers in service and non-service industries, of the ratio of professional to technical and other personnel, and of the changes in educational levels of R&D personnel. Using Canadian experience as a guide, some conclusions are drawn about the measurement challenges in producing indicators of the transition to a service economy.

We study the patterns of engagement in collaborative research among university researchers. We investigate how frequently researchers engage in collaborative research with third parties, as well as the factors that affect the probability of interacting with industry as compared to other types of partners. By focusing on a large sample of recipients of collaborative research grants, we examine how environmental and individual characteristics impact on both the size of the network set up by academic researchers, and the type of partner chosen. Our results demonstrate that individual characteristics matter both for the size and the type of the network, while department-level characteristics are particularly important for the type of the network set up by the researcher.

The growing importance of technology-relevant non-publication output of university research has come into the focus of policy-makers' interest. A fierce debate arose on possible negative consequences of the increasing commercialization of science, as it may come with a reduction in research performance. This paper investigates the relationship between publishing as a measure of scientific output and patenting for German professors active in a range of science fields. We combine bibliometric/technometric indicators and econometric techniques to show that patenting positively correlates with the publication output and quality of patenting researchers.

This paper describes the development of research evaluation in Spain. It assumes that research evaluation, R&D policy and programme evaluation are embedded in the development of and R&D system and are characterised by general Spanish policy-making. Research evaluation in a context of delegation and as a self-organising system for research actors guaranteed by the state, has been strongly developed in the last few years: R&D policy and programmme evaluation in less institutionalised. The explanation is linked to the sequence of reforms of the R&D system and to the set up of the first Spanish science and technology policy. The support of the European Commission is acknowledged (HCM contract CHRX-CT93-0240) and the Spanish National R&D Plan (projects SEC 93-0688 and SEC 94-0796).

The overarching aim of this paper is to contribute to a better understanding of how the main target variable of innovation policy – change in behaviour – can be better conceptualized and put into practice in evaluation and policy making. The paper first develops a theoretical framework of the concept of behavioural additionality. On that basis it looks in detail at the way behavioural additionality is operationalized in evaluation practice and how the concept is discussed and applied in the interaction between policy-makers and evaluators. The paper utilizes a statistical analysis of 171 innovation policy evaluations, a text analysis of selected behavioural additionality evaluation reports and finally a number of in-depth case studies of evaluations. Based on the theoretical considerations and the empirical findings, the paper identifies three different uses of behavioural additionality in innovation policy evaluations. It concludes that despite the widespread use of the behavioural additionality concept, an improved theoretical basis and serious methodological improvements are needed to realize the full potential of this concept for evaluation and policy practice.Those who are interested in this article or the underlying study are welcome to contact the authors.

A broad definition of innovation input is used, in which R&D is one of several sources of innovation. A quantitative innovation output measure is used in the analysis, which is based on a large representative sample of firms, including small firms. An econometric framework based on the knowledge-production function accounting for both selectivity and simultaneity bias is employed. The results from Nordic countries show that, given difficulties in pooling the data, it is important to identify country-specific models to account for country-specific effects and differences in countries' national innovation systems.