ArticlePDF Available

Quality Criteria and Indicators for Responsible Research and Innovation: Learning from Transdisciplinarity


Abstract and Figures

The concept of ‘responsible innovation’ (RI) or ‘responsible research and innovation’ (RRI) is rapidly gaining currency. However, a persistent critique is that without more concrete elaboration, the interpretive flexibility of the concept is so broad as to effectively render it meaningless. The articulation of quality criteria and indicators therefore seems crucial for RRI to be understood and operationalized by researchers, research funders, innovators and other relevant stakeholders. In this paper, we specifically draw on our knowledge and experience from the transdisciplinary research community, combined with recent multi-stakeholder deliberative work on the concrete case of nanoremediation, to make an offering on the challenge of articulating quality criteria and approaches to evaluate RRI. In doing so, we present an iteratively arrived at set of quality criteria, designate significant elements of each of these, and then develop an evaluative rubric of performance indicators across them. While the criteria and rubric we present were initiated through the specific context of our work on advancing RI in the research, development and use of nanoparticles for environmental remediation, we believe that they can serve as a useful example for how evaluative criteria and approaches can be developed and offer a helpful frame for sponsoring and structuring the ongoing conversations on quality criteria and indicators that are necessary if RRI is to reach its full potential.
Content may be subject to copyright.
This article was downloaded by: [Fern Wickson]
On: 14 October 2014, At: 03:11
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Responsible Innovation
Publication details, including instructions for authors and
subscription information:
Quality criteria and indicators for
responsible research and innovation:
Learning from transdisciplinarity
Fern Wicksonab & Anna L. Carewc
a GenØk Centre for Biosafety, Forskningsparken, PB 6418, Tromsø
9294, Norway
b Oslo Group on Responsible Innovation, Oslo and Akershus
University College of Applied Sciences, Postboks 4, St. Olavs plass,
NO-0130 Oslo, Norway
c Tasmanian Institute of Agriculture, University of Tasmania,
Private Bag 98, Hobart, TAS 7001, Australia
Accepted author version posted online: 24 Sep 2014.Published
online: 08 Oct 2014.
To cite this article: Fern Wickson & Anna L. Carew (2014): Quality criteria and indicators for
responsible research and innovation: Learning from transdisciplinarity, Journal of Responsible
Innovation, DOI: 10.1080/23299460.2014.963004
To link to this article:
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or
howsoever caused arising directly or indirectly in connection with, in relation to or arising
out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Quality criteria and indicators for responsible research and innovation:
Learning from transdisciplinarity
Fern Wickson
*and Anna L. Carew
GenØk Centre for Biosafety, Forskningsparken, PB 6418, Tromsø 9294, Norway;
Oslo Group on
Responsible Innovation, Oslo and Akershus University College of Applied Sciences, Postboks 4, St. Olavs
plass, NO-0130 Oslo, Norway;
Tasmanian Institute of Agriculture, University of Tasmania, Private Bag 98,
Hobart, TAS 7001, Australia
(Received 26 February 2014; accepted 5 September2014)
The concept of responsible innovation(RI) or responsible research and innovation(RRI) is
rapidly gaining currency. However, a persistent critique is that without more concrete
elaboration, the interpretive exibility of the concept is so broad as to effectively render it
meaningless. The articulation of quality criteria and indicators therefore seems crucial for
RRI to be understood and operationalized by researchers, research funders, innovators and
other relevant stakeholders. In this paper, we specically draw on our knowledge and
experience from the transdisciplinary research community, combined with recent multi-
stakeholder deliberative work on the concrete case of nanoremediation, to make an offering
on the challenge of articulating quality criteria and approaches to evaluate RRI. In doing so,
we present an iteratively arrived at set of quality criteria, designate signicant elements of
each of these, and then develop an evaluative rubric of performance indicators across them.
While the criteria and rubric we present were initiated through the specic context of our
work on advancing RI in the research, development and use of nanoparticles for
environmental remediation, we believe that they can serve as a useful example for how
evaluative criteria and approaches can be developed and offer a helpful frame for
sponsoring and structuring the ongoing conversations on quality criteria and indicators that
are necessary if RRI is to reach its full potential.
Keywords: responsible innovation; transdisciplinary research; nanoremediation; evaluation;
1. Introduction: the need for quality criteria and indicators
The concept of responsible innovation(RI) or responsible research and innovation(RRI) is
rapidly gaining currency in European policy discourse as well as in international scholarship
on the governance of new and emerging technologies. This rising emphasis on having the knowl-
edge economy develop responsiblyis arguably the latest manifestation of a longer historical
trend reimagining the relationship between science and society in ways that depart from the tra-
ditional linear modelor received view(Guston 2000). This historical development of attempts
to re-imagine the sciencesociety relationship and indeed, to enact it in new ways, has been
described by Stilgoe, Owen, and Macnaghten (2013) as observable through the development
of practices such as technology assessment in its various forms, the increasing institutionalization
© 2014 Taylor & Francis
*Corresponding author. Email:
Journal of Responsible Innovation, 2014
Downloaded by [Fern Wickson] at 03:11 14 October 2014
of public engagement, the embedding of research on ethical, legal and social aspects into large
technology development initiatives, and an increasing use of socio-technical integration and mid-
stream modulation. In a similar vein but emphasizing elds of practice rather than particular
methods or approaches, Grunwald (2011) has suggested that RRI extends a general model and
philosophy of technology assessment by incorporating perspectives of applied ethics and
science and technology studies. Despite a sense in which RRI is therefore an amalgamation, cul-
mination, or latest manifestation of various activities and elds of practice, a singular universally
accepted denition of RRI has yet to fully crystallize.
Various actors and initiatives have recently proposed denitions for RI/RRI (e.g. see EPSRC
2013; Jacob et al. 2013; Owen et al. 2013; von Schomberg 2013), and while these denitions may
not completely converge, they do have overlapping features that suggest a loosely agreed set of
core characteristics. The proposed denitions differ in the terminology they use, the orientation
they adopt, the depth of description they provide and where they place emphasis. However,
certain characteristics are shared across them and can therefore be identied as central to the emer-
ging concept. These include:
(1) A focus on addressing signicant socio-ecological needs and challenges;
(2) A commitment to actively engaging a range of stakeholders for the purpose of substan-
tively better decision-making and mutual learning;
(3) A dedicated attempt to anticipate potential problems, assess available alternatives and
reect on underlying values, assumptions and beliefs; and
(4) A willingness among all participants to act and adapt according to these ideas.
Although not specically mentioned by Stilgoe, Owen, and Macnaghten (2013) in their overview
of the historical roots grounding the concept of RI, the core characteristics identied above also
have strong resonance with the characteristics of transdisciplinary (TD) research. Transdiscipli-
narity specically emerged as an attempt to operationalize the sustainability imperative that
arose from the attainment of international consensus in the 1990s on the need for co-operative,
proactive action on global environmental and social challenges (e.g. Brundtlandt Report)
(WCED 1987). TD research has been characterized as involving (see Wickson, Carew, and
Russell 2006; Carew and Wickson 2010):
(1) A focus on complex and multidimensional real-world problems;
(2) Transcending academic boundaries to incorporate collaboration and mutual learning
between a range of researchers and relevant stakeholders; and
(3) An iteratively evolving method that reects on the problem and its context from a range
of perspectives and responds accordingly.
A recent report on options for strengthening RRI, under the auspices of the European Commission
(Jacob et al. 2013), specically highlights TD research (and the related approach of sustainability
science) as potentially important components of RRI. However, one of the specic obstacles
identied in the report is that there are no common quality standards or criteria for these types
of research that can serve as a useful guide for researchers, project managers or research
funding organizations. For those working within sustainability science and TD research, the
lack of appropriate quality criteria has been recognized as a signicant problem for some time
(e.g. see Komiyama et al. 2011; Jahn, Bergmann, and Keil 2012), and over the last decade a
number of attempts have been made to address it (Boix Mansilla and Gardner 2003; Mitrany
and Stokols 2005; Spaapen, Wamelink, and Dijstelbloem 2003;Bergmann et al. 2005; Carew
and Wickson 2010; Pohl et al. 2011).
2F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Evaluation criteria and schemas that have been proposed for TD research vary between those
advocating close attention to the institutional and interpersonal preconditions for successful TD
research, those more focused on the importance of good process, and those encouraging close
attention to the nal products or outcomes of the research. There is also substantial variation
in the balance different authors achieve between comprehensiveness and over-prescription. For
example, Bergmann et al. (2005) offered a particularly complex and time-intensive evaluative
schema, while the prompts suggested by Boix Mansilla and Gardener (2003) were provocative
as general guidance but were arguably not rich enough to support the full-scale design and
execution of integrative research. One of the core challenges is that given TD is an innately tai-
lored and necessarily responsive approach to research, any quality criteria need to be uniquely
exible to provide valid, reliable means to evaluate and compare projects, while not stiing
the evolution and responsiveness of the approach.
Beyond what has been taking place in the TD research community, in recent years a range of
different analytical and reporting methods have emerged to try and meet demands for responsi-
bility and responsiveness in the operations and decision-making of public and private entities.
Such schemas include triple bottom line sustainability reporting within Corporate Social Respon-
sibility frameworks (CSR) (e.g. see Labuschagne, Brent, and Erck 2005; Szekely and Knirsch
2005), Public Value Failure Mapping within science policy analysis (PVM)) (e.g. see Bozeman
and Sarewitz 2005,2011) and formalized processes of stakeholder consultation through
methods of multi-criteria decision analysis (MCDA) (e.g. see Petrie et al. 2004; Burgess et al.
2007; Elghali et al. 2008). Each of these approaches has also been considered and discussed in
relation to nanotechnology as a particular case of interest (see Groves et al. (2011) and Kuzma
and Kuzhabekova (2011) for CSR; Linkov et al. (2007) and Linkov and Seager (2011) for
MCDA; and Slade (2011) for PVM). Each of these elds of work has recognized the need for
and sought to prescribe processes for evaluative analysis and reporting, although each with its
own particular area of emphasis and applicability.
For example, CSR is a concept that is focused on corporate actors and has traditionally prior-
itized reporting frameworks for assessing the environmental, economic and social dimensions of
the companys contribution to sustainability (e.g. using reporting frameworks such as that of the
Global Reporting Initiative or international standards such as ISO 14000 and/or 26000). PVM, on
the other hand, has specically focused on public actors engaged in science policy processes and
has developed criteria for mapping when and how these policies fail to articulate, meet and
advance social or public values. In contrast, MCDA represents a constellation of different
methods for specically enhancing stakeholder consultation in the decision-making of either
public or private actors and has been particularly focused on managing tradeoffs in high-stakes
decisions shrouded in scientic uncertainty and subject to contested social values.
While each of the existing elds of CSR, PVM and MCDA clearly shares an overlapping ter-
ritory of interest with the emerging frameworks and denitions of RRI, again, RRI appears to be a
nascent umbrella concept seeking to bring together and unite the different areas of focus, dimen-
sions of interest and actors of attention present across these other areas; for example, combining
interests in sustainability, social values and stakeholder engagement, and seeking applicability
across both public and private actors engaged in research and innovation and their governance
and regulation. However, just as each of the elds mentioned above has recognized and been
working with the signicant challenge of providing quality criteria, indicators and assessment
methods to improve performance (either in corporate sustainability, the public value of science
policies or stakeholder involvement in decision-support), a common or easy critique of the emer-
ging umbrella notion of RRI is that it remains overly vague (e.g. see Bensuade Vincent 2014;
Oftedal 2014) and that without concrete elaboration and conceptual development, the interpretive
exibility of RRI will be so broad as to render the concept meaningless. Indeed, as Zwart and
Journal of Responsible Innovation 3
Downloaded by [Fern Wickson] at 03:11 14 October 2014
colleagues see it (Zwart, Landeweerd, and van Rooij 2014), the term RRI was introduced as a
signier in a top-down manner before the concept signied by it had a clear and stable prole,
and therefore the research community is now being explicitly invited to populate the term with
The articulation of relevant quality criteria and indicators, and particularly those that can unite
and encompass previously disparate areas of interest and investigation, therefore seems to be a
crucial task to lend the notion of RRI meaning and allow it to be understood and operationalized
by researchers, research funders, innovators and other relevant stakeholders. The task of develop-
ing quality criteria and indicators for RRI, however, meets with similar problems to those that
have been facing these related elds over recent decades. For example, one signicant challenge
is clearly the need to answer an initial question concerning whether the focus should be on eval-
uating the preconditions,process,orproducts of research and innovation (or perhaps even the
people populating all three). Furthermore, any emerging quality criteria need to be concrete
enough to provide real guidance but exible enough to adapt to the specicities of varying
In this paper, we specically draw on our knowledge and experience in the TD research com-
munity and eld to make an offering on the challenge of developing quality criteria for RRI that
we hope is a useful starting point on this issue and one that can helpfully prompt and facilitate
ongoing discussion, development and debate. We do this by rst proposing a set of dening
quality criteria that emerged through an iterative, multi-stakeholder process that we conducted
on the particular case of RRI in nanoremediation. We then identify some potentially signicant
elements to help denote and elaborate the content of each of these criteria, before nally demon-
strating how these can be developed into a qualitative evaluative rubric of performance indicators
for RRI. We consider the rubric we present as an outcome developed within our particular context
of research (the relatively narrow case of nanoremediation) drawing on our related area of exper-
tise (TD research). However, we believe that the presentation and description of this approach to
quality criteria and the evaluation of RRI, can serve as a useful starting point for an important
conversation that we think needs to take place if the notion of RRI is to reach its full potential.
We therefore particularly welcome interactions, alterations, critique and further elaboration of
both the rubric approach and the evaluative criteria and indicators for RRI that we propose and
present herein.
2. Method: iterative development of the RRI quality criteria and indicators
The quality criteria and indicators for RRI that we propose were derived through a series of events
and activities over several months. These events and activities were aimed at elucidating through
conversation and deliberation with a range of stakeholders what criteria might be used to under-
stand, explain, judge and approach RRIin the context of the development and use of nanopar-
ticles for environmental remediation (nanoremediation).
In the rst instance, the process of developing the schema was informed by our reading and
interpretation of emerging work published on the dening characteristics of RRI, as well as the
concepts relationship to related terms and elds, including general ideas concerning good
scienceand good technologyand more specic literature on concepts of quality that have
been proposed in elds such as transdisciplinarityand sustainability science. From this con-
ceptual and theoretical basis, our next step was to design and run an international and interdisci-
plinary workshop with a focus on articulating RRI quality criteria and understanding how they
may play out when applied to a particular case (nanoremediation). This workshop was held in
May 2013 at Sundvolden, Norway as part of a project on responsible ecological governance of
emerging technologies (funded by the Norwegian Research Council) and also to contribute to
4F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
the NANoREG project to develop a common approach to the regulatory testing of nanomaterials
(funded by the European Commission).
The workshop gathered 17 participants from different elds but all with expertise working
with either the specic issue of nanoremediation or the environmental governance of nanotech-
nology. These participants came from eight different countries (Norway, Denmark, Australia,
Brazil, England, Czech Republic, Germany and the USA) and were brought together to
discuss, debate and attempt to advance RRI in the eld of nanoremediation. The gathered
group included actors specialized in (with some actors holding more than one nominated area
of specialized expertise): nanoecotoxicology (seven individuals working within academia
across four different institutions and three national contexts: Norway, Denmark and the Czech
Republic), environmental chemistry (four individuals working within two institutional and
national contexts in Australia and Norway), environmental remediation technologies (four indi-
viduals, two working in the corporate sector, in Germany and the Czech Republic respectively,
one working in academia in Denmark and one working across academia and the corporate
sector in the Czech Republic), environmental policy and/or science and technology governance
(three individuals working primarily within academia but distributed across three institutions in
Brazil, the UK and Australia) and environmental philosophy (one individual working within aca-
demia in the USA). Approximately one third of the invited participants were graduate students or
early career researchers, while the remaining members were senior staff with a high status in their
eld or institution. This generational mix was considered crucial given the emerging nature of
nanotechnology and our explicit focus on building responsible futures.
With expertise across more than one of the listed elds and a status arguably somewhere
between senior and junior levels, the authors interchanged positions throughout the different
activities of the event to act as both facilitators and participants in the workshop. Acting as facil-
itators and participants was both challenging and interesting as it served to blur power relation-
ships within the process. This embedding of ourselves in the activity of co-creation of RRI
criteria serves to illustrate a point of difference between more structured consultation approaches
like MCDA, where stakeholder views are elicited by a supposedly neutral facilitator, and the TD
approach used in this study which attempted some democratization of the research process, some
deconstruction of power relations within the negotiation of criteria and an intimate engagement of
the researchers in the research context (Carew and Wickson 2010).
Given the well-recognized and documented problem of attaining conditions able to facilitate
situations resembling those of Habermasideal speech in multi-stakeholder deliberative exercises
(e.g. see Burgess and Chilvers 2006), participants in our workshop were intentionally not invited
to declare their position and status in an introductory round the table. This was due to our inten-
tion to disrupt the power relationship that often impinges on the free exchange of ideas in aca-
demic discourse. Rather, in the rst event of the workshop, participants were introduced to one
another through a World Café-style activity (see Brown 2005). In this activity, they generated
answers (rst individually, then in small mixed groups, and nally in plenary) to prompts such
as: ‘“The universe is made of stories, not of atomsAgree/Disagree?,I feel condent in my
knowledge when …’,Science is …’,Good Science is …’,Good technology is …’, and
How would you describe the relationship between humanity and nature?Using these
prompts and their discussion as a way to introduce participants to one another was informed
by a phenomenographic enquiry approach, which aims to map the qualitatively different ways
in which people conceptualize, perceive, and understand various aspects of the world
around them(Marton 1986, 31). Following small group and plenary discussions on each of
the questions, we developed the participantsresponses into an outcome spacethat mapped
the range of qualitatively different ways in which some of the fundamental issues and underlying
beliefs that we opened for discussion were conceived and held by the actors present. Outcome
Journal of Responsible Innovation 5
Downloaded by [Fern Wickson] at 03:11 14 October 2014
space posters were then put on display around the meeting room to help inform the workshops
subsequent activities and conversations; they proved particularly useful as the workshop
progressed when competing conceptions on these issues could be linked to variation in the
assumptions and objectives that each participant brought to later tasks, such as that focused on
co-developing quality criteria for RRI.
Interestingly, the World Café introductions revealed that participants gathered for the work-
shop were evenly divided on the question of whether they saw the world as made of atoms or
stories, indicating a signicant divergence in basic ontologies (e.g. whether participants were
more committed to materialism or idealism in their philosophical views and approaches to the
world). Perhaps unsurprisingly, the World Café activities also saw signicant divergence in
relation to the question of science. Some participants held that science was essentially explora-
tion, while others thought that it was a specic and highly structured process. Some felt that
it was a process that delivered reliable and elegant answers, while others emphasized that it
was a process that necessarily and characteristically led to new questions. Some participants
painted science in opposition to religion and, at the same time, others characterized it as the reli-
gion of the twenty-rst century. When it came to the question of what constituted good science
and how this differed from sciencein these rst-impression discussions, some participants felt
that good science needed to be socially relevant, seriously consider its limitations and be open to
interrogation. Others felt that it was simply that which was performed by virtuous scientists or
that it essentially involved following a high-quality process.
After canvassing the diversity of the workshop group and its membersunderlying views
and assumptions in these introductory conversations, it was fascinating to be part of the
group navigating the values landscape that the process had unearthed when we later asked
the group to focus on the specic task of developing quality criteria and indicators for RRI. Par-
ticipants who believed that the world was made of atoms conversed openly with colleagues for
whom the world was constructed through stories; participants who originally saw good science
as purely dened by good process happily negotiated quality criteria with those wanting science
to deliver socially relevant answers; and those who held science to be the main counterpoint to
religion sought consensus positions with participants who held science to be the twenty-rst
centurys religion. We believe that using the World Café approach as a way to introduce
participants to each other helped to render explicit the diversity of the group and by
making this diversity transparent from the very beginning allowed for greater acceptance
of and willingness to negotiate across the range of views and positions that emerged on the pur-
poses and processes of RRI. As postulated by Ulrich (2000), this laying bare of participants
fundamentally different views and values at the outset of a co-operative process appeared to
build understanding and facilitate respectful and constructive dialogue throughout the later
workshop tasks.
On the second day of the Sundvolden workshop, participants focused on the specictaskof
developing quality criteria and indicators for RRI. For this task, we assembled participants into
three mixed-disciplinary groups and asked them to nominate the range of criteria they saw as
relevant for dening good science. The full range of criteria was then shared, discussed and
debated in plenary. We then asked each group to select one draft criterion that had been ident-
ied and work together to develop a rubric of performance for that criterion. Rubrics are a
higher education assessment format for which standards of performance are spelled out
along a gradation that would constitute poor performance against the criterion, to the charac-
teristics or indicators of exemplary performance. While the Sundvolden participants were
mostly unfamiliar with rubrics, they embraced the task and worked to develop a gradation
scale for the criterion that their group had selected. The three criteria selected by the different
groups were honesty,diversityand social relevance. The range of criteria originally
6F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
articulated and the draft standards of performance that were developed for the three of these
formed the rst iteration of the emerging quality criteria and indicators for RRI that are the
subject of this paper.
In the months following the Sundvolden workshop, the authors reected on how the work-
shop outcomes related to the literature on both TD research quality and the emerging concept
of RRI and its norms. We conversed and debated the emerging list of potential RRI quality criteria
in several meetings with each other, as well as discussed various iterations with other interested
researchers and colleagues working in diverse elds (e.g. rural sociology, marine ecology, agri-
cultural economics, science and technology studies, ethics and organizational management).
This process of reection, debate, explanation and discussion developed the Sundvolden out-
comes into a supplemented and extended set of criteria. We then advanced the Sundvolden
work by developing more elaborated indicators of performance for each criterion and formatting
these into a draft rubric for RRI. Finally, we sought feedback on this draft rubric through: (a) infor-
mal discussion with colleagues in a range of disciplines and with RRI theorists and practitioners;
(b) presenting the criteria and rubric at a session on RI at the international conference of the
Society for the Study of Nanoscience and Emerging Technologies (S.Net); and (c) using the
draft rubric as a prompt for group work at a professional development workshop on evaluating
TD undergraduate science learning. Following this phase of gathering feedback on the draft
rubric, the authors added two additional criteria to the list, and edited and further developed
the rubric into Table 1.
2.1. Qualifying statement
Throughout the course of developing the RRI rubric presented in this paper, we consistently
encountered the question of whether we were developing a schema to evaluate research or inno-
vation. Adopting a Mode 2 (Gibbons et al.1994) or post academic (Ziman 1996) view, we found
it impractical, and indeed increasingly unhelpful, to try and separate research from innovation. We
therefore rejected the notion of a sharp separation between research and innovation. The rubric
presented here is therefore oriented toward RRI. While this may be criticized as creating a
beast that is far too large and diverse an entity to usefully engage and practically manage, we
believe that the systemic entanglement of these activities is so extensive in late modernity that
to create an articial separation between them would do more harm than confronting the beast
A similar conundrum we identied in the introduction to this paper was whether evaluative
criteria for RRI and related concepts (e.g. TD research, sustainability science) which are
rich, complex and systemic in practice should focus on institutional and interpersonal precondi-
tions for success, the importance of good process, and/or the nal products or outcomes of the
research. We agreed that any product developed through an innovation process will inevitably
enter a complex web of interactions of use, the outcomes of which will be inherently uncertain.
Given this, we soon lost faith with the sense of mandating that a particular innovation product
could or should be evaluated against the emerging norms. We therefore chose to focus more
directly on developing evaluative criteria for RRI as a process.
Despite making this choice, however, we found ourselves consistently drawn back into dis-
cussing the criteria as they related to products and felt a persistent need to acknowledge the impor-
tance of contextual preconditions and the characteristics of people actually conducting the
process. Through this struggle, we have come to realize that in focusing on the evaluation of a
process of innovation, one still needs the capacity to also consider existing preconditions, envi-
saged products and engaged people, since all of these elements shape, guide and, to some extent,
generate and characterize the RRI processes that the schema is primarily designed to evaluate.
Journal of Responsible Innovation 7
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Here we inadvertently walked in the footsteps of Bohr (1949) and Barad (2007) in accepting
the impossibility of a sharp separation between process and product. Bohrs ideas of complemen-
tarity would allow that these two phenomena offer unique and necessary, but potentially contra-
dictory, insights into the overarching focus the generation of novel knowledge objects. Barads
concept of agential realismextended Bohrs validation of complementary-but-discordant con-
ceptualizations by suggesting that the separation and naming of concepts like process and
product may be a cultural artifact of our need to isolate aspects of complex phenomena for the
purposes of examination and understanding. Hence, although we chose to orient ourselves pri-
marily toward the concept of process, we also acknowledged the value and place of product, pre-
conditions and people as complementary to understanding RRI. In short, although the schema we
offer is primarily oriented toward evaluating RRI as a process, it also contains elements that relate
to preconditions, products and people. It is perhaps worth noting that a schema primarily oriented
toward one or other of these later elements may well look different to the one we propose here (e.
g. see Glerup and Horst 2014).
Finally, we wish to comment upon how some of our own positions, beliefs and assumptions
might have inuenced our framing of the activities and outcomes that resulted. For example, one
of us (Wickson) is a member of the board of two different programs of the Norwegian Research
Council. Recent government commitment to responsible technology development (see The Nor-
wegian Ministry of Education and Research 2011; The Norwegian Ministries 2012), without the
articulation of a specicdenition or set of evaluative criteria, has created a signicant problem of
how to operationalize RRI within this context, including within the prioritization of elds in
research program planning and individual research project funding. This experience from practice
can be seen to have created a desire to arrive at a criteria set and approach that could be utilized
within a research council setting. Furthermore, that each of us has been committed to the norm of
sustainability across our research careers and personal lives, as well as has a history in teaching
ethics to students in the natural sciences, may also be seen to have informed a willingness to unite
these disparate elds and encapsulate both in our list of criteria. This is not to say that we con-
sciously set out with these intentions, but rather that we have come to see their potential inuence
when we have reected upon our own role in facilitating and performing the deliberative and
iterative aspects of our method. We lay them bare here as both evidence of the importance of
reexive learning in participatory research and a reminder that although we hope that it has
broader usefulness, what follows has emerged from a particular and unique set of contextual
3. Results: a rubric for RRI
Based on the method described above, we propose seven paired and interwoven quality criteria
for RRI. We propose that RRI can be evaluated according to whether it is
.Socially relevant and Solution oriented
.Sustainability centered and Future scanning
.Diverse and Deliberative
.Reexive and Responsive
.Rigorous and Robust
.Creative and Elegant
.Honest and Accountable
These criteria effectively unite demands from different communities concerning what good
science and responsibleresearch and innovation should entail. For example, the criteria of
8F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
creative and elegantand rigorous and robustare features that are traditionally valued within the
scientic community and are already used to assess the quality of scientic projects. That is, they
are factors that scientists themselves typically emphasize as important quality criteria for evalu-
ating research and development. The characteristics of honesty and accountabilityare already
emphasized in the eld of research ethics as important if science and innovation are to develop
responsibly and link to broader calls for transparency as an ethical value. The criteria for research
and innovation to be diverse and deliberativeand reective and responsiveare powerful norms
in the eld of science and technology studies and a key feature of MCDA approaches, while a
demand for innovation to be sustainability centered and future scanninghas emerged in
response to global environmental concerns and movements and is often emphasized within
CSR practices. Finally, the sense that RI should be socially relevant and solution oriented
echoes the values of PVM and is an emerging demand within science policy arenas and, arguably,
also one that industry is also often oriented toward. By combining quality criteria from across
different elds and communities, our proposed criteria unite these varying demands under a
common umbrella of RRI. This unication is crucial to achieve acceptance and uptake of the
concept of RRI since it enables these different elds of practice and communities of stakeholders
to be able to identify their existing norms within the conceptual framework but also to see how
these become extended under the concept of RRI.
While the criteria listed above make a start on articulating what good RRI might entail, the
terms used remain somewhat opaque and open to varying interpretations. Although some scholars
have recently suggested that this vagueness is exactly what gives buzzwordslike RRI their cur-
rency across different groups of actors and arenas (Bensaude Vincent 2014), as indicated above,
in this paper we have chosen to develop an inclusive group of dening quality criteria as a way to
try and unite different elds of practice and communities of stakeholders, but we are also speci-
cally interested in developing more concrete elaboration so as to avoid the concepts being a
walking shadow, full of talk and promise but signifying nothing (to paraphrase Shakespeare).
Of course some may view our proposed criteria as confounded and prefer to separate them,
pair them differently or (re)interpret them in the context of their own evaluative tasks and projects.
We believe that in any process of interpreting, rearranging and/or applying these criteria, open
negotiation on their meaning and application would only positively foster transformative learning,
which is a valued outcome of good TD processes (Pennington et al. 2013).
Within our own context, an intermediate step toward developing indicators of performance for
each paired set of criteria involved identifying relevant elements of each to give them enhanced
denitional clarity and meaning. In the case and process we were working within, we arrived at
the elements listed below as important components for consideration under each of the criteria. It
is entirely possible, however, that alternative elements for each criterion could (and arguably
should) be developed in specic relation to different contextual situations and by different
actors, and ideally through deliberative processes involving a range of relevant stakeholders to
the identied problem and context. The elements we present below are, however, those that we
used to delimit the criteria from each other and guide the development of performance indicators
in our own rubric.
.Socially relevant and Solution oriented
(a) Type of problem addressed
(b) Type of solution sought
.Sustainability centered and Future scanning
(a) Anticipating potential futures
(b) Identifying potential risks and benets
(c) Considering social, economic and environmental sustainability
Journal of Responsible Innovation 9
Downloaded by [Fern Wickson] at 03:11 14 October 2014
.Diverse and Deliberative
(a) Level of cross-disciplinarity involved
(b) Where stakeholders are involved
(c) How stakeholders are involved
.Reexive and Responsive
(a) Recognition of preconditions in context and group
(b) Exploration of underlying values, assumptions and choices
(c) Openness to critical scrutiny
(d) Ability to change after internal reective practice and external feedback
.Rigorous and Robust
(a) Aspects of the problem considered
(b) Repeatability across actors and settings
(c) Reliability of outcomes under real-world conditions
.Creative and Elegant
(a) Novelty and daring
(b) Sufciency and beauty
.Honest and Accountable
(a) Identication of uncertainties and limitations
(b) Lines of delegation and ownership
(c) Compliance with research ethics and governance requirements
(d) Policies on open access and information sharing
(e) Ownership over positive and negative outcomes
The elements listed under each criterion assisted the process of developing the quality criteria into an
RRI performance rubric. We used these elements to guide the development and renement of state-
ments that explained what could be considered routine,good,great and exemplary examples of
practice against each quality criterion. The outcome is presented in Tabl e 1 as a rubric for assessing
4. Discussion: use of the RRI rubric
The rubric we have developed for RRI and presented in Table 1 is intended to represent a useful
approach to clarify what is expected by and of different parties in the RRI process, and to provide
inspiration, concrete guidelines and direction for improvement for those seeking to innovate in
more responsible ways. An RRI rubric such as that presented here can, however, also be used
to evaluate and compare the quality of RRI in different projects and initiatives, or be used to
evaluate progress in a single project over time.
The development of rubrics such as this would offer much needed clarity on what good
quality RRI looks like. This clarity is urgently needed for:
(a) research funding bodies that need to compare and decide between different project appli-
cations in terms of the likely contribution each might make to meeting and advancing
(b) researchers and/or innovators needing to articulate either their capacity to undertake and/
or report on their ongoing work toward RRI, so as to satisfy institutional, public, policy
or funding body demands, and
(c) researchers, innovators and project managers looking to continually improve the per-
formance of their work over time.
10 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Table 1. A rubric for assessing RRI.
Criteria Exemplary Great Good Routine
Socially relevant
and Solution
Addressing a grand social
Ongoing analysis of objectives
and processes to favor the
delivery of wicked solutions
(solving multiple challenges
Addressing a signicant social
Ongoing analysis of objectives
and processes to maintain a
focus on delivering a
successful solution
Focused on a marginal or self-
dened problem.
Employing processes aimed at
generating insights toward a
solution, or a partial solution
Pursuing a purely personal
Possibility that process/product
will only result in the creation
of decontextualized knowledge
or new problems
centered and
Inclusion of formal processes of
future casting to at various
points throughout the research
and innovation process.
Generating a range of positive and
negative future scenarios and
identifying and assessing
associated risks and benets of
these for social, environmental
and economic sustainability.
Clear avenues for embedding
responses to these possible
futures and risk/benet
assessments into the project
Inclusion of future casting
activities at some point during
the research and innovation
Some attempt to integrate an
assessment of the risks and
benets for social,
environmental and economic
Identiable points and
possibilities for adaptation of
the process to respond to the
future scanning and risk/
benet assessment activities
Informal attempts to future cast at
limited points in the project.
A consideration of some
associated risks and benets in
terms of one or more of the three
dimensions of sustainability.
Little indication of how the
research and innovation process
may adapt and respond to either
identied possible futures or
their risks to sustainability
A singular optimistic prognosis
for future project outcomes
with no clear effort to identify
risks or survey possible future
Diverse and
Openly and actively seeking
ongoing critical input, feedback
and feed-forward from a range
of stakeholders.
Encouraging and rewarding
transformative mutual learning.
Employing an evolving
integrative method and
consciously employing a TD
Inviting, incorporating and
integrating stakeholder views
at various points along the
research and innovation
Actively seeking dialogic
interaction with stakeholders
and open to mutual learning.
Encompassing a wide range of
methods and adopting an
interdisciplinary process
Limited stages of the research and
innovation process open for
stakeholder engagement.
Tendency toward one-way forms
of communication with
stakeholders but open to some
Involving some level of
methodological diversity and
multidisciplinary practice
Communicating with
stakeholders only toward the
end of the research and
innovation process.
Use of one-way communication
approaches and defensive in
the face of counter-views or
stakeholder questions.
Mono-methodological and mono-
Journal of Responsible Innovation 11
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Table 1. Continued.
Criteria Exemplary Great Good Routine
Reexive and
Clear and explicit identication of
institutional and contextual
limitations and a structured
effort to acknowledge and
improve upon these conditions.
Structured, purposeful periodic
analytical review of underlying
values, assumptions and
Actively seeking critical feedback
from a wide group of sources
and actors.
Evidence of potential to adapt at a
range of points in response to
in-train reective practice and
external review/input/feedback
Explicit effort to identify
institutional and contextual
limitations and awareness of
their signicance for practice.
Occasional use of structured
process for reecting on
underlying values,
assumptions and choices.
Actively seeking critical
feedback from select sources
and actors.
Clear indications of a capacity to
adapt in response to reective
practice and external feedback
Some indication of awareness
concerning limitations posed by
institutional structure and
contextual realities.
Informal, one-off or ad hoc process
for considering underlying
values, assumptions and
Accepting critical feedback when
Stated willingness to accept
change in response to internal
reective practice or external
review and critique
No explicit consideration or
recognition of the limitations
posed by institutional structure
and contextual realities.
No process for facilitating
reective practice.
No critique sought.
No evidence for potential for
change in response to criticism/
unsolicited feedback
Rigorous and
Comprehensive investigation of
all aspects of the problem and
the interconnections between
Results repeatable by a variety of
different actors operating across
a range of relevant conditions.
Outcomes work reliably under
real-world conditions
Considering multiple
dimensions of the problem
and their interrelations.
Results repeatable by the same
actors operating under a range
of relevant conditions.
Outcomes with demonstrated
functionality under real-world
Interest in several dimensions of
the problem although not
necessarily their interrelations.
Results repeatable by the same
actors operating under similar
Outcomes remain untested under
real-world conditions
Narrow focus on one element or
aspect of a problem.
Results not able to be replicated.
Outcomes unable to be reliably
applied in real-world contexts
Creative and
The problem has been reframed in
innovative directions, with new
ideas being pursued through
appropriate methods.
Resources are carefully
considered and allocated to
efciently achieve maximum
utility and impact.
Esthetical consideration is given
to preconditions, process and
New methods are being
developed according to new
ideas within an established
problem framing.
The use of resources is explicitly
Esthetical consideration is given
to preconditions and products
New ideas are being pursued
through established methods
within an accepted problem
Considerable resources are
inefciently employed.
Esthetical consideration is given to
envisaged products
Problem framing, ideas and
methods fall within established
Extensive resources (e.g. time,
money, personnel, etc.) are
dedicated to work with
minimal signicance or
potential impact.
No consideration is given to the
esthetics of operating
preconditions, research and
innovation process or
envisaged products
12 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Honest and
Transparent identication of a
range of uncertainties and
limitations that may be relevant
for various stakeholders.
Openly communicated lines of
delegation and ownership able
to respond to process dynamics
and contextual change.
Documented compliance with
highest-level governance
requirements, research ethics
and voluntary codes of conduct,
all actively monitored
Consistent use of open access to
information policies.
Preparedness to accept
accountability for both
potential positive and negative
Identication of uncertainties
and limitations deemed to be
signicant by those involved.
Established lines of delegation
and ownership.
Compliance with governance
requirements and research
ethics with evidence of active
monitoring throughout.
Favoring open access to
information policies.
Willingness to accept
accountability for potential
positive and negative impacts
Some statement indicating
uncertainties and limitations.
Indications of potential lines of
delegation and ownership.
Complying with minimum
standards of governance
requirements and research
Occasionally employing open
access to information policies
Willingness to accept
accountability for positive
impacts and some negative
No transparency concerning
limitations and uncertainties.
Untraceable ownership of
No specic acknowledgement of
standards concerning
governance requirements or
research ethics.
No demonstrated commitment to
open access information
Accountability only accepted for
positive outcomes
Journal of Responsible Innovation 13
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Rubrics such as that shown in Table 1 could be used by funding bodies, researchers, innova-
tors or managers in much the same way that rubrics in higher education are used to assess and
provide feedback on student work. In that case, the assessor generally marks which performance
standard (routine, good, great, exemplary) the piece of work has demonstrated against each cri-
terion, and the student is provided the marked rubric to understand how their performance was
rated. Alternatively, funding bodies, researchers, innovators and project managers may wish to
visualize and compare performance against the criteria listed in Table 1. Such a visual represen-
tation may be easier to digest and discuss, especially in the case where multiple projects are under
consideration or a project is reporting against the criteria over time. Figure 1 shows a spider chart
method used to map several projects according to the proposed RRI criteria, and Figure 2 shows a
spider chart of performance against the criteria at two time periods in the life of an RRI initiative
or project.
What the spider charts in Figures 1 and 2illustrate is that certain projects and initiatives will
likely occupy different spaces across the range of RRI criteria proposed here, emphasizing certain
aspects perhaps to the neglect of others. It seems unlikely to us, and perhaps unreasonable to
expect, that any particular project or initiative would be able to reach an exemplary standard
across all the RRI criteria proposed. Reaching an exemplary standard for all criteria, however,
may not be necessary for a project to still deliver signicant value and be legitimately recognized
as RRI (e.g. see the third example below). Such visualization processes may be particularly useful
for highlighting and facilitating deliberation about the necessary tradeoffs across the criteria that
could be involved. This could also be conducted using the type of tradeoff analysis typically per-
formed within policy settings (e.g. see Bardach 2000) or it could be taken up in and elaborated
through processes and methods such as MCDA in which the weighting of the different criteria
could be usefully experimented with and expounded upon (by both experts and other stake-
holders). Compromises and tradeoffs are inevitable and while any negotiation of the relative
weighting of such quality criteria will always be contested and often interest based, having a
clear set of criteria and evaluative scales at least enables this process to be an increasingly trans-
parent one. That said, it might also be the case that if the same tradeoffs were systematically being
made and shown in the spider charts (e.g. within the portfolio of research within a company or
research council program) then this may indicate a need to consider rebalancing actions.
Spider charts of RRI could also work to identify obstacles that may be inhibiting the ability of
actors to reach exemplary standards of performance in RRI. In this way, the proposed rubric
would help to identify limitations and opportunities across different arenas for improvement to
Figure 1. Comparing projects.
14 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
the design and conduct of research and innovation. If used such that actors felt free to honestly
identify where a particular RRI effort sat according to the evaluative criteria and to openly
discuss why this was the case, the rubric and associated spider charting process could facilitate
the identication of broader socio-economic, political or institutional factors and cultures that cur-
rently constrain individual actors, projects or institutions from reaching the (ideal) exemplary
standard for RRI (e.g. see Wickson 2014).
An example of the value of the RRI rubric for catalyzing the identication and open discus-
sion of such constraints occurred during our development of the rubric at the Sundvolden work-
shop referred to earlier. A participant group who were drafting the exemplary performance
standard for the draft criterion of honestyobserved that participating researchers could
clearly identify with the ideal of open and transparent communication, particularly concerning
uncertainties and limitations. However, the group noted that in practice, current publication cul-
tures inhibited their ability to reach an exemplary practice as it was described in the draft rubric.
To elaborate, nanoecotoxicologists studying the environmental impact of nanoremediation found
it difcult to publish their work when they could not explain the results generated from their
experiments and could therefore not openly express in the public realm (i.e. through published
literature) the uncertainties and limitations they faced when using the existing methodologies
originally established for the testing of chemical compounds for the testing of novel nanomater-
ials. Additionally, the participating researchers explained that expectations from policy-makers
that they deliver clear and certain answers to the question of environmental impacts also
worked against them honestly reporting their uncertainties and limitations. These challenges
did not prohibit the group nominating open and transparent communication as an exemplary stan-
dard. Rather, the statement of this ideal helped them to identify and articulate socio-political and
institutional-level obstacles to their attaining this exemplary level of performance in their own
work and projects.
In a second example from the Sundvolden workshop, both industry actors and nanoecotoxi-
cologists suggested that for their work to meet exemplary standards of what it meant to be rig-
orous and reliable, they would have to engage in real-world experimental eld trials. However,
regulatory uncertainties concerning the use of nanoparticles in environmental remediation made
this difcult, or impossible in some cases. That is, the specicity of the behavior of nanoparticles
to the context and conditions within which they are used meant that laboratory experiments to test
the effectiveness of different options for nanoremediation were never well correlated to how the
materials would actually behave under real-world conditions. Therefore to achieve outcomes that
could reliably work under real-world conditions required testing under real-world conditions.
Figure 2. Comparing timepoints.
Journal of Responsible Innovation 15
Downloaded by [Fern Wickson] at 03:11 14 October 2014
This was, however, suggested as not always being possible, due to either regulatory restrictions on
the deliberate environmental release of free nanoparticles in some nations or due to the uncertain-
ties in the specic regulatory requirements for using nanoparticles in remediation and/or the local
or national political level at which such requirements were established and applied.
In a third example from the same case, nanoremediation was seen to be an example of inno-
vation directly oriented toward addressing a signicant social need (i.e. remediating polluted
groundwater systems) and although it was not able to reach an exemplary standard of addressing
agrand societal challenge, it could still be considered to have signicant value as a research and
innovation project and potentially be legitimately recognized as RRI. However, when evaluated
according to some of the other criteria, room for improvement was clearly possible. This was the
case, for example, in terms of the diverse and deliberativecriterion where developers of nanor-
emediation technologies present at the Sundvolden workshop identied a clear potential for
tighter integration and enhanced interaction with other scienticelds (such as nanoecotoxicol-
ogy), where working together would help identify and reduce potential risks and advance a
benign by designapproach. However, challenges to improved performance against this criterion
included limited opportunities for the different types of actors to nd each other, and varying
capabilities in communicating across different scientic cultures and terminologies when they
did. These limitations appear to be related to an emphasis on segregation and separation in edu-
cational practices within science and engineering, and limited opportunities for interdisciplinary
interactions at the institutional level.
In each of these examples from the Sundvolden workshop, asking researchers to consider
nanoremediation initiatives in terms of the emerging quality criteria of the RRI rubric catalyzed
important discussion. Signicantly, this discussion unearthed communities of practice and inu-
ence outside the circle of participating researchers and innovators who were inuencing the ability
of the participants to achieve exemplary standards of RRI. Working on and with the rubric there-
fore highlighted a broader range of actors and factors that would need to change if truly RRI were
to be achieved and, therefore, that the responsibility for advancing RRI needed to be distributed
across a wide collective. For us, the process of developing the rubric in co-operation with col-
leagues who might one day (need to) apply it to evaluate their own projects demonstrated the
range of benets it might offer, including this identication of obstacles, barriers and limitations
to the practice of RRI that stem not just from within particular projects or initiatives but are con-
nected to broader and more deeply rooted socio-political factors and institutional cultures.
5. Conclusion
Our intention in this paper has been to catalyze thought and debate among researchers, innovators,
project leaders, funding bodies, regulators and reviewers about what evidence and practices they
would associate with exemplary processes and outcomes in RRI that is, to consider what exemp-
lary practice in RRI entails and how we might develop concrete criteria and parameters for eval-
uating it. As the RRI concept is increasingly taken up in policy discourse and attempts are made to
implement it in practice, this concretization of quality conceptions and evaluative criteria is
urgently required.
In this paper, we have presented an example of how this challenging task may be tackled in
practice. Drawing on our work within the TD research community and through a multi-stake-
holder deliberative exercise focused on the case of nanoremediation, we have proposed a set of
quality criteria that usefully integrate demands from across different communities. We have
offered an example of how these general criteria can be further claried through the articulation
of important elements for consideration, and we have elaborated these into performance indicators
and statements that show their gradation across a scale of routine, good, great and exemplary
16 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
practice. On the basis of the proposed rubric, we have shown how projects can be mapped, visu-
ally represented and compared against each other and over time. Finally we have demonstrated
how the rubric can serve the dual purpose of not only evaluating the quality of different nanor-
emediation initiatives but also facilitating a discussion of factors beyond the initiatives themselves
that may work to inhibit the advance of RRI, including institutional, cultural and broad-ranging
socio-economic or political factors.
Through this article we offer our example rubric as the start of an important conversation that
needs to happen within the eld of RRI. Of course, both this rubric and the approach we have
outlined should remain open to evolve and be adapted, critiqued and amended, as appropriate
to different contexts. We specically see scope for different research groups, innovation organiz-
ations, funding bodies and interested stakeholders to engage in analytic-deliberative processes to
create their own criteria, and/or indicators for the quality criteria we present, and to articulate these
statements across an evaluative scale. It will be a particularly interesting question for future
research as to whether and/or how such evaluative criteria, approaches and schema may vary
depending on which aspect of the complex system of innovation is taken as a primary area of
focus (e.g. process versus product) or what specic sector or eld is under the spotlight. Addition-
ally, it would also be relevant for future research to empirically analyze where possible the per-
formative nature of evaluative criteria and standards, including how they necessarily bring about
changes in practice through their very articulation and existence. Of course, to engage in any
deliberative development of evaluative criteria for RRI will require the active creation and enact-
ment of new spaces for dialogue, which need to be supported despite the awareness that they will
open for disagreement and contestation and will likely entail a messy process of negotiation
leading to outcomes involving compromise. Rather than shying away from the task on this
basis, though, we need to embrace such contestation and struggle as central to the democratic
process (Stirling 2014).
If democracy is about contestation and struggle, quality is about tness for purpose. There-
fore, we encourage any group engaging with the emerging concept of RRI to actively engage
the challenge of evaluative processes and to experiment with both the approach of rubric devel-
opment and the concrete details of our own evaluative schema so as to help us advance ongoing
improvement and exemplary practice within RRI.
Both authors particularly acknowledge the detailed and useful work of two anonymous reviews as well as the
productive discussions had with all participants at the Sundvolden workshop and the Oslo Group on Respon-
sible Innovation and the signicant contribution these had for developing the ideas presented in this paper.
The work detailed in this paper was supported by the Norwegian Research Council [Grant No. 203288] and
the EC FP7 project NANoREG [Grant No. 310584]. A. L. C.s participation in the Sundvolden workshop
was also supported by the Australian Learning and Teaching Council (now the Ofce of Learning and
Teaching). Feedback on the RRI rubric was also gathered during an event supported by the University of
Tasmanias internal Teaching and Learning Grants scheme.
Barad, K. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning. Durham: Duke University Press.
Bardach, E. 2000. A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem
Solving. New York: Seven Bridges Press.
Journal of Responsible Innovation 17
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Bensuade Vincent, B. 2014. The Politics of Buzzwords at the Interface of Technoscience, Market
and Society: The Case of Pubic Engagement in Science.Public Understanding of Science 23 (3):
238253. doi:10.1177/0963662513515371
Bergmann, M., B. Brohmann, E. Hoffmann, M. C. Loibl, R. Rehaag, E. Schramm, and J.-P. Voss. 2005.
Quality Criteria for Transdisciplinary Research: A Guide for Formative Evaluation of Research
Projects. Frankfurt: Institute for Social-Ecological Research. Accessed February 25, 2014. http://
Bohr, N. 1949. Discussions with Einstein on Epistemological Problems in Atomic Physics.In Albert
Einstein: Philosopher-Scientist, edited by P. A. Schilpp, 199241. La Salle, IL: Open Court.
Boix Mansilla, V., and H. Gardner. 2003. Assessing Interdisciplinary Work at the Frontier: An Empirical
Exploration of Symptoms of Quality.Interdisciplines. Accessed February 25, 2014. http://www.mtu.
Bozeman, B., and D. Sarewitz. 2005. Public Values and Public Failure in US Science Policy.Science and
Public Policy 32 (2): 119136.
Bozeman, B., and D. Sarewitz. 2011. Public Value Mapping and Science Policy Evaluation.Minerva 49:
123. doi:10.1007/s11024-011-9161-7
Brown, J. 2005. The World Café: Shaping Our Futures Through Conversations That Matter. San Fransico:
Berrett Koehler.
Burgess, J., and J. Chilvers. 2006. Upping the Ante: A Conceptual Framework for Designing and
Evaluating Participatory Technology Assessments.Science and Public Policy 33 (10): 713728.
Burgess, J., A. Stirling, J. Clark, G. Davies, M. Eames, K. Staley, and S. Williamson. 2007. Deliberative
Mapping: A Novel Analytic-Deliberative Methodology to Support Contested Science-Policy
Decisions.Public Understanding of Science 16: 299322. doi:10.1177/0963662507077510
Carew, A. L., and F. Wickson. 2010. The TD Wheel: A Heuristic to Shape, Support and Evaluate
Transdisciplinary Research.Futures 42: 11461155. doi:10.1016/j.futures.2010.04.025
Elghali, L., R. Clift, K. G. Begg, and S. McLaren. 2008. Decision Support Methodology for Complex
Contexts.Engineering Sustainability 161 (1): 722.
EPSRC (Engineering and Physical Sciences Research Council). 2013. Framework for Responsible
Innovation.Accessed February 10, 2014.
Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, and M. Trow. 1994. The New Production
of Knowledge: The Dynamics of Science and Research in Contemporary Societies. London: Sage.
Glerup, C., and M. Horst. 2014. Mapping Social Responsibilityin Science.Journal of Responsible
Innovation 1 (1): 3150. doi:10.1080/23299460.2014.882077
Groves, C., L. Frater, R. Lee, and E. Stokes. 2011. Is There Room at the Bottom for CSR? Corporate Social
Responsibility and Nanotechnology in the UK.Journal of Business Ethics 101: 525552. doi:10.1007/
Grunwald, A. 2011. Responsible Innovation: Bringing Together Technology Assessment, Applied Ethics
and STS Research.Enterprise and Work Innovation Studies 7: 931. Accessed October 2, 2014.
Guston, D. 2000. Between Politics and Science: Assuring the Integrity and Productivity of Research.
Cambridge: Cambridge University Press.
Jacob, K., J. Van Den Hoven, L. Nielsen, F. Roure, L. Rudze, J. Stilgoe, K. Blind, A.-L. Guske, and
C. Martinez Riera. 2013. Options for Strengthening Responsible Research and Innovation: Report of
the Expert Group on the State of the Art in Europe on Responsible Research and Innovation.Brussels:
European Commission.
Jahn, T., M. Bergmann, and F. Keil. 2012. Transdisciplinarity: Between Mainstreaming and
Marginalization.Ecological Economics 79: 110. doi:10.1016/j.ecolecon.2012.04.017
Komiyama, H., K. Takeuchi, H. Shiroyama, and T. Mino, eds. 2011. Sustainability Science: A Mulitdisciplinary
Approach. Tokyo: United Nations University Press.
Kuzma, J., and Kuzhabekova, A. 2011. Corporate Social Responsibility for Nanotechnology Oversight.
Medicine, Health Care and Philosophy 14 (4): 407419.
Labuschagne, C., A. C. Brent, and R. P. G. Erck. 2005. Assessing the Sustainability Performances of
Industries.Journal of Cleaner Production 13: 373385. doi:10.1016/j.jclepro.2003.10.007
Linkov, I., F. K. Satterstrom, J. Steevens, E. Ferguson, and R. C. Pleus. 2007. Multi-Criteria Decision
Analysis and Environmental Risk Assessment for Nanomaterials.Journal of Nanoparticle Research
9: 543554. doi:10.1007/s11051-007-9211-0
18 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Linkov, I., and T. P. Seager. 2011. Coupling Multi-Criteria Decision Analysis, Life-Cycle Assessment and
Risk Assessment for Emerging Threats.Environmental Science and Technology 45: 50685074.
Marton, F. 1986. Phenomenography A Research Approach Investigating Different Understandings of
Reality.Journal of Thought 21 (3): 2849.
Mitrany, M., and D. Stokols. 2005. Gauging the Transdisciplinary Qualities and Outcomes of Doctoral
Training Programs.Journal of Planning Education and Research 24: 437449. doi:10.1177/
Oftedal. G. 2014. The Role of Philosophy of Science in Responsible Research and Innovation (RRI): The
Case of Nanomedicine.Life Sciences, Society and Policy 10 (5): 112.
Owen, R., J. Stilgoe, P. Macnaghten, M. Gorman, E. Fisher, and D. Guston. 2013. A Framework for
Responsible Innovation.In Responsible Innovation: Managing the Responsible Innovation of
Science and Innovation in Society, edited by R. Owen, J. Bessant, and M. Heintz, 2750. London:
John Wiley.
Pennington, D. D., G. L. Simpson, M. S. McConnell, J. M. Fair, and R. J. Baker. 2013. Transdisciplinary
Research, Transformative Learning, and Transformative Science.BioScience 63 (7): 564573.
Petrie, J., L. Basson, P. Notten, and M. Stewart. 2004. Multi-criteria Decision Analysis: The Case of
Power Generation in South Africa.In Sustainable Development in Practice: Case Studies for
Engineers and Scientists, edited by A. Azapagic, S. Perdan, and R. Clift, 367396. London: John
Wiley & Sons.
Pohl, C., P. Perrig-Chiello, B. Butz, G. Hirsch Hadorn, D. Joye, R. Lawrence, M. Nentwich, et al. 2011.
Questions to Evaluate Inter- and Transdisciplinary Research Proposals. Berne: td-net for
Transdisciplinary Research.
Slade, C. 2011. Public Value Mapping of Equity in Emerging Nanomedicine.Minerva 49: 7186. doi:10.
Spaapen, J. F., F. Wamelink, and H. Dijstelbloem. 2003. Towards the Evaluation of Transdisciplinary
Research.InInterdisciplinary and Transdisciplinary Landscape Studies: Potential and
Limitations, edited by B. Tress, G. Tress, A. Van Der Valk, and G. Fry, 148159. Wageningen: Delta
Series 2.
Stilgoe, J., R. Owen, and P. Macnaghten. 2013. Developing a Framework for Responsible Innovation.
Research Policy 42 (9): 15681580.
Stirling, A. 2014. Emancipating Transformations: From Controlling The Transitionto Culturing Plural
Radical Progress.Climate Engineering Governance Working Paper Series 012. Accessed July 25, 2014.
Székely, F., and M. Knirsch. 2005. Responsible Leadership and Corporate Social Responsibility: Metrics
for Sustainable Performance.European Management Journal 23 (6): 628647.
The Norwegian Ministries. 2012. Strategi. Regjeringens FoU-strategi for nanoteknologi 20122021 [The
Governments R&D Strategy for Nanotechnology 20122021]. Oslo: Nærings- og handelsdepartementet.
The Norwegian Ministry of Education and Research. 2011. Strategi 20112020. Nasjonal strategi for
bioteknologi. Oslo: Kunnskapsdepartementet.
Ulrich, W. 2000. Reective Practice in the Civil Society: The Contribution of Critically Systemic
Thinking.Reective Practice 1 (2): 247268.
Von Schomberg, R. 2013. A Vision of Responsible Innovation.In Responsible Innovation: Managing
the Responsible Innovation of Science and Innovation in Society, edited by R. Owen, J. Bessant, and
M. Heintz, 5174. London: John Wiley.
WCED (World Commission on Environment and Development). 1987. Our Common Future. Accessed
September 19, 2014.
Wickson, F. 2014. Watching the Regulatory Tail Wag the Scientic Dog.Research Europe 390: 7.
Accessed October 2, 2014.
Wickson, F., A. L. Carew, and A. W. Russell. 2006. Transdisciplinary Research: Characteristics, Quandaries
and Quality.Futures 38 (9): 10461059.
Ziman, J. 1996. Is Science Losing Its Objectivity?Nature 382: 751754. doi:10.1038/382751a0
Zwart, H., L. Landeweerd, and A. Van Rooij. 2014. Adapt or Perish? Assessing the Recent Shift in the
European Research Funding Arena from ELSAto RRI.Life Sciences, Society and Policy 10 (11):
Journal of Responsible Innovation 19
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Notes on contributors
Fern Wickson is a cross-disciplinary scholar interested in the governance of emerging technologies for sus-
tainable and resilient socio-ecological futures. Trained in both the natural and social sciences, her award-
winning interdisciplinary research ranges across such topics as environmental philosophy and ethics, the
politics of risk and uncertainty, and the theory and practice of cross-disciplinary research.
Anna L. Carew is a wine scientist leading distributed industry trials investigating chemical and sensory out-
comes from novel grape processing technologies. Anna is also an award-winning textbook writer and reci-
pient of an Australian Government Citation for her contribution to engineering education. She publishes and
reviews in engineering education, academic development, transdisciplinarity, wine science and sustainability
20 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
... While this has dominated the policy discourse, including innovation policies (Fitjar, Benneworth, and Asheim 2019), it also applies to innovation and entrepreneurial activities. These activities can be impactful if entrepreneurs and firms are responsive to stakeholders' concerns, needs, values and expectations (Owen, Macnaghten, and Stilgoe 2012;von Schomberg 2013;Wickson and Carew 2014). Also, the RI discourse argues that public deliberation and responsiveness must be prioritised to deal with 'questions of uncertainty, motivations, social and political constitutions, trajectories and directions of innovation' Macnaghten 2013, 1570). ...
... While social interaction is the base to build OC (Gemmell, Boland, and Kolb 2012), the dimensions of RI principles can play an instrumental role. Thus, they may help evaluate socio-economic, socio-ecological and socio-ethical factors to build OC that enables sustainable and responsible venture creation and firm development (Long and Blok 2018;Lubberink et al. 2019;Wickson and Carew 2014;Voegtlin et al. 2022). ...
... They can build OC in venture creation and firm development, thus increasing firm and entrepreneurial success (Scholten and Van Der Duin 2015;Brand and Blok 2019). Moreover, the RI framework plays an instrumental role in interacting with socioeconomic, socio-ethical, and socioecological factors (Long and Blok 2018;Lubberink et al. 2019;Wickson and Carew 2014;Voegtlin et al. 2022) critical to building OC in venture creation and firm development (Gemmell, Boland, and Kolb 2012). ...
Full-text available
The increasing adaptation of digitalisation has engendered numerous venture ideas and entrepreneurial opportunities. Many of these ideas bear the potential to address grand societal challenges. However, perceived opportunities can be elusive, especially in the context of complex problems. Opportunity confidence (OC) can be essential to venture creation and firm development. OC depends on evaluating socioeconomic, socio-ethical, and socioecological factors, which are not straightforward. Responsible innovation (RI) can be a viable approach to building OC. However, whether or how firms and entrepreneurs pursue RI to build OC needs to be clarified. Accordingly, we explore these issues through a case study of nine for-profit firms in digital healthcare and welfare services. The findings reveal that although the firms do not integrate RI principles in innovation and entrepreneurial activities per se, they practice them to varying degrees. This helps them to build OC. The study contributes to theory, practice, and policy.
... Incorporating stakeholder viewpoints into technology development has been recommended as a means of enhancing stakeholder confidence in the innovation process (Asveld et al. 2015). As examples of different approaches, Wickson and Carew (2014) cited the purposeful use of transdisciplinary processes, publicly and actively seeking critical feedback, and enabling transformational mutual learning. Citizen panels, focus groups, lay representation of governance groups, and user-centred design are all techniques that promote inclusion ). ...
Technical Report
Full-text available
JUST2CE Project Deliverable 3.2. H2020-SC5-2020-2 scheme, grant agreement number 101003491, A Just Transition to the Circular Economy
... Incorporating stakeholder viewpoints into technology development has been recommended as a means of enhancing stakeholder confidence in the innovation process (Asveld et al., 2015). As examples of different approaches, transdisciplinary procedures, openly soliciting critical input, and facilitating transformative mutual learning are all characteristics of inclusive practices highlighted by Wickson and Carew (2014). User-centred design, focus groups, citizen panels, and governance with a layperson's perspective are also techniques that promote inclusion . ...
... One cannot defend expenditures of public money without somehow measuring or monitoring the effectiveness of the funded activities. Also scholars within a typical aRRI discourse have argued for the reasonableness of defining success criteria and performance indicators for RRI (Wickson and Carew, 2014;Yaghmaei and van den Poel, 2020). However, the tension in how to understand the content of RRI-summarized in the aRRI/pRRI-distinction-was reproduced in the debates on how to evaluate RRI, even within the attempts taken by DG RTD to clarify the issue. ...
Full-text available
Introduction By a series of calls within the Horizon 2020 framework programme, the EU funded projects intended to deploy Responsible Research and Innovation (RRI) at a territorial level, in regional research and innovation ecosystems. This paper presents efforts to document and evaluate the achievements in TRANSFORM, one of these projects. Methods Evaluative inquiry and theoretical reasoning. Results Noting the need for a general principle to be interpreted, adapted and translated in order to be rendered meaningful at a local level, we studied precisely these multiple territorial translations of RRI, the organizational and institutional orderings with which they co-emerge and the challenges that come with these translations. An important shared feature is that RRI work does not start from zero, but rather builds on pre-existing relationships and repertoires of collaboration. The RRI project is hence a way to continue ongoing work and follow pre-set purposes, aims and objectives, as a form of “maintenance work”. In this very human sense, RRI is deployed with a logic of care in the regional context, while the Horizon 2020 calls and proposals above all are formulated in a logic of choice, to be assessed by indicators. Discussion We warn against undue standardization of RRI by toolification and use of quantitative indicators, and recommend that RRI performance is monitored by methods of evaluative inquiry.
Sustainability‐related developments become a differentiating factor in development processes. An area with a strong focus on sustainability‐related issues is packaging. In current packaging development processes, stakeholders focus on solving sustainability issues within their own boundaries. However, the complexities surrounding circular packaging can only be overcome by transdisciplinary collaboration. ‘Traditional’ collaboration shows to be incapable of overcoming packaging‐specific complexities. Therefore, we launch Packalicious, a research initiative aiming to establish transdisciplinary innovation as a collaborative learning ecosystem. In the initial research phase, a core stakeholder group developed the framework in which Packalicious operates. In the second (current) phase, the developed Packalicious framework is tested and improved. This design iteration builds on a transdisciplinary group‐based case study, where real‐life packaging challenges are tackled by diverse stakeholders. In this paper, we define and measure the efficacy of collaborative learning within Packalicious by means of three innovation indicators. The first results indicate that the approach yields more diverse solutions, and a positive connotation with on‐the‐spot transdisciplinary collaboration. However, it also exposes the differences in discipline‐related language and jargon. This paper contributes to academic insights by the establishment of a self‐sustaining transdisciplinary learning ecosystem, and the ways in which this bridges gaps between disciplines and stakeholders.
Purpose Responsible innovation assessment tools (RIATs) are key instruments that can help organizations, associations and individuals measure responsible innovation. Accordingly, this study aims to review the current status of research on responsible innovation and, in particular, of studies that either present the relevance of RIATs or provide empirical evidence of their adoption. Design/methodology/approach A systematic literature review is conducted to identify and review how RIATs are being addressed in academic research and the applications that are proposed. A systematic process is implemented using the Web of Science and Scopus bibliographic databases, aiming not only to summarize existing studies, but also to include a perspective on gaps and future research. Findings A total of 119 publications were identified and included in the review process. The study identifies that RIATs have attracted growing interest from the scientific community, with a greater predominance of studies involving qualitative and mixed methods. A well-balanced mix of conceptual and exploratory studies is also registered, with a greater predominance of analysis of RIATs application domains in the past years, with greater incidence in the finance, water, energy, construction, manufacturing and health sectors. Originality/value This study is pioneering in identifying 16 dimensions and 60 sub-dimensions for measuring responsible innovation. It also suggests the need to include multidimensional perspectives and individuals with interdisciplinary competencies in this process.
In the past decades, controversies in science and technology have fuelled the call for a research and innovation system that is open to public scrutiny and sensitive to public needs and values. These calls have been translated into a framework for responsible research and innovation (RRI). Despite its crucial role in RRI, the dimension of responsiveness is still considered to be the least conceptualized. In this paper, we analyse the mutual search for the meaning of `responsiveness' during a multi-stakeholder dialogue on responsiveness in nanotechnology innovation. We reconstructed five frames of responsiveness: science and technology, market, expert, network, and society-driven responsiveness. We highlighted and reflected on the three main strategies for dealing with differences: avoidance, polarization, and reframing. On the basis of our findings, we argue that in order to generate a collective meaning of responsiveness with a clear direction, conversations are needed in which fundamental assumptions can be questioned.
Full-text available
Lab spaces can teach STEM ethics by highlighting and creating opportunities to practice the four key dimensions of the AIRR framework of Responsible Innovation (RI): anticipation, inclusion, responsiveness, and reflexivity. In this paper, we introduce ‘Futures Labs’∼ as a pedagogical approach to train students in RI skills and encourage them to become aware of, and more concerned about, the broad social, political, ethical, and environmental dimensions of innovation. Our approach additionally trains students in a wide range of explicitly employable capacities (such as scenario building, foresight, communication, and collaboration skills). We conceive of the Futures Lab as a part of a broader permeation of RI-oriented educational practices within STEM institutions. Through practical experience in Futures Labs, students learn not only how to apply RI frameworks, but also get a feel for why doing so matters. The paper concludes with a practically-oriented discussion of exercises employed at each institution's Futures Lab.
Full-text available
Research on ethical, legal and social aspects (ELSA) of life sciences and new technologies has mainly been focused on impacts and consequences, while the emerging framework of Responsible Research and Innovation (RRI) focuses rather on increased involvement and reflexivity in research processes to foster science and technology that better answers the needs of society. I argue that philosophy of science should be a central feature of RRI and demonstrate how the philosophy of science can contribute in this sense. I show how investigating basic assumptions in research, here exemplified by reductive assumptions in causal modeling, can have important ethical and societal implications.
Full-text available
This article employs the Foucauldian notion of ‘political rationality’ to map discussions and ideals about the responsibility of science toward society. By constructing and analyzing an archive of 263 journal papers, four political rationalities were identified: the Demarcation rationality, which aims to exclude the social from the scientific production in order to make it objective and thereby responsible; the Reflexivity rationality, which sees it as science's responsibility to let itself be guided by problems in society in choice of research focus and methods; the Contribution rationality, which insists that responsible science should live up to public demands for innovation and democracy; and the Integration rationality, which advocates that science should be co-constructed with societal actors in order to be socially responsible. While each rationality is distinct, the article argues that all of them address the issue of a boundary (or integration) between science and society. Hence, it is not possible for scientists to avoid ‘a relationship’ with society. The political question is how this relationship is to be defined and regulated.
Full-text available
2.1 Introduction Few would disagree that science and innovation should be undertaken responsibly. "Responsible innovation" intuitively feels right in sentiment, as an ideal or aspiration. It has positive, constructive overtones, where science and innovation are directed at, and undertaken towards, socially desirable and socially acceptable ends, with connotations of trust and integrity. However, in reality, it lacks definition and clarity, both in concept and practice: What might it involve? Who might it involve? When might it be applied? In this chapter we explore these questions, proposing a framework for responsible innovation and highlighting some examples of its translation into practice. Before doing this we first need to provide some context. Why is there a need for a framework for responsible innovation, and what are the deficits of our current approach to innovation governance? In this chapter we will begin by emphasizing that science and innovation have not only produced understanding, knowledge, and value (economic, social, or otherwise), but also questions, dilemmas, and unintended (and sometimes undesirable) impacts. This is well understood. Some impacts, such as those associated with the financial crisis of 2008, have been both profound and global in nature, see Muniesa and Lenglet, Chapter 8. They have highlighted inefficiencies, and even failures, in the principle of regulation by market choice in liberal economies, which struggles, Responsible Innovation, First Edition. Edited by
This book combines political-economic, sociological and historical approaches to provide a coherent framework for analysing the changing relationship between politics and science in the United States. Fundamental to this relationship are problems of delegation, especially the integrity and productivity of sponsored research: politicians must see that research is conducted with integrity and productivity, and scientists must be able to show it. A science policy regime changes when solutions to these problems change. After World War II, the 'social contract for science' assumed that the integrity and productivity of research were automatic and, despite many challenges, that contract endured for four decades. However in the 1980s, as rich empirical studies show, cases of misconduct in science and flagging economic performance broke the trust between politics and science. New 'boundary organizations', in which scientists and nonscientists collaborate to assure the integrity and productivity of research, were created to mend the relationship.
How does one ascertain the quality of interdisciplinary work, when criteria from individual disciplines do not suffice? Assessment is one of the most important and least understood aspects of interdisciplinary research. An empirical study of interdisciplinary work in five established research institutions, reveals that experts prioritize peer review, journal prestige, citation patterns, and successful patent filing, as indicators of quality interdisciplinary work, while also viewing these indicators with skepticism. Three epistemic criteria to assess interdisciplinary work are revealed: (1) the degree to which new insights relate to antecedent disciplinary knowledge in multiple disciplines involved, (2) the sensible balance reached in weaving disciplinary perspectives together, and (3) the effectiveness with which the integration of disciplines advances understanding and inquiry. These criteria may inform the task of reviewers and evaluators of interdisciplinary research outcomes by attending to some of the unique epistemic demands that this type of work presents.
In October 2013, the UK Engineering and Physical Sciences Research Council (ESPRC) made a formal policy commitment to a framework for responsible innovation. I describe the development of this framework over a four-year period since 2009 and key inputs which shaped its evolution and framing. I conclude with some thoughts about future directions, as EPSRC itself transitions from understanding what RI means, to how it could be meaningfully implemented.