Content uploaded by Fern Wickson
Author content
All content in this area was uploaded by Fern Wickson on Oct 14, 2014
Content may be subject to copyright.
This article was downloaded by: [Fern Wickson]
On: 14 October 2014, At: 03:11
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Responsible Innovation
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/tjri20
Quality criteria and indicators for
responsible research and innovation:
Learning from transdisciplinarity
Fern Wicksonab & Anna L. Carewc
a GenØk Centre for Biosafety, Forskningsparken, PB 6418, Tromsø
9294, Norway
b Oslo Group on Responsible Innovation, Oslo and Akershus
University College of Applied Sciences, Postboks 4, St. Olavs plass,
NO-0130 Oslo, Norway
c Tasmanian Institute of Agriculture, University of Tasmania,
Private Bag 98, Hobart, TAS 7001, Australia
Accepted author version posted online: 24 Sep 2014.Published
online: 08 Oct 2014.
To cite this article: Fern Wickson & Anna L. Carew (2014): Quality criteria and indicators for
responsible research and innovation: Learning from transdisciplinarity, Journal of Responsible
Innovation, DOI: 10.1080/23299460.2014.963004
To link to this article: http://dx.doi.org/10.1080/23299460.2014.963004
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or
howsoever caused arising directly or indirectly in connection with, in relation to or arising
out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
RESEARCH ARTICLE
Quality criteria and indicators for responsible research and innovation:
Learning from transdisciplinarity
Fern Wickson
a,b
*and Anna L. Carew
c
a
GenØk Centre for Biosafety, Forskningsparken, PB 6418, Tromsø 9294, Norway;
b
Oslo Group on
Responsible Innovation, Oslo and Akershus University College of Applied Sciences, Postboks 4, St. Olavs
plass, NO-0130 Oslo, Norway;
c
Tasmanian Institute of Agriculture, University of Tasmania, Private Bag 98,
Hobart, TAS 7001, Australia
(Received 26 February 2014; accepted 5 September2014)
The concept of ‘responsible innovation’(RI) or ‘responsible research and innovation’(RRI) is
rapidly gaining currency. However, a persistent critique is that without more concrete
elaboration, the interpretive flexibility of the concept is so broad as to effectively render it
meaningless. The articulation of quality criteria and indicators therefore seems crucial for
RRI to be understood and operationalized by researchers, research funders, innovators and
other relevant stakeholders. In this paper, we specifically draw on our knowledge and
experience from the transdisciplinary research community, combined with recent multi-
stakeholder deliberative work on the concrete case of nanoremediation, to make an offering
on the challenge of articulating quality criteria and approaches to evaluate RRI. In doing so,
we present an iteratively arrived at set of quality criteria, designate significant elements of
each of these, and then develop an evaluative rubric of performance indicators across them.
While the criteria and rubric we present were initiated through the specific context of our
work on advancing RI in the research, development and use of nanoparticles for
environmental remediation, we believe that they can serve as a useful example for how
evaluative criteria and approaches can be developed and offer a helpful frame for
sponsoring and structuring the ongoing conversations on quality criteria and indicators that
are necessary if RRI is to reach its full potential.
Keywords: responsible innovation; transdisciplinary research; nanoremediation; evaluation;
quality
1. Introduction: the need for quality criteria and indicators
The concept of ‘responsible innovation’(RI) or ‘responsible research and innovation’(RRI) is
rapidly gaining currency in European policy discourse as well as in international scholarship
on the governance of new and emerging technologies. This rising emphasis on having the knowl-
edge economy develop ‘responsibly’is arguably the latest manifestation of a longer historical
trend reimagining the relationship between science and society in ways that depart from the tra-
ditional ‘linear model’or ‘received view’(Guston 2000). This historical development of attempts
to re-imagine the science–society relationship and indeed, to enact it in new ways, has been
described by Stilgoe, Owen, and Macnaghten (2013) as observable through the development
of practices such as technology assessment in its various forms, the increasing institutionalization
© 2014 Taylor & Francis
*Corresponding author. Email: fern.wickson@genok.no
Journal of Responsible Innovation, 2014
http://dx.doi.org/10.1080/23299460.2014.963004
Downloaded by [Fern Wickson] at 03:11 14 October 2014
of public engagement, the embedding of research on ethical, legal and social aspects into large
technology development initiatives, and an increasing use of socio-technical integration and mid-
stream modulation. In a similar vein but emphasizing fields of practice rather than particular
methods or approaches, Grunwald (2011) has suggested that RRI extends a general model and
philosophy of technology assessment by incorporating perspectives of applied ethics and
science and technology studies. Despite a sense in which RRI is therefore an amalgamation, cul-
mination, or latest manifestation of various activities and fields of practice, a singular universally
accepted definition of RRI has yet to fully crystallize.
Various actors and initiatives have recently proposed definitions for RI/RRI (e.g. see EPSRC
2013; Jacob et al. 2013; Owen et al. 2013; von Schomberg 2013), and while these definitions may
not completely converge, they do have overlapping features that suggest a loosely agreed set of
core characteristics. The proposed definitions differ in the terminology they use, the orientation
they adopt, the depth of description they provide and where they place emphasis. However,
certain characteristics are shared across them and can therefore be identified as central to the emer-
ging concept. These include:
(1) A focus on addressing significant socio-ecological needs and challenges;
(2) A commitment to actively engaging a range of stakeholders for the purpose of substan-
tively better decision-making and mutual learning;
(3) A dedicated attempt to anticipate potential problems, assess available alternatives and
reflect on underlying values, assumptions and beliefs; and
(4) A willingness among all participants to act and adapt according to these ideas.
Although not specifically mentioned by Stilgoe, Owen, and Macnaghten (2013) in their overview
of the historical roots grounding the concept of RI, the core characteristics identified above also
have strong resonance with the characteristics of transdisciplinary (TD) research. Transdiscipli-
narity specifically emerged as an attempt to operationalize the sustainability imperative that
arose from the attainment of international consensus in the 1990s on the need for co-operative,
proactive action on global environmental and social challenges (e.g. Brundtlandt Report)
(WCED 1987). TD research has been characterized as involving (see Wickson, Carew, and
Russell 2006; Carew and Wickson 2010):
(1) A focus on complex and multidimensional real-world problems;
(2) Transcending academic boundaries to incorporate collaboration and mutual learning
between a range of researchers and relevant stakeholders; and
(3) An iteratively evolving method that reflects on the problem and its context from a range
of perspectives and responds accordingly.
A recent report on options for strengthening RRI, under the auspices of the European Commission
(Jacob et al. 2013), specifically highlights TD research (and the related approach of sustainability
science) as potentially important components of RRI. However, one of the specific obstacles
identified in the report is that there are no common quality standards or criteria for these types
of research that can serve as a useful guide for researchers, project managers or research
funding organizations. For those working within sustainability science and TD research, the
lack of appropriate quality criteria has been recognized as a significant problem for some time
(e.g. see Komiyama et al. 2011; Jahn, Bergmann, and Keil 2012), and over the last decade a
number of attempts have been made to address it (Boix Mansilla and Gardner 2003; Mitrany
and Stokols 2005; Spaapen, Wamelink, and Dijstelbloem 2003;Bergmann et al. 2005; Carew
and Wickson 2010; Pohl et al. 2011).
2F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Evaluation criteria and schemas that have been proposed for TD research vary between those
advocating close attention to the institutional and interpersonal preconditions for successful TD
research, those more focused on the importance of good process, and those encouraging close
attention to the final products or outcomes of the research. There is also substantial variation
in the balance different authors achieve between comprehensiveness and over-prescription. For
example, Bergmann et al. (2005) offered a particularly complex and time-intensive evaluative
schema, while the prompts suggested by Boix Mansilla and Gardener (2003) were provocative
as general guidance but were arguably not rich enough to support the full-scale design and
execution of integrative research. One of the core challenges is that given TD is an innately tai-
lored and necessarily responsive approach to research, any quality criteria need to be uniquely
flexible to provide valid, reliable means to evaluate and compare projects, while not stifling
the evolution and responsiveness of the approach.
Beyond what has been taking place in the TD research community, in recent years a range of
different analytical and reporting methods have emerged to try and meet demands for responsi-
bility and responsiveness in the operations and decision-making of public and private entities.
Such schemas include triple bottom line sustainability reporting within Corporate Social Respon-
sibility frameworks (CSR) (e.g. see Labuschagne, Brent, and Erck 2005; Szekely and Knirsch
2005), Public Value Failure Mapping within science policy analysis (PVM)) (e.g. see Bozeman
and Sarewitz 2005,2011) and formalized processes of stakeholder consultation through
methods of multi-criteria decision analysis (MCDA) (e.g. see Petrie et al. 2004; Burgess et al.
2007; Elghali et al. 2008). Each of these approaches has also been considered and discussed in
relation to nanotechnology as a particular case of interest (see Groves et al. (2011) and Kuzma
and Kuzhabekova (2011) for CSR; Linkov et al. (2007) and Linkov and Seager (2011) for
MCDA; and Slade (2011) for PVM). Each of these fields of work has recognized the need for
and sought to prescribe processes for evaluative analysis and reporting, although each with its
own particular area of emphasis and applicability.
For example, CSR is a concept that is focused on corporate actors and has traditionally prior-
itized reporting frameworks for assessing the environmental, economic and social dimensions of
the company’s contribution to sustainability (e.g. using reporting frameworks such as that of the
Global Reporting Initiative or international standards such as ISO 14000 and/or 26000). PVM, on
the other hand, has specifically focused on public actors engaged in science policy processes and
has developed criteria for mapping when and how these policies fail to articulate, meet and
advance social or public values. In contrast, MCDA represents a constellation of different
methods for specifically enhancing stakeholder consultation in the decision-making of either
public or private actors and has been particularly focused on managing tradeoffs in high-stakes
decisions shrouded in scientific uncertainty and subject to contested social values.
While each of the existing fields of CSR, PVM and MCDA clearly shares an overlapping ter-
ritory of interest with the emerging frameworks and definitions of RRI, again, RRI appears to be a
nascent umbrella concept seeking to bring together and unite the different areas of focus, dimen-
sions of interest and actors of attention present across these other areas; for example, combining
interests in sustainability, social values and stakeholder engagement, and seeking applicability
across both public and private actors engaged in research and innovation and their governance
and regulation. However, just as each of the fields mentioned above has recognized and been
working with the significant challenge of providing quality criteria, indicators and assessment
methods to improve performance (either in corporate sustainability, the public value of science
policies or stakeholder involvement in decision-support), a common or easy critique of the emer-
ging umbrella notion of RRI is that it remains overly vague (e.g. see Bensuade Vincent 2014;
Oftedal 2014) and that without concrete elaboration and conceptual development, the interpretive
flexibility of RRI will be so broad as to render the concept meaningless. Indeed, as Zwart and
Journal of Responsible Innovation 3
Downloaded by [Fern Wickson] at 03:11 14 October 2014
colleagues see it (Zwart, Landeweerd, and van Rooij 2014), the term RRI was introduced as a
signifier in a top-down manner before the concept signified by it had a clear and stable profile,
and therefore the research community is now being explicitly invited to populate the term with
meaning.
The articulation of relevant quality criteria and indicators, and particularly those that can unite
and encompass previously disparate areas of interest and investigation, therefore seems to be a
crucial task to lend the notion of RRI meaning and allow it to be understood and operationalized
by researchers, research funders, innovators and other relevant stakeholders. The task of develop-
ing quality criteria and indicators for RRI, however, meets with similar problems to those that
have been facing these related fields over recent decades. For example, one significant challenge
is clearly the need to answer an initial question concerning whether the focus should be on eval-
uating the preconditions,process,orproducts of research and innovation (or perhaps even the
people populating all three). Furthermore, any emerging quality criteria need to be concrete
enough to provide real guidance but flexible enough to adapt to the specificities of varying
contexts.
In this paper, we specifically draw on our knowledge and experience in the TD research com-
munity and field to make an offering on the challenge of developing quality criteria for RRI that
we hope is a useful starting point on this issue and one that can helpfully prompt and facilitate
ongoing discussion, development and debate. We do this by first proposing a set of defining
quality criteria that emerged through an iterative, multi-stakeholder process that we conducted
on the particular case of RRI in nanoremediation. We then identify some potentially significant
elements to help denote and elaborate the content of each of these criteria, before finally demon-
strating how these can be developed into a qualitative evaluative rubric of performance indicators
for RRI. We consider the rubric we present as an outcome developed within our particular context
of research (the relatively narrow case of nanoremediation) drawing on our related area of exper-
tise (TD research). However, we believe that the presentation and description of this approach to
quality criteria and the evaluation of RRI, can serve as a useful starting point for an important
conversation that we think needs to take place if the notion of RRI is to reach its full potential.
We therefore particularly welcome interactions, alterations, critique and further elaboration of
both the rubric approach and the evaluative criteria and indicators for RRI that we propose and
present herein.
2. Method: iterative development of the RRI quality criteria and indicators
The quality criteria and indicators for RRI that we propose were derived through a series of events
and activities over several months. These events and activities were aimed at elucidating through
conversation and deliberation with a range of stakeholders what criteria might be used to under-
stand, explain, judge and approach ‘RRI’in the context of the development and use of nanopar-
ticles for environmental remediation (nanoremediation).
In the first instance, the process of developing the schema was informed by our reading and
interpretation of emerging work published on the defining characteristics of RRI, as well as the
concept’s relationship to related terms and fields, including general ideas concerning ‘good
science’and ‘good technology’and more specific literature on concepts of quality that have
been proposed in fields such as ‘transdisciplinarity’and ‘sustainability science’. From this con-
ceptual and theoretical basis, our next step was to design and run an international and interdisci-
plinary workshop with a focus on articulating RRI quality criteria and understanding how they
may play out when applied to a particular case (nanoremediation). This workshop was held in
May 2013 at Sundvolden, Norway as part of a project on responsible ecological governance of
emerging technologies (funded by the Norwegian Research Council) and also to contribute to
4F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
the NANoREG project to develop a common approach to the regulatory testing of nanomaterials
(funded by the European Commission).
The workshop gathered 17 participants from different fields but all with expertise working
with either the specific issue of nanoremediation or the environmental governance of nanotech-
nology. These participants came from eight different countries (Norway, Denmark, Australia,
Brazil, England, Czech Republic, Germany and the USA) and were brought together to
discuss, debate and attempt to advance RRI in the field of nanoremediation. The gathered
group included actors specialized in (with some actors holding more than one nominated area
of specialized expertise): nanoecotoxicology (seven individuals working within academia
across four different institutions and three national contexts: Norway, Denmark and the Czech
Republic), environmental chemistry (four individuals working within two institutional and
national contexts in Australia and Norway), environmental remediation technologies (four indi-
viduals, two working in the corporate sector, in Germany and the Czech Republic respectively,
one working in academia in Denmark and one working across academia and the corporate
sector in the Czech Republic), environmental policy and/or science and technology governance
(three individuals working primarily within academia but distributed across three institutions in
Brazil, the UK and Australia) and environmental philosophy (one individual working within aca-
demia in the USA). Approximately one third of the invited participants were graduate students or
early career researchers, while the remaining members were senior staff with a high status in their
field or institution. This generational mix was considered crucial given the emerging nature of
nanotechnology and our explicit focus on building responsible futures.
With expertise across more than one of the listed fields and a status arguably somewhere
between senior and junior levels, the authors interchanged positions throughout the different
activities of the event to act as both facilitators and participants in the workshop. Acting as facil-
itators and participants was both challenging and interesting as it served to blur power relation-
ships within the process. This embedding of ourselves in the activity of co-creation of RRI
criteria serves to illustrate a point of difference between more structured consultation approaches
like MCDA, where stakeholder views are elicited by a supposedly neutral facilitator, and the TD
approach used in this study which attempted some democratization of the research process, some
deconstruction of power relations within the negotiation of criteria and an intimate engagement of
the researchers in the research context (Carew and Wickson 2010).
Given the well-recognized and documented problem of attaining conditions able to facilitate
situations resembling those of Habermas’ideal speech in multi-stakeholder deliberative exercises
(e.g. see Burgess and Chilvers 2006), participants in our workshop were intentionally not invited
to declare their position and status in an introductory ‘round the table’. This was due to our inten-
tion to disrupt the power relationship that often impinges on the free exchange of ideas in aca-
demic discourse. Rather, in the first event of the workshop, participants were introduced to one
another through a World Café-style activity (see Brown 2005). In this activity, they generated
answers (first individually, then in small mixed groups, and finally in plenary) to prompts such
as: ‘“The universe is made of stories, not of atoms”Agree/Disagree?’,‘I feel confident in my
knowledge when …’,‘Science is …’,‘Good Science is …’,‘Good technology is …’, and
‘How would you describe the relationship between humanity and nature?’Using these
prompts and their discussion as a way to introduce participants to one another was informed
by a phenomenographic enquiry approach, which aims to ‘map the qualitatively different ways
in which people conceptualize, perceive, and understand various aspects of …the world
around them’(Marton 1986, 31). Following small group and plenary discussions on each of
the questions, we developed the participants’responses into an ‘outcome space’that mapped
the range of qualitatively different ways in which some of the fundamental issues and underlying
beliefs that we opened for discussion were conceived and held by the actors present. Outcome
Journal of Responsible Innovation 5
Downloaded by [Fern Wickson] at 03:11 14 October 2014
space posters were then put on display around the meeting room to help inform the workshop’s
subsequent activities and conversations; they proved particularly useful as the workshop
progressed when competing conceptions on these issues could be linked to variation in the
assumptions and objectives that each participant brought to later tasks, such as that focused on
co-developing quality criteria for RRI.
Interestingly, the World Café introductions revealed that participants gathered for the work-
shop were evenly divided on the question of whether they saw the world as made of atoms or
stories, indicating a significant divergence in basic ontologies (e.g. whether participants were
more committed to materialism or idealism in their philosophical views and approaches to the
world). Perhaps unsurprisingly, the World Café activities also saw significant divergence in
relation to the question of science. Some participants held that science was essentially explora-
tion, while others thought that it was a specific and highly structured process. Some felt that
it was a process that delivered reliable and elegant answers, while others emphasized that it
was a process that necessarily and characteristically led to new questions. Some participants
painted science in opposition to religion and, at the same time, others characterized it as the reli-
gion of the twenty-first century. When it came to the question of what constituted ‘good science’
and how this differed from ‘science’in these first-impression discussions, some participants felt
that good science needed to be socially relevant, seriously consider its limitations and be open to
interrogation. Others felt that it was simply that which was performed by virtuous scientists or
that it essentially involved following a high-quality process.
After canvassing the diversity of the workshop group and its members’underlying views
and assumptions in these introductory conversations, it was fascinating to be part of the
group navigating the values landscape that the process had unearthed when we later asked
the group to focus on the specific task of developing quality criteria and indicators for RRI. Par-
ticipants who believed that the world was made of atoms conversed openly with colleagues for
whom the world was constructed through stories; participants who originally saw good science
as purely defined by good process happily negotiated quality criteria with those wanting science
to deliver socially relevant answers; and those who held science to be the main counterpoint to
religion sought consensus positions with participants who held science to be the twenty-first
century’s religion. We believe that using the World Café approach as a way to introduce
participants to each other helped to render explicit the diversity of the group and –by
making this diversity transparent from the very beginning –allowed for greater acceptance
of and willingness to negotiate across the range of views and positions that emerged on the pur-
poses and processes of RRI. As postulated by Ulrich (2000), this laying bare of participants’
fundamentally different views and values at the outset of a co-operative process appeared to
build understanding and facilitate respectful and constructive dialogue throughout the later
workshop tasks.
On the second day of the Sundvolden workshop, participants focused on the specifictaskof
developing quality criteria and indicators for RRI. For this task, we assembled participants into
three mixed-disciplinary groups and asked them to nominate the range of criteria they saw as
relevant for defining ‘good science’. The full range of criteria was then shared, discussed and
debated in plenary. We then asked each group to select one draft criterion that had been ident-
ified and work together to develop a rubric of performance for that criterion. Rubrics are a
higher education assessment format for which standards of performance are spelled out
along a gradation that would constitute poor performance against the criterion, to the charac-
teristics or indicators of exemplary performance. While the Sundvolden participants were
mostly unfamiliar with rubrics, they embraced the task and worked to develop a gradation
scale for the criterion that their group had selected. The three criteria selected by the different
groups were ‘honesty’,‘diversity’and ‘social relevance’. The range of criteria originally
6F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
articulated and the draft standards of performance that were developed for the three of these
formed the first iteration of the emerging quality criteria and indicators for RRI that are the
subject of this paper.
In the months following the Sundvolden workshop, the authors reflected on how the work-
shop outcomes related to the literature on both TD research quality and the emerging concept
of RRI and its norms. We conversed and debated the emerging list of potential RRI quality criteria
in several meetings with each other, as well as discussed various iterations with other interested
researchers and colleagues working in diverse fields (e.g. rural sociology, marine ecology, agri-
cultural economics, science and technology studies, ethics and organizational management).
This process of reflection, debate, explanation and discussion developed the Sundvolden out-
comes into a supplemented and extended set of criteria. We then advanced the Sundvolden
work by developing more elaborated indicators of performance for each criterion and formatting
these into a draft rubric for RRI. Finally, we sought feedback on this draft rubric through: (a) infor-
mal discussion with colleagues in a range of disciplines and with RRI theorists and practitioners;
(b) presenting the criteria and rubric at a session on RI at the international conference of the
Society for the Study of Nanoscience and Emerging Technologies (S.Net); and (c) using the
draft rubric as a prompt for group work at a professional development workshop on evaluating
TD undergraduate science learning. Following this phase of gathering feedback on the draft
rubric, the authors added two additional criteria to the list, and edited and further developed
the rubric into Table 1.
2.1. Qualifying statement
Throughout the course of developing the RRI rubric presented in this paper, we consistently
encountered the question of whether we were developing a schema to evaluate research or inno-
vation. Adopting a Mode 2 (Gibbons et al.1994) or post academic (Ziman 1996) view, we found
it impractical, and indeed increasingly unhelpful, to try and separate research from innovation. We
therefore rejected the notion of a sharp separation between research and innovation. The rubric
presented here is therefore oriented toward ‘RRI’. While this may be criticized as creating a
beast that is far too large and diverse an entity to usefully engage and practically manage, we
believe that the systemic entanglement of these activities is so extensive in late modernity that
to create an artificial separation between them would do more harm than confronting the beast
directly.
A similar conundrum we identified in the introduction to this paper was whether evaluative
criteria for RRI and related concepts (e.g. TD research, sustainability science) –which are
rich, complex and systemic in practice –should focus on institutional and interpersonal precondi-
tions for success, the importance of good process, and/or the final products or outcomes of the
research. We agreed that any product developed through an innovation process will inevitably
enter a complex web of interactions of use, the outcomes of which will be inherently uncertain.
Given this, we soon lost faith with the sense of mandating that a particular innovation product
could or should be evaluated against the emerging norms. We therefore chose to focus more
directly on developing evaluative criteria for RRI as a process.
Despite making this choice, however, we found ourselves consistently drawn back into dis-
cussing the criteria as they related to products and felt a persistent need to acknowledge the impor-
tance of contextual preconditions and the characteristics of people actually conducting the
process. Through this struggle, we have come to realize that in focusing on the evaluation of a
process of innovation, one still needs the capacity to also consider existing preconditions, envi-
saged products and engaged people, since all of these elements shape, guide and, to some extent,
generate and characterize the RRI processes that the schema is primarily designed to evaluate.
Journal of Responsible Innovation 7
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Here we inadvertently walked in the footsteps of Bohr (1949) and Barad (2007) in accepting
the impossibility of a sharp separation between process and product. Bohr’s ideas of complemen-
tarity would allow that these two phenomena offer unique and necessary, but potentially contra-
dictory, insights into the overarching focus –the generation of novel knowledge objects. Barad’s
concept of ‘agential realism’extended Bohr’s validation of complementary-but-discordant con-
ceptualizations by suggesting that the separation and naming of concepts like process and
product may be a cultural artifact of our need to isolate aspects of complex phenomena for the
purposes of examination and understanding. Hence, although we chose to orient ourselves pri-
marily toward the concept of process, we also acknowledged the value and place of product, pre-
conditions and people as complementary to understanding RRI. In short, although the schema we
offer is primarily oriented toward evaluating RRI as a process, it also contains elements that relate
to preconditions, products and people. It is perhaps worth noting that a schema primarily oriented
toward one or other of these later elements may well look different to the one we propose here (e.
g. see Glerup and Horst 2014).
Finally, we wish to comment upon how some of our own positions, beliefs and assumptions
might have influenced our framing of the activities and outcomes that resulted. For example, one
of us (Wickson) is a member of the board of two different programs of the Norwegian Research
Council. Recent government commitment to responsible technology development (see The Nor-
wegian Ministry of Education and Research 2011; The Norwegian Ministries 2012), without the
articulation of a specificdefinition or set of evaluative criteria, has created a significant problem of
how to operationalize RRI within this context, including within the prioritization of fields in
research program planning and individual research project funding. This experience from practice
can be seen to have created a desire to arrive at a criteria set and approach that could be utilized
within a research council setting. Furthermore, that each of us has been committed to the norm of
sustainability across our research careers and personal lives, as well as has a history in teaching
ethics to students in the natural sciences, may also be seen to have informed a willingness to unite
these disparate fields and encapsulate both in our list of criteria. This is not to say that we con-
sciously set out with these intentions, but rather that we have come to see their potential influence
when we have reflected upon our own role in facilitating and performing the deliberative and
iterative aspects of our method. We lay them bare here as both evidence of the importance of
reflexive learning in participatory research and a reminder that although we hope that it has
broader usefulness, what follows has emerged from a particular and unique set of contextual
conditions.
3. Results: a rubric for RRI
Based on the method described above, we propose seven paired and interwoven quality criteria
for RRI. We propose that RRI can be evaluated according to whether it is
.Socially relevant and Solution oriented
.Sustainability centered and Future scanning
.Diverse and Deliberative
.Reflexive and Responsive
.Rigorous and Robust
.Creative and Elegant
.Honest and Accountable
These criteria effectively unite demands from different communities concerning what ‘good’
science and ‘responsible’research and innovation should entail. For example, the criteria of
8F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
‘creative and elegant’and ‘rigorous and robust’are features that are traditionally valued within the
scientific community and are already used to assess the quality of scientific projects. That is, they
are factors that scientists themselves typically emphasize as important quality criteria for evalu-
ating research and development. The characteristics of ‘honesty and accountability’are already
emphasized in the field of research ethics as important if science and innovation are to develop
responsibly and link to broader calls for transparency as an ethical value. The criteria for research
and innovation to be ‘diverse and deliberative’and ‘reflective and responsive’are powerful norms
in the field of science and technology studies and a key feature of MCDA approaches, while a
demand for innovation to be ‘sustainability centered and future scanning’has emerged in
response to global environmental concerns and movements and is often emphasized within
CSR practices. Finally, the sense that RI should be ‘socially relevant and solution oriented’
echoes the values of PVM and is an emerging demand within science policy arenas and, arguably,
also one that industry is also often oriented toward. By combining quality criteria from across
different fields and communities, our proposed criteria unite these varying demands under a
common umbrella of RRI. This unification is crucial to achieve acceptance and uptake of the
concept of RRI since it enables these different fields of practice and communities of stakeholders
to be able to identify their existing norms within the conceptual framework but also to see how
these become extended under the concept of RRI.
While the criteria listed above make a start on articulating what good RRI might entail, the
terms used remain somewhat opaque and open to varying interpretations. Although some scholars
have recently suggested that this vagueness is exactly what gives ‘buzzwords’like RRI their cur-
rency across different groups of actors and arenas (Bensaude Vincent 2014), as indicated above,
in this paper we have chosen to develop an inclusive group of defining quality criteria as a way to
try and unite different fields of practice and communities of stakeholders, but we are also specifi-
cally interested in developing more concrete elaboration so as to avoid the concept’s being a
walking shadow, full of talk and promise but signifying nothing (to paraphrase Shakespeare).
Of course some may view our proposed criteria as confounded and prefer to separate them,
pair them differently or (re)interpret them in the context of their own evaluative tasks and projects.
We believe that in any process of interpreting, rearranging and/or applying these criteria, open
negotiation on their meaning and application would only positively foster transformative learning,
which is a valued outcome of good TD processes (Pennington et al. 2013).
Within our own context, an intermediate step toward developing indicators of performance for
each paired set of criteria involved identifying relevant elements of each to give them enhanced
definitional clarity and meaning. In the case and process we were working within, we arrived at
the elements listed below as important components for consideration under each of the criteria. It
is entirely possible, however, that alternative elements for each criterion could (and arguably
should) be developed in specific relation to different contextual situations and by different
actors, and ideally through deliberative processes involving a range of relevant stakeholders to
the identified problem and context. The elements we present below are, however, those that we
used to delimit the criteria from each other and guide the development of performance indicators
in our own rubric.
.Socially relevant and Solution oriented
(a) Type of problem addressed
(b) Type of solution sought
.Sustainability centered and Future scanning
(a) Anticipating potential futures
(b) Identifying potential risks and benefits
(c) Considering social, economic and environmental sustainability
Journal of Responsible Innovation 9
Downloaded by [Fern Wickson] at 03:11 14 October 2014
.Diverse and Deliberative
(a) Level of cross-disciplinarity involved
(b) Where stakeholders are involved
(c) How stakeholders are involved
.Reflexive and Responsive
(a) Recognition of preconditions in context and group
(b) Exploration of underlying values, assumptions and choices
(c) Openness to critical scrutiny
(d) Ability to change after internal reflective practice and external feedback
.Rigorous and Robust
(a) Aspects of the problem considered
(b) Repeatability across actors and settings
(c) Reliability of outcomes under real-world conditions
.Creative and Elegant
(a) Novelty and daring
(b) Sufficiency and beauty
.Honest and Accountable
(a) Identification of uncertainties and limitations
(b) Lines of delegation and ownership
(c) Compliance with research ethics and governance requirements
(d) Policies on open access and information sharing
(e) Ownership over positive and negative outcomes
The elements listed under each criterion assisted the process of developing the quality criteria into an
RRI performance rubric. We used these elements to guide the development and refinement of state-
ments that explained what could be considered routine,good,great and exemplary examples of
practice against each quality criterion. The outcome is presented in Tabl e 1 as a rubric for assessing
RRI.
4. Discussion: use of the RRI rubric
The rubric we have developed for RRI and presented in Table 1 is intended to represent a useful
approach to clarify what is expected by and of different parties in the RRI process, and to provide
inspiration, concrete guidelines and direction for improvement for those seeking to innovate in
more responsible ways. An RRI rubric such as that presented here can, however, also be used
to evaluate and compare the quality of RRI in different projects and initiatives, or be used to
evaluate progress in a single project over time.
The development of rubrics such as this would offer much needed clarity on what good
quality RRI looks like. This clarity is urgently needed for:
(a) research funding bodies that need to compare and decide between different project appli-
cations in terms of the likely contribution each might make to meeting and advancing
RRI,
(b) researchers and/or innovators needing to articulate either their capacity to undertake and/
or report on their ongoing work toward RRI, so as to satisfy institutional, public, policy
or funding body demands, and
(c) researchers, innovators and project managers looking to continually improve the per-
formance of their work over time.
10 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Table 1. A rubric for assessing RRI.
Criteria Exemplary Great Good Routine
Socially relevant
and Solution
oriented
Addressing a grand social
challenge.
Ongoing analysis of objectives
and processes to favor the
delivery of ‘wicked solutions’
(solving multiple challenges
simultaneously)
Addressing a significant social
need.
Ongoing analysis of objectives
and processes to maintain a
focus on delivering a
successful solution
Focused on a marginal or self-
defined problem.
Employing processes aimed at
generating insights toward a
solution, or a partial solution
Pursuing a purely personal
interest.
Possibility that process/product
will only result in the creation
of decontextualized knowledge
or new problems
Sustainability
centered and
Future
scanning
Inclusion of formal processes of
future casting to at various
points throughout the research
and innovation process.
Generating a range of positive and
negative future scenarios and
identifying and assessing
associated risks and benefits of
these for social, environmental
and economic sustainability.
Clear avenues for embedding
responses to these possible
futures and risk/benefit
assessments into the project
development
Inclusion of future casting
activities at some point during
the research and innovation
process.
Some attempt to integrate an
assessment of the risks and
benefits for social,
environmental and economic
sustainability.
Identifiable points and
possibilities for adaptation of
the process to respond to the
future scanning and risk/
benefit assessment activities
Informal attempts to future cast at
limited points in the project.
A consideration of some
associated risks and benefits in
terms of one or more of the three
dimensions of sustainability.
Little indication of how the
research and innovation process
may adapt and respond to either
identified possible futures or
their risks to sustainability
A singular optimistic prognosis
for future project outcomes
with no clear effort to identify
risks or survey possible future
scenarios
Diverse and
Deliberative
Openly and actively seeking
ongoing critical input, feedback
and feed-forward from a range
of stakeholders.
Encouraging and rewarding
transformative mutual learning.
Employing an evolving
integrative method and
consciously employing a TD
process
Inviting, incorporating and
integrating stakeholder views
at various points along the
research and innovation
process.
Actively seeking dialogic
interaction with stakeholders
and open to mutual learning.
Encompassing a wide range of
methods and adopting an
interdisciplinary process
Limited stages of the research and
innovation process open for
stakeholder engagement.
Tendency toward one-way forms
of communication with
stakeholders but open to some
interaction.
Involving some level of
methodological diversity and
multidisciplinary practice
Communicating with
stakeholders only toward the
end of the research and
innovation process.
Use of one-way communication
approaches and defensive in
the face of counter-views or
stakeholder questions.
Mono-methodological and mono-
disciplinary
(Continued)
Journal of Responsible Innovation 11
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Table 1. Continued.
Criteria Exemplary Great Good Routine
Reflexive and
Responsive
Clear and explicit identification of
institutional and contextual
limitations and a structured
effort to acknowledge and
improve upon these conditions.
Structured, purposeful periodic
analytical review of underlying
values, assumptions and
choices.
Actively seeking critical feedback
from a wide group of sources
and actors.
Evidence of potential to adapt at a
range of points in response to
in-train reflective practice and
external review/input/feedback
Explicit effort to identify
institutional and contextual
limitations and awareness of
their significance for practice.
Occasional use of structured
process for reflecting on
underlying values,
assumptions and choices.
Actively seeking critical
feedback from select sources
and actors.
Clear indications of a capacity to
adapt in response to reflective
practice and external feedback
Some indication of awareness
concerning limitations posed by
institutional structure and
contextual realities.
Informal, one-off or ad hoc process
for considering underlying
values, assumptions and
choices.
Accepting critical feedback when
offered.
Stated willingness to accept
change in response to internal
reflective practice or external
review and critique
No explicit consideration or
recognition of the limitations
posed by institutional structure
and contextual realities.
No process for facilitating
reflective practice.
No critique sought.
No evidence for potential for
change in response to criticism/
unsolicited feedback
Rigorous and
Robust
Comprehensive investigation of
all aspects of the problem and
the interconnections between
them.
Results repeatable by a variety of
different actors operating across
a range of relevant conditions.
Outcomes work reliably under
real-world conditions
Considering multiple
dimensions of the problem
and their interrelations.
Results repeatable by the same
actors operating under a range
of relevant conditions.
Outcomes with demonstrated
functionality under real-world
conditions
Interest in several dimensions of
the problem although not
necessarily their interrelations.
Results repeatable by the same
actors operating under similar
conditions.
Outcomes remain untested under
real-world conditions
Narrow focus on one element or
aspect of a problem.
Results not able to be replicated.
Outcomes unable to be reliably
applied in real-world contexts
Creative and
Elegant
The problem has been reframed in
innovative directions, with new
ideas being pursued through
appropriate methods.
Resources are carefully
considered and allocated to
efficiently achieve maximum
utility and impact.
Esthetical consideration is given
to preconditions, process and
products
New methods are being
developed according to new
ideas within an established
problem framing.
The use of resources is explicitly
justified.
Esthetical consideration is given
to preconditions and products
New ideas are being pursued
through established methods
within an accepted problem
framing.
Considerable resources are
inefficiently employed.
Esthetical consideration is given to
envisaged products
Problem framing, ideas and
methods fall within established
paradigms.
Extensive resources (e.g. time,
money, personnel, etc.) are
dedicated to work with
minimal significance or
potential impact.
No consideration is given to the
esthetics of operating
preconditions, research and
innovation process or
envisaged products
12 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Honest and
Accountable
Transparent identification of a
range of uncertainties and
limitations that may be relevant
for various stakeholders.
Openly communicated lines of
delegation and ownership able
to respond to process dynamics
and contextual change.
Documented compliance with
highest-level governance
requirements, research ethics
and voluntary codes of conduct,
all actively monitored
throughout.
Consistent use of open access to
information policies.
Preparedness to accept
accountability for both
potential positive and negative
impacts
Identification of uncertainties
and limitations deemed to be
significant by those involved.
Established lines of delegation
and ownership.
Compliance with governance
requirements and research
ethics with evidence of active
monitoring throughout.
Favoring open access to
information policies.
Willingness to accept
accountability for potential
positive and negative impacts
Some statement indicating
uncertainties and limitations.
Indications of potential lines of
delegation and ownership.
Complying with minimum
standards of governance
requirements and research
ethics.
Occasionally employing open
access to information policies
Willingness to accept
accountability for positive
impacts and some negative
impacts
No transparency concerning
limitations and uncertainties.
Untraceable ownership of
components.
No specific acknowledgement of
standards concerning
governance requirements or
research ethics.
No demonstrated commitment to
open access information
policies.
Accountability only accepted for
positive outcomes
Journal of Responsible Innovation 13
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Rubrics such as that shown in Table 1 could be used by funding bodies, researchers, innova-
tors or managers in much the same way that rubrics in higher education are used to assess and
provide feedback on student work. In that case, the assessor generally marks which performance
standard (routine, good, great, exemplary) the piece of work has demonstrated against each cri-
terion, and the student is provided the marked rubric to understand how their performance was
rated. Alternatively, funding bodies, researchers, innovators and project managers may wish to
visualize and compare performance against the criteria listed in Table 1. Such a visual represen-
tation may be easier to digest and discuss, especially in the case where multiple projects are under
consideration or a project is reporting against the criteria over time. Figure 1 shows a spider chart
method used to map several projects according to the proposed RRI criteria, and Figure 2 shows a
spider chart of performance against the criteria at two time periods in the life of an RRI initiative
or project.
What the spider charts in Figures 1 and 2illustrate is that certain projects and initiatives will
likely occupy different spaces across the range of RRI criteria proposed here, emphasizing certain
aspects perhaps to the neglect of others. It seems unlikely to us, and perhaps unreasonable to
expect, that any particular project or initiative would be able to reach an exemplary standard
across all the RRI criteria proposed. Reaching an exemplary standard for all criteria, however,
may not be necessary for a project to still deliver significant value and be legitimately recognized
as RRI (e.g. see the third example below). Such visualization processes may be particularly useful
for highlighting and facilitating deliberation about the necessary tradeoffs across the criteria that
could be involved. This could also be conducted using the type of tradeoff analysis typically per-
formed within policy settings (e.g. see Bardach 2000) or it could be taken up in and elaborated
through processes and methods such as MCDA in which the weighting of the different criteria
could be usefully experimented with and expounded upon (by both experts and other stake-
holders). Compromises and tradeoffs are inevitable and while any negotiation of the relative
weighting of such quality criteria will always be contested and often interest based, having a
clear set of criteria and evaluative scales at least enables this process to be an increasingly trans-
parent one. That said, it might also be the case that if the same tradeoffs were systematically being
made and shown in the spider charts (e.g. within the portfolio of research within a company or
research council program) then this may indicate a need to consider rebalancing actions.
Spider charts of RRI could also work to identify obstacles that may be inhibiting the ability of
actors to reach exemplary standards of performance in RRI. In this way, the proposed rubric
would help to identify limitations and opportunities across different arenas for improvement to
Figure 1. Comparing projects.
14 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
the design and conduct of research and innovation. If used such that actors felt free to honestly
identify where a particular RRI effort sat according to the evaluative criteria and to openly
discuss why this was the case, the rubric and associated spider charting process could facilitate
the identification of broader socio-economic, political or institutional factors and cultures that cur-
rently constrain individual actors, projects or institutions from reaching the (ideal) exemplary
standard for RRI (e.g. see Wickson 2014).
An example of the value of the RRI rubric for catalyzing the identification and open discus-
sion of such constraints occurred during our development of the rubric at the Sundvolden work-
shop referred to earlier. A participant group who were drafting the exemplary performance
standard for the draft criterion of ‘honesty’observed that participating researchers could
clearly identify with the ideal of ‘open and transparent communication’, particularly concerning
uncertainties and limitations. However, the group noted that in practice, current publication cul-
tures inhibited their ability to reach an exemplary practice as it was described in the draft rubric.
To elaborate, nanoecotoxicologists studying the environmental impact of nanoremediation found
it difficult to publish their work when they could not explain the results generated from their
experiments and could therefore not openly express in the public realm (i.e. through published
literature) the uncertainties and limitations they faced when using the existing methodologies
originally established for the testing of chemical compounds for the testing of novel nanomater-
ials. Additionally, the participating researchers explained that expectations from policy-makers
that they deliver clear and certain answers to the question of environmental impacts also
worked against them honestly reporting their uncertainties and limitations. These challenges
did not prohibit the group nominating open and transparent communication as an exemplary stan-
dard. Rather, the statement of this ideal helped them to identify and articulate socio-political and
institutional-level obstacles to their attaining this exemplary level of performance in their own
work and projects.
In a second example from the Sundvolden workshop, both industry actors and nanoecotoxi-
cologists suggested that for their work to meet exemplary standards of what it meant to be ‘rig-
orous and reliable’, they would have to engage in real-world experimental field trials. However,
regulatory uncertainties concerning the use of nanoparticles in environmental remediation made
this difficult, or impossible in some cases. That is, the specificity of the behavior of nanoparticles
to the context and conditions within which they are used meant that laboratory experiments to test
the effectiveness of different options for nanoremediation were never well correlated to how the
materials would actually behave under real-world conditions. Therefore to achieve outcomes that
could reliably work under real-world conditions required testing under real-world conditions.
Figure 2. Comparing timepoints.
Journal of Responsible Innovation 15
Downloaded by [Fern Wickson] at 03:11 14 October 2014
This was, however, suggested as not always being possible, due to either regulatory restrictions on
the deliberate environmental release of free nanoparticles in some nations or due to the uncertain-
ties in the specific regulatory requirements for using nanoparticles in remediation and/or the local
or national political level at which such requirements were established and applied.
In a third example from the same case, nanoremediation was seen to be an example of inno-
vation directly oriented toward addressing a significant social need (i.e. remediating polluted
groundwater systems) and although it was not able to reach an exemplary standard of addressing
a‘grand societal challenge’, it could still be considered to have significant value as a research and
innovation project and potentially be legitimately recognized as RRI. However, when evaluated
according to some of the other criteria, room for improvement was clearly possible. This was the
case, for example, in terms of the ‘diverse and deliberative’criterion where developers of nanor-
emediation technologies present at the Sundvolden workshop identified a clear potential for
tighter integration and enhanced interaction with other scientificfields (such as nanoecotoxicol-
ogy), where working together would help identify and reduce potential risks and advance a
‘benign by design’approach. However, challenges to improved performance against this criterion
included limited opportunities for the different types of actors to find each other, and varying
capabilities in communicating across different scientific cultures and terminologies when they
did. These limitations appear to be related to an emphasis on segregation and separation in edu-
cational practices within science and engineering, and limited opportunities for interdisciplinary
interactions at the institutional level.
In each of these examples from the Sundvolden workshop, asking researchers to consider
nanoremediation initiatives in terms of the emerging quality criteria of the RRI rubric catalyzed
important discussion. Significantly, this discussion unearthed communities of practice and influ-
ence outside the circle of participating researchers and innovators who were influencing the ability
of the participants to achieve exemplary standards of RRI. Working on and with the rubric there-
fore highlighted a broader range of actors and factors that would need to change if truly RRI were
to be achieved and, therefore, that the responsibility for advancing RRI needed to be distributed
across a wide collective. For us, the process of developing the rubric in co-operation with col-
leagues who might one day (need to) apply it to evaluate their own projects demonstrated the
range of benefits it might offer, including this identification of obstacles, barriers and limitations
to the practice of RRI that stem not just from within particular projects or initiatives but are con-
nected to broader and more deeply rooted socio-political factors and institutional cultures.
5. Conclusion
Our intention in this paper has been to catalyze thought and debate among researchers, innovators,
project leaders, funding bodies, regulators and reviewers about what evidence and practices they
would associate with exemplary processes and outcomes in RRI –that is, to consider what exemp-
lary practice in RRI entails and how we might develop concrete criteria and parameters for eval-
uating it. As the RRI concept is increasingly taken up in policy discourse and attempts are made to
implement it in practice, this concretization of quality conceptions and evaluative criteria is
urgently required.
In this paper, we have presented an example of how this challenging task may be tackled in
practice. Drawing on our work within the TD research community and through a multi-stake-
holder deliberative exercise focused on the case of nanoremediation, we have proposed a set of
quality criteria that usefully integrate demands from across different communities. We have
offered an example of how these general criteria can be further clarified through the articulation
of important elements for consideration, and we have elaborated these into performance indicators
and statements that show their gradation across a scale of routine, good, great and exemplary
16 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
practice. On the basis of the proposed rubric, we have shown how projects can be mapped, visu-
ally represented and compared against each other and over time. Finally we have demonstrated
how the rubric can serve the dual purpose of not only evaluating the quality of different nanor-
emediation initiatives but also facilitating a discussion of factors beyond the initiatives themselves
that may work to inhibit the advance of RRI, including institutional, cultural and broad-ranging
socio-economic or political factors.
Through this article we offer our example rubric as the start of an important conversation that
needs to happen within the field of RRI. Of course, both this rubric and the approach we have
outlined should remain open to evolve and be adapted, critiqued and amended, as appropriate
to different contexts. We specifically see scope for different research groups, innovation organiz-
ations, funding bodies and interested stakeholders to engage in analytic-deliberative processes to
create their own criteria, and/or indicators for the quality criteria we present, and to articulate these
statements across an evaluative scale. It will be a particularly interesting question for future
research as to whether and/or how such evaluative criteria, approaches and schema may vary
depending on which aspect of the complex system of innovation is taken as a primary area of
focus (e.g. process versus product) or what specific sector or field is under the spotlight. Addition-
ally, it would also be relevant for future research to empirically analyze where possible the per-
formative nature of evaluative criteria and standards, including how they necessarily bring about
changes in practice through their very articulation and existence. Of course, to engage in any
deliberative development of evaluative criteria for RRI will require the active creation and enact-
ment of new spaces for dialogue, which need to be supported despite the awareness that they will
open for disagreement and contestation and will likely entail a messy process of negotiation
leading to outcomes involving compromise. Rather than shying away from the task on this
basis, though, we need to embrace such contestation and struggle as central to the democratic
process (Stirling 2014).
If democracy is about contestation and struggle, quality is about fitness for purpose. There-
fore, we encourage any group engaging with the emerging concept of RRI to actively engage
the challenge of evaluative processes and to experiment with both the approach of rubric devel-
opment and the concrete details of our own evaluative schema so as to help us advance ongoing
improvement and exemplary practice within RRI.
Acknowledgments
Both authors particularly acknowledge the detailed and useful work of two anonymous reviews as well as the
productive discussions had with all participants at the Sundvolden workshop and the Oslo Group on Respon-
sible Innovation and the significant contribution these had for developing the ideas presented in this paper.
Funding
The work detailed in this paper was supported by the Norwegian Research Council [Grant No. 203288] and
the EC FP7 project NANoREG [Grant No. 310584]. A. L. C.’s participation in the Sundvolden workshop
was also supported by the Australian Learning and Teaching Council (now the Office of Learning and
Teaching). Feedback on the RRI rubric was also gathered during an event supported by the University of
Tasmania’s internal Teaching and Learning Grants scheme.
References
Barad, K. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning. Durham: Duke University Press.
Bardach, E. 2000. A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem
Solving. New York: Seven Bridges Press.
Journal of Responsible Innovation 17
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Bensuade Vincent, B. 2014. “The Politics of Buzzwords at the Interface of Technoscience, Market
and Society: The Case of ‘Pubic Engagement in Science’.”Public Understanding of Science 23 (3):
238–253. doi:10.1177/0963662513515371
Bergmann, M., B. Brohmann, E. Hoffmann, M. C. Loibl, R. Rehaag, E. Schramm, and J.-P. Voss. 2005.
Quality Criteria for Transdisciplinary Research: A Guide for Formative Evaluation of Research
Projects. Frankfurt: Institute for Social-Ecological Research. Accessed February 25, 2014. http://
www.isoe.de/ftp/evalunet_guide.pdf
Bohr, N. 1949. “Discussions with Einstein on Epistemological Problems in Atomic Physics.”In Albert
Einstein: Philosopher-Scientist, edited by P. A. Schilpp, 199–241. La Salle, IL: Open Court.
Boix Mansilla, V., and H. Gardner. 2003. “Assessing Interdisciplinary Work at the Frontier: An Empirical
Exploration of Symptoms of Quality.”Interdisciplines. Accessed February 25, 2014. http://www.mtu.
edu/research/administration/sponsored-programs/enhancement/pdf/assessing-interdisciplinary-work.pdf
Bozeman, B., and D. Sarewitz. 2005. “Public Values and Public Failure in US Science Policy.”Science and
Public Policy 32 (2): 119–136.
Bozeman, B., and D. Sarewitz. 2011. “Public Value Mapping and Science Policy Evaluation.”Minerva 49:
1–23. doi:10.1007/s11024-011-9161-7
Brown, J. 2005. The World Café: Shaping Our Futures Through Conversations That Matter. San Fransico:
Berrett Koehler.
Burgess, J., and J. Chilvers. 2006. “Upping the Ante: A Conceptual Framework for Designing and
Evaluating Participatory Technology Assessments.”Science and Public Policy 33 (10): 713–728.
Burgess, J., A. Stirling, J. Clark, G. Davies, M. Eames, K. Staley, and S. Williamson. 2007. “Deliberative
Mapping: A Novel Analytic-Deliberative Methodology to Support Contested Science-Policy
Decisions.”Public Understanding of Science 16: 299–322. doi:10.1177/0963662507077510
Carew, A. L., and F. Wickson. 2010. “The TD Wheel: A Heuristic to Shape, Support and Evaluate
Transdisciplinary Research.”Futures 42: 1146–1155. doi:10.1016/j.futures.2010.04.025
Elghali, L., R. Clift, K. G. Begg, and S. McLaren. 2008. “Decision Support Methodology for Complex
Contexts.”Engineering Sustainability 161 (1): 7–22.
EPSRC (Engineering and Physical Sciences Research Council). 2013. “Framework for Responsible
Innovation.”Accessed February 10, 2014. http://www.epsrc.ac.uk/research/framework/Pages/
framework.aspx
Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, and M. Trow. 1994. The New Production
of Knowledge: The Dynamics of Science and Research in Contemporary Societies. London: Sage.
Glerup, C., and M. Horst. 2014. “Mapping ‘Social Responsibility’in Science.”Journal of Responsible
Innovation 1 (1): 31–50. doi:10.1080/23299460.2014.882077
Groves, C., L. Frater, R. Lee, and E. Stokes. 2011. “Is There Room at the Bottom for CSR? Corporate Social
Responsibility and Nanotechnology in the UK.”Journal of Business Ethics 101: 525–552. doi:10.1007/
s10551-010-0731-7
Grunwald, A. 2011. “Responsible Innovation: Bringing Together Technology Assessment, Applied Ethics
and STS Research.”Enterprise and Work Innovation Studies 7: 9–31. Accessed October 2, 2014.
http://run.unl.pt/bitstream/10362/7944/1/Grunwald9-31.pdf
Guston, D. 2000. Between Politics and Science: Assuring the Integrity and Productivity of Research.
Cambridge: Cambridge University Press.
Jacob, K., J. Van Den Hoven, L. Nielsen, F. Roure, L. Rudze, J. Stilgoe, K. Blind, A.-L. Guske, and
C. Martinez Riera. 2013. Options for Strengthening Responsible Research and Innovation: Report of
the Expert Group on the State of the Art in Europe on Responsible Research and Innovation.Brussels:
European Commission.
Jahn, T., M. Bergmann, and F. Keil. 2012. “Transdisciplinarity: Between Mainstreaming and
Marginalization.”Ecological Economics 79: 1–10. doi:10.1016/j.ecolecon.2012.04.017
Komiyama, H., K. Takeuchi, H. Shiroyama, and T. Mino, eds. 2011. Sustainability Science: A Mulitdisciplinary
Approach. Tokyo: United Nations University Press.
Kuzma, J., and Kuzhabekova, A. 2011. “Corporate Social Responsibility for Nanotechnology Oversight.”
Medicine, Health Care and Philosophy 14 (4): 407–419.
Labuschagne, C., A. C. Brent, and R. P. G. Erck. 2005. “Assessing the Sustainability Performances of
Industries.”Journal of Cleaner Production 13: 373–385. doi:10.1016/j.jclepro.2003.10.007
Linkov, I., F. K. Satterstrom, J. Steevens, E. Ferguson, and R. C. Pleus. 2007. “Multi-Criteria Decision
Analysis and Environmental Risk Assessment for Nanomaterials.”Journal of Nanoparticle Research
9: 543–554. doi:10.1007/s11051-007-9211-0
18 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Linkov, I., and T. P. Seager. 2011. “Coupling Multi-Criteria Decision Analysis, Life-Cycle Assessment and
Risk Assessment for Emerging Threats.”Environmental Science and Technology 45: 5068–5074.
doi:10.1021/es100959q
Marton, F. 1986. “Phenomenography –A Research Approach Investigating Different Understandings of
Reality.”Journal of Thought 21 (3): 28–49.
Mitrany, M., and D. Stokols. 2005. “Gauging the Transdisciplinary Qualities and Outcomes of Doctoral
Training Programs.”Journal of Planning Education and Research 24: 437–449. doi:10.1177/
0739456X04270368
Oftedal. G. 2014. “The Role of Philosophy of Science in Responsible Research and Innovation (RRI): The
Case of Nanomedicine.”Life Sciences, Society and Policy 10 (5): 1–12.
Owen, R., J. Stilgoe, P. Macnaghten, M. Gorman, E. Fisher, and D. Guston. 2013. “A Framework for
Responsible Innovation.”In Responsible Innovation: Managing the Responsible Innovation of
Science and Innovation in Society, edited by R. Owen, J. Bessant, and M. Heintz, 27–50. London:
John Wiley.
Pennington, D. D., G. L. Simpson, M. S. McConnell, J. M. Fair, and R. J. Baker. 2013. “Transdisciplinary
Research, Transformative Learning, and Transformative Science.”BioScience 63 (7): 564–573.
Petrie, J., L. Basson, P. Notten, and M. Stewart. 2004. “Multi-criteria Decision Analysis: The Case of
Power Generation in South Africa.”In Sustainable Development in Practice: Case Studies for
Engineers and Scientists, edited by A. Azapagic, S. Perdan, and R. Clift, 367–396. London: John
Wiley & Sons.
Pohl, C., P. Perrig-Chiello, B. Butz, G. Hirsch Hadorn, D. Joye, R. Lawrence, M. Nentwich, et al. 2011.
Questions to Evaluate Inter- and Transdisciplinary Research Proposals. Berne: td-net for
Transdisciplinary Research.
Slade, C. 2011. “Public Value Mapping of Equity in Emerging Nanomedicine.”Minerva 49: 71–86. doi:10.
1007/s11024-011-9163-5
Spaapen, J. F., F. Wamelink, and H. Dijstelbloem. 2003. “Towards the Evaluation of Transdisciplinary
Research”.InInterdisciplinary and Transdisciplinary Landscape Studies: Potential and
Limitations, edited by B. Tress, G. Tress, A. Van Der Valk, and G. Fry, 148–159. Wageningen: Delta
Series 2.
Stilgoe, J., R. Owen, and P. Macnaghten. 2013. “Developing a Framework for Responsible Innovation.”
Research Policy 42 (9): 1568–1580.
Stirling, A. 2014. “Emancipating Transformations: From Controlling ‘The Transition’to Culturing Plural
Radical Progress.”Climate Engineering Governance Working Paper Series 012. Accessed July 25, 2014.
http://www.geoengineering-governance-research.org/perch/resources/workingpaper12stirlingemancipatingt
ransformations.pdf
Székely, F., and M. Knirsch. 2005. “Responsible Leadership and Corporate Social Responsibility: Metrics
for Sustainable Performance.”European Management Journal 23 (6): 628–647.
The Norwegian Ministries. 2012. Strategi. Regjeringens FoU-strategi for nanoteknologi 2012–2021 [The
Government’s R&D Strategy for Nanotechnology 2012–2021]. Oslo: Nærings- og handelsdepartementet.
The Norwegian Ministry of Education and Research. 2011. Strategi 2011–2020. Nasjonal strategi for
bioteknologi. Oslo: Kunnskapsdepartementet.
Ulrich, W. 2000. “Reflective Practice in the Civil Society: The Contribution of Critically Systemic
Thinking.”Reflective Practice 1 (2): 247–268.
Von Schomberg, R. 2013. “A Vision of Responsible Innovation.”In Responsible Innovation: Managing
the Responsible Innovation of Science and Innovation in Society, edited by R. Owen, J. Bessant, and
M. Heintz, 51–74. London: John Wiley.
WCED (World Commission on Environment and Development). 1987. Our Common Future. Accessed
September 19, 2014. http://www.un-documents.net/our-common-future.pdf
Wickson, F. 2014. “Watching the Regulatory Tail Wag the Scientific Dog.”Research Europe 390: 7.
Accessed October 2, 2014. http://www.researchresearch.com/index.php?option=com_news&
template=rr_2col&view=article&articleId=1345017%20
Wickson, F., A. L. Carew, and A. W. Russell. 2006. “Transdisciplinary Research: Characteristics, Quandaries
and Quality.”Futures 38 (9): 1046–1059.
Ziman, J. 1996. “Is Science Losing Its Objectivity?”Nature 382: 751–754. doi:10.1038/382751a0
Zwart, H., L. Landeweerd, and A. Van Rooij. 2014. “Adapt or Perish? Assessing the Recent Shift in the
European Research Funding Arena from ‘ELSA’to ‘RRI’.”Life Sciences, Society and Policy 10 (11):
1–19.
Journal of Responsible Innovation 19
Downloaded by [Fern Wickson] at 03:11 14 October 2014
Notes on contributors
Fern Wickson is a cross-disciplinary scholar interested in the governance of emerging technologies for sus-
tainable and resilient socio-ecological futures. Trained in both the natural and social sciences, her award-
winning interdisciplinary research ranges across such topics as environmental philosophy and ethics, the
politics of risk and uncertainty, and the theory and practice of cross-disciplinary research.
Anna L. Carew is a wine scientist leading distributed industry trials investigating chemical and sensory out-
comes from novel grape processing technologies. Anna is also an award-winning textbook writer and reci-
pient of an Australian Government Citation for her contribution to engineering education. She publishes and
reviews in engineering education, academic development, transdisciplinarity, wine science and sustainability
education.
20 F. Wickson and A.L. Carew
Downloaded by [Fern Wickson] at 03:11 14 October 2014
