ArticlePDF Available

Abstract and Figures

Information Systems (IS) research is often conducted under the assumption that technology use leads to positive outcomes for different stakeholders. However, many IS studies demonstrate limited evidence of having engaged with the stakeholders that they claim benefit and speak on behalf of. It is therefore not surprising that examples abound of where technology use has failed 'to make the world a better place', or worse still has contributed towards unintended negative outcomes. In light of these concerns, calls have recently emerged for responsible research and innovation (RRI) studies in IS to understand how different stakeholder groups can have a voice in complex socio-technical issues. In this paper, we take steps towards addressing this call by presenting case study findings from a responsible IS research project which combined 'blended' face-to-face and online participatory techniques. The case study relates to a large-scale consultation undertaken in a 24-month European project involving 30 countries. The project engaged over 1,500 stakeholders in the co-creation of future research agendas for the European Union. We discuss case study findings using Stilgoe, Owen, & Macnaghten's (2013) RRI framework (Anticipation, Reflexivity, Inclusion, Deliberation, Responsiveness) and reflect on lessons learned for responsible IS research going forward.
Content may be subject to copyright.
ommunications of the
ssociation for
Accepted Manuscript
Accepted Manuscript
Blended Stakeholder Participation for Responsible Information Systems Research
Stephen McCarthy
Department of Business Information Systems,
Cork University Business School,
University College Cork, Ireland.
Wendy Rowan
Health Information Systems Research Centre,
Department of Business Information Systems,
University College Cork, Ireland.
Laura Lynch
Health Information Systems Research Centre,
Department of Business Information Systems,
University College Cork, Ireland.
Ciara Fitzgerald
Department of Business Information Systems,
Cork University Business School,
University College Cork, Ireland.
Please cite this article as: McCarthy, Stephen; Rowan, Wendy; Lynch, Laura; Fitzgerald, Ciara: Blended Stakeholder
Participation for Responsible Information Systems Research, Communications of the Association for Information
Systems (forthcoming), In Press.
This is a PDF file of an unedited manuscript that has been accepted for publication in the Communications of the
Association for Information Systems. We are providing this early version of the manuscript to allow for expedited
dissemination to interested readers. The manuscript will undergo copyediting, typesetting, and review of the resulting
proof before it is published in its final form. Please note that during the production process errors may be discovered,
which could affect the content. All legal disclaimers that apply to the Communications of the Association for
Information Systems pertain. For a definitive version of this work, please check for its appearance online at
ommunications of the
ssociation for
Research Paper ISSN: 1529-3181
Accepted Manuscript
Blended Stakeholder Participation for Responsible
Information Systems Research
Stephen McCarthy
Department of Business Information Systems,
Cork University Business School,
University College Cork, Ireland.
Wendy Rowan
Health Information Systems Research Centre,
Department of Business Information Systems,
University College Cork, Ireland.
Laura Lynch
Health Information Systems Research Centre,
Department of Business Information Systems,
University College Cork, Ireland.
Ciara Fitzgerald
Department of Business Information Systems,
Cork University Business School,
University College Cork, Ireland.
Information Systems (IS) research is often conducted under the assumption that technology use leads to
positive outcomes for different stakeholders. However, many IS studies demonstrate limited evidence of
having engaged with the stakeholders that they claim benefit and speak on behalf of. It is therefore not
surprising that examples abound of where technology use has failed ‘to make the world a better place’, or
worse still has contributed towards unintended negative outcomes. In light of these concerns, calls have
recently emerged for responsible research and innovation (RRI) studies in IS to understand how different
stakeholder groups can have a voice in complex socio-technical issues. In this paper, we take steps
towards addressing this call by presenting case study findings from a responsible IS research project
which combined ‘blended’ face-to-face and online participatory techniques. The case study relates to a
large-scale consultation undertaken in a 24-month European project involving 30 countries. The project
engaged over 1,500 stakeholders in the co-creation of future research agendas for the European Union.
We discuss case study findings using Stilgoe, Owen, & Macnaghten’s (2013) RRI framework (Anticipation,
Reflexivity, Inclusion, Deliberation, Responsiveness) and reflect on lessons learned for responsible IS
research going forward.
Keywords: Citizen participation, stakeholder engagement, co-creation, Delphi study, foresight, scenario
planning, IT ethics, responsible research and innovation; health IT
[Department statements, if appropriate, will be added by the editors. Teaching cases and panel reports will have a
statement, which is also added by the editors.]
[Note: this page has no footnotes.]
This manuscript underwent [editorial/peer] review. It was received xx/xx/20xx and was with the authors for XX months for XX
revisions. [firstname lastname] served as Associate Editor.] or The Associate Editor chose to remain anonymous.]
Communications of the Association for Information Systems
Accepted Manuscript
1 Introduction
“It takes two of us to discover the truth: one to utter it and one to understand it”
~Kahlil Gibran
Technology use has the potential to contribute towards positive economic, social, personal, and
environmental outcomes for different stakeholder groups (cf. Davison et al., 2019; Chen et al., 2017; Turel
et al., 2011; Walsham, 2012). Previous studies have shown how technology use can open up new
channels of communication and give a ‘voice’ to marginalised social groups (Bedeley et al., 2019),
improve standards of healthcare quality (Damberg et al., 2009), and drive process innovation (Bilgeri et
al., 2019). However, for each success story, there are contrasting cases where technology use has also
led to negative outcomes. This includes outcomes such as smartphone addiction (Busch & McCarthy,
2018), increased social control through data monitoring of employees (Howcroft and Trauth, 2005), and
reduced citizen privacy through surveillance capitalism (Zuboff, 2015). These latter examples call into
question the assumption that technology use always benefits society, illuminating IT outcomes that can
instead negatively impact the quality of life for different stakeholders.
In order to understand both the positive and negative consequences of technology use, literature asserts
the need for IS researchers to engage in dialogue with diverse stakeholder groups early in the system
development process to explore direct and indirect effects of use (Chen et al., 2017; Markus and Mentzer,
2014; Poser, Küstermann, & Bittner, 2019). For instance, Markus and Mentzer (2014) note that while
negative consequences are rarely intended, they can be foreseen by IS researchers using analysis
techniques in advance of system building. This requires targeted methods which allow researchers to
constructively engage with stakeholders and gather first-hand insights into how technology might impact
their daily lives. However, to date, many IS studies demonstrate limited evidence of having engaged with
the stakeholders that they claim to benefit and speak on behalf of (cf. Peticca-Harris et al., 2019).
Considering these concerns, IS scholars have recently called for new approaches which enable different
stakeholder groups to express their opinion on complex socio-technical issues (Davison et al., 2019).
Such approaches can potentially help IS researchers maximise the positive consequences, while adhering
to ethical principles which minimise the potential negative consequences of technology use (Markus and
Mentzer, 2014; Someh, Davern, Breidback & Shanks, 2019; Walsham, 2012).
Stakeholder participation approaches are well established in the policy making domain as a means of
engaging different groups in decision-making processes (Cornwall and Coelho, 2007). For instance,
approaches such as Foresight Processes, Scenario Planning, and Delphi Studies offer a means of
engaging different stakeholders in decision-making to gather their diverse views on a topic (cf. Fouché &
Light, 2011; Skinner et al., 2015). In addition, the more recent use of ‘blended’ participation approaches
provide new opportunities for citizens to participate in policy making through a mix of face-to-face and
online techniques (cf. Poser, Küstermann, & Bittner, 2019; Sæbø et al. 2008). These complementary
techniques can broaden the representation of diverse stakeholder groups at different stages of the
consultation process using a mix of co-located workshops, and information and communication
technologies (ICT) enabled consultations (Sæbø et al., 2008).
However, our understanding of responsible research and innovation (RRI) within the IS field remains
nascent (Davison et al., 2019). In particular, further research is needed on how stakeholder participation
approaches can be used for exploring the socio-technical issues which the IS field is synonymous with.
Based on this gap in both IS literature and practice, we seek to answer with the following research
question: How can stakeholder participation approaches be used to support RRI in IS research?
We provide empirical insights into this research question by presenting findings from the case study of a
24-month responsible IS research project which involved partners from 30 countries across Europe and
engaged diverse stakeholder groups. This paper presents findings from the use of a blended approaches
where both face-to-face and online techniques were used to engage citizens, practitioners, and
policymakers during the project.
This paper makes three primary contributions which will be of interest to both academics and practitioners.
Firstly, we present insights into how responsible IS research can be conducted using face-to-face and
online stakeholder participation techniques within a large-scale consultation process. We outline the steps
included in this approach, together with the strengths and weaknesses perceived by the participants.
Secondly, we discuss lessons learned based on a set of criteria from RRI, a domain of study which
Error! Reference source not found.
Accepted Manuscript
concerns the ethics of research and technology development. Our discussion was guided by constructs
taken from the literature, in particular those developed by Stilgoe, Owen, & Macnaghten (2013). These
criteria can inform responsible IS research efforts in the future. Thirdly, we recommend future research
directions for responsible IS research, including studies on user engagement in online consultations,
incentives for stakeholder engagement, and the development of an evaluation framework for guiding
responsible research and innovation in IS going forward. This will be of interest to academics and
practitioners in the IS field who hope to undertake similar blended consultation approaches.
The remainder of the paper is structured as follows: Section 2 provides the background to the paper by
reviewing literature on responsible research and innovation, and stakeholder participation approaches
(Foresight Processes, Scenario Planning, and Delphi Studies). Section 3 then introduces the approach
adopted in the case study which combines face-to-face and online consultation techniques for responsible
IS research. Section 4 presents case study findings and Section 5 discusses these findings as relevant to
academic and policymaking communities. Section 6 brings the paper to a close with a conclusion.
2 Background
2.1 Responsible Research & Innovation (RRI)
Responsible Research and Innovation (RRI) is a domain of study which concerns the ethics of research
and technology development (Stahl, Jirotka & Coeckelbergh, 2014). The aim of RRI is to prevent harm
occurring from research and innovation activities by bridging any perceived knowledge gaps between
stakeholders. This requires a comprehensive understanding of knowledge gaps, starting at the beginnings
of research right up to the point at when individuals or organisations use the outputs of research (Peter,
van der Veen, Doranova & Miedzinski, 2013; Stahl, et al 2014). Von Schomberg (2013, pg. 19) defines
responsible research and innovation as: a transparent, interactive process by which societal actors and
innovators become mutually responsive to each other with a view to the (ethical) acceptability,
sustainability and societal desirability of the innovation process and its marketable product”.
The RRI discourse revolves around three features:
1. Science for society and its impacts on society RRI proposes innovative science that is ethical,
inclusive, democratic and equitable. It aims to open up and realise new areas of public value for
science and innovation (Wilsdon et al., 2005).
2. Science with society and its responsiveness to society RRI seeks the integration and
incorporation of mechanisms such as anticipation, reflection and inclusive deliberation by relevant
the stakeholders of research and the innovation process (Owen, Macnaghten, & Stilgoe, 2012).
3. Reframing responsible research RRI aims to scrutinise potential impacts and risks associated
with emerging technologies in order to ensure accountability (Delpy, 2011). In responding to grand
challenges, RRI also seeks to maintain communication with policymakers at both national and
international levels (Owen et al., 2012).
RRI requires social actors to work together in improving the relationship between research and innovation
processes, and outcomes that meet the needs of society (Geoghegan-Quinn, 2014). By involving a range
of stakeholders (e.g. citizens, practitioners, and policymakers) in the research process, there is a
collective responsibility over the control and direction of outcomes that are ethically acceptable (von
Schomberg, 2012). For instance, social and sustainable innovation can aim to address societal challenges
such as the environmental, ethical, and economic impacts of technology (Bryant et al,, 2009; Lubberink,
Blok, van Ophem & Omta, 2017; Stahl, 2012, 2014).
RRI and the related area of technology assessment have closely aligned motivations including citizen
engagement, interdisciplinary collaborations, socio-technical imagery, and the consideration of wider
impacts from emerging technologies. The use of forecasting techniques to anticipate the potential
consequences of new technologies dates back to the work of Schot & Rip (1997), and more recently
technology assessment has moved towards more participatory and reflexive approaches to promote a
positive impression of technological impact (Genus, 2006). These new approaches place an increased
emphasis on reflection and action early in the development lifecycle to prevent irreversible technological
lock-in (Mingers and Walsham, 2010; Stahl et al., 2014).
The dominant framework for RRI in the literature, put forward by Stilgoe et al. (2013), highlights five
dimensions (see Table 1).
Communications of the Association for Information Systems
Accepted Manuscript
Table 1. Summary of RRI Criteria (adapted from Stilgoe et al. (2013))
RRI Construct
Thinking about the known, likely and possible implications of research
and innovation. This is guided by the involvement and early inclusion of
diverse stakeholder groups.
Stilgoe et al. (2013);
Wickson & Carew
Critical review of one’s own activities and assumptions, recognising
limitations in knowledge and lack of universal applicability of one’s own
outlook. This is important for social actors to acknowledge their role
responsibility and wider moral obligations.
Stilgoe et al. (2013);
Pavie, Scholten &
Carthy (2014).
Representation and engagement from diverse stakeholder groups (e.g.
citizens, practitioners, policymakers) to ensure their views and
perspectives are considered during the participation process.
Stilgoe et al. (2013);
Fitzgerald, et al. (2016);
Enserink & Monnikhof
Utilising different approaches to facilitate discussions and support
participants in investigating the ethical, social and political implications
that the innovation in question could produce.
Stilgoe et al. (2013;
Fitzgerald et al. (2016);
Ianniello et al. (2019)
Promoting the capacity to change and adapt the innovation in reaction to
stakeholder values. This includes the principle of co-responsibility for the
development of research and innovation by all stakeholders.
Stilgoe et al. (2013);
Owen et al. (2013).
Stakeholder participation is essential for preventing such scenarios by expanding our research knowledge
on what is possible and desirable within the wider societal context. The next section looks at stakeholder
participation in more detail.
2.1.1 Stakeholder Participation
The philosophy of RRI promotes the ethos of ‘science with and for society’ (Laroche, 2011; Von
Schomberg, 2013), with stakeholder participation seen as a vital part of the social engagement process.
Stakeholder participation can be traced back in the literature to Arnstein’s (1969) ladder of citizen
participation, with the highest level of citizen involvement resulting in citizens being in full charge of a
specific program or institution. However, since 1990, stakeholder engagement has also expanded to non-
institutional matters such as cooperatives, community enterprises, and services for communities
(Klingemann & Fuchs, 1995; Defourny & Nyssens, 2010; Teasdale, 2012). More recently, we have seen
the graduation of stakeholder participation into the design of research agendas such as the European
Union’s (EU) Horizon 2020 research framework.
Stakeholder participation requires careful planning. Firstly, the stakeholder groups invited to engage
requires consideration. We must first identify groups (both experts and lay people) who may be directly or
indirectly affected by a problem and ensure that they are able to contribute. Involving a wide range of
people in the process stimulates the sharing of perspectives and garners more information on prospective
goals and objectives (Bergvall-Kåreborn et al, 2014; Enserink & Monnikhof, 2003; Fitzgerald et al., 2016).
All too often participatory processes end up recruiting people who are easy to recruit, who are comfortable
at vocalising their opinions and speaking up in public arenas. Therefore, inclusive and diverse recruitment
drives are essential to ensure representativeness; this can be aided by choosing convenient meeting
times and places for citizens in different catchment areas (Laurian at al. 2004). Additionally, technology
has a role to play in engaging the public by providing opportunities for e-participation through computer
generated visualisations and interactive websites (Conroy & Evans-Cowley, 2006; Howard & Gaborit,
2007; Loukis & Wimmer, 2012). Technology can also aid recruitment using online campaigns.
When designing stakeholder participation processes, it has been noted that different types of problems
require different solution responses (Bryson, Quick, Slotterback & Crosby, 2013). The chosen objective of
participation therefore guides the strategy for engagement whether it be to inform, to collaborate, or to
empower stakeholders to decide for themselves (Cooper, Bryer & Meek, 2006; Kautz, 2011). For
instance, diverse stakeholders can be recruited to ensure adequate and diverse representation, with the
end goal of informing the distribution of benefits and the reduction of harms ensuing from decisions. In
contrast to this, small group formats can be used to ensure collaboration among participants through the
exchange of nuanced views. In this scenario, participants learn from each other, and represent a diverse
mixture of expertise within broader stakeholder groups. The outcome would be to change collective
assumptions, transform participant knowledge, and generate new solutions (Bryson, Quick, Slotterback &
Error! Reference source not found.
Accepted Manuscript
Crosby, 2013). These improvements can be ensured by designing approaches which allow for both
exploration and exploitation i.e. searching for new ideas and solutions, as well as refining existing ideas
and solutions (Kim & Schachter, 2013). It is important for participants to also know the purpose of their
In any group situation, effective leadership is required. With stakeholder participation, three types of
leadership styles have been identified sponsors, champions, and facilitators (Crosby & Bryson, 2005;
Morse, 2010; Schwarz et al. 2011). Each role is deemed equally important to ensure success. Sponsors
are identifiable as those with formal authority to justify participation providing funds, staff, and having the
power to translate results into policy. Champions manage day to day activities but lack resources, relying
instead on informal power created through competence and trust. The facilitator’s role is to build
processes, maintain neutrality, assisting with group cohesion and productivity. In participatory processes,
leadership is required to provide equal opportunity for meaningful exchange around decision outcomes
(Bryson, Quick, Slotterback & Crosby, 2013). This can be achieved via the co-production of agendas,
sharing of policy decisions and preventing value differences between the views expressed by experts and
other stakeholders (Quick & Feldman, 2011; Crewe 2001). Group dynamics can also be guided by equal
participation rules, with facilitators ensuring these rules are abided to (Callalan, 2005; Juarez & Brown,
2008). By providing an opportunity for all stakeholders to engage in open dialogue and interconnection, ‘a
space for genuine collaboration’ can be created (Boxelaar, Paine & Beilin, 2006, 121).
2.2 Stakeholder Participation Approaches for RRI
There are numerous established approaches which can guide stakeholder participation for RRI in IS. We
will discuss three such approaches: Foresight Processes, Scenario Planning, and Delphi Studies. We also
outline the relative advantages and limitations of each.
2.2.1 Foresight Processes
Foresight processes aim to explore future designs through participatory approaches for creative thinking
and the inclusion of multiple perspectives (Barré & Keenan, 2006). Information garnered from these
multiple perspectives can be converted into scenarios and shared visions, and eventually into strategies
and actions for policy creation (Caracostas, 2003; Havas, 2005). Foresight processes are most useful for
addressing novel issues with high levels of public concern, or known issues where public opinion has not
previously been considered (Amanatidou, 2014). To do this, foresight processes seek to build networks,
support knowledge creation, and transform participation into action (Amanatidou, 2014). From a collective
learning perspective, foresight can achieve a better match between the environment and its actors,
leading to more adaptive behaviours. The cross-boundary nature of foresight is also useful for creating
new relationships, through linking research and innovation to socio-economic needs, and policy formation
(Brown et al., 2001; Georghiou & Keenan, 2006).
Some of the participatory approaches used in foresight programmes include stakeholder panels,
brainstorming, expert panels, stakeholder consultation and analysis, SWOT analysis, interviews, surveys,
voting/polling, and road mapping (FNR et al., 2007). Prior cases studies suggest several positive
outcomes from foresight processes, both intended and unintended (e.g. knowledge societies; networking
and collective learning; public participation). In one such case, participants expressed that the foresight
process had enabled them to build trust with others and raise their interest in the subject, inspiring them to
improve collaboration and networking with other individuals and organisations. From this raised
awareness, they had not only become more informed but also more concerned and eager to engage in
foresight processes in the future (FNR et al., 2007). Foresight can also help raise public awareness on
science, technology and innovation policy concerns, and promote increased transparency through public
inclusion in democratic policy making (Cassingena & Pace, 2004).
Critics have questioned the ability of foresight processes to predict, given the uncertain and complex
nature of such planning activities (Wright, Cairns, & Goodwin, 2009). However, foresight processes do not
necessarily make claims of prediction, with the aim instead to focus participants on an imagined, and
possible world in the future. This can in turn help guide strategy by planning alternatives for uncertain
futures. Vision building presents an opportunity for participants to evaluate a range of possible futures,
allowing alternatives to be investigated in a systematic way. Creativity and prospective evaluations are
enabled by formulating many versions of the future, through branch analysis, areas of plausibility, cause-
effect generation, and back casting, to name a few (Foresight, 2009). These techniques utilise
Communications of the Association for Information Systems
Accepted Manuscript
participatory exercises with stakeholders to stimulate creativity and dialogue. In turn, foresight processes
seek to engage multiple views of the future through democratic communication.
2.2.2 Scenario Planning
Scenario planning meanwhile seeks to explore the current social world, as a complement to the more
future oriented perspective of foresight processes. Scenario planning promotes many versions of the
social world through communication processes. Differing perspectives can then be converted into
workable scenarios through negotiating varying interpretations of the world so that people come to
understand bounded truths (Dennis, 2013). In other words, truths are a matter of degree, consisting of
levels of objectivity. Truth and power are intimately related and scenario planning techniques seek to
redress the truth-power balance by giving voice to an otherwise unheard group/s of people (Green, 2008;
Habermas, 1984).
Scenario planning is guided by Habermas’ (1984) vision of consensus democracy which asserts that
rational debate, mutual learning, and argumentation are crucial for genuine consensus. When the speech
act of one person is accepted as a position, a process of implicit understanding is explicitly articulated.
Understanding is therefore a dialogue between speakers and hearers, in which the expression is made
understandable to both parties (Habermas, 1984). Shared understanding can therefore only be achieved
when people reach agreement on shared knowledge of the current situation (Van Bouwel & Van
Oudheusden, 2017; Bittner and Leimeister, 2014), and ideal speech is not imposed during communication
but rests on common convictions of stakeholders.
Critics of scenario planning have queried the credibility and legitimacy of scenario planning for policy and
decision-making (Clark, Mitchell, & Cash., 2002). One of the recognised constraints of scenario planning
is the limited worldview of participants taking part (O’Brien, 2004). Habermas’ theory of communicative
action has also been criticized for putting too much emphasis on community, with Maxwell (2012)
questioning whether the focus on reaching agreement detracts from the power of heterogeneous
responses and multiple visions of the future. However, proponents of scenario planning have countered
these criticisms by demonstrating the power of citizen participation in policy and planning processes. Such
that, scenario planning has reported successes in increased social learning, enlightening participants, and
providing an opportunity for new relationship and network building (Reed et al., 2010). Transparency
around recruitment and inclusivity practices are another example of rebuffing the concerns of critics by
broadening participation to capture inputs from diverse stakeholder groups (Long, 2015).
2.2.3 Delphi Studies
Delphi Studies are designed to structure communication processes for large groups of people and assess
the potential for new technological innovations (Skinner et al., 2015; Turoff, 1970). Groups of experts from
many different disciplines are asked to vote on whether they think certain events will occur based on the
evaluation of all group input. They are then later permitted to edit or revise these judgements to arrive at a
consensus (Mitroff & Turoff, 2012). However, since the initial use of this technique in forecast planning,
adaptations have been made such that compromise and consensus at the first round may not be
appropriate. In technology assessment, it is deemed more appropriate to generate several alternative
options for further discussion and debate. The focus on expert participation has also broadened, it is now
questionable as to who or what constitutes an ‘expert’. There has been a move towards more inclusion in
the design of Delphi and increased reflexivity by those taking part (i.e. how do we learn about ourselves
from this experience? In what way is a group of reflective minds better than one mind?) (Mitroff & Turoff,
2012; Linstone & Turoff, 2011). Delphi is therefore primarily aimed at generating ideas and evaluating
alternatives, through the creation of a venue for critical debate (Skinner et al., 2015; Turoff, 1970). These
aims are achieved through the following features (Linstone & Turoff, 2011; Skinner et al., 2015):
A group of experts engaged in anonymous, multi-round discussions.
Two or more rounds that move from open-ended to a narrower focus.
Evaluation of responses by participants, using rating systems to extrapolate written reviews.
Later rounds to refine evaluations and open new lines of enquiry.
Moving away from the traditional format of face-to-face meetings among experts, technology now allows
for the performance of online Delphi studies. This minimizes the time delays between the first, second or
third round of face-to-face Delphi designs. The online method allows participants to complete a survey
Error! Reference source not found.
Accepted Manuscript
(Likert type questions are linked to open-ended “reason questions”), which is stored and altered over time
following reflection and review by other participants within the group. Respondents’ identities are kept
private and anonymous. The online Delphi process can be synchronous or asynchronous, involving
participants from a worldwide panel (Gordon, 2009; Lee and Fedorowicz, 2018). Participant reactions are
also part of the process and included in the design (Linstone & Turoff, 1975; Skinner et al., 2015).
The Delphi method holds several advantages. Firstly, interactionism is supported by protecting the
anonymity of participants and hiding their hierarchical status. Ideas can therefore originate from any
participant and are free flowing. Delphi is also by its very nature a qualitative approach and thus involves
personal and subjective views. Feedback is aimed at explaining and clarifying but does not seek to judge
representations of problems. There is space to contribute both positive and negative perspectives, and
heterogeneity in responses is desired. The Delphi method asserts that different participants will hold
different views over the past and present, which in turn can impact on the future (Linstone, 1984). The
Delphi method can therefore help address complex issues and unexpected consequences by drawing on
the views of different actors. Nevertheless, there are certain limitations associated with Delphi studies.
One limitation concerns the short-term planning horizons and short-term memories of participants which
must be addressed through the communication of possible future scenarios. Objectivity may also be an
illusion as expert opinion can often be unconsciously biased by prior experience.
The next section describes the case study of this paper the “Visionary Depiction Project”
, which adapted
stakeholder participation approaches (e.g. foresight processes, scenario planning, and Delphi studies) to
guide a large-scale responsible IS research process.
3 The Vision Depiction Project
This section describes the Vision Depiction Project, a large-scale responsible research and innovation
project funded by the European Union (EU) Commission. Over a two-year period, the project engaged
more than 1,500 stakeholders (i.e. citizens, practitioners, and policymakers) from across 30 European
countries. The project involved partners from 30 countries across Europe (see Appendix A) who were
tasked with organising face-to-face and online consultations in their respective countries with citizens,
practitioners, and policymakers. The project was guided by the following objectives:
To create visions and scenarios that connect societal needs (e.g. Grand Challenges) with future
expected advances in technology, society, environment etc.
To provide concrete input to Horizon 2020 through recommendations and policy options for
research and innovation (R&I) and calls for the Horizon 2020 Work Programmes.
To engage citizens and stakeholders in a highly participatory consultation process on scenarios
for desirable sustainable futures.
To facilitate dialogue and shared understanding between policymakers, citizens, and
To reveal the relative merits of stakeholder focussed consultations.
The project ran between June 2015 to December 2017 and consisted of two consultation phases. Each
consultation phase was then followed by a clustering workshops where the results from across Europe
were aggregated by the project partners, and a group of invited researchers and citizens.
The first consultation phase began in September 2015 and engaged over 1,000 citizens from across 30
European countries. The purpose of the first consultation phase was to produce visions for desirable
futures in which different societal challenges are addressed through technological innovations. All 30
countries involved in the project produced aggregated results from 36 participants at each national
consultation workshop. The same consultation approach was adopted by all 30 European countries to
ensure that results could be aggregated. Clustering workshop 1 then took place between April 21th-23rd,
2016. During this two-day clustering workshop in Milan, Italy, results from the first consultation phase were
clustered into overarching social needs by the project partners, and over 40 citizens, and researchers
from across Europe. The workshop sought to combine expert opinion with the perspectives of citizens to
Name of the project, project roles, and organisations have all been disguised. They have no relationship to similarly named
organizations or projects that might exist in the real world.
Communications of the Association for Information Systems
Accepted Manuscript
better inform recommendations going forward. Overarching social needs were identified and clustered
thematically by participants to produce a catalogue of visions.
The second consultation phase began in September 2016 and employed blended face-to-face and online
consultations across all 30 countries to validate, enrich, and prioritise the clustered social needs from the
first clustering workshop. The purpose of the second consultation phase was to enhance the clustered
social needs from the first clustering workshop through further discussion. Participants were asked to
generate recommendations on the social and technological issues which they found most important for
the future of research and innovation. Clustering workshop 2 then took place in Brussels, Belgium in
December 2016 and sought to analyse output from the second consultation phase based on input from
researchers and EU policymakers from across Europe. These participants transformed the validated and
prioritised results phase into policy options and prioritised actions and research agendas for Horizon 2020.
The Vision Depiction Project’s phrases are illustrated in Figure 1.
Citizens across
30 countries
produce national
visions (36
citizens and 6
visions per
Citizens and
experts cluster
outputs from the
first phase into
social needs.
across 30
validate, enri ch,
and prioritise the
clustered social
Researchers and
EU policymakers
transform the
validated and
results Into
policy options.
1st Consultation
(September 2015)
2nd Consultation
(September 2016)
workshop 1
(April 2016)
workshop 2
(December 2016)
Stage 1 Stage 2
Figure 1. Timeline of Case Study
This paper presents findings from the second consultation phase in which a blended approach was
adopted to consult citizens, practitioners, and policymakers. The remainder of this section provides
descriptions of how the blended stakeholder consultation was undertaken by the project partner in Ireland.
3.1 Face-to-face Consultation (Foresight Processes and Scenario Planning)
The Irish face-to-face consultation utilised foresight processes and scenario planning to explore the theme
“Holistic Health and Technology Empowerment”. This theme centred on the future social implications of
modern health information systems i.e. privacy, clinical effectiveness, quality. A targeted recruitment
strategy was developed to engage citizens, practitioners, and policymakers. Target groups were recruited
to provide diverse viewpoints on the research scenarios assigned. For instance, the recruited citizens
came from a variety of different backgrounds and demographics to create as varied a representation as
possible i.e. based on age, gender, education etc. Meanwhile, the recruited practitioners consisted of
clinicians, healthcare practitioners, researchers, Health IT professionals, IT services staff, and academics.
Finally, recruited policymakers included Members of the European Parliament, Senators, and an advisor
to the government from the Office of Science, Technology and Innovation. In total, 48 people were
recruited across all target groups and in the end, 34 stakeholders attended the consultation (See Figure
2). Appendix B outlines the breakdown of participants groups in more detail.
Error! Reference source not found.
Accepted Manuscript
Figure 2. Breakdown of Face-to-Face Participants (n=34)
The consultation began with an ice-breaker session, during which time the participants were invited to
introduce themselves and briefly share their motivation for attending the consultation. The professional
title of each participant was not mentioned during the consultation and only first names were used to refer
to individuals during the entire day. Participants were then seated at six roundtables, and an assigned
seating plan ensured representation of citizens, practitioners, and policymakers at each table. A
moderator was also seated at each table and invited participants to discuss the future of research and
innovation in the European Union as relevant to the aforementioned theme. Three rounds of discussions
were held during the day (see Appendix C) in which participants discussed different research scenarios
using a designed template, see Appendix D. The objective was to answer the questions in this template in
order to promote discussion and enrich each research scenario. Special knowledge on the topics was not
required prior to the consultation. Instead the goal was for participants to contribute personal views on the
different research scenarios based on their own personal experience. Participants used post-it notes to
record ideas which were placed on flipcharts for further discussion. The table moderator also asked one
participant at each table to act as a scribe and document the main points from the discussion at the table.
Each table was then allocated one research scenario, and participants were asked to move to a different
table after each discussion round. Places were limited at each table to ensure that each participant would
have a chance to contribute.
3.2 Online Consultation (Delphi Study)
In parallel with the face-to-face consultations, an online consultation was run to engage additional citizens,
practitioners, and policymakers in the consultation process. The online consultation ran between
September and October 2016 and was targeted towards citizens, practitioners, and policymakers across
the 30 EU countries represented by the project consortium. The online consultations began by asking
participants two questions: “How important do you think each proposed research scenario is for society?”
and “What research questions do you think are most relevant for this proposed research scenario?”.
The online consultation utilised a Delphi study approach. Participants were presented with 2 to 5 ‘default’
pros and cons arguments for each chosen research scenario which they could rate in terms of perceived
likelihood of occurrence and potential impact. Participants could also add additional arguments which
would be visible to and rated by other participants going forward, once screened by the platform
administrator for each country. The maximum number of arguments that could be rated and added by
each participant was limited to three. Finally, the number of votes gathered by each argument during the
exercise was summed to provide an overall ranking for each argument. 372 distinct users were recruited
to participate in the Irish online consultation from a database of 444 citizens, practitioners, and
policymakers. In the end, 168 Irish users completed the process by providing responses to all steps in the
online consultation (see Figure 3).
Communications of the Association for Information Systems
Accepted Manuscript
Figure 3. Breakdown of Online Consultation Participants (n=168)
4 Data Analysis
Qualitative thematic analysis (Patton, 2002) was used by the authors to analyse participants’ responses
from both the face-to-face and online consultation. The authors began by continuously rereading the
transcribed content from both consultations to generate a set of codes which were judged as meaningful
and important to the study in question. These initial codes were then grouped together to form overarching
categories of codes which helped organise the content according to similar themes. New categories were
created as necessary to help further analyse the content. The authors continued this process of thematic
analysis until a point of saturation was reached and further analysis did not contribute new insights and
interpretations, but rather supported existing ones made by the authors (Patton, 2002). During each
consultation, participants were asked to offer their unique perspectives on research scenarios around
technology empowerment in healthcare.
In addition, each participant was provided with the opportunity to contribute towards the accuracy of the
collected data at the end of the consultation period and provide further feedback. For instance, at the end
of the face-to-face consultation, a plenary session was held where each participant was asked for
feedback on the findings and was allowed to vote for 5 research scenarios they felt were most important in
terms of impact and probability of occurrence. Participants were provided with 5 coloured dot stickers
(Total votes available = 165 (5 votes x 34 participants)) and participants could place only 1 dot sticker on
one research scenario. This evaluation contributed further insights to support the data analysis and
generated additional insights into the research question. The findings presented in the next section centre
on the most salient quotes selected to further support and illustrate results from the two consultations.
5 Findings
This section reports on findings from the second consultation phase. The project built on the conviction
that the collective intelligence of society could strengthen the relevance of the European science and
technology system. To achieve this, the project sought to establish genuine dialogue between citizens,
practitioners, and policymakers, and collect actors’ visions around the social implications of science,
technology and innovation. The findings focus on the topic of technology empowerment in healthcare, with
specific relevance to three research scenarios: 1) Quantitative person-centred health, 2) Data for all
Share the power of data, and 3) Equal access to holistic health services and resources for all citizens.
5.1 Quantitative Person-centred Health
Several participating countries looked at the scenario “Quantitative person-centred health” (e.g. Cyprus,
Finland, Ireland and Slovakia), focusing on the following 5 questions:
Error! Reference source not found.
Accepted Manuscript
5.1.1 What challenge(s) does this research scenario address?
Participants spoke about the need for an overall change in the way we think about health and wellbeing,
and how health information systems need to become more patient centric. There was consensus that the
delivery of efficient, cost effective and accessible healthcare services can be supported by technology i.e.
the use of e-health solutions, electronic health records. Discussions also centred on how the use of e-
health solutions will result in an explosion of data collection and support improved diagnosis and
treatment. However, the general sentiment was that patient data protection, security and integrity must be
at the forefront of a patient centric healthcare environment and technology use should be regulated,
inclusive, and follow a holistic approach which reflects the ever-changing world, we live in.
5.1.2 Is it important from your point of view to address this challenge? Why?
Participants felt that in order to assess the challenges facing the healthcare system, it was important to
benchmark the current healthcare service offered to patients using technology. This could help determine
where best /worst practices exist and whether services should be implemented elsewhere in the
healthcare system. Benchmarking could provide the opportunity to improve patient healthcare outcomes
and healthcare services using data on process efficiency, cost, data protection, and use of technology.
5.1.3 How could it be approached?
Participants agreed that healthcare systems in each country must be brought to a similar level across the
EU. They discussed how this will require regulations to be put in place to protect patient data privacy and
integrity, and in doing so allow for patient records to be shared between healthcare providers, both public
and private, in all EU countries. Practitioners argued that data availability could greatly improve the overall
standard of care through medical diagnostics. However, others noted that education and the use of
targeted research funding were essential to ensure that best practices were achieved for data
management system design and healthcare services efficiency.
5.1.4 Who should be involved in solving the problem?
For this scenario to be fully implemented, participants asserted that each member state of the EU must
set minimum standard requirements for technology implementation. They noted that relevant bodies must
be actively involved in this process, including policy makers, member-state non-governmental
organisations (NGOs), citizens, healthcare professionals, universities, IT professionals, legal professionals
and other specialist groups.
5.1.5 What should be the main goals/impacts of the research activity?
In terms of goals and expected impacts from this scenario, participants hoped for an “integrated
healthcare systems (public and private)” which would deliver “Improved health care services and
systems… [and] patient healthcare outcomes.” To achieve this, the consensus was that “Improved access
to healthcare services.” as well as “Improved ability to measure healthcare service efficiency and patient
satisfaction.” were essential. Some participants also pointed out the need to include privacy as a
measured impact of the scenario: through “Improved patient data management, protection and validation.”
5.2 Data for All Share the Power of Data
Several participating countries looked at scenario Data for all Share the power of data (e.g. Croatia,
Germany and Ireland), again focusing on the following 5 questions:
5.2.1 What challenge(s) does this research question address?
Participants noted that this scenario should address the availability, transparency and reliability of data in
healthcare systems. They asserted that there was a need to relook at data management and security in
the healthcare system as well as the ethics behind data use, in order to ensure equality around data use
and accessibility. There were also discussions on how citizens’ need to be educated around both the
creation and use of healthcare data to improve data management and security overall.
Communications of the Association for Information Systems
Accepted Manuscript
5.2.2 Why would it be important to address this challenge?
There was consensus that it was important to address this challenge in order to foster informed decision
making and to improve data transparency and protection. Participants were confident that more informed
decision-making can in turn help increase the overall quality of life for patients through the application of
targeted interventions. However, again citizens felt that education was needed to understand data
ownership and data management to ensure that they could be empowered during healthcare decisions.
5.2.3 How could this challenge be approached?
The participants noted that addressing this challenge required increasing citizen knowledge on their rights
in relation to healthcare data. Policymakers must also ensure that everyone has access to the internet
through the provision of high-speed broadband and affordable Wi-Fi Internet connections. It was also
discussed how further research was needed to determine what “the correct use of data” means and
provide guidelines on how to identify real problems and issues, as well as desired outcomes. Participants
noted that there was a need to ensure relevant policy exists to match new technologies to these problems.
5.2.4 Who should be involved in solving the problem?
Citizens were very vocal that different stakeholders should be involved in healthcare decision-making
including the citizens, policy makers, state institutions / decision makers, Civil society organisations
(CSOs), stakeholders and experts, public service providers, IT experts, the media and user groups. This
vision for engaged involvement would require new processes to manage decision-making.
5.2.5 What should be the main goals/impacts of the research activity?
For the second scenario, the participants identified the following primary goal: “Develop the [capability] of
citizens for participating in decision-making processes and for validating the relevance and meaning [of]
metrics”. In order to help citizens to validate the relevance and meaning of metrics, participants asserted
the need for “reliable data sources [and] enhanced systems across Europe. Providing enhanced...
awareness in the general public [about] personal health and maintenance.” Participants also spoke about
the opportunities in this scenario for solutions to be co-created by “enabling citizens and public service
providers (for example, CSOs) to use and create data through open source digital tools and platforms.”
Most participants agreed, co-creation processes could enable the “usage of data to solve social problems
and inequalities with the aim of development of sustainable society.” However, again, there was cautions
raised about the need to include “improved data security and protection”.
5.3 Equal Access to Health Services and Resources for all Citizens
Participating countries also looked at scenario Access to equal and holistic health services and resources
for all citizens (e.g. Austria and Ireland), focusing on the following 5 questions:
5.3.1 What are the problems/challenges of this scenario?
Participants noted that external cost pressures from sources such as the pharmaceutical industry were
creating many challenges around equal access to healthcare services. They discussed how healthcare
services were facing continuous cost-cutting measures and increasing pressures on healthcare staff. In
addition, participants noted that overall awareness of health was limited due to a lack of focus on health
promotion and many were unaware of the effect that changes in lifestyles could have on their health.
5.3.2 Why is it important to find solutions to these challenges?
The importance of solving these challenges centred on the need to improve the healthcare systems as a
whole by ensuring equal access to treatment for everyone. Participants noted that there is potential for
technology help reduce wait times, improve inefficiencies, to minimize the negative economic impact of
the current healthcare service through the prevention of illnesses. In turn, timely interventions can
ultimately lead to an increase in the quality of life for all and reduce the costs of healthcare delivery.
5.3.3 How could it be approached?
Participants noted that the harmonization of medical care (e.g. training standards, product standards,
etc.). across the European market, as well as improved working conditions for staff were key antecedents
Error! Reference source not found.
Accepted Manuscript
for ensuring equal access to treatment for everyone. Technology could also support closer cooperation
between medical staff and a closer exchange of knowledge. However, there is also a need for more
investment in research and development on how medical staff can adopt a focused patient orientation
through technology. Participants felt that these challenges could be addressed through increased
education, improved care environments, and targeted policymaking for improving the entire healthcare
service through technological developments.
5.3.4 Who should be involved in solving the problem?
Participants asserted that multiple stakeholder groups should be involved in solving this problem including
citizens, researchers, healthcare service providers including General Practitioners (GPs), and public
health agencies. Participants also noted that the government should play an important role in delivering
policy to fund research at a national and local level. Supporting roles were also discussed for the IT
industry, national statistics offices, teachers, and other private companies.
5.3.5 What should be the main goals/impacts of the research activity?
In the third scenario, participants placed an emphasis on the goal of “Improved awareness and education
[on] health and wellness” among citizens as well as “Improved [access to] healthcare services.” To
achieve this, the group discussed the need for policy makers to recognise national differences across the
EU and “to note that in the case of EU policies one size does not fit all.” Participants asserted the need for
“Increased research into the healthcare as a whole” in order to further support this goal.
The next section discusses findings from the Vision Depiction Project in more detail.
6 Discussion
The following section provides an in-depth discussion of our findings in relation to the following research
question: How can stakeholder participation approaches be used to support RRI in information systems
research? The discussion is guided by Stilgoe et al.’s (2013) RRI framework as presented in Table 1.
6.1 Anticipation
The use of foresight processes and scenario planning proved effective for supporting anticipation in the
Vision Depiction Project. There was a palpable enthusiasm among participants as individuals engaged in
dialogue around the future of technology empowerment in healthcare and discussed visions for the next
20 to 30 years. Questionnaire results from the Irish consultation suggest participants were excited by the
opportunity to contribute towards the future of science and technology in Europe and found the
stakeholder participation techniques useful for promoting creative thinking. However, there was some
divergence between participants’ views of outcome effectiveness. For instance, around 5% of
respondents disagreed with the questions “Overall, I was satisfied with the results of the event” and “It is
clear to me what will be done with the results of the discussions, which suggests uncertainty around what
the outcome from the Vision Depiction Project was, and how results would be used going forward. This is
a common criticism of foresight processes and scenario planning and suggests the need for more
exploitation focus (cf. Kim & Schachter, 2013) to refine existing ideas ensuring both RRI process and
outcome effectiveness (cf. Geoghegan-Quinn, 2014).
Overall, the results suggest that the Vision Depiction Project provided a structured process for participants
to discuss future socio-technical issues and engage in dialogue around how technology can produce both
positive outcome (utopian visions) and negative outcomes (dystopian visions). Table 2 provides sample
responses from the Irish consultation across each stakeholder group to illustrate utopian and dystopian
visions of technology empowerment in healthcare.
Communications of the Association for Information Systems
Accepted Manuscript
Table 2. Utopian and Dystopian Visions of Technology Empowerment in Healthcare
Utopian Vision
Dystopian Vision
Technology can help make universal healthcare a
reality and promote greater social inclusion. We can
overcome equality divides, geographical divides, age
divides, empower students with disabilities. We can
address key challenges through awareness, discovery,
and inclusivity.’
There are a lot of “Tech-phobic citizens” out there, we
need to overcome this fear of technology through
greater digital literacy across all age groups. The profit
motive of big pharma is also a concern. Things like
patient consent, data protection, confidentiality; we
need to enforce policies and provide education on
evidence-based medicine’.
It’s about personalised medicine, supporting
preventative health. Patients can take personal
responsibility for one’s health. Keeping citizens in
control will promote better conversation. We’ll also
have a wider evidence base for research.
Storing poor quality data will result in misinformation.
We’re an over medicalised society so health illiteracy is
important, as well as engagement and communication
across geography, age, cultural differences, and
I’d hope for better value from money spent. We can
reduce costs to the economy by having a healthier
population. We need to think about peoples’ longevity
that they’re healthy for longer not just living for
longer. This requires a bottom-up approach - public
and private mix involved. We need societal discussion
we can’t ignore the issue’.
The short political cycle is a challenge for ensuring
sustainability and the future proofing of outcomes.
There is also a complex regulatory environment and a
lack of standards e.g. Unique health identifier.
Anything that improves lives is important but if we don’t
focus on barriers, we fail before we start.
6.2 Reflexivity
Literature suggests that creative responses can be supported by engaging the diverse perspectives of
different stakeholder groups (Enserink & Monnikhof, 2003; von Schomberg, 2012). In the Vision Depiction
Project, participants noted that the involvement of citizens, practitioners, and policymakers was very
positive for dialogue and helped generate interesting and diverse discussions during the consultation. In
terms of reflexivity, most participants agreed that they had a clear understanding of the purpose of the
event and felt comfortable when voicing their opinions during the process. However, given the diverse
background of participants, respect for the opinions of others turned out to be a crucial concern, and
power dynamics sometimes arose between citizens, practitioners, and policymakers. For instance,
practitioners at times asserted their position as healthcare experts on a discussion topic, which drew
criticism from citizens who countered that practitioners did not understand the patient perspective.
Power dynamics require strong management to provide the opportunity for meaningful exchange and
bearing on decision outcomes (Bryson, Quick, Slotterback & Crosby, 2013). This signals the need for
strong facilitation while implementing participatory approaches (Callalan, 2005; Juarez & Brown, 2008). In
the Vision Depiction Project, training sessions were organised prior to each consultation phase to help
prepare facilitators for running the consultation using rules for equal participation. For instance, rule sets
sought to generate a commitment to discussion and ensure the integrity / authenticity of participants”.
This proved important for both exploration and exploitation processes to allow citizens, practitioners, and
policymakers a chance to explore and refine existing ideas and solutions (Kim & Schachter, 2013). As
stated by one participant “the main challenge is to keep discussions on topic. (We) need to always draw
each point back to question. Another participant noted the importance of good facilitation and observed
that all facilitators were excellent very professional and friendly”. The use of templates (see Appendix
D) also helped guide stakeholders through the steps involved in the approach using targeted questions
and motivated each stakeholder group to stay focused. Table 3 provides an overview of the opportunities
and challenges for reflexivity based on quotations from participants in the Irish consultation.
Error! Reference source not found.
Accepted Manuscript
Table 3. Quotations from Irish Participant on Reflexivity
Please state something you liked about the
Is there anything we could improve about the
“Very diverse range of participants made this
most interesting and open-minded.”
“Engagement, multi-disciplinary approach,
energy” ... “The enthusiasm was infectious.”
“Group of bright and interesting people
stimulating and enjoyable.”
“Good discussion, personalities and different
experiences of people.”
“How respectful everyone was of the opinion of
Need “more time given for summary of ideas
and consensus-forming”
Need “more time devoted to an overall proposal”
“Some topics had questions to drive it others did
not. The ones with [questions] worked better.”
“Having a prior commitment from people to stay
until the end” is important.
“More digital approach e.g. electronic voting.”
6.3 Inclusivity
The Vision Depiction Project aimed to provide ample opportunities for inclusiveness by inviting diverse
stakeholder groups to attend the consultation. The interest generated during the recruitment process
came as a welcome surprise to the organisers as diverse groups were eager to engage in the process.
Nevertheless, significant challenges were experienced in ensuring a context-sensitive design that catered
to the diverse representation of different target groups (Laurian at al. 2004). While a good level of diversity
was achieved overall, the number of attending citizens from the primary and secondary level education
categories was lower than the tertiary level education category. In the Irish context, this may have been
impacted by the fact that the event was hosted on a university campus, and therefore citizens with a
university education may have been more likely to attend. However, there was also an
underrepresentation of citizens aged 66+, despite a targeted communication strategy in a national
newspaper to address low number of applications received from citizens in this age bracket. This again
points to the challenge of designing a context-sensitive approach which caters to all demographics.
Our results suggest that combining digital and face-to-face approaches can be more effective for
inclusiveness than face-to-face approaches alone and helps increase participation among certain minority
groups. However, there was still a significant challenge in getting all participants to actively engage with
the online consultation and remain engaged over time. While the online consultation garnered some
interest, the level of engagement fell far below the project partners’ ambitious targets. Each partner had
expected to engage 300 participants in their country, leading to a total user base of around 9,000
participants across 30 European countries. However, in the end only 3,461 participants were engaged,
with some partners even failing to engage more than 30 participants. It should be noted that some
participants felt that digital communication was not a substitute for face-to-face communication, and senior
citizens in particular noted a preference for paper-based media. One participant suggested after the face-
to-face event that: “Info might be handed out to all participants in paper form on the dayas they were less
comfortable interacting with the PDF copies that had been made available by the organisers via email. We
find that a blended approach may cater better to the needs of diverse stakeholder groups, as the use of
eParticipation alone may exclude groups who are less tech savvy (Mahrer and Krimmer, 2005).
6.4 Deliberation
The Vision Depiction Project utilised numerous deliberation tools, techniques, and approaches in order to
support RRI process and outcome effectiveness. Literature suggests that the use of multiple participation
approaches can ensure a collective responsibility over the control and direction of RRI processes so that
they become ethically acceptable for the diverse stakeholder groups involved (von Schomberg, 2012). In
particular, the Vision Depiction Project adopted ‘blended’ approaches such as face-to-face workshops and
e-participation to facilitate deliberation among citizens, practitioners, and policymakers. Time turned out to
be a critical constraint during the face-to-face consultation as the diverse views of stakeholders meant that
it was difficult to reach consensus during the allotted schedule. It is therefore not surprising that more
mixed responses were received around the output of the face-to-face event; while 77.27% of participants
were satisfied with the results of the event, some participants indicated afterwards that they were unclear
what will be done with the results of the discussions given the constrained time for consensus building.
Table 4 provides an overview of questionnaire results from the Irish consultation.
Communications of the Association for Information Systems
Accepted Manuscript
Table 4. Results from Irish Participant Questionnaire (Deliberation)
1. I felt sufficiently comfortable and at ease
to speak out and voice my opinions during
the process.
2. The moderators did a good job in
ensuring a constructive and fair process
during the discussions.
3. The timing of the programme was
perfect: I didn’t feel stressed or in a rush.
4. I had a clear understanding of the
purpose of the event.
5. Overall, I was satisfied with the results of
the event.
6. It is clear to me what will be done with
the results of the discussions.
7. I would take part in such an event again
E-participation techniques helped address timing issues by using technology to engage stakeholders after
the event via interactive media such as instant messaging, computer generated visualisations, and e-
voting (Conroy & Evans-Cowley, 2006; Loukis & Wimmer, 2012; Howard & Gaborit, 2007). Three
participants left the face-to-face consultation before the programme ended due to other personal
commitments; a common challenge faced when organising face-to-face workshops (Laurian at al. 2004).
E-participation provided opportunities for them to re-engage with the process after leaving and continue
the conversation using digital tools. Nevertheless, the online consultation carried out in the Vision
Depiction Project suffered from a lack of clear rules, as the process was more open ended and less
consensus based than traditional Delphi studies. Part of the problem centred on the fact that in-depth user
testing had not been carried out on the online consultation platform and as a result, many participants
noted that the online consultation platform was not designed in an intuitive way and took too long to
complete. Another major challenge was representing the high volume of information contained in the
online consultation’s research scenarios in an accessible way. Unfortunately, the online consultation
placed a large burden on the user to understand the rules and process high volumes of information which
lead to information overload. In hindsight, this issue could have been resolved through the involvement of
a user experience expert to make the user interface more accessible and clarify rules. In particular, users
noted that a lot of text could have been hidden, especially text, which was repeated across all screens, in
order to streamline the process. This would better assist participants in working together to improve the
relationship between outcomes and the needs of society (Geoghegan-Quinn, 2014).
6.5 Responsiveness
Lastly, the Vision Depiction Project proved effective in terms of responsiveness as it supported a strong
future-oriented focus for research and innovation in Europe. Responsiveness requires that stakeholders
take co-responsibility for the development of innovation agendas, and drive change by adapting
deliverables to the reaction of stakeholder values (Owen et al., 2013; Stilgoe et al, 2013). In the Vision
Depiction Project, the needs and concerns of citizens, practitioners, and stakeholders were transformed
into recommendations and suggestions for research and innovation policies in the EU. This output was
later delivered to the EU Commission and supported the processes of setting the scientific course of
funding calls in the EU going forward. The Vision Depiction Project therefore delivered on the RRI
principle of ‘science for society’ which concerns innovative science that is ethical, inclusive, democratic
and equitable by including public values in science and innovation (Wilsdon et al., 2005).
Initial evidence suggests that outcomes from the project have had an impact on the topics that will be
included in the next research funding programme for the EU. In addition, policy options have been
extracted from the validated and prioritised scenarios to form recommendations which were directly
delivered to policymakers at a national and EU level. Results from the two consultation phases were
disseminated to key stakeholders with an involvement in research and innovation, including government
officials, national research councils, and scientific bodies. The Vision Depiction Project’s steering
Error! Reference source not found.
Accepted Manuscript
committee were therefore satisfied that responsiveness was achieved through the delivery and
communication of responsible research agendas for grand challenges at national and international policy-
making levels. The project also increased accountability by allowing stakeholders to scrutinise potential
impacts and risks from emerging technologies, a key criterion for responsiveness in RRI (Delpy, 2011;
Owen et al., 2012). Participant responses from the Irish consultation support the need for similar
approaches in the future, with one participant asserting that “the format… is a good formula for improving
future policy decisions”.
7 Conclusion
In this paper, we presented findings from a 24-month RRI project which engaged citizens, practitioners,
and policymakers from across 30 European countries in the co-creation of research agendas. Overall, the
Vision Depiction Project was deemed a success both by participants, and the partners involved. The face-
to-face and online consultation techniques provided a rich catalyst for discussion between stakeholders on
diverse socio-technical issues, and the feedback from participants indicates a strong desire for similar IS
research consultations in the future.
In terms of theoretical contributions in this paper, we present insights into the use of stakeholder
participation approaches for RRI in IS research. We assess these lessons learned based on criteria from
Stilgoe et al.’s (2013) RRI framework: Anticipation, Reflexivity, Inclusion, Deliberation, and
Responsiveness. Based on this discussion, the relative strengths and weaknesses of stakeholder
participation for RRI were analysed and recommendations were derived for IS research. In terms of
practical contributions, we provided an account of how IS researchers and practitioners might foster
responsible IS research through engaging diverse stakeholder groups. We draw on case study findings
from the second consultation phase, where blended face-to-face and online techniques were used to
engage citizens, practitioners, and policymakers. We hope the learnings described in the paper will assist
IS researchers and practitioners in navigating the promises and pitfalls of RRI in information systems and
assist them in undertaking multi-stakeholder consultations of their own going forward.
One limitation of the paper is that the case study was primarily focused on the initial stages of engaging
stakeholders in the design of IS solutions; therefore, an in-depth study of the impact derived from project
outcomes on the future development of IS solutions was outside the scope of our paper. Future studies
can seek to provide a longitudinal analysis of the impact of responsible IS research approaches from
design to the implementation stages of IS development. The findings of our case study are also specific to
the healthcare sector therefore, future research is needed to investigate the application of responsible IS
research approaches to other domains and compare idiosyncrasies between them. Similarly, an
evaluation framework, grounded in RRI and information systems literature, is needed to compare different
stakeholder participation approaches for responsible IS research in different contexts.
In terms of other future research agendas, we suggest that there is an urgent need for user engagement
studies on blended consultations to increase their effectiveness going forward. One lesson learned was
that online consultation platforms must be designed in a way which motivates users to remain
continuously engaged in the participatory process. Our findings suggest that this can be achieved through
the delivery of brief but informative prompts to guide users through the online consultations. The inclusion
of incentives can also help gain individuals’ attention. However, monetary incentives will likely be
constrained by the budget available to organisers, and the Vision Depiction Project instead relied on
individuals’ inherent motivation to engage with the topics. Future research can investigate the use of
different forms of incentives for user engagement in blended consultations.
The authors would like to thank Professor Graeme Shanks for his guidance in shaping the manuscript. We
would also like to acknowledge the helpful comments provided by Professor Jan Recker, the editor in
chief, and the associate editor, and two anonymous reviewers.
Communications of the Association for Information Systems
Accepted Manuscript
Amanatidou, E. (2014). Beyond the veilThe real value of Foresight. Technological Forecasting and
Social Change, 87, 274-291.
Arnstein, S. R. (1969). A ladder of citizen participation. Journal of the American Institute of planners,
35(4), 216-224.
Barre, R., & Keenan, M. (2006). Evaluation, impact and learning. In Seville FTA conference, September.
Bedeley, R., Carbaugh, D., Chughtai, H., George, J., Gogan, J., et al. (2019). Giving voice to the
voiceless: The use of digital technologies by marginalized groups. Communications of the
Association for Information Systems, 45(1).
Bergvall-Kåreborn, B., D. Howcroft and A. Ståhlbröst (2014). "Disregarding History: Contemporary IS
Contexts and Participatory Design." Communications of the Association for Information Systems 34,
Bilgeri, D., Gebauer, H., Fleisch, E., and Wortmann, F. (2019), Driving Process Innovation with IoT Field
Data, MIS Quarterly, 18(3).
Bittner, E. A. C., & Leimeister, J. M. (2014). Creating shared understanding in heterogeneous work
groups: Why it matters and how to achieve it. Journal of management information systems, 31(1),
Boxelaar, L., Paine, M., & Beilin, R. (2006). Community engagement and public administration: Of silos,
overlays and technologies of government. Australian Journal of Public Administration, 65(1), 113-
Brown, N., Rappert, B., Webster, A., Cabello, C., Sanz-Menéndez, L., Merkx, F., & Van de Meulen, B.
(2001). Foresight as a Tool for the Management of Knowledge Flows and Innovation (FORMAKIN).
Work Package, 5(6).
Bryant, A.; Land, F.; and King, J., L., (2009) Editor’s introduction to the special issue on ethical issues in
IS research. Journal of the Association for Information Systems 10(11), 782786.
Bryson, J. M., Quick, K. S., Slotterback, C. S., & Crosby, B. C. (2013). Designing public participation
processes. Public administration review, 73(1), 23-34.
Buckwalter, N. D. 2014. “The Potential for Public Empowerment through Government-Organized
Participation.” Public Administration Review 74 (5): 573584. doi:10.1111/puar.2014.74.issue-5.
Busch, P. A., & McCarthy, S. (2020). Antecedents and consequences of problematic smartphone use: A
systematic literature review of an emerging research area. Computers in Human Behavior, 106414.
Callanan, M. (2005). Institutionalizing participation and governance? New participative structures in local
government in Ireland. Public Administration, 83(4), 909-929.
Cassingena Harper, J., & Pace, G. (2004). ‘The creative processes in policy making: a case for context in
foresight. In Proceedings of the Fifth International Conference on Creative Thinking, Malta (pp. 21-
Caracostas, P. (2003). Sharing Visions: Towards European Area for Foresight. Brussels: European
Chen, W.S. and Hirschheim, R. (2004) A Paradigmatic and Methodological Examination of Information
Systems Research from 1991 to 2001, Information Systems Journal 14, 197235.
Clark, W. C., Mitchell, R. B., & Cash, D. W. (2006). Evaluating the Influence of Global Environmental
Assessments. Global environmental assessments: information and influence. MIT, Cambridge, 1-
Conroy, M. M., & Evans-Cowley, J. (2006). E-participation in planning: an analysis of cities adopting on-
line citizen participation tools. Environment and Planning C: Government and Policy, 24(3), 371-
Cooper, T. L., Bryer, T. A., & Meek, J. W. (2006). Citizencentered collaborative public management.
Public Administration Review, 66, 76-88.
Error! Reference source not found.
Accepted Manuscript
Cornwall, A., & Coelho, V. S. (Eds.). (2007). Spaces for change?: the politics of citizen participation in new
democratic arenas (Vol. 4). Zed Books.
Crewe, K. (2001). The quality of participatory design: The effects of citizen input on the design of the
Boston Southwest Corridor. Journal of the American Planning Association, 67(4), 437-455.
Crosby, B. C., & Bryson, J. M. (2005). Leadership for the common good: Tackling public problems in a
shared-power world (Vol. 264). John Wiley & Sons.
Damberg, C. L., Ridgely, M. S., Shaw, R., Meili, R. C., Sorbero, M. E., Bradley, L. A., & Farley, D. O.
(2009). Adopting information technology to drive improvements in patient safety: lessons from the
Agency for Healthcare Research and Quality health information technology grantees. Health
services research, 44(2p2), 684-700.
Davison, R., Hardin, A., Majchrzak, A., Ravishankar, MN (2019), Responsible IS Research for a Better
World: A Special Issue of the Information Systems Journal, Call for Papers, Information Systems
Defourny, J., & Nyssens, M. (2010). Social enterprise in Europe: At the crossroads of market, public
policies and third sector. Policy and Society, 29(3), 231-242.
Delpy, D. (2011). Synthetic Biology Dialogue and What is the Public Perspective. Science in Parliament:
The Journal of the Parliamentary and Scientific Committee, Spring, 2011, 41-42.
Dennis, B. (2013). “Validity crisis” in qualitative research: Still? Movement toward a unified approach. In
Dennis, B., Carspecken, L., Carspecken, P. (Eds.), Qualitative research: A reader in philosophy,
core concepts, and practice (SeriesCounter points) (pp. 3-37). New York, NY: Peter Lang.
Enserink, B., & Monnikhof, R. A. (2003). Information management for public participation in co-design
processes: evaluation of a Dutch example. Journal of Environmental Planning and Management,
46(3), 315-344.
Fitzgerald, C., McCarthy, S., Carton, F., Connor, Y. O., Lynch, L., & Adam, F. (2016). Citizen participation
in decision-making: can one make a difference? Journal of Decision systems, 25(sup1), 248-260.
Fouché, C., & Light, G. (2011). An Invitation to Dialogue: ‘The World Café’ In Social Work Research.
Qualitative Social Work, 10(1), 28-48.
French, S., & Bayley, C. (2011). Public participation: comparing approaches. Journal of Risk Research,
14(2), 241-257.
Foresight, (2009). Scenario Planning: Guidance Notes.
FNR, VDI, Z-Punkt (2007). FNR Foresight Final Report National Priorities for Public Research and Other
Findings. June 2007.
Geoghegan-Quinn, M. (2014). Horizon 2020A Paradigm Shift for Funding Research and Innovation in
Europe. Speech, 14, 65.
Georghiou, L., & Keenan, M. (2006). Evaluation of national foresight activities: Assessing rationale,
process and impact. Technological Forecasting and Social Change, 73(7), 761-777.
Genus, A. (2006). Rethinking constructive technology assessment as democratic, reflective, discourse.
Technological Forecasting and Social Change, 73(1), 13-26.
Gordon, T. J. (2009). The real-time Delphi method. Futures research methodology version, 3, 19.
Greene, J. C. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of mixed
methods research, 2(1), 7-22.
Gusmano, M. K. 2013. “FDA Decisions and Public Deliberation: Challenges and Opportunities.” Public
Administration Review 73 (S1): S115S126. doi:10.1111/puar.2013.73.issue-s1
Habermas, J. (1984). The theory of communicative action (T. McCarthy, Trans.). Boston, MA: Beacon.
Havas, A. (2005). Terminology and methodology for benchmarking foresight programmes. Available at
SSRN 1735023.
Communications of the Association for Information Systems
Accepted Manuscript
Howard, T. L., & Gaborit, N. (2007). Using virtual environment technology to improve public participation
in urban planning process. Journal of Urban Planning and Development, 133(4), 233-241.
Howcroft, D. and Trauth, E.M. (eds), Handbook of Critical Information Systems Research: Theory and
Application, Edward Elgar Publishing, Cheltenham, UK.
Ianniello, M., Iacuzzi, S., Fedele, P., & Brusati, L. (2019). Obstacles and solutions on the ladder of citizen
participation: a systematic review. Public Management Review, 21(1), 21-46.
Jørgensen, M.-L. and S. Schøning. 2016. Vision Catalogue. Encompassing the Visions From All 30
Countries. CIMULACT project, deliverable 1.3.
Juarez, J. A., & Brown, K. D. (2008). Extracting or empowering? A critique of participatory methods for
marginalized populations. Landscape Journal, 27(2), 190-204.
Kautz, K. (2011). "Investigating the design process: participatory design in agile software development."
Information Technology & People 24(3), 217-235.
Khalil Gibran Sand and Foam. accessed
Klingemann, H. D., & Fuchs, D. (Eds.). (1995). Citizens and the State (Vol. 1). OUP Oxford.
Kim, S., & Schachter, H. L. (2013). Citizen participation in the budget process and local government
accountability: Case studies of organizational learning from the United States and South Korea.
Public Performance & Management Review, 36(3), 456-471.
Kohlegger, M., Meier, R. and Thalmann, S. (2009). Understanding Maturity Models. Results of a
Structured Content Analysis. In Proceedings of the 9th International Conference on Knowledge
Management and Knowledge Technologies (IKNOW '09), Graz, Austria.
Laurian, L., Day, M., Berke, P., Ericksen, N., Backhurst, M., Crawford, J., & Dixon, J. (2004). Evaluating
plan implementation: A conformance-based methodology. Journal of the American Planning
Association, 70(4), 471-480.
Laroche, G. (2011) Presentation at the Responsible Innovation Workshop, London.
Lee, J., & Fedorowicz, J. (2018). Identifying issues for the bright ICT initiative: a worldwide delphi study of
IS journal editors and scholars. Communications of the Association for Information Systems, 42(1),
Linstone, H. A. (1984). Multiple perspectives for decision making; bridging the gap between analysis and
Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi Method. Reading, MA: Addison-Wesley.
Linstone, H. A., & Turoff, M. (2011). The Delphi method: techniques and applications, Elsevier Science
Pub. Co.
Long, H. (2017). Validity in mixed methods research in education: the application of Habermas' critical
theory. International Journal of Research & Method in Education, 40(2), 201-213.
Loukis, E., & Wimmer, M. (2012). A multi-method evaluation of different models of structured electronic
consultation on government policies. Information Systems Management, 29(4), 284-294.
Lubberink, R., Blok, V., van Ophem, J., & Omta, O. (2017). A framework for responsible innovation in the
business context: lessons from responsible-, social-and sustainable innovation. In Responsible
Innovation 3 (pp. 181-207). Springer, Cham.
Mahrer, H., & Krimmer, R. (2005). Towards the enhancement of edemocracy: identifying the notion of the
middleman paradox. Information systems journal, 15(1), 27-42.
Markus, M. L., & Mentzer, K. (2014). Foresight for a responsible future with ICT. Information Systems
Frontiers, 16(3), 353-368.
Maxwell, J. A. (2012). A realist approach for qualitative research. Sage.
Error! Reference source not found.
Accepted Manuscript
Mingers, J. and Walsham, G. (2010) Towards ethical information systems: the contribution of
discourse ethics. MIS Quarterly 34(4), 833854.
Mitroff, I., & Turoff, M. (2012). Philosophical and methodological foundations of Delphi. The Delphi
method: Techniques and applications, 30.
Morse, R. S. (2010). Integrative public leadership: Catalyzing collaboration to create public value. The
Leadership Quarterly, 21(2), 231-245.
O’Brien, F. A. (2004). Scenario planning––lessons for practice from teaching and learning. European
Journal of Operational Research, 152(3), 709-722.
Owen, R., Macnaghten, P., & Stilgoe, J. (2012). Responsible research and innovation: From science in
society to science for society, with society. Science and public policy, 39(6), 751-760.
Owen, R., Stilgoe, J., Macnaghten, P., Gorman, M., Fisher, E., & Guston, D. (2013). A framework for
responsible innovation. Responsible innovation: managing the responsible emergence of science
and innovation in society, 31, 27-50.
Pavie, X., Scholten, V., & Carthy, D. (2014). Responsible Innovation. From Concept to Practice. World
Peter, V., van der Veen, G., Doranova, A., & Miedzinski, M. (2013). Screening of Regulatory Framework.
Final Report.
RegulatoryScreening.pdf Accessed 22/8/2019
Peticca-Harris, A., deGama, N. and Ravishankar, M.N. (2018) Postcapitalist Precarious Work and those in
the ‘Drivers’’ Seat: Exploring the Motivations and Lived Experiences of Uber Drivers in Canada,
Poser, M., Küstermann, G., & Bittner, E. (2019). Citizen E-Participation: Bringing the “E” to Facilitated
Workshops. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
Quick, K. S., & Feldman, M. S. (2011). Distinguishing participation and inclusion. Journal of Planning
Education and Research, 31(3), 272-290.
Reed, M., Evely, A. C., Cundill, G., Fazey, I. R. A., Glass, J., Laing, A., ... & Stringer, L. (2010). What is
social learning? Ecology and society.
Rissgaard, K., Schoning, C and CIMULACT Consortium partners, 2017. Vision Catalogue- Encompassing
the Visions from all 30 Countries. Document originated from D1.3. CIMULACT project. Retrieved
Sæbø, Ø., Rose, J., & Flak, L. S. (2008). The shape of eParticipation: Characterizing an emerging
research area. Government information quarterly, 25(3), 400-428.
Schot, J., & Rip, A. (1997). The past and future of constructive technology assessment. Technological
forecasting and social change, 54(2-3), 251-268.
Schwarz, R. M., Davidson, A., Carlson, P., & McKinney, S. (2011). The skilled facilitator fieldbook: Tips,
tools, and tested methods for consultants, facilitators, managers, trainers, and coaches. John Wiley
& Sons.
Skinner, R., Nelson, R. R., Chin, W. W., & Land, L. (2015). The Delphi Method Research Strategy in
Studies of Information Systems. Communications of the Association for Information Systems, 37, 2.
Someh, I., Davern, M., Breidbach, C. F., & Shanks, G. (2019). Ethical issues in big data analytics: A
stakeholder perspective. Communications of the Association for Information Systems, 44(1), 34.
Stahl, B. C. (2012). Responsible research and innovation in information systems, European Journal of
Information Systems, 21, 207211.
Stahl, B. C., Eden, G., Jirotka, M., & Coeckelbergh, M. (2014). From computer ethics to responsible
research and innovation in ICT: The transition of reference discourses informing ethics-related
research in information systems. Information & Management, 51(6), 810-818.
Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation.
Research Policy, 42(9), 1568-1580.
Communications of the Association for Information Systems
Accepted Manuscript
Teasdale, S. (2012). What’s in a name? Making sense of social enterprise discourses. Public policy and
administration, 27(2), 99-119.
Turel, O., Serenko, A., & Bontis, N. (2011). Family and work-related consequences of addiction to
organizational pervasive technologies. Information & Management, 48(2-3), 88-95.
Turoff, M. (1970). The design of a policy Delphi. Technological forecasting and social change, 2(2), 149-
Van Bouwel, J., & Van Oudheusden, M. (2017). Participation beyond consensus? Technology
assessments, consensus conferences and democratic modulation. Social Epistemology, 31(6),
Von Schomberg, R. (2013). A vision of responsible research and innovation. Responsible innovation:
Managing the responsible emergence of science and innovation in society, 51-74.
Von Schomberg, R. (2012). Prospects for technology assessment in a framework of responsible research
and innovation. In Technikfolgen abschätzen lehren (pp. 39-61). VS Verlag für
Walsham, G. (2012). Are We Making a Better World with ICTs? Reflections on a Future Agenda for the IS
Field, Journal of Information Technology 27, 2, 87-93.
Wickson, F., & Carew, A. L. (2014). Quality criteria and indicators for responsible research and innovation:
Learning from transdisciplinarity. Journal of Responsible Innovation, 1(3), 254-273.
Wilsdon, J., Wynne, B., & Stilgoe, J. (2005). The public value of science. London: Demos.
Wright, G., Cairns, G., & Goodwin, P. (2009). Teaching scenario planning: lessons from practice in
academe and business. European journal of operational research, 194(1), 323-335.
Zuboff, S. (2015). Big other: surveillance capitalism and the prospects of an information civilization.
Journal of Information Technology, 30(1), 75-89.
Error! Reference source not found.
Accepted Manuscript
Appendix A: List of Countries Involved in Project
Figure A1. Countries Involved in Vision Depiction Project
Countries Involved (In alphabetical order):
Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Greece,
Germany, Hungary, Italy, Ireland, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland,
Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom.
Communications of the Association for Information Systems
Accepted Manuscript
Appendix B: Participants in the Irish Face-to-face Consultation
Table B1. List of Participants at Face-to-Face Consultation
Participant 1
Member of European Parliament
Participant 2
H2020 National Contact Point
Participant 3
Participant 4
Healthcare Worker
Participant 5
Senior Lecturer
Participant 6
Operations Manager
Participant 7
Participant 8
Participant 9
Senior Lecturer
Participant 10
Participant 11
Participant 12
IT Director
Participant 13
Medical Representative
Participant 14
Senior Postdoctoral Fellow
Participant 15
Clinical Research Centre Manager
Participant 16
IT/EU Projects Manager
Participant 17
Project Administrator
Participant 18
PhD Research Student
Participant 19
Non-executive chairman
Participant 20
Participant 21
Masters student
Participant 22
Human performance
Participant 23
Participant 24
Account Manager
Participant 25
Retired telecoms technician
Participant 26
Participant 27
Participant 28
Private Teacher
Participant 29
Participant 30
Adult Literacy Organiser
Participant 31
Participant 32
Participant 33
Participant 34
Error! Reference source not found.
Accepted Manuscript
Appendix C: Schedule of the Face-to-Face Consultation
Table C1. Schedule of Face-to-Face Consultation
Time and Activity
9.00 - 9.30 Welcoming participants
9.30 - 9.40 Presentation of the consultation objectives and expected outcomes
9.50 - 10.10 Get to know each other: ice breaker
10.10-10.30 Overview of research scenarios
10.30 - 11.00 Coffee break
11.00 - 12.45 Discussion round 1 & 2
12.45 - 13.45 Lunch break
13.45 - 14.15 Discussion round 3
14.15 - 14.50 Summary of the results
14.50 - 15.45 Finalization of the template
15.45 - 16.00 Coffee break
16.00 - 16.30 Exhibition of the 5 enriched research topics and then prioritization
16.30 - 17.00 Closing session: feedback on the day
Communications of the Association for Information Systems
Accepted Manuscript
Appendix D: Discussion Template
Table D1. Discussion Template
Error! Reference source not found.
Accepted Manuscript
About the Authors
Dr. Stephen McCarthy is a lecturer and researcher in the Department of Business Information Systems,
University College Cork (UCC) and is a co-director of the MSc Business Information and Analytics
Systems postgraduate programme in UCC. He holds a Ph.D. and MBS in Information Systems (IS), and
Bachelor of Commerce (Management and Information Systems) degree in the National University of
Ireland. Stephen's research centers on five areas: (i) knowledge integration in distributed teams, (ii)
artefacts for participatory design, (iii) Health IT use across contexts, (iv) IT ethics and responsible IS
research, and (v) practice-oriented research. He has published in leading outlets within the IS and
management field, including: Computers in Human Behavior, the International Conference on Information
Systems, Journal of Total Quality Management and Business Excellence, European Conference on
Information Systems, the Hawaii International Conference on Systems Science.
Dr. Wendy Rowan is a Postdoctoral Researcher in Business Information Systems at Cork University
Business School, University College Cork, Ireland. Her research focuses on the use of Information
Systems in ethical practices and health domains. Currently, she works within the Ethical Research Centre
(EITIC) and the Health Information Systems Research Centre (HIRSC) at UCC. She has several
publications and conference papers in the area of Information Systems.
Laura Lynch is a Research Support Officer, Project Manager in the Health Information Systems
Research Centre (HISRC), in Business Information Systems at University College Cork. Her research to
date has focused on the use of information systems in the healthcare industry. She has published and
presented at a number of conferences in the area of Health and Information Systems.
Dr. Ciara Fitzgerald is a faculty member at the Cork University Business School. Her research and
teaching interests lie at the intersection of technology innovation, entrepreneurship and strategy. Her
research investigates (1) strategies used by universities and firms to manage intellectual property and the
commercialisation process, (2) strategies to engage citizens and policymakers in technology assessment,
and (3) exploratory and applied research of innovative health information systems. Ciara has published
widely in leading journals and influential volumes such as Research Policy, Journal of Technology
Transfer, Edward Elgar, Routledge and Cambridge University Press. She is Co-Director of the Health
Information Systems Research Centre in the Department of Business Information Systems in UCC.
Copyright © 2020 by the Association for Information Systems. Permission to make digital or hard copies of
all or part of this work for personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear this notice and full citation on
the first page. Copyright for components of this work owned by others than the Association for Information
Systems must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
servers, or to redistribute to lists requires prior specific permission and/or fee. Request permission to
publish from: AIS Administrative Office, P.O. Box 2712 Atlanta, GA, 30301-2712 Attn: Reprints or via e-
mail from
... Design Science Research (DSR) targets to solve real-world problems in business and society. From a societal perspective, engaging with citizens to understand problems and provide solutions is critical and constitutes a prerequisite for responsible research and innovation in IS and beyond [1,2]. Citizen Science, "the (large-scale) involvement of citizens in scientific endeavors not only as participants but as co-researchers" [3 p. 273] is a prominent approach to foster extensive public participation in research projects with the aim to close the gap between scientific and public perspectives on real-world problems [3]. ...
... Identifying and formulating problems that matter requires substantial engagement with the stakeholders [9]. However, scholars rarely involve those affected by their research [1,10]. Approaches to leverage creativity, wisdom, and experience of the society for identifying relevant research problems are still scarce. ...
... Based on the existing DSR, Citizen Science, and crowdsourcing literature introduced above, we derive an initial set of design requirements (DR) for our participatory problem discovery system following a CDSR paradigm. As argued by [1] and [2] the identification of relevant real-world problems requires the involvement of those affected already in the initial problem identification phase which is crucial for the success of a DSR project [7]. We thus articulate the first and second design requirements as follows: DR1: The system should enable the cooperation between researchers and citizens to jointly explore the problem space of a DSR project. ...
Full-text available
To solve societal problems, it is essential to engage with actual problem owners in society in a scalable way. In this paper, we follow a Citizen Design Science Research (CDSR) paradigm proposing to more actively involve society in DSR projects. Specifically, we present, a prototypical system that supports participatory problem discovery by enabling the involvement of citizens in problem awareness in a scalable way. The system allows researchers and citizens to cooperate on the exploration of a given problem space, leveraging the creativity and wisdom of the crowd for identifying and describing relevant DSR problems. The evaluation of the prototype with 30 representative citizens points to the system’s strengths and opportunities related to its ease of use and the problem articulation feature but also reveals weaknesses and threats concerning the problem exploration features and issues about platform abuse.
... The identification of relevant real-world problems requires the involvement of those affected already in the initial problem identification phase [14]. In order to engage with different stakeholders we invited citizens and collected problems and challenges using the digital citizen science application MyResearchChallenge [8]. ...
Conference Paper
Full-text available
Design-oriented research typically involves some kind of research problem discovery activity in order to identify and understand the problem space. Researchers can apply different methods to explore the problem space, for instance, interviews or focus groups. However, these methods are time consuming and do not scale well. Especially when it comes to discovering socially relevant real-world problems they require access to the general public to reach domain experts that is often difficult to achieve for researchers. Citizen science offers a promising approach for research problem discovery by actively involving citizens into the scientific inquiry to access knowledge on a large scale. In this paper, we report on a participatory action following a digital citizen science approach by specifically exploring the topic "home office" and corresponding challenges along four different subtopics. We report on (1) our approach and process to involve citizens in the problem discovery phase, (2) the implementation of the process in the web-based digital citizen science application MyResearchChallenge to enable citizens to register, collect, discuss, and vote challenges, and (3) provide a summary on the collected challenges.
... Following our goal of designing a digitization pipeline for documents that contain both printed text and symbols/objects, we define design requirements (DR) for a corresponding prototype ( Fig. 1 -Step 2). Here, as argued by [27], it is crucial for the success of the DSR project to involve those affected by the practical problem that was characterized in the initial problem statement ( Fig. 1 -Step 1). To do so, we first interviewed employees of a practitioner to get practical requirements from the end-users firsthand. ...
Although digitization is advancing rapidly, a large amount of data processed by companies is in printed format. Technologies such as Optical Character Recognition (OCR) support the transformation of printed text into machine-readable content. However, OCR struggles when data on documents is highly unstructured and includes non-text objects. This, e.g., applies to documents such as medical prescriptions. Leveraging Design Science Research (DSR), we propose a flexible processing pipeline that can deal with character recognition on the one hand and object detection on the other hand. To do so, we derive Design Requirements (DR) in cooperation with a practitioner doing prescription billing in the healthcare domain. We then developed a prototype blueprint that is applicable to similar problem formulations. Overall, we contribute to research and practice in multiple ways. First, we provide evidence for selected OCR methods provided by previous research. Second, we design a machine-learning-based digitization pipeline for printed documents containing both text and non-text objects in the context of medical prescriptions. Third, we derive a nascent design pattern for this type of document digitization. These patterns are the foundation for further research and can support the development of innovative information systems leading to more efficient decision making and thus to economic resource usage.KeywordsDocument image analysisOptical character recognitionDigitizationMachine learningPreprocessingPostprocessing
... To address this gap in research, we adhere to design science research by following the methodology formulated by Peffers et al. (2007) and seek to design actionable guidance for sourcing and managing external data by using a rigorous research process. McCarthy et al. (2020) argue that the successful identification of relevant real-world problems in DSR relies on the engagement of stakeholders (i.e., practitioners) in all phases, starting with the initial problem identification phase. Our multi-year research project debuted in February 2020, when we formed an expert group with practitioners from nine high-profile companies to investigate the challenges related to external data sourcing and management. ...
Conference Paper
Full-text available
External data has become an indispensable pillar in state-of-the-art decision-making and value creation in an enterprise context. Despite the increasing motivation to use external data, information systems (IS) research still lacks an adequate data sourcing perspective. This study aims to address this gap by investigating the practical challenges in this emerging field and developing a reference process for sourcing and managing external data. To this end, we adopt a design science research approach leveraging collaboration with practitioners from nine high-profile companies. Our findings contribute to the scarce body of knowledge on data sourcing in IS by proposing explicit prescriptions in the form of a reference process for sourcing and managing external data.
Full-text available
Social media platforms are a pervasive technology that continue to define our modern world. While social media has no doubt brought many benefits to society in terms of connection and content sharing, numerous concerns remain for the governance of social media platforms going forward; including (but not limited to) the spread of misinformation, hate speech, and online surveillance. However, the voice of citizens and other non-experts is often missing from such conversations in information systems literature which has led to an alleged gap between research and the everyday life of citizens. Our qualitative study presents findings from 16 hours of online dialogue with 25 citizens on the dark sides of digitalisation and social media platform governance. The online dialogue was undertaken as part of a worldwide consultation project called “We, the Internet” which sought to provide citizens with a voice on a range of topics such as “Digitalisation and Me”, “My Data, Your Data, Our Data”, and “A Strong Digital Public Sphere”. Five phases of thematic analysis were undertaken by the authors to code the corpus of qualitative data. Building on the Theory of Communicative Action, we discuss three dialogical processes critical to citizen discourse: lifeworld reasoning, rationalisation, and moral action. Our findings point towards citizens’ perspectives of current and future issues associated with social media platform governance, including concerns around the multiplicity of digital identities, consent for vulnerable groups, and transparency in content moderation. We also reveal citizen’s rationalisation of the dilemmas faced in addressing these issues going forward including tensions such as digital accountability vs. data privacy, protection vs. inclusion, and algorithmic censorship vs. free speech. Based on outcomes from this dialogical process, moral actions in the form of policy recommendations are proposed by citizens, and for citizens. We find that tackling these dark sides of digitalisation is something too important to be left to ‘Big Tech’ and equally requires an understanding of citizens’ perspectives to ensure an informed and positive imprint for change.
Full-text available
This article provides a systematic review of existing research on problematic smartphone use (PSU) to guide other researchers in search of relevant studies, and to propose areas for future research. In total, 293 studies were analyzed leading to the development of an overview model in the field of PSU, presenting findings on demographic factors, explanations for smartphone use and why this use becomes problematic, consequences of PSU, and how such use can be corrected. In addition, we considered in which contexts, with which methods, and with which theoretical lenses this stream of research has been studied to date. Smartphone use is most often explained by the smartphone design, and users' emotional health and their ability to control smartphone use. Our review suggests that people who are young, female, and highly educated are more prone to PSU. Emotional health issues are the most frequently identified consequence of PSU. Strategies for correcting PSU fall into three categories: information-enhancing, capacity-enhancing, and behavior reinforcement strategies. The studies on PSU are most often conducted using quantitative surveys with university and college participants considering their personal smartphone use. Whereas a variety of theoretical frameworks have been adopted to investigate PSU, they are often related to identifying factors explaining use and problematic use, and more seldom to analyze the findings. A future research agenda for PSU is proposed consisting of seven key research questions which can be investigated by researchers going forward.
Full-text available
Big data analytics is a fast-evolving phenomenon shaped by interactions among individuals, organizations, and society. However, its ethical implications for these stakeholders remain empirically underexplored and not well understood. We present empirical findings from a Delphi study that identified, defined, and examined the key concepts that underlie ethical issues in big data analytics. We then analyze those concepts using stakeholder theory and discourse ethics and suggest ways to balance interactions between individuals, organizations, and society in order to promote the ethical use of big data analytics. Our findings inform practitioners and policymakers concerned with ethically using big data analytics and provide a basis for future research.
Full-text available
This paper reports on a workshop hosted at the Isenberg School of Management - University of Massachusetts Amherst in September 2018. The workshop title, “Giving Voice to the Voiceless: The Use of Digital Technologies by Marginalized Groups,” was organized to discuss how marginalized groups use digital technologies to raise their voices. At the workshop, a diverse group of scholars and Ph.D. students presented research projects and perspectives on the role of digital technologies on activist projects representing marginalized groups that have gained momentum in the last few years. The studies and viewpoints presented shed light on four areas in which IS research can expand our understanding of the use of digital technologies by marginalized groups to address societal challenges: 1) The Rise of Cyberactivism; 2) Resource Mobilization for Cyberactivism; 3) Cyberactivism by and with Marginalized Groups; and 4) Research Methods for Collaborations with Marginalized Groups.
Full-text available
Information and communication technology (ICT) continues to change business as we know it. As ICT further integrates into our daily lives, it creates more opportunities to both help and hinder fundamental social problems throughout the world. In response to these growing and urgent societal needs, the Association for Information Systems approved the Bright ICT Initiative to extend IS research beyond a focus on business to take on the broader challenges of an ICT-enabled bright society. We conducted a Delphi study to provide guidance on where bright ICTminded researchers might focus to produce their greatest impact. In this paper, we report on our findings. The Delphi panel comprised 182 globally distributed IS journal editors who participated in a three-round consensus-building process via the Internet. Our results provide a framework of eleven research priority areas and specific research topics for those engaged in future-oriented, socially conscious IS research.
Full-text available
In this inductive, qualitative study, we observe how Uber, a company often hailed as being the poster-child of the sharing economy facilitated through a digital platform may also at times represent and reinforce postcapitalist hyper-exploitation. Drawing on the motivations and lived experiences of 31 Uber drivers in Toronto, Canada, we provide insights into three groups of Uber drivers: (1) those that are driving part-time to earn extra money in conjunction with studying or doing other jobs, (2) those that are unemployed and for whom driving for Uber is the only source of income, and (3) professional drivers, who are trying to keep pace with the durable digital landscape and competitive marketplace. We emphasize the ways in which each driver group simultaneously acknowledges and rejects their own precarious employment by distancing techniques such as minimizing the risks and accentuating the advantages of the driver role. We relate these findings to a broader discussion about how driving for Uber fuels the traditional capitalist narrative that working hard and having a dream will lead to advancement, security and success. We conclude by discussing other alternative economies within the sharing economy.
Full-text available
While the concept of Responsible Innovation is increasingly common among researchers and policy makers, it is still unknown what it means in a business context. This study aims to identify which aspects of Responsible Innovation are conceptually similar and dissimilar from social- and sustainable innovation. Our conceptual analysis is based on literature reviews of responsible-, social-, and sustainable innovation. The insights obtained are used for conceptualising Responsible Innovation in a business context. The main conclusion is that Responsible Innovation differs from social- and sustainable innovation as it: (1) also considers possible detrimental implications of innovation, (2) includes a mechanism for responding to uncertainties associated with innovation and (3) achieves a democratic governance of the innovation. However, achieving the latter will not be realistic in a business context. The results of this study are relevant for researchers, managers and policy makers who are interested in responsible innovation in the business context.
This article presents a systematic review of the English-language empirical literature about citizen participation to identify the obstacles to its implementation and the most successful ways to address them. Three sets of variables seem to impact effectiveness: contextual factors, including information asymmetries and public officials’ attitude; organizational arrangements, including community representation criteria and process design; and process management patterns, including group dynamics and collaboration quality. Two recommendations stem from our analysis: internalize decisions in organizational procedures, and establish ongoing interactions between government bodies and their stakeholders. We conclude that half-hearted engagement is unlikely to lead to successful citizen participation.
In this article, we inquire into two contemporary participatory formats that seek to democratically intervene in scientific practice: the consensus conference and participatory technology assessment (pTA). We explain how these formats delegitimize conflict and disagreement by making a strong appeal to consensus. Based on our direct involvement in these formats and informed both by political philosophy and science and technology studies, we outline conceptions that contrast with the consensus ideal, including dissensus, disclosure, conflictual consensus and agonistic democracy. Drawing on the notion of meta-consensus and a distinction between four models of democracy (aggregative, deliberative, participatory and agonistic), we elaborate how a more positive valuation of conflict provides opportunities for mutual learning, the articulation of disagreement, and democratic modulation – three aspirations that are at the heart of most pTAs and consensus conferences. Disclosing the strengths and weaknesses of these different models is politically and epistemically useful, and should therefore be an integral part of the development of participation theory and process in science and technology.
At a time when the limits on human progress are ever-decreasing and the consequences of human actions have never been so critical to predict and manage, the responsible nature of innovations has acquired a whole new importance. Firms now need to find the appropriate balance between achieving positive growth and performance while integrating the concept of responsible innovation at the very core of innovation strategies and processes. Based on extensive academic research and illustrated by multiple case studies and examples, this book will help understand the challenges and issues of responsible innovation for leaders and managers. It will also provide a process for implementing responsible innovation into an organization while improving performance sustainably. © 2014 by World Scientific Publishing Co. Pte. Ltd. All rights reserved.