Conference PaperPDF Available

REVISITING THE PROBLEM OF THE PROBLEM - AN ONTOLOGY AND FRAMEWORK FOR PROBLEM ASSESSMENT IN IS RESEARCH

Authors:

Abstract and Figures

A comprehensive understanding of how to achieve relevance and practical impact with our work remains elusive within the information systems (IS) community. While we know that finding or constructing important research problems sets the bar for the potential impact that research can have, we know little about how to support research problem assessment and selection in practice. This paper address this gap by presenting the problem assessment framework (PAF) and outlining its application for the assessment, selection, and justification of important research problems. The PAF builds on the problem assessment ontology, which explicates the domain of problem assessment based on a synthesis of extant research. We have instantiated the PAF in the problem assessment canvas to make it more accessible. Altogether, we contribute three novel artifacts that support researchers looking to work on the most important research problems as the basis for more relevant and impactful IS research.
Content may be subject to copyright.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 1
REVISITING THE PROBLEM OF THE PROBLEM
AN ONTOLOGY AND FRAMEWORK FOR PROBLEM
ASSESSMENT IN IS RESEARCH
Research Paper
Herwix, Alexander, University of Cologne, Cologne, Germany,
herwix@wiso.uni-koeln.de
Amir Haj-Bolouri, Department of Informatics, University West, Sweden,
amir.haj-bolouri@hv.se
Abstract
A comprehensive understanding of how to achieve relevance and practical impact with our work re-
mains elusive within the information systems (IS) community. While we know that finding or con-
structing important research problems sets the bar for the potential impact that research can have, we
know little about how to support research problem assessment and selection in practice. This paper
address this gap by presenting the problem assessment framework (PAF) and outlining its application
for the assessment, selection, and justification of important research problems. The PAF builds on the
problem assessment ontology, which explicates the domain of problem assessment based on a synthe-
sis of extant research. We have instantiated the PAF in the problem assessment canvas to make it more
accessible. Altogether, we contribute three novel artifacts that support researchers looking to work on
the most important research problems as the basis for more relevant and impactful IS research.
Keywords: Relevance, Rigor, Problem Assessment, Problem Selection, Ontology, Design Science Re-
search.
1 Introduction
The quality, relevance and practical impact of information systems (IS) research has been a prime
concern for the IS research community since at least Peter Keen’s (1991) seminal call to focus on the
topic. Keen (1991) proposed that relevance and a focus on practical impact must drive IS research be-
cause purposive and impactful research is a desire that is shared by the majority of researchers that
identify with the IS research community. Thus, “until relevance is established, rigor is irrelevant”
(Keen, 1991, p. 27). Since Keen’s (1991) work, many debates have been undertaken to advance and
reflect upon the topic of relevance and practical impact in IS research (Applegate, 1999; Agarwal and
Lucas Jr, 2005; Desouza et al., 2006; Wiener et al., 2018). Yet, as recent contributions in major IS
journals demonstrate (e.g., Nunamaker, Briggs, Derrick and Schwabe, 2015; Nunamaker, Twyman,
Giboney and Briggs, 2017; Davison and BjørnAndersen, 2019; Gable, 2020; Mohajeri, Mesgari and
Lee, 2020; Pan and Pee, 2020), a comprehensive understanding of how to achieve relevance and prac-
tical impact remains elusive within the IS community.
Against this backdrop, this paper builds on and contributes to the ongoing debate about relevance and
practical impact in IS research, by zooming in on the role of research problems. It is generally agreed
that finding or constructing an important research problem (whether this happens in a planned or un-
planned manner) is a necessary precursor before any relevant and impactful research can take place
(Getzels, 1975; Weber, 2003; Van de Ven, 2007; Rai, 2017). Nevertheless, extant research has high-
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 2
lighted that there seems to be much confusion and little shared understanding on how to rigorously
assess or effectively justify the value and importance of research problems (e.g., Hassan, 2014; Rai,
2017; Tremblay, VanderMeer and Beck, 2018; Weber, 2003; Wiener et al., 2018; Winter and Butler,
2011; Moeini, Rahrovani and Chan, 2019; Gable, 2020).
For instance, Weber (2003) already reflected that finding deep and important research problems often
remains more of dark art rather than a systematic and rigorous endeavor. As recent investigations and
discussions highlighted, not much seems to have changed in this regard (e.g., Tremblay et al., 2018;
Wiener et al., 2018). Others, such as Winter and Butler (2011), highlight how important research prob-
lems can act as coordination mechanisms for impactful research programs but suggest that typical IS
research problems do not fully live up to this potential. Nunamaker et al. (2017) detail pragmatic guid-
ance on how this situation could be improved by focusing on systematic programs of high-impact re-
search but remain largely silent on what characterizes high-impact research problems in the first place.
Most recently, Moeini et al. (2019) synthesized extant research into a framework to assess potential
for practical relevance. Their work highlights the importance of research topic selection but also sug-
gests a need for a better evidencing of problem importance and relevance in practice. Gable (2020)
builds on these insights and forcefully advocates the value of being strategic in IS research. His pro-
posal emphasizes the need for an explicit discourse on the strategic direction of the IS field and, there-
by, acknowledges the importance of systematic problem assessment and selection.
Altogether, the given examples attest to the ongoing relevance of the problem of the problem: know-
ing that the selection of problems is important but not knowing enough about how to support the iden-
tification and selection of important problems. Thus, we heed prior calls to revisit the problem of the
problem by investigating the following research question:
How can we assess and select research problems as well as justify their importance and relevance
in a more rigorous way?
We answer the research question by presenting the problem assessment framework (PAF) and demon-
strate its applicability for the assessment, selection, and justification of important research problems.
The PAF is built on the problem assessment ontology (PAO), which explicates the domain of problem
assessment based on a synthesis of extant research. The PAF consists of four modules (sic., perspec-
tive, definition, barriers, importance) and nine related components that help to systematically charac-
terize, assess, and compare research problems. We have also instantiated the PAF in a proof-of-
concept visual inquiry tool (Avdiji, Elikan, Missonier and Pigneur, 2020), the so called problem as-
sessment canvas (PAC) to make the PAF more accessible and improve its ease of use. Altogether, we
contribute three novel artifacts that support researchers looking to work on the most important re-
search problems as the basis for more relevant and impactful IS research.
The rest of the paper is structured as follows. First, we outline related work that has informed our
work. Second, we explain our research approach. Third, we present the problem assessment frame-
work. Fourth, we demonstrate the use of the framework and articulate guidelines for the selection, as-
sessment, and justification of important research problems that can help to improve the relevance and
impact of IS research. Fifth, we discuss the implications and limitations of our work. Sixth, we con-
clude the paper with a short summary and outline of future research directions.
2 Related Work
In the following sections. we describe related work that has informed our enterprise. In general, we
build on insights and perspectives from three distinct research areas, namely, IS research, effective
altruism, and social-ecological systems research.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 3
2.1 Information Systems Research
As mentioned in the introduction, IS research has a long history of engaging with the question of rele-
vance and impact of IS research (e.g., Keen, 1991; Applegate, 1999; Agarwal and Lucas Jr, 2005;
Desouza et al., 2006; Wiener et al., 2018). However, even though it is widely acknowledged that find-
ing or constructing an important research problem sets the bar for the potential impact that research
can have (Getzels, 1975; Weber, 2003; Van de Ven, 2007; Rai, 2017), comparatively little IS research
has examined the topic of research problem assessment and selection.
In a laudable effort, Rai (2017) makes nine general suggestions to help with articulating research
questions that matter. While we generally agree with the suggestions offered, we observed that they
do not lend themselves to the assessment of the relative importance of research problems and, thus,
have limited value for the comparative appraisal of research problems that is needed when deciding
between different research options. Specifically, they do not help to answer the question of how im-
portant or relevant additional work on a research problem is expected to be. Thus, the work illustrates
the need for more systematic engagement with the topic of research problem selection.
Extending this line of inquiry, Gable (2020) advocated the value of being strategic in IS research and
having an explicit discourse on the strategic direction of the IS field. Thus, he goes beyond a focus on
research problem selection as a challenge for individual IS research (as implicit in Rai, 2017) and fol-
lows Moeini et al. (2019) in proposing a multilevel perspective that also includes the perspectives of
collectives such as systematic research programs and research communities. In particular, it is high-
lighted that systematic discourse and cumulative progress is needed to engage with research problems
that are larger in scale and, thus, provide more opportunity for impact (Winter and Butler, 2011).
However, as of yet it can be observed that the IS research community has not had a systematic focus
on addressing the most important and relevant research problems (Moeini et al., 2019) but has rather
been dependend on the preferences and expertise of individual researchers or research teams to initiate
such efforts (Nunamaker et al., 2017).
Based on our engagement with the IS community, we propose that an important determinant for this
situation is the fact that most discussion on the importance and relevance of research questions has
been based upon only implicit assumptions about what those terms actually mean. This leaves much
wiggle room for vague justifications of potential importance or relevance that are hard to evaluate be-
cause it is unclear what is actually claimed (e.g., Moeini et al., 2019). Moreover, because there is no
explicit and shared understanding of importance and relevance in the IS research community, there are
also no clear benchmarks that IS researchers or research teams could use to decide whether a specific
problem is worthwhile to investigate. As a famous quote (often attributed to Lord Kelvin) says: what
is not measured cannot be improved”.
A related effort in this space is the recent work by Maedche et al. (2019) who have put forth a frame-
work to conceptualize the problem space in design science research (DSR). While their framework
seems useful to help formalize and ground discussions about problematic situations, it remains agnos-
tic about the importance or relevance of problems. Thus, it cannot help to address the challenges out-
lined above.
2.2 Effective Altruism
The effective altruism (EA) community (MacAskill, 2015, 2019b) is an applied research community at
the intersection of philosophy and economics that engages with the question of how to do as much
good as possible given the resources that are available. Moreover, it aims to actively translate its re-
search insights into practice to improve the world. Thus, the vision of EA is in important ways similar
to the vision of the IS community that Peter Keen (1991) and others (e.g., Hevner, March, Park and
Ram, 2004; Walsham, 2012) have outlined.
One important cornerstone of EA is the development and use of rigorous methods and tools that help
to effectively prioritize the use of resources for the greatest altruistic benefit (MacAskill, 2015,
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 4
2019b). As part of this effort, EA is emphasizing the use of cost-effectiveness (CE) analyses (Garber
and Phelps, 1997) to inform the allocation of available resources (MacAskill, 2015). Specifically,
while CE analyses can be misleading if not interpreted carefully (Simcikas, 2019), they can help to
assess the gains in a dependent variable (e.g., healthy lifeyears or any other quantity) that are expected
to be gained by investing additional resources into a specific course of action (Garber and Phelps,
1997). Thus, they lend themselves to the assessment of a set of alternative options against a common
benchmark (i.e., the same dependent variable). This allows for a systematic and cumulative approach
to research as demonstrated by the field of development economics, which has been pioneering the use
of CE analyses to generate important insights about the relative effectiveness of particular courses of
actions (Olken, 2020).
Going beyond the traditional use of CE analyses in the retrospective evaluation of particular courses of
actions, the EA community has started to develop ways of applying CE analyses prospectively to es-
timate the expected value of engaging with problems in the first place (e.g., Cotton-Barratt, 2014;
Wiblin, 2017; Wiebe, 2019). For instance, Wiblin (2017) demonstrates how to use estimations of the
importance, tractability, and neglectedness of problems to compare them in terms of potential impact
of additional resource investments. Wiebe (2019) proposes an alternative model which only requires
the estimation of importance and tractability by integrating considerations of neglectedness into them.
The main intent behind the use of such methods is to leverage the full potential of the EA community
by facilitating the proactive coordination of its members around the worlds most pressing problems.
CE analysis is often quite useful for this case because, in a world of entangled complex systems (Liu
et al., 2007; Benbya, Nan, Tanriverdi and Yoo, 2020), huge (i.e., often multiple orders of magnitude)
differences in CE between the best and the worst opportunities for action are likely to arise, as regular-
ly observed in multiple domains of practice (e.g., Hattie, 2012; Ord, 2013). Thus, just having a general
idea of the CE of a problem can already help to prioritize the best opportunities for action. In sum, we
were inspired by the success of the EA community in applying prospective CE analyses and adapted
as well as improved existing approaches for application in the IS research community.
2.3 Social-Ecological Systems Research
Social-ecological systems (SES) research is an interdisciplinary field that is concerned with the ana-
lyzing, comparing, and diagnosing of complex SES (e.g., Ostrom, 2009; Binder, Hinkel, Bots and
Pahl-Wostl, 2013). SES are generally understood to be complex systems that are composed of multiple
natural and artificial subsystems at multiple levels analogous to living organisms that are composed of
cells, tissues, organs, etc. (Ostrom, 2009). Thus, a major challenge for SES research is the integration
of work from a variety of perspectives and disciplines into a cohesive and cumulative body of
knowledge.
A major progression in the field of SES research has been the development of the SES framework
(Ostrom, 2009, 2010; Cox, 2014; McGinnis and Ostrom, 2014), which provides a conceptual lens
through which SES can be comparatively studied. In particular, the SES framework organizes the
analysis of SES around the core concept of action situation (Ostrom, 2010; McGinnis and Ostrom,
2014). An action situation describes focal situations or interactions that are repeatedly occurring and
deemed interesting enough for study. Ostrom (2010) demonstrates how action situations can be for-
malized using the logic of game theory by specifying seven rule sets to describe the game dynamics
that are inherent in all interactions. This formalization has proven invaluable for the highly cumulative
and successful research program on sustainable goverance arrangements for common pool resources
(Ostrom, 2010). Although we do not explicitly use the action situation lens in our work as of yet, we
were inspired by it and have developed our framework around the core idea of problems being linked
to focal situations.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 5
3 Research Approach
This project follows other methodological contributions in the IS field (Gregor and Hevner, 2013) in
applying a design science research (DSR) approach to framework development. The onlinde appendix
to this paper (Herwix and Haj-Bolouri, 2021) provides a comprehensive description of the research
process as well as a design justification. Thus, for the description in this section we rely on Hevner et
al.’s (2004, p. 83) seven guidelines for DSR to outline our research approach.
First, our research produced several useful artifacts in the form of the PAO (a model), the PAF (a
model), and the PAC (an instantiation).
Second, the problem we addressed is relevant to the IS research community because it currently does
not have a systematic approach to assess research problems in terms of importance and relevance,
which is an important gap if cumulative research around the most pressing issues is to arise. Specifi-
cally, it stands to reason that without a systematic discourse about the research problems we should be
tackling, research efforts will not be systematically aimed at the most important and relevant topics,
thus, diminishing the potential impact IS research could have.
1
Third, the PAF has been formatively evaluated with expert feedback by senior IS scholars as well as
practitioners experienced in problem assessment and is currently undergoing additional testing in an
undergraduate university course. It is important to note that the PAF, PAO, and PAC should be con-
sidered as living artifacts that are meant to be used, extended, and adapted by the community to solve
emergent needs. Thus, the traditional guideline of conducting rigorous summative evaluations
(Venable, Pries-Heje and Baskerville, 2014) are less applicable to our case. Rather, it seems useful to
consider the take up of the artifacts by the IS research community as the best evaluation of their utility.
Fourth, we contribute three artifacts to the IS research community that, at the same time, exaptate ex-
isting knowledge on problem assessment to IS research as well as improve upon this existing
knowledge by making it more accessible to non-experts in problem assessment (Gregor and Hevner,
2013). In terms of Gregor and Hevner’s (2013) contribution types, they can be classified as level 1
(instantiation) and level 2 (nascent design theory) contributions.
Fifth, to ensure the rigor of our research approach, the development of the artifacts was informed by
the design theory for visual inquiry tools (Avdiji et al., 2020), which provided a set of design princi-
ples that acted as a point of reference for our design activities. Although this design theory has been
developed to help with the development of visual inquiry tools for strategic management problems in
business, we propose that it is still applicable for the case of problem assessment in research as there is
considerable overlap between strategic management problems in business and the problem of problem
assessment and selection in research. For instance, both problems are complex in nature, concern the
effective allocation of resources and often affect a team of people. Table 1 gives an overview of how
we addressed the main design principles that Avdiji et al. (2020, p. 716) have articulated.
Sixth, our research approach followed an iterative search process. While developing the artifacts, we
constantly moved back and forth between the literature and our personal experience from the field un-
til we converged on a coherent design. For instance, the results presented in this paper are informed by
several presentations of earlier versions of this work at research seminars, workshops, and a confer-
ence.
1
We do not wish to imply that all IS research should necesarrily follow a systematic or mechanistic approach to optimize
problem selection but highlight that without a systematic discourse around what the most important and relevant topics are,
we are obviously leaving potential impact on the table as it is highly unlikely that everyone would intuitively work out where
to invest their resources to optimize for impact.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 6
Seventh, we are using a variety of channels to communicate our research, for instance, by using it in
teaching, presentating it at academic workshops and conferences as well as writing papers for academ-
ic and practitioners audiences. The PAC is licensend under a creative commons license and will be
made available on the web to encourage use and adaptation. Professional development workshops to
train IS researchers in the use of the artifacts are also being considered.
Design Principles
Implementation
DP1 Conceptual model: To structure the
[…] problem, frame it with a conceptual
model describing the relevant building
blocks (components) of the problem. The
conceptual framework should be modeled
according to academic justificatory
knowledge and be kept parsimonious.
We developed the PAO to frame the problem situation with a con-
ceptual model. The PAO is informed by the practices of the ap-
plied research field called Effective Altruism (MacAskill, 2015,
2019b) and the Social-Ecological Systems (SES) framework
(Binder et al., 2013; McGinnis and Ostrom, 2014).
DP2 Shared visualization: To facilitate
communication between users, represent
the conceptual model as a shared visuali-
zation by structuring the components logi-
cally into a visual problem space.
The PAC provides a visualization of the PAF that can facilitate
the communication between users. For instance, it provides nine
empty problem spaces for each of the nine properties in the PAO.
The properties are logically grouped and related in four modules
that are visually highlighted. Textual prompts help to orient and
direct users while filling out the canvas.
DP3 Directions for use: Define and spec-
ify techniques that allow for joint inquiry.
This paper illustrates several use cases for the framework and how
to apply it. Regarding the PAC, short prompts for each property
provide instructions for how to think about the problem space.
Table 1. Overview of how this work implemented the main design principles articulated by
Avdiji et al. (2020, p. 716).
4 The Problem Assessment Framework
The main goal of the problem assessment framework (PAF) is to provide a more rigorous foundation
for the systematic assessment of research problems that is conducive to more relevant and impactful
research. The PAF is grounded in the problem assessment ontology (PAO) which synthesizes extant
research into a coherent perspective. Figure 1 shows how the PAO breaks up the domain of problem
assessment into four entities (white boxes), three major relationships (labeled arrows), and nine prop-
erties (white ellipses). A description and justification for each element can be found in the online ap-
pendix to this paper (Herwix and Haj-Bolouri, 2021).
Reading from left to right, the PAO proposes that actor entities can be affected by a problem entity,
which describes in some way unsatisfactory situations and frames potential solution entities. The PAF
then uses four modules (shaded boxes) to organize the domain of problem assessment and unpack the
mentioned entities and high-level relationships in more detail:
A module that explicates the perspective that is taken when assessing the problem.
A module that encourages consistent definitions of problems.
A module that considers barriers, which could limit the impact of solutions.
A module that assess the overall importance of a problem from a given perspective.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 7
Figure 1. The problem assessment ontology describes the most relevant features that a systemat-
ic problem assessment should consider. The problem assessment framework groups
those elements into four distinct but related modules.
4.1 Module 1: Perspective on the Problem
The first module describes that problems are always viewed from a specific perspective because prob-
lems are relational constructs that are defined in relation to who they affect (Maedche et al., 2019). A
problem which will never affect anyone is simply not a problem. Moreover, different actors have dif-
ferent roles and priorities in the world and are, thus, likely to evaluate problems differently (Maedche
et al., 2019). Thus, an important challenge in this regard is to find a way to characterize actors that is
mindful of different roles and priorities but still allows for the recognition of regularities across con-
texts and time.
We propose that institutional logics (Seidel and Berente, 2013) provide a suitable abstraction for char-
acterizing the perspectives of actors. In particular, institutional logics describe the ensemble of core
goals, values, and prescriptions that are associated with general roles in organizations and society
(Berente and Yoo, 2012). Thus, they represent an intersubjective or societal perspective on appropriate
goals, values, and behaviors for an actor in a given role (Berente and Yoo, 2012). Importantly, actors
are often exposed to a multiplicity of sometimes conflicting institutional logics that they are ex-
pected to draw upon and reconciliate as part of their practice (Berente and Yoo, 2012). In sum, using
institutional logics can help us to aggregate micro-level behavior into higher-level constructs that are
applicable across a wider range of contexts (Seidel and Berente, 2013). Moreover, taking the perspec-
tives of a variety of institutional logics can help to uncover conflicting institutional logics as sources of
problems or barriers to solutions.
For instance, when assessing IS research problems, academics and practitioners are generally
acknowledged as relevant stakeholder groups (Keen, 1991). Both can be described with different insti-
tutional logics. First, we propose that the institutional logic of an academic highlights knowledge as a
core value that should be optimized for (Klingbeil, Semrau, Ebers and Wilhelm, 2019). From this per-
spective problems become more relevant the more they pertain to the academic discourse and state of
Modle 1: Pepecie
Aco
Aco
Modle 3: Baie
Modle 4: Impoance
Modle 2: Deniion
affec PoblemAco
Iniional
Logic
Scale
Impoance ifm Baie
Tacabili
Tajeco
Limiing
Faco
Rik
Solion
Deniion
Cone
Siaion
Bonda
Solion
Siaion
Siaion
Siaion
afm
Solion
decibe
fame
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 8
the art. Second, we propose that the institutional logic of a practitioner highlights satisfaction with
performance in their respective domain of expertise as a core value that should be optimized for
(Simon, 1955; Berente and Yoo, 2012). From this perspective problems become more relevant the
more they affect the state of the art of the profession of the practitioner. These institutional logics can
be complementary if, for instance, a practitioner values academic knowledge as a contributor to her
excellence. They can even be held simultaneously if an academic understands herself to be a practi-
tioner or a practitioner takes on the role of an academic (e.g., as part of a PhD program). However,
they can also be in conflict if practitioners do not perceive the academic knowledge pursued by aca-
demics to be valuable to their practice (Rosemann and Vessey, 2008).
Furthermore, going beyond the traditional confines of IS research being focused on academics and
practitioners, it is increasingly recognized that additional perspectives such as those of affected indi-
viduals or societal institutions ought to be considered (Davison and BjørnAndersen, 2019). Institu-
tional logics are a flexible framework that can be easily extended to accommodate multiple perspec-
tives even across multiple levels of abstraction (Seidel and Berente, 2013). Thus, as described in Table
2, in addition to academic and practitioner, we suggest six additional institutional logics that we be-
lieve provide a comprehensive yet parsimonious overview of perspectives which could be considered
for a holistic problem assessment. This list is derived from iterative discussion and reflection between
the authors and should not be viewed as complete. It merely aims to provide a thought provoking start-
ing point for the deliberate engagement with the question of what actually makes a problem matter.
Institutional
Logic
Description
Academic
A generic academic is generally interested in advancing
knowledge.
Practitioner
A generic practitioner is generally interested in achieving satis-
faction with her performance.
Individual
A generic individual is generally interested in personal wellbeing.
Organization
A generic organization is generally interested in realizing value
for its members.
Business
A generic business is generally interested in success within a
competitive environment.
Government
A generic government is generally interested in managing and
increasing societal welfare.
Civil Society
A generic civil society organization is generally interested in in-
creasing justice in relation to a particular cause.
Future People
Future people are generally interested in the long-term sustaina-
bility of the human project.
Table 2. Proposed set of eight institutional logics for a holistic problem assessment.
4.2 Module 2: Definition of the Problem
The second module captures the definition of a problem through three complementary components,
namely, context, situation, and boundary. Altogether they allow for the succinct definition of a prob-
lem and provide an anchor for the cumulative and comparative assessment of problems.
First, context clarifies important aspects of the environment of the problem (e.g., Davison and
Martinsons, 2016). In particular, Table 3 details four general scopes of environment that we suggest
provide a holistic but concise first order classification for problem contexts. As presented in more de-
tail elsewhere (Herwix and Haj-Bolouri, 2020), this classification organizes problems in terms of the
scope of the environment that they relate to, ranging from the universal to the local environment. In
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 9
general, the higher up a problem is in terms of scope, the more foundational the effect of solving it.
2
Thus, this classification allows for a quick but systematic positioning of problems in the “grand
scheme of things”.
Scope
Focus
Description
Universal
Value systems
The universal scope encompasses problems that engage with foundational
questions around the nature of the “goodness”, “badness” or “value” of ac-
tions and their outcomes.
Global
Global priorities
The global scope encompasses problems that transcend particular domain
scopes and help to identify, understand, and improve global priorities.
Domain
Problem classes
The domain scope encompasses problems that persist across a range of local
contexts.
Local
Instantiations
The local scope encompasses problems that are idiosyncratic to the local envi-
ronment.
Table 3. Four general scopes of environment to classify problem contexts.
Second, situation describes the problem situation and should detail the reason why it ought to be tack-
led from the perspective selected in module 1. Being clear about what the problematic situation is and
why it is important from the perspective of a given institutional logic is a precondition for an intersub-
jective assessment. Specifically, it should be possible to work on problem assessments collaboratively
(Avdiji et al., 2020) as well as evaluate them through peer-review (Hecht et al., 2018). Theoretical
lenses such as Maedche et al.s (2019) conceptualization of the problem space or the analytic structure
of the action situation from the SES framework (Ostrom, 2010) could be used to support a systematic
description.
Third, boundary intends to further refine the problem definition through the explicit documentation of
what is not considered to be part of the problem. The specification of clear boundaries has been recog-
nized as a necessary component of rigorous academic work in IS research (Weber, 2012).
4.3 Module 3: Barriers to Solutions for the Problem
The third module prepares the overall assessment of the importance of working on a problem with the
identification of potential barriers to solutions for the problem in the form of limiting factors and
risks. Such barriers are important determinants of the overall tractability of working on a problem and
can, thus, inform a subsequent estimation of tractability.
First, limiting factors identify those aspects of potential solutions that are least likely to scale to the
full extent of a problem. This helps to better understand the bottlenecks that solutions are most likely
to run into and prioritize our efforts (Savoie, 2019). Some general limiting factors that can be consid-
ered as starting points for this analysis are (Savoie, 2019): talent pool, political support, funding, room
for more funding (i.e., how much funding can be effectively absorbed by organizations working on the
problem), and logistical capacity (i.e., how many resources can be effectively used). For instance, on
the one hand, if the problem assessed is climate change” through green IS then academic incentives
seems to be a major limiting factor that blocks solutions to the problem much earlier than either talent,
room for more funding, and logistical capacity would require (Gholami et al., 2016). On the other
hand, if the problem assessed is “risks from emerging technologies” then the talent pool seems to be a
2
For instance, solving a problem in the local scope generally only affects the people who are directly involved in the situa-
tion, solving a problem in the domain scope can generalize to people in similar situations, solving a problem in the global
scope is likely to affect at least large parts of the world, and solving a problem in the universal scope might even radically
alter our very understanding of what is valuable.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 10
major limiting factor because time is needed to develop experts capable of effectively working on such
topics (80,000 Hours, 2020).
Second, risks describe the potential negative effects that could be triggered by working on solving a
problem. It is prudent to explicitly think about risk before steps are taken to work on a problem
(Jirotka et al., 2017). For instance, in the context of risks from emerging technologiesIS research
has started to highlight the potential dark sides and unintended consequences of novel IT use
(Tarafdar, Gupta and Turel, 2013). Becoming aware of such risks and potential opposition to further
research and IT development because of them is very helpful when thinking about the tractability of
problems and is deemed an important aspect of responsible research efforts (Jirotka et al., 2017).
4.4 Module 4: Importance of the Problem
The fourth module assess the overall importance of a problem from a given perspective with the help
of three components, namely, scale, tractability, and trajectory. This assessment of importance is in-
formed by the logic of CE analysis (Garber and Phelps, 1997) and the basic economic premise that one
should prefer investing into solving those problems that promise the best return on investment (ROI).
Specifically, we suggest to look at the scale, tractability, and trajectory of problems to make a rough
assessment of the ROI that could be expected from working on a problem (Wiblin, 2017; Halstead,
2019; Wiebe, 2019). Given problem scale measured in the form of utility gained / % of problem
solved and problem tractability in the form of % of problem solved / extra resources”, the CE of the
investment of extra resources can be calculated with the formula scale * tractability”, which simpli-
fies to “utility gained / extra resources” (Wiebe, 2019).
Moreover, assessing the trajectory of expected changes in scale and tractability (i.e., roughly how
much are scale and tractability going to increase or decrease in the future?) can help to interpret the
estimate of CE in the light of expectations about how the importance of a problem will develop over
time. This is important because the marginal CE of extra resources depends on the amount of re-
sources which are and will be invested into solving a problem in addition to the scale of the potential
benefits to be gained (Caspar, 2017). This means that CE analyses can only provide a snapshot of CE
at a specific point in time. Thus, in times of quickly changing environments CE estimations might
need to be updated in relatively short intervals. We propose that the simple approach of estimating the
problem scale, tractability, and trajectory can act as a quick and useful proxy for assessing the im-
portance of problems from a given perspective that is conducive to rapid iteration and feedback.
5 Application of the Framework
In the following sections, we describe the application of the PAF in the context of three different IS
research scenarios: problem assessment, problem selection, and problem justification.
5.1 Scenario 1: Problem Assessment
Figure 2 details how a basic application of the PAF works. In general, one would start with a problem
that is deemed interesting enough to require a more detailed assessment and work through all of the
modules of the PAF in turn. The first module is used to clarify the perspective that is taken for the as-
sessment of the problem. The second module is used to capture a consistent definition of the problem.
The third module is used to identify barriers, which could affect the tractability of working on the
problem. The fourth module is used to assess the importance of the problem from the chosen perspec-
tive. Altogether, these modules provide a systematic way to create a comprehensive problem assess-
ment for a specific perspective. To create a holistic assessment from multiple perspectives, multiple
applications of the framework can be carried out for the same problem.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 11
Figure 2. A summary view of how to apply the problem assessment framework.
Going beyond this underlying logic, we also instantiated the PAF in the form of the PAC (see the
online appendix; Herwix and Haj-Bolouri, 2021) to make it more intuitive and useful in collaborative
research team settings. In particular, the PAC aggregates all of the relevant information for a systemat-
ic problem assessment on one page or canvas, which makes it easier to work together in collaborative
problem assessment sessions. Moreover, the PAC provides instructions for how to carry out the as-
sessment and links to a supplementary online tool to help with the calculation of the expected CE of
working on a problem based on estimates of problem scale and tractability. Together these tools lend
themselves to a systematic and comprehensive, yet relatively quick assessment of problems. For in-
stance, a tentative problem assessment from one perspective facilitated by someone who is familiar
with the PAF should not take longer than about an hour. However, more thorough assessments could
require the conduct of additional data gathering (e.g., literature reviews) or even additional empirical
measurements. Thus, the PAC can be seen as a starting point for systematic and iterative investiga-
tions of the problem space.
5.2 Scenario 2: Problem Selection
As we have already highlighted in the introduction, the selection of important research problem is
generally a quite opaque and not well understood part of the IS research process (Weber, 2003; Winter
and Butler, 2011; Rai, 2017). Against this backdrop, the PAF affords a more rigorous approach to
problem selection that is built around the cumulative use of systematic problem assessments for the
benchmarking of problems in terms of importance. In general, two different modes of using the PAF
can be distinguished. On the one hand, it is possible to start with the identification of broad global pri-
orities and then, in a second iteration, look for important IS research problems within these broad
problem areas. We term this mode of using the PAF priorities-led problem selection. It could be par-
ticularly interesting for researchers that are willing to explore new areas and push the IS field in new
directions. On the other hand, it is possible to start by looking at important research problems within a
specific IS subcommunity and then, in a second iteration, look for important global priorities as appli-
cation areas that they can contribute to. We term this mode of using the PAF domain-led problem se-
lection. It could be particularly interesting for researchers that are looking to exploit existing IS re-
search expertise for maximum impact.
Both modes of using the PAF require an iterative application of the framework and the buildup of a
problem assessment portfolio (cf. Gable, 2020). While it is informative to assess a problem on its own,
looking at and comparing a variety of different options is necessary to identify the most important
problems. Specifically, given potentially vast differences in expected CE of working on problems
(e.g., Ord, 2013; Wiblin, 2017) investing at least some resources into building up an understanding of
the most important ones seems like a worthwhile investment. In practice this could be realized through
the publication of problem assessments or comparisons of problem assessments that are informed by
the PAF. This would also mean that not all IS researchers would need to apply the PAF themselves,
rather thoughtleaders within the field could use the PAF to inform their suggestions for new research
directions that could then be taken up by other researchers.
Perspective Denition Barriers ImportanceProblem
Institutional
Logic
Context
Situation
Boundary
Limiting
Factors
Risk
Scale
Tractability
Trajectory
Assessment
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 12
5.3 Scenario 3: Problem Justification
In addition to facilitating a more systematic and cumulative problem selection process, the PAF could
also be used to ground the justification of research problems. As Spindeldreher et al. (2020) illustrate,
research problem justification in IS is generally concentrated around business logic and neglects other
perspectives which could also reveal important problems. While we do not see the PAF as a normative
framework in the sense that it suggests to put certain perspectives over others, it does help with ap-
praising different perspectives in a systematic way. Specifically, it provides a common theoretical
foundation for the assessment of problems across perspectives that can encourage discourse ethics fo-
cused on the justification of IS research priorities (Mingers and Walsham, 2010). For instance, IS re-
searchers could use the PAF and the suggested perspectives within it as a starting point for the justifi-
cation of their work from multiple perspectives: How does a research problem relate not only to aca-
demics, practitioners and businesses but also to governments, civil society organizations or, maybe
most importantly, future people? How are potential value conflicts handled? The PAF can help to sys-
tematically develop answers to such questions and, thus, help to clearly justify the importance and rel-
evance of a research problem.
Beyond this, the PAF also encourages an empirical look at the importance and relevance of research
problems. As Moeini et al. (2019) highlights empirical evidence is thought to be conducive to a prac-
tice-oriented framing of research problems but not used to the extent that would be desireable. The
PAF provides a clear rationale and strategy for using empirical data to justify problem importance and
relevance that could help to improve the status quo.
6 Discussion and Limitations
We have presented the PAF and the underlying PAO as a more rigorous foundation for the systematic
assessment of research problems that is conducive to more relevant and impactful research. In particu-
lar, we outlined three scenarios in IS research where we expect the PAF to make significant contribu-
tions: problem assessment, problem selection, and problem justification.
In terms of problem assessment, extant research has highlighted that it is generally treated as an un-
structured process that is mostly guided by an intuitive understanding and perception of phenomena
and problems (Weber, 2003). Although broad recommendations for how to increase the importance
and relevance of problems exist (Davis, 1971; e.g., Alvesson, 2011; Winter and Butler, 2011; Rai,
2017), we are not aware of any framework that provides a theoretical foundation for the systematic
assessment of research problems from multiple perspectives. This is an important gap because it
means that the IS research community is unlikely to engage in a cumulative discourse about IS re-
search priorities that could systematically identify and promote the most important research problems.
The PAF provides the theoretical foundation necessary to close this gap. The feasibility of PAF sup-
ported problem assessment can be verified through the application of the PAC.
In terms of problem selection, we have argued that this can be supported systematically through the
iterative and cumulative application of the PAF. Currently, problem selection is mostly facilitated by
disconnected research agendas that do not attempt to enable (or even intentionally obscure) the com-
parative assessment of research problems in terms of importance or relevance. The paper by Moeini et
al. (2019) provides an interesting illustration of this dilemma. They develop a framework of potential
practical relevance of IS research and aim to demonstrate its usefulness by outlining future research
directions. However, all topics that they suggest for future study are simply listed without reference to
justification of their importancea practice that they criticized just a few pages before. This illustra-
tion is not meant to paint the work of Moeini et al. (2019) in a bad light but simply to highlight the
entranched nature of how little we, as a research community, currently support problem selection. Alt-
hough this state of affairs is very understandable from a pragmatic or evolutionary perspective, we
question its efficacy for leveraging the full potential of IS research. If the very best research problems
in terms of relevance and importance are likely to be few and far better than the average problem (e.g.,
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 13
Ord, 2013; Wiblin, 2017), it seems important and worthwhile to invest at least some resources into the
systematic comparison of problems and identification of priorities (cf. Gable, 2020). We propose that
the PAF provides the theoretical foundation necessary to facilitate this kind of discourse in a rigorous
and systematic way.
In terms of problem justification, we have argued that the PAF helps to clearly and rigorously justify
the importance and relevance of research problems from different perspectives. Currently, it often
seems like the justification of IS research is an afterthought that is only done in reference to market
logic and business values (Spindeldreher et al., 2020). Again, we question the efficacy of this state of
affairs for leveraging the full potential of IS research. While existing businesses are certainly im-
portant stakeholders in our world, recent developments in moral theory suggest that from an impartial
ethical perspective the needs of future generations and, thus, the sustainability of the human project
might be even much more important (e.g., Beckstead, 2013; Bostrom, 2013; MacAskill, 2019a). Thus,
it seems fruitful to encourage the exploration of new scripts that go beyond the standard focus on near-
term business value and embrace other perspectives such as those of future people (Gholami et al.,
2016; Seidel et al., 2017). The PAF provides a theoretical foundation to support and guide this process
toward the most important research problems. Encouragingly, given a trend toward earlier peer-review
on research projects through registered reports (Weinhardt, van der Aalst and Hinz, 2019; Doyle and
Luczak-Roesch, 2020) and recommendations to emphasize problem justification (Moeini et al., 2019)
the attention paid to the topic could grow in the future.
In terms of the limitations of our work, we note that the PAF is not a panacea for solving the issues
and realizing the opportunities outlined in this paper. While we would argue that a theoretical frame-
work for assessing problems such as the PAF is a necessary ingredient to start with the systematic
identification of the most relevant and important research problems, it is by no means sufficient by
itself. For the promise of more relevant research on important problems to be realized the IS research
community, or rather a substantial portion of individual IS researchers, needs to start engaging in sys-
tematic and rigorous discourse about research priorities. Although, we take first steps on this road, we
have only invested a small amount of resources into making it easy and desirable for people to walk
with us. The instantiation of the PAF in the form of the PAC is only a first attempt and proof-of-
concept in the much larger project of providing adequate support that empowers IS researchers to
make the most of their efforts. In particular, future research should more deeply investigate the utility
of the PAF by building on it or comparing it to alternatives.
7 Conclusion
In this paper, we have presented the PAF and the underlying PAO as a more rigorous foundation for
the systematic assessment of research problems that is conducive to more relevant and impactful re-
search. We have discussed its use in the IS research scenarios of problem assessment, problem selec-
tion, and problem justification and made the feasibility of the PAF easily verifiable by making the
PAC freely available and encouraging its use. Altogether, we make an important contribution by pre-
senting a set of artifacts that together provide a foundation for the development of more relevant IS
research through a focus on more important problems. As Winter and Butler (2011) have highlighted,
the IS research community has tremendous methods, capabilities and insights to offer, let us apply
them wisely.
Future research is encouraged to further improve on our work. For instance, currently a structured de-
scription of the problem situation is not suggested by the PAF despite potential options for doing so
because additional tooling would likely need to be developed to make the result accessible to re-
searchers. Moreover, currently the PAO only highlights the major relationships that are necessary for
the purposes of this paper. Future research is encouraged to further explore the domain of problem
assessment and to extend and refine the considerations offered in this paper.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 14
References
80,000 Hours. (2020). “Our list of high-impact careers.” Retrieved from
https://80000hours.org/career-reviews/
Agarwal, R. and H. C. Lucas Jr. (2005). “The information systems identity crisis: Focusing on high-
visibility and high-impact research.” MIS Quarterly, 381398.
Alvesson, M. (2011). “Generating Research Questions Through Problematization.Academy of Man-
agement Review, 36(2), 247271.
Applegate, L. M. (1999). “Rigor and Relevance in Mis Research -- Introduction.” MIS Quarterly,
23(1), 12.
Avdiji, H., D. Elikan, S. Missonier and Y. Pigneur. (2020). “A Design Theory for Visual Inquiry
Tools.” Journal of the Association for Information Systems, 22(2), 247265.
Beckstead, N. (2013). On the Overwhelming Importance of Shaping the Far Future (PhD). Rutgers,
The State University of New Jersey, New Brunswick, New Jersey, USA. Retrieved from
https://rucore.libraries.rutgers.edu/rutgers-lib/40469/pdf/1/
Benbya, H., N. Nan, H. Tanriverdi and Y. Yoo. (2020). “Complexity and Information Systems Re-
search in the Emerging Digital World.” MIS Quarterly, 44(1), 117.
Berente, N. and Y. Yoo. (2012). “Institutional Contradictions and Loose Coupling: Postimplementa-
tion of NASA’s Enterprise Information System.” Information Systems Research, 23(2), 376396.
Binder, C., J. Hinkel, P. Bots and C. Pahl-Wostl. (2013). “Comparison of Frameworks for Analyzing
Social-ecological Systems.” Ecology and Society, 18(4).
Bostrom, N. (2013). “Existential Risk Prevention as Global Priority.” Global Policy, 4(1), 1531.
Caspar. (2017, June 25). “Complications in evaluating neglectedness.” Retrieved from
https://casparoesterheld.com/2017/06/25/complications-in-evaluating-neglectedness/
Cotton-Barratt, O. (2014, December 4). “Estimating cost-effectiveness for problems of unknown diffi-
culty.” Retrieved from https://www.fhi.ox.ac.uk/estimating-cost-effectiveness/
Cox, M. (2014). “Understanding large social-ecological systems: introducing the SESMAD project.”
International Journal of the Commons, 8(2), 265.
Davis, M. S. (1971). “That’s Interesting: Towards a Phenomenology of Sociology and a Sociology of
Phenomenology.” Philosophy of the Social Sciences; Newbury Park, Calif., 1(4), 309344.
Davison, R. M. and N. Bjørn‐Andersen. (2019). “Do we care about the Societal Impact of our re-
search?” Information Systems Journal, 29(5), 989993.
Davison, R. M. and M. G. Martinsons. (2016). “Context is king! Considering particularism in research
design and reporting.” Journal of Information Technology, 31(3), 241249.
Desouza, K. C., O. A. El Sawy, R. D. Galliers, C. Loebbecke and R. T. Watson. (2006). “Beyond Ri-
gor and Relevance Towards Responsibility and Reverberation: Information Systems Research That
Really Matters.” Communications of the Association for Information Systems, 17, 226.
Doyle, C. and M. Luczak-Roesch. (2020). “This paper is an artefact: On open science practices in de-
sign science research using registered reports.” In: Proceedings of the 53rd Hawaii International
Conference on System Sciences.
Gable, G. G. (2020). “Viewpoint: Information systems research strategy.” The Journal of Strategic
Information Systems, 29(2), 101620.
Garber, A. M. and C. E. Phelps. (1997). “Economic foundations of cost-effectiveness analysis.” Jour-
nal of Health Economics, 16(1), 131.
Getzels, J. W. (1975). “Problem-Finding and the Inventiveness of Solutions.” The Journal of Creative
Behavior, 9(1), 1218.
Gholami, R., R. Watson, H. Hasan, A. Molla and N. Bjorn-Andersen. (2016). “Information Systems
Solutions for Environmental Sustainability: How Can We Do More?” Journal of the Association
for Information Systems, 17(8), 521536.
Gregor, S. and A. R. Hevner. (2013). “Positioning and Presenting Design Science Research for Maxi-
mum Impact.” MIS Quarterly, 32(2), 337355.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 15
Halstead, J. (2019). “The ITN framework, cost-effectiveness, and cause prioritisation.” EA Forum.
Retrieved from https://forum.effectivealtruism.org/posts/Eav7tedvX96Gk2uKE/the-itn-framework-
cost-effectiveness-and-cause
Hassan, N. (2014). “Value of IS Research: Is there a Crisis?” Communications of the Association for
Information Systems, 34(1), 801816.
Hattie, J. (2012). Visible Learning for Teachers: Maximizing Impact on Learning. Routledge.
Hecht, B., L. Wilcox, J. P. Bigham, J. Schöning, E. Hoque, J. Ernst, … C. Wu. (2018, March 29). “It’s
Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the
Peer Review Process.” Retrieved from https://acm-fca.org/2018/03/29/negativeimpacts/
Herwix, A. and A. Haj-Bolouri. (2020). “Having a positive impact with Design Science Research
Learning from Effective Altruism.” Presented at the 15th International Conference on Design Sci-
ence Research in Information Systems and Technology, Kristiansand, Norway.
Herwix, A. and A. Haj-Bolouri. (2021). “Revisiting the Problem of the Problem An Ontology and
Framework for Problem Assessment in IS Research (Online Appendix).” OSF. Retrieved from
https://osf.io/urdqb/
Hevner, A. R., S. T. March, J. Park and S. Ram. (2004). “Design Science In Information Systems Re-
search.” MIS Quarterly, 28(1), 75105.
Jirotka, M., B. Grimpe, B. Stahl, G. Eden and M. Hartswood. (2017). “Responsible research and inno-
vation in the digital age.” Communications of the ACM, 60(5), 6268.
Keen, P. G. (1991). “Relevance and rigor in information systems research: improving quality, confi-
dence, cohesion and impact.” In: H.-E. Nissen, H. K. Klein, & R. Hirschheim (Eds.), Information
systems research: Contemporary approaches and emergent traditions (Vol. 27, p. 49).
Klingbeil, C., T. Semrau, M. Ebers and H. Wilhelm. (2019). “Logics, Leaders, Lab Coats: A Multi-
Level Study on How Institutional Logics are Linked to Entrepreneurial Intentions in Academia.”
Journal of Management Studies, 56(5), 929965.
Liu, J., T. Dietz, S. R. Carpenter, M. Alberti, C. Folke, E. Moran, W. W. Taylor. (2007). “Com-
plexity of Coupled Human and Natural Systems.” Science, 317(5844), 15131516.
MacAskill, W. (2015). Doing good better: Effective altruism and a radical new way to make a differ-
ence. Guardian Faber Publishing.
MacAskill, W. (2019a). ““Longtermism.”” EA Forum. Retrieved from
https://forum.effectivealtruism.org/posts/qZyshHCNkjs3TvSem/longtermism
MacAskill, W. (2019b). “The Definition of Effective Altruism.” In: H. Greaves & T. Pummer (Eds.),
Effective Altruism Philosophical Issues. Oxford University Press.
Maedche, A., S. Gregor, S. Morana and J. Feine. (2019). “Conceptualization of the Problem Space in
Design Science Research.” Presented at the DESRIST 2019.
McGinnis, M. D. and E. Ostrom. (2014). “Social-ecological system framework: initial changes and
continuing challenges.” Ecology and Society, 19(2).
Mingers, J. and G. Walsham. (2010). “Toward ethical information systems: the contribution of dis-
course ethics.” MIS Quarterly, 34(4), 833854.
Moeini, M., Y. Rahrovani and Y. E. Chan. (2019). “A review of the practical relevance of IS strategy
scholarly research.” The Journal of Strategic Information Systems, 28(2), 196217.
Mohajeri, K., M. Mesgari and A. Lee. (2020). “When Statistical Significance Is Not Enough: Investi-
gating Relevance, Practical Significance, and Statistical Significance.” MIS Quarterly, 44(2), 525
559.
Nunamaker, J. F., R. O. Briggs, D. C. Derrick and G. Schwabe. (2015). “The Last Research Mile:
Achieving Both Rigor and Relevance in Information Systems Research.” Journal of Management
Information Systems, 32(3), 1047.
Nunamaker, J. F., N. W. Twyman, J. S. Giboney and R. O. Briggs. (2017). “Creating High-Value Re-
al-World Impact Through Systematic Programs of Research.” MIS Quarterly, 41(2), 335351.
Olken, B. A. (2020). “Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics*.”
The Scandinavian Journal of Economics, 122(3), 853878.
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 16
Ord, T. (2013). The Moral Imperative toward Cost-Effectiveness in Global Health (Center for Global
Development Essay). Retrieved from
https://www.cgdev.org/sites/default/files/1427016_file_moral_imperative_cost_effectiveness.pdf
Ostrom, E. (2009). “A General Framework for Analyzing Sustainability of Social-Ecological Sys-
tems.” Science, 325(5939), 419422.
Ostrom, E. (2010). “Beyond Markets and States: Polycentric Governance of Complex Economic Sys-
tems.” American Economic Review, 100(3), 641672.
Pan, S. L. and L. G. Pee. (2020). “Usable, in-use, and useful research: A 3U framework for demon-
strating practice impact.” Information Systems Journal, 30(2), 403426.
Rai, A. (2017). “Editor’s Comments: Avoiding Type III Errors: Formulating IS Research Problems
that Matter.” MIS Quarterly, 41(2), iiivii.
Rosemann, M. and I. Vessey. (2008). “Toward improving the relevance of information systems re-
search to practice: the role of applicability checks.” MIS Quarterly, 32(1), 122.
Savoie, J. (2019). “Why we look at the limiting factor instead of the problem scale.” Retrieved from
http://www.charityentrepreneurship.com/1/post/2019/01/why-we-look-at-the-limiting-factor-
instead-of-the-problem-scale.html
Seidel, S. and N. Berente. (2013). “Toward “Third Wave” Information Systems Research: Linking
Sociomaterial Practice with Broader Institutional Logics.” In: ICIS 2013 Proceedings (p. 15).
Seidel, S., P. Bharati, G. Fridgen, R. T. Watson, A. Albizri, M.-C. (Maric) Boudreau, … S. Watts.
(2017). “The Sustainability Imperative in Information Systems Research.” Communications of the
Association for Information Systems, 40, 4052.
Simcikas, S. (2019). “List of ways in which cost-effectiveness estimates can be misleading.” EA Fo-
rum. Retrieved from https://forum.effectivealtruism.org/posts/zdAst6ezi45cChRi6/list-of-ways-in-
which-cost-effectiveness-estimates-can-be
Simon, H. A. (1955). “A Behavioral Model of Rational Choice.” The Quarterly Journal of Economics,
69(1), 99.
Spindeldreher, K., D. Schlagwein and D. Schoder. (2020). “How is Information Systems Research
Justified? An Analysis of Justifications Given by Authors.” In: Proceedings of the 53rd Hawaii In-
ternational Conference on System Sciences (p. 10).
Tarafdar, M., A. Gupta and O. Turel. (2013). “The dark side of information technology use.” Infor-
mation Systems Journal, 23(3), 269275.
Tremblay, M., D. VanderMeer and R. Beck. (2018). “The Effects of the Quantification of Faculty
Productivity: Perspectives from the Design Science Research Community.” Communications of the
Association for Information Systems, 43(1).
Van de Ven, A. H. (2007). Engaged scholarship: a guide for organizational and social research. Ox-
ford ; New York: Oxford University Press.
Venable, J., J. Pries-Heje and R. Baskerville. (2014). “FEDS: a Framework for Evaluation in Design
Science Research.” European Journal of Information Systems, 25(1), 7789.
Walsham, G. (2012). “Are we making a better world with ICTs? Reflections on a future agenda for the
IS field.” Journal of Information Technology, 27(2), 8793.
Weber, R. (2003). “The Problem of the Problem.” MIS Quarterly, 27(1), iiiix.
Weber, R. (2012). “Evaluating and Developing Theories in the Information Systems Discipline.”
Journal of the Association for Information Systems, 13(1), 130.
Weinhardt, C., W. M. P. van der Aalst and O. Hinz. (2019). “Introducing Registered Reports to the
Information Systems Community.” Business & Information Systems Engineering, 61(4), 381384.
Wiblin, R. (2017). “How to compare different global problems in terms of impact?” Retrieved from
https://80000hours.org/articles/problem-framework/
Wiebe, M. (2019). “Formalizing the cause prioritization framework.” EA Forum. Retrieved from
https://forum.effectivealtruism.org/posts/fR55cjoph2wwiSk8R/formalizing-the-cause-
prioritization-framework
Wiener, M., C. Saunders, S. Chatterjee, A. Dennis, S. Gregor, M. hring and P. Mertens. (2018).
“Information Systems Research: Making an Impact in a Publish-or-Perish World.” Communica-
tions of the Association for Information Systems, 43(1).
Herwix and Haj-Boluri / The Problem of the Problem
Twenty-Ninth European Conference on Information Systems (ECIS 2021), Marrakesh, Morocco. 17
Winter, S. J. and B. S. Butler. (2011). “Creating Bigger Problems: Grand Challenges as Boundary Ob-
jects and the Legitimacy of the Information Systems Field.” Journal of Information Technology,
26(2), 99108.
... Alexander Herwix and Amir Haj-Bolouri strongly advocate for the centrality of ethical considerations for any type of research (Herwix & Haj-Bolouri, 2021) and DSR in particular (Herwix & Haj-Bolouri, 2020). At heart, all decisions that we are making are either implicitly or explicitly a reflection of our values (Ulrich, 2006). ...
... At heart, all decisions that we are making are either implicitly or explicitly a reflection of our values (Ulrich, 2006). As such, an understanding of ethics is paramount for the assessment of IS research and can help to improve its relevance and impact (Herwix & Haj-Bolouri, 2021. In particular, they argue that we need to be explicit about the values we consider to be the motivation of our work. ...
... Considering the impact that the choice of research problems has on the potential relevance of the IS field, we join others in advocating for a more systematic and serious engagement with problem choice, framing, and prioritization (e.g., Becker et al., 2015;Chen, 2011;Gable, 2020;Herwix & Haj-Bolouri, 2021Mertens & Barbian, 2015;Purao, 2021;Rai, 2017;Winter & Butler, 2011). Put simply, our work can only be as important as the problem which it is addressing. ...
Article
Full-text available
While ethics are recognized as an integral part of information systems (IS) research, many questions about the role of ethics in research practice remain unanswered. Our report responds to this emerging set of concerns with a broad and integrative account of five perspectives on ethics in IS research and design science research (DSR) in particular. Our report is informed by a broad literature review, a panel discussion at DESRIST 2020, and substantial personal experience from wrestling with ethical considerations in the field. The report provides a comprehensive discussion of prevailing perspectives on ethics and draws implications for IS research. Together, we hope the report will inspire more ethics-conscious and responsible IS research.
... We ground our research in the problem space defined according to (Herwix and Haj-Bolouri 2021). The scope of our targeted problem is global, because combating climate change and managing the respective changes in the energy system is a global matter. ...
Article
Full-text available
The transition from fossil fuels to renewable energy sources poses major challenges for balancing increasingly weather-dependent power supply and demand. Although demand-side energy flexibility, offered particularly by industrial companies, is seen as a promising and necessary approach to address these challenges and realize benefits for companies, its implementation is not yet common practice. Often facing highly complex process landscapes and operational systems, process mining provides significant potential to increase transparency of actual process flows and to discover or reflect existing dependencies and interrelationships of activities, instances or resources. It facilitates the implementation of energy flexibility measures and enables the realization of monetary benefits associated with flexible process operation. This paper contributes to the successful integration of energy flexibility into process operations by presenting a design science research artifact called PM4Flex. This is a prescriptive process monitoring approach that uses linear programming to generate recommendations for pending process flows optimized under fluctuating power prices by utilizing established energy flexibility measures. Thereby, event logs and corresponding company- as well as process-specific constraints are considered. PM4Flex is demonstrated and evaluated based on its implementation as a software prototype, its application to exemplary data from two real-world processes exhibiting power cost savings of up to 75% compared to the original execution, and based on semi-structured expert interviews. PM4Flex provides new design knowledge at the interface of prescriptive process monitoring and the energy domain providing decision support to optimize industrial energy procurement costs.
... As shown in the case evaluation, the components still require researchers to select more concrete techniques. Here, we see different integration potentials depending on the abstraction level of a tool: Our tool allows for combinations with more specific techniques in each component, such as for formalizing the problem (e.g., Herwix & Haj-Bolouri, 2021), grounding the design knowledge (e.g., Goldkuhl, 2004), formulating DSR research questions (e.g., Thuan et al., 2019), and evaluating the design principles (e.g., Iivari et al., 2020). Vice versa, our tool can serve as an input for more generic tools, such as the DSR Grid that captures "output knowledge" (vom Brocke & Maedche, 2019) or the Design Canvas that presents "design knowledge" as an outcome (Morana et al., 2018b). ...
Article
Full-text available
Particularly young researchers face challenges in organizing large design science research (DSR) projects and often struggle to capture, communicate, and reflect on important components to produce purposeful outcomes. Making informed decisions at the project start, such as selecting suitable kernel theories and development procedures, is of great relevance because they affect the entire design process and the resulting design products. Although DSR can produce different types of outcomes, from more situational artifacts to more abstract design knowledge, scholars point to the need for generalizing insights collected in such projects to advance the knowledge base. As design principles are among the prevailing forms of such design knowledge, this paper builds a visual inquiry tool—represented as a canvas—that navigates researchers through common components for crafting design principles and leverages collaborative reflections on essential project decisions. To build our canvas, we adapt inquiry-based learning (IBL) guidelines and visual inquiry tools to DSR education. Evaluations with doctoral students revealed promising indications for the canvas’s applicability and usefulness in guiding iterative DSR projects, reflecting on basic components, and communicating work-in-progress to other scholars and practice. Overall, we complement the body of DSR literature by providing an educational visual inquiry tool for producing design principles.
... We structure the remainder of the review as follows. Section 2 provides a description of the DSR project as a foundation for its interplay problem's importance as well as extant barriers for a problem solution (Herwix and Haj-Bolouri, 2021). Beyond the definition of the problem itself, there are two key design knowledge components that describe a project's problem space -the application context and the goodness criteria for solution acceptance. ...
Book
Researchers travel on paths of knowledge throughout life and the outcomes of rigorous scientific investigation result in contributions of new knowledge to the world. The Information Systems (IS) discipline is particularly suited for contributing to digital innovations and the corresponding knowledge growth. IS research develops not only knowledge in the form of understanding and designing digital technologies but also the implementation and use of actual socio-technical systems. In this review, the authors integrate the current thinking in the design science research (DSR) literature around the conceptual and methodological foundations of these high-level topics into a conceptual knowledge path framework. The authors position DSR at the intersection of science and technology where the interplay of descriptive and prescriptive knowledge is most active. They delineate the various forms of prescriptive design knowledge and examine the knowledge paths that utilize and produce the varied forms of knowledge in a DSR project. They define, analyze, and expand the ideas of knowledge gaps and journeys and argue that more attention to design postulates in DSR along the outlined knowledge paths can contribute to an increase in actionable and sustainable digital innovations within the IS discipline. By doing so, the authors aim to guide and inspire design-oriented IS researchers to actively and deliberately consider and incorporate a greater variety of existing knowledge into their designs, reflect even more thoroughly and systematically on their knowledge usage and contributions, and explicate and document these reflections in their publications.
... We structure the remainder of the review as follows. Section 2 provides a description of the DSR project as a foundation for its interplay problem's importance as well as extant barriers for a problem solution (Herwix and Haj-Bolouri, 2021). Beyond the definition of the problem itself, there are two key design knowledge components that describe a project's problem space -the application context and the goodness criteria for solution acceptance. ...
... In these context configurations, the goal is to improve the status quo by developing and evaluating a more useful problem representation that effectively corresponds to the underlying problem situations in question (i.e., the problem representation is insightful [40]). Again, this should ideally be done for each context scope to understand the problem in depth [41]. ...
Conference Paper
Full-text available
One of the open methodological concerns for design science research (DSR) in information systems is how to think about and deal with the notion of context. This paper takes an important step toward clarifying the notion of context and elaborates how it can be dealt with from a DSR perspective. In particular, we present a coherent theoretical account of context grounded in pragmatism. Moreover, we also reify this understanding into a context taxonomy and context framework for DSR. Altogether, we intend to provide a sound foundation and a fruitful platform for DSR that is more attuned to the particularities of context.
Chapter
Postpartum depression (PPD) for men is a significant but little-understood public health concern that affects ~14% of men in the US. It has not received adequate attention from society, researchers or health practitioners. This paper describes results from problem space exploration for this concern as the first step in a design science research process. Following the double-diamond model of design thinking, we describe two iterations. The first relies on qualitative analysis of data obtained from a social media platform to extract themes that describe pain points of new fathers. The second uses a participatory design exercise to identify personas and meta-requirements. Member-checking and triangulation efforts following the two iterations validate our findings that provide a rich understanding of this public health concern. A secondary contribution of our work is a demonstration of how design thinking techniques can be used within a design science research process to enhance the relevance cycle. We conclude by pointing to next steps for developing design science solutions in response to the problem.KeywordsProblem spaceTheory of the problemPostpartum depression among menDesign thinkingDouble diamond approach
Article
Full-text available
The Business Model Canvas project cleared the path for the development of a new tool type which we call visual inquiry tools. Such tools build on design thinking techniques to allow management practitioners to jointly inquire into specific strategic management problems. As the interest in and the emergence of visual inquiry tools gains momentum, it is important to formalize the design knowledge that future designers can build on to develop such tools. Thus, we propose a design theory for visual inquiry tools based on the design knowledge accumulated within and across three projects: the Business Model Canvas, the Value Proposition Canvas, and the Team Alignment Map. We outline the design principles (among others) that should be followed for developing visual inquiry tools for other strategic management problems. Our study addresses the lack of guidance in the development of visual inquiry tools and the lack of methodological guidance in design science research on how to theorize and formalize knowledge across multiple projects. We provide a methodological process for analyzing multiple-project data by bridging methodological insights from design science research and qualitative methods from the social sciences.
Article
Full-text available
In addition to innate curiosity, many of us also see scientific research as a way of making the world a better place. There has been a drive to better understand and observe the practical and societal impact of research, led by researchers seeking to find meaning and purpose in their work, as well as government agencies responsible for allocating research funding to maximum effect. Despite a wealth of guidance from researchers discussing impact and agencies evaluating impact, making practice impact visible and demonstrable remains arduous to researchers because it appears to be possible only at the end of a long and winding pathway to impact. This article presents a framework for demonstrating practice impact as it is being realized progressively, rather than only at the end of the pathway. It identifies usable, in-use, and useful research outputs, with each having cumulative and demonstrable practice impact. Our analyses of the guidelines of existing impact evaluations and top-ranked impact cases submitted to REF show that all three forms of impact can be demonstrated and are recognized as practice impact. Framing impact in terms of "use" inherently connects the perspectives of researchers and beneficiary users and positions users as co-producers of impact rather than passive objects and recipients of research. The 3U framework is descriptive as well as prescriptive. It identifies impact indicators for each form of impact. It also indicates the necessary actions for strengthening impact. When applied iteratively, the 3U framework facilitates the identification and pursuit new research questions that will further solidify a research endeavor's practice impact.
Article
Full-text available
Digitally induced complexity is all around us. Global digital infrastructure, social media, Internet of Things, robotic process automation, digital business platforms, algorithmic decision making, and other digitally enabled networks and ecosystems fuel complexity by fostering hyper-connections and mutual dependencies among human actors, organizations, structures, processes, and objects. Complexity affects human agencies and experiences in all dimensions including market and economic behaviors, political processes, entertainment, social interactions, etc. Individuals and organizations turn to digital technologies for exploiting emerging new opportunities in the digital world. They also turn to digitally-enabled solutions to cope with the wicked problems arising out of digitally-induced complexity. In the digital world, complexity and solutions based on digital technologies present new opportunities and challenges for information systems (IS) research. The purpose of this special issue is to foster the development of new IS theories on the causes, dynamics, and consequences of digitally-induced complexity in socio-technical systems. In this essay, we discuss the key concepts and methods of complexity science, and illustrate emerging new IS research challenges and opportunities in complex socio-technical systems. We also provide an overview of the five articles included in the special issue. These articles illustrate how IS researchers build on theories and methods from complexity science to study wicked problems arising out of digitally induced complexity. They also illustrate how IS researchers leverage the uniqueness of the IS context to generate new insights to contribute back to complexity science.
Conference Paper
Full-text available
This study analyses how Information Systems (IS) research is justified by authors. We assess how authors justify their research endeavors based on published IS research papers. We use justification theory [11], which along with later work, identifies seven different value systems (i.e., orders of worth) as co-existing in society, as a conceptual foundation. We qualitatively and quantitatively analyze the justifications in published IS research papers. We provide a breakdown of the justifications used in IS research. Our findings show that the importance and relevance of IS research is predominantly justified in reference to three orders of worth (market, industrial and civic values) at the neglect of the four other orders of worth (domestic, inspiration, fame, green) that equally exist in society. We provide suggestions to stimulate a broader consideration of research topics in relation to these other orders of worth and hence alternative sources of justification for authors.
Article
This article¹,² has two aligned aims: (i) to espouse the value of a strategic research orientation for the Information Systems Discipline; and (ii) to facilitate such a strategic orientation by recognising the value of programmatic research and promoting the publication of such work. It commences from the viewpoint that Information Systems (IS) research benefits from being strategic at every level, from individual researcher, to research program, to research discipline and beyond. It particularly advocates for more coordinated programs of research emphasising real-world impact, while recognising that vibrant, individual-driven and small-team research within broad areas of promise, is expected to continue forming the core of the IS research ecosystem. Thus, the overarching aim is the amplification of strategic thinking in IS research – the further leveraging of an orientation natural to the JSIS community, with emphasis on research programs as a main strategic lever, and further considering how JSIS can be instrumental in this aim.
Article
In 2019, Abhijit Banerjee, Esther Duflo, and Michael Kremer received the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel. These three scholars were recognized “for their experimental approach to alleviating global poverty”. This paper reviews the contributions of these three scholars in the field of development economics, to put this contribution in perspective. I highlight how the experimental approach helped to break down the challenges of understanding economic development into a number of component pieces, and I contrast this to understanding development using macroeconomic aggregates. I discuss pioneering contributions in understanding the challenges of education, service delivery, and credit markets in developing countries, as well as how the experimental approach has spread to virtually all aspects of development economics.
Chapter
The term “effective altruism” has no official definition, meaning that different authors will inevitably understand the term in different ways. Since this harbours the potential for considerable confusion, William MacAskill, one of the leaders of the effective altruism movement, has contributed a chapter aimed at forestalling some of these potential confusions. In this chapter, MacAskill first outlines a brief history of the effective altruism movement. He then proposes his preferred definition of “effective altruism”, aiming to capture the central activities and concerns of those most deeply involved in the movement. Finally, he replies to various common misconceptions about the movement. These include the views that effective altruism is just utilitarianism, that it is purely about poverty alleviation, that it is purely about donations, and that it in principle ignores possibilities for systemic change.
Chapter
In this chapter, Nick Beckstead argues that the best available interventions gain most of their expected value via the effects that they have on the long-run future, rather than via their more immediate effects. Because of the vastness of humanity’s possible future, this line of argument tends to favour actions that reduce risks of premature extinction, and actions that increase probabilities of other significantly beneficial “trajectory changes” to the course of humanity’s long-run future, even where the change in probabilities that we are able to bring about is very small.